TWI842416B - Crop harvester - Google Patents

Abstract

The present invention provides a crop harvester, comprising: a traveling device; a robotic arm device; at least one image-capturing device for capturing in farmland to obtain a two-dimensional image and a three-dimensional image corresponding to the two-dimensional image; and a computing device comprising: an image aligning module for aligning the three-dimensional image with the two-dimensional image; an object detecting module for detecting the two-dimensional image to mark at least one marking area corresponding to a crop in the two-dimensional image and obtaining a two-dimensional coordinate of the marking area; a coordinate analysis module for corresponding the marking area of the two-dimensional image to the three-dimensional image to obtain a depth coordinate of the marking area and then combine with the two-dimensional coordinate as a three-dimensional coordinate for clamping the crop; and a control module for controlling the robotic arm device to harvest the crop according to the three-dimensional coordinate.

Description

Crop harvester

The present invention relates to a crop harvester capable of automatically harvesting crops.

Asparagus cultivation requires a lot of manpower, and harvesting relies on a large number of people. However, in recent years, Taiwan's farmer labor force has gradually become short. In addition, since the harvestable asparagus stems are located near the ground, farmers need to bend and squat frequently to harvest, which will cause serious physical burdens and occupational injuries in the long run.

However, although there are currently asparagus harvesting aids to assist in harvesting asparagus, they are mainly for improving harvesting, and most of them are for improving the mechanism to reduce the labor time or improve the harvesting method, but still require manual identification, and cannot fully automatically identify and harvest asparagus stems. Therefore, it is hoped that the framework of smart agriculture can be introduced into the asparagus industry to improve the current asparagus harvesting situation, so as to achieve the effect of saving labor and stabilizing quality.

The main purpose of the present invention is to provide a crop harvester, comprising: a walking device for walking on a farmland planted with crops; a machine arm device, arranged on the walking device; at least one shooting device, arranged on the walking device, and used to shoot in the farmland to obtain a two-dimensional image and a three-dimensional image corresponding to the two-dimensional image; and a computing device, arranged on the walking device, and comprising: an image alignment module, used to align the two-dimensional image with the three-dimensional image; an object detection module, used to detect the A two-dimensional image is used to mark at least one marked area corresponding to the crop in the two-dimensional image to obtain the two-dimensional coordinates of the marked area; a coordinate analysis module is used to correspond the marked area of the two-dimensional image to the three-dimensional image aligned with the two-dimensional image to obtain the depth coordinates of the marked area from the three-dimensional image to combine the depth coordinates and the two-dimensional coordinates as the three-dimensional coordinates for clamping the crop; and a control module controls the robot arm device to move according to the three-dimensional coordinates to harvest the crop.

As in the aforementioned crop harvester, the coordinate analysis module uses the smallest depth information of the lower half of the marked area as the depth coordinate.

As in the aforementioned crop harvester, there are two camera devices, which are configured to shoot from different viewing angles to obtain two two-dimensional images at different viewing angles for the image alignment module and the object detection module to align and detect.

As in the aforementioned crop harvester, the computing device is an edge computing device, which is used to enable the shooting device to continuously shoot while the walking device is walking to form an image stream, which is then used by the object detection module for real-time detection.

As in the aforementioned crop harvester, the object detection module sends a stop signal to the walking device when it is able to mark the marking area.

As in the aforementioned crop harvester, after the machine arm device has harvested the crop, the control module sends a walking signal to the walking device to enable the shooting device to continuously shoot again to form an image stream, so that the object detection module can perform real-time detection again.

As in the aforementioned crop harvester, the photographing device is a depth camera.

As in the aforementioned crop harvester, the object detection module is an artificial intelligence model trained by YOLOv5x6.

As in the aforementioned crop harvester, the AI model is trained with multiple images containing the crop being obstructed or overlapped with other non-harvesting objects.

As in the aforementioned crop harvester, the artificial intelligence model is further trained by the multiple images of the crop labeled according to its height.

As in the aforementioned crop harvester, the crop corresponding to the marked area is asparagus stems.

As in the aforementioned crop harvester, the machine arm device has multi-axis degrees of freedom, and a clamping unit is provided at the end thereof, and the clamping unit includes a soft pad and a blade.

In summary, the crop harvester of the present invention can accurately and effectively identify the crops to be harvested, and automatically harvest them accurately accordingly, thereby improving the convenience and efficiency of asparagus harvesting and effectively achieving the purpose of labor-saving harvesting.

1:Crop harvester

10: Walking device

11: Track mechanism

20: Robot arm device

21: Vertical lifting mechanism

22: The first robot arm

23: Second robot arm

24: The third robot arm

25: Clamping unit

26: Soft padding

27: Blade

30: Filming equipment

40: Computing device

41: Image alignment module

42: Object detection module

43: Coordinate analysis module

44: Control module

50: Harvest box

B: Marking area

C: Crops

I ₂ : Two-dimensional image

I ₃ ,I ₃ ': 3D image

P: Depth coordinate

S11-S15: Steps

Figure 1 is a schematic diagram of the crop harvester of the present invention.

Figure 2 is a schematic diagram of the clamping unit of the present invention.

Figure 3 is a diagram of the architecture of the computing device of the present invention.

Figure 4(a) is a schematic diagram of the present invention before the two-dimensional image and the three-dimensional image are aligned.

Figure 4(b) is a schematic diagram of the image alignment module of the present invention after aligning the two-dimensional image and the three-dimensional image.

Figure 5 is a schematic diagram of the object detection module of the present invention marking the marked area after detecting a two-dimensional image.

Figure 6 is a flow chart of the operation of the crop harvester of the present invention.

The following is a specific embodiment to illustrate the implementation of the present invention. People familiar with this technology can easily understand the other advantages and effects of the present invention from the content disclosed in this manual, and can also implement or apply it through other different specific embodiments.

Please refer to Figure 1. The crop harvester 1 of the present invention includes a walking device 10, a machine arm device 20, at least one camera device 30, a computing device 40 and a harvesting box 50. The machine arm device 20, the camera device 30, the computing device 40 and the harvesting box 50 are arranged on the walking device 10. The walking device 10 can walk in the farmland where the crop C is planted, so that the camera device 30 can take pictures in the farmland, and the computing device 40 can analyze the images taken by the camera device 30 to control the machine arm device 20 to harvest the crop C into the harvesting box 50, thereby realizing fully automatic crop harvesting.

As shown in FIG1 , the walking device 10 includes a crawler mechanism 11, and the rotation speed of the crawler mechanism 11 is preferably 250 rpm, but not limited thereto. With the crawler mechanism 11, the crop harvester 1 can stably travel between fragmented and uneven fields without excessive bumps affecting the crop harvester 1's identification and harvesting of the crop C. In this embodiment, the walking device 10 is provided with an ultrasonic sensor (not shown), which can sense the surrounding environmental information, so that the walking device 10 automatically avoids obstacles to travel smoothly and steadily between farmlands. However, the present invention is not limited thereto, and the walking device 10 can also use other methods to sense the surrounding environmental information.

As shown in FIG1 , the robot arm device 20 includes a vertical lifting mechanism 21, a first robot arm 22, a second robot arm 23, a third robot arm 24, and a clamping unit 25. The vertical lifting mechanism 21 is rotatably arranged on the walking device 10, and can be horizontally rotated relative to the walking device 10. One end of the first robot arm 22 is connected to the vertical lifting mechanism 21, and can be vertically lifted relative to the vertical lifting mechanism 21. One end of the second robot arm 23 is connected to the other end of the first robot arm 22, and can be horizontally rotated relative to the first robot arm 22. One end of the third robot arm 24 is connected to the other end of the second robot arm 23, and can be horizontally rotated relative to the second robot arm 23. The clamping unit 25 is connected to the other end of the third robot arm 24, and can be vertically rotated relative to the third robot arm 24. With the above design, the robot arm device 20 has multi-axis degrees of freedom, and can plan different movement paths for crops C at different positions. In addition, the robot arm device 20 can cooperate with the movement of the walking device 10 to avoid obstacles such as branches and leaves, and accurately move the clamping unit 25 to the position of the crop C for harvesting. The robot arm device 20 of the present invention is not limited to the above structure, as long as it has a multi-axis degree of freedom.

As shown in FIG2 , the clamping unit 25 is a V-shaped clamp that can be opened and closed to clamp the crop C. A soft pad 26 and a blade 27 are provided on both sides of the V-shaped clamp of the clamping unit 25, wherein the blade 27 is provided at the lower end of the clamping unit 25 to cut the bottom of the crop C, and the soft pad 26 is provided above the blade 27. In this embodiment, the soft pad 26 can be a high-density sponge. When the two sides of the V-shaped clamp are close to each other so that the blade 27 cuts the crop C, the clamping unit 25 can clamp the crop C through the soft pad 26 and send it to the harvesting box 50 without damaging the crop C.

In the present embodiment, the photographing device 30 is a depth camera (e.g., Intel RealSense D435i), which has two lenses and can obtain a two-dimensional image I ₂ and a three-dimensional image I ₃ corresponding to the two-dimensional image I ₂ , as shown in FIG4(a). In the present embodiment, the depth camera obtains depth information by measuring distance using infrared rays. For example, the D435i has two sets of infrared receivers and can synthesize a depth image by obtaining two infrared parallax images, but the present invention is not limited thereto. In addition, there are two photographing devices 30, and the two photographing devices 30 can shoot from different viewing angles (in FIG1 , they are located at the front and rear sides of the robot arm device 20, respectively) to obtain two two-dimensional images I ₂ at different viewing angles for analysis by the computing device 40. Furthermore, the two-dimensional image I ₂ can be an RGB image with a resolution of 1920*1080 pixels, and the three-dimensional image I ₃ can be a depth image with a resolution of 1280*720 pixels.

Please refer to Figure 3, the computing device 40 includes an image alignment module 41, an object detection module 42, a coordinate analysis module 43 and a control module 44. In this embodiment, the computing device 40 is an edge computing device (Nvidia Jetson Xavier NX 16G). The shooting device 30 continuously shoots when the walking device 10 walks to form an image stream (Video stream), and the computing device 40 receives the image stream for real-time analysis.

As shown in FIG4(a), the two-dimensional image _I2 and the three-dimensional image _I3 taken by the shooting device 30 may not correspond to each other in size or position before calibration and alignment, because the resolutions of the RGB image and the depth image taken by the depth camera are different. As shown in FIG4(b), the image alignment module 41 aligns the three-dimensional image _I3 with the two-dimensional image _I2 and adjusts it to a three-dimensional image _I3 ' corresponding to the two-dimensional image _I2 , for example, adjusting the scale to be the same and performing conversion between different coordinate systems. Among them, the X and Y coordinates of the two-dimensional image _I2 are in pixels, while the X and Y coordinates of the three-dimensional image _I3 ' are in pixels, and the Z coordinate is in centimeters.

In this embodiment, coordinate conversion can be performed using the following formula.

為用以將世界座標系 轉換為相機座標系；

為用以將該相機座標系進一步轉 換為圖像座標系；

為用以將該圖像座標系再進一步轉換為像素座標系；

為表示相機內參數，而

為表示相機外參數。 Among them, O _W (X _W ,Y _W ,Z _W ) is the world coordinate system, whose origin is usually set to the robot base (i.e., the robot arm device), and its unit is millimeter (mm); O _C (X _C ,Y _C ,Z _C ) is the camera coordinate system, whose origin is the camera optical center, and its unit is millimeter; o(x,y) is the image coordinate system, and its unit is millimeter; and (u,v) is the pixel coordinate system, and its unit is pixel. In the above formula,

For converting the world coordinate system into the camera coordinate system;

for further converting the camera coordinate system into an image coordinate system;

For further converting the image coordinate system into a pixel coordinate system;

represents the camera internal parameters, and

To represent the camera external parameters.

As shown in FIG5 , the object detection module 42 can be used to detect the two-dimensional image I ₂ to mark at least one marked area B corresponding to the crop C in the two-dimensional image I ₂ and obtain the two-dimensional coordinates of the marked area B. When the object detection module 42 can mark the marked area B, it sends a stop signal to the walking device 10. After the walking device 10 stops, the robot arm device 20 can harvest the crop C. In this embodiment, the object detection module 42 is an artificial intelligence model trained by YOLOv5x6, but the present invention is not limited thereto. The object detection module 42 can also use other types of artificial intelligence models. The training method of the artificial intelligence model will be described in detail later.

The coordinate analysis module 43 can be used to correspond the marked area B of the two-dimensional image I ₂ to the three-dimensional image I ₃ ' aligned with the two-dimensional image I ₂ , so as to obtain the depth coordinate P of the marked area B from the three-dimensional image I ₃ ', and combine the depth coordinate P and the two-dimensional coordinate as the three-dimensional coordinate of the crop C. In this embodiment, the coordinate analysis module 43 uses the smallest depth information of the lower half of the marked area B (i.e., the object closest to the shooting module 30) as the depth coordinate P. As shown in FIG. 5, even if the asparagus stem is crooked, the coordinate analysis module 43 can correctly determine the position of the asparagus stem. In other embodiments, the coordinate analysis module 43 can also use the depth information of the center point of the bottom of the marked area B as the depth coordinate P, but the present invention is not limited thereto. In addition, the present invention is not limited to the above-mentioned determination method, and a detection module such as Yolact-Edge can also be used to enclose the edge of the asparagus and directly detect the bottom point in the image to avoid incorrect clamping position.

The control module 44 can control the robot arm device 20 to move according to the three-dimensional coordinates, so that the gripping unit 25 of the robot arm device 20 moves to the crop C to harvest the crop C and send the harvested crop C to the harvesting box 50 for collection. After the robot arm device 20 harvests the crop C, the control module 44 can send a walking signal to the walking device 10 and make the shooting device 30 shoot continuously again to form an image stream for the object detection module 42 to perform real-time detection again, until the object detection module 42 can mark the marking area B again, and then send a stop signal to the walking device 10 to stop harvesting.

In this embodiment, the crop C corresponding to the marked area B is asparagus stems, but the present invention is not limited to this.

The following further describes the training method of the artificial intelligence model. In this embodiment, YOLOv5x6 is used to train the artificial intelligence model. In the training of YOLOv5x6, asparagus stems can be actually photographed to obtain multiple images containing asparagus stems, and then a labeling software such as LabelImg is used to manually label the labeled area corresponding to the asparagus stems in the multiple images containing asparagus stems, and the multiple images containing asparagus stems and the corresponding labeled area are used as the training set of the artificial intelligence model, thereby training the artificial intelligence model to identify asparagus stems.

The multiple images for training YOLOv5x6 may include multiple images in which asparagus stems are covered by leaves or overlap with asparagus stems. This is closer to the actual application situation, so it can have a better recognition accuracy in actual application. In addition, when marking the multiple images for training YOLOv5x6, the asparagus stems can also be marked according to the height, and the asparagus stems of different harvesting levels can be further classified according to the height of the crop C, so that the object detection module 42 can mark the asparagus stems that meet the harvesting level.

In addition, in the training of YOLOv5x6, the hyperparameters that can be adjusted are shown in Table 1 below. The following parameters and their values are only examples, and the present invention is not limited to them.

Table 1

The performance verification results of the artificial intelligence model trained by the YOLOv5x6 are as follows. The artificial intelligence model was trained and performance verified using 2573 images containing asparagus stems, including 300 images of special cases such as asparagus stems being covered by leaves or overlapping with asparagus stems. The recognition accuracy of asparagus stems was about 96.6%, and the accuracy of the robot arm device 20 based on the three-dimensional coordinate positioning of asparagus stems was about 93.3%. The computing device 40 can calculate a single frame in the image stream in 0.3 seconds, which can effectively perform real-time detection.

Please refer to FIG. 6 , the operation flow of the crop harvester 1 of the present invention when it is actually applied is as follows. First, in step S11, the walking device 10 is first made to walk in the farmland where the crop C is planted, and at the same time, the shooting device 30 is made to continuously shoot while the walking device 10 is walking to form an image stream, and the computing device 40 receives the image stream for real-time analysis. Among them, the computing device 40 aligns the two-dimensional image I ₂ and the three-dimensional image I ₃ shot by the shooting device 30 through the image alignment module 41, and detects the two-dimensional image I ₂ through the object detection module 42 to determine whether the marking area B corresponding to the crop C can be marked in the two-dimensional image I ₂ . In step S12, when the object detection module 42 of the computing device 40 can mark the marked area B corresponding to the crop C, a stop signal is sent to the walking device 10. In step S13, the walking device 10 receives the stop signal, stops walking, and sends a signal back to the computing device 40.

In step S14, after the computing device 40 receives the signal that the walking device 10 has stopped walking, the coordinate analysis module 43 maps the marked area B of the two-dimensional image I ₂ to the three-dimensional image I ₃ ' aligned with the two-dimensional image I ₂ , so as to obtain the depth coordinate P of the marked area B from the three-dimensional image I ₃ ', and combines the depth coordinate P with the two-dimensional coordinate as the three-dimensional coordinate for gripping the crop C, and the control module 44 controls the robot arm device 20 to move the gripping unit 25 to the position of the crop C for harvesting based on the three-dimensional coordinate of the crop C analyzed by the coordinate analysis module 43. In step S15, after the robot arm device 20 finishes harvesting the crop C, it sends a walking signal to the walking device 10 and the computing device 40. Therefore, the process returns to step S11 and repeats the above steps, thereby achieving fully automatic crop harvesting.

In this embodiment, since the shooting device 30 continuously shoots and the computing device 40 continuously analyzes during the walking process of the walking device 10, compared with stopping to shoot after walking a fixed distance, it can avoid missing crops C in the middle of the fixed distance. In addition, when the computing device 40 finds the crop C, it immediately sends a stop signal to stop the walking device 10, so the position where the walking device 10 stops is the position of the crop C, and the problem of the walking device 10 stopping too far from the position of the crop C will not occur.

In summary, the crop harvester of the present invention can accurately and effectively identify the crops to be harvested, and automatically harvest them accurately accordingly, thereby improving the convenience and efficiency of asparagus harvesting and effectively achieving the purpose of labor-saving harvesting.

The above implementation forms are only illustrative of the technical principles, features and effects of the present invention, and are not intended to limit the scope of implementation of the present invention. Anyone familiar with this technology can modify and change the above implementation forms without violating the spirit and scope of the present invention. However, any equivalent modifications and changes completed by using the teachings of the present invention should still be covered by the following patent application scope. The scope of protection of the present invention should be as listed in the following patent application scope.

1:Crop harvester

10: Walking device

11: Track mechanism

20: Robot arm device

21: Vertical lifting mechanism

22: The first robot arm

23: Second robot arm

24: The third robot arm

25: Clamping unit

30: Filming equipment

40: Computing device

50: Harvest box

Claims (12)

A crop harvester includes: a walking device for walking on a farmland planted with crops; a machine arm device disposed on the walking device; at least one shooting device disposed on the walking device and used to shoot in the farmland to obtain a two-dimensional image and a three-dimensional image corresponding to the two-dimensional image; and a computing device disposed on the walking device and including: an image alignment module for aligning the two-dimensional image with the three-dimensional image; an object detection module for detecting the two-dimensional image, At least one marked area corresponding to the crop is marked in the two-dimensional image to obtain the two-dimensional coordinates of the marked area; a coordinate analysis module is used to correspond the marked area of the two-dimensional image to the three-dimensional image aligned with the two-dimensional image to obtain the depth coordinates of the marked area from the three-dimensional image, so as to combine the depth coordinates and the two-dimensional coordinates as the three-dimensional coordinates for clamping the crop; and a control module controls the robot arm device to move according to the three-dimensional coordinates to harvest the crop. A crop harvester as described in claim 1, wherein the coordinate analysis module uses the smallest depth information of the lower half of the marked area as the depth coordinate. The crop harvester as described in claim 1, wherein the number of the photographing devices is two, and the photographing devices are configured to be able to photograph from different viewing angles to obtain two two-dimensional images at different viewing angles for the image alignment module and the object detection module to align and detect. The crop harvester as described in claim 1, wherein the computing device is an edge computing device, which is used to enable the shooting device to continuously shoot when the walking device is walking to form an image stream, and then provide the object detection module with real-time detection. A crop harvester as described in claim 4, wherein the object detection module sends a stop signal to the walking device when the marking area is marked. The crop harvester as described in claim 5, wherein the control module sends a walking signal to the walking device after the machine arm device has harvested the crop, so that the shooting device can continue shooting again to form an image stream, so that the object detection module can perform real-time detection again. A crop harvester as described in claim 1, wherein the photographing device is a depth camera. A crop harvester as described in claim 1, wherein the object detection module is an artificial intelligence model trained by YOLOv5x6. A crop harvester as described in claim 8, wherein the artificial intelligence model is trained with multiple images including the crop being obstructed or overlapped with other non-harvesting objects. A crop harvester as described in claim 9, wherein the artificial intelligence model is further trained by the plurality of images of the crop labeled according to its height. A crop harvester as described in claim 1, wherein the crop corresponding to the marked area is asparagus stems. A crop harvester as described in claim 1, wherein the machine arm device has multi-axis degrees of freedom, and a clamping unit is provided at the end thereof, and the clamping unit includes a soft pad and a blade.

Images