KR100433272B1 - An apparatus of detecting 3-D position of fruits and method of detecting 3-D position of it - Google Patents

Abstract

The present invention relates to a three-dimensional position detection apparatus and method thereof of fruits and vegetables, comprising: independent first and second cameras (2, 4) for detecting the X and Y axis distances of a harvest object such as fruits and fruits; A laser sensor (3) attached between the first and second cameras (2,4) for detecting a straight line distance to a harvesting object which is a Z axis; A six-axis manipulator 20 to which the first and second cameras 2 and 4 and the laser sensor 3 are attached; First and second image boards 31 and 32 connected to the first and second cameras 2 and 4 to process images of the harvest object; A six-axis manipulator adjusting device 40 connected to and controlled by the six-axis manipulator 20; It is composed of a computer 50 connected to the six-axis manipulator adjusting device 40 and the first and second image boards 31 and 32 to perform a series of control.

Therefore, the present invention has the effect of detecting the three-dimensional position of the harvest object by stereo image processing and pan tilting technique using two cameras and a laser sensor.

Description

An apparatus of detecting 3-D position of fruits and method of detecting 3-D position of it}

The present invention relates to a three-dimensional position detection device and method thereof of the fruit vegetables and fruits, and more particularly to the three-dimensional position of the fruit vegetables and fruits that can detect the fruit vegetables and fruits overlapping or partially obscured by obstacles such as leaves and branches A detection apparatus and a method thereof are provided.

In general, fruits and vegetables require considerable labor to harvest. Accordingly, recent researches are being conducted to facilitate the harvesting of fruits and vegetables, fruits, and transplanting using robots.

In this study on the visual device of the fruit harvesting robot, the visual device for detecting the three-dimensional distance of the mini-tomato using the phase difference of the image projected on the two-dimensional plane of the camera, the articulated manipulator, the traveling device and the vacuum object And a method of extracting color components from an image input from a camera to identify ripe tomatoes, immature tomatoes, and foliage. The study of cutting the nipple with the blade rotation has already been completed.

By the way, the apparatus and method of harvesting the fruit has a problem that can not be detected when the harvest target is overlapped or covered with leaves.

Accordingly, the present invention has been made in order to solve the above-mentioned conventional problems, and includes independent cameras for detecting the X and Y axis distances of harvested objects such as fruits and vegetables, and a laser sensor for detecting the Z axis distance. In order to provide a three-dimensional position detection device of fruits and vegetables capable of accurately detecting the harvested objects that are overlapped or covered by leaves by detecting a three-dimensional position of the harvested objects by stereo image processing and pan tilting techniques using a 6-axis manipulator. The purpose is.

Another object of the present invention is to detect the X, Y-axis distance of the harvesting object, such as fruits and vegetables by a stereo image processing method, the Z-axis distance is superimposed by a stereo image processing and pan tilting technique to detect the linear distance, or covered with leaves It is to provide a three-dimensional position detection method of fruits and vegetables to accurately detect the harvesting objects present.

In order to achieve the above object, the three-dimensional position detection device of the fruit vegetable and fruit of the present invention comprises: independent first and second cameras (2, 4) for detecting the X, Y-axis distance of the harvest object such as fruit and fruit; A laser sensor (3) attached between the first and second cameras (2,4) for detecting a straight line distance to a harvesting object which is a Z axis; A six-axis manipulator 20 to which the first and second cameras 2 and 4 and the laser sensor 3 are attached; First and second image boards 31 and 32 connected to the first and second cameras 2 and 4 to process images of the harvest object; A six-axis manipulator adjusting device 40 connected to and controlled by the six-axis manipulator 20; The six-axis manipulator adjusting device 40 and the first and second image boards 31 and 32 are connected to the computer 50 to perform a series of control.

According to another object of the present invention, a three-dimensional position detection method of fruits and fruits is a harvesting object such as fruits and fruits with a first camera 2 and a second camera 4 attached to an upper portion of three axes of a six-axis manipulator 20. Obtaining a video of the first step; The beam of the laser sensor 3 attached between the first and second cameras 2 and 4 by a stereo image processing method using a positional change amount of an image formed on the imaging surface of the first and second cameras 2 and 4. Z-axis distance (Rz) from the launch point to the harvest object, X-axis distance (Rx) from the horizontal direction point where the harvest object is present at the beam firing point and the center point of the harvest object at the horizontal direction point where the harvest object is present A second step of obtaining a three-dimensional distance that is a Y-axis distance Ry up to; The first angle θx from the beam emission point to the horizontal direction point of the harvesting object is obtained from the Z axis distance Rz and the X axis distance Rx, and the Z axis distance Rz and the X axis distance ( A third step of obtaining a second angle θy from the horizontal direction point to the center point of the harvest object based on the beam emission point based on Rx) and the Y axis distance Ry; The laser sensor attached between the first and second cameras 2 and 4 by moving one axis and three axes of the six-axis manipulator 20 by the first angle θx and the second angle θy. A fourth step of matching the beam firing point of (3) with the center point of the harvest object; The laser sensor 3 detects the Z-axis distance Rz between the beam firing point and the center point of the harvest object, and uses a stereo image using the position change amount of the image formed on the imaging surface of the first and second cameras 2 and 4. The fifth step is to obtain the X-axis distance (Rx) and Y-axis distance (Ry) as a treatment method.

1 is a schematic configuration diagram of a three-dimensional position detection device of fruits and vegetables according to the present invention.

2 is a diagram illustrating a relationship between a camera for detecting a three-dimensional position according to the present invention.

3 is a view showing the principle of the stereo imaging method according to the present invention.

Figure 4 is a flow chart of the three-dimensional position detection method of fruits and vegetables according to the present invention.

5 is a photograph of a state where the fruit is located in the center of the two cameras by the image and the pan tilting of the harvest object according to the present invention.

6 is a photograph of a state in which the beam firing center of the laser sensor according to the present invention is fired at the center of the fruit.

<Description of the symbols for the main parts of the drawings>

1: object 2, 4: first and second camera

3: laser sensor 20: 6-axis manipulator

31,32: 1st, 2nd image board 40: 6-axis manipulator adjusting device

50: computer

Hereinafter, exemplary embodiments of the present invention will be described with reference to the accompanying drawings.

1 is a schematic configuration diagram of a three-dimensional position detection device of fruits and vegetables according to the present invention, the independent first and second cameras 2,4 for the X, Y-axis distance of the harvesting object 1 such as fruits and vegetables ), And the linear distance to the Z-axis harvesting object is detected by the laser sensor 3 attached between the first and second cameras 2 and 4.

The laser sensor 3 attached between the two cameras 2 and 4 keeps the optical axis of the camera horizontal and at the same time the beam is emitted at the same height in the vertical direction.

The first and second cameras 2 and 4 and the laser sensor 3 are attached to an upper portion of the three axes of the six-axis manipulator 20, and the first and second cameras 2 and 4 are attached to the first and second cameras 2 and 4. Image boards 31 and 32 are connected to each other to process an image of the harvest object transmitted from the first and second cameras 2 and 4. The six-axis manipulator 20 is connected to the six-axis manipulator adjusting device 40 to receive a control signal. The six-axis manipulator adjusting device 40 and the first and second image boards 31 and 32 are also connected to the computer 50 and subjected to a series of control.

Images of the harvest objects such as fruits and vegetables input from the first and second cameras 2 and 4 are input to the first and second image boards 31 and 32, respectively, and are input by the computer 50. The six-axis manipulator 20 moves to the harvest object under the control of the six-axis manipulator adjusting device 40 according to the three-dimensional distance and the result of the image processing.

The six-axis manipulator adjusting device 40 is sequentially controlled by using the serial communication with the computer 50. Therefore, the images of the fruits and vegetables harvested object 1, such as fruits and vegetables input from the first and second cameras 2 and 4, are input to the first and second image boards 31 and 32, respectively. Acquisition time is faster.

FIG. 2 is a diagram illustrating a relationship between a camera for detecting a three-dimensional position according to the present invention, and includes a first camera 2 and a second camera 4 attached to a three-axis upper portion of the six-axis manipulator 20. Images of harvesting objects such as fruits and vegetables are obtained, and the first and second cameras 2 and 4 are subjected to stereo image processing using a positional change amount of the image formed on the image pickup surface of the first and second cameras 2 and 4. Z-axis distance Rz of the harvesting object at the beam emission point 8 of the laser sensor 3 attached in between and the horizontal direction point 7 at which the harvesting object exists at the beam emission point 8 The three-dimensional distance, which is the Y-axis distance Ry from the X-axis distance Rx and the horizontal direction point 7 at which the harvesting object is present, to the center point 9 of the harvesting object, is obtained.

In FIG. 2, the angle from the beam emission point 8 of the laser sensor 3 to the horizontal direction point 7 of the harvesting object is referred to as a first angle θx and is referred to the beam emission point 8. The angle from the horizontal direction point 7 to the center point 9 of the harvesting object is referred to as a second angle θy, which is called a pan tilting angle.

The first angle θx can be obtained from the Z-axis distance Rz and the X-axis distance Rx of the laser sensor 3, and the Z-axis distance Rz and the X-axis distance Rx. And the 2nd angle (theta) y are calculated from and Y-axis distance Ry.

Fig. 3 shows the principle of the stereo imaging method according to the present invention, in which the harvesting object 1 is picked up by the first and second cameras, and shows the first and second images.

Among the three-dimensional distances Zz (Rz), X-axis distance (Rx) and Y-axis distance (Ry), which are three-dimensional distances to the harvest object, the X-axis distance (Rx) is Obtain it using

Where Imgx is the X-axis variation of the reference camera.

LF is the focal length.

Y axis distance Ry Obtain it using

Where Imgy is the Y-axis variation of the reference camera.

In addition, Z-axis distance (Rz) is Obtain using

Where Imgz is the amount of change in the X-axis position of the first and second cameras 2 and 4,

Lb is the distance of the first and second cameras 2 and 4.

4 is a flowchart of a three-dimensional position detection method of fruits and vegetables according to the present invention, by moving the X-axis of the first manipulator horizontally by -15 ° (step 61), and harvesting such fruits and fruits from the first and second cameras. If the image of the object exists, it is acquired. (Step 62) With the obtained image of the harvested object, three-dimensional distance in the X, Y, and Z directions is obtained by using stereo image processing method, and the pan tilting angle is determined by the distance. The first angle θx and the second angle θy are calculated (step 63), and the first and second cameras 2 are moved by moving one axis and three axes of the manipulator 20 by this panting angle. And 4, the beam firing point of the laser sensor 3, which is attached between them, coincides with the center point of the harvest object (step 64).

The laser sensor 3 detects the Z-axis distance Rz, which is the distance between the beam firing point and the center point of the harvesting object, and uses the position change amount of the image formed on the imaging surface of the first and second cameras 2 and 4. The X-axis distance Rx and the Y-axis distance Ry are calculated by the stereo image processing method (step 65).

In this way, after determining the exact X-axis distance Rx, Y-axis distance Ry, and Z-axis distance Rz of the harvest object, the manipulator 20 can be moved to harvest the object (step 66).

In step 66, if the manipulator 20 is moved to complete harvesting of the object, or if the object does not exist in the first and second cameras in step 62, the manipulator 20 is moved by 5 ° until it becomes + 15 °. Find images of objects to harvest as you move.

The manipulator 20 is moved by 5 degrees, and if the image of the harvest object exists, steps 63, 64, 65 and 66 are repeatedly performed.

On the other hand, after completing step 66, the stereo image processing is performed again to calculate the pan tilting angle, and then the manipulator 20 is moved again. Then, the beam is emitted from the laser sensor, and the three-dimensional position of the precise harvest object can be detected by calculating the three-dimensional distance whose Z-axis distance is corrected.

FIG. 5 is a photograph of a state in which fruit is placed at the center of two cameras by pantilting and an image of a harvesting object according to the present invention. States 71 and 72 are shown, and the lower pictures with the designations of the X-axis and Y-axis adjustments show states 73 and 74 where the fruit is located at the center of the two cameras by pan tilting.

6 is a photograph of a state in which the beam firing center of the laser sensor according to the present invention is fired at the center of the fruit, even if both sides of the fruit are covered by a leaf or the like. The beam firing center shows the state that the beam was correctly projected to the fruit, and the laser beam firing can obtain the distance along the Z axis.

As described in detail above, the present invention can detect the three-dimensional position of the harvest object by stereo image processing and pan tilting using two cameras and a laser sensor, and even if the object is partially covered by obstacles such as leaves, There is a possible effect.

Although the invention has been described in detail only with respect to specific examples, it will be apparent to those skilled in the art that various modifications and variations are possible within the spirit of the invention, and such modifications and variations belong to the appended claims.

Claims (4)

delete delete A first step of acquiring an image of a harvest object such as fruits and vegetables with a first camera (2) and a second camera (4) attached to the three axes of the six-axis manipulator (20); The beam of the laser sensor 3 attached between the first and second cameras 2 and 4 by a stereo image processing method using a positional change amount of an image formed on the imaging surface of the first and second cameras 2 and 4. Z-axis distance (Rz) from the launch point to the harvest object, X-axis distance (Rx) from the horizontal direction point where the harvest object is present at the beam firing point and the center point of the harvest object at the horizontal direction point where the harvest object is present A second step of obtaining a three-dimensional distance that is a Y-axis distance Ry up to; The first angle θx from the beam emission point to the horizontal direction point of the harvesting object is obtained from the Z axis distance Rz and the X axis distance Rx, and the Z axis distance Rz and the X axis distance ( A third step of obtaining a second angle θy from the horizontal direction point to the center point of the harvest object based on the beam emission point based on Rx) and the Y axis distance Ry; The laser sensor attached between the first and second cameras 2 and 4 by moving one axis and three axes of the six-axis manipulator 20 by the first angle θx and the second angle θy. A fourth step of matching the beam firing point of (3) with the center point of the harvest object; The laser sensor 3 detects the Z-axis distance Rz between the beam firing point and the center point of the harvest object, and uses a stereo image using the position change amount of the image formed on the imaging surface of the first and second cameras 2 and 4. And a fifth step of obtaining an X-axis distance (Rx) and a Y-axis distance (Ry) as a processing method. The method of claim 3, wherein In the second and fifth steps, the X-axis distance Rx and the Y-axis distance Ry are obtained by stereo image processing. X-axis distance (Rx) is Obtain it using Where Imgx is the X-axis variation of the reference camera. LF is the focal length. Y axis distance Ry Obtain it using Where Imgy is the Y-axis variation of the reference camera. Obtaining the Z-axis distance (Rz) by the stereo image processing method of the second step, Z-axis distance (Rz) is Using Where Imgz is the amount of change in the X-axis position of the first and second cameras 2 and 4, Lb is the distance of the first and second cameras 2 and 4. The three-dimensional position detection method of fruit vegetables and fruits characterized by the above-mentioned.