JP3396920B2 - Harvesting robot imaging method - Google Patents

Description

Description: BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an imaging method of a harvesting robot, and more particularly to a visual section of a harvesting robot that performs stereoscopic imaging with a CCD camera and detects a distance to an object to be imaged. Etc. can be used. 2. Description of the Related Art In the visual part of a conventional harvesting robot, the sliding direction of a CCD camera when performing stereo imaging is always constant. In the case of cucumber, if the CCD camera is slid only to the right as shown in FIG. 12, the fruit existing at the left end in the first image as in the A image is missing in the second image. As a result, the distance may not be detected, or a fruit different from the fruit of the first image in the second image, such as the B image, may be erroneously detected and approached at an incorrect distance to damage the fruit. [0003] Accordingly, the present invention is based on a CCD camera.
At the time of position stereo imaging, the same object existing in the first image must be always present in the second image so that the distance to the imaging object can be accurately detected. According to the present invention, a CCD camera 1 is slid a fixed distance to capture a first image and a second image from two positions and detect a distance D to an object 2 to be imaged. In the visual part 3 of the harvesting robot using the stereo imaging method, if the position of the object 2 in the first image is shifted to the left or right from the center line of the image plane, the CCD camera 1 is displaced. The imaging method of the harvesting robot is characterized in that the imaging apparatus captures the second image by sliding to the side of the harvesting robot. When a harvest robot works to harvest fruits, for example, when the harvest target 2 is a fruit,
The fruit is imaged by the CCD camera 1 of the visual section 3 provided near the manipulator that grasps and cuts the fruit to remove the fruit. At the time of this imaging, as shown in FIG. Using a stereo imaging method in which the first image and the second image are slid to perform two-position imaging, the object distance is calculated from the sliding distance and the focal distance of the CCD camera 1 and the number of pixels of the object 2 moving on the image plane. The distance D to the fruit as 2 can be obtained. In this stereo imaging, if the imaging position of the fruit is shifted to the left from the center line of the image plane in the first image, CC
By moving the D camera 1 to the left and rightward if it is at a position deviated rightward, the D object 1 is slid to the right by a certain distance to capture the second image, so that the imaging target 2 is positioned at the center line of the image plane of the first image. Therefore, the same object 2 can be imaged on the image plane of the second image, no matter how much the image is displaced. As described above, even when the object 2 picked up by the first image is displaced to the left or right from the center line of the image plane when the image is picked up by the CCD camera 1, the CCD camera 1 is displaced to this position. The CCD camera 1 can be slid a predetermined distance in the left or right direction in accordance with the position direction, so that the sliding direction of the CCD camera 1 is limited to one direction as in the related art. Object 2 deviated to
Is not lost in the second image, and the same object 2 existing in the first image
It can be present in the image, and the distance from the CCD camera 1 to the imaging target 2 can be accurately detected. An embodiment of the present invention will be described below with reference to a harvesting robot that uses cucumber as an object 2 among fruits.
(Hereafter, cucumber is a specific name and the symbol is omitted)
A pair of left and right front wheels 5 and rear wheels 6 are respectively disposed at lower front and rear positions of the chassis 4, and a wheel drive motor 7 is interlocked and connected to each of the four front wheels 5 and rear wheels 6 in the traveling direction. On the other hand, the vehicle body 9 is constructed by supporting the vehicle freely in a turning direction and by providing a steering angle sensor 8 for detecting a steering angle for controlling the steering of the four wheels 5 and 6 individually. A front cover 10 is provided at an upper front position of the undercarriage 4, and a conical revolving cover 12 having a manipulator 11 for grasping and removing cucumber as a fruit is provided behind the front cover 10 so as to be revolvable. At the same time, a carrier supply device 14 for storing a large number of carriers 13 for transporting the harvested cucumber and supplying it in a timely manner is arranged behind the revolving cover 12, that is, at the rear position of the chassis 4. The carrier supply device 14
An operation panel 15 for manually controlling the operation of the harvesting robot and a directional lever 16 are provided on the rear side surface. The manipulator 11 has a shoulder motor 11a, an elbow motor 11b, a wrist motor 11c and a wrist motor 11c for each joint from the base side to the distal end side in order to perform an action similar to the movement of a human arm and a hand by multiple joints.
The hand unit 17 and the hand unit 17 are linked to each other. A slider 18 composed of a ball screw or the like for raising and lowering the manipulator 11 is fixedly provided inside the revolving cover 12 along the inclination of the revolving cover 12, and the manipulator 11 and the revolving cover 1 are provided.
2 is fixed to the chassis 4 at the center of the turning cover 12, and the waist motor 19 is connected to the slider 18 and the turning cover 12 so as to be turnable. The base is attached to the slider 18 with the dustproof rubber 12 of the turning cover 12.
A slide guide 20 is provided to penetrate through the slide groove of a and to be able to move up and down, and an up and down slide motor 21 for moving the manipulator 11 up and down is interlockingly connected to the slider 18. The hand section 17 of the manipulator 11 includes a fruit detection sensor 17a for detecting cucumber, a gripping motor 17b for gripping the detected cucumber, a fruit pattern detection sensor 17c for detecting the fruit pattern of the gripped cucumber, A fruit pattern detecting slide motor 17d for this detection and a cutting motor 17e for cutting the detected fruit pattern are provided, and as shown in FIG. 8, the slide rail 22 is laterally arranged at an upper position near the base of the manipulator 11. A CCD camera 1 (hereinafter simply referred to as a camera) that captures an image of a cucumber and detects the direction and distance where the cucumber exists so that the cucumber can be slid uniformly on the slide rail 22 in the horizontal direction by a fixed distance. A left / right slide motor 23 for sliding the camera 1 in the left / right direction is connected to one end of the slide rail 22. Linked to constitute a visual portion 3. [0011] As shown in FIG.
A running unit controller 24 that controls the running relationship, a harvesting unit controller 25 that controls the harvesting relationship, a visual unit controller 26 that controls the visual unit 3, and a general control by connecting these controllers 24, 25, and 26 The main controller 27 is provided inside the front cover 10. The running controller 24 is connected to the wheel drive motor 7, the steering angle sensor 8, the carrier supply device 14 and the like so as to be controllable.
7 and the main controller 27
Is connected to the operation panel 15 and the safety device 28 at the same time. As shown in FIG. 4, the harvesting unit controller 25 has the waist motor 1 as a servomotor.
9, vertical slide motor 21, shoulder motor 11a,
The elbow motor 11b and the wrist motor 11c are connected via the control driver 29, respectively, and the fruit detection sensor 17a, the gripping motor 17b, the fruit pattern detection sensor 17c, the fruit pattern detection slide motor 17d, and the cutting motor 17e are respectively connected. Connect and configure. Each of the servomotors 19, 21, 11a, 11b, and 11c is provided with a combination of an encoder and an origin detection sensor. As shown in FIG. 5, the visual unit controller 26 receives a signal from the OCR 30 and generates a signal necessary for operating the camera 1 from a driving pulse generating circuit 31 to a receiver 33 via a driver 32. The receiver 33 is connected to the camera 1. Further, the camera 1 is connected to a signal processing circuit 34 that converts a signal detected by driving the camera 1 into a video signal, and an A / D converter 35 that converts a video signal from the signal processing circuit 34 to a digital signal. And an image processing circuit 37 for performing high-speed image processing and a CPU 38 for performing central processing. The CPU 38 is connected to a timing circuit 39 for setting a strobe light emission timing and a start timing of the A / D converter 35 based on a video signal, and the timing circuit 39, the A / D converter 35, and a strobe drive circuit 40, the signal processing circuit 34 is connected to the timing circuit 39, and the strobe 41 is driven via the strobe drive circuit 40. Next, the operation of the above configuration will be described. In order for the harvest robot to harvest cucumber among fruits as the target object 2, as shown in FIG. 6, the traveling unit is operated manually or automatically by the traveling unit controller 24 via the traveling unit controller 24 or the steering angle. The traveling of the traveling vehicle body 9 is controlled by controlling the sensor 8 and the like. The traveling of the cucumber cultivated on the cucumber cultivation shelf B while traveling along the ridge groove A causes the manipulator 11 itself to turn left and right by calculation of the harvesting unit controller 25. By controlling the up and down movement and the free movement of the hand unit 17, the cucumber is selected and harvested at an appropriate time for harvesting, and the harvested cucumber is collected in the carrier 13, and the carrier 13 is filled. And the next carrier 13 is set by the carrier supply device 14. During a series of harvesting operations, the camera 1 serving as an eye of the manipulator 11 is moved to the visual controller 26.
When performing stereo imaging in which the camera 1 captures images from two positions, the first stereo imaging is performed as shown in the flowchart of FIG.
In response to an image pickup input of an image, a signal processing circuit 34 ⇒ A / D converter 35 ⇒ image memory 36 ⇔ image processing circuit 37 回路 CPU 3
The image processing is performed by the operation of each circuit 8 and the cucumber is recognized by the image processing. From this recognition, a cucumber of | Xmax | is selected, and it is checked whether the selected cucumber is on the right or left side of the center line of the image plane. As shown in FIG. In some cases, the search area of the second image is set to the left of the cucumber, and as shown in FIG. 8, the camera 1 is slid to the right by a certain distance (for example, 150 mm). The camera 3 is set further to the right, and the camera 3 is slid 150 mm to the left to capture and input the second image. The same image processing as described above is performed only in the area of the input second image, and as shown in FIG.
Is calculated by dividing a product obtained by multiplying the slide distance L by the focal length d of the camera 1 by the number of pixels moved by the cucumber in the first image and the second image, that is, the distance between the camera 1 and the cucumber, The length of the cucumber is calculated from the calculated distance. If the calculated cucumber length is longer than a predetermined reference length (about 170 mm), it is determined that harvesting is possible, and the manipulator 11 is moved to the shoulder motor 11.
a, the elbow motor 11b and the wrist motor 11c are operated to cause the hand portion 17 to approach the cucumber of the harvest mark. With this approach, the gripping motor 17b is operated by the detection of the fruit detection sensor 17a of the hand portion 17 to grip the cucumber. The fruit of the cucumber is grasped by a fruit detection sensor 17c and a fruit detection slide motor 17c.
d, and the detected peduncle is detected by the cutting motor 17.
e is applied and cut to complete the fruiting. Further, as shown in FIG. 10, when a plurality of, for example, two cucumber are present in the first image at the time of imaging of the first image and are divided into right and left with respect to the center line of the image plane, the center line On the other hand, when the cucumber a having a large separation distance is on the right, the camera 1 is slid right to set the search area of the second image to the left of the cucumber a. However, the cucumber b moves to the left more and more disadvantageously due to this correspondence, but the separation distance from the center line is small, so that the probability that the cucumber b is dropped from the image plane at the time of capturing the second image is small. If the cucumber a is on the left side of the center line in the first image, the reverse of the above is performed. As described above, the cucumber a having a large separation distance from the center line at the time of capturing the first image is prioritized.
Is slid, the separation distance from the center line is small even if the sliding direction of the camera 1 with respect to the other cucumber b is reversed. The distance between a plurality of cucumbers can be detected simultaneously by the imaging operation. For example, as shown in FIG. 11, after detecting the distance between two cucumbers and removing the cucumber a, if the cucumber a is approached without confirmation, the slack of the main stem is reduced by an amount corresponding to the loss of the weight of the cucumber a. Changes, the position of the cucumber b may change, so that the cucumber a cannot be grasped, or the fist or the like may be damaged by the hand unit 17, so after the cucumber a is plucked, an image is taken again at the second image position,
Image processing is performed only in the search area, and it is confirmed whether or not the position (direction) of the cucumber b is shifted. When there is no displacement, the cucumber b is approached as it is, and when there is a displacement, the cucumber b is returned to the previously captured first image position, and the cucumber b is imaged at the first image position and searched. Image processing is performed only in the area, the distance is calculated, and cucumber b is approached. As described above, when the imaging operation of the camera 1 is performed individually for each cucumber, it takes time and time, and therefore, the distance is detected only when the position of the cucumber is misaligned. It is possible to increase the efficiency of the position detection operation and accurately hold and extract the cucumber. In addition, when determining whether or not harvesting is possible based on the length of the cucumber, a necessary condition is that the upper and lower ends of the cucumber are not hidden, but when the upper and lower ends are hidden by foliage and the like, It is not possible to determine whether or not harvesting is possible based on the length of the cucumber. However, the length and thickness of the cucumber are 0.
The cucumber has a correlation coefficient of 9 or more, and the length can be estimated to some extent based on the thickness, so that if a part of the cucumber is visible, it can be recognized as a cucumber in a suitable harvest time. Thus, as shown in the flow chart of FIG. 13, image input is performed at wavelengths of 850 nm and 550 nm, an arithmetic image is detected by calculating this input image, and the arithmetic image is binarized by calculating a threshold value. An overlapping area is set from the binary image (1), and the image having the wavelength of 850 nm is binarized and superimposed on the binary image (1). After performing fine processing (shrinkage) and thick processing (diffusion) for smoothing the binary image, labeling processing is performed to calculate the perimeter of the cucumber, and the Feret length ratio is calculated by deleting the perimeter of a cucumber or less. I do. If the calculated value of the feret length ratio is equal to or less than the constant value, the FLAG is calculated and the FLAG is checked. If FLAG = 1 is not satisfied, it is determined that the cucumber is not a cucumber and deleted. When the calculated value of the Ferre length ratio is equal to or greater than the constant value, the center line is extracted, the noise component is removed, the center lines are combined, and if the length is equal to or greater than the constant value, it is determined to be a cucumber. The approach for harvesting is performed, and if it is less than the constant value, it is determined that it is not a cucumber and deleted. When gripping a cucumber hidden by leaves, as shown in the flowchart of FIG.
An image of a cucumber is input, and image processing of the input image is performed to determine whether or not harvesting is possible based on information on the top (lower end) of the cucumber. A high-speed approach is made to a position near the, and after this approach, a further low-speed approach is made to the apex. According to this approach, the hand unit 17 is slid upward along the cucumber by estimating the length based on the thickness of the cucumber, and the cucumber is gripped by the gripping motor 17b of the hand unit 17 to act. Pattern detection sensor 17
The stem is detected by c, and the cutting motor 17e is operated to cut the stem. Thus, it is not necessary to approach while selecting the avoiding direction, and it is possible to shorten the harvesting time by the straight-line approach, and also possible to harvest the cucumber hidden by the leaves, thereby improving the yield. In this embodiment, the object 2 to be harvested is limited to cucumber. However, it goes without saying that the gist of the present invention can be applied to fruits other than cucumber.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a perspective view showing an entire harvesting robot. FIG. 2A is a front view showing the entire harvesting robot. (B) A plan view showing the entire harvest robot. (C) A side view showing the entire harvest robot. FIG. 3 is a block diagram showing a control relationship of the whole harvesting robot. FIG. 4 is a block diagram showing a control relationship of a harvesting unit. FIG. 5 is a block diagram showing a control relationship of a visual unit. FIG. 6 is a front view showing a working state of the harvesting robot. FIG. 7 is a flowchart illustrating a processing procedure of a visual unit. FIG. 8 is a calculation diagram showing a method of calculating a distance by a visual part. FIG. 9 is an image diagram showing a displacement relationship between a first image and a second image. FIG. 10 is an image diagram showing a displacement relationship between a first image and a second image. FIG. 11 is an image diagram showing a displacement relationship between a first image and a second image. FIG. 12 is an image diagram showing a displacement relationship between a first image and a second image. FIG. 13 is a flowchart illustrating a processing procedure of a visual unit. FIG. 14 is a flowchart showing a manipulator approach procedure. [Description of Signs] 1 CCD camera 2 Object to be imaged 3 Visual part

Claims (1)

(57) [Claim 1] A stereo that slides a CCD camera 1 a fixed distance to capture a first image and a second image from two positions, and detects a distance D to an object 2 to be imaged. In the visual part 3 of the harvesting robot using the imaging method, if the position of the object 2 in the first image is shifted to the left or right from the center line of the image plane, the CCD camera 1 is shifted. An imaging method of a harvesting robot, wherein the imaging apparatus captures a second image by sliding the imaging apparatus to a side where the user is.