JPH1042665A - Image pick up device of robot for agricultural use - Google Patents

Abstract

PROBLEM TO BE SOLVED: To shorten a harvesting work by carrying out a second input of an image within a treating time of a first input of the image by an image pick up of a visual sensor. SOLUTION: In this image pick up device of a robot for agricultural use, a first input of an image is carried out by a visual sensor 1 which picks up the image of an objective substance in work by moving to slide upward and downward, then, a second input of the image can be carried out by moving the visual sensor 1 upward and downward parallel to an image treatment of the first input image by installing a built-in memory for images.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image pickup apparatus for an agricultural robot, and belongs to a technical field such as an apparatus for efficiently reducing the time required for image input and image processing by a visual sensor.

[0002]

2. Description of the Related Art In an agricultural robot, a visual sensor capable of sliding and moving in an up-and-down direction is provided. Then, the harvesting work is performed by the manipulator after the first image input / image processing, and upon completion of this work, the visual sensor and the manipulator are slid upward or downward to perform the harvest work after the second image input / image processing. In this way, since the operations from image input to harvesting are consistently performed independently, the manipulator remains stopped during image processing, and it is difficult to shorten the harvesting time.

Accordingly, the present invention shortens the harvesting time by performing the second image input within the image processing time of the first input image.

[0004]

According to the present invention, after a first image input is performed by a visual sensor 1 which slides up and down and captures an image of a work object, the first image input is performed by a built-in image memory. The imaging device of the agricultural robot is configured to move the visual sensor 1 upward or downward in parallel with the image processing of the input image to enable the second image input.

[0005]

With the above arrangement, the visual sensor 1 for imaging the work object in the agricultural robot, for example, in the imaging area divided into the lower position and the upper position, firstly, the work object at the lower position by the first imaging. Immediately after the image input is performed, the visual sensor 1 is slid upward immediately to input an image at the upper position by the second imaging. At this time, by incorporating a memory for the image, the first image is input. Since the second image can be input in parallel with the image processing of the input image, the second image input time is actually zero within the first image processing time and becomes zero. The time required for image input and image processing can be reduced.

The image processing of the second input image can be performed in parallel with the operation of the manipulator 5 for performing the harvest approach based on the information of the first image, thereby reducing the time required for image processing. Can be achieved.

[0007]

Since the second image input time by the visual sensor 1 is performed in parallel with the image processing of the first input image, it is not necessary to provide a separate image input time, and the second image input time is shortened. In addition, since the second image processing time can be processed in parallel with the operation of the manipulator 5 at the time of harvesting, there is no need to provide a separate image processing time, and the time for the second image processing can be reduced. Therefore, it is possible to significantly reduce the recognition of the work target and the total harvest time. Since the first and second image inputs are both performed in the initial state in which the manipulator 5 is stopped, there is no blurring or displacement in the image, and the direction of the harvest approach of the manipulator 5 does not change. .

[0008]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described below with reference to the drawings, referring to an agricultural robot for harvesting fruits and vegetables as work objects. A pair of left and right front wheels 10a and rear wheels 10b are provided at lower front and rear positions of the chassis 9, respectively.
And a wheel drive motor 11 for each of the four wheels 10b and the rear wheel 10b, and also supports the steering wheel so as to be steerable and revolvable with respect to the traveling direction, and controls the front and rear wheels 10a and 10b separately. A traveling device 13 is provided by providing a steering angle sensor 12 for detecting a steering angle, and storing a battery 13a as a power source between the front and rear wheels 10a and 10b.

A front cover 14 containing a controller and the like for controlling the operation of the agricultural robot is provided at a front position on the undercarriage 9, and an image of cucumber as fruit and vegetables is taken behind the front cover 14. The work revolving drum 15 is mounted on a visual sensor 1 by a CCD camera 2 and a manipulator 5 for grasping and removing cucumber by the image of the visual sensor 1. A rear cover 16 is provided at a rear position on the undercarriage 9 on the rear side of the work turning cylinder 15 and the rear cover 16 stores and discharges a carry for storing harvested cucumbers. The work device 17 is configured by disposing the carry control unit 16a. After this, cover 1
6, an operation panel 17a of the robot is provided on the rear side surface.

Since the manipulator 5 performs an action similar to the movement of a human arm and a hand by multiple joints, each of the manipulators 5 includes a shoulder joint S, an arm joint A, a wrist joint W, and the like from the base side to the distal end side. The joints are connected and arranged. The shoulder joint S is provided with a shoulder motor 18 by a servo motor in a shoulder joint for rotating the manipulator 5 up and down.
Incorporates an arm motor 19 by a servo motor into the telescopic linear motion arm 6 that expands and contracts in two stages, and the wrist joint W
Is a wrist pitch motor by a servo motor for pitch or roll rotation of an end effector 8 for gripping and picking a cucumber at one end of a wrist pitch joint shaft 7 internally supported near the distal end of a telescoping linear arm 6. 20 or a wrist roll motor 21 are incorporated and are connected to each other.

A waist motor 25, which is a servomotor for horizontally rotating the manipulator 5, is provided in the base 5a on which the shoulder joint S is mounted, in interlocking connection. A plurality of cucumber can be stored on the lower surface of the base portion 5a. The auxiliary storage case 4, which extends to the position of the end effector 8 at the time of picking the cucumber and which can open and close the bottom surface, is driven by a rack and a pinion. A case slide motor 4a is disposed on the upper surface of the storage case 4 and configured.

The manipulator 5 and the auxiliary storage case 4
Is rotatably supported along the inner wall surface of the cutout portion of the work turning cylinder 15 so as to be vertically movable by sliding the slider 22 in the vertical direction. The slide guide 23 is provided in bearing. The base 5a of the manipulator 5 is provided on the slide guide 23 along the outer wall surface of the cutout portion of the work turning cylinder 15 at right angles.
a, and a slide motor 24 by a servo motor for rotating the slider 22 is connected to the lower end of the slider 22 in an interlocking manner, and the visual sensor 1 and the manipulator 5 are moved together with the work turning cylinder 15 by approximately 180. An undercarriage motor 9a for turning the vehicle is arranged at the center position of the work turning cylinder 15 of the undercarriage 9 and connected in an interlocking manner.

The end effector 8 of the manipulator 5
The gripper 8a grips the cucumber, a fruit detection sensor 8b that detects the cucumber, a gripping motor 8c that grips the detected cucumber, a fruit pattern detection sensor 8d that detects the fruit pattern of the gripped cucumber, A fruit pattern detection slide motor 8e for detection and a cutting motor 8f for cutting the detected fruit pattern are provided.

As shown in FIG. 1, the visual sensor 1 includes an interference filter 3a on the inside of the front surface of a cylindrical outer case 43, which can capture a density value image at a wavelength of 550 nm in a visible light region to a near infrared region. A disk-shaped interference filter 3 formed by half-split and an interference filter 3b capable of imaging a density value image at a wavelength of 850 nm is arranged, and the axis of the interference filter 3 is positioned rearward by a filter drive shaft 44. The CCD camera 2 is attached to the lower side of the filter drive shaft 44 in conjunction with the motor 45, and the front surface of the outer case 43 is provided with an imaging hole 43a only in the lens portion of the CCD camera 2.

A filter position sensor 46 such as a photomicro sensor for detecting the rotational position of the interference filter 3 is disposed above the filter drive shaft 44.
A photosensor 47 for detecting the intensity of light at the time of imaging is arranged on the lower side of the outer periphery of the lens, and the halogen lamp 41 as a light source is mounted on the upper side of the outer periphery of the outer case 43 to transmit light through the interference filters 3a and 3b. As a result, a two-wavelength imaging device that inputs two types of images having different wavelengths is configured.

As shown in FIG. 2, the CCD camera 2, the filter motor 45, the filter position sensor 46, the photo sensor 47, the halogen lamp 41 and the like are connected to a camera controller 49 via an interface 48. The visual sensor 1 rotatably supports a slider 26 made of a ball screw or the like that slides in the vertical direction along the inner wall surface of the cutout portion of the work turning cylinder 15 so as to rotate. A slide guide 27 is provided so as to be vertically movable by movement. The upper surface of the slide guide 27 penetrates a sliding groove 15b provided along an outer wall surface displaced to a substantially right angle position with respect to an outer wall surface having a sliding groove 15a of the manipulator 5 to engage the visual sensor 1. A slide motor 28 by a servomotor for rotating the slider 26 is connected to the lower end of the slider 26 in conjunction with the engagement of the arm 1a. A movable guide rail is provided.

As shown in FIG. 9, the control unit includes a traveling unit controller 29a for controlling the traveling device 13, a visual unit controller 29b for controlling the visual sensor 1, and a knob controller 29c for controlling the manipulator 5.
And a main controller 29 that connects these controllers 29a, 29b, and 29c and controls them in an integrated manner in the front cover 14.

The main controller 29 is connected to the operation panel 17a and the safety device 14a. The traveling unit controller 29a is configured by controllably connecting the wheel drive motor 11, the steering angle sensor 12, the carry control unit 16a, and the like. The visual unit controller 29b includes:
As shown in FIG. 10, a receiver 3 receives a signal from a drive pulse generation circuit 31 via a driver 32 and receives a signal from the OCR 30.
3 from the receiver 33 to the CCD camera 2. A signal processing circuit 34 for converting a signal detected by driving the camera 2 into a video signal is connected to the camera 2,
The signal processing circuit 34 is connected to an image memory 36 via an A / D converter 35, and the image memory 36, an image processing circuit 37 for performing high-speed image processing, and a CPU 38 are communicably connected. The CPU 38 is connected to a timing circuit 39 for setting the lamp emission timing and the start timing of the A / D converter 35 based on the video signal, and the timing circuit 39, the A / D converter 35, and the lamp drive circuit 40 are connected. Connection and the timing circuit 39
, A signal processing circuit 34 is connected, the lamp driving circuit 40 is connected to drive the halogen lamp 41, and the photosensor 47 is connected to the CPU 38 through A / D conversion.

As shown in FIG. 11, the thinning section controller 29c includes the shoulder motor 1 as a servomotor.
8, an arm motor 19, a wrist pitch motor 20, a wrist roll motor 21, a slide motor 24, a waist motor 25, and a slide motor 28 are connected via a control driver 42, respectively, and the fruit detection sensor 8b, gripping motor 8c, The pattern detection sensor 8d, the fruit pattern detection slide motor 8e, and the cutting motor 8f are connected to each other. The servo motors 18, 19, 20, 2
Each of 1, 24, 25, and 28 is provided with a combination of an encoder and an origin detection sensor.

In order to harvest a cucumber as a work object by the agricultural robot, the traveling unit controller 29a
While traveling along the ridge groove R under the control of the wheel drive motor 11 and the steering angle sensor 12, etc., the cucumber cultivated on the inclined shelf C at the optimal harvest time is calculated by the visual controller 29b and the thinning controller 29c. And a vertical approach and a horizontal turn of the visual sensor 1 and the manipulator 5 to perform an imaging and a harvest approach. With this approach, the end effector 8 is freely used, the gripping motor 8c is actuated by the detection of the fruit detection sensor 8b, and the cucumber is gripped by the gripping portion 8a. And a fruit pattern detection slide motor 8e, and the detected fruit pattern is cut by a cutting motor 8f to cut the fruit. The picked cucumber is collected and delivered to the carry, and is discharged to the ground by the carry control unit 16a when the carry is full.

The range in which the cucumber is distributed during the optimal harvesting period depends on the height of the ridge from the ridge R, the height of the cultivation bed in hydroponics, the height of the worker's height, and the like. Since the conditions are different, when the image input is performed with the lower end position and the upper end position of the imaging by the CCD camera 2 limited to a certain height, the image of the inclined shelf C of the cucumber is divided into two upper and lower stages (the number of divided stages is Arbitrarily set), the image area at the lower position contains a lot of ridge soil, which is wasted, and the image area at the upper position includes no foliage and directly inputs sunlight. The possibilities are great.

For this reason, by setting the height at the time of image input to be as small as possible in the area where cucumber is determined to be present, useless images of the soil portion can be captured and the CCD camera 2 can be used. It is possible to avoid such a problem that direct sunlight enters and the entire screen becomes saturated (white), making it difficult to recognize cucumber. In addition, the cultivation style of cucumber is basically cultivated so as to minimize the rate of cucumber hiding behind the foliage by adopting an inclined shelf C that inclines the cultivation surface. Cultivation can be broadly divided into pinching and cultivation. In pinching cultivation, cucumber in the optimal harvesting season is distributed in almost the entire cultivated area in the prime season, so that harvesting work is required at the upper position and the lower position. In descent cultivation, cucumber at the right time for harvesting is not present near the upper growing point, and each vine is distributed in the approximately same height area below. Therefore, harvesting work at a certain height in the lower position Only that would be good.

From the above, the operation panel 17a
Is provided with a cultivation changeover switch 50 that can be set to switch between pinching cultivation and invasion cultivation. Since the image input range is determined in accordance with the cultivation style by switching this switch 50, a wasteful image of a portion where no cucumber is clearly present There is no need for input and processing, and the total harvest time can be shortened in the infestation cultivation. In addition, pinching cultivation is mainly performed in summer, and infestation cultivation is mainly performed in winter,
Since the upper and lower operations and the operation of only the lower operation can be performed by switching the switch 50, the robot can be utilized throughout the year.

In the harvesting operation in the pinching cultivation, as shown in the flow chart of FIG. 12, the visual sensor 1 and the manipulator 5 are each set to the initial position, and the inclined shelf C is used.
An image is input by the visual sensor 1 at the lower position of the image. Only the visual sensor 1 is slid upward in parallel with the image processing for cucumber recognition using the input image at the lower position, and image input is performed by the visual sensor 1 at the upper position upon completion of the image processing at the lower position.

Upon completion of the image input at the upper position, as a result of the image processing at the lower position, if there is a cucumber in the image area, the harvest approach at the lower position is performed. When the harvesting at the lower position is completed and the result of the image processing at the upper position indicates that there is a cucumber in the image area, the manipulator 5 is slid upward to perform the harvesting approach at the upper position. . Cucumbers harvested by the harvest approach at the upper and lower positions are stored in the auxiliary storage case 4.

When the harvesting at the upper position is completed, the visual sensor 1 and the manipulator 5 are returned to their initial posture positions, respectively.
The work turning body 15 is horizontally rotated by approximately 180 degrees. At this time, the timing at which the auxiliary storage case 4 passes above the carry of the carry control section 16a is detected, and the detection is performed so that the auxiliary storage case 4 is stored in the auxiliary storage case 4. The cucumber is stored in the carry by opening the bottom of the case 4. With the completion of the turning of the work turning cylinder 15, the harvesting operation on the inclined shelf C of the cucumber on the opposite side is performed in the manner described above.

The harvesting operation in the infestation cultivation is shown in FIG.
As shown in the flowchart, the visual sensor 1 and the manipulator 5 are each set to the initial posture, and an image is input by the visual sensor 1 at the lower position of the inclined shelf C. As a result of the image processing for cucumber recognition using the input image at the lower position, if there is a cucumber in the image area, a harvest approach at the lower position is performed. The cucumber harvested by the harvest approach at the lower position is stored in the auxiliary storage case 4.

When the harvesting at the lower position is completed, the visual sensor 1 and the manipulator 5 are returned to their initial posture positions, respectively.
The work turning body 15 is horizontally rotated by approximately 180 degrees. At this time, the timing at which the auxiliary storage case 4 passes above the carry of the carry control section 16a is detected, and the detection is performed so that the auxiliary storage case 4 is stored in the auxiliary storage case 4. The cucumber is stored in the carry by opening the bottom of the case 4. With the completion of the turning of the work turning cylinder 15, the harvesting operation on the inclined shelf C of the cucumber on the opposite side is performed in the manner described above.

As described above, in the infestation cultivation, the cucumber can be harvested only at the lower position, so that the harvesting time can be shortened. However, in the pinching cultivation, the cucumber is harvested from the lower position to the upper position. Cucumber harvesting has to be done and requires a lot of harvest time. By processing the harvest time that requires much of this by the work process of the procedure shown in the flowchart of FIG. 13, the image input time at the upper position is parallel to the image processing time of the input image at the first lower position. And the image processing time of the input image at the upper position can be made substantially zero in parallel with the harvesting time of the manipulator 5 at the lower position. Can be greatly reduced. Since image input at the lower position and the upper position are both performed in the initial state where the manipulator 5 is stopped, there is no blurring or displacement of the image, and the manipulator 5
No misalignment in the direction of the harvest approach.

Image processing based on these input images is shown in FIG.
As shown in FIG. 4, in the reflectance at a wavelength of 550 nm, both cucumber and foliage show a low value of about 20 to 30%, but in the reflectance at a wavelength of 850 nm, the foliage is 50%.
%, The cucumber shows a high value of about 80%, and the density value ratio based on the reflectance is calculated as: density value (850) / density value (850) + density value (55
0) The luminance level (25)
By the calculated value multiplied by 5), a calculated image P as shown in FIG. 15 can be obtained.

As shown in FIG. 16, a binary image a obtained by binarizing a pixel having a density value (within a set range) for extracting cucumber as 1 is shown on the left side of the computed image P as shown in FIG.
On the right side of the figure, a binary image b obtained by binarizing a pixel having a density value (within the setting range) for extracting foliage and petiole as 1 is shown. , B, the upper left and right images a,
If the stem is behind the cucumber, as in b, the left image a
At the position of the cucumber recognized in the above, there is no stem in the right image b, so it is possible to approach the cucumber from directly in front, and as in the middle left and right images a and b, the cucumber is located behind the stem. In some cases, there is a stem in the right image b at the position recognized in the left image a, so it is necessary to approach the cucumber avoiding the stem, and the lower left and right images a, b When the stalk is on the left side of the cucumber, the cucumber is recognized in the left image a, and the stalk exists on the left side of the cucumber in the right image b. It can be judged that approaching from a direction is better.

As described above, by comparing the left and right images a and b, conventionally, only the cucumber is recognized from the binary image including the cucumber and a part of the foliage, so that the stem and petiole and the cucumber are recognized. The position relationship between the cucumber and the cucumber was first recognized by the left image a, and the cucumber was recognized by the right image b. By grasping the positional relationship between the stalk or petiole and the cucumber, which become obstacles during fruit picking, it is possible to determine an appropriate approach and determine whether or not harvesting is possible.

Further, when recognizing the stem, a binary image b (the right image shown in FIG. 16) obtained by binarizing the above-mentioned computed image P
Then, the number of pixels in the horizontal direction is calculated for each block figure, and a histogram is created. The average value or the peak value of the number of pixels is detected whether it is within the set value of the number of pixels indicating the stem, and it is determined whether or not the standard deviation is created and is within the set value. By recognizing the stem as a vertically elongated massive figure, the positional relationship with the cucumber can be grasped.

As described above, by grasping the positional relationship between the cucumber and the stem, it is possible to avoid damage to the stem by setting the approach direction accurately, so that it is possible to prevent the stem from being damaged and the tree vigor to be weakened. In addition, it is possible to determine whether or not harvesting is possible before approaching, so that the number of unnecessary approaches can be suppressed and the total harvesting time can be reduced. In addition, a clustered figure recognized as other than a cucumber is extracted from a binary image a obtained by binarizing the operation image P, and is recognized as a part other than a stem and a petiole from a binary image b obtained by binarizing the operation image P. By extracting a lump figure and recognizing the lump figure formed by ANDing both lump figures as leaves, if there is a stem or petiole in front of the cucumber, the cucumber can be recognized even if it can be recognized. In many cases, it is impossible, but when a part of the cucumber is hidden by the leaves, it is possible to harvest while pushing the leaves by the end effector 8. By recognizing the obstacle as a leaf, the obstacle can be used as a criterion for determining the approach direction, and the yield can be improved.

In this embodiment, the object to be harvested is limited to cucumber. However, it is needless to say that the gist of the present invention can be applied to fruits and vegetables other than cucumber.

[Brief description of the drawings]

FIG. 1 is a front view and a side view showing a detailed structure of a visual sensor.

FIG. 2 is a block diagram showing a control relationship of a visual sensor.

FIG. 3 is a work diagram showing an imaging state of a lower and upper inclined shelf by a visual sensor.

FIG. 4 is a work diagram showing a state of picking cucumber on the upper stage of the inclined shelf by the manipulator and an auxiliary storage state.

5A and 5B are a plan view and a side view showing a state in which a telescopic translation arm of the manipulator is extended.

FIGS. 6A and 6B are a plan view and a side view showing a state in which the cucumber is pinched and gripped by the manipulator to shorten the telescopic translation arm and the auxiliary storage case is extended.

FIG. 7 is a side view showing the whole of the agricultural robot.

FIG. 8 is a back sectional view showing the whole of the agricultural robot.

FIG. 9 is a block diagram showing a control relationship of the entire agricultural robot.

FIG. 10 is a block diagram showing a control relationship of a visual unit.

FIG. 11 is a block diagram showing a control relationship of a fruit cutting unit.

FIG. 12 is a flowchart showing a harvesting operation (pinching cultivation) procedure of the agricultural robot.

FIG. 13 is a flowchart showing a procedure of harvesting (panning cultivation) performed by the agricultural robot.

FIG. 14 is a diagram showing spectral reflection characteristics of cucumber and its foliage.

FIG. 15 is an image diagram showing a calculation image based on a luminance level.

FIG. 16 (a) is a binarized image obtained by binarizing an operation image mainly for extracting cucumber; (B) A binary image obtained by binarizing the operation image mainly for extracting foliage and petiole.

[Explanation of symbols]

 1. Vision sensor

Claims (1)

[Claims] 1. A first image input is performed by a visual sensor 1 that slides up and down to capture an image of a work object, and is built in an image memory in parallel with image processing of the first input image. An image pickup device for an agricultural robot that moves the visual sensor 1 upward or downward to enable a second image input.