JPH0585129B2 - - Google Patents

Description

[Detailed Description of the Invention] [Industrial Application Field] The present invention provides an imaging means for imaging fruits and vegetables to be worked, extracts a region corresponding to the fruits and vegetables based on the imaging information, and brightens the area within the region. an image processing means for determining the direction in which the visible highlighted portion is located; and a hand guiding means for guiding the working hand for capturing the fruits and vegetables in the direction in which the highlighted portion is located based on the detected information. The present invention relates to a guidance device for a working machine for fruits and vegetables provided with the following.

[Conventional technology]

In the above-mentioned guidance device for a working machine for fruits and vegetables, when a plurality of fruits and vegetables are crowded together, there is a problem that the plurality of fruits and vegetables are extracted as one area.

By the way, if we take fruits such as mandarin oranges and apples as an example, they are generally spherical in shape, so they are high in color, with the closest part of the surface appearing brighter than the surrounding area. It is known that there is a light part.

Therefore, by using the direction in which this highlighted part is located as information indicating the direction in which one fruit and vegetable is located, multiple fruits and vegetables extracted as one area can be separated and identified. Even when fruits and vegetables are crowded together, the working hand can be guided in the direction where one fruit and vegetable is located.

However, if multiple fruits and vegetables are located adjacent to each other, multiple highlighted portions may exist within one area, and in that case, which highlight portions It is necessary to decide whether to approach the parts first.

Conventionally, the closer the distance, the brighter the brightness appears, so the brightness of multiple highlight areas is compared and the brighter highlights are approached first. It has been considered to determine the approach order based on brightness.

[Problem that the invention seeks to solve]

However, when multiple fruits and vegetables are clustered together, the highlighted part corresponding to the fruits and vegetables located in the center of the cluster appears the brightest, and If you approach it first, there is a risk that surrounding fruits and vegetables will get in the way and cause you to fail to capture it.

In addition, when multiple fruits and vegetables are lined up in a row horizontally or vertically, the difference in distance to the imaging position is small, so there is no difference in the brightness of their highlighted parts. In that case, there is a disadvantage that the approach order cannot be determined using the difference in brightness of the highlighted portions.

The present invention has been made in view of the above circumstances, and its purpose is to reduce the number of failures in capture.
To enable accurately determining the order of approaches to a plurality of fruits and vegetables in close proximity.

[Means for solving problems]

A characteristic configuration of the guiding device for a working machine for fruits and vegetables according to the present invention is that when a plurality of the highlighted portions are present in the extracted region, the guidance device for the fruit and vegetable working machine according to the present invention Approach order determining means is provided for determining the approach order so that the work hand approaches the work hand earlier as the work hand is located closer to the outer periphery of the area or as the work hand is located furthest from the center of gravity of the area. Its functions and effects are as follows.

[Operation] When multiple fruits and vegetables are arranged in a horizontal direction and are crowded together to be extracted as one area, it is possible to approach the fruit from the edge of the fruit and vegetables, thereby increasing the capture speed. This means that you can approach vegetables that are less likely to get in the way first.

In other words, if the approach order is determined so that the highlighted part located on the outer periphery of the extracted area or the farthest from the center of gravity of the area is approached first, the number of other fruits and vegetables that may interfere with capture can be reduced. The fewer things you have, the more you can approach them first.
The approach order can be appropriately determined.

〔Effect of the invention〕

As a result, it has become possible to accurately determine the approach order for a plurality of fruits and vegetables, such that the fruits and vegetables located at positions with fewer capture failures are approached first.

〔Example〕

Embodiments of the present invention will be described below based on the drawings.

As shown in FIG. 5, a boom 1 is attached to a vehicle body 2 so as to be able to rise and fall and turn freely, an auxiliary boom 3 is attached to the boom 1 so as to be able to swing freely in the horizontal direction, and a work hand H for fruit harvesting is provided. A working manipulator 4 is attached to the tip of the auxiliary boom 3 to constitute a working machine for fruit harvesting.

In the figure, 5 is a hydraulic cylinder for raising and lowering the boom, and 6 is a hydraulic cylinder for raising and lowering the boom.
7 is an electric motor for swinging the boom, 7 is an electric motor for swinging the auxiliary boom, and K is a box for storing the harvested fruit.

The working manipulator 4 includes a multi-jointed telescopic arm 8 and a working hand H attached to the distal end of the telescopic arm 8. The telescoping arm 8 is rotated around the vertical axis X by the electric motor 9a, and swung around the horizontal axis Y by the electric motor 9b.
It is configured to be expanded and contracted at c.

To explain the working hand H, as shown in FIGS. 6 and 7, a vacuum pad 10 made of soft rubber is externally fitted onto the tip of the ventilation pipe 11 with its suction port facing the front side of the hand. A branch pipe 12 for ventilation is provided at the base of the ventilation pipe 11, a telescopic bellows type hose 13 is connected to the branch pipe 12, and the base end of the branch pipe 12 is It is bolted to the tip of the telescoping arm 8. However, the bellows type hose 13 is
As shown in FIG. 3, piping is connected to the intake pump 14.

Further, a stalk cutting device for cutting the stalk, or so-called stalk, of the fruit captured while being attracted to the vacuum pad 10, and a trapping part case 15 are provided.

To explain the trapping part case 15, as shown in FIGS. 6 to 8, it includes a proximal end portion 15a that is externally supported on the vent pipe 11 so as to be slidable in the front-back direction of the hand, and an upper portion on the distal side thereof. an upper cover portion 15b fixed to the base end portion 15a;
The lower cover portion 15c is fixed to the lower part of the tip side of the
An air cylinder 16 for sliding operation of the trapping part case 15 is interposed between the base end portion 15a and the branch pipe 12, and the suction port of the vacuum pad 10 is connected to the suction port of the vacuum pad 10. It is configured so that it can be freely switched between a state where it is located on the front side of the trapping part case 15 and a state where the fruit adsorbed by the vacuum pad 10 is taken into the inside of the trapping part case 15.

To explain the upper cover part 15b and the lower cover part 15c, they are formed into a hemispherical shape so as to cover the outer periphery of the hand, and a piece of fruit is placed in the part facing the suction port of the vacuum pad 10. An opening 26 for fruit passage is formed in a size that allows the fruit to pass therethrough. In addition, a slit-like groove 27 is formed in a portion of the upper cover portion 15b close to the opening 26, through which a fruit stalk is inserted. It is designed so that only the adsorbed fruits to be harvested can be taken into the trapping part case 15, and the handle can be smoothly moved when taking them in (see FIG. 8C).

Further, an opening for discharging the fruit with the stem cut off held in the trapping part case 15 is formed on the lower rear side of the lower cover part 15c, and an opening is formed for discharging the fruit downward. A shutter 29 is fixedly attached to the lower part of the proximal end portion 15a of the capture portion case 15 so as to be openable and closable by an air cylinder 30 for opening and closing.

To explain the handle cutting device, as shown in FIG. 6, both ends of the clipper-shaped cutter 17 in the shape of a strip and a semicircular arc, and the handle support member 18 in the shape of a strip and a semicircle, The cutter 17 and the handle support member 18 are pivotably attached to the proximal end portion 15a of the catcher case 15 so as to be swingable about the horizontal axis, and
A state in which the handle of the fruit caught in the trapping part case 15 is projected toward the front side of the hand so as to be supported between them, and a state in which it is retracted toward the rear side of the hand to form a passage for introducing the fruit between them. It is designed to be able to swing in and out so that it can be switched.

An electric motor 19 for swinging the cutter 17 and the handle support member 18, and an air cylinder 20 for sliding the cutter 17 are provided. The handle is supported between the cutter 17 and the handle support member 18, and the supported handle is cut by the sliding operation of the cutter 17 (see FIG. 8).

However, the cutter 17 and handle support member 18 are
When either the cutter 17 or the handle support member 18 contacts the handle first, the cutter 17 or handle support member 18 is connected to the electric motor 19 via the differential mechanism 21. With the raising operation of 18 stopped, the handle support member 1 on the other side
The eight-pronged cutter 17 can continue to be operated in an upright position, and the protruding positions of the cutter 17 and handle support member 18 can be automatically changed in accordance with the position of the captured fruit handle. .

Further, various sensors for controlling the operation of each part of the work hand H are provided.

As shown in FIG. 6, the vacuum pad 1
Negative pressure sensor that detects when fruit is absorbed by 1
S ₁ is attached to the branch pipe 12 .

As shown in FIGS. 6 and 7, a light reflection type proximity sensor S ₂ detects when the work hand H approaches the fruit to be harvested within a set distance, and
An image sensor _S3 serving as an imaging means for taking an image of the fruit to be harvested is provided inside the vacuum pad 10 with its detection direction facing toward the front side of the suction port of the vacuum pad 10, and the upper cover portion 15b A strobe light emitting device 28 that emits light at a set light amount in synchronization with the imaging processing by the image sensor S ₃ is provided at the rear of the flash light emitting device 28 .
It is designed to allow imaging processing to be performed with an appropriate amount of light regardless of the surrounding brightness.

Further, a potentiometer _S4 that detects the swinging positions of the cutter 17 and handle support member 18 is interlocked with the differential mechanism 21, and is connected to the differential mechanism 21 to determine when the raising operation or retraction operation of the cutter 17 and handle support member 18 is complete. It is designed so that it can be detected.

To explain the proximity sensor S ₂ , a sensor main body 22 housing a light emitting part, a light receiving part, a signal processing part therefor, etc. is held at the rear upper part of the branch pipe 12 , and the emitted and received light emitted from the sensor main body 22 is The optical fiber 23 for
The optical fiber 2 extends into the vacuum pad 10 through the inside of the ventilation pipe 11 and serves as a detection working surface of the proximity sensor _S2 .
The extending end of the image sensor S 3 is supported on the upper part of the image sensor S ₃ by the holder 24 for supporting the image sensor S ₃ in a state facing the front side of the hand.

Next, a description will be given of the configuration of a control means for causing the work hand H to approach a fruit to be harvested, cutting the stalk of the captured fruit, and harvesting the fruit.

As shown in FIG. 3, image processing means 1 performs image processing based on the imaging information from the image sensor _S3 to determine the direction in which the fruit to be harvested is located.
00, a guiding means 101 for guiding the work hand H toward the fruit to be harvested, and determining an approach order for a plurality of fruits based on the position of the highlight portion _P1 extracted by the image processing means 100; A control device 32 is provided, which constitutes each of the approach order determining means 102.

In other words, the control device 32 uses the imaging information of the image sensor S ₃ , the detection information of the negative pressure sensor S ₁ and the proximity sensor S ₂ , and the potentiometer.
Based on the detection information of _S4 and the information set and stored in advance, the controller 33 for driving the cylinder 5 for lifting and lowering the boom, the motor 6 for rotating the boom
, a drive controller 35 for the auxiliary boom swing motor 7, a drive controller 36a, 36b, 36c for each electric motor 9a, 9b, 9c of the telescoping arm 8, and a drive controller 36a, 36b, 36c for moving the catching part case 15 in and out. A control valve 37 for the operating air cylinder 16, a control valve 38 for the operating air cylinder 20 of the cutter 17, and a control valve 38 for the operating air cylinder 20 of the cutter 1.
7 and the electric motor 1 for raising and lowering the handle support member 18
9, a control valve 40 for controlling the suction operation of the vacuum pad 10, a control valve 41 for the air cylinder 30 for opening and closing the shutter 29, and each of the image sensor _S3 and the strobe light emitting device 28, Outputting an operation command, detecting the direction in which the fruit to be harvested is located, determining the approach order, and moving the work hand H
It is configured to automatically approach and harvest one fruit to be harvested, and to discharge the harvested fruit at a predetermined position.

However, the image sensor _S3 at the initial position
The imaging field of view includes the movement of the vehicle body 2, the boom 1
setting by changing the direction of the work hand H, that is, the vacuum pad 10, by turning or raising and lowering the auxiliary boom 3, horizontally swinging the auxiliary boom 3, and rotating the telescopic arm 8 with respect to the auxiliary boom 3 or swinging back and forth. become.

Next, the operation of the control device 32 will be explained based on the flowchart shown in FIGS. 4A and 4B.

First, the vacuum pad 10 is set at a preset initial position and facing a predetermined range in which the fruits to be harvested are located, and then the strobe light emitting device 28 is set so that its light intensity is maximum. Light is emitted in such a state that the image sensor _S3 performs imaging processing.

Next, the captured image information is taken in, backgrounds such as branches and leaves are removed, and an area corresponding only to the target fruit is created.
Image processing is performed to extract F ₀ (see Figure 2, b) to determine the direction in which the fruit to be harvested is located.

To explain about the image processing means 100 that removes backgrounds such as branches and leaves and extracts image information corresponding only to the target fruit, the image processing means ₁₀₀ extracts image information corresponding only to the target fruit. Binarization will be performed based on color, and by extracting a region F ₀ that corresponds only to the fruit F and finding the position of the center of gravity P ₀ , which is the geometric feature point of that region F ₀ ,
The direction in which the fruit to be harvested is located is determined based on the coordinate values Gx and Gy on the image in which the center of gravity P ₀ is located (see FIG. 2 B and C).

Next, since the position of the vacuum pad 10 in the determined direction, that is, the approach start position, and the center of the imaging field of view of the image sensor _S3 match, the vacuum pad ₁₀ is The work hand H is approached in a state of moving at high speed toward the direction in which the fruit to be harvested is located while maintaining the state in which the suction port 10 is directed toward the direction in which the center of gravity P ₀ is located, that is, toward the fruit to be harvested.

However, if multiple areas _F0 are recognized from the image information captured by the image sensor _S3 ,
Taking advantage of the fact that objects that are closer to each other appear brighter, the brighter objects are approached first based on the difference in average brightness of each of the extracted regions _F0 , or the objects located on the lower side are approached first. Multiple areas to approach as soon as possible
This will set the approach order among F ₀ . In other words, the closer to the vacuum pad 10 or the easier it is to capture the harvest, the earlier it will be harvested.

Then, after starting the approach, the proximity sensor
The telescopic arm 8 is arranged so that the work hand H moves at a low speed when it is detected that the S ₂ approaches the fruit to be harvested within a set distance.
After switching the extension speed to a low speed, the robot moves forward by a set distance that corresponds to the working distance of the proximity sensor _S2 until the negative pressure sensor _S1 detects that the fruit F has been adsorbed.

However, if the negative pressure sensor _S1 does not operate even if the work hand H is moved forward a set distance,
After stopping the operation of the telescoping arm 8 and once stopping the approach of the work hand H, the work hand H is moved back a set distance so that the amount of light received by the image sensor _S3 does not become saturated. In addition, the light amount of the strobe device 28 is reduced to a preset light amount based on the retreating distance and the working distance of the proximity sensor S ₂ so that the change in light amount with respect to the imaging distance becomes large, and the light amount is re-emitted. , reimaging processing is performed (see Fig. 8A).

Then, by performing image processing again on the re-imaged image information, one region F ₀ corresponding to the fruit is re-extracted, and within the re-extracted region F ₀ , the highlight that appears brightest is The approach direction is corrected so that the vacuum pad 10 approaches the highlighted portion P ₁ by extracting the portion P ₁ (see FIG. 2 C), and the proximity sensor S ₂ is turned ON again. ,
Then, the approach is restarted while moving at a low speed until the negative pressure sensor _S1 detects that the fruit F has been adsorbed.

However, if there are multiple highlighted portions P ₁ within the re-extracted area F ₀ (see Figure 2 C), it is determined that the multiple fruits are grown adjacent to each other in a plane. , as described below, based on the position of the highlighted portion P ₁ within the area F ₀ ,
The approach order is determined so that the fruit located at the edge with fewer other fruits getting in the way of the approach is approached first, and one highlighted part is
Only P ₁ will be selected.

To explain why a plurality of highlight parts _P1 are extracted, based on the brightness of each quantized pixel in one extracted region _F0 , the brightness is equal to or higher than a set threshold, and The coordinate values of each pixel that has the maximum value are calculated, but due to the fact that multiple fruits F are lined up and adjacent to each other with no difference in distance to them, multiple highlighted parts When the brightness of P ₁ becomes the same value,
There are cases where the brightest part cannot be distinguished, and in that case, it is necessary to decide which highlight part to approach.

Note that when a plurality of fruits grow adjacent to each other in the front-back direction, the amount of light reflected by the strobe light emitting device 28 is larger from the fruit located at the front than from the fruit located at the rear. Then, the position information for the fruit located on the rear side is automatically removed, and even if multiple fruits are adjacent to each other, only one highlighted portion _P1 is extracted, and the work hand H Only the position corresponding to one fruit located on the near side of the image will be automatically extracted.

When the proximity sensor _S2 is turned on again, the suction operation of the vacuum pad 10 is started, and the work hand H is moved forward at a low speed.

After the suction operation of the vacuum pad 10 starts, the work hand H is moved forward by a set distance (for example, a distance corresponding to the operating distance of the proximity sensor _S2 ) toward the fruit to be harvested (see FIG. 8B), and the Based on the detection information of the pressure sensor _S1 , it is determined whether or not the fruit has been adsorbed.

However, if the proximity sensor _S2 does not operate even if the approach distance exceeds the set value corresponding to the position where the telescoping arm 8 is extended to its maximum extension position, it is determined that harvesting is not possible, and the above-mentioned The telescoping arm 8 is shortened and the work hand H is returned to the approach starting position.

When the negative pressure sensor S ₁ is activated and the fruit F is adsorbed as desired, the vacuum pad 10 is stopped, and the trapping part case 1 is stopped.
5 protrudes forward to capture the fruit to be harvested into the trapping case 15 so that the fruit is located within the operating range of the handle cutting device (see Figure 8C). .

However, if the negative pressure sensor S ₁ does not operate,
Determining that it is impossible to harvest, the trapping part case 1
5 is retracted to the rear side, the suction operation of the vacuum pad 10 is stopped, the telescoping arm 8 is shortened, and the work hand H is returned to the approach starting position.

After the fruit is taken into the catcher case 15, the motor 19 for raising and lowering is driven in normal rotation to raise the cutter 17 and handle support member 18 forward. Then, based on the information detected by the potentiometer _S4 , the rising operation is temporarily stopped at an intermediate position on the front side of the rising operation completion position, and the vacuum pad 1
Stop the suction operation of 0 and move the entire work hand H.
By retreating a set distance toward the front and lower side of the fruit, the length of the handle to be cut by the cutter 17 is adjusted so that the cutter 17 comes into contact with the captured fruit. The position of the handle relative to the cutter 17 is corrected so that the length is as short as possible (see FIG. 8D).

Thereafter, based on the information detected by the potentiometer S ₄ , the cutter 17 and the handle support member 18 are continuously operated to stand up forward until the raising operation is completed, and the handle of the fruit is moved from the cutter 1 to the front side.
7 and the handle support member 18, and cut the handle with the cutter 17 (8th
(See Figure E).

However, the cutter 17 and handle support member 18 are
If it cannot be detected that the standing operation has reached that position even after the set time required to reach the completion position or the intermediate position has elapsed, it is determined that harvesting is not possible, and the operation is performed based on the information detected by the potentiometer _S4 . Then, the cutter 17 and the handle support member 18 are
A state in which the suction port of the vacuum pad 10 protrudes to the front side of the opening 26 of the trapping part case 15 by retiring it backward to the retirement completion position and retiring the trapping part case 15 to the rear side of the hand. After returning to the initial position, the telescopic arm 8 is shortened and returned to the approach starting position.

After cutting the handle, while maintaining the state in which the fruit is held in the catching part case 15, the telescopic arm 8 is shortened, the working hand H is returned to a predetermined position, and the shutter 29 is moved downward. By opening the fruit towards the vehicle body 2, the harvested fruits are discharged into the storage box K placed on the side of the vehicle body 2.

After the harvested fruit is discharged, the cutter 17 and the handle support member 18 are retracted rearward to the retracted completion position based on the information detected by the potentiometer _S4 , and the catcher case 15 is is retracted to the rear side of the hand, so that the suction port of the vacuum pad 10 is connected to the catching part case 1.
The harvesting operation for one fruit is completed by returning it to the initial position in which it protrudes forward from the opening 26 of No. 5.

After the harvesting operation for one fruit is completed,
Return the work hand H to the approach starting position,
Then, for example, it is determined whether or not the work has been completed by determining whether or not there are fruits to be harvested based on the information on the fruit F extracted based on the image information captured at the initial position. . If there is a fruit to be harvested, the next fruit F will be harvested by repeating the above-described series of processes.

Consequently, a series of processes for guiding the working hand H toward the direction in which the highlighted portion P ₁ is located corresponds to the hand guiding means 101 .

Next, based on the flowchart shown in FIG. 1, approach order determining means determines the approach order when there are a plurality of highlighted portions P ₁ in the area F ₀ (see FIG. 2 C). 102 will be explained.

In other words, the area F ₀ extracted in the re-imaging process
The number N of highlight parts P ₁ in the area, the number i indicating their order, and the coordinate value of each highlight part P ₁
Hxi, Hyi, and coordinate values Gx, Gy of the center of gravity P ₀
Based on , the distance Di of each of the plurality of highlight portions P ₁ to the center of gravity P ₀ is determined.

Then, by comparing the magnitudes of the obtained distances Di, and finding the number j of the one with the largest distance Di, the position farthest from the center of gravity P ₀ in the area F ₀ , that is, located at the end The highlighted part corresponding to the fruit is found, and the work hand H is made to approach the direction in which the highlighted part corresponding to the number j is located.

However, the approach order for other highlighted parts that have not been approached will be re-recognized during the next harvesting operation.

[Another example]

In the above example, the center of gravity P ₀ of the extracted area F ₀
In the above example, the approach order is determined based on the distance of the highlighted portion P ₁ to the area F ₀ , but for example, the position of the highlighted portion P ₁ in the longitudinal direction of the area F ₀ is Based on the distance from the outer periphery of the area F ₀ , so that the closer to the outer periphery the object approaches first,
The approach order may also be determined.

In addition, in the above embodiment, if acquisition fails in the first approach, the area extracted by re-imaging
Although we have illustrated a case in which the approach order is determined only when a plurality of highlight parts P ₁ exist within F ₀ , the approach order may be determined using image information captured at the initial position. After approaching the extracted region F ₀ to a predetermined distance, reimaging processing may be performed every time, and the approach order may be determined for each harvesting operation.

Further, in carrying out the present invention, various changes may be made to the configuration of each part, such as the specific configuration of the working hand H and the specific configuration of the telescoping arm 8 provided for guiding the working hand H toward the harvesting target.

Further, in the above embodiment, the case of harvesting fruits was illustrated, but the present invention can also be applied to harvesting of vegetables such as tomatoes. Further, the present invention can be applied to works other than harvesting work, and the present invention is not limited to the above embodiments.

Incidentally, although reference numerals are written in the claims section for convenient comparison with the drawings, the present invention is not limited to the structure shown in the accompanying drawings by the reference numerals.

[Brief explanation of the drawing]

The drawings show an embodiment of the guidance device for a working machine for fruits and vegetables according to the present invention, in which FIG. 1 is a flowchart for determining the approach order, FIG. 2 A, B, and C are explanatory diagrams of image processing, and FIG. 4A and 4B are flowcharts showing the operation of the control device; FIG. 5 is an overall side view of the fruit harvesting work machine; FIG. The figure is a front view of the same, and Fig. 8 A, B, C, D, and Ho are explanatory diagrams of the harvesting operation. H...Working hand, S ₃ ... Imaging means, F ₀ ...
Area corresponding to the target fruits and vegetables, P ₀ ...center of gravity,
P ₁ ... Highlighted part, 100 ... Image processing means, 101 ... Hand guiding means, 102 ... Approach order determining means.

Claims (1)

[Claims] 1 Imaging means S ₃ that images the fruits and vegetables to be worked on, and an area F ₀ corresponding to the fruits and vegetables based on the imaging information
an image processing means 100 that extracts the area F ₀ and determines the direction in which the highlight portion P ₁ that appears bright is located; A guiding device for a fruit and vegetable work machine is provided with a hand guiding means 101 for guiding in a direction in which a light portion P ₁ is located, wherein a plurality of highlight portions P ₁ are located within an extracted region F ₀ . If there is a highlighted part in the area F ₀
Based on the position of P ₁ , the closer to the outer periphery of the area F ₀ or the farthest from the center of gravity P ₀ of the area F ₀ the earlier the work hand H is approached. A guidance device for a working machine for fruits and vegetables, which is provided with an approach order determining means 102 for determining an approach order.