JP2018206015A - Fruit detector and fruit detecting method - Google Patents

Abstract

To provide a fruit detector and a fruit detecting method that can distinguish between fruits subject to harvest and the highlight area of other than the fruits subject to harvest, and that can harvest the fruits subject to harvest in a stable manner by precisely detecting the fruits subject to harvest.SOLUTION: A fruit detector comprises: an image acquisition part 102 for acquiring an image; a bright spot detection part 105 for detecting the bright spot of the image obtained by the image acquisition part 102; a dispersion calculation part 106 for calculating the dispersion value of the color distribution in a prescribed area around the bright spot detected by the bright spot detection part 105; and a determination part 107 for determining whether or not the bright spot is a bright spot corresponding to fruits on the basis of the dispersion value calculated by the dispersion calculation part 106.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a fruit detection apparatus mounted on a harvesting robot or the like that automatically harvests fruits such as tomatoes, and a fruit detection method used in a harvesting robot or the like.

  The environment surrounding Japan's agriculture is in a very difficult situation due to the labor shortage due to the aging population. The number of farmers and new farmers is decreasing, the average age is aging, and the labor shortage is highlighted. As a result, the number of abandoned cultivated land increases, which adversely affects the local farming environment.

  Under such circumstances, dealing with labor shortages through automation of agriculture is an important issue, and according to the “2012 Robot Industry Market Trends” by the Ministry of Economy, Trade and Industry, domestic agriculture-related robots from 2018 to 2024 The market is expected to grow significantly and is expected to reach approximately 220 billion yen. For this reason, efforts are being made to automate the harvesting of fruits and the like, and development of the harvesting robot is underway.

  In the harvesting robot, image processing for identifying whether the fruit is fruit is necessary to harvest the fruit.

  In conventional image processing, image information obtained by an imaging unit that images a fruit to be harvested is processed, and a specific color area corresponding to the color of the fruit to be harvested and brightness within the specific color area Processing for obtaining a large highlight area is performed.

  Then, in order to detect a fruit to be harvested, when a plurality of highlight areas are detected in the same specific color area, a virtual area having a predetermined shape inscribed in the outline of the specific color area centered on the highlight area There is a method of detecting fruits to be harvested by estimating those virtual regions as specific color regions corresponding to the highlight regions, that is, regions corresponding to fruits (for example, patents) Reference 1).

Japanese Patent No. 2813697

  However, in the conventional detection method, when the color of the fruit to be harvested and the part other than the fruit to be harvested, for example, the leaves and stems are the same color as the fruit to be harvested, the highlight region is located at a part other than the fruit to be harvested. When detected, a part other than the fruit to be harvested is erroneously detected as a fruit.

  The present invention solves the above-mentioned problem, and makes it possible to distinguish between a fruit to be harvested and a highlight area other than the fruit to be harvested (hereinafter referred to as “bright spot”), and is a target to be harvested. It is an object of the present invention to provide a fruit detection apparatus and a fruit detection method that enable a fruit to be harvested stably by reliably detecting a fruit.

  In order to achieve the above object, the fruit detection device of the present invention includes an image acquisition unit that acquires an image, a bright spot detection unit that detects a bright spot of the image acquired by the image acquisition unit, and the bright check. A dispersion calculating unit that calculates a dispersion value of a color distribution in a predetermined region centered on the bright spot detected by the output unit; and the bright spot is a fruit based on the dispersion value calculated by the dispersion calculating unit And a determination unit for determining whether or not the bright spot corresponds to the above.

  In order to achieve the above object, the fruit detection method of the present invention includes an image acquisition step of acquiring an image, a bright spot detection step of detecting a bright spot of the image acquired by the image acquisition step, A dispersion calculating step for calculating a dispersion value of a color distribution in a predetermined region centered on the bright spot detected by the bright spot detecting step, and the bright spot based on the dispersion value calculated by the dispersion calculating step Determining whether or not is a bright spot corresponding to a fruit.

  According to the fruit detection device and the fruit detection method of the present invention, it becomes possible to distinguish the fruit to be harvested from the bright spot other than the fruit to be harvested, and to reliably detect the fruit to be harvested. A stable harvest of berries.

The block diagram which shows an example of a structure of a fruit detection apparatus The flowchart which shows an example of the process sequence of a fruit harvesting process The figure explaining the flow which calculates the threshold value of hue Illustration explaining how to grow a bunch The figure explaining the composition of a bunch An example of an IR image obtained by irradiating IR light is shown. The figure which shows the field which calculates the dispersion value of hue Diagram showing the distribution of hues in the bright spot area of fruits Diagram showing the distribution of hues in bright spot areas other than fruits

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the drawings, the same components are denoted by the same reference numerals.

  First, in order to help the understanding of the fruit detection method according to the embodiment of the present invention, the form of tomatoes to be harvested will be described with reference to FIGS. FIG. 4 is a diagram for explaining how a tuft grows, and FIG. 5 is a diagram for explaining the configuration of the tuft.

  As shown in FIG. 4, the tomato that is the harvest target of the present invention is composed of a plurality of fruits, and grows like a bunch of grapes. And the tuft grows so that the orchard 305 continues to the main stem 201. Also, when harvesting, the degree of maturity of the fruits attached to the bunches may differ, and instead of harvesting them as bunches, they are harvested by picking up the red ripe fruits to be harvested one by one. In other words, it is necessary to detect red ripe fruits to be harvested when tearing.

  The configuration of the tomato bunch will be described with reference to FIG. As shown in FIG. 5, one tuft is configured around a stem called the fruit fold 305. From this incision 305, small infarction 301 and gaku 303 are grown, and a fruit 304 is growing ahead. There is a portion called a delamination 302 in the middle portion of the prunus 301, and when the force is applied to the prunus 301, the prunus 301 is divided by the delamination 302.

  Further, in order to detect the fruit to be harvested and harvest the detected fruit, it is necessary to know what kind of structure the fruit is growing in the bunch. This is because it is relatively difficult to harvest the fruit at the bottom of the bunches. Even if the fruit is red and ripe, it must be harvested as long as it grows at the bottom of the bunches. This is because it may be necessary to judge.

  Therefore, it is determined whether or not red ripe fruits are located in the lower part of the tress by detecting all the berries growing in the tress, including the green fruits.

  That is, by knowing the position of the fruit in the whole cluster, if there is a green fruit at the bottom of the cluster, the detected red fruit is at the top of the green fruit, that is, not growing at the bottom of the cluster I understand that. In this case, it is determined that the red fruit is easy to harvest, and the red fruit is harvested.

  In addition, if there are no green fruits at the bottom of the bunches, among the detected red fruits, the red berries located at the bottom are at the bottom of the bunches, so it may be determined that the red fruits will not be harvested. is there.

  Thus, in the embodiment of the present invention, it is necessary to detect a green fruit. In addition to the green fruit growing in the bunches, there are also green spots such as the main stem 201 around it, so that the image acquisition unit 102 (see FIG. 1) described later can display the main stem 201 or the like. Acquire images.

  And when a bright spot appears in the location of the main stem 201, in the conventional method, since it is judged that it is a fruit only by highlight area, ie, the presence or absence of a bright spot, the main stem 201 which is the same green. Is erroneously determined to be the fruit 304.

  Further, if the color of the fruit to be detected is limited to red in order not to mistakenly determine the main stem 201 as the fruit 304 to be harvested, the green fruit cannot be detected. That is, this method detects only the fruit and prevents a part other than the fruit, for example, the main stem 201 from being erroneously detected as the fruit to be harvested.

  The structure of the fruit detection apparatus which implement | achieves this method is demonstrated using FIG. This fruit detection apparatus is mounted on a fruit harvesting apparatus or the like that harvests fruits. FIG. 1 is a block diagram illustrating an example of a configuration of a fruit detection apparatus.

  The fruit detection apparatus includes a light source 101, an image acquisition unit 102, an image processing unit 103, and a determination unit 107. The image processing unit 103 includes a hue conversion unit 104, a bright spot detection unit 105, and a variance calculation unit 106.

  The light source 101 irradiates light toward the bunch of tomatoes to be harvested. The image acquisition unit 102 images a region irradiated with light.

  Here, an IR (InfraRed) light source is used as the light source 101 in order to detect the fruit 304, but a light source corresponding to the fruit to be harvested is used in order to make bright spots appear easily on the fruit to be harvested. May be.

  The image acquisition unit 102 uses a camera that can acquire RGB (Red, Green, Blue) color images and IR images. The image acquisition unit 102 inputs the acquired RGB color image and IR image to the image processing unit 103.

  The image processing unit 103 performs image processing described below, and the determination unit 107 determines whether the imaged object is a fruit to be harvested based on an image obtained as a result of the image processing, and detects the fruit. .

  Next, the process procedure of the fruit harvesting process concerning this embodiment is demonstrated using FIG. FIG. 2 is a flowchart illustrating an example of a processing procedure of the fruit harvesting process.

  First, IR light is emitted from the light source 101 (step S10). The image acquisition unit 102 acquires the RGB color image and the IR image at the same timing and on the same axis (step S10). The RGB color image is obtained by reflection by sunlight, and the IR image is obtained by reflection of IR light emitted from the light source 101.

  In the present invention, a camera that acquires the RGB color image and the IR image at the same timing and coaxial is used as the image acquisition unit 102, but the camera that acquires the RGB color image and the camera that acquires the IR image are separate cameras. There may be.

  In addition, in the present invention, the reflected light of the light emitted from the light source 101 is used because the bright spot where the image acquisition unit 102 appears more noticeably by irradiating light from the direction of the image acquisition unit 102. This is because the image can be captured.

  That is, sunlight is emitted from all directions to the tuft and the main stem 201, and the bright spot is difficult to appear clearly. In addition, by using IR light, the image acquisition unit 102 captures light of a specific wavelength, so that there is an advantage that the influence of disturbance light (such as irregular reflection of sunlight) can be reduced.

  Subsequently, the hue conversion unit 104 converts the color representation of the RGB color image obtained by the image acquisition unit 102 into a hue representation (step S11).

  Further, the bright spot detection unit 105 detects a bright spot from the IR image obtained by the image acquisition unit 102 (step S12).

  FIG. 6 is a diagram showing an example of an IR image obtained by irradiating IR light. As shown in FIG. 6, the brightness of the IR image is higher than the surroundings at the bright spots such as the fruit bright spot 401 and the main stem bright spot 402.

  The fruit part bright spot 401 is a part where the brightness is increased by reflection of IR light in the convex part of the fruit. The main stem portion bright spot 402 is a portion where the bunch of fruit 305 born on the main stem 201 is cut off, and the luminance is high due to reflection of IR light.

  In addition to the fruit portion bright spot 401 and the main stem bright spot 402, bright spots may occur in the gourd 303 and leaves. Here, the fruit spot bright spot 401 and the main stem bright spot 402 will be described.

  Next, the variance calculation unit 106 acquires the hue information of a predetermined region centered on the bright spot detected by the bright spot detection unit 105 from the hue conversion unit 104, and calculates the hue variance value from the information ( Step S13).

  FIG. 7 is a diagram illustrating a region for obtaining a hue dispersion value. 7 is an area for calculating the hue dispersion value. The variance detection unit 106 calculates a variation in the frequency of hues appearing in this region as a variance value. In FIG. 7, an area 501 for calculating the hue dispersion value of the fruit portion luminescent spot 401 and an area 502 for calculating the hue dispersion value of the main stem luminescent spot 402 are shown.

  In the present embodiment, a rectangle having a short side as an average value of the radii of a plurality of fruits 304 measured in advance is used as the predetermined region for calculating the hue dispersion value. However, a square may be used instead of such a rectangle. Further, the length of the short side may be set in consideration of the size and shape of the bright spot or the fruit 304.

  FIG. 8 is a diagram showing the hue distribution of a region centering on the bright spot of the fruit 304, that is, the fruit bright spot 401. FIG. Since the fruit 304 becomes uniform red or green, the variation in the distribution of hue frequency is small.

  That is, red fruits have a high frequency of red as a hue, and green fruits have a high frequency of green as a hue. Therefore, in any case, variation in the frequency distribution becomes small and the variance value becomes small.

  FIG. 9 is a diagram showing a hue distribution in the bright spot region other than the fruit 304. Here, the distribution of hues in the region centered on the bright spot appearing on the main stem 201, that is, the main stem bright spot 402 is shown.

  The main stem 201 includes red and blue color components in addition to the green color component. In particular, the cut portion after cutting off the curd 305 is changed to white or brown, and many red and blue color components are included in addition to the green color component.

  In addition, the main stem 201 may be thinner than the radius of the fruit 304. In this case, since the diameter of the main stem 201 is shorter than the length of the short side of the set rectangle, the background of the main stem 201 is also reflected in the set rectangular area.

  As a result, color components other than the green color of the main stem 201 are likely to be included, the variation in hue distribution in the region centered on the bright spot appearing on the main stem 201 becomes large, and the variance value also increases.

  The determination unit 107 determines whether or not the hue dispersion value of each bright spot calculated by the dispersion calculation unit 106 is equal to or greater than a threshold value (step S14).

  If the hue variance value of each bright spot calculated by the variance calculation unit 106 is less than the threshold value (NO in step S14), the determination unit 107 determines that the bright spot is the bright spot of the fruit 304. (Step S15).

  If the variance value is equal to or greater than the threshold value (YES in step S14), determination unit 107 determines that the bright spot is a bright spot other than fruit 304 (step S16), and ends the fruit harvesting process as it is. .

  When it is determined in step S15 that the bright spot is a bright spot of the fruit 304, the determination unit 107 determines whether the hue corresponding to the position of the bright spot is red, thereby determining the bright spot. It is determined whether or not the corresponding fruit 304 is a red fruit to be harvested (step S17).

  If the fruit 304 corresponding to the bright spot is a red fruit to be harvested (YES in step S17), the fruit harvesting device equipped with the fruit detection device harvests the fruit (step S18). ).

  If the fruit 304 corresponding to the bright spot is not a red fruit to be harvested (NO in step S17), the fruit harvesting process is terminated without performing harvesting as an immature fruit 304.

  The fruit 304 is harvested using an actuator or the like. In that case, the fruit harvesting device obtains the position of the RGB color image corresponding to the position of the bright spot of the fruit 304 to be harvested from the image information, operates the actuator with that position as the target position, and harvests the fruit 304. .

  In addition, the fruit harvesting device can operate the actuator with higher accuracy by using a sensor capable of measuring the distance and obtaining the distance to the fruit to be harvested.

  Here, the threshold value calculation method used in step S14 in FIG. 2 will be described with reference to FIG.

  First, the variance calculation unit 106 calculates the variance value of the hue in the region centered on the bright spots of the plurality of fruits 304 in advance (step S20). In addition, the variance calculating unit 106 calculates the variance value of the hue in the region centered on the bright spot generated in a place other than the fruit 304 (step S21).

  Then, the variance calculating unit 106 calculates the maximum value Va of the hue dispersion value in the region centered on the bright spot of the fruit 304 (step S22), and further the variance of the hue in the region centered on a place other than the fruit 304. The minimum value Vb is calculated (step S23).

Then, the variance calculation unit 106 obtains the threshold value Vt from the following formula 1 (step S24).
Vt = (Va + Vb) / 2 (1)

  The fruit 304 for which the variance value is calculated includes not only the fruit 304 to be harvested that is ripe in red, but also the fruit 304 that is green and immature for harvesting. This is because it is necessary to detect all the fruits 304 growing in the bunches as described above.

  Further, the hue dispersion value of the bright spots other than the fruit 304 is also calculated. Specifically, the image acquisition unit 102 previously captures a part other than the tuft, for example, the main stem 201 and leaves, and the equipment of the cultivation facility of the fruit 304 reflected behind the tuft, and the bright spot detection unit 105 shines. The point is detected, and the variance calculation unit 106 obtains the variance value of the hue in the area centered on the detected bright spot.

  In this embodiment, the hue conversion unit 104 is provided to convert the color expression of the RGB color image into the hue expression. However, the hue conversion unit 104 is not provided, and each color of R, G, and B in the RGB color image is provided. The dispersion value of the value calculated from the component value or the value of each color component of R, G, B is calculated as the dispersion value of the color distribution, and based on the dispersion value, the bright spot corresponds to the fruit. It may be determined whether or not.

  In this case, in step S13 of FIG. 2, the variance calculating unit 106 calculates the variance value of the color distribution from the RGB values of a predetermined area centered on the bright spot detected by the bright spot detecting unit 105. In step S15, the determination unit 107 determines whether the dispersion value of the color distribution of each bright spot calculated by the dispersion calculation unit 106 is equal to or greater than a threshold value.

  Note that, in the present embodiment, description has been given by taking, as an example, a fruit that grows in a tuft, but the scope of application of the present invention is not limited to a fruit that grows in a tuft, and the color of the fruit to be harvested Is not limited to red.

  Further, the present invention can be applied not only to tomatoes that are in bunches, but also to other berries that have gathered in the fruit stem and constitute bunches, as well as in the case where the fruits are growing individually.

  The present invention is suitable for use in a fruit detection device that detects fruits.

DESCRIPTION OF SYMBOLS 101 Light source 102 Image acquisition part 103 Image processing part 104 Hue conversion part 105 Bright spot detection part 106 Variance calculation part 107 Judgment part 201 Main stem 301 Small pedicel 302 Delamination 303 Gaku 304 Fruit 305 Infection 401 Fruit part bright spot 402 Main Stem luminescent point 501 Area for calculating the variance value of the hue of the fruit 502 Area for calculating the variance value of the hue of the main stem

Claims (5)

An image acquisition unit for acquiring images;
A bright spot detection unit for detecting a bright spot of the image acquired by the image acquisition unit;
A dispersion calculation unit that calculates a dispersion value of a color distribution in a predetermined region centered on the bright spot detected by the bright spot detection unit;
A determination unit that determines whether the bright spot is a bright spot corresponding to a fruit based on the variance value calculated by the variance calculation unit;
A fruit detection apparatus comprising:   The determination unit compares the variance value calculated by the variance calculation unit, the variance value of the fruit color distribution, and a threshold value determined from the variance value of the color distribution other than the fruit, The fruit detection apparatus according to claim 1, wherein it is determined whether or not the bright spot is a bright spot corresponding to a fruit.   The fruit detection apparatus according to claim 1, wherein the bright spot is an area in which luminance generated by reflection of light emitted from a light source is larger than that of the surrounding area.   A hue conversion unit that converts the color representation of the image acquired by the image acquisition unit into a hue representation; and the distribution calculation unit calculates a variance value of the distribution of the hue in a predetermined region centered on the bright spot. The fruit detection apparatus of any one of Claims 1-3 calculated. An image acquisition step of acquiring an image;
A bright spot detecting step for detecting a bright spot of the image acquired by the image acquiring step;
A dispersion calculating step for calculating a dispersion value of a color distribution in a predetermined region centered on the bright spot detected by the bright spot detecting step;
A determination step of determining whether the bright spot is a bright spot corresponding to a fruit based on the variance value calculated in the variance calculating step;
A fruit detection method comprising: