JP2020155060A - Recognition method and recognition device of head-forming center of head-forming vegetable, and harvester - Google Patents

Abstract

To provide a recognition method and a recognition device capable of shortening total processing time compared to a conventional device regarding a head-forming center of head-forming vegetable (especially, lettuce) planted in a field, and a harvester including the recognition device.SOLUTION: A recognition method of a head-forming center of a head-forming vegetable: creating a resized image reducing the number of RGB image pixels of the head-forming vegetable (an image acquiring step and an image resizing step); creating an edge mask image from the resized image (an edge step, a color tone mask step, an opening/closing step, and an edge mask step); carrying out normal line voting processing regarding the edge mask image and extracting a center candidate region by carrying out binarization processing by superposing of the normal lines (a normal line drawing step and a center candidate extracting step); and recognizing an outline center of gravity of the center candidate region as the head-forming center of the head-forming vegetable (a center recognition step).SELECTED DRAWING: Figure 1

Description

The present invention acquires an image of a heading vegetable such as lettuce, cabbage, and Chinese cabbage planted in a field, and obtains an image of the heading center of the heading vegetable, and more specifically, the heading center of the heading vegetable shown in the image. The present invention relates to a method for recognizing Chinese cabbage and a recognition device using the method.

Heading vegetables such as lettuce, cabbage, and Chinese cabbage are widely cultivated in the field in Japan. These headed vegetables are relatively heavy, and for example, manual harvesting work is a heavy labor because it is performed by bending over. Therefore, it is expected that labor saving will be established by mechanization of harvesting work.

In the mechanization of the harvesting work of headed vegetables, a technique of acquiring the position information of the headed vegetables planted in the field with high accuracy in real time and cutting one by one is required. Therefore, the harvester must be operated so that the harvesting blade hits the proper position of each strain. However, the planted plants are not necessarily located on the linear axis, and some individuals are located off the linear axis. Therefore, if a harvester equipped with a harvesting blade is simply moved straight along the line of headed vegetables, the harvesting blade may not hit the proper position for harvesting, or the harvesting blade may hit the wrong position and damage the leaves. .. From the above, if the position information of the headed vegetables being planted can be acquired with high accuracy in real time, it is possible to operate the harvesting blade of the harvester so as to hit the appropriate position of each strain according to the information. ..

Here, in Non-Patent Document 1 (Journal of Agricultural Machinery Society, Agricultural Food Engineering Society, July 1996, Vol. 58, No. 4, p.35-43), lettuce in the field is used as a video camera. An example of attempting recognition by two-dimensional image processing and an example of attempting recognition from a three-dimensional image using a three-dimensional visual sensor are described. However, in the example using a two-dimensional image, there is a problem in detecting the position and size as accurate as practical, and in the example using a three-dimensional image, the cost is high and the device tends to be large. There are challenges.

Takaemin, Takeshi Fujiura, Kiyoji Nakao, Makoto Dohi, "Study of selective harvesting of headed vegetables by robot (1st report) -2D recognition and 3D shape measurement-", Journal of Agricultural Machinery Society, Agricultural Food Engineering Society, Heisei July 2008, Vol. 58, No. 4, p. 35-43

Of the headed vegetables planted in the field, especially lettuce, it is difficult to distinguish between headed and outer leaves compared to cabbage, and it is difficult to detect the position and size as accurate as practical. At the same time, the device tends to be large. Therefore, the inventors of the present application have conducted diligent research on a method of recognizing the heading center of lettuce, with the aim of enabling the route generation of the harvester based on the information by recognizing the heading center of lettuce. .. As a result, we found a method with a relatively high discrimination rate, but the number of non-heading recognition points (the number of points that detect a position other than the heading center as the heading center) is high, the overall discrimination accuracy is low, and the total processing time Long was a challenge.

The present invention has been made in view of the above circumstances, and is a recognition method capable of shortening the total processing time and improving the identification accuracy of the heading center of headed vegetables (particularly lettuce) planted in the field as compared with the conventional method. , A recognition device, and a harvester equipped with the recognition device.

The present invention solves the above-mentioned problems by means of a solution as described below as an embodiment.

The method for recognizing the heading center of headed vegetables according to the present invention includes an image acquisition step of acquiring an RGB image of headed vegetables planted in a field and image resizing to create a resized image by reducing the number of pixels of the RGB image. The process, the edge step of performing edge detection on the resized image to create an edge image, and the resized image being masked by a hue with a threshold value to create a hue mask image. The hue mask step, the opening closing step of creating a hue mask filter image by performing either or both of the opening and closing processes on the hue mask image, and the hue mask for the edge image. An edge mask step of performing mask processing using a filter image to create an edge mask image, a normal drawing step of performing a normal voting process for each pixel in the edge mask image to create a normal image, and the normal line. A central candidate extraction step of extracting a central candidate region by performing a binarization process on the image by overlapping normal lines with thresholds, and using the contour center of the central candidate region as the heading center of the headed vegetable. It is a requirement to have a central recognition process for recognition.

According to this, by processing the resized image in which the number of pixels of the acquired image is reduced, the total processing time can be shortened as compared with the conventional case. Further, by reducing the number of pixels, the image can be roughened and the boundary region can be blurred, and the image can be smoothed. Therefore, the rounded edge makes it possible to draw the normal toward the center point of the head in the normal voting process, and the number of pixels is reduced to reduce the number of normals themselves to obtain a normal image. It is possible to suppress the complexity of. Therefore, it is possible to create a normal image capable of extracting a central candidate region more accurately than in the past. As a result, the number of recognition points outside the head can be reduced, and the identification accuracy can be improved.

Further, the heading center recognition device for headed vegetables according to the present invention controls the operations of a camera capable of acquiring RGB images, an image processing device, a calculation device, the camera, the image processing device, and the calculation device. The control unit is provided with a control unit for controlling the camera to acquire an RGB image of the headed vegetables planted in the field, and the image processing device controls the RGB image. A resized image is created by reducing the number of pixels, edge detection is performed on the resized image to create an edge image, and the resized image is masked by a hue with a threshold value. A hue mask image is created, the hue mask image is processed with either or both of opening and closing to create a hue mask filter image, and the hue mask filter image is applied to the edge image. A method in which an edge mask image is created by performing mask processing to be used, a normal voting process is performed for each pixel in the edge mask image to create a normal image, and a threshold value is provided for the normal image. Controlling to extract the central candidate region by performing binarization processing by overlapping lines, and controlling the arithmetic unit to calculate the contour center of gravity of the central candidate region as the heading center of the heading vegetable. Make it a requirement.

According to this, the position of the heading center of the headed vegetables planted in the field can be recognized accurately in a relatively short time.

Further, the headed vegetable harvester according to the present invention has a body that is equipped with a power source and can travel, a recognition device for the heading center of the headed vegetable according to the present invention, and a harvesting blade that cuts the stem of the headed vegetable. It is a requirement that a control unit that receives recognition information of the heading center of the headed vegetable from the recognition device and controls a route for positioning the harvesting blade along the stem of the headed vegetable is provided.

According to this, the position information of the heading center of the headed vegetables planted in the field is acquired with high accuracy in real time, and the harvesting blade is moved to the optimum position for cutting the stem of the headed vegetables according to the information. Becomes possible.

According to the present invention, a recognition method, a recognition device, and a recognition device capable of shortening the total processing time and improving the identification accuracy of the heading center of headed vegetables (particularly lettuce) planted in the field as compared with the conventional case. A harvester equipped with a recognition device becomes feasible.

It is a flowchart which shows the example of the recognition method of the heading center of the heading vegetable which concerns on embodiment of this invention. It is an image explaining from the image acquisition process to the image resizing process in the method of recognizing the heading center of a headed vegetable according to the embodiment of the present invention. It is an image explaining the edge root and the hue mask root in the method of recognizing the heading center of a headed vegetable which concerns on embodiment of this invention. It is an image explaining from the edge masking step to the center candidate extraction step in the method of recognizing the heading center of a headed vegetable which concerns on embodiment of this invention. It is an image explaining the center recognition process in the recognition method of the heading center of the headed vegetable which concerns on embodiment of this invention. It is another example 1 of the method of recognizing the heading center of a headed vegetable which concerns on embodiment of this invention, and is the flowchart which shows the example which includes the depth hue masking process. It is another Example 2 of the method of recognizing the heading center of the headed vegetable which concerns on embodiment of this invention, and is the flowchart which shows the example which includes the center identification process. It is an image which shows the result of Example 1 of the recognition method of the heading center of the headed vegetable which concerns on this invention. It is an image which shows the result of Example 5 of the recognition method of the heading center of the headed vegetable which concerns on this invention. It is an image which shows the result of the comparative example 1 of the recognition method of the heading center of the headed vegetable which concerns on this invention.

(How to recognize the heading center of headed vegetables)
The present invention is an invention of a method of acquiring an image of headed vegetables (particularly lettuce) planted in a field and recognizing the heading center of the headed vegetables shown in the image. Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a flowchart showing an example of a method of recognizing the heading center of headed vegetables according to the present embodiment. Further, FIG. 2 is an image illustrating the steps from the image acquisition step (S100) to the image resizing step (S102) in the recognition method according to the present embodiment. FIG. 3 is an image illustrating an edge root (R1) and a hue mask root (R2) in the recognition method according to the present embodiment. FIG. 4 is an image illustrating the steps from the edge mask step (S108) to the center candidate extraction step (S112) in the recognition method according to the present embodiment. FIG. 5 is an image illustrating the central recognition step (S114) in the recognition method according to the present embodiment. Further, FIG. 6 is a flowchart showing another example 1 of the recognition method according to the present embodiment, which includes a depth hue masking step. Further, FIG. 7 is a flowchart showing another example 2 of the recognition method according to the present embodiment, which includes a central identification step. The images displayed in all the drawings are displayed by appropriately enlarging or reducing the horizontal and vertical dimensions in order to make them easier to see or compare. Further, the images displayed in all the drawings are output for explanation, and it is optional whether or not to output (visualize) each image created in each step of the present invention.

Further, since the embodiment of the present invention is particularly preferably applied to a problem peculiar to lettuce, lettuce planted in a ridge covered with a multi-sheet in a field as an application target will be described as an example. The embodiment of the present invention can be applied not only to lettuce but also to various cultivation forms of cabbage, Chinese cabbage and other headed vegetables. Further, the "total processing time" in the text represents the total time of all the steps of the recognition method according to the embodiment of the present invention. The "identification rate" is a value [%] calculated by the identification rate = [(number of heads-undetected number) / number of heads] × 100. Here, the number of heads represents the number of lettuce to be recognized, and the undetected number represents the number of lettuce (center of head) that could not be detected. Therefore, the "discrimination rate" is an index indicating the strength of recognition of the head center. Further, the "identification accuracy" is a value [%] calculated by [(points within the heading center error 25.5 [pixels]) / (number of heads + number of non-heading recognition points)] × 100. Here, the non-heading recognition point represents a point in which a position other than the heading center is detected as the heading center, that is, a false recognition point, and specifically, a score exceeding the heading center error of 25.5 [pixels]. Therefore, "discrimination accuracy" is an index indicating the accuracy of recognition of the head center. Hereinafter, [pixel] is referred to as [px].
Hereinafter, each step will be described in detail with reference to the flowchart of FIG.

As a recognition method according to the present embodiment, first, an image acquisition step (S100) for acquiring an RGB image (FIG. 2A) of lettuce planted in a field is carried out. As a result, it is possible to acquire an object for which the recognition method according to the present embodiment is used. A known camera capable of acquiring an RGB image (FIG. 2A) can be used to acquire the image. The number of lettuce shown in the image is not limited, but it is usually considered to be about 2 to 6 when the above camera is mounted on a heading vegetable harvester and used for path generation of a forward harvester (this embodiment). In, it is set to 3). The RGB image (FIG. 2A) acquired in this step (S100) may be hereinafter referred to as "unconverted image (FIG. 2A)".

After the image acquisition step (S100), an image resizing step (S102) is carried out to create a resized image (FIG. 2B) by reducing the number of pixels of the RGB image (FIG. 2A), that is, the unconverted image (FIG. 2A). As a result, it is possible to perform arithmetic processing on the resized image (FIG. 2B) having a small amount of data in the processing after the next step. Therefore, the total processing time can be significantly shortened as compared with the method of performing arithmetic processing on the unconverted image (FIG. 2A). In FIG. 2, the unconverted image (FIG. 2A) and the resized image (FIG. 2B) are displayed in the same size (horizontal and vertical dimensions).

Further, the resized image (FIG. 2B) is rougher than the unconverted image (FIG. 2A) and the boundary region is blurred, and the lettuce head is closer to the rounded sphere. That is, according to this step (S102), the image can be smoothed by reducing the number of pixels.

The resize (reduction) of the unconverted image (FIG. 2A) is not limited to the bilinear interpolation method as an example in the present embodiment, and other processing for reducing the number of pixels may be used. Further, in the present embodiment, as an example, the image size (width x height) is 180 for the unconverted image (FIG. 2A) in which the image size (width x height) is 720 [px] x 540 [px]. It is resized to [px] x 135 [px], but it is not limited to this. The reduction ratio in this step (S102) will be described in detail in Examples described later. Further, when resizing, in the present embodiment, the aspect ratio is fixed as an example, but the aspect ratio is not limited to this, and may be changed as appropriate.

After the image resizing step (S102), an edge image (FIG. 3B) and a hue mask filter image (FIG. 3D) are created from the appropriately duplicated resizing image (FIG. 2B), respectively. Here, a series of steps for creating an edge image (FIG. 3B) from the resized image (FIG. 2B) is referred to as an edge root (R1), and a series for creating a hue mask filter image (FIG. 3D) from the resized image (FIG. 2B). This process is referred to as a hue mask root (R2). The edge route (R1) and the hue mask route (R2) may be performed first or later, or may be performed at the same time.

First, in the edge root (R1), an edge image (FIG. 3B) is created from the resized image (FIG. 2B) by the edge step (104a). Specifically, after the image resizing step (S102), an edge step (104a) is performed on the resized image (FIG. 2B) to perform edge detection and create an edge image (FIG. 3B). As a result, the circumference of the head (outline of the head) can be detected as an edge. In this embodiment, a sobel filter is used as an example of the filter used for edge detection, but the present invention is not limited to this. The Sobel filter calculates the horizontal and vertical gradients and calculates the square of the edge strength. Here, it is preferable to output in a squared state without further performing route calculation. As a result, the time spent on route calculation can be reduced and the total processing time can be shortened.

Here, in this step (104a), it is preferable to appropriately perform one or both of grayscale and smoothing on the resized image (FIG. 2B) before performing the edge detection. .. When both grayscale and smoothing are performed, either may be performed first or later. In the present embodiment, as an example, the resized image (FIG. 2B) is grayscaled and then smoothed. Next, the edge detection is performed on the “resized image (FIG. 3A) that has been smoothed” to create an edge image (FIG. 3B). However, grayscale and smoothing are arbitrary processes.

According to the grayscale, color information unnecessary for this step (S104a) can be removed. Further, according to smoothing, the image can be roughened to blur the boundary region, and the edges can be rounded. Here, especially for lettuce, since the leaf tip of the outer leaf has a shape finely carved into a saw blade shape, the leaf tip of the outer leaf is regarded as a high-frequency edge (position with strong edge strength) in edge detection described later. Easy to detect. As a result, in the normal voting process described later, the normal is drawn based on the value calculated as the circumference of the head (contour of the head) at such a position where the edge strength is relatively strong, so that the identification accuracy is finally lowered. Will be done. Therefore, by rounding the edges by smoothing, it is possible to suppress the detection of non-heading objects such as outer leaves as high-frequency edges, and it is possible to prevent a decrease in final identification accuracy.

In this embodiment, a Gaussian filter is used as an example of the filter used for smoothing, but the present invention is not limited to this. For example, an averaging filter or the like may be used. Further, when resizing the image, it is preferable to adjust the parameters of the filter used for smoothing to be appropriately small according to the reduction ratio of the image size. As a result, it is possible to prevent the edge distinction from becoming unclear due to excessive smoothing, or the edge from being enlarged too much. As a specific example, if the unconverted image (FIG. 2A) is smoothed, the filter size is 11 × 11, and the standard deviation is 11. If the resized image (FIG. 2B) is smoothed, the filter size is 3 × 3. , Standard deviation 3, and so on.

Next, in the hue mask root (R2), a hue mask filter image (FIG. 3D) is created from the resized image (FIG. 2B) by the steps from the hue masking step (104b) to the opening closing step (S106b). Specifically, after the image resizing step (S102), a hue masking step (104b) for creating a hue mask image (FIG. 3C) by performing mask processing with a hue set with a threshold value is performed. Specifically, as an example, an HSV image (not shown) obtained by HSV-converting a resized image (FIG. 2B) which is an RGB image is created, and then the HSV image is masked by hue to perform a hue mask image (FIG. 2B). Figure 3C) is created. However, the present invention is not limited to this, and for example, a method of performing HSL conversion instead of HSV conversion, or a method of calculating only the hue and performing mask processing based on the result to create a hue mask image (FIG. 3C) is used. You may. Here, a value of a predetermined color (for example, green), which is the color of the head, is set as the threshold value. Thereby, the head region can be extracted from the resized image (FIG. 2B). Further, by performing the mask processing based on hue, the influence of weather conditions such as fine weather, cloudy weather, and rainy weather can be suppressed as compared with the mask processing using RGB information, for example, and the heading region can be stabilized regardless of the weather. Can be extracted.

After the hue mask step (S104b), the hue mask image (FIG. 3C) is subjected to either or both of the opening and closing processes to create a hue mask filter image (FIG. 3D). S106b) is carried out. As a result, in the hue mask image (FIG. 3C), a part of the heads (mainly the region formed as a hole in the extracted region) that could not be extracted by the mask processing is filled as noise (head region). be able to. In addition, non-heading areas (for example, weeds and outer leaves, as well as the boundary area between the multi-sheet and the ground) extracted by the masking process can be erased as noise (areas other than the heading area).

The opening / closing process in this step (S106b) may be performed as appropriate for either or both of the opening and closing processes. Further, the opening generally means that the shrinkage treatment is performed a predetermined number of times, and then the expansion treatment is performed the same number of times as the shrinkage treatment. However, in the text including the scope of claims, the expansion treatment is followed by the shrinkage treatment. Including the case where it is performed a different number of times (including 0 times). Further, closing generally means that the expansion treatment is performed a predetermined number of times, and then the contraction treatment is performed the same number of times as the expansion treatment. However, in the text including the scope of claims, the contraction treatment is followed by the expansion treatment. Including the case where it is performed a different number of times (including 0 times).

Further, when resizing the image, it is preferable to adjust the parameters of the opening / closing process appropriately small according to the reduction ratio of the image size. As a result, it is possible to prevent the shape of the head from being maintained due to the excessive opening / closing treatment, or to prevent a part of the head (for example, a peripheral portion) from being erased. As a specific example, when the opening / closing process of the hue mask image (FIG. 3C) based on the non-conversion image (FIG. 2A) is performed, the closing is performed 3 times and then the contraction 3 times, and the opening is 7 times contraction and then 0 expansion. Do it times. On the other hand, when opening and closing the hue mask image (FIG. 3C) based on the resized image (FIG. 2B), the closing is performed once for expansion and then once for contraction, and the opening is performed for two contractions and then 0 expansions. It is in good condition.

After the edge root (R1) and the hue mask root (R2), the edge image (FIG. 3B) is masked using the hue mask filter image (FIG. 3D) to create an edge mask image (FIG. 4A). The edge mask step (S108) is carried out. As a result, the edge around the head (contour of the head) further selected can be detected by performing the mask processing in the head region extracted as the hue around the head (contour of the head) detected as the edge.

After the edge masking step (S108), a normal drawing step (S110) is performed to create a normal image (FIG. 4B) by performing a normal voting process for each pixel in the edge mask image (FIG. 4A). In the normal voting process in this step (S110), the thickness of the normal is determined by the brightness value provided with the threshold value. Specifically, since there is a high possibility that the position where the edge strength is relatively strong is the shadow around the head (outline of the head), the thickness of the normal is set thicker as the edge strength is stronger. That is, in the present embodiment, as an example, the brightness that correlates with the edge strength is set as a threshold value, and three types of thickness are set according to the brightness value, and the higher the brightness value, the thicker the setting. Further, when the brightness value is less than or equal to a predetermined value, the normal is not drawn (the process proceeds to the next pixel). As a result, the influence of voting can be greatly exerted on the position where the edge strength is relatively strong. The number of types of normal thickness is not limited. Further, even when the luminance value is equal to or less than a predetermined value (relatively low), the thickness of the normal may be relatively thin and the normal may be drawn for all the pixels.

The angle of the normal is determined by using a sobel filter. The horizontal and vertical gradients are calculated by the Sobel filter, and the normal angle is calculated. As for the direction of the normal, the center of the head is relatively brighter than the periphery of the head, so the direction of the normal is set to the direction of strong brightness. Further, the length of the normal is set to a predetermined length that reaches from the periphery of the head to the center of the head and does not become excessive in consideration of the image size, the size of the head that appears in the image, and the like.

With the thickness, angle, direction and length determined by the above method, one normal line is drawn for each pixel on an image for drawing (an image in which nothing is drawn). Such a normal voting process is performed on all pixels to create a normal image (FIG. 4B).

In the present embodiment, since the number of pixels is reduced by the image resizing step (S102) described above, the number of pixels for performing the normal voting process can be reduced. Specifically, in the reduction ratio (0.25) in the present embodiment, assuming that the normal voting process (processing from the calculation of each parameter of the normal to the normal drawing) in one pixel is one loop, the number of loops is set. It is possible to reduce it to 1/16. Thereby, the time (voting processing time) required for this step (S110) can be reduced. Further, when the image is not resized by the inventors of the present application (the image resizing step (S102) is not performed), 99 [%] or more of the total processing time (the center recognition step (S116) described later) is performed. Even in this case, it has been clarified that about 65 [%]) is the voting processing time. Therefore, by reducing the voting processing time by resizing the image, the total processing time can be significantly reduced.

Further, as described above, the image can be smoothed according to the image resizing step (S102). Further, in the edge step (104a), the resized image (FIG. 2B) is smoothed as an arbitrary process. Therefore, since the edges of the image are sufficiently rounded by these smoothings, the normal can be drawn toward the center of the head in the normal voting process, and the normal can be drawn by reducing the number of pixels. It is possible to reduce the number of itself and suppress the complexity of the normal image. Therefore, it is possible to create a normal image (FIG. 4B) capable of more accurately extracting the center candidate region as compared with the method without resizing the image. As a result, the number of recognition points outside the head can be reduced, and the identification accuracy can be improved.

When resizing the image, it is preferable to adjust the normal length and thickness parameters to be appropriately smaller according to the reduction ratio of the image size. As a result, it is possible to prevent the normals from overlapping excessively or protruding from the image. As a specific example, when the normal voting process of the edge mask image (FIG. 4A) based on the unconverted image (FIG. 2A) is performed, the length 120 [px] and the thickness 20 [px] · 40 [px] ·. When performing the normal voting process of the edge mask image (FIG. 4A) based on the resized image (FIG. 2B) for the three types of 60 [px], the length is 40 [px] and the thickness is 5 [px]. There are three types, 10 [px] and 15 [px].

After the normal drawing step (S110), the center candidate extraction step (center candidate extraction step) of extracting the center candidate region by performing a binarization process on the normal image (FIG. 4B) by overlapping normals provided with a threshold value. S112) is carried out. FIG. 4D shows an image (hereinafter, referred to as “center candidate image”) in which the center candidate region extracted by this step (S112) is output. In this step (S112), since there is a high possibility that the region where the overlap of normals is relatively strong is the center of the head, a binarization process is performed by overlapping the normals with a threshold value, and the normals are obtained. The region where the overlap of is relatively large is extracted as the central candidate region. The P-tile method is used for the binarization process. The P-tile method is a method of determining a threshold value at which the area after binarization is constant with respect to the total area. This makes it possible to prevent over-extraction of the central candidate region to be extracted. In the present embodiment, as an example, the area area after binarization is set to be 3 [%] with respect to the total area area, but the present invention is not limited to this. For example, it may be set to about 5 [%].

Here, in this step (S112), it is preferable to appropriately smooth the normal image (FIG. 4B) before performing the binarization process. In this embodiment, as an example, a normal image (FIG. 4B) is smoothed. Next, the above binarization process is performed on the "smoothed normal image (FIG. 4C)" to extract the central candidate region. However, smoothing is just an arbitrary process. According to the smoothing, it is possible to reduce the difference in the brightness of adjacent pixels and prevent a defect from occurring in the region to be extracted in the binarization process in the next step. Therefore, the heading region can be brought closer to a rounded sphere. Here, as the filter used for smoothing, an averaging filter is used as an example in the present embodiment, but the present invention is not limited to this. For example, a Gaussian filter or the like may be used. Further, when resizing the image, it is preferable to adjust the parameters of the filter used for smoothing to be appropriately small according to the reduction ratio of the image size. As a result, it is possible to prevent the shape of the region to be extracted in the binarization process of the next step from being relatively significantly changed due to excessive smoothing. As a specific example, when smoothing a normal image (FIG. 4B) based on an unconverted image (FIG. 2A), a normal image (FIG. 2B) based on a resized image (FIG. 2B) is used for a filter size of 11 × 11. When smoothing 4B), the filter size is 3 × 3, and so on.

After the center candidate extraction step (S112), a center recognition step (S114) for recognizing the contour center of gravity of the center candidate region as the heading center of the headed vegetable is carried out. In this step (S114), as an example, a threshold value (for example, 10 [px]) is appropriately provided for the size of the central candidate region extracted in the central candidate extraction step (S112) to make it relatively small. It is preferable to exclude the region. As a result, the jump value (outlier) extracted as a relatively small region can be removed as noise, and the heading center can be recognized more accurately. This filtering may be performed when calculating the contour center of gravity, or may be performed before or after the calculation of the contour center of gravity. Further, the threshold value may be appropriately changed according to the reduction ratio of the image size. However, filtering is just an arbitrary process.

Further, when the heading center recognized in this step (S114) is output (visualized), as an example, the RGB image (FIG. 2A) (non-conversion image (FIG. 2A)) of the headed vegetable acquired in the image acquisition step (S100) is used. It is preferable to hit and output. As a result, the center of the head can be spotted on the image whose image is not roughened by resizing, and the recognition result can be easily seen. FIG. 5 shows an image (hereinafter, referred to as “center recognition image”) output by spotting the heading center recognized in this step (S114) on a non-conversion image (FIG. 2A). However, the image on which the recognition result is punctured is not limited to the unconverted image (FIG. 2A), and may be arbitrarily selected. Further, the output format of the recognition result itself is not limited, and whether or not the recognition result itself is output is not limited.

According to the above steps, an image of the headed vegetables planted in the field can be acquired, the heading center of the headed vegetables reflected in the image can be recognized, the total processing time can be shortened as compared with the conventional case, and the total processing time can be shortened. It becomes possible to improve the identification accuracy.

As another example 1, after the image resizing step (S102), instead of the hue masking step (S104b), the resized image (FIG. 2B) is masked by a depth with a threshold value and a threshold. A depth hue masking step (S103b) is carried out to create a depth hue mask image (not shown) by performing mask processing with hues provided with values. Then, after the depth hue masking step (S103b), in the opening closing step (S106b), one or both of the opening and closing processes are performed on the depth hue mask image instead of the hue mask image (FIG. 3C). May be used to create a hue mask filter image (FIG. 3D) (see FIG. 6). According to the depth hue masking step (S103b) in this example, outer leaves and weeds having a height different from that of the head are excluded by masking the resized image (FIG. 2B) according to the depth in addition to the hue. Can be done. Therefore, it is possible to extract a more selected heading region as compared with the hue masking step (S104b) in which only the hue masking process is performed. The depth information in this example can be easily acquired in the image acquisition step (S100) by using a known camera capable of acquiring a depth image synchronized with the RGB image. As the camera, for example, Intel RealSense Depth Camera D435 (manufactured by Intel) or the like can be used. Further, as the depth threshold value, a predetermined depth may be set in consideration of the distance between the camera and the lettuce. For example, when the camera is mounted on a headed vegetable harvester and used to generate a route for a harvester moving forward, the acquisition distance may be set to about 500 [mm] to 780 [mm] as an example. Either the hue-based masking process or the depth-based masking process may be performed first or later.

Further, as another example 2, after the center recognition step (S114), identification processing by machine learning is performed on the contour center of gravity of the center candidate region recognized as the heading center of the headed vegetable in the center recognition step (S114). It may be a method of carrying out the center identification step (S115) for determining the heading center of the final headed vegetable (see FIG. 7). As a result, the non-heading recognition point can be excluded. However, the processing time becomes longer due to the central identification step (S115). It is also necessary to be aware of the possibility of overfitting. As an example of machine learning, a support vector machine, deep learning, or the like can be adopted. The center identification step (S115) of this example (another example 2) and the depth hue masking step (S103b) of the above-mentioned (another example 1) are both arbitrary steps. Moreover, each of these examples (another example 1 and another example 2) may carry out not only one but both.

Subsequently, a recognition device using the recognition method according to the present invention and a headed vegetable harvester equipped with the recognition device will be described.

(Recognizing device for heading center of heading vegetables)
First, a heading center recognition device for headed vegetables according to an embodiment of the present invention will be described. The heading center recognition device for headed vegetables according to the present embodiment includes a camera capable of acquiring RGB images, an image processing device, and an arithmetic unit. Further, it includes a control unit that controls the operation of these cameras, an image processing device, and an arithmetic unit. This control unit controls the camera to acquire an RGB image of the headed vegetables planted in the field. Further, the image processing apparatus is controlled to perform image processing by the same method as the above-mentioned recognition method, and the central candidate region is extracted. Further, the arithmetic unit is controlled to calculate the contour center of gravity of the center candidate region ring as the heading center of the headed vegetable. This makes it possible to accurately recognize the position of the heading center of the headed vegetables planted in the field in a relatively short time. When the recognition result is output, a monitor or the like for outputting the result is provided as appropriate.

(Heading vegetable harvester)
Next, the headed vegetable harvester according to the embodiment of the present invention will be described. The headed vegetable harvester according to the present embodiment has the above-mentioned recognition device, a harvesting blade for cutting the stem of the headed vegetable, and a heading center of the headed vegetable from the recognition device on a vehicle body that can run with a power source. A control unit that receives recognition information and controls a route for positioning the harvesting blade along the stem of the headed vegetable is provided. According to this, the position information of the heading center of the headed vegetables planted in the field is acquired with high accuracy in real time, and the harvesting blade is moved to the optimum position for cutting the stem of the headed vegetables according to the information. Becomes possible. More specifically, for example, the structure may be such that the traveling direction (advancing angle) of the vehicle body is changed by the control unit that receives the position information by the recognition device, or the harvesting blade is slid and moved in the left-right direction. It will be possible.

Subsequently, an example of the method for recognizing the heading center of the headed vegetable according to the present invention will be described. However, the scope of the present invention is not limited to the embodiment.

[Method]
Based on the above-described embodiment of the present invention, the heading center of lettuce planted in the field was recognized.
Here, as an example, the reduction ratio of the image size in the image resizing step (S102) is set to the value shown in Table 1 (Examples 1 to 9). Although only the width is shown in Table 1, the aspect ratio is always constant, and the height is also reduced according to the reduction ratio shown in Table 1 when resizing.
On the other hand, as a comparative example, an example in which the image resizing step (S102) is not performed, that is, the unconverted image (FIG. 10A) acquired in the image acquisition step (S100) is subjected to each processing in the subsequent steps while maintaining the original image size. (Comparative example 1). The size (width x height) of the unconverted image (FIG. 10A) is 720 [px] x 540 [px].

In all the examples and comparative examples, the same processing was performed in all the steps other than the image resizing step (S102). In the edge step (S104a), the resized image was grayscaled, then smoothed, and then edge detection was performed. A Gaussian filter was used for smoothing, and a Sobel filter was used for edge detection. Further, in the hue masking step (S104b), the image was converted to HSV, and the HSV image was masked. Further, a sobel filter was used to determine the normal angle in the normal drawing step (S110). In the center candidate extraction step (S112), the normal image was smoothed and then binarized. An average filter was used for smoothing, and a P-tile method was used for binarization. Further, in the center recognition step (S114), a relatively small area is excluded with the center candidate area 10 [px] as a threshold value. Other steps and treatments were carried out according to the above-described embodiment. However, as another example, the depth hue masking step (S103b) and the center identification step (S115) described above are not carried out.

[result]
The results are shown in Table 1 and FIGS. 8-10. Table 1 shows the identification rate [%], the identification accuracy [%], the total processing time [s], and the like of Examples 1 to 9 and Comparative Example 1. Further, FIG. 8 shows an image of the result of Example 1, and displays a resized image (FIG. 8A), a normal image (FIG. 8B), a center candidate image (FIG. 8C), and a center recognition image (FIG. 8D), respectively. ing. Further, FIG. 9 shows an image of the result of Example 5, and displays a resized image (FIG. 9A), a normal image (FIG. 9B), a center candidate image (FIG. 9C), and a center recognition image (FIG. 9D), respectively. ing. Further, FIG. 10 shows an image of the result of Comparative Example 1, and displays an unconverted image (FIG. 10A), a normal image (FIG. 10B), a center candidate image (FIG. 10C), and a center recognition image (FIG. 10D), respectively. are doing. Here, the center recognition images (FIGS. 8D, 9D, and 10D) are all images in which the recognition result (recognized position of the heading center) is dotted on the unconverted image (FIG. 10A). The images displayed in all the drawings are displayed by appropriately enlarging or reducing the horizontal and vertical dimensions in order to make them easier to see or compare.

As shown in Table 1, the larger the reduction ratio, the shorter the voting processing time and the total processing time. In addition, in all the examples, the number of non-heading recognition points decreased as compared with Comparative Example 1. Further, in each of Examples 2 to 9, the identification accuracy was improved while maintaining the identification rate at about the same level as compared with Comparative Example 1. From these results, it was shown that the voting processing time and the total processing time can be shortened by performing the image resizing (implementation of the image resizing step (S102)). Further, it is shown that by performing image resizing (implementation of the image resizing step (S102)), it is possible to reduce the number of recognition points outside the head while maintaining the identification rate, and thereby improve the identification accuracy. Was done.

Specifically, when FIG. 9 (Example 5) and FIG. 10 (Comparative Example 1) are compared, in Comparative Example 1 in which the image was not resized, the central candidate was compared with Example 5 in which the image was resized. It was shown that the extraction of the region was insufficient, which resulted in a high number of out-of-head recognition points. On the other hand, in Example 5 in which the image was resized, it was shown that the edges had moderate roundness and the central candidate region was extracted more accurately, which resulted in a reduction in the number of out-of-head recognition points. ..

Further looking at Table 1 in detail, in Examples 3 to 7 (the width of the unconverted image (FIG. 10A) was reduced from 720 [px] to 90 [px] to 240 [px]), the identification rate was 96 [%]. ] And the identification accuracy of 80 [%] were both exceeded and suitable. Further, in Examples 4 to 6 (the width is reduced from 720 [px] to 120 [px] to 180 [px]), both the identification rate 96 [%] and the identification accuracy 85 [%] are exceeded, which is more preferable. there were. In particular, in Example 5 (width reduced from 720 [px] to 150 [px]), both the identification rate (100.00 [%]) and the identification accuracy (86.54 [%]) are all Examples and Comparative Examples. It showed the highest value with respect to, and was the most suitable.

On the other hand, in Example 1 (the width was reduced from 720 [px] to 50 [px]), the identification rate and the identification accuracy were both lower than those in Comparative Example 1. In this regard, as shown in FIG. 8, if the reduction ratio is too large, the image becomes excessively coarse and the edges collapse, making it impossible to accurately extract the central candidate region, which causes undetection. In addition, as shown in Table 1, it was shown that the number of undetected spheres was large. Further, in Examples 8 to 9 (the width was reduced from 720 [px] to 360 [px] to 540 [px]), the degree of increase in the identification accuracy was lower than that in Examples 2 to 7. .. From the above results, regarding image resizing, if the reduction ratio is too large, the identification rate and identification accuracy are lowered, which is unsuitable. On the other hand, if the reduction ratio is too small, the effect of image resizing is not sufficiently exerted, which is inappropriate. It was shown to be.

Here, 1.0 [mm] ≈ 0.889 [px] (1.0 [px] ≈ 1.125 [mm] based on the actual size value in the unconverted image (720 [px] × 540 [px]). ) Was calculated. Therefore, the following numerical range is shown based on the results of this example, assuming that the diameter of the head of the headed vegetable is 150 [mm] (133 [px]). That is, in the image resizing step, the RGB image is preferably resized within the range of 15 [px] to 45 [px] in the head diameter of one strain of the headed vegetable, and more preferably 20 [. It is preferable to resize within the range of px] to 35 [px].

As described above, according to the method for recognizing the heading center of headed vegetables according to the present invention, it is possible to recognize the heading center of headed vegetables planted in a field with a relatively high discrimination rate and discrimination accuracy. .. In particular, by processing a resized image in which the number of pixels of the acquired image is reduced, the total processing time can be shortened as compared with the conventional case. Further, by reducing the number of pixels, the image can be smoothed, the image can be roughened, and the edges can be rounded. Therefore, it is possible to extract the central candidate region more accurately than in the conventional case, reduce the number of recognition points outside the head, and improve the identification accuracy.

Further, according to the heading center recognition device of the headed vegetable according to the present invention, the position of the heading center of the headed vegetable planted in the field can be recognized accurately in a relatively short time. Further, according to the heading vegetable harvester equipped with the recognition device, it is possible to acquire the position information of the heading center of the heading vegetables planted in the field with high accuracy in real time. Therefore, it is possible to move forward by the optimum route according to the information. In addition, the harvesting blade can be moved to the optimum position according to the information.

The present invention is not limited to the examples described above, and various modifications can be made without departing from the present invention. In particular, the present invention is preferably applied to lettuce, but it can also be applied to cabbage and Chinese cabbage.

Claims (10)

An image acquisition process for acquiring RGB images of headed vegetables planted in the field,
An image resizing step of reducing the number of pixels of the RGB image to create a resized image, and
An edge process of performing edge detection on the resized image to create an edge image, and
A hue masking step of creating a hue mask image by masking the resized image with a hue set with a threshold value, and
An opening / closing step of performing either one or both of the opening and closing processes on the hue mask image to create a hue mask filter image, and
An edge masking step of creating an edge mask image by performing mask processing using the hue mask filter image on the edge image, and
A normal drawing step of creating a normal image by performing a normal voting process for each pixel in the edge mask image,
A central candidate extraction step of extracting a central candidate region by performing a binarization process on the normal image by overlapping normals provided with a threshold value, and
A method for recognizing a heading center of a heading vegetable, which comprises a center recognition step of recognizing the contour center of gravity of the center candidate region as the heading center of the heading vegetable. The headed vegetable according to claim 1, wherein in the image resizing step, the RGB image is resized within the range of 20 [pixels] to 35 [pixels] in the heading diameter of one strain of the headed vegetable. How to recognize the center of the head. After the image resizing step and before the opening closing step, instead of the hue masking step,
The resized image is provided with a depth hue masking step of creating a depth hue mask image by performing mask processing by a depth with a threshold value and mask processing with a hue with a threshold value.
A claim characterized in that, in the opening closing step, a hue mask filter image is created by performing either or both of opening and closing processing on the depth hue mask image instead of the hue mask image. The method for recognizing the heading center of a heading vegetable according to claim 1 or 2. Claims 1 to 1, wherein in the edge step, one or both of grayscale and smoothing are performed on the resized image, and then edge detection is performed to create an edge image. 3. The method for recognizing the heading center of headed vegetables according to any one of the items. In the center candidate extraction step, the normal image is smoothed, and then binarization processing is performed by overlapping normals provided with threshold values to extract the center candidate region. The method for recognizing the heading center of a heading vegetable according to any one of claims 1 to 4. The heading center of a headed vegetable according to any one of claims 1 to 5, wherein in the normal drawing step, the thickness of each normal is determined by a brightness value provided with a threshold value. Recognition method. The method for recognizing a heading center of a headed vegetable according to any one of claims 1 to 6, wherein in the center candidate extraction step, the binarization process is performed by using the P tile method. After the center recognition step
Further, a center identification step of determining the final heading center of the heading vegetable by performing identification processing by machine learning on the contour center of gravity of the center candidate region recognized as the heading center of the heading vegetable in the center recognition step. The method for recognizing a heading center of a heading vegetable according to any one of claims 1 to 7, wherein the heading vegetable is provided. A camera that can acquire RGB images and
Image processing equipment and
Arithmetic logic unit and
The camera, the image processing device, and a control unit that controls the operation of the arithmetic unit are provided.
The control unit
The camera is controlled to acquire an RGB image of the headed vegetables planted in the field, and the image processing device is used to reduce the number of pixels of the RGB image to create a resized image.
Edge detection is performed on the resized image to create an edge image, and the image is created.
The resized image is masked by a hue with a threshold value to create a hue mask image.
The hue mask image is processed by either one or both of opening and closing to create a hue mask filter image.
An edge mask image is created by performing mask processing using the hue mask filter image on the edge image.
A normal voting process is performed for each pixel in the edge mask image to create a normal image.
The normal image is controlled to extract a central candidate region by performing a binarization process by overlapping normals provided with a threshold value, and the contour of the central candidate region is controlled by the arithmetic unit. A device for recognizing the heading center of a heading vegetable, which controls the calculation of the center of gravity as the heading center of the heading vegetable. For a car body that can run with a power source
The recognition device for the heading center of the heading vegetable according to claim 9,
A harvesting blade that cuts the stems of the headed vegetables,
A heading feature is provided with a control unit that receives recognition information of the heading center of the heading vegetable from the recognition device and controls a route for positioning the harvesting blade along the stem of the heading vegetable. Vegetable harvester.