JP7479007B2 - Information processing device, program, system, and method for detecting grapes from an image - Google Patents

Description

The present invention relates to an information processing device, program, system, and method for detecting grapes from an image.

Patent Document 1 discloses a grape counting device that places a bunch of grapes in a photography space created by partitions and counts the number of grapes through image analysis.

Patent Publication 2019-200563

However, in order to measure the grapes on the grape bunches without interrupting the normal flow of the picking work, it was necessary to be able to detect the grapes belonging to the bunch being worked on even from an image that contains multiple grape bunches, similar to the field of vision of the picker, without placing them in the shooting space created by a partition or the like.

The present invention provides an information processing device, program, system, and method that can detect grapes belonging to a bunch of grapes being worked on from an image that includes multiple bunches of grapes.

According to the present invention, there is provided an information processing device that detects grapes from an image, the information processing device comprising a grape detection unit, a bunch identification unit, and an integration processing unit, the grape detection unit detects the grapes contained in the image, the bunch identification unit detects the grape bunches contained in the image, and identifies the bunch of grapes currently being worked on from the detected bunches based on the positions of the bunches of grapes and the sizes of the bunches of grapes in the image, and the integration processing unit determines the grapes that belong to the bunch of grapes currently being worked on based on the results of the grape detection and the results of the grape bunch identification.

In the present invention, grape berries and grape bunches are detected, and the bunch being worked on is identified from the detected bunches based on the position and size of the bunch, and the grapes belonging to the bunch being worked on are determined. Therefore, even if the bunch being worked on is not placed in a shooting space created by a partition or the like, it is possible to identify the bunch being worked on from an image containing multiple bunches and detect the grapes belonging to the bunch being worked on.

Various embodiments of the present invention will be described below. The embodiments described below can be combined with each other.
Preferably, in the information processing device, the grain detection unit detects the grape grains contained in the image based on a first learning model, and the first learning model is a learning model including an object detector that detects objects using features of the image, and a classifier that classifies objects using the features of the image.
Preferably, in the information processing device, the classifier included in the first learning model has a non-positional classifier that is a classifier that does not use location information as a feature for classification, and a positional classifier that is a classifier that uses location information as a feature for classification, and classifies based on the classification results by the non-positional classifier and the classification results by the positional classifier.
Preferably, in the information processing device, the bunch identification unit detects grape bunches included in the image based on a second learning model, the second learning model being a learning model including an object detector that detects objects using features of the image and a classifier that classifies objects using the features of the image, and the second learning model being the same as or different from the first learning model.
Preferably, the information processing device further includes a grape number calculation unit that counts the number of grapes detected in the image as grapes belonging to the bunch of grapes under work, and calculates a range of the total number of grapes in the bunch of grapes based on the counted number of grapes and a predetermined coefficient.

Furthermore, according to the present invention, there is provided a system for detecting grapes from an image, the system comprising an image capturing unit and an image analysis unit, the image capturing unit captures the image, the image analysis unit comprises a grape detection unit, a bunch identification unit and an integration processing unit, the grape detection unit detects the grapes contained in the image, the bunch identification unit detects the grape bunches contained in the image, and identifies a bunch of grapes currently being worked on from the detected bunches based on the positions of the bunches and the sizes of the bunches, and the integration processing unit determines which grapes belong to the bunch currently being worked on based on the results of the grape grain detection and the results of the bunch identification.
Preferably, the system further comprises an analysis result display unit, the analysis result display unit displaying the range of the total number of grapes in the grape bunch under operation.

The present invention also provides a program for detecting grapes from an image, which causes a computer to execute a grape detection step, a bunch identification step, and an integration process step, in which the grape detection step detects the grapes contained in the image, the bunch identification step detects the bunches of grapes contained in the image, and identifies the bunch of grapes currently being worked on from the detected bunches based on the positions of the bunches of grapes and the sizes of the bunches of grapes in the image, and the integration process determines the grapes that belong to the bunch of grapes currently being worked on based on the results of the grape detection and the results of the bunch identification.

The present invention also provides an information processing method for detecting grapes from an image, the information processing method comprising a grape detection step, a bunch identification step, and an integration step, in which the grape detection step detects the grapes contained in the image, the bunch identification step detects the grape clusters contained in the image, and identifies the bunch of grapes currently being worked on from the detected bunches based on the positions of the bunches and the sizes of the bunches in the image, and the integration step determines the grapes that belong to the bunch of grapes currently being worked on based on the results of the grape detection and the results of the grape cluster identification.

1 is a diagram showing an overview of a grape grain detection system 1 according to a first embodiment. FIG. 1 is a block diagram showing the hardware configuration of an information processing device 10 and a user terminal 20 according to a first embodiment. 1 is a block diagram showing functional configurations of an information processing device 10 and a user terminal 20 according to a first embodiment. 1 is a conceptual diagram illustrating the results of grape grain detection by the grain detection unit 11a. FIG. 11 is a conceptual diagram for explaining the results of detection and identification of grape bunches during operation by the bunch identification unit 11b. FIG. 11 is a conceptual diagram illustrating detection of overlapping grape bunches by the bunch specifying unit 11b and processing for eliminating the overlapping grape bunches. FIG. 11 is a conceptual diagram illustrating the detection result of grapes belonging to a grape bunch during work by the integrated processing unit 11c. FIG. 13 is a diagram showing a verification result of particle number calculation by the particle number calculation unit 11d. FIG. FIG. 13 is a schematic diagram of the structure of a convolutional neural network according to a second embodiment. FIG. 1 is a schematic diagram of the flow from the region proposal network to the output of location information. FIG. 2 is a schematic diagram illustrating an example of a learning model. FIG. 13 is a schematic diagram showing the detection result of grapes represented by a bounding box. FIG. 2 is a schematic diagram of an example of a learning model including a mask. This is a schematic diagram showing the results of classification for each subdivided region by filling in grapes. FIG. 1 is a schematic diagram of an example of a learning model having multiple classifiers and object detectors. FIG. 1 is a schematic diagram of an example of a learning model having multiple classifiers, object detectors, and masks. 13A to 13C are diagrams illustrating the results of verification of the effect of image analysis using a model of the second modified example. FIG. 13 is a conceptual diagram illustrating an image synthesis process according to the third embodiment.

Below, several embodiments of the present invention will be described with reference to the drawings. The various features shown in the embodiments below can be combined with each other. In addition, each feature can be an invention independently.

<1. First embodiment>
(1-1. Grape Grain Detection System 1)
An information processing device according to one embodiment of the present invention is an information processing device 10 such as a server constituting a part of a grape grain detection system 1 as shown in Fig. 1. The grape grain detection system 1 includes the information processing device 10 and a user terminal 20.

The information processing device 10 is configured to be able to communicate with the user terminal 20 via the communication line 5. The user terminal 20 takes an image P1, which may include multiple grape bunches, and transmits it to the information processing device 10. The information processing device 10 analyzes the image P1 received from the user terminal 20. Based on the detection of grape berries, detection of grape bunches, and identification of the grape bunch being worked on, detection of grape berries belonging to the grape bunch being worked on is realized. Each component will be described below.

(1-2. Hardware configuration of grape grain detection system 1)
The hardware configuration of the grape grain detection system 1 will be described with reference to FIG.

(1-2-1. Hardware configuration of information processing device 10)
2 is a block diagram showing the hardware configuration of the information processing device 10 and the user terminal 20 according to this embodiment. The information processing device 10 includes a control unit 11, a storage unit 12, and a communication unit 13. The information processing device 10 may also include an operation input unit 14 configured with a keyboard, a mouse, etc., that receives input of various operations, and a monitor 15, such as a liquid crystal display device, that displays various images.

The control unit 11 is, for example, a CPU (Central Processing Unit), a microprocessor, or a DSP (Digital Signal Processor), and controls the overall operation of the information processing device 10.

A part of the storage unit 12 is composed of, for example, RAM (Random Access Memory) or DRAM (Dynamic Random Access Memory), and is used as a work area when the control unit 11 executes processes based on various programs. In addition, a part of the storage unit 12 is, for example, a non-volatile memory such as ROM (Read Only Memory) or an HDD (Hard Disk Drive), and stores various data and programs used in the processing of the control unit 11.

The programs stored in the storage unit 12 include, for example, an OS (Operating System) for implementing the basic functions of the information processing device 10, drivers for controlling various hardware, programs for implementing various functions, etc., and include the computer program according to this embodiment.

The communication unit 13 is, for example, a NIC (Network Interface Controller) and has a function of connecting to the communication line 5. Instead of or together with the NIC, the communication unit 13 may have a function of connecting to a wireless LAN (Local Area Network), a function of connecting to a wireless WAN (Wide Area Network), a function of enabling short-range wireless communication such as Bluetooth (registered trademark), and infrared communication. The information processing device 10 is connected to other information processing devices such as a user terminal 20 via the communication line 5, and can transmit and receive various data to and from the other information processing devices.

The control unit 11, memory unit 12, communication unit 13, operation input unit 14, and monitor 15 are electrically connected to each other via a system bus 16. Therefore, the control unit 11 can access the memory unit 12, display images on the monitor 15, grasp the operation state of the user on the operation input unit 14, and access various communication networks and other information processing devices via the communication unit 13.

(1-2-2. Hardware Configuration of User Terminal 20)
The user terminal 20 is, for example, an information processing terminal such as AR (Augmented Reality) glasses, MR (Mixed Reality) glasses, smart glasses, a smartphone, or a tablet terminal, and includes a control unit 21, a storage unit 22, a communication unit 23, an image capturing unit 24, and a display unit 25. The user terminal 20 may also include a speaker 26 that outputs sound, an operation unit (not shown) configured with a power button and other operation buttons, etc. The following description will focus on the differences from the information processing device 10.

The photographing unit 24 includes a camera capable of photographing still images, videos, etc. If the user terminal 20 is AR (augmented reality) glasses, MR (mixed reality) glasses, smart glasses, etc., the display unit 25 may include a display that combines a portion equivalent to a lens of glasses with a portion having a function such as projection. In addition, AR (augmented reality) glasses, MR (mixed reality) glasses, smart glasses, etc. may include a portion that projects an image directly onto the retina. If the user terminal 20 is a smartphone, tablet terminal, etc., the display unit 25 may be a touch panel display that can display images, etc. and accept operations.

The speaker 26 may also be used to communicate information regarding the total number of grapes in the grape bunch being worked on to the worker W by voice, signal sound, etc., as described below.

The control unit 21, memory unit 22, communication unit 23, image capture unit 24, display unit 25, and speaker 26 are electrically connected to each other via a system bus 27. Therefore, the control unit 21 can access the memory unit 22, control the image capture unit 24, display images on the display unit 25, grasp the operation state of the operator, output sound from the speaker 26, and access various communication networks and other information processing devices via the communication unit 23.

(1-3. Functional configuration of information processing device 10)
3, the control unit 11 of the information processing device 10 has a kernel detection unit 11a, a bunch identification unit 11b, and an integration processing unit 11c. The control unit 11 may further have a kernel number calculation unit 11d. The kernel detection unit 11a, bunch identification unit 11b, and integration processing unit 11c may be collectively referred to as an image analysis unit 30.

The grape detection unit 11a detects grapes contained in the image P1. The bunch identification unit 11b detects grape bunches contained in the image P1, and identifies the bunch currently being worked on from the detected bunches based on the position of the bunches and the size of the bunches in the image P1. The integration processing unit 11c determines the grapes that belong to the bunch currently being worked on based on the results of grape detection and bunch identification.

The grape number calculation unit 11d also counts the number of grapes detected in the image P1 as belonging to the grape bunch being worked on, and calculates the range of the total number of grapes in the grape bunch based on the counted number of grapes and a predetermined coefficient. The details of each function will be described later.

(1-4. Functional configuration of user terminal 20)
As shown in Fig. 3, the control unit 21 of the user terminal 20 has an image capturing unit 21a. The image capturing unit 21a captures an image P1. The control unit 21 may also have an analysis result display unit 21b. The analysis result display unit 21b displays the range of the total number of grapes in the grape bunch under work. In other words, the analysis result display unit 21b communicates the range of the total number of grapes to the worker W, who is the user of the user terminal 20, by displaying it on the display unit 25.

As described above, the grape grain detection system 1 includes an image capture unit 21a and an image analysis unit 30. The grape grain detection system 1 may further include an analysis result display unit 21b.

The above-mentioned functional configuration may be realized by software (including so-called apps) that is appropriately installed on the information processing device 10 or the user terminal 20, or may be realized by hardware. When realized by software, the various functions can be realized by the control unit 11 or the control unit 21 executing the programs that constitute the software.

When it is realized by executing a program, the program may be stored in the memory unit 12 or memory unit 22 built into the information processing device 10 or the user terminal 20, or may be stored in a non-transitory recording medium that is readable by a computer. It may also be realized by reading out a program stored in an external storage device and using so-called cloud computing. Or, when it is realized by hardware, it can be realized by various circuits such as ASIC, SOC, FPGA, or DRP. Also, some of the functional configurations described as functions of the information processing device 10 may be processed by the user terminal 20, etc., by software or hardware. Conversely, some of the functional configurations described as functions of the user terminal 20 may be processed by the information processing device 10, etc., by software or hardware.

(1-5. Functions of the grain detection unit 11a)
The function of the grain detection unit 11a will be described with reference to Fig. 4. The grain detection unit 11a detects grapes contained in an image P1. The image P1 is an image captured by the image capture unit 24 of a user terminal 20 held by a worker W performing the picking, and transmitted to the information processing device 10. If the user terminal 20 is an AR glass or the like, the range captured in the image P1 may be a range close to the field of view of the worker W.

Grape detection by the grain detection unit 11a is performed based on various feature quantities obtained by image analysis, such as the hue, brightness, saturation, and other features of image P1. For example, image P1 may be binarized, and then image analysis may be performed based on the contours, size, etc. to detect grapes.

The grape detection results can be obtained as the position, size, range, etc. of each grape and can be recorded in the memory unit 12. When image P1 is processed based on such grape detection results, it becomes, for example, image P2 shown in FIG. 4. The areas of grapes detected as grapes (detected grapes DG) are filled in black. The areas of grapes not detected as grapes (non-detected grapes DG) are not filled in and remain white.

(1-6. Function of the tuft specifying unit 11b)
The function of the bunch identification unit 11b will be described with reference to Fig. 5. The bunch identification unit 11b detects grape bunches contained in the image P1. The detection of grape bunches by the bunch identification unit 11b is performed based on various feature quantities obtained by image analysis, such as the hue, brightness, saturation, etc., of the image P1, in the same manner as the detection of grape grains. For example, the image P1 may be subjected to binarization processing, and then image analysis based on the contour, size, etc. may be performed to detect the grape bunches.

The grape bunch detection results can be obtained as the position, size, range, etc. of each grape bunch and can be recorded in the memory unit 12. When image P1 is processed based on such grape bunch detection results, it becomes, for example, image P3 shown in FIG. 5. The grape bunches detected as grape bunches are shown surrounded by frames as grape bunch B1, grape bunch B2, and grape bunch B3.

The bunch identification unit 11b identifies the bunch of grapes being worked on from the detected bunches based on the position of the bunch of grapes in the image P1 and the size of the bunch of grapes. In FIG. 5, bunch of grapes B1, bunch of grapes B2, and bunch of grapes B3 have been detected as bunches of grapes, and from these, the bunch of grapes being worked on is identified.

Here, the "bunch of grapes being worked on" refers to the bunch of grapes that the worker W is working on. The bunch of grapes that the worker W is working on refers to, for example, a bunch of grapes about to be thinned, a bunch of grapes on which the thinning work is being performed, a bunch of grapes on which the worker W is checking the status of the thinning work, etc. It can also be said to be a bunch of grapes that the worker W is paying attention to and gazing at in order to thin the grapes.

The criterion for determining the "bunch position" to be considered in identifying the bunch being worked on is, for example, the proximity of each detected bunch to the center of image P1. As shown in FIG. 5, the bottom left of image P1 is set as the origin, the right direction is the X axis, the top direction is the Y axis, the right end is X=1, and the top end is Y=1. In this case, the center of image P1 is point D1 where (X, Y)=(0.5, 0.5). The proximity can be calculated based on the distance between this point D1 and the center of each detected bunch of grapes (image bunch center distance). For example, a value obtained by dividing 1 by the image bunch center distance, in other words, the "proximity rate to the image center" can be used as an index of proximity.

The "size of the bunch of grapes" considered to identify the bunch of grapes being worked on is, for example, the area of the area belonging to each detected bunch of grapes in image P1. As an index representing the size of each bunch of grapes in image P1, for example, the value obtained by dividing the "area of the area belonging to the bunch of grapes" by the "area of image P1", in other words, the "occupancy rate in the image", etc. can be used.

An example of a method for the bunch identification unit 11b to identify bunches of grapes in work based on the "position of the bunch" and the "size of the bunch" is to identify the bunches of grapes in work based on the proximity of each detected bunch of grapes to the center of image P1 and the area of the area belonging to each detected bunch of grapes in image P1. More specifically, for example, the standard is the sum of the "proximity rate to the image center" and the "occupancy rate in the image". In one aspect, it can be determined that the larger the sum, the higher the probability that the bunch is in work.

In FIG. 5, let us assume that the "proximity rate to the image center" of grape bunches B1 to B3 is 0.5, 0.3, and 0.1, respectively, and that the "occupancy rate in the image" of grape bunches B1 to B3 is 0.2, 0.15, and 0.1, respectively. In such a case, the sum of the "proximity rate to the image center" and the "occupancy rate in the image", that is, the probability that a bunch is being worked on, is 0.7, 0.45, and 0.2 for grape bunches B1 to B3, respectively. In other words, grape bunch B1 has the highest probability of being a bunch being worked on in image P1, and is identified as a bunch being worked on.

The process of identifying a bunch of grapes being worked on from the detected bunches based on the position and size of the bunch by the bunch identification unit 11b also contributes to excluding the same bunch of grapes when it is detected as bunch B1 and bunch B1' in duplicate, as shown in image P3' in FIG. 6. Bunch B1' is a part of bunch B1 that has been mistakenly detected as a bunch of grapes. However, when the position and size of the bunch of grapes are taken into consideration in order to identify the bunch of grapes being worked on, bunch B1' is excluded as a bunch of grapes being worked on, and the bunch of grapes being worked on can be identified with high accuracy.

(1-7. Functions of the integration processing unit 11c)
The function of the integration processing unit 11c will be described with reference to Fig. 7. The integration processing unit 11c determines which grapes belong to the grape bunch under processing based on the grape grape detection results and the grape bunch identification results. The determination is made by comprehensively considering the grape grape detection results and the grape bunch identification results.

Of the grape berries DG that are considered to be grape berries based on the "grape berry detection result," those that are highly likely to belong to the grape bunch B1 during work that is identified based on the "grape bunch identification result" are determined to be grape berries DG that belong to the grape bunch B1 during work. When image P1 is processed based on such a determination, it becomes, for example, image P4 shown in FIG. 7.

There are 26 grape berries DG (black berries) detected in the entire image P4. However, the grape bunch B1 identified as being in work is within the solid line frame, and the grape berries determined to belong to the grape bunch B1 in work are the nine grape berries DG within the range enclosed by the solid line frame.

(1-8. Functions of particle number calculation unit 11d)
The grape number calculation unit 11d counts the number of grapes detected as belonging to the grape bunch under work in the image P1. For example, in Fig. 7, the number of grapes detected as belonging to the grape bunch under work is 9 for grape bunch B1 within the range surrounded by a solid line frame.

The grape number calculation unit 11d also calculates the range of the total number of grapes in the grape bunch B1 based on the number of measured grapes and a predetermined coefficient. The predetermined coefficient is a coefficient determined based on the relationship between the number of grapes belonging to the grape bunch detected and measured from the two-dimensional image, and the actual number of grapes belonging to the grape bunch, including grapes located on the back side that are not detected because they are not visible in the two-dimensional image, for example.

The determination of the predetermined coefficient in one embodiment will be described. First, for multiple grape bunches for which the actual number of grapes belonging to the bunch is known, multiple sample images including those grape bunches are prepared, and the number of grapes detected as grapes belonging to the bunch under operation is counted. For each number of measured grapes, the actual number of grapes belonging to the bunch is tallied (clustered). By such tallying, it is possible to determine a coefficient for calculating the range of the total number of grapes in the bunch, which corresponds to the number of measured grapes. Note that some of the multiple grape bunches may be the same bunch. However, even if they are the same bunch, it is preferable that the number of grapes on the bunch differs due to the thinning operation.

For example, suppose that the number of grapes measured from multiple sample images is 44, and it is found that the actual number of grapes belonging to the grape cluster is between 44 and 52. If the number of grapes measured is 44, the predetermined coefficient can be determined to be between 1 (=44/44) and 1.8 (=52/44). In other words, the predetermined coefficient may be determined separately for each number of grapes measured.

The results of tallying up the actual number of grapes belonging to a grape cluster for each number of grapes measured can be represented as in the graph in Figure 8. Let X be the number of grapes measured, and Y be the actual number of grapes belonging to a grape cluster. After plotting all points (X, Y), plot the minimum and maximum points of Y at the same X. This plot makes it possible to draw a regression curve based on each minimum point and a regression curve based on each maximum point. The regression curve based on each minimum point becomes the lower limit curve for the actual number of grapes belonging to a grape cluster, and the regression curve based on each maximum point becomes the upper limit curve for the actual number of grapes belonging to a grape cluster.

<Verification of grain number calculation>
FIG. 8 shows the results of calculating the number of grapes when a test was conducted using 100 sample images and 26 verification images. A lower limit curve 41 and an upper limit curve 42 are drawn based on the analysis of the 100 sample images. In addition, the actual number of grapes belonging to a grape cluster is plotted for each number of grapes measured from the verification images. It can be seen that the plots are almost within the range bounded by the lower limit curve 41 and the upper limit curve 42 created from the sample images.

(1-8. Processing flow)
The following describes the flow of processing by the grape grain detection system 1. First, an image capturing step is executed. In the image capturing step, an image P1 is captured by the image capturing unit 24 of the user terminal 20. The image P1 is transmitted from the user terminal 20 to the information processing device 10.

Next, a grain detection process is performed. In the grain detection process, grapes contained in image P1 are detected. A bunch identification process is performed before or after the grain detection process, or in parallel. In the bunch identification process, grape bunches contained in image P1 are detected, and a bunch of grapes currently being worked on is identified from the detected bunches based on the position of the bunch and the size of the bunch in image P1. Then, an integration process is performed. In the integration process, grapes belonging to the bunch currently being worked on are determined based on the results of grape grain detection and bunch identification. The process including the grain detection process, bunch identification process, and integration process is sometimes referred to as an image analysis process.

Next, a grape number calculation process is executed. In the grape number calculation process, the number of grapes detected in image P1 as belonging to the grape bunch being worked on is counted, and the range of the total number of grapes in the grape bunch is calculated based on the counted number of grapes and a predetermined coefficient.

Then, an analysis result display process is executed. In the analysis result display process, the range of the total number of grapes in the grape bunch being worked on is displayed on the display unit 25 of the user terminal 20. In the analysis result display process, it is also possible to display the analysis results, such as the area of the grapes and the position of the grape bunch, superimposed on the field of view using AR technology.

<2. Second embodiment>
(2-1. Grape Grain Detection System 1 According to the Second Embodiment)
A grape grain detection system 1 according to a second embodiment of the present invention will be described below. The grape grain detection system 1 according to the second embodiment differs from the first embodiment in that it performs various detections, classifications, etc. based on a learning model. The following description will focus on the differences.

(2-1. Functions of the Grain Detector 11a)
The grain detection unit 11a detects grapes contained in the image P1 based on the first learning model. That is, the grain detection unit 11a performs image recognition processing based on machine learning, deep learning, etc., and detects and classifies objects. The first learning model is not particularly limited as long as it is a learning model that can detect grapes, but for example, it is a learning model that trains a model using a large amount of teacher data (a set of known input data and correct answer data) and makes it possible to predict future outputs.

For the first learning model, a deep learning model using a convolutional neural network (CNN) can be adopted. In such a learning model, the input image (image P1) is processed by the convolutional neural network. The convolutional neural network is composed of one or more combinations of convolutional layers and pooling layers. For example, as shown in FIG. 9, the input image is processed by a convolutional neural network composed of a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, ... an nth convolutional layer, and an nth pooling layer, and a feature map is generated.

The first learning model is a learning model that includes an object detector that detects objects using the features of image P1, and a classifier that classifies objects using the features of image P1. If the first learning model is a deep learning model that uses a convolutional neural network, the object detector and the classifier use the feature map.

The classifier included in the first learning model may have a non-positional classifier that is a classifier that does not use positional information as a feature for classification, and a positional classifier that is a classifier that uses positional information as a feature for classification, and may be configured to perform classification based on the classification results by the non-positional classifier and the classification results by the positional classifier. In a learning model, information about position is usually excluded from the features for classifying objects. This is because the position in an image, for example an object being located in the upper right corner of an image, usually has a negative effect on a classifier that determines what the object is.

However, the inventors discovered that the recall rate, which is the rate at which grapes are correctly detected as grapes, is improved when the first learning model also includes a positional classifier, which is a classifier that uses position information as a feature for classification, compared to when the first learning model includes only a non-positional classifier, which is a classifier that does not use position information as a feature for classification. In other words, it can be said that objects can be more correctly classified as grapes.

In addition, when a positional classifier, which is a classifier that uses position information as a feature for classification, was included, the detection of grapes other than those belonging to the bunch being worked on was suppressed. As a result, the proportion of grapes belonging to the bunch being worked on among the detected grapes was improved.

Here, the location information as a feature can be obtained from the input image by various methods, but for example, it may be obtained by using a region proposal network (RPN: Region Proposal Network). The region proposal network may have a network structure similar to that of a convolutional neural network. Also, as shown in FIG. 10, the region proposal network may be configured to receive a feature map generated by a convolutional neural network as input and output location information.

The first learning model, which uses a convolutional neural network (CNN) and a region proposal network (RPN) and further includes a classifier that uses location information as a feature for classification, can be expressed as shown in Figure 11. That is, the region proposal network outputs location information using the feature map output by the convolutional neural network as input. The feature map and location information are input to an RoI pooling layer that integrates them, where extraction processing is performed and features are output.

The object detector and the non-positional classifier are input with the features extracted in the RoI pooling layer. Meanwhile, the positional classifier is input with features including position information output from the region proposal network in addition to the features extracted in the RoI pooling layer. The object detector, the non-positional classifier, and the positional classifier may be a fully connected layer composed of one or more layers.

The object detector outputs information such as the position and area of the detected object in image P1. In the example of Fig. 12, an image P5 processed based on the result of object detection is shown. In this image P5, the coordinates of point D2 in the lower left and point D3 in the upper right of the area are output in order to represent the area of the detected object as a rectangle (bounding box BB1). Point D2 is ( _x1 , _y1 ), and point D3 is ( _x2 , _y2 ). The coordinates of these two points are an example of information output from the object detector.

The non-positional classifier outputs the probability that a detected object is a specific object. The grain detection unit 11a uses a learning model that has been trained to detect grapes as the first learning model, and outputs the probability that the detected object is a grape. At this time, the probability that it is the background may also be output at the same time and taken into consideration when classifying.

The positional classifier differs from the non-positional classifier in that it also uses input position information, but like the non-positional classifier, it outputs the probability that a detected object is a specific object. The grain detection unit 11a uses a learning model that has been trained to detect grapes as the first learning model, so it outputs the probability that the detected object is a grape. At this time, the probability that it is the background may also be output at the same time and taken into consideration when classifying.

When the classifier only includes a non-positional classifier, it classifies whether or not it is a grape based on whether the probability output by the non-positional classifier is a certain level or higher. When the classifier includes a positional classifier together with a non-positional classifier, it classifies whether or not it is a grape based on both the probability output by the non-positional classifier and the probability output by the positional classifier. More specifically, for example, it classifies based on the sum (i.e., the average value) of the value obtained by multiplying the "probability output by the non-positional classifier" by 0.5 and the value obtained by multiplying the "probability output by the positional classifier" by 0.5. In other words, the sum is the probability that it is a grape, and it is classified whether or not it is a grape based on whether or not this probability is a certain level or higher.

(2-2. Function of the tuft specification unit 11b)
The bunch identification unit 11b may be configured to detect grape bunches included in the image P1 based on the second learning model. That is, the bunch identification unit 11b performs image recognition processing based on machine learning, deep learning, etc., and detects and classifies objects. The second learning model is a learning model including an object detector that detects objects using the feature amount of the image P1, and a classifier that classifies objects using the feature amount of the image P1. A deep learning model using a convolutional neural network can be adopted as the second learning model.

The second learning model may be the same as or different from the first learning model. If the second learning model is the same as the first learning model, the detection of grapes and the detection of grape clusters may be configured to be performed using the same learning model.

(2-3. Modification 1 of the second embodiment)
The grain detection unit 11a and the bunch identification unit 11b may be configured to perform classification for each subdivided region. "For each subdivided region" refers to, for example, for each pixel. Classification for each subdivided region may be performed for the entire image, but classification for each subdivided region may also be performed for only detected objects, such as grapes or grape bunches. The process of classifying for each subdivided region is sometimes called segmentation.

When classifying each subdivided region, for example, a model such as that shown in FIG. 13 can be used. The model in FIG. 13 has a mask, and differs from the model in FIG. 11 in that the features extracted in the RoI pooling layer are also input to the mask. This mask is used to classify each subdivided region.

Based solely on the detection results from the object detector, it is only possible to recognize the object as being surrounded by a bounding box BB2, as shown in image P6 in Figure 14, but by using a mask, it is possible to recognize the object (grapes) in a form that is closer to its actual contours.

(2-4. Modification 2 of the Second Embodiment)
As shown in Fig. 15, the first learning model may be configured to include a first object detector and a second object detector as object detectors. A threshold value for detection by the second object detector is configured to be different from a threshold value for detection by the first object detector. In addition, the second object detector can use the detection result of the first object detector. With such a configuration, object detection is performed using a plurality of different threshold values, and it is expected that oversight in object detection is suppressed.

The first object detector and the second object detector, which have different detection thresholds, can be trained independently during the training phase. Different detection thresholds are set for the first object detector and the second object detector, and training is performed. Here, the detection threshold is, for example, a value for IoU (Intersection over Union), and is the ratio of overlap of the images of the detected objects. It may also be rephrased as the overlap between bounding boxes.

The first and second object detectors are then used in conjunction with each other when making inferences about data for which the correct answer is unknown. For example, when using a model such as that shown in FIG. 15 during inference, the first object detector trained with a low threshold and the second object detector trained with a higher threshold are linked via a Rol pooling layer. The first object detector trained with a low threshold will have a low detection threshold and will obtain an output with relatively high detection noise. The second object detector trained with a higher threshold will have a higher detection threshold and can obtain an output with relatively low detection noise, and since the output results of the first object detector are used, improved detection accuracy can be expected.

As shown in FIG. 15, the first learning model may include a first classifier and a second classifier as classifiers, and the second classifier may be configured to use the detection results of the first object detector. More specifically, a second positional classifier included in the second classifier uses the detection results of the first object detector. The second positional classifier receives an output including position information from the first object detector. With this configuration, it is expected that the recall rate, which is the rate at which grapes are correctly detected as grapes, will be improved.

The control unit 11 may further include a subdivided area detection unit that performs classification for each subdivided area. The subdivided area detection unit detects grape grain areas included in the image for each subdivided area based on the first learning model. The first learning model may be configured to include a mask that uses the features of the image P1 to classify (segment) objects for each subdivided area.

As shown in FIG. 16, the first learning model may be configured such that the second RoI pooling layer receives an output including position information from the first object detector, and the first mask outputs a result of classifying objects for each subdivided region based on the output from the second RoI pooling layer.

Similarly, the second mask is processed based on the output from the third RoI pooling layer, and the third mask is processed based on the output from the fourth RoI pooling layer. By configuring it in this way, the output from the RoI pooling layer located behind the model is used, and it is expected that better segmentation results will be obtained by using more refined features.

When the first learning model is a learning model including a positional classifier that uses positional information as a feature for classification, a term based on the positional information may be added to a loss function used for adjusting weights in the learning stage. The loss function L may be set, for example, as shown in the following formula (1).

The loss function L is explained when the first learning model is composed of multiple stages, from the first stage to the Tth stage. In FIG. 16, T=3, which means that the model is composed of the first to third stages.

L _bbox ^t represents the loss of the bounding box (object detector) in the tth stage. L _cls ^t represents the loss of the non-positional classifier in the tth stage. L _scls ^t represents the loss of the positional classifier in the tth stage. L _mask ^t represents the loss of the mask in the tth stage. The coefficient α _t is a coefficient for adjusting the contribution in the loss function of each stage. β is a weighting coefficient for each loss (L _bbox ^t , L _cls ^t , and L _scls ^t ). The design of such a loss function and the setting of each coefficient value can be referred to Cascade R-CNN (Cai and Vasconcelos 2019), Mask R-CNN (He et al. 2017), Hybrid Task Cascade (HTC) (Chen, Ouyang, et al. 2019), etc.

Each loss is defined as in the following (2), (4), (5), and (6): "smooth _L1 " in the following formula (2) is defined as in the following formula (3).

_Lbbox can be expressed as a four-dimensional vector (position information x, y and size information w, h) for the correct bounding box and the predicted bounding box, and the loss function is calculated as the Manhattan distance between the two vectors (the sum of the absolute values of the differences in the coordinates of each dimension). Here, the correct bounding box can be expressed as v = ( _vx , _vy , _vw , _vh ), and the predicted bounding box can be expressed as b = ( _bx , _by , _bw , _bh ).

_Lcls ^t and _Lscls ^t are defined as cross-entropy (CE) (Equation (4) below). The output of each classifier is the probability of a certain class (represented by a label), and the evaluation loss function is expressed as the cross-entropy between the probability of actually obtaining the classification result (prediction probability) and the probability of the correct answer. The more the predicted result deviates from the correct answer, the larger the value of this cross-entropy. K is the number of classes in the model, p is the prediction probability calculated by the softmax function of the fully connected layer, and u is the ground truth for each class.

For binary cross-entropy loss (BCE) when K is 2, the cross-entropy can be calculated using the following formula (5):

The mask outputs a result for each RoI, and outputs a K x m ^two- dimensional result. The output is the value of each dimension (value for each pixel), which is a real number between 0 and 1. The ground truth of the mask is a binary image of 0 or 1, and L _mask can be calculated using sigmoid cross entropy and can be defined as the average of binary cross-entropy loss (BCE). m _pred is the predicted mask, and m _gt is the ground truth.

<Verification of the effect of image analysis using the model of modification example 2>
The inventors conducted an experiment on a first learning model that is composed of the first to third stages as shown in FIG.

First, the model was trained using 790 images. During this training, each stage was trained separately. The thresholds for UoI of the object detectors were set to increase in the order of the first object detector, the second object detector, and the third object detector. The first object detector was set to be most tolerant of noise related to object detection.

After learning, each stage was constructed as shown in Figure 16 and tested with 198 images. The results are shown in Table 1. Compared to the model reported by Hybrid Task Cascade (HTC) (Chen, Ouyang, et al. 2019), the false positive rate (the proportion of non-grape objects among those detected as grapes) decreased by 0.07%, but the recall rate (the proportion of grapes that were detected as grapes) improved by 1.77%. In addition, grapes that could not be detected by existing methods could be detected (Figure 17). The grapes present in the three circled areas in Figure 17A could not be detected by the conventional model, but were successfully detected by the first learning model of the present invention (Figure 17B).

<3. Third embodiment>
(3-1. Grape Grain Detection System 1 According to the Third Embodiment)
A grape grain detection system 1 according to a third embodiment of the present invention will be described below. The grape grain detection system 1 according to the third embodiment is characterized in the preparation of images as teacher data used for model learning. The following description will focus on the differences from the first and second embodiments.

(2-2. Functional configuration of information processing device 10)
The control unit 11 further includes a teacher data generation unit and a model learning unit. The teacher data generation unit generates teacher data (a set of known input data and correct answer data) used by the learning model in the learning stage. The model learning unit trains the model using the generated teacher data.

(2-3. Teacher data generation unit)
The teacher data generating unit synthesizes an image of a process of removing grapes one by one from the image (teacher data generating step). The teacher data generating step may include a circularity calculating step, a removal candidate extracting step, and a removal complementing step.

In the circularity calculation process, the circularity is calculated for the grape berry regions in the image that are known to be grape berries. In the removal candidate extraction process, grape berries whose grape berry region has a circularity below a threshold are extracted as candidates for removal. In the removal and completion process, one grape berry is randomly selected from the extracted grape berries and removed, and the removed part is image-completed with the background or the like. Image completion can be performed using the technology described in Iizuka, Satoshi, Edgar Simo-Serra, and Hiroshi Ishikawa. 2017. "Globally and Locally Consistent Image Completion." ACM Transactions on Graphics (Proc. of SIGGRAPH) 36(4): 107. The removal and completion process is repeated until there are no more candidates extracted. An image with a different berry-picking state can be generated with each repetition. Also, by starting over from the beginning or partway through the process before removal, an image with a different pattern of random berry-picking can be generated.

In Figure 18A, grape 52 is located behind grape 51. In such an image, the region of grape 52 has a lower circularity than grape 51 and the like. In the circularity calculation process, the circularity of the region of grape 52 is calculated, and if the circularity of the region of grape 52 is equal to or less than a threshold, it is extracted as a removal candidate in the removal candidate extraction process (Figure 18B). Then, in the removal and completion process, grape 52 is deleted, and the hole (blank region) in the image created by the deletion is completed based on the characteristics of the background image that is not a grape (Figure 18C).

<Verification of the effect of learning using synthetic images>
The inventors trained the model using 50 original images and 790 synthetic images synthesized from them. Then, the performance of the model to detect grapes was verified using another 10 original images and 198 synthetic images synthesized from them. It can be seen that both the recall rate and the false positive rate are improved compared to when only 50 original images are used for learning. For the verification, the deep learning model of Chen, Kai, Wanli Ouyang, et al. 2019. "Hybrid Task Cascade for Instance Segmentation." Proceedings of the IEEE International Conference on Computer Vision: 4969-78. http://arxiv.org/abs/1901.07518 (November 21, 2019). was used.

By using such image synthesis technology to generate a large number of synthetic images from original images that can serve as training data, it is possible to improve the performance of the first learning model used in the second embodiment, etc.

1: Grain detection system, 5: Communication line, 10: Information processing device, 11: Control unit, 11a: Grain detection unit, 11b: Bunch identification unit, 11c: Integration processing unit, 11d: Grain number calculation unit, 12: Memory unit, 13: Communication unit, 14: Operation input unit, 15: Monitor, 16: System bus, 20: User terminal, 21: Control unit, 21a: Image capture unit, 21b: Analysis result display unit, 22: Memory unit, 23: Communication unit, 24: Capture unit, 25: Display unit, 26: Speaker, 27: System bus, 30: Image analysis unit, 41: Lower limit curve, 42: Upper limit curve, 51, 52, DG, UG: Grape grain, B1 to B3, B1: Grape bunch, BB1, BB2: Bounding box, D1 to D3: Points, P1 to P6, P3': Images, W: Operator, WAN: Wireless

Claims (9)

An information processing device for detecting grapes from an image,
The apparatus includes a grain detection unit, a bunch identification unit, and an integration processing unit,
The grain detection unit detects the grapes included in the image,
The tuft specifying unit is
Detecting grape clusters contained in the image;
Identifying a bunch of grapes that is to be subjected to a grape thinning operation from among the detected bunches of grapes based on the positions of the bunches of grapes and the sizes of the bunches of grapes in the image;
The integrated processing unit determines grapes belonging to a grape bunch that is to be thinned based on the detection result of the grapes and the identification result of the grape bunch.
Information processing device. 2. The information processing device according to claim 1,
The particle detection unit is
Detecting the grapes included in the image based on a first learning model;
The first learning model is a learning model including an object detector that detects an object by using a feature amount of the image, and a classifier that classifies the object by using the feature amount of the image.
Information processing device. 3. The information processing device according to claim 2,
The classifier included in the first learning model is
A non-positional classifier that is a classifier that does not use position information as a feature for classification, and a positional classifier that is a classifier that uses position information as a feature for classification,
classifying based on the classification result by the non-positional classifier and the classification result by the positional classifier;
Information processing device. 4. The information processing device according to claim 2,
The tuft specifying unit is
Detecting grape bunches included in the image based on a second learning model;
The second learning model is a learning model including an object detector that detects an object by using a feature amount of the image, and a classifier that classifies the object by using the feature amount of the image,
The second learning model is the same as or different from the first learning model.
Information processing device. The information processing device according to any one of claims 1 to 4,
Further comprising a particle number calculation unit,
The particle number calculation unit is
Counting the number of grapes detected in the image as belonging to the grape bunch that is to be thinned ;
Calculating the range of the total number of grapes in the grape cluster based on the number of grapes measured and a predetermined coefficient;
Information processing device. 1. A system for detecting grapes from an image, comprising:
The apparatus includes an image capturing unit and an image analyzing unit,
The image capturing unit captures the image,
The image analysis unit includes a grain detection unit, a bunch identification unit, and an integration processing unit.
The grain detection unit detects the grapes included in the image,
The tuft specifying unit is
Detecting grape clusters contained in the image;
Identifying a bunch of grapes that is to be subjected to a grape thinning operation from among the detected bunches of grapes based on the positions of the bunches of grapes and the sizes of the bunches of grapes in the image;
The integrated processing unit determines grapes belonging to a grape bunch that is to be thinned based on the detection result of the grapes and the identification result of the grape bunch.
system. 7. The system of claim 6,
Further comprising an analysis result display unit;
The analysis result display unit displays the range of the total number of grapes in the grape clusters that are the subject of the grape thinning operation .
system. A program for detecting grapes from an image, comprising:
A computer is caused to execute a kernel detection step, a bunch identification step, and an integration processing step;
In the grain detection step, the grape grains included in the image are detected,
In the tuft identification step,
Detecting grape clusters contained in the image;
Identifying a bunch of grapes that is to be subjected to a grape thinning operation from among the detected bunches of grapes based on the positions of the bunches of grapes and the sizes of the bunches of grapes in the image;
In the integration process, grapes belonging to a grape bunch that is to be thinned are determined based on the detection result of the grapes and the identification result of the grape bunch.
program. An information processing method for detecting grapes from an image, comprising the steps of:
The method includes a grain detection step, a bunch identification step, and an integration processing step,
In the grain detection step, the grape grains included in the image are detected,
In the tuft identification step,
Detecting grape clusters contained in the image;
Identifying a bunch of grapes that is to be subjected to a grape thinning operation from among the detected bunches of grapes based on the positions of the bunches of grapes and the sizes of the bunches of grapes in the image;
In the integration process, grapes belonging to a grape bunch that is to be thinned are determined based on the detection result of the grapes and the identification result of the grape bunch.
Information processing methods.