JP6964316B1 - Artificial intelligence (AI) estimation system, learning data generator, learning device, fruit thinning object estimation device, learning system, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an AI for estimating a fruit-picking object with high accuracy in a fruit-picking operation of an agricultural product. SOLUTION: The estimation system 501 has a learning data generation device, a learning device, and a fruit thinning object estimation device, and the learning data generation device shows before (first) and after (second) fruit thinning of agricultural products. An extraction unit that inputs image data and extracts the target object that is the difference between the two as a fruit-picking target, a generation unit that learns using the first learning data including the extraction result and estimates the fruit-picking target, and a generation unit. The estimation result from is provided with an identification unit that identifies the image data and generates the second training data. The learning device has a generation unit that generates an estimation result image data of a fruit-picking object by the second learning data, and a discrimination unit that feeds back the identification result based on the training data and comparison to the generation unit to learn the learning model. It has a learning department. The fruit-picking target estimation device estimates the fruit-picking target from the unknown image data showing before the fruit-picking by the trained model. [Selection diagram] FIG. 20

Description

The present invention relates to an AI estimation system, a learning data generation device, a learning device, a fruit thinning object estimation device, a learning system, and a program.

There is known a technique of estimating based on the current situation or recognizing various objects by artificial intelligence (hereinafter referred to as "AI").

For example, there is a robot system used in a factory or the like in which a robot hand is installed on a conveyor. Specifically, the robot system first photographs an object with a camera. After shooting, the object is image-recognized based on the shot image. Then, the robot system calculates the position of the center of gravity of the object based on the captured image. Based on the position of the center of gravity calculated in this way, the robot system determines an accurate position for gripping the object with the robot hand. A technique for stably gripping an object with a robot hand in this way is known (see, for example, Patent Document 1 and the like).

The recognition of objects by AI is also used in agricultural situations. Specifically, there is known a technique in which AI automatically determines the number of grains in a grape picking operation (see, for example, Non-Patent Document 1 and the like).

Japanese Unexamined Patent Publication No. 2019-185204

Kenichi Shoji, "AI to automatically determine" how many grape grains are there? "To be put into practical use next summer-joint development by Yamanashi University and agricultural production corporation", [online], August 17, 2020, DG Lab House, [ Search on December 2, 2nd year of Reiwa], Internet, <URL: https://media.dglab.com/2020/08/17-grape-01/>

The technology described in Patent Document 1 described above is a technology assuming a lighting environment such as in a factory. That is, the lighting environment such as in a factory is an environment in which the conditions such as light are stable as compared with the lighting environment such as under natural light such as outdoors for processing such as shooting and image recognition. In many cases. Therefore, there is a problem that it is difficult to apply the technology assuming the lighting environment such as in a factory to the lighting environment such as handling agricultural products.

Further, in the technique as described in Non-Patent Document 1 above, sufficient learning data is secured in order to train AI. In particular, in order to improve the accuracy of AI, it is desirable to secure a large amount of learning data. Therefore, in the technique as described in Non-Patent Document 1 above, there is a problem that it is difficult to estimate the fruit-picking object with high accuracy by AI.

An object of the present invention is to estimate the fruit-picking target in the fruit-picking work of agricultural products with high accuracy by AI.

In order to solve the above problems, in one aspect of the present invention,
A learning data generation device that generates learning data, a learning device that has a generation unit and an identification unit and trains a learning model using the learning data, and fruit picking using a learned model trained by the learning device. In an estimation system with an object estimation device,
The learning data generator is
An image data input unit for inputting a first input image data which is image data showing an agricultural product before fruit thinning and a second input image data which is an image data showing the agricultural product after fruit thinning.
Extraction of the target object which is the difference between the first input image data and the second input image data among the target objects which are the fruits, flowers, leaves, or a combination thereof of the agricultural crop as the fruit-picking target. Department and
A generation unit that learns the first input image data and a set of extraction results by the extraction unit as the first learning data, and generates estimation result image data indicating the result of estimating the fruit-picking object.
It is provided with an identification unit that identifies the estimation result image data and generates a second learning data based on the identification result.
The learning device is
A learning data input unit for inputting the second learning data,
The generation unit that generates the estimation result image data showing the result of estimating the fruit-picking object by the second learning data, and the generation unit.
It is provided with the identification unit that identifies the estimation result image data as compared with the learning data and feeds back the identification result to the generation unit to train the learning model.
The fruit thinning object estimation device is
An image data input unit that inputs unknown image data indicating an unknown crop before fruit thinning,
An estimation unit that estimates the fruit-picking object using the trained model,
It includes an output unit that outputs an estimation result by the estimation unit.

According to the present invention, the fruit-picking target in the fruit-picking work of agricultural products can be estimated with high accuracy by AI.

It is a figure which shows the whole structure example of the learning data generation apparatus for AI. It is a figure which shows the hardware configuration example of an information processing apparatus. It is a figure which shows the whole processing example of 1st Embodiment. It is a figure which shows the configuration example of a hostile generation network. It is a figure which shows the example of the photographing method. It is a figure which shows the whole processing example of 2nd Embodiment. It is a figure which shows the example of the extraction process. It is a figure which shows the processing example of the instance segmentation, and the example of the mask image data. It is a figure which shows the processing example of illustration. It is a figure which shows the modification example of the illustrated image data or the mask image data. It is a figure which shows the recognition example of the target object. It is a figure which shows the processing result example of the whole processing. It is a figure which shows the configuration example of the learning apparatus. It is a figure which shows the example of the structure which performs learning by a learning device. It is a figure which shows the functional configuration example of a learning apparatus. It is a figure which shows the structural example of the fruit thinning object estimation apparatus. It is a figure which shows the example of the structure which estimates by the fruit thinning object estimation apparatus. It is a figure which shows the functional composition example of the fruit thinning object estimation apparatus. It is a figure which shows the functional structure example of a learning system. It is a figure which shows the functional configuration example of an estimation system. It is a figure which shows the example of a network structure.

Specific examples will be described below with reference to the attached drawings. In the following description, the reference numerals described in the drawings refer to the same elements when the reference numerals are the same.

[First Embodiment]
FIG. 1 is a diagram showing an overall configuration example of a learning data generation device for AI. For example, the learning data generation device for AI (hereinafter referred to as "learning data generation device 10") is used as follows.

The learning data generation device 10 is, for example, the following information processing device.

[Hardware configuration example of information processing device]
FIG. 2 is a diagram showing a hardware configuration example of the information processing device. For example, the learning data generation device 10 has a hardware configuration including a Central Processing Unit (CPU, hereinafter referred to as “CPU 10H1”), a storage device 10H2, an interface 10H3, an input device 10H4, an output device 10H5, and the like. Further, it is desirable that the learning data generation device 10 has a hardware configuration having a Graphics Processing Unit (GPU, hereinafter referred to as "GPU10H6").

CPU10H1 is an example of an arithmetic unit and a control unit. For example, the CPU 10H1 performs an operation based on a program, an operation, or the like.

The storage device 10H2 is a main storage device such as a memory. The storage device 10H2 may have an auxiliary storage device such as an SSD Solid State Drive (SSD) or a hard disk.

The interface 10H3 transmits / receives data to / from an external device via a network, a cable, or the like. For example, the interface 10H3 is a connector, an antenna, or the like.

The input device 10H4 is a device for inputting an operation by the user. For example, the input device 10H4 is a mouse, a keyboard, or the like.

The output device 10H5 is a device that outputs a processing result or the like to the user. For example, the output device 10H5 is a display or the like.

The GPU 10H6 is an arithmetic unit for image processing. The GPU 10H6 may be called a graphic controller or the like. In particular, GPU10H6 is used when performing image processing in real time, or for parallel calculation in learning.

The learning data generation device 10 may have a hardware configuration in which hardware resources other than the above are further provided inside or outside. Further, the learning data generation device 10 may be a plurality of devices.

[About crops, target objects, fruit-picking objects, and fruit-picking work]
The learning data generation device 10 captures image data (hereinafter, "first input image data") of the crop before the fruit-picking work (hereinafter, the crop in the state before the fruit-picking work is referred to as "first crop 12") with the camera 11. 11D1 ") is input. The photographing device such as the camera 11 may have a configuration included in the learning data generation device 10.

Further, the learning data generation device 10 captures image data (hereinafter, "second") of the crop after the fruit-picking work (hereinafter, the crop in the state after the fruit-picking work is referred to as "second crop 13") with the camera 11. Input image data 11D2 ”) is input.

Hereinafter, the first input image data 11D1 and the second input image data 11D2 may be collectively referred to as “input image data”.

The first input image data 11D1 and the second input image data 11D2 are a moving image, a still image, or a combination thereof. When inputting in the form of a moving image, for example, one frame or a predetermined number of frames out of a plurality of frames constituting the moving image are cut out and used as input image data.

The fruit-picking operation is an operation of thinning out fruits, flowers, leaves, or a combination thereof (hereinafter referred to as "target object") possessed by or around the crop. That is, the fruit-picking work is a work of grain-picking, fruit-picking, flower-picking, or a combination thereof.

Hereinafter, among the target objects, the objects to be thinned out by the fruit thinning work are referred to as "fruit thinning objects". That is, the fruit-picking work is a work of selecting and thinning out some fruit-picking objects from a plurality of target objects. In the figure, the fruit-picking object is indicated by "x" to indicate that it is in a thinned state. However, the form of how to distinguish the target object from the fruit-picking target does not matter.

The worker 14 determines which of the target objects is to be the fruit-picking target.

For example, even if the crops are the same, the fruit-picking objects may differ depending on the purpose. First, the objectives are, for example, to ensure that the crops are generally evenly lit, to adjust the taste, to ensure that the crops are dense to some extent, to keep the crops within a predetermined size, or to harvest. Occasionally, the appearance of crops (color, shape, scratched target objects are reduced, or the overall appearance of these is reduced) is improved.

The worker 14 performs a sample fruit-picking operation on the first crop 12 based on the purpose of fruit-picking. Then, the worker 14 separately photographs before and after the fruit-picking operation. Input image data is generated by each such shooting.

In addition, the input image data is separately photographed according to the purpose of fruit thinning, the type of crop, and the like. That is, the content of the fruit thinning work may differ depending on the purpose. Therefore, the input image data is generated separately according to the purpose, the type of agricultural product, and the like. The worker 14 is, for example, a skilled farmer or the like in order to show a sample fruit-picking work.

Comparing the first input image data 11D1 and the second input image data 11D2, the learning data generation device 10 and the like determine which target object is the fruit picking target and how much is the fruit picking target. It is possible to grasp whether or not to do so.

Agricultural crops are crops that bear fruit, such as tomatoes. Hereinafter, the case where the agricultural product is tomato will be described as an example. However, crops are not limited to tomatoes. For example, crops are fruits such as persimmons, cherries, strawberries, grapes, or tangerines. Alternatively, the crop may be a flower, a vegetable, or the like. Even if the crop is tomato or the like, the fruit-picking object may include leaves, stems or the like existing around the fruit.

The first input image data 11D1 and the second input image data 11D2 captured as described above are input to the learning data generation device 10. Next, when the first input image data 11D1 and the second input image data 11D2 are input, the learning data generation device 10 performs the whole processing to obtain image data for learning (hereinafter referred to as "learning data 15"). To generate. The learning data 15 generated in this way is input, and the AI 16 performs learning.

[Overall processing example]
FIG. 3 is a diagram showing an example of overall processing of the first embodiment.

In step S0301, the worker 14 captures the first input image data 11D1. That is, the worker 14 photographs the first crop 12 before performing the fruit-picking operation to generate the first input image data 11D1.

In step S0302, the worker 14 performs the fruit picking work. By this fruit-picking operation, the first crop 12 is in a state in which the fruit-picking target is excluded, and becomes the second crop 13. After such fruit thinning work, step S0303 is performed.

In step S0303, the worker 14 captures the second input image data 11D2. That is, the worker 14 photographs the second crop 13 after performing the fruit thinning work to generate the second input image data 11D2.

In step S0304, the learning data generation device 10 inputs the first input image data 11D1 and the second input image data 11D2.

In step S0305, the learning data generation device 10 extracts the fruit-picking object. For example, the learning data generation device 10 compares the first input image data 11D1 and the second input image data 11D2, and among all the target objects indicated by the first input image data 11D1, the second input image data 11D2. The target object that is no longer above is extracted as the fruit-picking target.

Therefore, the extraction result is in the form of image data or the like indicating the position of the fruit-picking object. Specifically, the extraction result is shown by processing the first input image data 11D1 and filling the area of the fruit-picking object with a predetermined color, hatching, or the like.

The extraction result is not limited to the image data format, and it is sufficient that the fruit-picking target can be specified. For example, when recognizing a target object in extraction, an identification number or a coordinate value in image data (a representative value such as a center of gravity may be used) is set for each target object. The extraction result may be generated in a format for specifying the fruit-picking target by designating such an identification number, a coordinate value, or the like.

However, it is assumed that the learning data generation device 10 can generate image data indicating the extraction result if there is data such as an identification number. Hereinafter, the extraction result will be described with an example in which the image data format is used.

The extraction result may be specified, corrected, or added by the user.

In step S0306, the learning data generation device 10 uses image data or the like indicating the extraction result as learning data and performs learning.

The learning data is image data or the like showing the extraction result or the like, that is, an image or the like in an illustrated format. However, the learning data may be image data in a plurality of formats. The format of the training data will be described later.

The learning may be repeated. That is, the learning may be repeated to the extent that step S0307 and step S0308, which will be described later, can be executed with a predetermined accuracy.

In step S0307, the learning data generation device 10 generates the estimation result image data.

In step S0308, the learning data generation device 10 identifies the estimation result image data.

It is desirable that step S0307 and step S0308 are realized, for example, in the following configuration.

[Example of generation and identification of image data by a hostile generation network (Generative Adversarial Networks, hereinafter referred to as "GAN")]
FIG. 4 is a diagram showing a configuration example of a hostile generation network. For example, it is desirable that the learning data generation device 10 has the following configuration by the extraction unit 10F2, the generation unit 10F3, the identification unit 10F4, and the like.

As shown in the figure, the GAN has a configuration in which the identification unit 10F4 distinguishes between the image data generated by the generation unit 10F3 and the image data indicating the extraction result by the extraction unit 10F2.

The generation unit 10F3 serves as a generator (also referred to as a Generator, a generation network, or the like) in a hostile generation network. That is, the generation unit 10F3 is a neural network model that creates image data.

The identification unit 10F4 serves as a discriminator (also referred to as a discriminator, an identification network, or the like) in a hostile generation network. That is, the identification unit 10F4 is a neural network model that discriminates whether or not the image data is the image data generated by the generator.

Hereinafter, the learning data used for learning the generator and the classifier constituting the GAN will be referred to as "first learning data". On the other hand, the learning data generated by the overall processing, that is, output based on the identification result of the identification unit 10F4, is referred to as "second learning data".

In the illustrated GAN, the image data indicating the extraction result (hereinafter, simply referred to as “extraction result 20”) is “genuine”. The extraction result 20 also serves as a "sample" of the generation unit 10F3. That is, the generation unit 10F3 has a configuration in which, for example, some extraction results 20 are learned in advance as the first learning data, and image data similar to the extraction results 20 can be generated with a certain degree of accuracy.

On the other hand, the image data (hereinafter referred to as "estimation result image data 21") showing the result of estimating the content of the fruit thinning work generated by the generation unit 10F3 is "fake".

In step S0307, the generation unit 10F3 generates the estimation result image data 21.

The estimation result image data 21 is image data generated by imitating the extraction result 20. Therefore, the estimation result image data 21 has the same format as the extraction result 20, and is image data for identifying the fruit-picking target. In this way, the generation unit 10F3 generates the estimation result image data 21 which is a “fake” with the aim of causing the identification unit 10F4 to identify it as the “genuine”.

However, since the estimation result image data 21 is image data generated by the generation unit 10F3, it is not image data indicating an actual agricultural product. In this way, the generation unit 10F3 and the identification unit 10F4, that is, GAN generate the composite image data.

Further, the estimation result image data 21 reproduces the fruit-picking operation indicated by the extraction result 20 in another crop. That is, the estimation result image data 21 shows the result of estimating the target object to be the fruit-picking target among all the target objects.

The generation unit 10F3 uses the extraction result 20 and the like as the first learning data in advance to learn the pattern of fruit-picking work and the like. Therefore, when the first input image data 11D1 indicating an unknown agricultural product is input, the generation unit 10F3 can first recognize the target object indicated by the first input image data 11D1 by prior learning.

Next, the generation unit 10F3 can estimate, by learning in advance, at what position the target object is to be picked, or how much is to be picked. .. Then, the generation unit 10F3 shows these estimation results in the form of image data, and generates the estimation result image data 21.

In step S0308, the extraction result 20 and the estimation result image data 21 are mixed, and the identification unit 10F4 identifies whether it is “genuine” or “fake”.

The generation unit 10F3 learns image processing and the like so as to generate the estimation result image data 21 so that the identification unit 10F4 can identify it as “genuine” as much as possible. On the other hand, the identification unit 10F4 learns to improve the accuracy of distinguishing the "counterfeit" from the "counterfeit" based on feedback or the like.

Specifically, with respect to the identification result by the identification unit 10F4, the first learning data indicates a "correct answer" as to whether the image data to be identified is "genuine" or "fake". Data (hereinafter referred to as "correct answer data 22") is prepared. Then, by collating the identification result with the correct answer data 22, it is possible to evaluate whether or not the identification unit 10F4 is the correct identification.

When such evaluation and identification results are fed back to the generation unit 10F3, the generation unit 10F3 aims to identify the estimation result image data 21 as "genuine" by the identification unit 10F4. Can be learned to generate. That is, the generation unit 10F3 learns so that it can generate a "counterfeit" that can be easily identified as a "genuine" by feedback.

Further, when the evaluation is fed back to the identification unit 10F4, the identification unit 10F4 can learn to improve the accuracy of distinguishing the "counterfeit" from the "counterfeit". That is, the identification unit 10F4 learns to reduce the probability of overlooking the "counterfeit" or misidentifying the "counterfeit" as the "genuine" by feedback.

The learning data generation device 10 may perform a learning process in which learning based on the first learning data in step S0306 is repeated in advance to give the generation unit 10F3 and the identification unit 10F4 a certain degree of accuracy.

Then, the estimation result image data 21 generated with a quality that can be identified as "genuine" by the identification unit 10F4 is used as the second learning data. By generating the learning data 15 in this way, the second learning data used by the AI 16 for learning can be increased.

On the other hand, the estimation result image data 21 identified as "fake" by the identification unit 10F4 is subject to "reuse". That is, the estimation result image data identified as "fake" is a result of insufficient learning.

Therefore, for example, an operation of correcting insufficient points is performed so that the estimation result image data identified as "fake" is identified as "genuine". In this way, the process of feeding back to the generation unit 10F3 by the image data or the like reflecting the manually operated contents is “reuse”. When such "reuse" is performed, the generation unit 10F3 can learn the insufficient points and generate the estimation result image data 21 that is more easily identified as "genuine".

Note that "reuse" is not limited to being used for learning of the generation unit 10F3. For example, “reuse” may include adding image data reflecting the manually operated contents to the learning data 15. However, if "reuse" is difficult, the estimation result image data identified as "fake" may be discarded.

The GAN as shown in the figure generates learning data 15 used for learning the AI 16. The second learning data generated in this way is for AI for estimating the fruit-picked portion of the crop, and is different from the image data evaluated visually by a human.

For example, when a general landscape or the like is photographed, the image data may have minute color changes or the like that are difficult for humans to visually judge. Such changes are less important in human visual assessment. On the other hand, in the evaluation by a computer, it may be possible to grasp by calculating the fluctuation of the pixel value and the like. As described above, in the generation of image data, the evaluation items to be emphasized may differ depending on whether the evaluation by a computer or the visual evaluation by a person is conscious.

[Example of shooting method]
It is desirable that the input image data such as the first input image data 11D1 and the second input image data 11D2 be photographed as follows, for example.

FIG. 5 is a diagram showing an example of a photographing method. Hereinafter, in the figure, the vertical direction is referred to as the "Z-axis direction". The Z-axis direction is the so-called gravity direction. Further, in the figure, the left-right direction is mainly defined as the "X-axis direction". The X-axis direction is the right-hand direction while facing the front of the crop. Further, the depth direction is defined as the "Y-axis direction".

Hereinafter, a case where the first crop 12 is photographed will be described as an example.

It is desirable that the input image data be photographed from a plurality of viewpoints around the Z axis. That is, it is desirable that the input image data is image data showing the first crop 12 from various viewpoints as much as possible.

Specifically, the camera 11 is moving toward the first crop 12 so as to rotate about the Z axis (so-called Yaw axis rotation, which is the rotation indicated by “Yaw” in the figure). It is desirable to shoot with.

When photographed in this way, the first crop 12 can be photographed from the entire circumference direction. The input image data may be a still image or the like that captures about 3 viewpoints out of 360 °.

The fruit-picking work may be carried out in consideration of the overall shape of the crop or the sunlight. Therefore, the fruit-picking object may exist at various angles. Therefore, the camera 11 may not be able to capture all the fruit-picking objects from one viewpoint. Therefore, it is desirable that the input image data be photographed from various viewpoints so as to have as few blind spots as possible.

It is more desirable that the input image data is further photographed from a plurality of viewpoints around the X axis. For example, the camera 11 has an optical axis directed toward the first crop 12, a viewpoint that is in front of the first crop 12, a viewpoint that photographs the first crop 12 from below (a so-called looking-up viewpoint), and a first. The photograph is taken from the viewpoint which is the back surface of the crop 12.

As described above, it is desirable that the camera 11 shoots so as to rotate about the X axis (so-called Pitch axis rotation, which is the rotation indicated by “Pitch” in the figure).

Further, it is desirable that the second input image data 11D2 is also photographed in the same manner.

As described above, it is desirable to rotate the Pitch or Yaw to photograph the crops from a plurality of viewpoints and capture the input image data. With such shooting, it is possible to grasp the fruit-picking work for adjusting the overall shape of the crop, the fruit-picking work for adjusting the sunnyness of the crop, and the like from the input image data.

Further, the input image data may be captured under different weather conditions, different arrangements of surrounding objects, and the like. That is, it is desirable that the input image data shows a state of being photographed under different ambient environments or different lighting conditions depending on the season, weather, and the like.

[Second Embodiment]
The second embodiment is different from the first embodiment in that the overall processing is as follows.

FIG. 6 is a diagram showing an example of overall processing of the second embodiment. Hereinafter, the points different from those of the first embodiment will be mainly described, and duplicate description will be omitted. The overall process in the second embodiment is different from the overall process in the first embodiment in that step S0601 is performed.

In step S0601, the learning data generation device 10 extracts the fruit-picking object. Specifically, the learning data generation device 10 extracts the fruit-picking object by performing the following extraction process.

FIG. 7 is a diagram showing an example of extraction processing. For example, step S0601 performs the following processing.

In step S0701, the learning data generation device 10 generates the first mask image data.

The first mask image data is mask image data that becomes learning data in the learning for instance segmentation (Instance Segmentation) performed in the subsequent step S0702. That is, the first mask image data is image data that serves as a "sample".

The first mask image data may be data that specifies a masked area such as filling a part or the whole of the image data.

Hereinafter, the first mask image data is referred to as training data for instance segmentation, and the mask image data generated by the instance segmentation is referred to as "second mask image data". The details of the mask image data will be described later.

In step S0702, the training data generation device 10 learns instance segmentation.

In step S0703, the training data generator 10 evaluates the instance segmentation.

In step S0704, the learning data generation device 10 generates the second mask image data for instance segmentation.

For example, instance segmentation and generation of mask image data are the following processes.

FIG. 8 is a diagram showing an example of instance segmentation processing and an example of mask image data. Hereinafter, the first input image data 11D1 shown in FIG. 8A will be described as an example.

For example, it is assumed that four target objects of the first object 31, the second object 32, the third object 33, and the fourth object 34 are photographed in the first input image data 11D1.

FIG. 8B is a diagram showing an execution result of instance segmentation and an example of mask image data 40 generated by the instance segmentation.

The instance segmentation is, for example, a process of generating the mask image data 40 shown in FIG. 8 (B) by executing the process on the first input image data 11D1 shown in FIG. 8 (A).

Specifically, the instance segmentation is a process of detecting an object and identifying a plurality of detected objects from different objects in the first input image data 11D1.

In the example shown in FIG. 8B, the area showing the first object 31, the second object 32, the third object 33, and the fourth object 34 (hereinafter, the area showing the target object in the image data is referred to as the “first area”. ”), Areas other than the first object 31, the second object 32, the third object 33, and the fourth object 34 (hereinafter referred to as“ second area ”. For example, the second area is a background or the like. ) Is an example of the mask image data 40 which is distinguished by two colors.

Specifically, as shown in FIG. 8B, in the mask image data 40, the first region is a region shown in white. On the other hand, in the mask image data 40, the second region is a region shown in black. As described above, the mask image data 40 is, for example, image data in which the first region and the second region are binarized and shown in different colors.

The mask image data 40 is not limited to the format shown in FIG. 8B. For example, the colors of the first region and the second region can be set in advance, and other color combinations may be used. Further, the mask image data 40 is not limited to a format in which regions are distinguished by color, and may be in a format such as the presence or absence of hatching or discrimination by identification data.

When the mask image data 40 is applied to the first input image data 11D1, the learning data generation device 10 can generate image data obtained by extracting the first region. That is, with reference to the mask image data 40, the learning data generation device 10 can recognize the target object in the first input image data 11D1 and generate image data obtained by extracting the target object.

By using the mask image data 40, the background and the like indicated by the first input image data 11D1 can be deleted. That is, if data other than the target object such as the background can be excluded in the learning, it is possible to prevent the AI from unnecessarily learning the object or the background which is not important in the fruit-picking work.

As described above, it is desirable that the mask image data 40 is image data capable of masking a region other than the first region, such as setting the background or the like as the second region.

Further, the mask image data 40 can identify individual target objects even if they are the same type of target objects. That is, when the mask image data 40 is applied, as shown in FIG. 8B, the first object 31, the second object 32, the third object 33, and the fourth object 34 are the first target objects 41 and the second object 34. It can be distinguished from different objects such as the target object 42, the third target object 43, and the fourth target object 44.

For example, in the process of semantic segmentation, the first target object 41, the second target object 42, the third target object 43, and the fourth target object 44 are classified into the same object or category and distinguished. Often not done.

On the other hand, in the case of instance segmentation processing, a plurality of pixels indicating one target object are collectively identified as one object, and if they are different objects of the same type, they are different objects. Can be identified.

That is, when the instance segmentation process is performed, so-called labeling becomes possible when there are a plurality of target objects of the same type in the image data. For example, in the example shown in FIG. 8B, the first target object 41, the second target object 42, the third target object 43, and the fourth target object 44 can be managed by different identification numbers or the like.

Therefore, the learning in step S0702 is performed to such an extent that the target object can be accurately identified. Then, the evaluation in step S0703 evaluates the accuracy of extracting the target object and the like. When such steps S0702 and S0703 are performed, in step S0704, the learning data generation device 10 can generate the second mask image data for instance segmentation.

Then, depending on the evaluation result of the instance segmentation, steps S0701 to S0703 are repeatedly executed. That is, the "learning process" and the process shown in FIG. 7 are repeatedly executed until a certain degree of accuracy is ensured, and then the "generation process" and the process shown in FIG. The process shown in 7 may be performed.

It is desirable that the learning data generation device 10 is further illustrated as in step S0705. For example, illustration is the following process.

FIG. 9 is a diagram showing an example of an illustration process. Hereinafter, a case where the first input image data 11D1 in the photographic format as shown in FIG. 9A is input will be described as an example.

The example shown in FIG. 9A shows an example in which the target object exists in the central portion of the image data (the portion in which the fruit is photographed. Hereinafter referred to as “target object region 51”). For example, the target object reflected in the target object area 51 is identified by object recognition such as instance segmentation.

The illustration process is, for example, a process of inputting the first input image data 11D1 and generating image data (hereinafter referred to as “illustrated image data 50”) as shown in FIG. 9 (B).

FIG. 9B is a diagram showing an example of the illustrated image data 50.

The illustrated image data 50 fills the area of the target object with a predetermined color. For example, as shown in FIG. 9B, the illustrated image data 50 is image data in which a region of a target object is filled, which is shown by hatching.

Hereinafter, in the example shown in FIG. 9B, the area that is identified as the area of the target object and is filled by the illustration process is referred to as “filled area 52”.

Further, in the illustrated image data 50, the area other than the filled area 52 (the area showing the background or the like) is white (the area is filled with a color different from the filled area 52, etc.).

As described above, the process of creating an illustration is a process of color-coding the region of the target object and the other regions with a predetermined color. By performing the illustration process in this way, the RGB value, the luminance value, and the like in the image data can be simplified.

In the case of image data in a photographic format such as the first input image data 11D1, there are many cases where there are small changes in RGB values or luminance values that are difficult for the human eye to understand.

For example, tomato fruits are simply one red color. In the case of indicating such an object, if the image data is in a photographic format, the pixel value indicating red in the same object may finely change the pixel value such as the RGB value. In many cases, it is better not to study such small changes in RGB values.

Therefore, in the illustration process, the target object is shown in the same color in a unified manner. Specifically, when instance segmentation or the like is performed on the first input image data 11D1, the pixels that can be identified as the target object are grouped.

The illustration process is a process of filling the same grouped pixels with the same color. Further, the illustration process is a process of painting an area such as a background with a color different from the area of the target object.

The illustration process may be a process other than filling with a predetermined color as long as it is a process for simplifying the image data. For example, the illustration process may be performed by making the background or the like a single color. Further, the illustration process may be a process using hatching or the like instead of painting with a color.

By making the image data into an illustration in this way, the extraction result and the like can be simplified and expressed. In many cases, the extraction result should be able to roughly express the position and shape of the target object. That is, in many cases, the extraction result does not require data such as fine color changes and backgrounds.

Therefore, if the target object is simply shown in a single color, it is possible to eliminate elements that hinder learning and to learn with high accuracy as compared with a photographic format or the like. That is, when the AI is made to learn the fruit-picking work by using the illustrated image data as the learning data, the AI can accurately learn the feature amount important for the fruit-picking work.

In addition, the amount of data can be reduced in the illustrated image data because the color expression and the like are simpler than the image data in the photographic format.

FIG. 10 is a diagram showing a modified example of the illustrated image data or the mask image data. For example, the mask image data may be generated as shown in FIG. 10 (B) or FIG. 10 (C). Hereinafter, the first input image data 11D1 shown in FIG. 10A will be described as an example.

FIG. 10A is a diagram showing an example of the first input image data 11D1 in which four apple fruits are targeted objects. Hereinafter, the learning data generation device 10 will be described with an example in which such first input image data 11D1 is input, and the learning data generation device 10 performs instance segmentation or the like.

For example, when performing the instance segmentation shown in FIG. 8, the second mask image data is generated as shown in FIG. 10 (B).

On the other hand, the second mask image data may be generated as shown in FIG. 10 (C).

FIG. 10B is a diagram showing an example of a format in which four target objects are collectively shown by one image data. As described above, the second mask image data may represent a plurality of target objects with one image data.

FIG. 10C is a diagram showing an example in which four target objects are divided into four image data for each target object and are in the form of an image data group. As described above, the second mask image data may be in the form of an image data group in which the image data is divided for each target object and a plurality of image data groups are used as one second mask image data.

As described above, the mask image data or the image data generated by making an illustration may be a plurality of target objects collectively as one image data, or may be separately divided for each target object as an image data group. May be good.

[Example of extraction result]
FIG. 11 is a diagram showing a recognition example of the target object. Hereinafter, the first input image data 11D1 shown in FIG. 11A will be described as an example.

When handling the target object shown in FIG. 11A, the learning data generation device 10 learns in advance the shape, color, combination thereof, and the like of the target object. When such learning is performed, for example, the learning data generation device 10 can recognize the target object as shown in FIG. 11B or FIG. 11C.

11 (B) and 11 (C) are examples in which the position where the target object is recognized, the range, and the like are surrounded by a broken line. The recognition result may be output in a format other than those shown in FIGS. 11 (B) and 11 (C).

FIG. 11B is a diagram showing a first example of the result of recognizing the target object. For example, as shown in FIG. 11B, the target objects are the first target object 101, the second target object 102, the third target object 103, the fourth target object 104, the fifth target object 105, and the sixth target object. Like 106 and the seventh object object 107, it is recognized by the learning data generator 10.

Further, the target object may be recognized in the form shown in FIG. 11C, for example.

FIG. 11C is a diagram showing a second example of the result of recognizing the target object. The second example is an example of a format in which the recognition result is divided into separate image data for each target object. Specifically, the learning data generation device 10 includes a first target object 101, a second target object 102, a third target object 103, a fourth target object 104, a fifth target object 105, a sixth target object 106, and the like. The recognition result of the seventh target object 107 is output separately for each target object.

The recognition result of the target object does not have to be in the form of image data as shown in FIG. 11 (B) or FIG. 11 (C). That is, the recognition result of the target object is an intermediate product, and parameters such as the position, size, range, number, or coordinates occupied by the target object in the image data (statistical value or representative value are used). Including)) may be in a format that can be grasped by the learning data generation device 10.

Therefore, the learning data generation device 10 does not have to store the parameters indicating the recognition result internally and output the image data or the like as shown in the figure.

In step S0306, the learning data generation device 10 trains the learning model using the learning data. For example, the learning data is the image data generated in step S0601, that is, the illustrated image data and the like. The learning data may be image data in a plurality of formats. The details of the training data will be described later.

[Example of processing result of whole processing]
FIG. 12 is a diagram showing a processing result example of the entire processing. Hereinafter, the case where FIGS. 12 (A) and 12 (B) are before and after fruit thinning will be described as an example.

FIG. 12A is a diagram showing an example of the first input image data 11D1.

FIG. 12B is a diagram showing an example of the second input image data 11D2.

FIG. 12C is a diagram showing an example of the second learning data.

Hereinafter, in the first input image data 11D1, that is, before the fruit thinning work, as shown in FIG. 12A, the first target object 101, the second target object 102, the third target object 103, and the fourth target object It is assumed that there are seven target objects of 104, a fifth target object 105, a sixth target object 106, and a seventh target object 107.

On the other hand, in the second input image data 11D2, that is, after the fruit thinning work is performed, as shown in FIG. 12B, the second target object 102, the third target object 103, and the fourth target object 104, In addition, four target objects of the sixth target object 106 are fruit-picking objects, and the fruit-picking target is an example of fruit-picking.

By comparing the first input image data 11D1 and the second input image data 11D2 in this way, the fruit-picking target can be extracted. When such an extraction result is learned, the learning data generation device 10 can estimate the fruit-picking operation and generate the estimation result image data when the unknown first input image data 11D1 is input.

The estimation result image data or the like generated in this way becomes the learning data 15. Then, the AI 16 learns the fruit-picking work by using the learning data 15 and the like as the second learning data.

FIG. 12C is a diagram showing an example of a format in which the target object is surrounded by a dotted line. Further, FIG. 12C is a diagram showing an example of a format in which the fruit-picking object is indicated by hatching.

The second learning data is not limited to the format shown in FIG. 12 (C). That is, for the second learning data, it is sufficient that AI16 can learn the position, number, arrangement, shape, range, etc. of the fruit-picking object. Therefore, in the second learning data, the fruit-picking target and the target object may be specified in other formats.

[Third Embodiment]
FIG. 13 is a diagram showing a configuration example of the learning device. Compared with the first embodiment and the like, the configuration of the learning data generation device 10 and the like in the third embodiment is the same as that of the first embodiment, for example. On the other hand, the learning device 301 is an information processing device or the like. The learning data generation device 10 and the learning device 301 may be the same information processing device or the like.

In the third embodiment, the learning model 302 is trained using the learning data 15 or the like generated by the configuration in the first embodiment or the second embodiment to generate the trained model 303.

Hereinafter, the AI during learning or before learning is simply referred to as "learning model 302". On the other hand, to some extent, the AI after learning with the second training data is called "trained model 303".

The learning device 301 inputs the learning data 15. Then, the learning device 301 trains the learning model 302 by the learning data 15.

In addition, data other than learning data 15 may be used for learning. For example, the learning device 301 may also input the first input image data 11D1 and the second input image data 11D2 to train the learning model 302. In addition, the learning device 301 may input the second learning data in the form of an extraction result or the like.

As described above, the learning device 301 trains the learning model 302 to generate the trained model 303. If such a trained model 303 can be generated, AI for estimating the fruit-picking object can be realized.

Specifically, the learning device 301 has the following configuration, for example.

FIG. 14 is a diagram showing an example of a configuration in which learning is performed by a learning device. As shown in the figure, the learning model 302 has at least a generation unit 10F3 and an identification unit 10F4.

Then, the generation unit 10F3 and the identification unit 10F4 are the generator and the classifier in the hostile generation network.

First, the learning device 301 inputs the first input image data 11D1.

Next, the generation unit 10F3 estimates the target object to be the fruit-picking target among the target objects indicated by the first input image data 11D1. Then, the generation unit 10F3 generates the estimation result image data 21. Hereinafter, as in FIG. 12C, an example of a format in which the target object is surrounded by a dotted line and the fruit-picking target is shown by hatching among the target objects will be described.

Next, when the estimation result image data 21 is generated, the identification unit 10F4 identifies the estimation result image data 21. Then, the identification unit 10F4 identifies the estimation result image data 21 by setting the learning data 15 as the “correct answer”.

Specifically, first, the estimation result image data 21 indicates the position, number, and the like of the fruit-picking object. On the other hand, the learning data 15 also shows the position, number, and the like of the fruit-picking object, like the estimation result image data 21.

In the example described below, the identification unit 10F4 refers to the estimation result image data 21 and identifies the “correct answer” when both the position and the number of the fruit-picking objects match the learning data 15.

On the other hand, if at least one of the position and the number of the fruit-picking target indicated by the estimation result image data 21 is different from the learning data 15, the identification unit 10F4 identifies it as an “wrong answer”.

Then, the identification unit 10F4 causes at least the generation unit 10F3 to feed back the identification result of the "correct answer" or the "wrong answer". As described above, the feedback is a process of transmitting the identification result from the identification unit 10F4 to at least the generation unit 10F3.

For the learning of the generation unit 10F3, the feedback may convey the identification process by the identification unit 10F4, the identification standard, the intermediate data generated during the identification, and the like. That is, the feedback may convey the process up to the output of the identification result and the data generated in the middle as a set with the identification result. Then, the generation unit 10F3 learns by referring to the feedback identification result. When data is transmitted as a set, the generation unit 10F3 may also refer to the set data for learning.

Specifically, in the example shown in FIG. 14, the estimation result image data 21 and the learning data 15 are shown by selecting a fruit-picking object from seven object objects. Then, when the estimation result by the estimation result image data 21 and the "correct answer" by the learning data 15 are compared, in this example, the target object located at the center (the target object indicated by the difference 151 in the figure) is picked. Whether it is an object or not is different.

Therefore, the comparison result of the estimation result image data 21 and the learning data 15 is identified as having a difference because the number of fruit-picking objects and the judgment result of the difference 151 are different. Therefore, based on the comparison result, the number and the position of the fruit-picking target are different from the reference learning data 15, so that the identification unit 10F4 identifies it as an "wrong answer".

The identification by the identification unit 10F4 may have an allowable range with respect to the standard. For example, the number may be set to allow as long as it is 2 or less with respect to the standard. In the case of setting such an allowable range, if there is only a difference of difference 151, the identification unit 10F4 identifies it as a "correct answer". In addition, there may be items that can be set in learning.

Then, for example, an expert looks at the plurality of estimation result image data 21 generated by the generation unit 10F3 and evaluates them. Specifically, if the generation unit 10F3 generates 100 estimation result image data 21 and the expert looks at the estimation result image data 21 and determines that all 100 images are okay, the generation unit 10F3 and the like can learn. Evaluated as completed.

By repeating the above generation and identification feedback, the learning device 301 can increase the accuracy of generating the estimation result image data 21.

In addition, it is desirable that the learning device 301 extracts the fruit-picking target in the generation or identification. Specifically, the learning device 301 performs processing such as generation of mask image data and illustration in generation or identification.

In this way, if the image data is masked, illustrated, or both are processed and extracted, the extraction result and the like can be simplified and expressed. In many cases, the extraction result only needs to be able to roughly express the position, shape, and the like of the target object. That is, in many cases, the extraction result does not require data such as fine color changes, subjects that are not related to fruit-picking work, and backgrounds.

In particular, in an environment with agricultural products, it is often difficult to adjust the surrounding environment for learning AI and for photography. In addition, the environment in which there are agricultural products is often an environment in which subjects with little relation to each other can easily enter. Therefore, if such disturbance can be reduced by the process of masking the image data, the AI can accurately learn the feature amount important for grasping the content of the fruit-picking work.

In addition, it is better to make the target object into an illustration and simply show it in a single color, or to use image data focusing on important parts, which eliminates elements that hinder learning the contents of fruit-picking work compared to photographic formats. And you can learn with high accuracy. That is, when the image data is subjected to the extraction process as the pre-process and the AI is made to learn the fruit-picking work, the AI can accurately learn the feature amount important for grasping the content of the fruit-picking work.

The identification unit 10F4 may improve the accuracy of identification by learning from the estimation result image data 21, the identification result, the learning data 15, and the like.

Further, the learning data 15 may be data generated by the learning data generation device 10, data generated by manipulating the first input image data 11D1, or a combination thereof.

Further, the formats of the estimation result image data 21 and the learning data 15 are not limited to the formats shown. That is, as for the format of the estimation result image data 21 and the learning data 15, it is sufficient that the content of the fruit thinning work can be specified. For example, the format of the estimation result image data 21 and the learning data 15 may be a format in which the position of the fruit-picking object and the content such as the number are numerical values (indicating the coordinates or quantity in the image).

The criteria for identification are not limited to the position and number of fruit-picking objects, and may be other criteria. Then, what is used as a reference for identification may be the subject of learning. In addition, a person may be able to set what criteria should be used for identification.

[Functional configuration example]
FIG. 15 is a diagram showing a functional configuration example of the learning device. For example, the learning device 301 has a functional configuration including an image data input unit 10F1, a learning data input unit 301F1, a generation unit 10F3, an identification unit 10F4, and the like. It is desirable that the learning device 301 has a functional configuration further including an extraction unit 10F2, a mask image data generation unit 10F5, and an illustration processing unit 10F6. Hereinafter, the illustrated functional configuration will be described as an example.

The image data input unit 10F1 performs an image data input procedure for inputting the first input image data 11D1. For example, the image data input unit 10F1 is realized by the camera 11, the interface 10H3, and the like.

The generation unit 10F3 performs a generation procedure for generating the estimation result image data 21. For example, the generation unit 10F3 is realized by the CPU 10H1 or the like.

The identification unit 10F4 performs an identification procedure of identifying the estimation result image data 21 by comparing with the learning data 15 and feeding back the identification result to the generation unit 10F3 to learn the learning model 302. For example, the identification unit 10F4 is realized by the CPU 10H1 or the like.

The estimation result image data 21 and the training data 15 are illustrated by either one or both of them generating mask image data by the extraction unit 10F2, the mask image data generation unit 10F5, and the illustration processing unit 10F6. It is desirable that an extraction process is performed in which the process is performed or both processes are performed.

In this way, when the fruit-picking target is extracted, the learning model 302 can accurately learn important features such as the fruit-picking target, as compared with the case where the image data obtained by simply photographing the crop is used as it is. .. That is, the learning device 301 can train the learning model 302 to generate a learned model 303 that can accurately estimate the fruit-picking operation.

[Fourth Embodiment]
FIG. 16 is a diagram showing a configuration example of a fruit thinning object estimation device. Hereinafter, an example of image data showing an unknown crop before fruit thinning will be referred to as “unknown image data 401”.

The fourth embodiment is an embodiment in which the trained model 303 generated by the learning according to the third embodiment is executed. Hereinafter, the fruit-picking object estimation device using the trained model 303 will be referred to as a “fruit-picking object estimation device 402”.

The fruit-picking object estimation device 402 is, for example, an information processing device such as a smartphone. The trained model 303 may be configured to be used by another server device or the like, and the fruit-picking object estimation device 402 may be configured to communicate with the server device to acquire and output the estimation result by the trained model 303. ..

Specifically, the trained model 303 is distributed via a network or the like. The trained model 303 may be in a format or the like incorporated in application software or the like. When the learned model 303 distributed in this way is installed in the fruit-picking object estimation device 402, the fruit-picking object estimation device 402 is in a state where it can perform estimation as shown in the figure and output of the estimation result.

The unknown image data 401 is image data captured by the fruit thinning object estimation device 402. Further, the crop shown by the unknown image data 401 is in a state before the fruit thinning work is performed. As described above, the crop indicated by the unknown image data 401 is an "unknown" crop different from the crop targeted for learning in the first embodiment or the second embodiment.

In addition, it is desirable that the fruit-picking target estimation device 402 extracts the fruit-picking target in the estimation. Specifically, it is desirable that the fruit-picking object estimation device 402 performs processing such as generation of mask image data and illustration in estimation. When such an extraction of the fruit-picking target is performed, the fruit-picking target estimation device 402 can perform the estimation with high accuracy.

The fruit thinning object estimation device 402 identifies the target object based on the unknown image data 401. Then, the fruit-picking object estimation device 402 estimates the fruit-picking target by the trained model 303. For example, the estimation result is output in the format of Augmented Reality (AR). Specifically, the fruit thinning object estimation device 402 displays the output screen 403 to the user 404.

The output screen 403 is a screen in which an “x” is superimposed on the unknown image data 401 to inform the user 404 of the fruit-picking target. The output may be in another display format or in a format such as using voice.

It is desirable that the fruit thinning object estimation device 402 has a configuration for accepting items on, for example, an operation screen of "optimization item setting" (hereinafter, simply referred to as "setting screen 405").

The fruit-picking work may be performed according to so-called preference. Therefore, the setting screen 405 is an interface for setting preferences and the like.

The setting screen 405 has items such as "sweetness", "sourness", "size (overall)", "size (grain)", "color", "uniformity", and "shape to fit in the case". This is an example of setting. The items and setting format will be determined in advance.

"Sweetness" and "sourness" are items for adjusting the taste of agricultural products at the time of harvesting.

"Size (overall)" is an item that adjusts the overall size of the crop at the time of harvest. For example, "size (overall)" is used to adjust the overall balance of a plurality of fruits in the case of a crop having a plurality of fruits.

"Size (fruit)" is an item for adjusting the size of one crop at the time of harvest. For example, "size (fruit)" is used to adjust the size of each fruit in the case of a crop having a plurality of fruits.

"Color" is an item for adjusting the color of crops at the time of harvest.

"Homogeneity" is an item for adjusting whether or not the size of the fruit of the crop at the time of harvest is made uniform.

"Making a shape that fits in a case" is an item for adjusting whether or not the size fits in a predetermined shape used for shipping. In this way, the items may be entered in the form of check boxes.

In addition, the size of the case can be specified numerically, for example, "to fit in a case" means "to fit in a case of length (mm) x width (mm) x height (mm)". The format may be used.

These items are items that can be adjusted by fruit thinning work. In addition, what kind of fruit-picking work affects which item is input to the learning data in the learning (that is, the third embodiment). For example, the learning model learns for each purpose of the fruit-picking work, such as a fruit-picking work in which the crop becomes sweet or a fruit-picking work in which the crop is enlarged. Therefore, the trained model can identify the fruit-picking operation that optimizes the item. Further, the degree (for example, sweetness, size, etc.) is input by, for example, a numerical value.

The reception unit that accepts items is not limited to the setting screen 405. That is, the items that can be set may include items other than those shown in the figure. Further, the reception unit may be an interface other than the taskbar or the check box. For example, the reception unit may be an interface for inputting in a text box or the like. Further, the item to be optimized may be fixed.

FIG. 17 is a diagram showing an example of a configuration in which estimation is performed by a fruit-picking object estimation device. For example, the trained model 302 has a configuration in which the identification unit 10F4 is removed from the generation unit 10F3 and the identification unit 10F4 that form the hostile generation network used in the third embodiment after learning according to the third embodiment. be.

That is, when the unknown image data 401 is input, the fruit-picking object estimation device 402 estimates the fruit-picking operation suitable for the target object indicated by the unknown image data 401. Then, the fruit-picking object estimation device 402 outputs the estimation result image data 21 showing the estimation result.

The function of the identification unit 10F4 may be stopped. That is, even if the learned model 302 has the identification unit 10F4 as in the learning model 302, the identification unit 10F4 may be stopped. On the other hand, the trained model 302 may have a configuration in which the identification unit 10F4 is removed or the identification unit 10F4 is not provided, and the identification unit 10F4 may not be provided at all.

[Functional configuration example]
FIG. 18 is a diagram showing a functional configuration example of the fruit thinning object estimation device. For example, the fruit thinning object estimation device 402 has a functional configuration including an image data input unit 10F1, an estimation unit 402F1, an output unit 402F2, and the like. It is desirable that the fruit thinning object estimation device 402 further includes an extraction unit 10F2, a mask image data generation unit 10F5, and an illustration processing unit 10F6. Hereinafter, the illustrated functional configuration will be described as an example.

The image data input unit 10F1 performs an image data input procedure for inputting the first input image data 11D1. For example, the image data input unit 10F1 is realized by the camera 11, the interface 10H3, and the like.

The estimation unit 402F1 performs an estimation procedure for estimating the fruit-picking target by the trained model 303. For example, the estimation unit 402F1 is realized by the CPU 10H1 or the like.

For example, the estimation unit 402F1 is composed of a generation unit 10F3 and the like.

The output unit 402F2 performs an output procedure for outputting the estimation result. For example, the output unit 402F2 is realized by an output device 10H5 or the like.

The unknown image data 401 is subjected to an extraction process in which the mask image data is generated, illustrated, or both are processed by the extraction unit 10F2, the mask image data generation unit 10F5, and the illustration processing unit 10F6. Is desirable.

In the estimation, the fruit-picking target estimation device 402 can accurately estimate the fruit-picking target and the like by paying attention to the learned elements as much as possible.

In this way, when the fruit-picking target is extracted from the unknown image data 401, the fruit-picking target estimation device 402 accurately adjusts the fruit-picking target, etc., as compared with the case where the image data obtained by simply photographing the crop is used as it is. Can be estimated well.

[Example of functional configuration of learning system]
FIG. 19 is a diagram showing a functional configuration example. For example, the learning data generation device 10 has a functional configuration including an image data input unit 10F1, an extraction unit 10F2, a generation unit 10F3, an identification unit 10F4, and the like. Further, as shown in the figure, the learning data generation device 10 preferably has a functional configuration further including a mask image data generation unit 10F5, an illustration processing unit 10F6, and the like.

The image data input unit 10F1 performs an image data input procedure for inputting the first input image data 11D1 and the second input image data 11D2. For example, the image data input unit 10F1 is realized by the camera 11, the interface 10H3, and the like.

The extraction unit 10F2 performs an extraction procedure for extracting the target object, which is the difference between the first input image data 11D1 and the second input image data 11D2, as the fruit-picking target object among the target objects. For example, the extraction unit 10F2 is realized by the CPU 10H1 or the like.

The generation unit 10F3 learns the image data indicating the extraction result as the first learning data, and performs a generation procedure for generating the estimation result image data. For example, the generation unit 10F3 is realized by the CPU 10H1 or the like.

The identification unit 10F4 performs an identification procedure of identifying the estimation result image data and generating the second learning data based on the identification result. For example, the identification unit 10F4 is realized by the CPU 10H1 or the like.

The mask image data generation unit 10F5 performs a mask image data generation procedure for generating mask image data that distinguishes between the target object and other than the target object. For example, the mask image data generation unit 10F5 is realized by the CPU 10H1 or the like.

The illustration processing unit 10F6 performs an illustration procedure for illustration of the target object and other than the target object. For example, the illustration processing unit 10F6 is realized by the CPU 10H1 or the like.

As described above, the learning data generation device 10 generates the second learning data such as the learning data 15. As described above, if the second learning data can be generated, the workload of preparing the learning data for AI for estimating the fruit-picked portion of the crop can be reduced as compared with the case where the learning data is manually generated. For example, it is necessary to prepare at least several thousand image data for the learning data for AI for estimating the fruit-picked part of the crop. It often takes at least one to several years to make such preparations.

In particular, agricultural products are often photographed under so-called natural light, such as outdoors. Under such a lighting environment, the lighting environment is often more unstable than in factories and the like. Specifically, it is difficult to artificially adjust sunlight and the like. Therefore, the natural light has various conditions such as the intensity, direction, and the presence or absence of shadows, as compared with the lighting of a factory or the like. Therefore, when shooting crops, the lighting environment is often stricter than indoors such as in factories. When AI is used under such a condition with a lot of disturbance, it is desirable that there is a large amount of training data.

The preparation period differs depending on the cycle of the target crop.

Further, if the estimation accuracy of AI is to be sufficiently improved, it is desirable that more training data are prepared. For example, based on Uncle Bernie's rule, etc., it is desirable to prepare training data that is 10 times or more the number of parameters in the neural network for AI training. Therefore, it may be desirable to prepare tens of thousands to hundreds of thousands or more image data for the learning data for AI for estimating the fruit-picked portion of the crop.

If the amount of learning data to be prepared is large, the preparation period becomes long and the workload tends to be large when the learning data is generated by photographing the actual agricultural products. As described above, when the workload becomes large, it causes an increase in development cost and a prolongation of development.

On the other hand, if the learning data can be generated as in the present embodiment, a large amount of learning data can be prepared with a small workload. Therefore, the workload of preparing the learning data can be reduced.

The learning device 301 has, for example, a functional configuration including a learning data input unit 301F1 and a learning unit 301F2.

The learning data input unit 301F1 performs a learning data input procedure for inputting the second learning data. For example, the learning data input unit 301F1 is realized by the interface 10H3 or the like.

The learning unit 301F2 performs a learning procedure for learning the learning model 302 using the second learning data. For example, the learning unit 301F2 is realized by the CPU 10H1 or the like.

As described above, the learning device 301 trains the learning model 302 using the second learning data or the like generated by the learning data generation device 10. By such learning, the learning device 301 can generate a trained model 303 that estimates the fruit-picking object. For example, the trained model 303 is used by the fruit thinning object estimation device 402 as follows.

The fruit-picking object estimation device 402 has a functional configuration including an image data input unit 10F1, an estimation unit 402F1, an output unit 402F2, and the like.

The image data input unit 10F1 performs an image data input procedure for inputting unknown image data 401. For example, the image data input unit 10F1 is realized by the camera 11, the interface 10H3, and the like.

The estimation unit 402F1 performs an estimation procedure for estimating the fruit-picking target by the trained model 303. For example, the estimation unit 402F1 is realized by the CPU 10H1 or the like.

The output unit 402F2 performs an output procedure for outputting the estimation result. For example, the output unit 402F2 is realized by an output device 10H5 or the like.

As described above, when the trained model 303 is mounted, the fruit-picking object estimation device 402 estimates the content of the fruit-picking work by the trained model 303, and the fruit-picking object (position, number, candidate, etc.) is estimated. Information is included.) Can be estimated. When such an estimation result is output, the user 404 can refer to the estimation result and perform an appropriate fruit-picking operation even if he / she is a beginner or the like. That is, even if the user 404 is a beginner or the like, the fruit left in the fruit-picking operation and the fruit to be picked can be grasped by referring to the estimation result.

The learning system 500 includes, for example, one of the functional configurations of the learning data generation device 10, the learning device 301, and the fruit-picking object estimation device 402.

Specifically, the learning system 500 is composed of a plurality of information processing devices such as a learning data generation device 10 and a learning device 301. With such a learning system 500, it is possible to generate learning data and train the learning model 302 to generate the trained model 303.

The learning system 500 is not limited to a plurality of information processing devices, and may be one information processing device.

Further, the learning system 500 may be a combination of the learning device 301 and the fruit thinning object estimation device 402.

[Example of functional configuration of estimation system]
FIG. 20 is a diagram showing a functional configuration example of the estimation system. For example, the estimation system 501 includes a learning data generation device 10, a learning device 301, a fruit thinning object estimation device 402, and the like. However, the estimation system 501 does not have to have the learning data generation device 10. That is, the estimation system 501 may use the captured image data as the learning data 15, the image data generated by the learning data generation device 10, or both.

The learning model 302 and the trained model 303 (including a program that uses the trained model 303) may be duplicated and have a plurality of learning devices 301, a fruit thinning object estimation device 402, and the like. ..

The learning device 301 has, for example, a functional configuration including an image data input unit 10F1, a learning data input unit 301F1, a learning unit 301F2, an extraction unit 10F2, a mask image data generation unit 10F5, an illustration processing unit 10F6, and the like.

The image data input unit 10F1 performs an image data input procedure for inputting the first input image data 11D1. For example, the image data input unit 10F1 is realized by the camera 11, the interface 10H3, and the like.

The learning data input unit 301F1 performs a learning data input procedure for inputting the learning data 15. For example, the learning data input unit 301F1 is realized by the interface 10H3 or the like.

The learning unit 301F2 performs a learning procedure for learning the learning model 302 based on the learning data 15. For example, the learning unit 301F2 is realized by the CPU 10H1 or the like.

The extraction unit 10F2 performs an extraction procedure for extracting the target object or the fruit-picking target from the first input image data 11D1 and the learning data 15. For example, the extraction unit 10F2 is realized by the CPU 10H1 or the like.

One or both of the first input image data 11D1 and the training data 15 generate mask image data by the extraction unit 10F2, the mask image data generation unit 10F5, and the illustration processing unit 10F6. It is desirable that an extraction process is performed in which either the conversion process is performed or both processes are performed.

In this way, when the target object or the fruit-picking target is extracted, the learning model 302 has important features such as the fruit-picking target, as compared with the case where the image data obtained by simply photographing the crop is used as it is. You can learn the amount accurately. That is, the learning device 301 can train the learning model 302 to generate a learned model 303 that can accurately estimate the fruit-picking operation.

As described above, the estimation system 501 trains the learning model 302 by the learning unit 301F2 to generate the learned model 303. In this way, the generated trained model 303 is sent to the fruit-picking object estimation device 402 via a network or the like.

The fruit-picking object estimation device 402 has a functional configuration including an image data input unit 10F1, an extraction unit 10F2, a mask image data generation unit 10F5, an illustration processing unit 10F6, an estimation unit 402F1, an output unit 402F2, and the like.

The image data input unit 10F1 performs an image data input procedure for inputting unknown image data 401. For example, the image data input unit 10F1 is realized by the camera 11, the interface 10H3, and the like.

The extraction unit 10F2 performs an extraction procedure for extracting the target object or the fruit-picking target in the unknown image data 401. For example, the extraction unit 10F2 is realized by the CPU 10H1 or the like.

The estimation unit 402F1 performs an estimation procedure for estimating the fruit-picking target by the trained model 303. For example, the estimation unit 402F1 is realized by the CPU 10H1 or the like.

The output unit 402F2 performs an output procedure for outputting the estimation result. For example, the output unit 402F2 is realized by an output device 10H5 or the like.

As described above, in the estimation system 501, the learning device 301 first trains the learning model 302 to generate the trained model 303. Next, in the estimation system 501, the trained model 303 thus generated is distributed to the fruit thinning object estimation device 402.

When the trained model 303 is mounted, the fruit-picking target estimation device 402 estimates the content of the fruit-picking work by the trained model 303, and the fruit-picking target (including information such as position, number, or candidate). Can be estimated. When such an estimation result is output, the user 404 can refer to the estimation result and perform an appropriate fruit-picking operation even if he / she is a beginner or the like.

That is, even if the user 404 is a beginner or the like, the fruit left in the fruit-picking operation and the fruit to be picked can be grasped by referring to the estimation result. Further, for example, when the learning device 301 uses a cloud environment or the like, data can be collected and the learned model 303 can be quickly distributed.

[About the format of learning data]
As the learning data such as the first learning data and the second learning data, it is desirable to use the image data in the format of extracting the agricultural products. However, the extraction may be performed in a plurality of stages. In such a case, the learning device 301 may include image data or the like in a format that is in the middle of the extraction in the learning data.

For example, the extraction process is divided into three stages, a first stage to a third stage.

The first stage is image data in the input state, that is, in the form of a photograph (however, white balance and the like may be adjusted).

The second stage is image data in a format in which a part other than the agricultural product is used as a background and the background is masked. For example, the background is masked to white (the color is set by the mask).

The third stage is image data in the form of illustrations of agricultural products and the like.

The learning data may be image data of any of the above-mentioned first to third stages. Further, the learning data may be not only the image data of any stage of the first to third stages described above, but also the image data of a plurality of stages, that is, both before and after the extraction process.

If the image data is in a format in which agricultural products are extracted by masking or the like, the learning device 301 can accurately learn the fruit-picking object in the learning model.

On the other hand, it may be desirable that the training data include image data in a format such as a photograph. For example, when illustrated, the image data may be caused by scratches or the like (for example, poor sunlight, salt damage, corrosion, illness, trauma, worm-eaten, etc.) occurring on the target object, or may be discolored or the like. ) May be omitted. On the other hand, in the fruit-picking operation, the target object having scratches or the like may be preferentially picked. It is desirable that the AI for such fruit-picking work is learned in the form of the first stage or the second stage, that is, the image data in the format of displaying scratches and the like. Therefore, the format of the training data may be selected according to the preference of the fruit-picking work and the like.

As described above, when the learning data is image data of a plurality of stages, the learning device 301 can make the learning model learn the fruit-picking work that more closely matches the preference.

[About AI]
AI processes image data and the like with the following network structure, for example.

FIG. 21 is a diagram showing an example of a network structure. For example, the AI may have a network structure having an input layer L1, a hidden layer L2, and an output layer L3.

Specifically, AI is a network structure having a Convolution Natural Network (convolutional neural network, CNN) or the like as shown in the figure.

The input layer L1 is a layer for inputting input data DIN.

The hidden layer L2 is a layer that performs processing such as convolution, pooling, normalization, or a combination thereof with respect to the input data DIN input from the input layer L1.

The output layer L3 is a layer that outputs the result processed by the hidden layer L2 as the output data DOUT. For example, the output layer L3 is composed of a fully connected layer or the like.

Convolution is, for example, an image or a feature map generated by performing a predetermined process on an image based on a filter, a mask, a kernel (hereinafter, simply referred to as "filter"), or the like. On the other hand, it is a process of performing a filter process and generating a feature map.

Specifically, the filter is data used for calculating the filter coefficient (sometimes referred to as "weight" or "parameter") by multiplying the pixel value of the image or feature map. The filter coefficient is a value determined by learning or setting.

Then, the convolution process is a process of multiplying each pixel value of the pixels constituting the image or the feature map by a filter coefficient to generate a feature map having the calculation result as a component.

When the convolution process is performed in this way, the features of the image or the feature map can be extracted. The feature is, for example, the edge component or the result of statistically processing the periphery of the target pixel.

Further, when the convolution process is performed, the target image or the subject indicated by the feature map is an image or feature map that shifts up and down, shifts left and right, shifts diagonally, rotates, or is a combination thereof. However, the same characteristics can be extracted.

Pooling is a process of extracting features and generating a feature map by performing processing such as average calculation, extraction of the minimum value, or extraction of the maximum value for a target range. That is, the pooling is max pooling, avg pooling, or the like.

The convolution and pooling may be pretreated such as zero padding.

By convolution, pooling, or a combination thereof as described above, a so-called data amount reduction effect, syntheticity, movement invariance, or the like can be obtained.

Normalization is, for example, a process of aligning variances and averaging values. The normalization includes the case where it is performed locally. Then, when normalization is performed, the data becomes a value or the like within a predetermined range. Therefore, the data can be easily handled in the subsequent processing.

Fully connected is a process of dropping data such as a feature map into the output.

For example, the output is in the form of a binary output, such as "YES" or "NO". In such an output format, full coupling is a process of joining nodes based on the features extracted in the hidden layer L2 so that one of the two conclusions can be reached.

On the other hand, when there are three or more types of outputs, the full coupling is a process of performing a so-called softmax function or the like. In this way, by full coupling, classification (including the case where an output indicating a probability is performed) can be performed by a maximum likelihood estimation method or the like.

[Other Embodiments]
The learning data generation device 10, the learning device 301, and the fruit thinning object estimation device 402 may be different types of information processing devices. That is, the learning data generation device 10, the learning device 301, and the fruit thinning object estimation device 402 may have different hardware configurations.

The learning data may be referred to as teacher data, training data, or the like.

The embodiment may be a combination of the above embodiments. That is, the device that generates the training data, the device that performs the learning process on the learning model to generate the trained model, and the device that performs the execution process using the trained model may be the same device or different devices. It may be. As described above, the learning of the learning model and the execution by the trained model do not have to be performed by the same information processing device. That is, the learning of the learning model and the execution by the trained model may be performed by different information processing devices.

In the case of different devices, the devices send and receive training data or data such as a trained model via a network or the like, for example.

Therefore, the trained model may be generated by learning, then distributed in the form of a program or the like via a network or the like, and executed by a device different from the trained information processing device. In addition, learning may be additionally performed with respect to the learning model generated by learning in another information processing apparatus.

The training data may be data-expanded (data augmentation). Specifically, in the case of image data, the training data may be data-expanded such as cutting out a part of the image indicated by the image data to generate new data.

Similarly, data expansion includes, for example, rotation, sliding, partial shearing of data, left-right reversal, upside-down, distortion, distortion correction, shading change, color correction, noise reduction, noise addition, etc. This is a process of randomly applying a filter, enlargement, reduction, edge enhancement, or a combination of these processes to image data.

If the learning data can be increased by data expansion in this way, the learning data used for learning the learning model can be increased.

In the embodiment, a process for reducing overfitting (also referred to as “overfitting” or “overfitting”, etc., overfitting) such as batch normalization or dropout may be performed. In addition, processing such as dimension reduction may be performed.

The network structure in the learning model, the trained model, and the like is not limited to the network structure of CNN. For example, the network structure may have a configuration such as RNN (Recurrent Neural Network), LSTM (Long Short-Term Memory), or Transformer.

Further, the learning model and the trained model may have a configuration having hyperparameters. That is, the learning model and the trained model may be configured in which the user makes some settings.

In addition, for example, when dealing with a graph (data composed of vertices and edges), the learning model and the trained model may be a Graph Neural Network (GNN) or the like. It may have a structure.

Further, the learning model and the trained model may use other machine learning. For example, the learning model and the trained model may be normalized or the like by preprocessing by an unsupervised model.

The present invention includes a learning data generation method, a learning method, an estimation method exemplified above, or a program (firmware and a program equivalent to the program) that executes a process equivalent to the process shown above. Hereinafter, simply "program". It may be realized by.).

That is, the present invention may be realized by a program or the like described in a programming language or the like so that a predetermined result can be obtained by giving a command to a computer. The program may be configured to execute a part of the processing by hardware such as an integrated circuit (IC) or an arithmetic unit such as a Graphics Processing Unit (GPU).

The program causes the computer to execute the above-mentioned processing and the like in cooperation with the arithmetic unit, the control device, the storage device, and the like possessed by the computer. That is, the program is loaded into the main storage device or the like, and issues an instruction to the arithmetic unit to perform an arithmetic operation to operate the computer.

The program may also be provided via a computer-readable recording medium or a telecommunication line such as a network.

The present invention may be realized in a system composed of a plurality of devices. That is, a system using a plurality of computers may execute the above-mentioned processes in a redundant, parallel, distributed manner, or a combination thereof. Therefore, the present invention may be realized by a device other than the hardware configuration shown above and a system other than the device shown above.

The present invention is not limited to the above-exemplified embodiments. Therefore, the present invention can add or modify components without departing from the technical gist. Therefore, all the technical matters included in the technical idea described in the claims are the subject of the present invention. The embodiments illustrated above are specific examples suitable for implementation. A person skilled in the art can realize various modified examples from the disclosed contents, and such modified examples are included in the technical scope described in the claims.

10: Learning data generation device 10F1: Image data input unit 10F2: Extraction unit 10F3: Generation unit 10F4: Identification unit 10F5: Mask image data generation unit 10F6: Illustration processing unit 11: Camera 11D1: First input image data 11D2: First 2 Input image data 12: 1st agricultural product 13: 2nd agricultural product 14: Worker 15: Learning data 20: Extraction result 21: Estimated result image data 22: Correct answer data 31: 1st object 32: 2nd object 33: 3rd Object 34: Fourth object 40: Mask image data 41: First object object 42: Second object object 43: Third object object 44: Fourth object object 50: Illustrated image data 51: Target object area 52: Filled area 101: 1st object 102: 2nd object 103: 3rd object 104: 4th object 105: 5th object 106: 6th object 107: 7th object 301: Learning device 301F1: Learning data Input unit 301F2: Learning unit 302: Learning model 303: Learned model 401: Unknown image data 402: Fruit thinning object estimation device 402F1: Estimating unit 402F2: Output unit 403: Output screen 404: User 405: Setting screen 500: Learning system

Claims (9)

A learning data generation device that generates learning data, a learning device that has a generation unit and an identification unit and trains a learning model using the learning data, and fruit picking using a learned model trained by the learning device. An estimation system that has an object estimation device.
The learning data generator is
An image data input unit for inputting a first input image data which is image data showing an agricultural product before fruit thinning and a second input image data which is an image data showing the agricultural product after fruit thinning.
Extraction of the target object which is the difference between the first input image data and the second input image data among the target objects which are the fruits, flowers, leaves, or a combination thereof of the agricultural crop as the fruit-picking target. Department and
A generation unit that learns using the first learning data including the extraction result by the extraction unit and generates an estimation result image data showing the result of estimating the fruit-picking object.
It is provided with an identification unit that identifies the estimation result image data and generates a second learning data based on the identification result.
The learning device is
A learning data input unit for inputting the second learning data,
The generation unit that generates the estimation result image data showing the result of estimating the fruit-picking object by the second learning data, and the generation unit.
It is provided with the identification unit that identifies the estimation result image data as compared with the learning data and feeds back the identification result to the generation unit to train the learning model.
The fruit thinning object estimation device is
An image data input unit that inputs unknown image data indicating an unknown crop before fruit thinning,
An estimation unit that estimates the fruit-picking object using the trained model,
An estimation system including an output unit that outputs an estimation result by the estimation unit. The learning device is
The estimation system according to claim 1, further comprising a mask image data generation unit that generates mask image data that distinguishes between the target object and other than the target object. The mask image data generation unit
The estimation system according to claim 2, wherein the mask image data for performing instance segmentation for identifying the target object is generated. The learning device is
The estimation system according to any one of claims 1 to 3, further comprising an illustration processing unit for illustration of the target object. In the learning device
The first input image data and the second input image data are
The estimation system according to any one of claims 1 to 4, which is image data showing a moving image or a plurality of still images taken from a plurality of viewpoints and showing the entire circumference of the crop. In the learning device
The generation unit and the identification unit
Configure a hostile generation network,
The generator
The estimation result image data is generated with the aim of being identified as genuine by the identification unit.
The identification unit
It is identified whether it is a genuine product showing the extraction result or a fake product which is the estimation result image data generated in the generation unit.
The generation unit and the identification unit
The estimation system according to any one of claims 1 to 5, which generates the estimation result image data identified as genuine by the identification unit as the second learning data. A learning system having a learning data generator that generates learning data and a learning device that trains a learning model using the learning data.
The learning data generator is
An image data input unit for inputting a first input image data which is image data showing an agricultural product before fruit thinning and a second input image data which is an image data showing the agricultural product after fruit thinning.
Extraction of the target object which is the difference between the first input image data and the second input image data among the target objects which are the fruits, flowers, leaves, or a combination thereof of the agricultural crop as the fruit-picking target. Department and
A generation unit that learns using the first learning data including the extraction result by the extraction unit and generates an estimation result image data showing the result of estimating the fruit-picking object.
With the identification unit that identifies the estimation result image data and generates the second learning data based on the identification result.
With
The learning device is
A learning data input unit for inputting the second learning data,
A generation unit that generates estimation result image data showing the result of estimating the fruit-picking object based on the second learning data, and a generation unit.
With the identification unit that identifies the estimation result image data in comparison with the learning data and feeds back the identification result to the generation unit of the learning device to learn the learning model.
Learning system with. An image data input unit for inputting a first input image data which is image data showing an agricultural product before fruit thinning and a second input image data which is an image data showing the agricultural product after fruit thinning.
Extraction of the target object which is the difference between the first input image data and the second input image data among the target objects which are the fruits, flowers, leaves, or a combination thereof of the agricultural crop as the fruit-picking target. Department and
A generation unit that learns using the first learning data including the extraction result by the extraction unit and generates an estimation result image data showing the result of estimating the fruit-picking object.
A learning data generation device including an identification unit that identifies the estimation result image data and generates a second learning data based on the identification result. A program that causes a computer to execute a learning data generation method.
An image data input procedure in which a computer inputs a first input image data which is image data showing an agricultural product before fruit thinning and a second input image data which is an image data showing the agricultural product after fruit thinning.
The computer uses the target object that is the difference between the first input image data and the second input image data among the target objects that are the fruits, flowers, leaves, or a combination thereof as the fruit picking target. Extraction procedure and extraction procedure
A generation procedure in which a computer learns from the first learning data including the extraction result by the extraction procedure and generates an estimation result image data showing the result of estimating the fruit-picking object.
A program for a computer to identify the estimation result image data and execute an identification procedure for generating second learning data based on the identification result.

Images