JP2022114418A - Training device of artificial intelligence (ai), picking object estimation device, estimation system, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a training device, a picking object estimation device, an estimation system, and a program for estimating the object to be picked in farm crop picking work with high accuracy by AI.

SOLUTION: An estimation system 501 comprises a training data generator, a training device and a picking object estimation device. The training data generator generates training data from image data before picking and image data after picking. In the training device, a training data input unit performs inputting the training data, estimation result image data is generated by a generation unit that is the estimation result, for the object to be picked, of having used the image data before picking and the training data; the estimation result image data is identified using the training data by an identification unit; and a learning model is trained with the identification result fed back to the generation unit. In the picking object estimation device, an image data input unit performs inputting unknown image data that indicates an unknown farm crop before picking. An estimation unit estimates the object to be picked by the trained learning model. An output unit outputs the result of estimation by the estimation unit.

SELECTED DRAWING: Figure 20

COPYRIGHT: (C)2022,JPO&INPIT

Description

The present invention relates to an artificial intelligence (AI) learning device, thinning target object estimation device, estimation system, and program.

Techniques for estimating or recognizing various objects based on the current state or the like using artificial intelligence (hereinafter referred to as "AI") are known.

For example, there is a robot system used in a factory or the like in which a robot hand is installed on a conveyor. Specifically, the robot system first photographs an object with a camera. After photographing, the object is image-recognized based on the photographed image. The robot system then calculates the position of the center of gravity of the object based on the captured image. Based on the position of the center of gravity calculated in this way, the robot system determines an accurate position for gripping the object with the robot hand. Techniques for stably gripping an object with a robot hand in this manner are known (see, for example, Patent Document 1 and the like).

Recognition of objects by AI is also used in agricultural situations. Specifically, there is known a technique in which AI automatically determines the number of grapes in grape picking (see, for example, Non-Patent Document 1).

JP 2019-185204 A

Kenichi Shoji, “Automatic judgment of “How many grapes are there?” AI will be put into practical use next summer-Joint development by University of Yamanashi and agricultural production corporation”, [online], August 17, 2020, DG Lab Haus, [ Retrieved December 2, 2020], Internet, <URL: https://media.dglab.com/2020/08/17-grape-01/>

The technology described in Patent Document 1 is a technology that assumes a lighting environment such as in a factory. In other words, the lighting environment such as in a factory is an environment in which conditions such as light are stable compared to the lighting environment under natural light such as outdoors for performing processing such as shooting and image recognition. often. Therefore, there is a problem that it is difficult to apply the technology that assumes the lighting environment such as in a factory to the lighting environment such as handling agricultural products.

In addition, in the technique as described in Non-Patent Document 1 above, sufficient learning data must be secured in order for AI to learn. In particular, in order to improve the accuracy of AI, it is desirable to secure a large amount of learning data. Therefore, with the technique as described in Non-Patent Document 1, it is difficult to estimate the thinning object with AI with high accuracy.

An object of the present invention is to highly accurately estimate a thinning object in thinning work of agricultural crops by AI.

In order to solve the above problems, in one aspect of the present invention,
A learning device for learning a learning model having a generation unit and a recognition unit,
an image data input unit for inputting first input image data, which is image data representing crops before thinning, and second input image data, which is image data representing the crops after thinning;
the generation unit that generates estimation result image data indicating a result of estimating a thinning target object in the crop;
and the identification unit that identifies the estimation result image data and feeds back the identification result to the generation unit to learn the learning model.

ADVANTAGE OF THE INVENTION According to this invention, the thinning target object in fruit thinning work of agricultural products can be estimated with high precision by AI.

It is a figure which shows the whole structural example of the learning data generation apparatus for AI. It is a figure which shows the hardware structural example of an information processing apparatus. It is a figure which shows the example of the whole process of 1st Embodiment. FIG. 4 is a diagram showing a configuration example of a generative adversarial network; It is a figure which shows the example of the imaging|photography method. It is a figure which shows the example of the whole process of 2nd Embodiment. It is a figure which shows the example of an extraction process. FIG. 10 is a diagram illustrating an example of instance segmentation processing and an example of mask image data; It is a figure which shows the processing example of illustration. FIG. 10 is a diagram showing a modified example of illustrated image data or mask image data; It is a figure which shows the recognition example of a target object. FIG. 10 is a diagram illustrating an example of a processing result of overall processing; It is a figure which shows the structural example of a learning apparatus. It is a figure which shows the example of the structure which learns by a learning apparatus. It is a figure which shows the functional structural example of a learning apparatus. It is a figure which shows the structural example of a thinning target object estimation apparatus. It is a figure which shows the example of the structure which estimates by a thinning target object estimation apparatus. It is a figure which shows the functional structural example of a thinning target object estimation apparatus. It is a figure which shows the functional structural example of a learning system. It is a figure which shows the functional structural example of an estimation system. It is a figure which shows an example of a network structure.

A specific example will be described below with reference to the accompanying drawings. In the following description, reference numerals in the drawings refer to the same elements when the reference numerals are the same.

[First embodiment]
FIG. 1 is a diagram showing an example of the overall configuration of a learning data generation device for AI. For example, a learning data generation device for AI (hereinafter referred to as "learning data generation device 10") is used as follows.

The learning data generation device 10 is, for example, an information processing device such as the following.

[Hardware Configuration Example of Information Processing Device]
FIG. 2 is a diagram illustrating a hardware configuration example of an information processing apparatus. For example, the learning data generation device 10 has a hardware configuration including a Central Processing Unit (CPU, hereinafter referred to as "CPU 10H1"), a storage device 10H2, an interface 10H3, an input device 10H4, an output device 10H5, and the like. Further, the learning data generation device 10 preferably has a hardware configuration including a Graphics Processing Unit (GPU, hereinafter referred to as "GPU 10H6").

The CPU 10H1 is an example of an arithmetic device and a control device. For example, the CPU 10H1 performs calculations based on programs, operations, or the like.

The storage device 10H2 is a main storage device such as a memory. The storage device 10H2 may be an SSD Solid State Drive (SSD) or an auxiliary storage device such as a hard disk.

The interface 10H3 transmits and receives data to and from an external device via a network, cable, or the like. For example, the interface 10H3 is a connector, an antenna, or the like.

The input device 10H4 is a device for inputting an operation by the user. For example, the input device 10H4 is a mouse, keyboard, or the like.

The output device 10H5 is a device that outputs processing results and the like to the user. For example, the output device 10H5 is a display or the like.

The GPU 10H6 is an arithmetic unit for image processing. Note that the GPU 10H6 may also be called a graphic controller or the like. In particular, the GPU 10H6 is used for real-time image processing or for parallel computation in learning.

Note that the learning data generation device 10 may have a hardware configuration that further includes internal or external hardware resources other than those described above. Also, the learning data generation device 10 may be a plurality of devices.

[Regarding crops, target objects, thinning targets, and fruit thinning work]
The learning data generating device 10 generates image data (hereinafter referred to as "first input image data 11D1”). Note that the photographing device such as the camera 11 may be configured in the learning data generation device 10 .

Furthermore, the learning data generation device 10 generates image data (hereinafter referred to as "second Input image data 11D2”) is input.

Hereinafter, the first input image data 11D1 and the second input image data 11D2 may be collectively referred to simply as "input image data".

The first input image data 11D1 and the second input image data 11D2 are moving images, still images, or a combination thereof. When inputting in the form of a moving image, for example, one frame or a predetermined number of frames are extracted from a plurality of frames constituting the moving image and used as input image data.

Fruit thinning work is work to thin out fruits, flowers, leaves, or a combination thereof (hereinafter referred to as “target objects”) that are present in or around the crops. That is, the fruit thinning work is grain thinning, fruit thinning, flower thinning, or a combination thereof.

Hereinafter, among the target objects, an object to be thinned out in the fruit thinning operation is referred to as a “thinning object”. In other words, the thinning work is a work of selecting and thinning some thinning objects among a plurality of target objects. In addition, in the figure, the thinning object is indicated by "x" to indicate that it is thinned out. However, it does not matter how the target object and the thinning target object are distinguished from each other.

The operator 14 decides which of the target objects is to be the thinning target.

For example, the thinning target may differ depending on the purpose even if the crop is the same. First, the purpose is, for example, to make the crops evenly exposed to the sun overall, to adjust the taste, to make the crops dense to some extent, to make the crops fit in a predetermined size, or to harvest. Sometimes, the appearance of crops (color, shape, fewer damaged target objects, or overall appearance of these, etc.) is improved.

The worker 14 performs a sample thinning operation on the first crop 12 based on the purpose of thinning. Then, the operator 14 separately photographs before and after the fruit thinning work. Each such capture produces input image data.

Also, the input image data is captured separately depending on the purpose of fruit thinning, the type of crops, or the like. In other words, the content of the fruit thinning work may differ depending on the purpose. Therefore, the input image data is generated separately according to the purpose, the type of crops, or the like. The worker 14 is, for example, a skilled farmer or the like in order to show the sample fruit thinning work.

When the first input image data 11D1 and the second input image data 11D2 are compared, the learning data generation device 10 or the like determines which part of the target object is the thinning target and how much the thinning target is. It is possible to grasp whether or not

Agricultural crops are, for example, agricultural crops that bear fruit such as tomatoes. In the following, the case where the crop is tomato will be described as an example. However, crops are not limited to tomatoes. For example, crops are fruits such as persimmons, cherries, strawberries, grapes, or oranges. Alternatively, the crops may be flowers, vegetables, or the like. Note that even if the crop is a tomato or the like, the thinning object may include leaves, stems, or the like around the fruit.

The first input image data 11D1 and the second input image data 11D2 photographed as described above are input to the learning data generating device 10. FIG. Next, when the first input image data 11D1 and the second input image data 11D2 are input, the learning data generation device 10 generates image data for learning (hereinafter referred to as "learning data 15") through overall processing. to generate The learning data 15 generated in this manner is input to the AI 16 to perform learning.

[Overall processing example]
FIG. 3 is a diagram illustrating an example of overall processing according to the first embodiment.

In step S0301, the operator 14 photographs the first input image data 11D1. That is, the worker 14 photographs the first crop 12 before performing the fruit thinning work to generate the first input image data 11D1.

In step S0302, the operator 14 performs fruit thinning work. As a result of this thinning operation, the first crop 12 becomes a second crop 13 from which the target of thinning has been removed. After such thinning work, step S0303 is performed.

In step S0303, the operator 14 photographs the second input image data 11D2. That is, the worker 14 photographs the second crop 13 after performing the fruit thinning work, and generates the second input image data 11D2.

In step S0304, the learning data generation device 10 inputs the first input image data 11D1 and the second input image data 11D2.

In step S0305, the learning data generation device 10 extracts a thinning object. For example, the learning data generation device 10 compares the first input image data 11D1 and the second input image data 11D2, and out of all the target objects indicated by the first input image data 11D1, the second input image data 11D2 In the above, the missing target object is extracted as the thinning target object.

Therefore, the extraction result is in the form of image data or the like indicating the position of the object to be thinned. Specifically, the extraction result is indicated by processing the first input image data 11D1 and filling in the region of the thinning object with a predetermined color, hatching, or the like.

Note that the extraction result is not limited to the image data format, as long as the thinning object can be specified. For example, when recognizing a target object in extraction, an identification number or a coordinate value in image data (a representative value such as the centroid may be used) is set for each target object. The extraction result may be generated in a format that identifies the thinning object by designating such an identification number or coordinate values.

However, it is assumed that the learning data generation device 10 can generate image data indicating the extraction result if there is data such as an identification number. In the following, extraction results will be described using an example in the form of image data.

Note that the extraction result may be specified, corrected, or added by the user.

In step S0306, the learning data generation device 10 performs learning using image data or the like representing the extraction result as learning data.

The learning data is image data or the like indicating the extraction result or the like, that is, an illustrated image or the like. However, the learning data may be image data in multiple formats. The format of learning data will be described later.

In addition, learning may be performed repeatedly. That is, the learning may be repeated to the extent that steps S0307 and S0308, which will be described later, can be executed with a predetermined accuracy.

In step S0307, the learning data generation device 10 generates estimation result image data.

In step S0308, the learning data generation device 10 identifies the estimation result image data.

Steps S0307 and S0308 are desirably realized by, for example, the following configuration.

[Example of generation and identification of image data by generative adversarial networks (hereinafter referred to as "GAN")]
FIG. 4 is a diagram showing a configuration example of a generative adversarial network. For example, the learning data generation device 10 preferably has the following configuration with the extraction unit 10F2, the generation unit 10F3, the identification unit 10F4, and the like.

As illustrated, the GAN has a configuration in which the identification unit 10F4 distinguishes between image data generated by the generation unit 10F3 and image data representing the extraction result of the extraction unit 10F2.

The generation unit 10F3 becomes a generator in the hostile generation network (also called a generator, a generation network, etc.). That is, the generation unit 10F3 is a neural network model that generates image data.

The identification unit 10F4 serves as a discriminator (also called a discriminator, identification network, etc.) in the hostile generation network. That is, the identification unit 10F4 is a neural network model that identifies whether image data is image data generated by a generator.

Hereinafter, the learning data used for learning the generator and classifier that constitute the GAN will be referred to as "first learning data". On the other hand, learning data generated by the overall process, that is, output based on the identification result of the identification unit 10F4 is referred to as "second learning data".

In the illustrated GAN, the image data representing the extraction result (hereinafter simply referred to as "extraction result 20") is "genuine". The extraction result 20 also serves as a "sample" for the generation unit 10F3. That is, the generation unit 10F3 is configured to learn in advance some extraction results 20 as first learning data, for example, and generate image data that resembles the extraction results 20 with a certain degree of accuracy.

On the other hand, the image data (hereinafter referred to as "estimation result image data 21") indicating the result of estimating the content of the fruit thinning work generated by the generation unit 10F3 is "fake".

In step S<b>0307 , the generation unit 10F3 generates estimation result image data 21 .

The estimation result image data 21 is image data generated by imitating the extraction result 20 . Therefore, the estimation result image data 21 has the same format as the extraction result 20, and is image data that specifies the thinning object. In this way, the generation unit 10F3 generates the estimation result image data 21, which is a "fake", with the aim of causing the identification unit 10F4 to identify it as "genuine".

However, since the estimation result image data 21 is image data generated by the generation unit 10F3, it is not image data representing an actual agricultural product. Thus, the generation unit 10F3 and the identification unit 10F4, that is, the GAN generate synthetic image data.

In addition, the estimation result image data 21 reproduces the fruit thinning work indicated by the extraction result 20 in another crop. That is, the estimation result image data 21 indicates the result of estimating the target object to be the thinning target among all the target objects.

The generation unit 10F3 previously learns the pattern of the fruit thinning work using the extraction result 20 and the like as first learning data. Therefore, when the first input image data 11D1 representing unknown crops is input, the generation unit 10F3 can first recognize the target object represented by the first input image data 11D1 through prior learning.

Next, the generation unit 10F3 can estimate, by prior learning, which position of the recognized target object is to be the thinning target object, or how much of the target object is to be the thinning target object. . The generation unit 10F3 generates estimation result image data 21 by presenting these estimation results in the form of image data.

In step S0308, the extraction result 20 and the estimation result image data 21 are mixed, and the identification unit 10F4 identifies whether it is "genuine" or "fake".

The generation unit 10F3 learns image processing and the like so as to generate the estimation result image data 21 so as to be identified as "genuine" by the identification unit 10F4 as much as possible. On the other hand, the identification unit 10F4 learns based on feedback and the like so as to improve the accuracy with which it can identify a "fake" as a "fake".

Specifically, for the identification result of the identification unit 10F4, the first learning data indicates the “correct answer” as to whether the image data to be identified is “genuine” or “fake”. Data (hereinafter referred to as "correct answer data 22") is prepared. Then, by collating the identification result with the correct answer data 22, it is possible to evaluate whether or not the identification part 10F4 has correctly identified.

When such evaluation and identification results are fed back to the generation unit 10F3, the generation unit 10F3 generates the estimation result image data 21 with the aim of being identified as "genuine" by the identification unit 10F4. It can be learned to generate. That is, the generation unit 10F3 learns so as to generate a "fake" that can be easily identified as a "genuine" by feedback.

Further, when the evaluation is fed back to the identification unit 10F4, the identification unit 10F4 can learn to improve the accuracy with which it can identify a "fake" as a "fake". That is, the identifying unit 10F4 learns by feedback so as to reduce the probability of overlooking the "fake" or misidentifying the "fake" as the "genuine".

Note that the learning data generation device 10 may repeat learning based on the first learning data in step S0306 in advance, and perform learning processing to give the generation unit 10F3 and the identification unit 10F4 a certain degree of accuracy.

Then, the estimation result image data 21 generated with such a quality as to be identified as "genuine" by the identification unit 10F4 is used as second learning data. By generating the learning data 15 in this way, the second learning data used by the AI 16 for learning can be increased.

On the other hand, the estimation result image data 21 identified as "counterfeit" by the identification unit 10F4 is targeted for "reuse". That is, the estimation result image data identified as "fake" is the result of insufficient learning.

Therefore, for example, the estimation result image data identified as "fake" is subjected to an operation of correcting the insufficiency so as to be identified as "genuine". In this way, processing such as feedback to the generation unit 10F3 by image data or the like reflecting the content of manual operation is "reuse". When such "reuse" is performed, the generation unit 10F3 can learn the insufficient point and generate the estimation result image data 21 that is more likely to be identified as "genuine".

Note that "reuse" is not limited to being used for learning by the generation unit 10F3. For example, "reuse" may be added to the learning data 15 by image data reflecting the content of manual operation. However, if "reuse" is difficult, the estimation result image data identified as "fake" may be discarded.

The illustrated GAN generates learning data 15 used for AI 16 learning. The second learning data generated in this way is for AI to estimate thinning locations of crops, and is different from image data visually evaluated by humans.

For example, when a general landscape is photographed, the image data may include minute changes in color that are difficult to determine by human eyes. Such changes are not given much importance in human visual evaluation. On the other hand, in evaluation by a computer, there are cases where it can be grasped by calculating fluctuations in pixel values. In this way, when generating image data, evaluation items to be emphasized may differ depending on whether evaluation by a computer or visual evaluation by a person is conscious.

[Example of shooting method]
The input image data such as the first input image data 11D1 and the second input image data 11D2 are desirably shot as follows, for example.

FIG. 5 is a diagram showing an example of an imaging method. Hereinafter, the vertical direction in the drawings will be referred to as the "Z-axis direction". The Z-axis direction is the so-called direction of gravity. Also, in the drawings, the horizontal direction is mainly defined as the "X-axis direction". The X-axis direction is the right-hand direction when facing the crops. Further, the depth direction is defined as "Y-axis direction".

A case of photographing the first crop 12 will be described below as an example.

The input image data is desirably photographed from a plurality of viewpoints around the Z-axis. That is, it is desirable that the input image data be image data showing the first crop 12 from as many viewpoints as possible.

Specifically, the camera 11 turns the optical axis toward the first crop 12 and rotates around the Z axis (so-called Yaw axis rotation. This is the rotation indicated by "Yaw" in the figure). It is preferable to shoot with

By photographing in this way, the first crop 12 can be photographed from all directions. Note that the input image data may be still images or the like obtained by photographing about three viewpoints out of 360 degrees.

The fruit thinning work may be performed with consideration given to the overall shape of the crops or the sun exposure. Therefore, the thinning object may exist at various angles. Therefore, the camera 11 may not be able to photograph all thinning objects from one viewpoint. Therefore, it is desirable that the input image data be shot from various viewpoints with as few blind spots as possible.

In addition, it is more desirable that the input image data is further photographed from a plurality of viewpoints around the X axis. For example, the camera 11 directs the optical axis to the first crop 12, and has a viewpoint that is the front of the first crop 12, a viewpoint that captures the first crop 12 from below (a so-called looking-up viewpoint), and a viewpoint that captures the first crop 12 from below. Photographing is performed from a viewpoint or the like that is the back of the crop 12 .

In this way, it is desirable that the camera 11 takes an image while rotating around the X axis (so-called Pitch axis rotation, which is indicated by "Pitch" in the figure).

Also, it is desirable that the second input image data 11D2 is similarly captured.

As described above, it is desirable that the input image data is photographed by photographing crops from a plurality of viewpoints by performing Pitch or Yaw rotation. With this type of photography, it is possible to grasp, from the input image data, the fruit thinning work for adjusting the overall shape of the crops, the fruit thinning work for adjusting the sunnyness of the crops, or the like.

Also, the input image data may be captured under different weather conditions, different arrangement of surrounding objects, or the like. In other words, it is desirable that the input image data show the state of being photographed under different ambient environments or different lighting conditions depending on the season, weather, or the like.

[Second embodiment]
The second embodiment differs from the first embodiment in that the overall processing is as follows.

FIG. 6 is a diagram illustrating an example of overall processing of the second embodiment. In the following, differences from the first embodiment will be mainly described, and redundant description will be omitted. The overall processing in the second embodiment differs from the overall processing in the first embodiment in that step S0601 is performed.

In step S0601, the learning data generation device 10 extracts a thinning object. Specifically, the learning data generation device 10 extracts the thinning object by performing the following extraction process.

FIG. 7 is a diagram illustrating an example of extraction processing. For example, step S0601 performs the following processing.

In step S0701, the learning data generation device 10 generates first mask image data.

The first mask image data is mask image data that becomes learning data in learning for instance segmentation performed in step S0702 below. That is, the first mask image data is image data that serves as a "sample".

It should be noted that the first mask image data may be data specifying a masked area, such as filling out part or all of the image data.

Hereinafter, the mask image data generated by using the first mask image data as learning data for instance segmentation and generating the instance segmentation will be referred to as "second mask image data". Details of the mask image data will be described later.

In step S0702, the learning data generation device 10 learns instance segmentation.

In step S0703, the learning data generation device 10 evaluates instance segmentation.

In step S0704, the learning data generation device 10 generates second mask image data for instance segmentation.

For example, instance segmentation and generation of mask image data are the following processes.

FIG. 8 is a diagram illustrating an example of instance segmentation processing and an example of mask image data. The first input image data 11D1 shown in FIG. 8A will be described below as an example.

For example, assume that four target objects, a first object 31, a second object 32, a third object 33, and a fourth object 34, are photographed in the first input image data 11D1.

FIG. 8B is a diagram showing an example of the execution result of instance segmentation and mask image data 40 generated by instance segmentation.

Instance segmentation is, for example, a process of generating the mask image data 40 shown in FIG. 8B by executing the process on the first input image data 11D1 shown in FIG. 8A.

Specifically, the instance segmentation is a process of detecting objects in the first input image data 11D1 and distinguishing a plurality of detected objects from separate objects.

In the example shown in FIG. 8(B), the area showing the first object 31, the second object 32, the third object 33, and the fourth object 34 (hereinafter, the area showing the target object in the image data is referred to as the "first area ), and regions other than the first object 31, the second object 32, the third object 33, and the fourth object 34 (hereinafter referred to as the “second region”. For example, the second region is the background or the like. ) are distinguished by two colors.

Specifically, as shown in FIG. 8B, in the mask image data 40, the first area is the area shown in white. On the other hand, in the mask image data 40, the second area is the area shown in black. Thus, the mask image data 40 is, for example, image data in which the first region and the second region are binarized and shown in different colors.

Note that the mask image data 40 is not limited to the format shown in FIG. 8B. For example, the colors of the first area and the second area can be set in advance, and other color combinations may be used. Also, the mask image data 40 is not limited to a format in which areas are distinguished by color, and may be in a format in which areas are distinguished by the presence or absence of hatching, identification data, or the like.

By applying the mask image data 40 to the first input image data 11D1, the learning data generation device 10 can generate image data in which the first region is extracted. That is, referring to the mask image data 40, the learning data generation device 10 can recognize the target object in the first input image data 11D1 and generate image data in which the target object is extracted.

By using the mask image data 40, the background or the like indicated by the first input image data 11D1 can be deleted. That is, if data other than the target object, such as the background, can be excluded in learning, it is possible to prevent AI from learning unimportant objects or the background in fruit thinning work in vain.

Thus, it is desirable that the mask image data 40 be image data capable of masking areas other than the first area, such as setting the background or the like as the second area.

Further, the mask image data 40 can identify individual target objects even if the target objects are of the same type. That is, when the mask image data 40 is applied, as shown in FIG. Different objects such as the target object 42 , the third target object 43 and the fourth target object 44 can be identified.

For example, in semantic segmentation processing, the first target object 41, the second target object 42, the third target object 43, and the fourth target object 44 are classified into the same object or category, and distinguished. often not.

On the other hand, in the case of instance segmentation processing, a plurality of pixels representing one target object are collectively identified as one object, and objects of the same type but different are regarded as different objects. Identifiable.

That is, when the instance segmentation process is performed, so-called labeling becomes possible when there are multiple target objects of the same type in the image data. For example, in the example shown in FIG. 8B, the first target object 41, the second target object 42, the third target object 43, and the fourth target object 44 can be managed with different identification numbers or the like.

Therefore, the learning in step S0702 is performed to the extent that the target object can be accurately identified. And the evaluation in step S0703 evaluates the accuracy etc. which extract a target object. When such steps S0702 and S0703 are performed, in step S0704, the learning data generation device 10 can generate second mask image data for instance segmentation.

Steps S0701 to S0703 are repeatedly executed depending on the evaluation result of the instance segmentation. That is, the "learning process" and the process shown in FIG. 7 are repeatedly executed until a certain degree of accuracy is ensured. 7 may be performed.

In addition, it is desirable that the learning data generation device 10 further performs illustration as in step S0705. For example, illustration processing is as follows.

FIG. 9 is a diagram showing an example of illustration processing. A case of inputting the first input image data 11D1 in the photograph format as shown in FIG. 9A will be described below as an example.

The example shown in FIG. 9A shows an example in which the target object exists in the central portion of the image data (the portion where the fruit is photographed in the drawing; hereinafter referred to as "target object region 51"). For example, the target object appearing in the target object region 51 is identified by object recognition such as instance segmentation.

The illustration process is, for example, a process of inputting the first input image data 11D1 and generating image data as shown in FIG. 9B (hereinafter referred to as "illustration image data 50").

FIG. 9B is a diagram showing an example of the illustrated image data 50. As shown in FIG.

The illustrated image data 50 fills in the area of the target object with a predetermined color. For example, as shown in FIG. 9B, the illustrated image data 50 is image data in which the area of the target object indicated by hatching is painted out.

Hereinafter, in the example shown in FIG. 9(B), the area identified as the area of the target object and painted in the illustration process is referred to as "filled area 52".

Further, in the illustrated image data 50, the area other than the painted area 52 (the area indicating the background, etc.) is made white (filled with a different color from the painted area 52, etc.).

In this way, the illustration process is a process of classifying the area of the target object and the other areas with a predetermined color. By performing the illustration processing in this manner, the RGB values or luminance values in the image data can be simplified.

Photographic image data such as the first input image data 11D1 often includes subtle changes in RGB values or luminance values that are difficult for the human eye to perceive.

For example, a tomato fruit is simply one color, red. In the case of showing such a target object, if the image data is in the form of a photograph, the pixel values such as the RGB values of the pixels showing red in the same target object may change minutely. In many cases, such fine changes in RGB values should not be learned.

Therefore, in the illustration processing, processing such as displaying the target objects in the same color is performed. Specifically, when instance segmentation or the like is performed on the first input image data 11D1, pixels that can be identified as a target object are grouped.

The illustration process is a process of filling in the same grouped pixels with the same color. Furthermore, the illustration process is a process of filling in a region such as a background with a color different from that of the region of the target object.

Note that the illustration process may be any process other than painting with a predetermined color as long as it simplifies the image data. For example, the illustration processing may be such that the background or the like is rendered in a single color. Further, the illustration process may be a process using hatching or the like instead of filling with color.

In this way, if the image data is illustrated, extraction results and the like can be expressed in a simplified manner. As for the extraction result, it is often sufficient if the position, shape, etc. of the target object can be roughly expressed. That is, extraction results often do not require data such as fine color changes and backgrounds.

Therefore, simply showing the target object in a single color eliminates factors that hinder learning and allows for more accurate learning than in a photographic format or the like. In other words, if the AI learns the fruit thinning work using the illustrated image data as learning data, the AI can accurately learn the feature quantities that are important for the fruit thinning work.

In addition, illustrated image data is easier to express colors than photographic image data, so that the amount of data can be reduced.

FIG. 10 is a diagram showing modified examples of illustrated image data or mask image data. For example, mask image data may be generated as shown in FIG. 10(B) or FIG. 10(C). The first input image data 11D1 shown in FIG. 10A will be described below as an example.

FIG. 10A is a diagram showing an example of the first input image data 11D1 with four apples as target objects. An example in which the learning data generation device 10 inputs such first input image data 11D1 and performs instance segmentation and the like will be described below.

For example, when performing the instance segmentation shown in FIG. 8, the second mask image data is generated as shown in FIG. 10(B).

On the other hand, the second mask image data may be generated as shown in FIG. 10(C).

FIG. 10B is a diagram showing an example of a format in which four target objects are collectively shown as one piece of image data. In this way, the second mask image data may represent a plurality of target objects with one image data.

FIG. 10C is a diagram showing an example of the format of an image data group, in which four target objects are divided into four image data. In this way, the second mask image data may be in the form of an image data group in which image data is divided for each target object and a plurality of image data groups are used as one second mask image data.

As described above, the mask image data or the image data generated by illustration may be a single image data group by grouping a plurality of target objects, or may be separately divided for each target object to form an image data group. good too.

[Extraction result example]
FIG. 11 is a diagram showing an example of target object recognition. The first input image data 11D1 shown in FIG. 11A will be described below as an example.

When the target object shown in FIG. 11A is handled, the learning data generation device 10 learns in advance the shape, color, or combination thereof of the target object. By performing such learning, for example, the learning data generation device 10 can recognize a target object as shown in FIG. 11(B) or FIG. 11(C).

11(B) and 11(C) are examples in which the position, range, etc., where the target object is recognized are shown enclosed by dashed lines. Note that the recognition result may be output in formats other than those shown in FIGS. 11B and 11C.

FIG. 11B is a diagram showing a first example of the result of recognizing the target object. For example, as shown in FIG. 11B, the target objects are a first target object 101, a second target object 102, a third target object 103, a fourth target object 104, a fifth target object 105, and a sixth target object. 106 and the seventh target object 107 are recognized by the learning data generation device 10 .

Also, the target object may be recognized in a format as shown in FIG. 11(C), for example.

FIG. 11C is a diagram showing a second example of the result of recognizing the target object. A second example is an example of a format in which recognition results are divided into separate image data for each target object. Specifically, the learning data generation device 10 includes a first target object 101, a second target object 102, a third target object 103, a fourth target object 104, a fifth target object 105, a sixth target object 106, and The recognition result of the seventh target object 107 is divided for each target object and output.

Note that the recognition result of the target object does not have to be in the form of image data as shown in FIG. 11(B) or FIG. 11(C). That is, the recognition result of the target object is an intermediate product, and parameters such as the position, size, range, number, or coordinates of the target object in the image data (statistical values or representative values may be used). ) can be used as long as the learning data generation device 10 can grasp the format.

Therefore, the learning data generation device 10 does not need to internally store parameters indicating recognition results and output image data and the like as illustrated.

In step S0306, the learning data generation device 10 makes the learning model learn using the learning data. For example, the learning data is image data generated in step S0601, that is, illustrated image data. Note that the learning data may be image data in a plurality of formats. Details of the learning data will be described later.

[Example of processing result of overall processing]
FIG. 12 is a diagram illustrating a processing result example of the overall processing. Hereinafter, a case where FIGS. 12(A) and 12(B) are before and after thinning will be described as an example.

FIG. 12A is a diagram showing an example of the first input image data 11D1.

FIG. 12B is a diagram showing an example of the second input image data 11D2.

FIG. 12C is a diagram showing an example of the second learning data.

Hereinafter, in the first input image data 11D1, that is, before the fruit thinning work, as shown in FIG. 104 , a fifth target object 105 , a sixth target object 106 and a seventh target object 107 .

On the other hand, in the second input image data 11D2, that is, after the fruit thinning work is performed, the second target object 102, the third target object 103, the fourth target object 104, In addition, it is assumed that the four target objects of the sixth target object 106 are the thinning target objects, and the thinning target objects are thinned.

In this way, by comparing the first input image data 11D1 and the second input image data 11D2, the thinning object can be extracted. By learning such extraction results, the learning data generation device 10 can estimate the fruit thinning work and generate estimation result image data when unknown first input image data 11D1 is input.

The estimation result image data and the like generated in this way become the learning data 15 . Then, the AI 16 uses the learning data 15 and the like as second learning data, and learns the fruit thinning work.

FIG. 12C is a diagram showing an example of a format in which the target object is surrounded by dotted lines. Also, FIG. 12C is a diagram showing an example of a format in which the thinning object is indicated by hatching.

Note that the second learning data is not limited to the format shown in FIG. 12(C). That is, as for the second learning data, it is sufficient that the AI 16 can learn the position, number, arrangement, shape, range, or the like of the thinning object. Therefore, the second learning data may specify the thinning target object and the target object in other formats.

[Third Embodiment]
FIG. 13 is a diagram illustrating a configuration example of a learning device. Compared with the first embodiment and the like, the configuration of the learning data generation device 10 and the like in the third embodiment is, for example, the same as in the first embodiment. On the other hand, the learning device 301 is an information processing device or the like. Note that the learning data generation device 10 and the learning device 301 may be the same information processing device or the like.

In the third embodiment, a learned model 303 is generated by learning a learning model 302 using the learning data 15 or the like generated by the configuration in the first embodiment or the second embodiment.

Hereinafter, AI during learning or before learning is simply referred to as "learning model 302". On the other hand, the AI after learning with the second learning data to some extent is referred to as a "learned model 303".

The learning device 301 inputs the learning data 15 . Then, the learning device 301 causes the learning model 302 to learn using the learning data 15 .

Note that data other than the learning data 15 may be used for learning. For example, the learning device 301 may also input the first input image data 11D1, the second input image data 11D2, etc., and allow the learning model 302 to learn. Alternatively, the learning device 301 may input the second learning data in the form of an extraction result or the like.

As described above, the learning device 301 trains the learning model 302 to generate the trained model 303 . If such a learned model 303 can be generated, an AI for estimating a thinning object can be realized.

Specifically, the learning device 301 has, for example, the following configuration.

FIG. 14 is a diagram showing an example of a configuration in which learning is performed by a learning device. As illustrated, the learning model 302 is configured to include at least a generation unit 10F3 and an identification unit 10F4.

The generation unit 10F3 and the identification unit 10F4 are generators and classifiers in the adversarial generation network.

First, the learning device 301 inputs the first input image data 11D1.

Next, the generation unit 10F3 estimates a target object to be a thinning target among the target objects indicated by the first input image data 11D1. Then, the generation unit 10F3 generates the estimation result image data 21. FIG. Hereinafter, similarly to FIG. 12(C), an example of a format in which target objects are indicated by dotted lines and thinning target objects are indicated by hatching among the target objects will be described.

Next, when the estimation result image data 21 is generated, the identification unit 10F4 identifies the estimation result image data 21 . Then, the identification unit 10F4 identifies the estimation result image data 21 with the learning data 15 as “correct”.

Specifically, first, the estimation result image data 21 indicates the positions, numbers, and the like of thinning objects. On the other hand, similarly to the estimation result image data 21, the learning data 15 also indicates the positions, numbers, and the like of thinning objects.

In the example described below, the identification unit 10F4 refers to the estimation result image data 21, and if both the position and the number of the thinning target objects match the learning data 15, the identification unit 10F4 identifies them as "correct".

On the other hand, if at least one of the positions and the number of thinning objects indicated by the estimation result image data 21 is different from the learning data 15, the identification unit 10F4 identifies it as an “wrong answer”.

Then, the identification unit 10F4 feeds back at least the generation unit 10F3 with the identification result of "correct answer" or "wrong answer". In this way, the feedback is a process of transmitting the identification result from the identification unit 10F4 to at least the generation unit 10F3.

For learning of the generation unit 10F3, the feedback may convey the process of identification by the identification unit 10F4, criteria for identification, intermediate data generated during identification, or the like. In other words, the feedback may include the process up to the output of the identification result and the data generated during the process, together with the identification result. Then, the generation unit 10F3 learns with reference to the feedback identification result. In addition, when data is transmitted in another set, the generation unit 10F3 may learn by referring to the data in the set as well.

Specifically, in the example shown in FIG. 14 , the estimation result image data 21 and the learning data 15 are selected from seven target objects and shown as thinning objects. Comparing the estimation result based on the estimation result image data 21 and the "correct answer" based on the learning data 15, in this example, the target object located in the center (the target object indicated by the difference 151 in the figure) is thinned. It is different whether it becomes an object or not.

Therefore, the comparison result of the estimation result image data 21 and the learning data 15 is identified as having a difference because the number of thinning objects and the judgment result of the difference 151 are different. Therefore, based on the comparison result, the identification unit 10F4 identifies "wrong answer" because both the number and position of the thinning objects are different from the reference learning data 15. FIG.

Note that the identification by the identification unit 10F4 may have an allowable range with respect to the reference. For example, the number may be set such that two or less is allowed with respect to the reference. In the setting of such an allowable range, if there is only the difference of the difference 151, the identification unit 10F4 identifies it as "correct". Also, in learning, there may be items that can be set.

Then, for example, an expert views and evaluates a plurality of pieces of estimation result image data 21 generated by the generation unit 10F3. Specifically, if the generation unit 10F3 generates 100 images of the estimation result image data 21, and the expert looks at the estimation result image data 21 and determines that all 100 images are satisfactory, the generation unit 10F3 and the like cannot learn. Evaluated as complete.

By repeating generation and identification feedback as described above, the learning device 301 can improve the generation accuracy of the estimation result image data 21 .

Note that the learning device 301 preferably extracts thinning objects during generation or identification. Specifically, the learning device 301 performs processing such as generation of mask image data and illustration during generation or identification.

In this way, if the image data is masked, illustrated, or both are processed for extraction, extraction results and the like can be expressed in a simplified manner. In many cases, it is sufficient that the extraction result can roughly express the position, shape, and the like of the target object. That is, in many cases, extraction results do not require data such as fine color changes, subjects with little relation to the fruit thinning work, backgrounds, and the like.

In particular, in environments with crops, it is often difficult to adjust the surrounding environment for AI learning and photography. In addition, in many cases, an environment with crops is an environment in which it is easy for subjects with little relation to enter unexpectedly. Therefore, if such a disturbance can be reduced by masking the image data, the AI can accurately learn the feature quantity that is important for grasping the content of the fruit thinning work.

In addition, it is better to illustrate the target object and show it in a single color in a simplified manner, or to use image data that focuses on the important part. and can learn with high accuracy. That is, if the image data is subjected to extraction processing as preprocessing and the AI learns the fruit thinning work, the AI can accurately learn the feature quantities that are important for grasping the details of the fruit thinning work.

Note that the identification unit 10F4 may improve the accuracy of identification by learning using the estimation result image data 21, the identification result, the learning data 15, and the like.

The learning data 15 may be data generated by the learning data generation device 10, data generated by manipulating the first input image data 11D1, or a combination thereof.

Furthermore, the formats of the estimation result image data 21 and the learning data 15 are not limited to the illustrated formats. That is, the format of the estimation result image data 21 and the learning data 15 should be able to specify the details of the fruit thinning work. For example, the format of the estimation result image data 21 and the learning data 15 may be a format using numerical values (indicating coordinates or quantities in the image) for contents such as the positions and numbers of thinning objects.

Note that the criteria for identification are not limited to the position and number of thinning objects, and other criteria may be used. What is used as a reference for identification may also be an object of learning. Moreover, what is used as a reference for identification may be set by a person.

[Example of functional configuration]
FIG. 15 is a diagram illustrating a functional configuration example of a learning device; For example, the learning device 301 has a functional configuration including an image data input unit 10F1, a learning data input unit 301F1, a generation unit 10F3, an identification unit 10F4, and the like. The learning device 301 preferably has a functional configuration further including an extraction unit 10F2, a mask image data generation unit 10F5, and an illustration processing unit 10F6. The illustrated functional configuration will be described below as an example.

The image data input unit 10F1 performs an image data input procedure for inputting the first input image data 11D1. For example, the image data input unit 10F1 is implemented by the camera 11, the interface 10H3, and the like.

The generation unit 10F3 performs a generation procedure for generating the estimation result image data 21. FIG. For example, the generation unit 10F3 is realized by the CPU 10H1 or the like.

The identification unit 10F4 compares the estimation result image data 21 with the learning data 15 and feeds back the identification result to the generation unit 10F3 to perform the identification procedure for learning the learning model 302 . For example, the identification unit 10F4 is implemented by the CPU 10H1 or the like.

Either or both of the estimation result image data 21 and the learning data 15 are illustrated by generating mask image data by an extraction unit 10F2, a mask image data generation unit 10F5, and an illustration processing unit 10F6. It is desirable to have an extraction process that does or does both.

In this way, when the thinning target is extracted, the learning model 302 can learn important feature amounts such as the thinning target with high accuracy, compared to the case where the image data of the crops is simply used as it is. . That is, the learning device 301 can make the learning model 302 learn and generate the learned model 303 that can accurately estimate the fruit thinning work.

[Fourth embodiment]
FIG. 16 is a diagram illustrating a configuration example of a thinning target object estimation device. Hereinafter, an example of image data representing an unknown crop before thinning is referred to as "unknown image data 401".

The fourth embodiment is an embodiment for executing the trained model 303 generated by learning according to the third embodiment. Hereinafter, the thinning target object estimation device using the learned model 303 is referred to as a "thinning target object estimation device 402".

The thinning target object estimation device 402 is, for example, an information processing device such as a smart phone. In addition, the learned model 303 may be configured to be used by another server device or the like, and the thinning target object estimation device 402 may communicate with the server device to acquire and output the estimated result of the learned model 303. .

Specifically, the trained model 303 is distributed via a network or the like. Note that the learned model 303 may be in a format incorporated in application software or the like. When the learned model 303 distributed in this way is installed in the thinning target object estimation device 402, the thinning target object estimation device 402 is put into a state in which it is possible to perform estimation as shown in the figure, output of the estimation result, and the like.

The unknown image data 401 is image data captured by the thinning object estimation device 402 . Also, the crop indicated by the unknown image data 401 is in a state before the fruit thinning work is performed. In this way, the crops indicated by the unknown image data 401 are "unknown" crops different from the crops that were the target of learning in the first embodiment or the second embodiment.

In addition, the thinning target object estimation device 402 preferably extracts the thinning target object in the estimation. Specifically, the thinning object estimation device 402 desirably performs processes such as generation of mask image data and illustration during estimation. When such a thinning target object is extracted, the thinning target object estimation device 402 can perform estimation with high accuracy.

A thinning object estimation device 402 identifies a target object based on the unknown image data 401 . Then, the thinning target object estimation device 402 estimates the thinning target object using the learned model 303 . For example, the estimation result is output in the form of Augmented Reality (AR). Specifically, the thinning object estimation device 402 displays the output screen 403 to the user 404 .

The output screen 403 is a screen that displays an “x” superimposed on the unknown image data 401 to inform the user 404 of the object to be thinned. Note that the output may be in another display format or in a format using voice.

In addition, it is desirable that the thinning object estimation device 402 has a configuration for accepting items, for example, through an operation screen for "optimization item setting" (hereinafter simply referred to as "setting screen 405").

Fruit thinning work may be performed according to so-called preference. Therefore, the setting screen 405 is an interface for setting preferences.

The setting screen 405 includes items such as "sweetness", "sourness", "size (whole)", "size (grain)", "color", "uniformity", and "shape to fit in a case". This is an example of setting The items and setting format are determined in advance.

“Sweetness” and “sourness” are items for adjusting the taste of crops at the time of harvest.

“Size (whole)” is an item for adjusting the overall size of crops at the time of harvest. For example, "size (whole)" is used to adjust the overall balance of a plurality of fruits in the case of crops having a plurality of fruits.

“Size (fruit)” is an item for adjusting the size of one fruit of crops at the time of harvest. For example, "size (fruit)" is used to adjust the size of each fruit in the case of crops having a plurality of seeds.

“Color” is an item for adjusting the color of crops at the time of harvest.

“Uniformity” is an item for adjusting whether or not the fruit size of crops at the time of harvesting is uniform.

“Make a shape to fit in a case” is an item for adjusting whether or not the size should fit within a predetermined shape used for shipping. Thus, items may be entered in the form of checkboxes.

In addition, "shape to fit in the case" can specify the size of the case with a numerical value, such as "to fit in a case of length (mm) x width (mm) x height (mm)". format, etc. may be used.

These items are items that can be adjusted in the fruit thinning operation. Also, what kind of fruit thinning work affects which item is input to learning data in learning (that is, in the third embodiment). For example, the learning model learns for each purpose of fruit thinning work, such as fruit thinning work that makes the crops sweeter or fruit thinning work that makes the crops larger. Therefore, the trained model can identify the thinning operation that optimizes the item. Also, the degree (for example, sweetness, size, etc.) is input, for example, as a numerical value.

Note that the reception unit that receives items is not limited to the setting screen 405 . That is, the items that can be set may include items other than those shown in the figure. Also, the reception unit may be an interface other than a taskbar or a check box. For example, the reception unit may be an interface for inputting using a text box or the like. Furthermore, the items to be optimized may be fixed.

FIG. 17 is a diagram illustrating an example of a configuration for performing estimation by the thinning target object estimation device. For example, after learning according to the third embodiment, the trained model 302 has a configuration in which the identification unit 10F4 is removed from the generation unit 10F3 and the identification unit 10F4 that constitute the adversarial generation network used in the third embodiment. be.

That is, when the unknown image data 401 is input, the thinning target object estimation device 402 estimates a thinning work suitable for the target object indicated by the unknown image data 401 . Then, the thinning target object estimation device 402 outputs the estimation result image data 21 indicating the estimation result.

Note that the function of the identification unit 10F4 may be stopped. That is, even if the trained model 302 has the identification unit 10F4 like the learning model 302, the identification unit 10F4 may be stopped. On the other hand, the trained model 302 may be configured without the identification unit 10F4 or without the identification unit 10F4 at all.

[Example of functional configuration]
FIG. 18 is a diagram illustrating a functional configuration example of a thinning target object estimation device. For example, the thinning object estimation device 402 has a functional configuration including an image data input unit 10F1, an estimation unit 402F1, an output unit 402F2, and the like. It is desirable that the thinning object estimation device 402 has a functional configuration further including an extraction unit 10F2, a mask image data generation unit 10F5, and an illustration processing unit 10F6. The illustrated functional configuration will be described below as an example.

The image data input unit 10F1 performs an image data input procedure for inputting the first input image data 11D1. For example, the image data input unit 10F1 is implemented by the camera 11, the interface 10H3, and the like.

The estimating unit 402F1 performs an estimating procedure for estimating the thinning object using the learned model 303 . For example, the estimation unit 402F1 is realized by the CPU 10H1 or the like.

For example, the estimation unit 402F1 is composed of the generation unit 10F3 and the like.

The output unit 402F2 performs an output procedure for outputting the estimation result. For example, the output unit 402F2 is realized by the output device 10H5 or the like.

The unknown image data 401 is subjected to extraction processing of generating mask image data, illustration, or both processing by the extraction unit 10F2, the mask image data generation unit 10F5, and the illustration processing unit 10F6. is desirable.

Also in estimation, the thinning target object estimation device 402 can accurately estimate the thinning target object and the like by paying attention to the learned elements as much as possible.

In this way, when the thinning target object is extracted from the unknown image data 401, the thinning target object estimating device 402 can accurately identify the thinning target object, etc., compared to the case where the image data obtained by simply photographing the crops is used as it is. can be estimated well.

[Example of functional configuration of learning system]
FIG. 19 is a diagram illustrating an example of a functional configuration; For example, the learning data generation device 10 has a functional configuration including an image data input unit 10F1, an extraction unit 10F2, a generation unit 10F3, an identification unit 10F4, and the like. Moreover, as shown in the drawing, the learning data generation device 10 preferably has a functional configuration further including a mask image data generation unit 10F5, an illustration processing unit 10F6, and the like.

The image data input unit 10F1 performs an image data input procedure for inputting first input image data 11D1 and second input image data 11D2. For example, the image data input unit 10F1 is implemented by the camera 11, the interface 10H3, and the like.

The extraction unit 10F2 performs an extraction procedure for extracting target objects that are different between the first input image data 11D1 and the second input image data 11D2 from among the target objects as thinning targets. For example, the extraction unit 10F2 is realized by the CPU 10H1 or the like.

The generation unit 10F3 learns the image data representing the extraction result as the first learning data, and performs a generation procedure of generating estimation result image data. For example, the generation unit 10F3 is realized by the CPU 10H1 or the like.

The identification unit 10F4 identifies the estimation result image data and performs an identification procedure of generating second learning data based on the identification result. For example, the identification unit 10F4 is implemented by the CPU 10H1 or the like.

The mask image data generation unit 10F5 performs a mask image data generation procedure for generating mask image data that distinguishes between a target object and non-target objects. For example, the mask image data generation unit 10F5 is realized by the CPU 10H1 or the like.

The illustration processing unit 10F6 performs an illustration procedure for illustrating a target object and objects other than the target object. For example, the illustration processing unit 10F6 is realized by the CPU 10H1 or the like.

As described above, the learning data generation device 10 generates the second learning data such as the learning data 15 and the like. In this way, when the second learning data can be generated, the workload of preparing learning data for AI for estimating thinning locations of crops can be reduced compared to the case of manually generating learning data. For example, it is necessary to prepare at least several thousand pieces of image data as learning data for AI for estimating thinning locations of crops. Such preparation often requires a preparation period of at least one to several years.

In particular, crops are often photographed under so-called natural light, such as outdoors. Under such a lighting environment, there are many cases where the lighting environment is not stable, such as in a factory. Specifically, sunlight and the like are difficult to adjust artificially. Therefore, in natural light, various conditions such as light intensity, direction, presence or absence of shadows, etc. fluctuate compared to lighting in a factory or the like. Therefore, when photographing agricultural products, the lighting environment is often harsher than indoors such as in a factory. When AI is used under such conditions with many disturbances, it is particularly desirable to have a large amount of learning data.

Note that the preparation period varies depending on the cycle of the target agricultural products.

Furthermore, if an attempt is made to sufficiently improve the AI estimation accuracy, it is desirable to prepare a larger amount of learning data. For example, based on Uncle Bernie's rule, it is desirable to prepare learning data ten times or more the number of parameters in the neural network for AI learning. Therefore, it may be desirable to prepare tens of thousands to hundreds of thousands of image data as AI learning data for estimating thinning locations of crops.

As the amount of learning data to be prepared increases, the preparation period becomes longer and the workload tends to increase when the learning data is generated by photographing the actual crops. As described above, when the work load increases, it causes an increase in development costs and a prolonged development period.

On the other hand, if learning data can be generated as in this embodiment, a large amount of learning data can be prepared with a small workload. Therefore, the workload of preparing learning data can be reduced.

The learning device 301 has, for example, a functional configuration including a learning data input unit 301F1, a learning unit 301F2, and the like.

The learning data input unit 301F1 performs a learning data input procedure for inputting second learning data. For example, the learning data input unit 301F1 is realized by the interface 10H3 or the like.

The learning unit 301F2 performs a learning procedure for learning the learning model 302 using the second learning data. For example, the learning unit 301F2 is realized by the CPU 10H1 or the like.

As described above, the learning device 301 makes the learning model 302 learn using the second learning data generated by the learning data generation device 10 and the like. Through such learning, the learning device 301 can generate a learned model 303 for estimating the thinning object. For example, the learned model 303 is used by the thinning object estimation device 402 as follows.

The thinning object estimation device 402 has a functional configuration including an image data input unit 10F1, an estimation unit 402F1, an output unit 402F2, and the like.

The image data input unit 10F1 performs an image data input procedure for inputting unknown image data 401. FIG. For example, the image data input unit 10F1 is implemented by the camera 11, the interface 10H3, and the like.

The estimating unit 402F1 performs an estimating procedure for estimating the thinning object using the learned model 303 . For example, the estimation unit 402F1 is realized by the CPU 10H1 or the like.

The output unit 402F2 performs an output procedure for outputting the estimation result. For example, the output unit 402F2 is realized by the output device 10H5 or the like.

As described above, when the learned model 303 is installed, the thinning target object estimation device 402 estimates the content of the thinning work using the learned model 303, information.) can be estimated. When such an estimation result is output, the user 404 can refer to the estimation result and perform an appropriate thinning operation even if the user is a beginner. That is, even if the user 404 is a beginner or the like, he/she can grasp the fruit left in the fruit thinning work and the fruit to be thinned by referring to the estimation result.

The learning system 500 includes, for example, any one of the functional configurations of the learning data generation device 10, the learning device 301, and the thinning target object estimation device 402.

Specifically, the learning system 500 includes a plurality of information processing devices such as the learning data generation device 10 and the learning device 301 . With such a learning system 500 , it is possible to generate learning data and to train the learning model 302 to generate the trained model 303 .

Note that the learning system 500 is not limited to a plurality of information processing devices, and may be a single information processing device.

Also, the learning system 500 may be a combination of the learning device 301 and the thinning object estimation device 402 .

[Function configuration example of estimation system]
FIG. 20 is a diagram illustrating a functional configuration example of an estimation system; For example, the estimation system 501 includes the learning data generation device 10, the learning device 301, the thinning object estimation device 402, and the like. However, the estimation system 501 does not need the learning data generation device 10 . That is, the estimation system 501 may use photographed image data as the learning data 15, use image data generated by the learning data generation device 10, or use both.

In addition, the learning model 302 and the learned model 303 (including a program that uses the learned model 303) may be duplicated so that there may be a plurality of learning devices 301, thinning object estimation devices 402, and the like. .

The learning device 301 has, for example, an image data input unit 10F1, a learning data input unit 301F1, a learning unit 301F2, an extraction unit 10F2, a mask image data generation unit 10F5, an illustration processing unit 10F6, and the like.

The image data input unit 10F1 performs an image data input procedure for inputting the first input image data 11D1. For example, the image data input unit 10F1 is implemented by the camera 11, the interface 10H3, and the like.

The learning data input unit 301 F 1 performs a learning data input procedure for inputting the learning data 15 . For example, the learning data input unit 301F1 is realized by the interface 10H3 or the like.

The learning unit 301F2 performs a learning procedure for learning the learning model 302 based on the learning data 15 . For example, the learning unit 301F2 is realized by the CPU 10H1 or the like.

The extraction unit 10F2 performs an extraction procedure for extracting a target object or a thinning target object from the first input image data 11D1 and the learning data 15 . For example, the extraction unit 10F2 is realized by the CPU 10H1 or the like.

Either or both of the first input image data 11D1 and the learning data 15 are extracted by an extraction unit 10F2, a mask image data generation unit 10F5, and an illustration processing unit 10F6 to generate mask image data. It is desirable to have an extraction process that converts or does both.

In this way, when the target object or the thinning target object is extracted, the learning model 302 can extract the important features of the thinning target object, etc. compared to the case where the image data of the crops is simply used as it is. Able to learn quantity accurately. That is, the learning device 301 can make the learning model 302 learn and generate the learned model 303 that can accurately estimate the fruit thinning work.

As described above, the estimation system 501 causes the learning unit 301F2 to learn the learning model 302 and generates the trained model 303. FIG. Thus, the generated learned model 303 is sent to the thinning object estimation device 402 via a network or the like.

The thinning object estimation device 402 has a functional configuration including an image data input unit 10F1, an extraction unit 10F2, a mask image data generation unit 10F5, an illustration processing unit 10F6, an estimation unit 402F1, an output unit 402F2, and the like.

The image data input unit 10F1 performs an image data input procedure for inputting unknown image data 401. FIG. For example, the image data input unit 10F1 is implemented by the camera 11, the interface 10H3, and the like.

The extraction unit 10F2 performs an extraction procedure for extracting a target object or a thinning target object from the unknown image data 401 . For example, the extraction unit 10F2 is realized by the CPU 10H1 or the like.

The estimating unit 402F1 performs an estimating procedure for estimating the thinning object using the learned model 303 . For example, the estimation unit 402F1 is realized by the CPU 10H1 or the like.

The output unit 402F2 performs an output procedure for outputting the estimation result. For example, the output unit 402F2 is realized by the output device 10H5 or the like.

As described above, in the estimation system 501 , the learning device 301 first learns the learning model 302 to generate the trained model 303 . Next, in the estimation system 501 , the learned model 303 generated in this way is distributed to the thinning target object estimation device 402 .

When the learned model 303 is installed, the thinning target object estimating device 402 estimates the content of the thinning work using the learned model 303, and identifies the thinning target object (including information such as position, number, or candidate). can be estimated. When such an estimation result is output, the user 404 can refer to the estimation result and perform an appropriate thinning operation even if the user is a beginner.

That is, even if the user 404 is a beginner or the like, he/she can grasp the fruit left in the fruit thinning work and the fruit to be thinned by referring to the estimation result. Further, for example, when the learning device 301 uses a cloud environment or the like, it is possible to quickly collect data, distribute the learned model 303, and the like.

[Regarding the format of learning data]
Learning data such as the first learning data and the second learning data are desirably image data in the form of extracted crops. However, extraction may be performed in multiple steps. In such a case, the learning device 301 may include, in the learning data, image data or the like in an intermediate stage of extraction.

For example, it is assumed that the extraction process is divided into three stages, ie, the first stage to the third stage.

The first stage is image data in the input state, that is, image data in a photograph format (however, white balance and the like may be adjusted).

The second stage is image data in a format in which portions other than crops are used as a background and the background is masked. For example, the background is masked to white (which color is set by the mask).

The third stage is image data in the form of illustrations of crops and the like.

The learning data may be image data at any stage among the first to third stages. Further, the learning data may be not only image data at any stage among the first to third stages, but also image data at a plurality of stages, that is, both before and after extraction processing.

If the image data is in a format in which crops are extracted by masking or the like, the learning device 301 can accurately learn thinning objects into the learning model.

On the other hand, it may be desirable for the training data to include image data in the form of photographs or the like. For example, if illustrated, the image data may be caused by scratches, etc. (for example, poor sunlight, salt damage, corrosion, disease, trauma, worm-eaten, etc.) occurring on the target object, or discoloration. ) may be omitted. On the other hand, in the fruit-picking operation, there are cases where target objects with scratches or the like are preferentially picked. It is desirable that the AI for such fruit thinning work is learned in the first stage or second stage format, that is, in the format of image data that displays scars and the like. Therefore, the format of the learning data may be selected according to the preference of the fruit thinning work.

In this way, when the learning data is image data of multiple stages, the learning device 301 can cause the learning model to learn the fruit thinning work that more closely matches the user's preference.

[About AI]
AI processes image data etc., for example, with the following network structure.

FIG. 21 is a diagram illustrating an example network structure. For example, an AI may have a network structure with an input layer L1, a hidden layer L2, and an output layer L3.

Specifically, AI is a network structure having a Convolution Neural Network (Convolution Neural Network, CNN) or the like as shown.

The input layer L1 is a layer for inputting input data DIN.

The hidden layer L2 is a layer that performs processing such as convolution, pooling, normalization, or a combination thereof on the input data DIN input from the input layer L1.

The output layer L3 is a layer that outputs the result processed by the hidden layer L2 as output data DOUT. For example, the output layer L3 is composed of a fully connected layer or the like.

Convolution is, for example, based on a filter, mask, or kernel (hereinafter simply referred to as "filter"), or the like, to an image or a feature map generated by performing a predetermined process on the image. On the other hand, it is a process of performing filtering and generating a feature map.

Specifically, a filter is data used to perform calculations for multiplying pixel values of an image or feature map by filter coefficients (sometimes referred to as "weights" or "parameters"). Note that the filter coefficient is a value determined by learning, setting, or the like.

The convolution process is a process of multiplying each pixel value of pixels constituting an image or a feature map by a filter coefficient to generate a feature map having the calculation results as constituent elements.

Thus, once the convolution process is performed, the features of the image or feature map can be extracted. A feature is, for example, an edge component, a result of statistical processing of the periphery of a target pixel, or the like.

Further, when the convolution process is performed, the subject or the like indicated by the target image or feature map shifts up and down, shifts left and right, shifts obliquely, rotates, or is an image or feature map that is a combination of these. Similar features can be extracted from

Pooling is a process of calculating the average, extracting the minimum value, or extracting the maximum value for a target range, extracting features, and generating a feature map. That is, the pooling is max pooling, avg pooling, or the like.

Note that convolution and pooling may be preprocessed such as zero padding.

The so-called data amount reduction effect, synthesizability, or movement invariance can be obtained by convolution, pooling, or a combination thereof as described above.

Normalization is, for example, a process of aligning variances and average values. It should be noted that normalization includes a case where it is performed locally. Then, normalization means that the data becomes a value or the like within a predetermined range. Therefore, the data can be easily handled in subsequent processing.

Fully connected is a process of putting data such as feature maps into an output.

For example, the output is in the form of a binary output, such as "YES" or "NO". In such an output format, full connection is a process of connecting nodes based on the features extracted in the hidden layer L2 so as to obtain either of the two types of conclusions.

On the other hand, when there are three or more types of outputs, the full combination is a process that performs a so-called softmax function. In this way, the full combination enables classification (including output indicating probability) by maximum likelihood estimation or the like.

[Other embodiments]
The learning data generation device 10, the learning device 301, and the thinning target object estimation device 402 may be different types of information processing devices. That is, the learning data generation device 10, the learning device 301, and the thinning target object estimation device 402 may have different hardware configurations.

The learning data may also be called teacher data, training data, or the like.

Embodiments may be combinations of the above embodiments. That is, a device that generates learning data, a device that performs learning processing on a learning model to generate a trained model, and a device that performs execution processing using the trained model may be the same device or different devices. may be In this way, the learning of the learning model and the execution by the trained model do not have to be performed by the same information processing device. That is, the learning of the learning model and the execution by the trained model may be performed by different information processing apparatuses.

If the devices are different devices, the devices transmit and receive data such as learning data or learned models via a network or the like.

Therefore, a trained model may be distributed in the form of a program or the like via a network or the like after being generated by learning, and may be executed by a device different from the information processing device in which the trained model was trained. In addition, learning may be performed in addition to a learning model generated by learning in another information processing apparatus.

Note that the learning data may be subjected to data augmentation. Specifically, when the learning data is image data, the data may be expanded by extracting a part of the image indicated by the image data to generate new data.

Similarly, data augmentation includes, for example, rotation, slide, partial shearing of data, horizontal flip, vertical flip, add distortion, correct distortion, change shade, correct color, reduce noise, add noise, This is a process of randomly applying filtering, enlargement, reduction, edge enhancement, or a combination thereof to image data.

By increasing the amount of learning data through data expansion in this way, it is possible to increase the amount of learning data used for learning a learning model.

In the embodiment, processing for reducing overfitting (also referred to as “overfitting” or “overfitting”) such as batch normalization or dropout may be performed. In addition, processing such as dimension reduction may be performed.

The network structure in the learning model, the trained model, etc. is not limited to the network structure of CNN. For example, the network structure may have a configuration such as RNN (Recurrent Neural Network), LSTM (Long Short-Term Memory), or Transformer.

Also, the learning model and the trained model may be configured to have hyperparameters. That is, the learning model and the learned model may be partially configured by the user.

In addition, for example, when dealing with a graph (data composed of vertices and edges), the learning model and the trained model are Graph Neural Network (Graph Neural Network, GNN), etc. It may have a structure.

Also, the learning model and the trained model may utilize other machine learning. For example, the learning model and the trained model may be subjected to preprocessing such as normalization by an unsupervised model.

The present invention includes a program (including firmware and programs equivalent to the learning data generation method, the learning method, the estimation method, or the processing equivalent to the processing shown above, hereinafter simply "program" ) may be implemented.

That is, the present invention may be realized by a program or the like written in a programming language or the like so as to issue a command to a computer and obtain a predetermined result. Note that the program may be configured such that part of the processing is executed by hardware such as an Integrated Circuit (IC) or an arithmetic unit such as a Graphics Processing Unit (GPU).

The program causes the computer to execute the processes described above by cooperating with the arithmetic device, the control device, the storage device, and the like of the computer. That is, the program is loaded into the main storage device or the like, issues instructions to the arithmetic unit to perform arithmetic operation, and operates the computer.

Also, the program may be provided via a computer-readable recording medium or an electric communication line such as a network.

The present invention may be implemented in a system composed of multiple devices. That is, a system of multiple computers may perform the processes described above redundantly, in parallel, distributed, or any combination thereof. Therefore, the present invention may be realized by devices with hardware configurations other than those shown above, and systems other than those shown above.

In addition, the present invention is not limited to the embodiments illustrated above. Therefore, the present invention can be added or modified without departing from the technical scope. Therefore, all the technical matters included in the technical idea described in the claims are covered by the present invention. In addition, the embodiment illustrated above is a specific example suitable for implementation. A person skilled in the art can implement various modifications from the disclosed contents, and such modifications are included in the technical scope described in the claims.

10: learning data generation device 10F1: image data input unit 10F2: extraction unit 10F3: generation unit 10F4: identification unit 10F5: mask image data generation unit 10F6: illustration processing unit 11: camera 11D1: first input image data 11D2: first 2 Input image data 12 : First crop 13 : Second crop 14 : Worker 15 : Learning data 20 : Extraction result 21 : Estimation result image data 22 : Correct data 31 : First object 32 : Second object 33 : Third Object 34 : Fourth object 40 : Mask image data 41 : First target object 42 : Second target object 43 : Third target object 44 : Fourth target object 50 : Illustrated image data 51 : Target object region 52 : Filled region 101: first target object 102: second target object 103: third target object 104: fourth target object 105: fifth target object 106: sixth target object 107: seventh target object 301: learning device 301F1: learning data Input unit 301F2 : Learning unit 302 : Learning model 303 : Learned model 401 : Unknown image data 402 : Thinning object estimation device 402F1 : Estimation unit 402F2 : Output unit 403 : Output screen 404 : User 405 : Setting screen 500 : Learning system

Claims (7)

A learning device for learning a learning model having a generation unit and an identification unit,
an image data input unit for inputting first input image data, which is image data representing the crops before thinning, and second input image data, which is image data representing the crops after thinning;
the generation unit that generates estimation result image data indicating a result of estimating a thinning target object in the crop;
A learning device comprising: the identification unit that identifies the estimation result image data, feeds back the identification result to the generation unit, and learns the learning model. A thinning target object estimation device that uses a learned model learned by the learning device according to claim 1,
an image data input unit for inputting unknown image data showing unknown crops before fruit thinning;
an estimating unit that estimates the thinning object using the learned model;
and an output unit that outputs an estimation result obtained by the estimation unit. 3. The thinning target object estimation apparatus according to claim 2, further comprising a mask image data generation unit that generates mask image data that distinguishes between a target object and objects other than the target object. The thinning target object estimation device according to claim 2 or 3, further comprising an illustration processing unit that illustrates the target object in the first input image data. A program for causing a computer having a generation unit and an identification unit to execute a learning method,
an image data input procedure in which a computer inputs first input image data, which is image data representing the crop before thinning, and second input image data, which is image data representing the crop after thinning;
a generation procedure in which a computer generates estimation result image data indicating a result of estimating a thinning object in the crop;
A program for causing a computer to identify the estimation result image data, feed back the identification result to the generation unit, and execute an identification procedure for learning a learning model. A program for causing a computer using a trained model trained by executing the program according to claim 5 to execute an estimation method,
An image data input procedure in which a computer inputs unknown image data showing an unknown crop before fruit thinning;
an estimation procedure in which a computer estimates the thinning object using the learned model;
A program for causing a computer to execute an output procedure for outputting an estimation result obtained by the estimation procedure. An estimation system having a learning device for learning a learning model having a generation unit and an identification unit, and a thinning target object estimation device that uses the learned model learned by the learning device,
The learning device
an image data input unit for inputting first input image data, which is image data representing the crops before thinning, and second input image data, which is image data representing the crops after thinning;
the generation unit that generates estimation result image data indicating a result of estimating a thinning target object in the crop;
the identification unit that identifies the estimation result image data, feeds back the identification result to the generation unit, and learns the learning model;
The thinning target object estimation device includes:
an image data input unit for inputting unknown image data showing unknown crops before fruit thinning;
an estimating unit that estimates the thinning object using the learned model;
An estimation system comprising an output unit that outputs an estimation result obtained by the estimation unit.

Images