JP7842774B2 - Learning device - Google Patents

Description

This disclosure relates to a learning device.

At waste disposal sites, large quantities of waste flow along conveyor belts and are processed daily. At these sites, waste sorting is carried out manually. While waste sorting is a simple task, it places a heavy burden on the workers who perform it (hereinafter sometimes referred to as "sorters"). Therefore, automated waste sorting devices (hereinafter sometimes referred to as "waste sorting devices") have been developed.

International Publication No. 2016/084336 Japanese Patent Publication No. 2019-074945

When a waste sorting machine replaces a sorting worker, it is conceivable that the machine would recognize each type of waste flowing on the conveyor belt and, based on the recognition results, use a robotic hand or suction pad to extract the desired waste (hereinafter sometimes referred to as "desired waste") from the group of waste flowing on the conveyor belt. Therefore, when multiple types of waste flow together on the conveyor belt, the waste sorting machine needs to be able to identify the types of waste. In order for the waste sorting machine to recognize various types of waste, recognition using a trained model generated by machine learning is effective.

However, since machine learning requires a vast amount of training data, if the annotation process for generating training data from image data of various types of waste is to be done manually by visually inspecting the images, generating the training data becomes extremely labor-intensive.

Therefore, this disclosure proposes a technology that can improve the efficiency of annotation work.

The learning device of this disclosure comprises a first annotation unit, a learning unit, and a model management unit. The first annotation unit performs a first annotation process to assign feature information, which is information indicating the features of the object image, to the object image using a second trained model replicated from a first trained model used by an object selection device that selects a desired object based on the recognition result of the object image. The learning unit updates the second trained model by performing machine learning using the object image and the feature information assigned by the first annotation process as training data. The model management unit updates the first trained model using the updated second trained model.

The disclosure technology can improve the efficiency of annotation work.

Figure 1 shows an example of the configuration of an object sorting system according to Embodiment 1 of this disclosure. Figure 2 shows an example of the operation of the learning device of Embodiment 1 of this disclosure. Figure 3 shows an example of the operation of the learning device according to Embodiment 1 of this disclosure. Figure 4 shows an example of the operation of the learning device according to Embodiment 1 of this disclosure. Figure 5 shows an example of the operation of the learning device according to Embodiment 1 of this disclosure. Figure 6 shows an example of the operation of the learning device according to Embodiment 1 of this disclosure. Figure 7 shows an example of the operation of the learning device of Embodiment 1 of this disclosure. Figure 8 shows an example of the operation of the learning device according to Embodiment 1 of this disclosure. Figure 9 shows an example of the operation of the learning device according to Embodiment 1 of this disclosure. Figure 10 shows an example of the operation of the learning device according to Embodiment 1 of this disclosure. Figure 11 shows an example of the operation of the learning device according to Embodiment 1 of the present disclosure. Figure 12 shows an example of the operation of the learning device according to Embodiment 1 of this disclosure.

The embodiments of this disclosure will be described below with reference to the drawings. In the following embodiments, identical components will be denoted by the same reference numerals.

[Example 1]
<Configuration of the object sorting system>
Figure 1 shows an example of the configuration of an object sorting system according to Embodiment 1 of this disclosure.

In Figure 1, the object sorting system 1 comprises a learning device 10, a camera 20, an object sorting device 30, a display 40, and an input device 50. The display 40 and the input device 50 are connected to the learning device 10. Examples of the input device 50 include a pointing device such as a mouse and a keyboard. The learning device 10, the camera 20, and the object sorting device 30 are connected to each other via a network.

The learning device 10 includes a first annotation unit 11, a training data storage unit 12, a machine learning unit 13, a pre-trained model storage unit 14 for annotation, a model management unit 15, a replication unit 16, and a second annotation unit 17.

The object sorting device 30 includes a learning model storage unit 31 for recognition, an image recognition unit 32, and a desired object extraction unit 33.

In the following explanation, we will use the example of the object sorting system 1 shown in Figure 1 being installed in a waste treatment plant where waste materials flow on a conveyor belt. In other words, the following explanation will use the example of the object sorting system 1 being used when the objects to be sorted are waste materials. However, the object sorting system 1 may also be installed in an assembly plant or similar facility where parts flow on a conveyor belt. In short, the objects to be sorted by the object sorting system 1 are not limited to waste materials, and the object sorting system 1 can be used for a variety of objects.

Camera 20 is positioned above the conveyor belt through which the waste materials are transported, and it photographs the waste materials as they are being transported by the conveyor belt. The images captured by camera 20 (hereinafter sometimes referred to as "captured images") are transmitted from camera 20 to both the learning device 10 and the object sorting device 30. The first annotation unit 11 and the image recognition unit 32 receive the captured images transmitted from camera 20.

In the object sorting device 30, the image recognition unit 32 uses the trained recognition model stored in the trained recognition model storage unit 31 to recognize each waste image (hereinafter sometimes referred to as "waste image") present in the captured image, and outputs the recognition result to the desired object extraction unit 33. The image recognition unit 32 recognizes the waste images, for example, by performing instance segmentation on the captured image.

The desired object extraction unit 33 extracts the desired waste from the group of waste being transported by the belt conveyor, according to the recognition results from the image recognition unit 32. Examples of the desired object extraction unit 33 include a robotic hand and a suction pad.

On the other hand, in the learning device 10, the duplication unit 16 operates only when the annotation-trained model is initialized. When the annotation-trained model is initialized, the duplication unit 16 duplicates the recognition-trained model stored in the recognition-trained model storage unit 31, and stores the duplicated recognition-trained model in the annotation-trained model storage unit 14, thereby initializing the annotation-trained model using the recognition-trained model.

The first annotation unit 11 uses the pre-trained annotation model stored in the pre-trained annotation model storage unit 14 to recognize each waste image present in the captured image and performs a process (hereinafter sometimes referred to as "first annotation processing") to assign information indicating the characteristics of each waste image (hereinafter sometimes referred to as "feature information") to each waste image. The first annotation unit 11 assigns feature information to waste images whose image recognition score is equal to or greater than the threshold TH1. The feature information includes information indicating the contour of the waste image (hereinafter sometimes referred to as "contour information") and information indicating the color of the waste image (hereinafter sometimes referred to as "color information"). The first annotation unit 11 associates the waste images with the feature information already assigned to them and generates training data (hereinafter sometimes referred to as "first training data") that includes the waste images and feature information. Therefore, the data in which feature information has been assigned to each waste image in the captured image becomes the first training data. The first annotation unit 11 stores the generated first training data in the training data storage unit 12. The first annotation unit 11 recognizes waste images, for example, by performing instance segmentation on the captured image.

The second annotation unit 17 acquires the first training data stored in the training data storage unit 12 and displays the captured images to which feature information has been added for each waste image on the display 40. The input device 50 is operated by an operator, who can specify any part of the captured image displayed on the display 40 using the input device 50. The second annotation unit 17 uses the trained annotation model stored in the trained annotation model storage unit 14 to recognize an image of an arbitrarily specified part of the captured image (hereinafter sometimes referred to as the "specified area image") and performs a process (hereinafter sometimes referred to as the "second annotation process") to add feature information to the specified area image. The second annotation unit 17 adds feature information to specified area images whose image recognition score is greater than or equal to a threshold TH2, which has a value smaller than the threshold TH1 used by the first annotation unit 11. The second annotation unit 17 associates the designated area image with the feature information already assigned to the designated area image, and generates training data (hereinafter sometimes referred to as "second training data") that includes the designated area image and the feature information. Therefore, the data on which feature information has been assigned to the designated area image in the captured image becomes the second training data. The second annotation unit 17 stores the generated second training data in the training data storage unit 12. The second annotation unit 17 recognizes waste images by, for example, performing instance segmentation on an arbitrarily designated area in the captured image.

The machine learning unit 13 performs machine learning using the first and second training data stored in the training data storage unit 12, and updates the annotation trained model stored in the annotation trained model storage unit 14 with the trained model after machine learning.

The model management unit 15 updates the recognition trained model stored in the recognition trained model storage unit 31 using the updated annotation trained model stored in the annotation trained model storage unit 14 (hereinafter sometimes referred to as the "updated annotation trained model"). The model management unit 15 performs a recognition test similar to the recognition process performed by the image recognition unit 32, for example, using the updated annotation trained model, and updates the recognition trained model with the updated annotation trained model when the recognition accuracy reaches the target accuracy.

<Operation of the learning device>
Figures 2 to 12 show examples of the operation of the learning device according to Embodiment 1 of this disclosure. In the following, we assume that empty bottles are used as waste, and that the object sorting device 30 sorts each empty bottle flowing on the conveyor belt into brown empty bottles (hereinafter sometimes referred to as "brown bottles") and empty bottles of other colors (hereinafter sometimes referred to as "other-colored bottles") (hereinafter sometimes referred to as "other-colored bottles"). In the following, we also assume that the desired waste is brown bottles, and that first and second training data for brown bottles are generated.

As shown in Figure 2, for example, the captured image I1 includes images of empty bottles (hereinafter sometimes referred to as "empty bottle images") B11 and B12 as waste images. Empty bottle image B11 is an image of a brown bottle (hereinafter sometimes referred to as "brown bottle image"), and empty bottle image B12 is an image of a bottle of another color (hereinafter sometimes referred to as "other color bottle image").

The first annotation unit 11 assigns contour information INb1 to the empty bottle image B11 and contour information INb2 to the empty bottle image B12 in the captured image I1. The contour information INb1 and INb2 are formed by a plurality of coordinate points [x1, y1], [x2, y2], ... Furthermore, the first annotation unit 11 assigns label information INa1 and image information INc1 to the empty bottle image B11 in the captured image I1, and assigns label information INa2 and image information INc2 to the empty bottle image B12. Label information INa1 includes the color information "brown bottle" for the empty bottle image B11, and label information INa2 includes the color information "other color bottle" for the empty bottle image B12. Furthermore, the image information INc1 and INc2 include the image file name "XXX.bmp" indicating the captured image I1, and height information "1536" and width information "2048" indicating the pixel size of the captured image I1. Then, annotation data AD1 for the empty bottle image B11 is formed from the label information INa1, contour information INb1, and image information INc1, and annotation data AD2 for the empty bottle image B12 is formed from the label information INa2, contour information INb2, and image information INc2. The first annotation unit 11 associates the captured image I1 with the annotation data AD1 and AD2 with each other and generates first training data including the captured image I1 and the annotation data AD1 and AD2. The annotation data AD1 and AD2 are generated by the first annotation unit 11, for example, as files in JSON format.

For example, as shown in Figure 3, the captured image I2 includes empty bottle images B21, B22, B23, B24, and B25 as waste images. The first annotation unit 11, which performs the first annotation process on the captured image I2, assigns label information "brown bottle" and contour information CO1 to empty bottle image B21, label information "other color bottle" and contour information CO2 to empty bottle image B22, label information "brown bottle" and contour information CO4A to empty bottle image B24, and label information "brown bottle" and contour information CO5 to empty bottle image B25.

Here, we assume that the correct color of empty bottle image B21 is brown, the correct color of empty bottle image B22 is another color, the correct color of empty bottle image B24 is brown, and the correct color of empty bottle image B25 is another color. Therefore, the label information assigned to empty bottle images B21, B22, and B24 is correct, while the label information assigned to empty bottle image B25 is incorrect, as shown in Figure 3. Thus, empty bottle image B25 is a waste image (hereinafter sometimes referred to as a "failed image") in which the assignment of feature information by the first annotation process failed.

Furthermore, as shown in Figure 3, the contour indicated by the contour information CO4A assigned to the empty bottle image B24 deviates from the correct contour of the empty bottle image B24. Therefore, the empty bottle image B24, like the empty bottle image B25, is a failed image.

Furthermore, although the captured image I2 includes the empty bottle image B23, as shown in Figure 3, label information and contour information have not been assigned to the empty bottle image B23. In other words, the first annotation unit 11 has not detected the empty bottle image B23. Therefore, the empty bottle image B23, like the empty bottle images B24 and B25, is a failed image.

Furthermore, as shown in Figure 3, even though region R6 in the captured image I2 does not contain an empty bottle image with a contour indicated by contour information CO6, the label information "brown bottle" and contour information CO6 are assigned to region R6. In other words, the first annotation unit 11 mistakenly detected region R6 as an empty bottle image.

As described above, in the captured image I2, the empty bottle images B23, B24, and B25 are failure images, and region R6 is incorrectly detected as an empty bottle image.

Therefore, for example, the operator uses the pointer PO displayed on the captured image I2 by operating the input device 50 to specify the empty bottle image B23 by clicking on the empty bottle image B23 in the captured image I2, for example as shown in Figure 4. Alternatively, the operator uses the pointer PO to specify the empty bottle image B23 by drawing a region RS surrounding the location where the empty bottle image B23 exists in the captured image I2, for example as shown in Figure 5. Once the empty bottle image B23 is specified by operating the input device 50, the second annotation unit 17 uses the annotation trained model stored in the annotation trained model storage unit 14 to assign contour information CO3 to the empty bottle image B23, as shown in Figure 6.

For example, the operator uses the pointer PO to specify region R6 by clicking on the area enclosed by the contour indicated by the contour information CO6 in the captured image I2, as shown in Figure 7. Once region R6 is specified by the operation of the input device 50, the second annotation unit 17 deletes the contour information CO6 that was assigned to region R6, as shown in Figure 8.

For example, the operator uses the pointer PO to click on the label information "brown bottle" attached to the empty bottle image B25 in the captured image I2, as shown in Figure 9, thereby specifying the label information for the empty bottle image B25. Once the label information is specified by the operation of the input device 50, the second annotation unit 17 enables the modification of the specified label information. The operator then uses, for example, the keyboard to modify the label information attached to the empty bottle image B25 from "brown bottle" to "other color bottle," as shown in Figure 10. In accordance with the operator's modification, the second annotation unit 17 modifies the label information "brown bottle" attached to the empty bottle image B25 to the label information "other color bottle" which indicates the correct color of the empty bottle image B25.

For example, the operator uses the pointer PO to specify the contour indicated by the contour information CO4A in the captured image I2, as shown in Figure 11. Once the contour is specified by the operation of the input device 50, the second annotation unit 17 enables the modification of the specified contour. The operator then uses the pointer PO to modify the contour of the empty bottle image B24 to the correct contour, as shown in Figure 12. For example, the operator modifies the contour by moving the vertices of the contour indicated by the contour information CO4A using the pointer PO. In accordance with the operator's modification, the second annotation unit 17 modifies the contour information CO4A attached to the empty bottle image B24 to contour information CO4B that indicates the correct contour of the empty bottle image B24.

The above describes Example 1.

[Example 2]
The training data storage unit 12, the annotation-trained model storage unit 14, and the recognition-trained model storage unit 31 are implemented as hardware, for example, by memory or storage. The first annotation unit 11, the machine learning unit 13, the model management unit 15, the replication unit 16, the second annotation unit 17, and the image recognition unit 32 are implemented as hardware, for example, by a processor such as a CPU (Central Processing Unit), DSP (Digital Signal Processor), FPGA (Field Programmable Gate Array), or ASIC (Application Specific Integrated Circuit).

The above describes Example 2.

As described above, the learning device of this disclosure (learning device 10 in the embodiment) comprises a first annotation unit (first annotation unit 11 in the embodiment), a learning unit (machine learning unit 13 in the embodiment), and a model management unit (model management unit 15 in the embodiment). The first annotation unit performs a first annotation process to attach feature information, which is information indicating the features of an object image, to an object image using a second trained model (trained model for annotation in the embodiment) which is duplicated from a first trained model (trained model for recognition in the embodiment) used by an object selection device (object selection device 30 in the embodiment) that selects a desired object based on the recognition result of an object image. The learning unit updates the second trained model by performing machine learning using the object image and the feature information attached by the first annotation process as training data. The model management unit updates the first trained model using the updated second trained model.

This allows the first annotation unit to automatically perform annotation on object images, thereby increasing the efficiency of the annotation process. Furthermore, by sharing the pre-trained model used for object image recognition in the object sorting device with the first annotation process, there is no need to prepare a separate pre-trained model for the first annotation process, further improving the efficiency of annotation. Additionally, since the second pre-trained model is updated as needed, the accuracy of the first annotation process improves with each update to the second pre-trained model. Moreover, since the first pre-trained model is updated as needed with each update to the second pre-trained model, the accuracy of object image recognition in the object sorting device also improves with each update to the second pre-trained model.

Furthermore, the learning device of this disclosure (learning device 10 in the embodiment) has a second annotation unit (second annotation unit 17 in the embodiment). The second annotation unit performs a second annotation process to arbitrarily designated failure images, which are object images for which the first annotation process failed to assign feature information, by using a second trained model to assign feature information. The learning unit updates the second trained model by performing machine learning using the object image, the feature information assigned by the first annotation process, and the feature information assigned by the second annotation process as training data. For example, the feature information includes contour information that shows the contour of the object image.

This improves the efficiency of annotation work performed by operators on failed images. In particular, it reduces the effort required to add contour information to failed images. Furthermore, the accuracy of the second pre-trained model is improved by the second annotation process, further improving the accuracy of the first annotation process.

Furthermore, the score threshold used when feature information is assigned by the second annotation process (threshold TH2 in the example) is smaller than the score threshold used when feature information is assigned by the first annotation process (threshold TH1 in the example).

By doing this, the success rate of recognizing failed images in the second annotation process can be increased compared to the success rate of recognizing object images in the first annotation process.

1. Object sorting system 10. Learning device 11. First annotation unit 12. Training data storage unit 13. Machine learning unit 14. Annotation trained model storage unit 15. Model management unit 16. Replication unit 17. Second annotation unit 20. Camera 30. Object sorting device 31. Recognition trained model storage unit 32. Image recognition unit 33. Desired object extraction unit.

Claims (1)

A first annotation unit uses a second trained model, which is a replica of a first trained model used by an object sorting device that sorts desired objects based on the recognition results of an object image, to add feature information, which is information indicating the features of the object image, to the object image.
A second annotation unit modifies the feature information based on the image specification, targeting the object image in which the first annotation unit failed to assign the feature information.
A learning unit updates the first trained model by performing machine learning using the object image, the feature information assigned by the first annotation unit, and the feature information modified by the second annotation unit as training data.
It is equipped with ,
The score threshold used when the feature information is modified by the second annotation unit is smaller than the score threshold used when the feature information is assigned by the first annotation unit.
Learning device.