JP2020024534A - Image classifier and program - Google Patents

Abstract

To reduce the time and labor in collecting useful teacher data when using teacher data to learn a learning model for classifying images.SOLUTION: An image classification unit 30 of an image classifier 3 estimates scores for respective categories by using a learning model with respect to each teacher candidate image of a plurality of collected teacher candidate images and classifies the teacher candidate image into a category giving a maximum score and sorts the teacher candidate images in the ascending order of score with respect to the respective categories to generate classification results for the respective categories. A correction unit 33 prompts an operator to confirm the classification results for the respective categories in the ascending order of score from the teacher candidate image having the lowest score and corrects the categories as needed in accordance with operator's operation to generate teacher data for the respective categories. A learning unit 35 uses the teacher data for the respective categories to learn the learning model.SELECTED DRAWING: Figure 1

Description

  The present invention relates to the field of image processing using a computer and a hard disk, and more particularly to an image classification device and a program for classifying collected images to generate teacher data and learning a learning model using the teacher data.

  In recent years, deep learning for inputting an image and directly outputting a classification result thereof has attracted attention. According to this technique, by using a learning model generated by deep learning, feature data useful for image classification can be automatically generated.

  For this reason, there is an advantage that the design and selection of the feature data by a person becomes unnecessary. There is also a report that classification using a learning model provides higher accuracy than classification of images using feature data manually designed or selected by a person.

  On the other hand, learning of an image classification device using deep learning requires a large amount of teacher data in which a set of an image and a correct answer label is used. However, collection of a large amount of teacher data is supposed to be performed manually, which requires a great deal of labor and time.

  A plurality of proposals have been made on teacher data generation techniques for image classification. For example, Patent Literature 1 proposes a technique for reducing the load on an operator when generating teacher data for automatically classifying a defect of a substrate.

  Patent Document 2 proposes a technique of generating teacher data by dividing an image into regions and performing clustering, and selecting the data as positive case data or negative case data according to an instruction from an operator or the like.

  Further, Patent Document 3 includes a plurality of detectors for detecting a region to be detected from an image in order to generate teacher data effective for learning, and integrates these detection results to generate teacher data. Techniques for selecting have been proposed.

  As a technology of image classification using deep learning, Patent Literature 4 proposes a technology of performing image classification processing and re-learning processing in parallel. Specifically, the control unit controls the operation, suspension, and the like of the plurality of determination units that perform the classification process, controls the re-learning unit that performs the re-learning, and controls the execution of the re-learning. In addition, control is performed such that the learning model is copied to a plurality of determination units and operated.

JP 2011-158373 A JP 2009-282660 A JP 2012-190159 A JP-A-2017-21690

  However, in the technique of Patent Document 1 described above, it is necessary to determine in advance a feature amount for automatically determining whether a label temporarily assigned by a classifier is correct or not. Further, in the technique of Patent Literature 2, in order to collect teacher data, automation is performed by clustering, but feature amounts used for clustering are set in advance. For this reason, these techniques are difficult to apply to an image classifying apparatus in which it is difficult to design and select a feature amount, and also difficult to apply to an advanced image classifying apparatus used for determining the state of an image.

  Further, in the technique of Patent Document 2, collection of teacher data and learning in a classifier are performed independently, so that useful teacher data cannot always be used in a classifier that performs learning.

  In the technique of Patent Document 3, an image to be added to teacher data is determined by integrating a plurality of detectors and integrating the detection results, but the detection result by one detector is used as teacher data. Not applicable in cases. In addition, there is no description about the parallel processing of checking teacher data and learning a model by an operator, and there is a problem that a series of processing from data collection to model learning requires time.

  Further, in the technique of Patent Document 4, parallel processing of the re-learning process is automatically performed. However, it is premised that correct labels are obtained in advance, and the labor for collecting teacher data is solved. It has not been.

  As described above, a large amount of teacher data is required for learning of the image classification device. However, even if a large amount of teacher data is collected, if the teacher data is not useful, a learning model for performing highly accurate classification cannot be generated. Therefore, a mechanism for collecting useful teacher data with low labor and in a short time has been desired.

  Therefore, the present invention has been made to solve the above-described problem, and an object of the present invention is to collect useful teacher data when learning a learning model for classifying images using the teacher data. To provide an image classification device and a program that can reduce the labor and time required for the image classification.

  In order to solve the problem, the image classification device according to claim 1, wherein the image classification device performs learning of a learning model for classifying images, wherein the learning model is generated for each of a plurality of collected teacher candidate images. An image classification unit that estimates a score for each category by using the category, classifies the plurality of teacher candidate images into categories having the highest score, sorts the plurality of teacher candidate images in descending order of the score for each category, and generates a classification result for each category And for the teacher candidate image of the classification result generated by the image classification unit, for each category, prompt the operator to confirm in the order of the lowest score, correct the category according to the operation of the operator, Using a correction unit that generates a teacher candidate image as teacher data, and using the teacher data for each category generated by the correction unit, A learning unit that performs learning of the serial learning model, characterized by comprising a.

  The image classification device according to claim 2 is the image classification device according to claim 1, further comprising a scheduler, wherein the image classification unit is configured to collect the plurality of teacher candidate images from the image collection device. Inputting the teacher candidate images, the scheduler collects the plurality of teacher candidate images by the image collection device, a classification process of generating the classification result by the image classification unit, and the teacher data by the correction unit. And a learning process for learning the learning model by the learning unit. The classification process by the image classification unit and the learning process by the learning unit are performed simultaneously. Output a classification start instruction for starting the classification process to the image classifying unit so as to prevent the learning process from starting. And it outputs a learning instruction to start the learning section, and wherein the.

  The image classification device according to claim 3 is the image classification device according to claim 2, wherein the scheduler outputs a collection start instruction for starting the collection processing to the image collection device. When a collection completion indicating that the collection processing has been completed is input, it is determined that the collection processing has been completed, the collection processing by the image collection device has been completed, and the learning processing by the learning unit has been completed. Is completed, the classification start instruction is output to the image classification unit, and when the classification completion indicating that the classification processing is completed is input from the image classification unit, it is determined that the classification processing is completed. If the classification process has been completed, a correction start instruction for starting the correction process is output to the correction unit, and the correction completion indicating that the correction process has been completed is output to the correction unit. When input from the main part, it is determined that the correction processing has been completed, and when the correction processing by the correction part has been completed and the classification processing by the image classification part has been completed, the learning start is started. An instruction is output to the learning unit, and when learning completion indicating that the learning process has been completed is input from the learning unit, it is determined that the learning process has been completed.

  Furthermore, a program according to a fourth aspect causes a computer to function as the image classification device according to any one of the first to third aspects.

  As described above, according to the present invention, when learning a learning model for classifying images using teacher data, it is possible to reduce labor and time for collecting useful teacher data. .

1 is a schematic diagram of an entire system including an image classification device according to an embodiment of the present invention. It is a flowchart explaining the flow of the whole process. FIG. 3 is a diagram illustrating an example of a processing flow of an image collection device and an image classification device. FIG. 9 is a diagram illustrating an example of processing of a learning model by an image classification unit and a learning unit. FIG. 6 is a diagram illustrating a configuration example of a classification result stored in a storage unit. FIG. 4 is a diagram illustrating a configuration example of teacher data stored in a storage unit. 5 is a flowchart illustrating a processing example of an image classification unit. It is a flowchart which shows the example of a process of a correction part. It is a flowchart which shows the example of a process of a learning part. FIG. 9 is a diagram illustrating an example of parallel processing by a scheduler. 5 is a flowchart illustrating a control example of an image collection unit and a preprocessing unit by a scheduler. 5 is a flowchart illustrating a control example of an image classification unit by a scheduler. It is a flowchart which shows the control example of the correction part by a scheduler. It is a flowchart which shows the control example of the learning part by a scheduler.

Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the drawings.
FIG. 1 is a schematic diagram of an entire system including an image classification device according to an embodiment of the present invention. The overall system includes a storage device 1 such as a server that holds images, an image collection device 2, and an image classification device 3.

  The storage device 1 such as a server and the image collection device 2 are connected via a transmission path 4 such as the Internet, and the image collection device 2 and the image classification device 3 are connected to a LAN (Local Area Network) or the like. Connected via.

  The storage device 1 stores images that are candidates for teacher data used in the learning process of the image classification device 3. The storage device 1 may be a server or the like connected to the image collection device 2 via the transmission path 4 as shown in FIG. 1, or may be directly connected to the image collection device 2 and store the image as a database. It may be a stored hard disk or the like.

  FIG. 2 is a flowchart for explaining the overall processing flow in the overall system shown in FIG. First, the operator prepares a predetermined number of teacher data with correct labels (images and scores). Before performing the actual processing, the image classification device 3 performs initial learning of a learning model using a predetermined number of teacher data with correct answers prepared in advance by an operator (step S201).

  The image collection device 2 collects images from the external storage device 1, performs preprocessing on the images, and converts the images into a form suitable for learning (step S202). The image classification device 3 estimates a score (reliability) for each category using a learning model for each image (step S203), and sets the category having the highest score as the category to which the image belongs (step S204). The score indicates the probability that the image belongs to the category.

  The image classification device 3 sorts the images for each category in ascending order of the score (step S205). Then, the image classification device 3 prompts the operator to confirm in order from the image with the lowest score for each category (confirms whether or not the image belongs to the category), and sorts the category as necessary according to the operation of the operator. Correct (step S206).

  The image classification device 3 generates teacher data for each category, using the category that has not been modified after confirmation by the operator and the category that has been modified after confirmation by the operator as the correct category (step S207). Then, the image classification device 3 learns the learning model based on the teacher data for each category (step S208).

  Thereby, learning of the learning model is performed using the image with a low score as teacher data. The reason why images with low scores are used as teacher data is to update the learning model so that the images can be classified more correctly. In the first place, an image with a low score is an image that is difficult to be classified into a correct category by the current learning model. The category of this image is correctly corrected by the operator, and learning of the learning model is performed using the corrected image as teacher data, whereby the classification accuracy of an image that has been difficult to correctly classify can be increased.

  In other words, an image with a low score can be said to be useful teacher data because an image with a low score can be updated to a learning model with high classification accuracy by using it as teacher data. As described above, since an image having a low score is an image for which the current learning model is not good at classifying, the classification accuracy of the learning model can be efficiently increased by preferentially setting this as teacher data. .

  The image classification device 3 determines whether or not to end the process (whether or not a predetermined end condition is satisfied) (step S209), and when it is determined that the process is not to be ended (step S209: N), The process proceeds to S202, and the processes of steps S202 to S208 are repeated. On the other hand, when the image classification device 3 determines in step S209 that the process is to be ended (step S209: Y), the process ends.

  The image classification device 3 ends the process in step S209, for example, when the accuracy of the image classification becomes sufficient by additional learning or when a sufficient number of teacher data is obtained.

  With reference to FIG. 1, the image collection device 2 includes an image collection unit 20, a storage unit 21 in which teacher candidate images are stored, and a preprocessing unit 22. The image classification device 3 includes an image classification unit 30, a storage unit 31 in which a learning model is stored, a storage unit 32 in which images and scores for each category are stored, a correction unit 33, and a storage unit 34 in which images for each category are stored. , A learning unit 35 and a scheduler 36.

  FIG. 3 is a diagram illustrating an example of a processing flow of the image collection device 2 and the image classification device 3. The scheduler 36 of the image classification device 3 schedules the operations of the image collection unit 20 and the pre-processing unit 22 of the image collection device 2, and the image classification unit 30, the correction unit 33, and the learning unit 35 of the image classification device 3, Overall control is performed (step S300). Details of the scheduler 36 will be described later.

Image acquisition of the image acquisition device 2 20 via the transmission path 4 from the storage device 1 collects N images, storing unit N images teacher candidate image I _1, · · ·, as I _N 21 (step S301). N is an integer of 1 or more.

  For example, the image collection unit 20 may download an image on a Web page based on a condition such as a size or an aspect ratio, or may randomly select and read an image to be classified from a database in which images to be classified are registered. It may be.

Pre-processing unit 22, N pieces of the teacher candidate images I ₁ from the storage unit 21, ..., reads the I _N, suitable teacher candidate image I _1, ..., a I _N input format of the image classification device 3 A pre-process for converting the data into a form is performed (step S302). Then, the preprocessing unit 22 transmits the _N teacher candidate images I ₁ ,..., IN after the preprocessing to the image classification device 3.

  The preprocessing unit 22 converts, for example, the size of the image to match the input size of the learning model, randomly deforms the image to improve the generalization performance of the learning model, and adds noise.

The image classification unit 30 of the image classification device 3 receives _N pre-processed teacher candidate images I ₁ ,..., IN from the pre-processing unit 22 of the image collection device 2. Then, the image classification section 30, the teacher candidate image I _1, · · ·, to calculate a feature amount for each of the I _N, estimates the scores for each category based on the feature quantity, to identify the highest category score . The number of categories is C, and C is an integer of 2 or more.

Specifically, the image classification section 30 uses the stored learning model in the storage unit 31, a teacher candidate image I _1, · · ·, estimates the scores for each category for each of I _N, most score Identify high categories.

  Before the process of the image classification unit 30 is performed, it is assumed that the learning model has already been initially learned. As described above, during the initial learning, a predetermined number of teacher data with correct labels are prepared and learning is performed.

Image classifying unit 30 in accordance with the identified category, a teacher candidate image I _1, · · ·, classifies the respective I _N to any of the C-number of categories (step S303). The image classifying unit 30 stores the teacher candidate images I _{k, 1} ,..., I _{k, Nk} which are the classification results for each category, and the scores S _{k, 1 ,.} It is stored in the unit 32 (step S304). The details of the image classification unit 30 will be described later.

k is a category number, and k = 1,..., C. Nk is the number of teacher candidate images I _{k, 1} ,..., I _{k, Nk} classified into the category k, and is an integer of 0 or more. That is, the classification result of the category k is Nk teacher candidate images I _{k, 1} ,..., I _{k, Nk} and their scores S _{k, 1 ,.}

FIG. 4 is a diagram illustrating an example of processing of a learning model by the image classification unit 30 and the learning unit 35. 4, the image classifying unit 30, the teacher candidate image I _1, ···, I _N (collectively referred to. Teacher candidate image I) each, to the input layer as input data for learning model Then, a score S for each category is obtained from the output layer as output data of the learning model.

  Thereby, the score S for each category is estimated for the teacher candidate image I using the learning model. In the example of the score S (0.3, 0.1, 0,..., 0.1) shown in FIG. 4, the score of category 1 of the teacher candidate image I is 0.3, and the score of category 2 is 0. ., The score of category 3 is 0,..., The score of category C is 0.1. The sum of the scores for all categories is 1. Assuming that the maximum score is 0.3, the image classification unit 30 classifies the teacher candidate image I into category 1 of the maximum score.

  Here, the feature amount is calculated from the teacher candidate image I input to the input layer by using the learning model. The feature amount is a vector reflecting a local feature of the image or a semantic feature of the entire image. For example, in a convolutional neural network, the feature amount is obtained by repeatedly connecting a convolutional layer and a pooling layer. Alternatively, a predetermined feature amount that is not updated by learning, such as an image gradient or a color histogram, may be used.

  Then, a score for each category is calculated from the feature amount. As a calculation method, for example, in a convolutional neural network, it can be obtained by connecting a plurality of all connected layers and using a layer having an element of the number of categories (C) as an output layer.

The learning model is a model capable of supervised learning, and may be any model that outputs the classification result of the image as a score. For example, a neural network is used as the learning model. In this case, any type of neural network may be used, but a convolutional neural network used in deep learning is preferable. For the convolutional neural network, refer to the following document.
A. Krizhevsky et al., “Imagenet classification with deep convolutional neural networks”, Advances in neural information processing systems, pp.1097-1105 (2012)

FIG. 5 is a diagram illustrating a configuration example of the classification result stored in the storage unit 32. As shown in FIG. 5, for the category 1, the teacher candidate image I _{1, 1,} · · ·, I _{1, N1} and scores S _{1, 1,} · · ·, S _{1, N1} is stored in the storage unit 32 . Further, the category 2, the teacher candidate image I _2,1, · · ·, I _{2, N2} and scores S _2,1, · · ·, S _{2, N2} is stored in the storage unit 32. Similarly, for category C, teacher candidate images I _{C, 1} ,..., I _{C, NC} and scores S _{C, 1 ,.}

N1 is the number of teacher candidate images I _1,1 ,..., I _{1, N1} classified into category 1, and is an integer of 0 or more. N2 is the number of teacher candidate images I _2,1 ,..., I _{2, N2} classified into category 2, and is an integer of 0 or more. Similarly, NC is the number of teacher candidate images I _{C, 1} ,..., I _{C, NC} classified into category C, and is an integer of 0 or more.

Returning to FIGS. 1 and 3, the correction unit 33 reads from the storage unit 32 the teacher candidate images I _{k, 1} ,..., I _{k, Nk} and the scores S _{k, 1 ,. Read out Sk} and _Nk . Then, the correction unit 33, for each category, the score _{S k, 1, ···, S} k, low teacher candidate image I _k of _{Nk, 1, ···, I k} , a confirmation to the operator in order from _Nk Then, according to the operation of the operator, the category is corrected as needed (step S305).

  As a result, the category of the teacher candidate image I whose category is confirmed to be correct is left as it is, and the category of the teacher candidate image I whose category is confirmed to be incorrect is corrected.

The correction unit 33 stores the teacher candidate images I _{k, 1 ′} ,..., I _{k, Nk ′} for each category after confirmation and correction as teacher data in the storage unit 34 (step S306). Details of the correction unit 33 will be described later.

The teacher candidate images I _{k, 1 ′} ,..., I _{k, Nk ′} for each category after confirmation and modification include images that have not been modified after confirmation and images that have been modified after confirmation. k is a category number, and k = 1,..., C. Nk ′ is the number of confirmed and corrected teacher candidate images I belonging to the category k, and is an integer of 0 or more.

  As a result, confirmation and correction are performed in order from the teacher candidate image I having the lowest score, and teacher data is generated. Therefore, with respect to a teacher candidate image I having a low score (a teacher candidate image I with a wrong classification or a teacher candidate image I near a classification boundary where classification of a category is difficult), the category can be corrected to a correct one. This can be preferentially added to the teacher data. As described above, the teacher candidate image I with a low score is an image for which the current learning model is not good at classification. By using this as teacher data, it is possible to update the learning model with high classification accuracy.

  Since learning is performed in the learning unit 35 using useful teacher data, it can be said that the processing of the correction unit 33 is necessary for updating to a learning model with high classification accuracy.

  The process of correcting the category by the correcting unit 33 for the teacher candidate image I to which the category is added is different from the process of adding a new category to an image to which no category (label) is added. Processing load is reduced.

FIG. 6 is a diagram illustrating a configuration example of the teacher data stored in the storage unit 34. As shown in FIG. 6, (category 1) teacher data (images thereof) I _{1,1 ′} ,..., I _{1, N1 ′} are stored in the storage unit 34. For category 2, the teacher data I _{2,1 ′} ,..., I _{2, N2 ′} are stored in the storage unit 34. Similarly, for the category C, the teacher data I _{C, 1 ′} ,..., I _{C, NC ′} is stored in the storage unit 34.

N1 ′ is the number of teacher data I _{1,1 ′} ,..., I _{1, N1 ′} belonging to category 1, and is an integer of 0 or more. N2 ′ is the number of teacher data I _{2,1 ′} ,..., I _{2, N2 ′} belonging to category 2, and is an integer of 0 or more. Similarly, NC ′ is the number of teacher data I _{C, 1 ′} ,..., I _{C, NC ′} belonging to category C, and is an integer of 0 or more.

Returning to FIGS. 1 and 3, the learning unit 35 reads the teacher data I _{k, 1 ′} ,..., I _{k, Nk ′} for each category from the storage unit. Then, the learning unit 35 learns the learning model stored in the storage unit 31 based on the teacher data I _{k, 1 ′} ,..., I _{k, Nk ′} for each category (step S307). Details of the learning unit 35 will be described later.

Referring to FIG. 4, learning unit 35 receives each of teacher data I _{k, 1 ′} ,..., I _{k, Nk ′} as input data, and scores S for each category reflecting the category to which the teacher data belongs. Is used as the correct answer data to learn the learning model. The score S for each category is set such that the score of the category to which the teacher data belongs is 1 and the scores of the other categories are 0.

  The example of FIG. 4 shows a case where the teacher data belongs to category 2. The score S of the teacher data is S (0, 1, 0,..., 0) where the score of category 2 is 1 and the scores of other categories are 0. The learning unit 35 gives the teacher data and the score S to the learning model. Then, the learning unit 35 forwardly propagates the teacher data from the input layer, obtains an error signal between the output signal of the output layer and the score S, and reverse propagates the error signal from the output layer, thereby obtaining parameters such as weights. To update.

  Thereby, learning is performed using the useful teacher data generated by the correction unit 33, so that the learning model can be updated to a learning model with high classification accuracy, and the classification accuracy in the image classification unit 30 can be increased.

  Returning to FIGS. 1 and 3, the processing of steps S301 to S307 by the image collection device 2 and the image classification device 3 is repeatedly performed according to the processing of step S300.

  Thereby, the teacher data generated by the correction unit 33 is sequentially increased, and the accuracy of the classification process by the image classification unit 30 can be improved.

[Image Classification Unit 30]
Next, the image classification unit 30 of the image classification device 3 shown in FIG. 1 will be described in detail. FIG. 7 is a flowchart illustrating a processing example of the image classification unit 30.

  The image classification unit 30 determines whether a classification start instruction has been input from the scheduler 36 (step S701). When determining that the classification start instruction has not been input in step S701 (step S701: N), the image classification unit 30 waits until the classification start instruction is input. The classification start instruction is a signal for the scheduler 36 to cause the image classification unit 30 to start the classification processing.

On the other hand, if the image classification unit 30 determines in step S701 that a classification start instruction has been input (step S701: Y), the preprocessing unit 22 of the image collection device 2 sends the teacher candidate images I ₁ ,. _N is input (step S702).

Image classifying unit 30, the teacher candidate image I _1, · · ·, for each of I _N, by using the stored learning model in the storage unit 31, estimates the scores of each category (step S703). Thus, the teacher candidate image I _1, · · ·, for each of I _N, the score S ₁ for each category, · · ·, S _N is obtained.

Image classifying unit 30, the teacher candidate image I _1, · · ·, for each of I _N, identifies a maximum score among the scores for each category, the category of maximum score is set to the category of the teacher candidate image I (Step S704).

The image classifying section 30 classifies the teacher candidate images I ₁ ,..., _IN for each category (step S705). Then, the image classification section 30, for each category, the teacher candidate image I ₁ in ascending order of score, ..., by sorting the _{I N, k (k = 1} , ···, C) for th category image I _k, 1 _of, ···, I _k, obtain _Nk (step S706).

The image classification unit 30 generates teacher candidate images I _{k, 1} ,..., I _{k, Nk} and scores S _{k, 1} ,..., S _k, N _k which are classification results for each category (step S707). ) And store it in the storage unit 32 (step S708).

Image classifying unit 30, the teacher candidate image I ₁ inputted from the image acquisition device 2, ..., as the classification process I _N is completed, the classification completion corresponding to the classification start instruction input at step S701, the scheduler 36 (step S709). Classification completion is a signal indicating that the classification processing by the image classification unit 30 has been completed.

In this way, the image classification unit 30, in accordance with the classification start instruction, teacher candidate image I ₁ by using a learning model,..., Performs a classification of I _N, teacher candidate image I _k is the classification results of each _{category, 1,} ···, I _{k, Nk} and score _{S k, 1, ···, S} k, generates _Nk, outputs a classification completion.

[Modification unit 33]
Next, the correction unit 33 of the image classification device 3 shown in FIG. 1 will be described in detail. FIG. 8 is a flowchart illustrating a processing example of the correction unit 33.

  The correction unit 33 determines whether a correction start instruction has been input from the scheduler 36 (step S801). When determining in step S801 that the correction start instruction has not been input (step S801: N), the correction unit 33 waits until the correction start instruction is input. The correction start instruction is a signal for the scheduler 36 to cause the correction unit 33 to start the correction processing.

On the other hand, when the correction unit 33 determines in step S801 that a correction start instruction has been input (step S801: Y), the storage unit 32 stores the teacher candidate images I _{k, 1} ,. , I _{k, Nk} and scores S _{k, 1} ,..., S _{k, Nk} are read (step S802).

The correction unit 33 prompts the operator to confirm in order from the teacher candidate images I _{k, 1} ,..., I _{k, Nk having} the lower scores S _{k, 1 ,.} According to the operation of the operator, the category is corrected as needed (step S803).

The correction unit 33 generates teacher candidate images I _{k, 1 ′} ,..., I _{k, Nk ′} for each category after confirmation and correction as teacher data (step S804), and stores them in the storage unit. (Step S805).

The teacher candidate images I _{k, 1 ′} ,..., I _{k, Nk ′} for each category after confirmation and correction are the teacher candidate images I _{k, 1 ,.} Only the images confirmed by the operator. Here, in the image confirmed by the operator, the category is determined to be incorrect by the confirmation, and the image in which the category is corrected thereafter, and the category is determined to be correct by the confirmation, and then the category is corrected. Includes images that were not performed.

The correction unit 33 determines that the correction processing of the teacher candidate images I _{k, 1} ,..., I _{k, Nk} for each category classified by the image classification unit 30 has been completed _{, and responds} to the correction start instruction input in step S801. The corresponding correction completion is output to the scheduler 36 (step S806). The correction completion is a signal indicating that the correction processing by the correction unit 33 has been completed.

As described above, the correction unit 33 corrects the teacher candidate images I _{k, 1} ,..., I _{k, Nk} for each category according to the correction start instruction _{, and obtains} the teacher data I _k which is the confirmation correction result for each category. _{, 1 ′} ,..., _{Ik, Nk ′} , and outputs correction completion.

Note that, in step S803, the correction unit 33 may set all the teacher candidate images I _{k, 1} ,..., I _{k, Nk} of all the categories as the check and correction targets, or may set a preset number of teacher candidate images. The images I _{k, 1} ,..., I _{k, Nk} may be targeted for confirmation and correction.

For example, it is assumed that the upper limit number of sheets is set in advance for each category by the operator. The correction unit 33 reaches the upper limit number in order from the teacher candidate images I _{k, 1} ,..., I _{k, Nk having} the lower scores S _{k, 1 ,.} Prompt for confirmation and correct the category.

Further, for example, it is assumed that a score threshold value is set in advance for each category by an operator. Correction unit 33, for each category, the score _{S k, 1, ···, S} k, the image that low teacher candidate image I _k of _{Nk, 1, ···, I k} , in order from the _Nk, below the threshold Prompt for confirmation only and correct the category.

Further, the scheduler 36 may determine the number of sheets to be checked and corrected by the correction unit 33. For example, the scheduler 36 estimates the time from when the correction is started by the correction unit 33 to when the learning unit 35 completes the current learning. Then, the scheduler 36 determines the number of confirmation correction targets and outputs the number of confirmation correction targets to the correction unit 33 so that the correction unit 33 completes the correction processing at the timing when the time elapses. The correction unit 33 divides the number of objects to be checked and corrected by the number of categories, and for each category, a teacher candidate image I _{k, 1} ,... _Having a low score S _{k, 1 ,.} Confirmation is urged in order from _{Ik, Nk} until the number of division results is reached, and the category is corrected.

  Specifically, the scheduler 36 obtains a time from outputting a learning start instruction described later to the learning unit 35 to inputting a learning completion described later from the learning unit 35, and calculates the time as the number of teacher data. The learning time per teacher data is totalized by dividing by, and the average is calculated to estimate the learning time per teacher data. The scheduler 36 holds the estimated learning time per one teacher data.

  In addition, the scheduler 36 obtains the time from when the correction instruction start is output to the correction unit 33 to when the correction completion is input from the correction unit 33, and divides the time by the number of images that have been checked and corrected. Then, the correction time per image is totaled, the average is calculated, and the correction time per image is estimated. The scheduler 36 holds the estimated correction time per image.

  The scheduler 36 inputs the number of remaining teacher data in the current learning from the learning unit 35 at the time of the start of the correction by the correction unit 33, and adds the number of remaining teacher data to the number of remaining teacher data per one teacher data held. By multiplying the learning time, the time from the timing to the completion of the current learning is estimated.

  The scheduler 36 determines the number of sheets to be checked and corrected by dividing the time from the timing to the completion of the current learning by the held correction time per image.

The correction unit 33 also specifies _{, for} each category, teacher candidate images I _{k, 1} ,..., I _{k, Nk} whose scores S _{k, 1 ,.} Alternatively, a part of the specified image may be randomly selected and added to the image to be checked and corrected. The predetermined threshold is set in advance by the operator.

  As a result, an image with a high score can be used as teacher data, and bias in teacher data based on the score can be reduced. Further, it is possible to increase the possibility of correcting an image having a high score but classified into an incorrect category.

  As described above, images classified into an incorrect category with a high score are images for which the current learning model is not good at classification, and by using this as teacher data, the classification accuracy of the learning model can be efficiently improved. Can be increased.

[Learning unit 35]
Next, the learning unit 35 of the image classification device 3 shown in FIG. 1 will be described in detail. FIG. 9 is a flowchart illustrating a processing example of the learning unit 35.

  The learning unit 35 determines whether a learning start instruction has been input from the scheduler 36 (step S901). When determining that the learning start instruction has not been input in step S901 (step S901: N), the learning unit 35 waits until the learning start instruction is input. The learning start instruction is a signal by which the scheduler 36 causes the learning unit 35 to start the learning process.

On the other hand, the learning section 35, in step S901, the case where it is determined that the input learning start instruction (step S901: Y), from the storage unit 34, the teacher data I _k, 1 for each category ', · · ·, I _{k , Nk ′} are read (step S902).

The learning unit 35 sets, for each of the teacher data I _{k, 1 ′} ,..., I _{k, Nk ′} , the score of the category to which the image belongs to 1 and sets the other scores to 0. Then, a score S is generated (step S903).

The learning unit 35 learns a learning model using each of the teacher data I _{k, 1 ′} ,..., I _{k, Nk ′} as input data and the score S for each category as correct data (step S904).

The learning unit 35 determines that the learning process using the teacher data I _{k, 1 ′} ,..., I _{k, Nk ′} for each category that has been confirmed and corrected by the correction unit 33 is completed, and the learning input in step S901. The completion of learning corresponding to the start instruction is output to the scheduler 36 (step S905). The learning completion is a signal indicating that the learning processing by the learning unit 35 has been completed.

As described above, the learning unit 35 performs learning using the teacher data I _{k, 1 ′} ,..., I _{k, Nk ′} for each category according to the learning start instruction, updates the learning model, and completes the learning. Output.

Note that the learning unit 35 adds the teacher data I _{k, 1 ′} ,..., I _{k, Nk ′} for each category read from the storage unit 34 and the teachers used in the learning so far in steps S903 and S904. The data may also be used as teacher data to be learned.

  As a result, learning of the learning model is performed using the teacher data used for the past learning as the teacher data for the current learning. If the teacher data used for the past learning is not used as the teacher data for the current learning, the classification accuracy of the image of the teacher data decreases. Therefore, by including the teacher data used in the past learning in the teacher data in the current learning, the classification accuracy of the image can be prevented from being reduced.

  In other words, it can be said that including teacher data used for past learning in teacher data for current learning is useful teacher data in that the classification accuracy of the image is not reduced. Thereby, the classification accuracy of the learning model can be efficiently increased.

  The learning unit 35 may select the teacher data based on the usage rate setting value for each teacher data set in advance by the operator. For example, the learning unit 35 thins out (excludes) the teacher data with respect to the teacher data with the usage rate set value of 50% in one of the two learning processes. As a result, teacher data used for learning can be thinned out according to the usage rate setting value, and an increase in learning time due to an increase in teacher data can be reduced.

[Scheduler 36]
Next, the scheduler 36 of the image classification device 3 shown in FIG. 1 will be described in detail. FIG. 10 is a diagram for explaining an example of parallel processing by the scheduler 36, and it is assumed that time elapses downward. As described above, the scheduler 36 totally controls the operations of the image collection unit 20, the preprocessing unit 22, the image classification unit 30, the correction unit 33, and the learning unit 35, and instructs the timing of the processing to perform these processing. Are performed in parallel.

  Referring to FIG. 10, first, in time zone T1, image collection unit 20 and preprocessing unit 22 perform processing of teacher candidate image A1, and at this time, learning unit 35 uses teacher data D1. It is assumed that a process for learning a learning model is performed.

  When the processing of the teacher data D1 by the learning unit 35 is completed, and the processing of the teacher candidate image A1 by the image collection unit 20 and the preprocessing unit 22 is completed, in the time zone T2, the image classifying unit 30 , A process of generating the classification result B1 using the learning model. In the time zones T2 and T3, the image collection unit 20 and the preprocessing unit 22 perform processing of the next teacher candidate image A2.

  In this case, the process using the learning model by the image classifying unit 30 and the process of learning the learning model by the learning unit 35 cannot be performed simultaneously and in parallel. This is because the use and learning cannot be performed simultaneously for one learning model. That is, the process by the image classifying unit 30 is performed after the completion of the process by the learning unit 35, and the process by the learning unit 35 is performed after the process by the image classifying unit 30 is completed.

  When the process of generating the classification result B1 using the learning model by the image classification unit 30 is completed, in the time zones T3 and T4, the correction unit 33 performs a process of generating the teacher data C1 based on the classification result B1. In the time zone T3, the learning unit 35 performs a process of learning a learning model using the teacher data D2.

  When the processing of the teacher data D2 by the learning unit 35 is completed, and the processing of the teacher candidate image A2 by the image collection unit 20 and the preprocessing unit 22 is completed, in the time zone T4, the image classifying unit 30 , A process of generating the classification result B2 using the learning model. In the time zones T4 and T5, the image collection unit 20 and the preprocessing unit 22 perform processing of the next teacher candidate image A3.

  When the process of generating the teacher data C1 by the correction unit 33 is completed and the process of generating the classification result B2 using the learning model by the image classification unit 30 is completed, in the time zones T5 and T6, the correction unit 33 A process of generating teacher data C2 based on B2 is performed. In the time zone T5, the learning unit 35 performs a process of learning a learning model using the teacher data C1.

  When the processing of the teacher data C1 by the learning unit 35 is completed and the processing of the teacher candidate image A3 by the image collection unit 20 and the preprocessing unit 22 are completed, in the time zone T6, the image classifying unit 30 , A process of generating the classification result B3 using the learning model. In the time zones T6 and T7, the image collection unit 20 and the preprocessing unit 22 perform processing of the next teacher candidate image A4.

  When the process of generating the teacher data C2 by the correction unit 33 is completed and the process of generating the classification result B3 using the learning model by the image classification unit 30 is completed, in the time zones T7 and T8, the correction unit 33 outputs the classification result. A process of generating teacher data C3 based on B3 is performed. In the time period T7, the learning unit 35 performs a process of learning a learning model using the teacher data C2.

As described above, when the processing of the teacher candidate image is completed, the image collection unit 20 and the preprocessing unit 22 perform the processing of the next teacher candidate image. Then, the image classification unit 30 performs the processing after the completion of the processing of the image collection unit 20 and the preprocessing unit 22, and the correction unit 33 performs the processing after the completion of the processing of the image classification unit 30. The processing 35 waits for the completion of the processing of the correction unit 33 and performs the processing.
In this case, since the image classification unit 30 and the learning unit 35 access the same learning model, they do not operate at the same time (see the hatched portions in FIG. 10).

  FIG. 11 is a flowchart illustrating an example of control of the image collection unit 20 and the preprocessing unit 22 by the scheduler 36. The scheduler 36 outputs a collection start instruction to the image collection unit 20 (Step S1101). The acquisition start instruction is a signal by which the scheduler 36 causes the image acquisition unit 20 to start an acquisition process.

  As a result, the image collection unit 20 collects the teacher candidate images, and then the preprocessing unit 22 performs preprocessing of the teacher candidate images. When completing the preprocessing of the teacher candidate image, the preprocessing unit 22 outputs the collection and the preprocessing completion to the scheduler 36. Alternatively, when completing the collection of the teacher candidate images, the image collection unit 20 outputs the completion of the collection to the scheduler 36, and when completing the preprocessing of the teacher candidate images, the preprocessing unit 22 outputs the completion of the preprocessing to the scheduler 36. I do.

  The scheduler 36 determines whether or not the collection and the preprocessing completion are input from the preprocessing unit 22 (or whether or not the collection completion is input from the image collection unit 20 and the preprocessing completion is input from the preprocessing unit 22). Is determined (step S1102).

  If the scheduler 36 determines in step S1102 that collection and preprocessing completion have been input (step S1102: Y), the process proceeds to step S1103. On the other hand, if the scheduler 36 determines in step S1102 that collection and preprocessing completion have not been input (step S1102: N), the scheduler 36 waits until collection and preprocessing completion is input.

  The scheduler 36 determines whether to end the control of the image collection unit 20 and the preprocessing unit 22 by the scheduler 36 (whether or not a predetermined end condition is satisfied) (step S1103). If the scheduler 36 determines in step S1103 that the control is not to be ended (step S1103: N), the process proceeds to step S1101, and outputs the next acquisition start instruction to the image acquisition unit 20.

  Thus, the image collection unit 20 collects the next teacher candidate image, and then the pre-processing unit 22 performs pre-processing of the next teacher candidate image.

  On the other hand, when the scheduler 36 determines in step S1103 to end the control (step S1103: Y), the control ends.

  FIG. 12 is a flowchart illustrating an example of control of the image classification unit 30 by the scheduler 36. The scheduler 36 determines whether the collection and preprocessing of the teacher candidate images by the image collection unit 20 and the preprocessing unit 22 have been completed (step S1201). Further, the scheduler 36 determines whether or not the learning of the learning model using the teacher data by the learning unit 35 has been completed (step S1202).

  When the scheduler 36 determines that the collection and the pre-processing have not been completed in Step S1201 or that the learning has not been completed in Step S1202 (Step S1201: N or Step S1202: N), the scheduler 36 waits until it is completed.

  On the other hand, if the scheduler 36 determines that the collection and the pre-processing have been completed in step S1201 and the learning has been completed in step S1202 (step S1201: Y and step S1202: Y), the scheduler 36 issues a classification start instruction. The image is output to the image classifying unit 30 (step S1203).

  Thereby, the image classification unit 30 classifies the teacher candidate images using the learning model. Then, when the classification of the teacher candidate images is completed, the image classification unit 30 outputs the classification completion to the scheduler 36.

  The scheduler 36 determines whether classification completion has been input from the image classification unit 30 (step S1204).

  If the scheduler 36 determines in step S1204 that classification completion has been input (step S1204: Y), the process proceeds to step S1205. On the other hand, if the scheduler 36 determines in step S1204 that classification completion has not been input (step S1204: N), the scheduler 36 waits until classification completion is input.

  The scheduler 36 determines whether to end the control of the image classification unit 30 by the scheduler 36 (whether or not a predetermined end condition is satisfied) (step S1205). If the scheduler 36 determines in step S1205 that the control is not to be ended (step S1205: N), the process proceeds to step S1201, and determines whether the condition for outputting the next classification start instruction is satisfied.

  On the other hand, when the scheduler 36 determines in step S1205 to end the control (step S1205: Y), the control ends.

  FIG. 13 is a flowchart illustrating an example of control of the correction unit 33 by the scheduler 36. The scheduler 36 determines whether the classification processing of the teacher candidate images by the image classification unit 30 has been completed (step S1301).

  If the scheduler 36 determines in step S1301 that the classification process has not been completed (step S1301: N), the scheduler 36 waits until it is completed.

  On the other hand, when determining in step S1301 that the classification process has been completed (step S1301: Y), the scheduler 36 outputs a correction start instruction to the correction unit 33 (step S1302).

  As a result, the correction unit 33 performs a correction process using the classification result. When the correction unit 33 completes the correction processing and generates the teacher data, the correction unit 33 outputs the correction completion to the scheduler 36.

  The scheduler 36 determines whether or not correction completion has been input from the correction unit 33 (step S1303).

  If the scheduler 36 determines in step S1303 that correction completion has been input (step S1303: Y), the process proceeds to step S1304. On the other hand, if the scheduler 36 determines in step S1303 that correction completion has not been input (step S1303: N), the scheduler 36 waits until correction completion is input.

  The scheduler 36 determines whether to end the control of the correction unit 33 by the scheduler 36 (whether or not a predetermined end condition is satisfied) (step S1304). If it is determined in step S1304 that the control is not to be ended (step S1304: N), the scheduler 36 proceeds to step S1301, and determines whether or not the condition for outputting the next correction start instruction is satisfied.

  On the other hand, when the scheduler 36 determines in step S1304 to end the control (step S1304: Y), the control ends.

  FIG. 14 is a flowchart illustrating an example of control of the learning unit 35 by the scheduler 36. The scheduler 36 determines whether or not the correction of the classification result by the correction unit 33 has been completed (generation of teacher data has been completed) (step S1401). Further, the scheduler 36 determines whether the classification using the teacher data by the image classification unit 30 has been completed (step S1402).

  If the scheduler 36 determines that the correction has not been completed in step S1401 or that the classification has not been completed in step S1402 (step S1401: N or step S1402: N), the scheduler 36 waits until the correction is completed.

  On the other hand, if the scheduler 36 determines that the modification has been completed in step S1401 and the classification has been completed in step S1402 (step S1401: Y and step S1402: Y), the learning unit 35 issues a learning start instruction. (Step S1403).

  Thus, the learning unit 35 learns the learning model using the teacher data. When the learning unit 35 completes the learning, the learning unit 35 outputs the learning completion to the scheduler 36.

  The scheduler 36 determines whether learning completion has been input from the learning unit 35 (step S1404).

  If the scheduler 36 determines in step S1404 that learning completion has been input (step S1404: Y), the process proceeds to step S1405. On the other hand, if it is determined in step S1404 that learning completion has not been input (step S1404: N), the scheduler 36 waits until learning completion is input.

  The scheduler 36 determines whether to end the control of the learning unit 35 by the scheduler 36 (whether or not a predetermined end condition is satisfied) (step S1405). If it is determined in step S1405 that the control is not to be ended (step S1405: N), the scheduler 36 proceeds to step S1401, and determines whether the condition for outputting the next learning start instruction is satisfied.

  On the other hand, when the scheduler 36 determines in step S1405 to end the control (step S1405: Y), the control ends.

  As described above, the scheduler 36 controls the respective operations of the image collection unit 20, the preprocessing unit 22, the image classification unit 30, the correction unit 33, and the learning unit 35, and causes these processes to be performed in parallel.

Thus, it is possible to shorten the overall processing time, the teacher candidate image I ₁ of the time (the image acquisition unit 20 of the N sheets per cycle, ..., learning unit 35 collects I _N learning The processing time until the model is learned can be reduced.

As described above, according to the image classification device 3 of the embodiment of the present invention, the image classifying unit 30, the teacher candidate image I _1, · · ·, for each of I _N, scores for each category using a learning model And classify it into the category with the highest score. Then, the image classification section 30 obtains for each category, the teacher candidate image I ₁ to a low score order, ..., by sorting the I _N, the image I _{k, 1,} ..., I k, the _Nk . The image classification unit 30 generates teacher candidate images I _{k, 1} ,..., I _{k, Nk} and scores S _{k, 1 ,.}

The correction unit 33 prompts the operator to confirm in order from the teacher candidate images I _{k, 1} ,..., I _{k, Nk having} the lower scores S _{k, 1 ,.} According to the operation of the operator, the category is corrected as necessary, and the teacher candidate images I _{k, 1 ′} ,..., I _{k, Nk ′} for each category after confirmation and correction are used as teacher data. The learning unit 35 learns a learning model using teacher data for each category.

  The scheduler 36 totally controls the operations of the image collection unit 20, the preprocessing unit 22, the image classification unit 30, the correction unit 33, and the learning unit 35, and causes these processes to be performed in parallel.

  As a result, the classification result using the learning model is corrected by the operator, and teacher data used for learning is generated. Therefore, useful teacher data is collected as compared with a case where teacher data is collected manually. Labor and time can be reduced.

  Also, under the control of the scheduler 36, image collection and pre-processing, classification processing using a learning model, correction processing, and learning processing of a learning model are performed in parallel, so that the overall processing time is reduced, Time per cycle can be reduced.

  Generally, as the teacher data for deep learning, thousands to tens of thousands of images are often required for each category. In the embodiment of the present invention, it is sufficient to prepare several hundred images for each category for the initial learning, and thereafter, collect the teacher data until the required amount is reached while reducing the processing effort and time. be able to.

  As described above, the present invention has been described with reference to the embodiment. However, the present invention is not limited to the above embodiment, and can be variously modified without departing from the technical idea thereof. In the above-described embodiment, the image classification device 3 includes the scheduler 36, but may not include the scheduler 36.

  Note that an ordinary computer can be used as a hardware configuration of the image classification device 3 according to the embodiment of the present invention. The image classification device 3 is configured by a computer including a CPU, a volatile storage medium such as a RAM, a non-volatile storage medium such as a ROM, and an interface.

  Each function of the image classification unit 30, the storage unit 31, the storage unit 32, the correction unit 33, the storage unit 34, the learning unit 35, and the scheduler 36 provided in the image classification device 3 executes a program describing these functions to the CPU. To be realized.

  These programs are stored in the storage medium, and are read and executed by the CPU. These programs can also be stored in a storage medium such as a magnetic disk (floppy (registered trademark) disk, hard disk, etc.), an optical disk (CD-ROM, DVD, etc.), a semiconductor memory or the like, and distributed via a network. You can also send and receive.

  The image classification device 3 according to the embodiment of the present invention is useful for analyzing situations using images, detecting abnormalities using images, organizing information using images, and the like.

REFERENCE SIGNS LIST 1 storage device 2 image collection device 3 image classification device 4 transmission path 20 image collection units 21, 31, 32, 34 storage unit 22 preprocessing unit 30 image classification unit 33 correction unit 35 learning unit 36 scheduler

Claims (4)

In an image classification device that learns a learning model for classifying images,
For each of the collected plurality of teacher candidate images, a score for each category is estimated using the learning model, the category is classified into the category having the highest score, and for each category, the plurality of teacher candidates are sorted in ascending order of the score. An image classification unit that sorts images and generates a classification result for each category;
For the teacher candidate image of the classification result generated by the image classifying unit, for each category, the operator is urged to confirm in the order of the score, the category is corrected according to the operation of the operator, and the teacher candidate for each category is corrected. A correction unit that generates an image as teacher data,
A learning unit that learns the learning model using the teacher data for each category generated by the correction unit;
An image classification device comprising: The image classification device according to claim 1,
In addition, it has a scheduler,
The image classification unit,
From the image collection device that collects the plurality of teacher candidate images, input the plurality of teacher candidate images,
The scheduler comprises:
A collection process of collecting the plurality of teacher candidate images by the image collection device; a classification process of generating the classification result by the image classification unit; a correction process of generating the teacher data by the correction unit; To control the timing of each learning process for learning a learning model, and to start the classification process so that the classification process by the image classification unit and the learning process by the learning unit are not performed simultaneously. The classification start instruction is output to the image classification unit, and a learning start instruction for starting the learning process is output to the learning unit. The image classification device according to claim 2,
The scheduler comprises:
A collection start instruction for starting the collection processing is output to the image collection apparatus, and when a collection completion indicating that the collection processing is completed is input from the image collection apparatus, it is determined that the collection processing is completed. ,
If the collection process by the image collection device has been completed, and the learning process by the learning unit has been completed, the classification start instruction is output to the image classification unit, and the classification is performed by the image classification unit. When inputting classification completion indicating that the processing is completed, it is determined that the classification processing is completed,
When the classification process is completed, a correction start instruction for starting the correction process is output to the correction unit, and when a correction completion indicating that the correction process is completed is input from the correction unit, the correction is performed. Judge that the process is completed,
When the correction processing by the correction unit is completed and the classification processing by the image classification unit is completed, the learning start instruction is output to the learning unit, and the learning processing is performed by the learning unit. An image classification device, wherein when learning completion indicating completion is input, it is determined that the learning processing is completed.   A program for causing a computer to function as the image classification device according to any one of claims 1 to 3.

Images