JP2021002270A - Image recognition learning device, image recognition learning method, image recognition learning program and terminal device - Google Patents

Abstract

To construct a multi-layer neutral network having an appropriate recognition ratio.SOLUTION: An image recognition learning device includes: an image acquisition unit 31 that acquires an image provided with a label indicating a first category and an image provided with a label indicating a second category; a reference image generation unit 20 that generates a reference image based on the image provided with the label indicating the first category; a difference image generation unit 33 that generates a first difference image calculated from a difference between the image provided with the label indicating the first category and the reference image, and a second difference image calculated from a difference between the image provided with the label indicating the second category and the reference image; and a learning unit 35 that generates a learned model obtained by learning weighting of a multi-layer neural network that identifies the first category and the second category, by using the first difference image and the second difference image.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image recognition learning device, an image recognition learning method, an image recognition learning program, and a terminal device.

As a method of visual inspection of a product, a method of determining the presence or absence of an abnormality by image recognition is known (see, for example, Patent Documents 1 to 3).

Japanese Unexamined Patent Publication No. 2006-047040 JP-A-2018-173814 Japanese Unexamined Patent Publication No. 10-289317

Patent Document 1 extracts features from an inspection target area of an inspection target and executes learning. In Patent Document 1, it is necessary to extract features by repeating trial and error for each part or product. In Patent Document 2, image recognition is performed using the teacher data to be recognized, but other than the essential features are also learned at the same time. As a result, a very large amount of teacher data is required, and it may be difficult to prepare enough teacher data to withstand practical use. Further, Patent Document 2 removes non-essential features by masking non-features. However, it is necessary to use the feature amount to discriminate other than the essential features. Patent Document 3 adjusts the pixel density by comparing a predetermined comparison area with an input image and a reference image. As a result, it is necessary to store a large amount of reference images in advance.

Deep learning using a multi-layer neural network generally requires a very large amount of teacher data, and it may be difficult to prepare enough teacher data to withstand practical use. Insufficient amount of teacher data makes it difficult to obtain an appropriate recognition rate. As described above, in deep learning using a multi-layer neutral network, there is room for improvement in improving the recognition rate.

The present invention has been made in view of the above, and an object of the present invention is to construct a multi-layer neutral network having an appropriate recognition rate.

In order to solve the above-mentioned problems and achieve the object, the image recognition learning device according to the present invention acquires an image with a label indicating the first classification and an image with a label indicating the second classification. The difference between the image acquisition unit, the reference image generation unit that generates a reference image based on the image with the label indicating the first classification, and the image with the label indicating the first classification and the reference image. A difference image generation unit that generates a second difference image calculated from the difference between the first difference image calculated from the above and the image with the label indicating the second classification and the reference image, and the first difference image. A learning unit that generates a trained model in which the weighting of the multi-layer neural network that distinguishes the first classification and the second classification is trained by using the difference image of the above and the second difference image. To be equipped.

The image recognition learning method according to the present invention includes an image acquisition step of acquiring an image with a label indicating the first classification and an image with a label indicating the second classification, and a label indicating the first classification. The first difference image calculated from the difference between the reference image generation step of generating the reference image based on the image to which the first classification is given, the image to which the label indicating the first classification is given, and the reference image, and the second Using the difference image generation step of generating the second difference image calculated from the difference between the image with the label indicating the classification and the reference image, and the first difference image and the second difference image. , A learning step of generating a trained model trained in the weighting of a multi-layer neural network that distinguishes the first classification from the second classification.

The image recognition learning program according to the present invention includes an image acquisition step of acquiring an image with a label indicating the first classification and an image with a label indicating the second classification, and a label indicating the first classification. The first difference image calculated from the difference between the reference image generation step of generating the reference image based on the image to which the first classification is given, the image to which the label indicating the first classification is given, and the reference image, and the second Using the difference image generation step of generating the second difference image calculated from the difference between the image with the label indicating the classification and the reference image, and the first difference image and the second difference image. , A learning step of generating a trained model trained in the weighting of a multi-layer neural network that distinguishes the first classification from the second classification.

The terminal device according to the present invention includes an image acquisition unit that acquires a specific image, a reference image acquisition unit that acquires a reference image based on an image with a label indicating the first classification, the specific image, and the reference. A difference image generation unit that generates a third difference image calculated from the difference between the images, and a first difference image calculated from the difference between the image with the label indicating the first classification and the reference image. , A multi-layer neural that discriminates between the first classification and the second classification by using a second difference image calculated from the difference between the image labeled indicating the second classification and the reference image. By inputting the third difference image into the trained model and the trained model acquisition unit that acquires the trained model in which the weighting of the network is trained, the specific image acquired by the image acquisition unit can be obtained. It is provided with a data inference unit that distinguishes between the first classification and the second classification.

According to the present invention, there is an effect that a multi-layer neutral network having an appropriate recognition rate can be constructed.

FIG. 1 is a schematic view showing an example of a configuration example of the image processing system according to the first embodiment. FIG. 2 is a diagram illustrating an image set. FIG. 3 is a schematic view showing a configuration example of the learning unit according to the first embodiment. FIG. 4 is a diagram illustrating a composite abnormality image according to the first embodiment. FIG. 5 is a schematic view showing a configuration example of the inference unit according to the first embodiment. FIG. 6 is a flowchart showing a processing flow in the image processing system according to the first embodiment. FIG. 7 is a flowchart showing a processing flow in the learning unit according to the first embodiment. FIG. 8 is a flowchart showing a processing flow in the inference unit according to the first embodiment. FIG. 9 is a schematic view showing an example of a configuration example of the image processing system according to the second embodiment. FIG. 10 is a schematic view showing an example of a configuration example of the image processing system according to the third embodiment.

Hereinafter, embodiments of the image recognition learning device, the image recognition learning method, the image recognition learning program, and the terminal device according to the present invention will be described in detail with reference to the accompanying drawings. The present invention is not limited to the following embodiments.

[First Embodiment]
FIG. 1 is a schematic view showing an example of a configuration example of the image processing system according to the first embodiment. The image processing system 1 executes image recognition by a multi-layer neural network (hereinafter referred to as “DNN”) trained by deep learning of supervised learning using teacher data.

Deep learning is a machine learning method that repeatedly updates the DNN weights so that the accuracy of inference using DNN is improved, and derives the DNN weights so that the inference result of DNN is good. is there.

The image processing system 1 is used, for example, when detecting an abnormality such as a scratch on the surface of a product. The image processing system 1 includes a recording unit 10, a reference image generation unit 20, a learning unit 30, an inference unit 40, and a display unit 50. The configuration including the recording unit 10, the reference image generation unit 20, and the learning unit 30 is referred to as an image recognition learning device 2. The configuration including the recording unit 10, the reference image generation unit 20, and the inference unit 40 is referred to as an image recognition device 3.

The recording unit 10 records an image set which is teacher data used in the image processing system 1. The recording unit 10 records a training (training) image set, a confirmation (validation) image set, and a test (test) image set. Each image set of training image set, confirmation image set, and test image set is an image set classification method generally used in deep learning. The training image set is used for learning. The confirmation image set evaluates the generalization performance, receives the evaluation result, and uses it for readjusting the parameters of the classifier of the trained model. Generalization performance is the ability to correctly predict class labels and function values not only for training data given at the time of training but also for unknown new data. The test image set evaluates the inference result using DNN. Each image set contains a combination of images and their classification type (label). The recording unit 10 is, for example, a semiconductor memory element such as a flash memory or a storage device such as a hard disk or an optical disk. In addition to the image set that is the teacher data, the recording unit 10 can also acquire an image from the outside.

The image set will be described with reference to FIG. FIG. 2 is a diagram illustrating an image set. In the image set, in the present embodiment, images are classified into five normal classification types that do not correspond to scratches, stains, color unevenness, and deformations. Images are classified by giving the images labels indicating their respective classification types. The number of images of each classification type of normal, scratch, stain, color unevenness, and deformation is 100, 5, 5, 5, 5, respectively. In the present embodiment, the number of images of each classification type of normal, scratch, stain, color unevenness, and deformation is 100, 5, 5, 5, 5, respectively, but is not limited thereto. The number of images of each classification type of normal, scratch, stain, color unevenness, and deformation may be more than or less than 100. An image having a normal classification type is defined as a normal image 100 (see FIG. 4), and an image having a classification type of scratches, stains, color unevenness, or deformation is defined as an abnormal image 110 (see FIG. 4). In other words, the image to which the label indicating the first classification is given is defined as the normal image 100. An image to which a label indicating the second classification is given is referred to as an abnormal image 110. In the present embodiment, there are five classification types, but the classification type is not limited to this. As the classification type, for example, scratches, stains, color unevenness, and deformation may be collectively set as one classification type (for example, abnormal). The classification type may be, for example, at least one or more of scratches, stains, color unevenness, and deformation. The classification type may be 6 or more.

In the present embodiment, the number of images of each classification type belonging to each image set of the training image set, the confirmation image set, and the test image set is the same. The images of each classification type belonging to each image set of the training image set, the confirmation image set, and the test image set are all different images. In the present embodiment, it is assumed that the number of images of each classification type of the training image set, the confirmation image set, and the test image set is the same, but if there is one or more images of each classification type, each is The number of images for each classification type in the image set does not have to be the same.

The reference image generation unit 20 generates a reference image based on the normal image 100 as an image with a label indicating the first classification. The reference image generation unit 20 generates a reference image which is an average image of all the normal images 100 included in the training image set recorded in the recording unit 10, and records the reference image in the recording unit 10. In the present embodiment, the reference image generation unit 20 generates a reference image by averaging each of the RGB of all the normal images 100 included in the training image set in pixel units. Alternatively, the reference image generation unit 20 may generate a reference image based on the average of each pixel for each YCbCr separated into a luminance component and a color difference component. Alternatively, when the color component of the reference image is unnecessary, the reference image generation unit 20 may convert the normal image 100 to grayscale and generate the reference image from the image after the conversion to grayscale.

In the present embodiment, the reference image is generated from the training image set, but it may be generated from the confirmation image set, or may be generated from both the training image set and the confirmation image set.

In the present embodiment, the reference image is the average image of all the normal images 100, but it may be an average image of one or more normal images 100.

Although the reference image generation unit 20 has been described as being separate from the learning unit 30, it may be included in the learning unit 30.

The learning unit 30 uses the first difference image and the second difference image to learn the weighting of the multi-layer neural network that distinguishes between the first classification and the second classification (hereinafter, trained model). "DNN model") is generated. The learning unit 30 uses the training image set and the confirmation image set to train the DNN model by deep learning as a classifier that classifies the DNN model into five classification types. The learning unit 30 stores the learned DNN model in the recording unit 10. The learning unit 30 outputs the loss value and the correct answer rate obtained in the middle of learning to the display unit 50.

The DNN model is composed of parameters necessary for reproducing the learned DNN, and includes at least the structure and weight of the DNN. The configuration of the DNN model has at least one convolutional layer, a pooling layer, and a fully connected layer as hidden layers between the input layer and the output layer. As a configuration of the DNN model, for example, as a hidden layer, there is a LeNET in which three fully connected layers are lined up after convolution and pooling are repeated twice.

The learning unit 30 will be described in detail with reference to FIG. FIG. 3 is a schematic view showing a configuration example of the learning unit according to the first embodiment. The learning unit 30 includes an image acquisition unit 31, an abnormal image generation unit 32, a difference image generation unit 33, a deformed image generation unit 34, a DNN learning unit 35, and a learned model storage unit 36.

The image acquisition unit 31 acquires an image set from the recording unit 10. The image acquisition unit 31 acquires a normal image 100 as an image with a label indicating the first classification and an abnormal image 110 as an image with a label indicating the second classification. In the present embodiment, the image acquisition unit 31 acquires the normal image 100 or the abnormal image 110 of the image set from the recording unit 10. The image acquisition unit 31 outputs the acquired normal image 100 or abnormal image 110 to the abnormal image generation unit 32.

The abnormal image generation unit 32 will be described with reference to FIG. FIG. 4 is a diagram illustrating a composite abnormality image according to the first embodiment. The abnormal image generation unit 32 cuts out an area including a specific target from the abnormal image 110 to which the label indicating the second classification is attached, and superimposes and synthesizes the normal image 100 to which the label indicating the first classification is attached. Anomalous image 120 is generated. In the present embodiment, the area including the specific target is, for example, the abnormal area A1 including scratches, stains, color unevenness, deformation, and the like. The abnormal image generation unit 32 superimposes the abnormal area A1 on the normal image 100 and synthesizes the abnormal area A1 to generate a composite abnormal image 120. The abnormal image generation unit 32 includes the generated composite abnormal image 120 in the abnormal image. The abnormal image generation unit 32 outputs the generated composite abnormal image 120 to the difference image generation unit 33.

More specifically, the abnormal image generation unit 32 cuts out the abnormal area A1 on the known abnormal image 110 and changes the position in the same position on the normal image 100 and the composite area A2 for synthesizing the abnormal area A1. In other words, while moving, the abnormal area A1 is combined to generate the combined abnormal image 120. The abnormal image generation unit 32 may rotate the abnormal area A1 on the known abnormal image 110 and synthesize it with the composite area A2 to generate the composite abnormal image 120. The abnormal image generation unit 32 may enlarge or reduce the abnormal area A1 on the known abnormal image 110 and combine it with the composite area A2 to generate the composite abnormal image 120. The abnormal image generation unit 32 may generate the composite abnormal image 120 by increasing or lowering the brightness level of the abnormal area A1 on the known abnormal image 110 and combining it with the composite area A2. Here, processing such as movement, rotation, enlargement, reduction, or change of brightness level may be executed individually to generate the composite abnormal image 120, and movement, rotation, enlargement, reduction, or change of brightness level may be generated. The composite abnormal image 120 may be generated by combining a plurality of processes such as.

The abnormal area A1 is an area in which abnormalities such as scratches, stains, color unevenness, and deformation are captured in the known abnormal image 110. In the abnormal area A1, the area to be cut out is designated in advance by the user. When the feature extraction process is possible for the abnormal image 110, the abnormal area A1 may be cut out by the feature extraction process.

The composite area A2 may be set individually for each classification type of the abnormal image 110. More specifically, the synthetic area A2 may be set by limiting the area where the abnormality can occur for each type of abnormality such as scratches, stains, color unevenness, and deformation. As a result, it is possible to suppress the generation of an inappropriate composite abnormality image 120.

It is preferable that the number of normal images 100 to be combined with the abnormal area A1 is 1 or more.

The composite abnormality image 120 is generated, for example, 20 for each image whose classification type is scratch, stain, color unevenness, or deformation. In this embodiment, it is assumed that 20 are generated at a time, but it is desirable that the number is 1 or more. However, when increasing the number of images of the composite abnormal image 120, it is desirable to increase the number of images of the normal image 100 as well.

Returning to FIG. 3, the difference image generation unit 33 includes a normal difference image as a first difference image calculated from the difference between the normal image 100 and the reference image with a label indicating the first classification, and a first. An abnormal difference image as a second difference image calculated from the difference between the abnormal image 110 to which the label indicating the classification of 2 is given and the reference image is generated. The difference image generation unit 33 includes the composite abnormal image 120 generated by the abnormal image generation unit 32 in the abnormal image 110 with a label indicating the second classification. In other words, the difference image generation unit 33 includes the difference image between the composite abnormality image 120 and the reference image in the abnormality difference image and generates it. The difference image generation unit 33 outputs the generated normal difference image and the abnormal difference image to the modified image generation unit 34.

In the present embodiment, the difference image generation unit 33 generates a difference image based on the absolute difference in pixel units for each RGB. Alternatively, the difference image generation unit 33 may generate a difference image for each YCbCr based on the absolute difference in pixel units. Further, or when the color component of the difference image is unnecessary, the difference image generation unit 33 converts the normal image 100, the abnormal image 110, and the reference image into grayscale, and the difference image is obtained for the image after the conversion to grayscale. May be generated.

The deformed image generation unit 34 generates a normal deformed image as a new normal difference image and an abnormally deformed image as a new abnormal difference image for the training image set. More specifically, the modified image generation unit 34 moves, rotates, enlarges, reduces, changes the brightness level, etc. of the normal difference image as the first difference image and the abnormal difference image as the second difference image, respectively. It is transformed to generate a new normal difference image and an abnormal difference image. Affine transformation is used for deformation processing such as movement, rotation, enlargement, or reduction. The movement, rotation, enlargement, reduction, or change of the brightness level may be individually transformed, or a plurality of movement, rotation, enlargement, reduction, or change of the brightness level may be combined and transformed. In this way, a large amount of normal deformed images are generated from the normal difference images, and a large amount of abnormally deformed images are generated from the abnormal difference images. The deformed image generation unit 34 outputs the generated normal deformed image and the abnormally deformed image to the DNN learning unit 35.

The DNN learning unit 35 uses a normal difference image including a normal deformed image generated by the deformed image generation unit 34 and an abnormal difference image including an abnormal deformed image generated by the deformed image generation unit 34 for the training image set. Learn DNN weights as a model for recognizing classification types. The learning in the DNN learning unit 35 may be performed in the same manner as the learning in the known deep learning. For example, the DNN learning unit 35 inputs a normal difference image having a classification type of "normal" into the DNN and executes deep learning learning. Then, when an erroneous inference result is output when the classification type is "abnormal", the weight during learning is updated based on the error back propagation method. Further, the DNN learning unit 35 inputs an abnormality difference image whose classification type is "abnormal" to the DNN and executes deep learning learning. Then, when an erroneous inference result is output when the classification type is "normal", the weight during learning is updated based on the error back propagation method. By repeating the process called epoch in this way, the learned weight is obtained as a result of learning. Repeat the epoch until the desired loss value and correct answer rate are obtained. The DNN learning unit 35 outputs a DNN model including the weight of the DNN to the trained model storage unit 36.

The DNN learning unit 35 infers the classification type of the normal difference image and the abnormal difference image for the confirmation image set. The inference in the DNN learning unit 35 may be executed in the same manner as the inference in the known deep learning. For example, the DNN learning unit 35 inputs a normal difference image having a classification type of “normal” into the DNN and confirms the generalization performance of the learned DNN. Further, the DNN learning unit 35 inputs an abnormality difference image whose classification type is "abnormal" into the DNN and confirms the generalization performance of the learned DNN. By repeating such processing for each epoch, the DNN learning unit 35 confirms the generalization performance. If the generalized performance of the trained model storage unit is not sufficient, it is necessary to review the DNN model and the training image set.

The learned model storage unit 36 records the DNN model output from the DNN learning unit 35 in the recording unit 10.

The inference unit 40 will be described with reference to FIG. FIG. 5 is a schematic view showing a configuration example of the inference unit according to the first embodiment. The inference unit 40 executes image recognition using DNN and infers. More specifically, the inference unit 40 reads the DNN model learned from the recording unit 10, reproduces the DNN, classifies the classification type of the test image set, and outputs the inference result to the display unit 50. The inference unit 40 includes an image acquisition unit 42, a difference image generation unit 43, a data inference unit 44, and a trained model acquisition unit 46.

The trained model acquisition unit 46 acquires the DNN model recorded in the recording unit 10. The trained model acquisition unit 46 outputs the acquired DNN model to the data inference unit 44.

The image acquisition unit 42 acquires the normal image 100 and the abnormal image 110 of the test image set from the recording unit 10 as evaluation images. The image acquisition unit 42 outputs the evaluation image to the difference image generation unit 43.

The difference image generation unit 43 generates an evaluation difference image which is a difference image between the evaluation image and the reference image. More specifically, the difference image generation unit 43 evaluates the normal difference image between the normal image 100 and the reference image in the evaluation image and the abnormal difference image between the abnormal image 110 and the reference image in the evaluation image. Generate as. The difference image generation unit 43 outputs the generated evaluation difference image to the data inference unit 44.

The data inference unit 44 infers the classification type for the evaluation difference image. The inference in the data inference unit 44 may be executed in the same manner as the inference in known deep learning. The data inference unit 44 uses the normal difference image and the abnormal difference image to infer with a DNN that recognizes whether the image is normal or abnormal. By such processing, the data inference unit 44 evaluates the inference result using DNN. The data inference unit 44 outputs the inference result to the display unit 50.

The display unit 50 is, for example, a display including a liquid crystal display (LCD: Liquid Crystal Display) or an organic EL (Organic Electro-Luminence) display. The display unit 50 displays the progress of learning of the learning unit 30 and the inference result of the inference unit 40.

Although the configuration of the image processing system 1 has been described so far, the reference image generation unit 20, the learning unit 30, the inference unit 40 are a ROM, a RAM, a CPU, a GPU, or the like, or a combination thereof.

Next, the image recognition method and operation of the image processing system will be described with reference to FIG. FIG. 6 is a flowchart showing a processing flow in the image processing system according to the first embodiment. When used as the image recognition learning device 2, the processes of steps ST1 to ST3 and step ST5 may be executed. When used as the image recognition device 3, the processes of steps ST1 to ST2 and steps ST4 to ST5 may be executed. The processes of steps ST1 to ST3 may be executed at least once when learning the DNN model.

The image processing system 1 records an image set by the recording unit 10 (step ST1). The image processing system 1 proceeds to step ST2.

The image processing system 1 generates a reference image by the reference image generation unit 20 (step ST2). The image processing system 1 proceeds to step ST3.

The image processing system 1 learns the DNN model by the learning unit 30 (step ST3). Details of the process in step ST3 will be described later. The image processing system 1 proceeds to step ST4.

The image processing system 1 infers whether it is normal or abnormal by the DNN model by the inference unit 40 (step ST4). Details of the process in step ST4 will be described later. The image processing system 1 proceeds to step ST5.

The image processing system 1 displays the inference result by the display unit 50 (step ST5). The image processing system 1 ends the processing.

The process in the learning unit 30 of step ST3 of the flowchart shown in FIG. 6 will be described with reference to FIG. 7. FIG. 7 is a flowchart showing a processing flow in the learning unit according to the first embodiment. E is a counter for the number of epochs. The number of epochs means the number of times of repeated learning using the same training image set or confirmation image set. In this embodiment, the number of epochs is 10. The process from step ST10 to step ST22 is repeated as many times as the number of epochs. S is a counter indicating the type of image set. When the counter S is 0, the training image set is shown, and when the counter S is 1, the confirmation image set is shown. The process from step ST11 to step ST21 is repeated for the number of types of image sets.

The reference image generation unit 20 generates a reference image before the learning unit 30 executes the process. Further, the image acquisition unit 31 acquires the normal image 100 of each image set from the recording unit 10.

The learning unit 30 clears the epoch number counter E to zero (step ST10). The learning unit 30 proceeds to step ST11.

The learning unit 30 clears the counter S of the image set to zero (step ST11). The learning unit 30 proceeds to step ST12.

The learning unit 30 synthesizes the abnormal area A1 with each normal image 100 by the abnormal image generation unit 32 to generate each composite abnormal image 120 (step ST12). The learning unit 30 proceeds to step ST13.

The learning unit 30 generates a normal difference image which is a difference image of each normal image 100 and a reference image by the difference image generation unit 33 (step ST13). The learning unit 30 proceeds to step ST14.

The learning unit 30 generates an abnormal difference image which is a difference image of each abnormal image 110 and a reference image by the difference image generation unit 33 (step ST14). The learning unit 30 proceeds to step ST15.

The learning unit 30 generates an abnormal difference image which is a difference image of each composite abnormal image 120 and a reference image by the difference image generation unit 33 (step ST15). The learning unit 30 proceeds to step ST16.

The learning unit 30 inspects whether the image set indicated by the counter S of the image set is a training image set by the deformed image generation unit 34 (step ST16). When the counter S of the image set is 0, the modified image generation unit 34 determines that the image set is a training image set (Yes in step ST16). When the counter S of the image set is not 0, the deformed image generation unit 34 determines that the image set is not the training image set (No in step ST16).

When it is determined that the image set indicated by the counter S of the image set is the training image set (YES in step ST16), the learning unit 30 moves, rotates, and enlarges each normal difference image by the deformed image generation unit 34. , Reduction, or change of the brightness level is added to generate a normally deformed image (step ST17). The learning unit 30 proceeds to step ST18.

The learning unit 30 generates an abnormally deformed image by adding a deforming process such as moving, rotating, enlarging, reducing, or changing the brightness level to each abnormal difference image by the deformed image generation unit 34 (step ST18). The learning unit 30 proceeds to step ST19.

The learning unit 30 learns the DNN weight as a model for recognizing the classification type by the DNN learning unit 35 using the normal difference image including each normal deformation image and the abnormality difference image including each abnormal deformation image (step). ST19). The learning unit 30 proceeds to step ST21.

In other cases where it is determined that the image set indicated by the counter S of the image set is not the training image set (NO in step ST16), the DNN learning unit 35 infers the classification type for the normal difference image and the abnormal difference image (step). ST20). Here, the inference result by the DNN learning unit 35 is sent to the display unit 50 by the inference unit 40, and the display unit 50 displays the inference result. Here, the inference result includes inference accuracy (Accuracy), precision rate (Precision), recall rate (Recall), and the like. The user of this device can check the generalization performance, and if the progress of the generalization performance is not good, stop the processing, readjust the parameters of the classifier of the trained model, and restart FIG. 7 from the beginning. Good. The learning unit 30 proceeds to step ST21.

When the counter S of the image set is less than 2, the learning unit 30 increments the counter S of the image set by +1 (step ST21), and executes the process of step ST11 again. If the counter S of the image set is not less than 2, the learning unit 30 proceeds to step ST22.

When the epoch number counter E is less than 10, the learning unit 30 increments the epoch number counter E by +1 (step ST22) and executes the process of step ST10 again. If the counter E of the number of epochs is not less than 10, the learning unit 30 proceeds to step ST23.

The learning unit 30 records the DNN model including the weight of the DNN in the recording unit 10 by the learned model storage unit 36 (step ST23). The learning unit 30 ends the process.

In FIG. 7, a composite abnormal image, a normal difference image, an abnormal difference image, a normal deformed image, and an abnormal deformed image of the training image set and the confirmation image set are generated for each epoch, but these are generated in the first epoch. They may be saved and then only read in the second and subsequent epochs, or they may be saved in advance and read from the beginning.

The process in the inference unit 40 of step ST4 of the flowchart shown in FIG. 6 will be described with reference to FIG. FIG. 8 is a flowchart showing a processing flow in the inference unit according to the first embodiment.

The inference unit 40 reads out the DNN model recorded in the recording unit 10 by the learned model acquisition unit 46 (step ST30). The inference unit 40 proceeds to step ST31.

The inference unit 40 acquires the normal image 100 and the abnormal image 110 of the test image set from the recording unit 10 as evaluation images by the image acquisition unit 42 (step ST31). The inference unit 40 proceeds to step ST32.

The inference unit 40 generates an evaluation difference image, which is a difference image between the evaluation image and the reference image, by the difference image generation unit 43 (step ST32). The inference unit 40 proceeds to step ST33.

The inference unit 40 infers the classification type for the evaluation difference image by the data inference unit 44 (step ST33). The inference unit 40 ends the process.

As described above, in the image recognition learning device 2, the reference image generation unit 20 generates a reference image from the normal image 100. The difference image generation unit 33 of the learning unit 30 generates a normal difference image between the normal image 100 and the reference image, and an abnormal difference image between the abnormal image 110 and the reference image. The DNN learning unit 35 learns the weight of the DNN so as to recognize the normal image 100 and the abnormal image 110 by using the normal difference image and the abnormal difference image.

In the image recognition device 3, the reference image generation unit 20 generates a reference image from the normal image 100. The difference image generation unit 43 of the inference unit 40 evaluates the normal difference image between the normal image 100 and the reference image in the evaluation image and the abnormal difference image between the abnormal image 110 and the reference image in the evaluation image. Generate as. The data inference unit 44 makes an inference using a DNN that recognizes whether the normal difference image and the abnormal difference image are normal or abnormal.

As described above, in the present embodiment, the reference image is generated from the normal image 100. In the present embodiment, a normal difference image between the normal image 100 and the reference image and an abnormal difference image between the abnormal image 110 and the reference image are generated. In this embodiment, the weight of the DNN can be learned so as to recognize the normal image 100 and the abnormal image 110 by using the normal difference image and the abnormal difference image. In this embodiment, during learning in the DNN learning unit 35 of the learning unit 30, only the abnormal portion can be learned by using the normal difference image and the abnormal difference image. In this embodiment, a DNN having an appropriate recognition rate can be constructed.

In the present embodiment, the abnormal image generation unit 32 of the learning unit 30 synthesizes the abnormal area A1 on the known abnormal image 110 with the normal image 100 to generate the composite abnormal image 120. At this time, the abnormal area A1 is repositioned within the composite area A2 to generate the composite abnormal image 120. The abnormal area A1 is enlarged or reduced to generate a composite abnormal image 120. The abnormal area A1 is rotated to generate a composite abnormal image 120. The composite abnormality image 120 is generated by increasing or decreasing the brightness level of the abnormality area A1. In this embodiment, a large amount of composite abnormal images 120 can be generated from a small number of known abnormal images 110. According to the present embodiment, for example, for an industrial product having a low occurrence rate of abnormality, a large amount of composite abnormal image 120 can be generated from the abnormal image 110 which is originally difficult to obtain. As described above, according to this embodiment, a DNN having an appropriate recognition rate can be constructed from a small amount of teacher data.

In the present embodiment, the abnormal image generation unit 32 of the learning unit 30 can individually set the composite area A2 for each classification type of the abnormal image 110. According to this embodiment, it is possible to suppress the generation of an inappropriate composite abnormality image 120. This embodiment can suppress learning based on inappropriate teacher data.

In the present embodiment, the deformed image generation unit 34 of the learning unit 30 adds deformation processing such as movement, rotation, enlargement, or reduction to the normal difference image to generate a normal deformed image. In the present embodiment, the deformed image generation unit 34 generates an abnormally deformed image by adding deformation processing such as moving, rotating, enlarging, reducing, or changing the brightness level to the abnormal difference image. In this way, the present embodiment can generate a large amount of normal deformed images from the normal difference image and a large amount of abnormally deformed images from the abnormal difference image. According to this embodiment, it is possible to improve the recognition rate by deep learning.

In addition, this embodiment generates a reference image from the normal image 100. In this embodiment, a normal difference image between the normal image 100 and the reference image in the evaluation image and an abnormal difference image between the abnormal image 110 and the reference image in the evaluation image are generated as the evaluation difference image. In this embodiment, when the data inference unit 44 of the inference unit 40 makes an inference, the normal difference image and the abnormal difference image can be used to make an inference using a DNN that recognizes whether the image is normal or abnormal. .. According to this embodiment, the inference result can be obtained with an appropriate recognition rate.

[Second Embodiment]
The image processing system 1A according to the present embodiment will be described with reference to FIG. FIG. 9 is a schematic view showing an example of a configuration example of the image processing system according to the second embodiment. The basic configuration of the image processing system 1A is the same as that of the image processing system 1 of the first embodiment. In the following description, the same components as those of the image processing system 1 are designated by the same reference numerals or corresponding reference numerals, and detailed description thereof will be omitted. The present embodiment is different from the first embodiment in that the image processing system 1A has a camera 60A.

The image processing system 1A is arranged on a production line or the like of a factory that produces products. The image processing system 1A includes a camera 60A, a recording unit 10A, a reference image generation unit 20A, a learning unit 30, and an inference unit 40A.

The camera 60A captures, for example, an image of the appearance of the product to be inspected. The camera 60A is arranged on a production line of a factory or the like. The camera 60A outputs the captured image to the recording unit 10A, the reference image generation unit 20A, and the inference unit 40A.

The recording unit 10A records the image input from the camera 60A as a training image set and a confirmation image set. Images of each classification type of normal, scratch, stain, color unevenness, and deformation are acquired from the camera 60A, and a predetermined number of images are recorded in the training image set and the confirmation image set, respectively.

The reference image generation unit 20A generates a reference image which is an average image of the normal image 100 input from the camera 60A, and records the reference image in the recording unit 10A. Here, the reference image is generated from the normal image 100 input from the camera 60A, but may be generated from the normal image 100 included in the training image set.

The inference unit 40A classifies the classification type of the image input from the camera 60A. More specifically, the inference unit 40A reads the DNN model learned from the recording unit 10A, reproduces the DNN, and infers whether the classification type of the image input from the camera 60A is "normal" or "abnormal". The product copied in the image in which the inference result of "normal" is obtained by the inference unit 40A is a normal product. The product copied in the image in which the inference result of "abnormal" is obtained by the inference unit 40A is an error product.

As described above, in the present embodiment, a DNN having an appropriate recognition rate can be constructed based on the image taken by the camera 60A.

In this embodiment, it is possible to infer at an appropriate recognition rate whether the product is normal or abnormal by using DNN for the image taken by the camera 60A. Here, the image processing system 1A includes the learning unit 30, but the learning unit 30 may not be included. In that case, the image processing system 1A includes a connection unit that connects to an external device or the like via USB, a network, or the like, and the recording unit 10A acquires a DNN model learned from the external device via the connection unit.

[Third Embodiment]
The image processing system 1B according to the present embodiment will be described with reference to FIG. FIG. 10 is a schematic view showing an example of a configuration example of the image processing system according to the third embodiment. In this embodiment, the camera 60B is arranged on the production line of the factory, the inference unit 40B is arranged on the terminal (terminal device) 90B or the like of the monitoring facility that monitors the production line of the factory, and the learning unit 30B is arranged on the cloud. In that respect, it differs from the first embodiment. The camera 60B, the recording unit 10B, and the inference unit 40B are connected so as to be able to transmit and receive data. The reference image generation unit 20B and the inference unit 40B are connected so that data can be transmitted and received. The learning unit 30B and the inference unit 40B are connected so that data can be transmitted and received.

The camera 60B transmits the captured image to the recording unit 10B and the inference unit 40B. The camera 60B acquires images of each classification type of normal, scratch, stain, color unevenness, and deformation.

The recording unit 10B is arranged on the cloud. The recording unit 10B records the image transmitted from the camera 60B as a training image set and a confirmation image set.

The reference image generation unit 20B is arranged on the cloud. The reference image generation unit 20B generates a reference image which is an average image of the normal image 100 transmitted from the camera 60B, and records the reference image in the recording unit 10B. Here, the reference image is generated from the normal image 100 input from the camera 60B, but may be generated from the normal image 100 included in the training image set.

The inference unit 40B is arranged on the terminal 90B arranged in the monitoring facility that monitors the production line of the factory. The inference unit 40B classifies the classification type of the image transmitted from the camera 60B. The inference unit 40B includes a reference image acquisition unit 41B, an image acquisition unit 42B, a difference image generation unit 43B, a data inference unit 44B, and a trained model acquisition unit 46B.

The reference image acquisition unit 41B acquires a reference image based on the normal image 100 as an image with a label indicating the first classification.

The image acquisition unit 42B acquires a specific image. The specific image is, for example, an image showing the product to be inspected among the products produced on the production line of the factory.

The difference image generation unit 43B generates a third difference image calculated from the difference between the specific image acquired by the image acquisition unit 42B and the reference image acquired by the reference image acquisition unit 41B.

The data inference unit 44B inputs a third difference image generated by the difference image generation unit 43B into the trained model acquired by the trained model acquisition unit 46B to obtain a specific image acquired by the image acquisition unit 42B. It is divided into a first classification and a second classification.

The trained model acquisition unit 46B is normal as a first difference image calculated from the difference between the normal image 100 with a label indicating the first classification and the reference image acquired by the reference image acquisition unit 41B. Using the difference image and the abnormal difference image as the second difference image calculated from the difference between the abnormal image 110 as the image with the label indicating the second classification and the reference image, the first classification and A trained model trained in the weighting of the multilayer neural network that distinguishes it from the second classification is acquired from the learning unit 30B.

As described above, in the present embodiment, in the terminal 90B arranged on the production line of the factory or the like, the image acquisition unit 42B acquires a specific image such as an image on which the product to be inspected is projected. The difference image generation unit 43B generates a third difference image calculated from the difference between the specific image and the reference image based on the normal image 100. The data inference unit 44B inputs a third difference image into the trained model acquired by the trained model acquisition unit 46B from the learning unit 30B on the cloud, thereby classifying the specific image into the first classification and the second classification. Identify in classification. According to this embodiment, it is possible to obtain an inference result with an appropriate recognition rate for a specific image taken by a camera 60B arranged on a production line or the like of a factory.

In the present embodiment, for the image taken by the camera 60B, the inference unit 40B arranged on the terminal 90B can infer whether the product is normal or abnormal by using DNN with an appropriate recognition rate. it can.

Although the image processing system 1 according to the present invention has been described so far, it may be implemented in various different forms other than the above-described embodiment.

Each component of the illustrated image processing system 1 is functionally conceptual and does not necessarily have to be physically configured as shown in the figure. That is, the specific form of each device is not limited to the one shown in the figure, and all or part of each device is functionally or physically dispersed or integrated in an arbitrary unit according to the processing load and usage status of each device. You may.

The configuration of the image processing system 1 is realized, for example, by executing a program or the like loaded in a memory such as ROM or RAM as software or firmware of a computer or the like by a device such as a CPU or GPU. In the above embodiment, it has been described as a functional block realized by linking these hardware or software. That is, these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof.

The above-mentioned components include those that can be easily assumed by those skilled in the art and those that are substantially the same. Further, the above configurations can be combined as appropriate. Further, various omissions, substitutions or changes of the configuration can be made without departing from the gist of the present invention.

Whether or not the color components of the reference image and the difference image are unnecessary may be determined based on the recognition rate by DNN. First, the learning unit 30 generates a reference image and a difference image on a gray scale, and learns the weight of DNN. Then, the inference unit 40 performs inference using DNN, and when the recognition rate of the inference result is equal to or higher than the threshold value, the color component is unnecessary, the reference image and the difference image are generated in gray scale, and the weight of DNN is generated. Is determined to be learned. When the recognition rate of the inference result is not equal to or more than the threshold value, it is determined that the color component is necessary and the reference image and the difference image should be generated by RGB or YCbCr to learn the weight of DNN. By doing so, the user does not have to determine the necessity of the color component, so that a DNN having an appropriate recognition rate can be easily constructed.

1 Image processing system 2 Image recognition learning device 3 Image recognition device 10 Recording unit 20 Reference image generation unit 30 Learning unit 31 Image acquisition unit 32 Abnormal image generation unit 33 Difference image generation unit 34 Deformed image generation unit 35 DNN learning unit 36 Learned Model storage unit 40 Reasoning unit 41 Learned model acquisition unit 42 Image acquisition unit 43 Difference image generation unit 44 Data inference unit 50 Display unit 100 Normal image 110 Abnormal image 120 Composite abnormal image A1 Abnormal area A2 Composite area

Claims (6)

An image acquisition unit that acquires an image with a label indicating the first classification and an image with a label indicating the second classification, and
A reference image generation unit that generates a reference image based on an image with a label indicating the first classification, and
Calculated from the difference between the first difference image calculated from the difference between the image with the label indicating the first classification and the reference image, and the image with the label indicating the second classification and the reference image. A difference image generation unit that generates a second difference image to be generated,
Learning to generate a trained model in which the weighting of the multi-layer neural network that distinguishes between the first classification and the second classification is trained using the first difference image and the second difference image. Department and
An image recognition learning device comprising. An abnormal image generation unit that cuts out an area including a specific object from an image with a label indicating the second classification and superimposes it on an image with a label indicating the first classification to generate a composite abnormal image. ,
With
The image recognition learning device according to claim 1, wherein the difference image generation unit includes the composite abnormal image in an image with a label indicating the second classification. Moving, rotating, enlarging, reducing, or changing the brightness level of the first difference image and the second difference image, respectively, to generate a new first difference image and the second difference image. The image recognition learning device according to claim 1 or 2, wherein the modified image generation unit is provided. An image acquisition step of acquiring an image with a label indicating the first classification and an image with a label indicating the second classification, and
A reference image generation step of generating a reference image based on an image with a label indicating the first classification, and
Calculated from the difference between the first difference image calculated from the difference between the image with the label indicating the first classification and the reference image, and the image with the label indicating the second classification and the reference image. The difference image generation step of generating the second difference image to be performed, and
Learning to generate a trained model in which the weighting of the multi-layer neural network that distinguishes between the first classification and the second classification is trained using the first difference image and the second difference image. Steps and
Image recognition learning method including. An image acquisition step of acquiring an image with a label indicating the first classification and an image with a label indicating the second classification, and
A reference image generation step of generating a reference image based on an image with a label indicating the first classification, and
Calculated from the difference between the first difference image calculated from the difference between the image with the label indicating the first classification and the reference image, and the image with the label indicating the second classification and the reference image. The difference image generation step of generating the second difference image to be performed, and
Learning to generate a trained model in which the weighting of the multi-layer neural network that distinguishes between the first classification and the second classification is trained using the first difference image and the second difference image. Steps and
An image recognition learning program that causes a computer that operates as an image recognition learning device to execute. An image acquisition unit that acquires a specific image,
A reference image acquisition unit that acquires a reference image based on an image with a label indicating the first classification, and
A difference image generation unit that generates a third difference image calculated from the difference between the specific image and the reference image, and
Calculated from the difference between the first difference image calculated from the difference between the image with the label indicating the first classification and the reference image, and the image with the label indicating the second classification and the reference image. A trained model acquisition unit that acquires a trained model trained in the weighting of a multi-layer neural network that distinguishes between the first classification and the second classification by using the second difference image.
By inputting the third difference image into the trained model, a data inference unit that distinguishes a specific image acquired by the image acquisition unit into a first classification and a second classification,
A terminal device comprising.

Images