JP2018097807A - Learning device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique for efficiently performing labelling by semi-supervised learning.SOLUTION: A learning device 1 includes: an input part 10 for inputting a plurality of labelled images L and a plurality of non-labelled images U; a CNN processing part 11 for generating a plurality of feature maps by subjecting the images to a CNN processing; summing the entropy obtained for each pixel with respect to the plurality of feature maps generated in the CNN processing part 11 and summing the cross entropy with a correct label attached for each element and the elements of the plurality of feature maps with respect to the plurality of feature maps generated from the labelled images L, an estimation value calculation part 12 for performing the process of drawing the cross entropy from the entropy with respect to the plurality of labeled images and the plurality of non-labeled images and calculating the estimation value by summing the obtained value; and a learning part 13 for performing the learning of a parameter Q used in the CNN process so as to minimize the evaluation value.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for labeling an unknown image, and more particularly, to a technique for learning parameters for classifying an unknown image for labeling.

  Semantic segmentation, which labels each pixel of an image, is an important technique for applications such as automatic driving. In general, a large amount of labeling is required to achieve high performance. In the semantic segmentation that requires a label for each pixel, the labeling cost per image is particularly high, and labor saving is required. Weakly supervise learning (Non-Patent Documents 1 and 2), micro annotation (Non-Patent Document 3), and the like have been proposed as methods for saving the labeling cost.

  Weakly supervise learning only includes a small amount of labeled data labeled for each pixel and the label of the object in the image with relatively low labeling costs (describes the presence / absence of each object in the entire image) There is an advantage that it is not necessary to label each pixel. The micro annotation labels only the correctness of the segmentation proposal output by the trained model.

Learning Transferrable Knowledge for Semantic Segmentation with Deep Convolutional Neural Network (CVPR2016) Decoupled Deep Neural Network for Semi-supervised Semantic Segmentation (NIPS 2015) Improving Weakly-Supervised Object Localization By Micro-Annotation (BMVC2016)

  However, no method has been proposed for utilizing unlabeled data in semi-supervised learning without using the above-mentioned weakly supervised data or micro annotation information in semantic segmentation. The present invention proposes a technique for efficiently labeling by semi-supervised learning.

  The learning device according to the present invention includes an input unit that inputs a plurality of labeled images and a plurality of unlabeled images, a CNN processing unit that generates a plurality of feature maps by CNN processing the images, and the CNN processing unit. The entropy obtained for each pixel for the generated feature maps is added together, and for the feature maps generated from the labeled images, the cross entropy with the correct label assigned for each pixel is added together. The cross entropy is subtracted from the entropy for a plurality of labeled images and a plurality of unlabeled images, and an evaluation value calculation unit that calculates an evaluation value by adding the obtained values; and A learning unit that learns parameters used in the CNN process.

  In this way, by reducing the entropy of the feature map and obtaining parameters that increase the cross-entropy of the labeled image with the correct answer label, even if there are not many labeled images, unlabeled images It is possible to obtain a parameter for appropriately classifying an unknown image by supplementing the accuracy of the parameter in terms of classifying the image.

  In the learning device of the present invention, the evaluation value calculation unit may calculate entropy based on an average value of pixels in a predetermined area in units of a predetermined area instead of a configuration for obtaining entropy for each pixel. Good. Further, the evaluation value calculation unit may calculate entropy based on a weighted sum of pixels in a predetermined area as a unit instead of a configuration for obtaining entropy for each pixel.

  In an image, it is common that the label is the same in an area where several pixels are gathered. Therefore, by performing processing based on the average value or weighted sum of pixels in a predetermined area, the parameters are appropriately set. Can learn.

  In the learning device of the present invention, the evaluation value calculation unit may calculate entropy based on a representative value of a super pixel instead of a configuration for obtaining entropy for each pixel.

  A parameter can be appropriately learned by performing processing using a super pixel which is a collection of pixels having similar characteristics.

  The learning device of the present invention includes a generator that generates a new image from arbitrary information, and an identifier that identifies whether the new image and a predetermined image are the same image, A GAN (Generative Adversarial Networks) device that performs learning of the generator so as to be identified by the classifier as being the same as the predetermined image, and using the parameters learned by the learning unit, Learning is performed so that the GAN device generates the predetermined image from a semantic image in which a label is assigned to each pixel, and the parameter is learned by back-propagating an error of the semantic image obtained as a result. Good.

  Thus, by updating the parameters so that the image restored with the semantic image as a condition is similar to the original predetermined image, the accuracy of the parameters can be further improved.

  The learning device of the present invention includes a plurality of feature maps obtained by processing the predetermined image by a CNN processing unit, and a plurality of feature maps obtained by processing a new image generated from the semantic image by a CNN processing unit. And a cross-entropy calculating unit for obtaining a cross-entropy between the generator and the generator may generate the new image using the cross-entropy information.

  The GAN discriminator has a configuration for determining whether or not they match by measuring the distance between the manifold on which the predetermined image is stretched and the new image, so that only the discriminator can identify a new image. It is not possible to correspond to a predetermined image in pixel units. With the configuration of the present invention, it is possible to generate an image that is semantically close to a predetermined image by using a new image generated from the semantic image and cross-entropy information of the predetermined image.

  The learning method of the present invention is a method for learning a parameter for classifying an unlabeled image based on a plurality of labeled images and a plurality of unlabeled images by a learning device. A step of inputting a labeled image and a plurality of unlabeled images; a step in which the learning device performs a CNN process on the image to generate a plurality of feature maps; and a step in which the learning device generates a plurality of feature maps. The entropy obtained for each pixel is added together, and for a plurality of feature maps generated from the labeled image, the cross entropy with the correct label attached to each pixel is further added, and the cross entropy is calculated from the entropy. The subtraction process is performed for multiple labeled images and multiple unlabeled images, and the calculated values are added together for evaluation. Comprising calculating a value, the learning apparatus, and performing learning of the parameters used in the CNN process so as to minimize the evaluation value.

  The program of the present invention is a program for learning parameters for classifying an unlabeled image using a plurality of labeled images and a plurality of unlabeled images. The step of generating a feature map and the entropy obtained for each pixel of the plurality of generated feature maps are added together, and for a plurality of images generated from the labeled image, a correct label attached to each pixel Summing the cross entropy and subtracting the cross entropy from the entropy is performed for a plurality of labeled images and a plurality of unlabeled images, and adding the obtained values to calculate an evaluation value; and Performing learning of parameters used in the CNN process so as to minimize the evaluation value To.

  According to the present invention, when the entropy of the feature map of the input image is reduced and the parameters that increase the cross entropy of the labeled image with the correct answer label are obtained, the number of labeled images is not abundant. Even so, it is possible to obtain parameters for appropriately classifying unknown images by supplementing the accuracy of the parameters in terms of classifying unlabeled images well.

It is a figure which shows the structure of the learning apparatus of 1st Embodiment. It is a figure which shows operation | movement of the learning apparatus of 1st Embodiment. It is a figure which shows the structure of the learning apparatus of 2nd Embodiment. It is a figure which shows the structure of the GAN apparatus used by 2nd Embodiment. It is a figure which shows operation | movement of the learning apparatus of 2nd Embodiment.

  Hereinafter, the learning apparatus according to the embodiment of the present invention will be described. The learning device according to the present embodiment learns parameters of convolutional neural networks (element values of each kernel in the convolution layer, coupling weights of each unit of all connection layers, bias, etc.) for performing semantic segmentation on the image. It is a device to do.

  FIG. 1 is a diagram illustrating a configuration of the learning device 1. The learning device 1 is connected to a database 14 that stores a large number of images. The database 14 stores a large number of image data as teacher data. The image includes a labeled image L to which a label indicating what image is given and an unlabeled image U to which no label is given.

  Since it takes a lot of time to label the images, all the images are not labeled. The number of labeled images L is significantly smaller than that of unlabeled images U. For example, there are 10,000 unlabeled images U and 1000 labeled images L. The learning device 1 according to the present embodiment performs semi-supervised learning using the labeled image L and the unlabeled image U so that an unknown unlabeled image can be automatically labeled. Learn the parameters.

  The learning apparatus 1 includes an input unit 10 that reads and inputs an image from the database 14 and a CNN processing unit 11 that performs a convolutional neural network process on the input image. The CNN processing unit 11 performs a CNN process on the image to generate a plurality of feature maps. The number of feature maps generated here is the same as the number of classes into which images are to be classified. The CNN processing unit 11 generates a feature map for both the labeled image and the unlabeled image input by the input unit 10.

  The learning device 1 has an evaluation value calculation unit 12 that calculates an evaluation value for performing parameter updating based on the generated feature map, and a learning unit 13 that updates a parameter so as to minimize the evaluation value. doing. The evaluation value calculation unit 12 calculates the entropy of the feature map of all the input images, and calculates the cross-entropy between the feature map and the correct answer label for the labeled image. The evaluation value calculation unit 12 calculates a value obtained by subtracting the cross entropy from the entropy of all images as an evaluation value. The evaluation value calculation unit 12 calculates entropy for each pixel.

  The learning unit 13 updates parameters used in the CNN processing unit 11 so as to minimize the evaluation value. The following formula represents the parameter update process performed by the learning unit 13.

  In the above equation, Q indicates a parameter used in the CNN processing unit 11, and C indicates a correct label given to the labeled image. L indicates a labeled image and U indicates an unlabeled image. The first term in parentheses represents the cross entropy between the labeled image and the correct label, and the second term represents the entropy of all the images of the labeled image and the unlabeled image.

  Qualitatively explaining this expression, the first term is the cross-entropy between the feature map of the labeled image L and the correct label. Therefore, the larger the cross-entropy, the more correct the feature map generated by the CNN processing unit 11 is. Means close to the label. The second term is the entropy of the feature map of the entire image. The smaller this entropy is, the more unknown the label is, but it means that some feature is remarkable. In other words, the smaller the entropy of the second term, the better the feature map is classified.

  The learning unit 13 obtains a parameter that minimizes the evaluation value shown in the above mathematical formula, thereby obtaining a parameter that can classify an unknown image and can give a label to the classified class.

  FIG. 2 is a flowchart showing an operation of performing learning in the learning device 1. First, a large number of labeled images and unlabeled images are read from the database 14 and input to the learning device 1 (S10). Next, the learning device 1 performs CNN processing on the input image to obtain a feature map of each image (S12), and the feature map entropy and the feature map generated from the labeled image cross with the correct label. Entropy is determined for each pixel. The evaluation value calculation unit 12 obtains an evaluation value by subtracting the cross entropy from the obtained entropy (S14).

  Next, the learning device 1 determines whether or not the end condition is satisfied (S16). The end condition may be, for example, that the evaluation value is equal to or less than a predetermined value, or that the number of parameter updates has reached a predetermined value. When it is determined that the end condition is satisfied (YES in S16), the learning device 1 determines the value of the parameter obtained at that time as the parameter of the CNN processing unit 11 (S18). If it is determined that the termination condition is not satisfied (NO in S16), the learning unit 13 updates the parameter in a direction in which the evaluation value decreases (S20), and returns to the step of performing the CNN process again (S12). ). The configuration and operation of the learning device 1 according to the first embodiment have been described above.

  The learning device 1 according to the present embodiment can obtain a parameter for classifying an image by updating the parameter in a direction that maximizes the cross entropy between the feature map of the labeled image and the correct label. At this time, by updating the parameter in the direction that minimizes the entropy of the feature map of the unlabeled image, by obtaining a parameter that makes a certain feature remarkable, even when the number of labeled images is small, By using the information of the unlabeled image, a parameter for appropriately classifying the image can be obtained.

(Second Embodiment)
FIG. 3 is a diagram illustrating a configuration of the learning device 2 according to the second embodiment. The learning device 2 of the second embodiment includes a GAN device 20 in addition to the configuration of the learning device 1 of the first embodiment. The learning device 2 of the second embodiment is different in that the learning result is fed back to the learning unit 13 by the identification result obtained by the classifier 23 provided in the GAN device 20.

  FIG. 4 is a diagram showing a detailed configuration of the GAN device 20. The semantic segmentation unit 21 has a function of performing CNN processing on the input image, classifying the input image into pixels, and generating a semantic image. In this CNN process, parameters learned by the learning unit 13 are used. Hereinafter, for convenience of description, an image input to the semantic segmentation unit 21 is referred to as “image A”.

  Here, an outline of the GAN device 20 will be described. The GAN device 20 verifies whether the semantic image generated from the input image A is close to the correct answer. Since it is considered that the closer the semantic image is to the correct answer, the closer the input image A is to the image A ′, the GAN device 20 calculates the error between the input image A and the restored image A ′ and learns. Back propagation to the unit 13 helps the learning in the learning unit 13.

  The GAN device 20 includes a generator 22 and a discriminator 23. The generator 22 has a function of generating a new image from arbitrary information, and the identifier 23 has a function of identifying whether or not the generated new image and the predetermined image are the same image. The generator 22 generates an image that is determined to be the same image by the classifier 23, that is, an image that tricks the classifier 23. The discriminator 23 inputs the discrimination result that discriminates the generated image and the predetermined image to the generator 22, and the generator 22 uses the information to generate an image closer to the predetermined image. 23. By repeatedly generating and identifying the image in this way, the generator 22 learns to generate the same image as the predetermined image. The above is the principle of GAN (Generative Adversarial Networks).

  In the present embodiment, the generator 22 of the GAN device 20 generates a new image A ′ from the semantic image of the image A generated by the semantic segmentation unit 21. Then, the discriminator 23 discriminates whether or not the new image A ′ is the same as the image A from which the semantic image is based. That is, it is determined how close to the original image A the image A ′ can be generated from the semantic image, and the result is fed back to the learning unit 13. If the image A ′ close to the image A can be generated, the semantic image is likely to be correct, and if only the image A ′ having a distance from the image A can be generated, the semantic image is determined from the correct answer. You can see that it was far away.

  The GAN device 20 performs a CNN processing unit 24 that performs CNN processing of the images A and A ′, and a cross-entropy calculation that calculates a cross-entropy of the feature map of the image A and the feature map of the image A ′ obtained by the CNN processing unit 24. Part 25. The cross entropy calculation unit 25 inputs the obtained cross entropy information to the generator 22. Thereby, since the generator 22 can generate a new image A ′ based on the cross-entropy information between the image A and the image A ′, an image A ′ that is semantically close to the image A can be generated.

  FIG. 5 is a flowchart illustrating an operation of performing learning in the learning device 2 according to the second embodiment. The operations until the parameters of the CNN processing unit 11 are determined based on the evaluation values of entropy and cross entropy are the same as the operations of the learning device 1 according to the first embodiment. However, in the second embodiment, the parameter obtained based on the evaluation value is determined as a temporary parameter (S18).

  The learning device 1 according to the second embodiment further updates the parameters by verifying the semantic image generated using the temporarily determined parameters in the GAN device 20 (S24). That is, the learning device 1 performs a CNN process using the temporarily determined parameters to generate a feature map, and classifies each pixel of the image. Then, a semantic image is generated based on the classification of each pixel. The generator 22 generates an image A ′ from the semantic image, and generates an image A ′ so that the image A ′ is identified by the classifier 23 as being the same as the original image A. The generator 22 repeatedly generates the image A ′ based on the identification result information from the classifier 23 and brings it closer to the image A. At this time, as described with reference to FIG. 4, the generator 22 generates an image using information on the cross entropy of the original image A and the restored image A ′. The GAN device 20 determines whether or not the identification result between the image A ′ thus generated and the original image A satisfies the end condition (S26).

  If it is determined that the end condition is satisfied (YES in S26), the temporarily determined parameter is determined as the final parameter (S28). If it is determined that the termination condition is not satisfied (NO in S26), the identification error is propagated back to the learning unit 13 to update the parameter, and the parameter is updated again based on the evaluation value (S20). The configuration and operation of the learning device 2 according to the second embodiment have been described above.

  As in the first embodiment, the learning device 2 according to the second embodiment appropriately classifies images using information on unlabeled images even when the number of labeled images is small. Parameters can be determined. In addition, the learning device 2 according to the second embodiment verifies the parameters temporarily generated using the GAN device 20, so that the parameters can be obtained with high accuracy.

(Modification)
In the above-described embodiment, an example in which entropy is obtained for each pixel has been described. However, entropy is obtained by using an average value of pixels in a predetermined area as a unit. You may calculate. Further, entropy may be calculated based on a weighted sum of pixels in a predetermined area as a unit. In an image, the label is generally the same in a region where several pixels are gathered, so that it is possible to learn parameters appropriately by performing processing based on the average value or weighted sum of neighboring pixels. it can.

  Further, instead of the configuration for obtaining entropy for each pixel, the entropy may be calculated based on the representative value of the super pixel. Thus, by performing processing using superpixels that are a collection of pixels having similar features, it is possible to appropriately learn parameters and reduce entropy calculation processing. Here, the representative value may be an average value of all the pixels in the super pixel, a maximum value, or another value.

  When entropy is calculated using super pixels and the parameters are updated, the parameters may be calculated using super pixels generated at different resolutions. That is, the CNN processing unit performs a process of dividing the labeled image and the unlabeled image into super pixels at different resolutions, and the evaluation value calculation unit and the learning unit perform learning using a plurality of patterns of super pixels having different resolutions. .

  Specifically, the CNN processing unit divides the image input from the input unit into superpixels with three resolutions, for example, 10,000 divisions, 1000 divisions, and 100 divisions, and performs CNN processing on the images with the respective resolutions. Go to generate a feature map. Subsequently, the evaluation value calculation unit obtains parameters for classifying the super pixels by performing the learning described in the above embodiment for each resolution image. By using the information of the labeled image, the labels of each class can be obtained, so that the label parameters are weighted according to the contents of the labels, and the results obtained at different resolutions are blended.

  For example, a thin and thin object such as a pole or a signal is likely to be buried in the background with a low resolution super pixel. Therefore, when the output label shows a high value for these objects, a label histogram obtained by high-resolution super pixels that are difficult to be buried is used with high weight. Conversely, a label such as a road that is difficult to be buried uses a label histogram obtained with low resolution superpixels with high weight in order to make it more consistent with surrounding labels. Note that the blending factor for each label is obtained in advance by learning from the rate at which the label changes when the labeled data is segmented at various resolutions.

  By learning using superpixels divided at multiple resolutions in this way and blending the learning results for each label, things that are difficult to bury, such as roads, can be made consistent with surrounding labels. It is possible to obtain an appropriate parameter such that a small object such as a pole or a signal is not buried in the surroundings.

  The present invention can be applied to a technique for classifying an image, and is particularly useful as a technique for labeling an unknown image.

1, 2 Learning device 10 Input unit 11 CNN processing unit 12 Evaluation value calculation unit 13 Learning unit 20 GAN device 21 Semantic segmentation unit 22 Generator 23 Discriminator 24 CNN processing unit 25 Cross entropy calculation unit

Claims (8)

An input unit for inputting a plurality of labeled images and a plurality of unlabeled images;
A CNN processing unit that CNN-processes an image to generate a plurality of feature maps;
The entropy obtained for each pixel for the plurality of feature maps generated by the CNN processing unit is summed, and for the plurality of feature maps generated from the labeled image, a correct label assigned to each pixel and An evaluation value calculation unit that performs a process of subtracting the cross entropy from the entropy and subtracting the cross entropy for a plurality of labeled images and a plurality of unlabeled images, and adding the obtained values to calculate an evaluation value; ,
A learning unit that learns parameters used in the CNN process so as to minimize the evaluation value;
A learning apparatus comprising:   The learning apparatus according to claim 1, wherein the evaluation value calculation unit calculates entropy based on an average value of pixels in a predetermined area as a unit instead of a configuration for obtaining entropy for each pixel.   The learning device according to claim 1, wherein the evaluation value calculation unit calculates entropy based on a weighted sum of pixels in a predetermined area as a unit instead of a configuration for obtaining entropy for each pixel. .   The learning apparatus according to claim 1, wherein the evaluation value calculation unit calculates entropy based on a representative value of a super pixel instead of a configuration for obtaining entropy for each pixel. A generator for generating a new image from arbitrary information, and an identifier for identifying whether the new image and the predetermined image are the same image, wherein the new image is the same as the predetermined image A GAN (Generative Adversarial Networks) device that trains the generator to be identified by the classifier,
Using the parameters learned by the learning unit, learning is performed so that the GAN device generates the predetermined image from a semantic image in which each pixel of the predetermined image is labeled, and the semantic obtained as a result is obtained. The learning apparatus according to claim 1, wherein the parameter is learned by back-propagating an image error. Cross entropy between a plurality of feature maps obtained by processing the predetermined image by the CNN processing unit and a plurality of feature maps obtained by processing the new image generated from the semantic image by the CNN processing unit. A cross entropy calculation unit
The learning device according to claim 5, wherein the generator generates the new image using the cross-entropy information. A method for learning parameters for classifying an unlabeled image based on a plurality of labeled images and a plurality of unlabeled images by a learning device,
The learning device inputs a plurality of labeled images and a plurality of unlabeled images;
The learning device CNN-processing the image to generate a plurality of feature maps;
The learning device adds the entropy obtained for each pixel with respect to the plurality of generated feature maps, and for the plurality of feature maps generated from the labeled image, further includes a correct label attached to each pixel. Adding cross entropy, subtracting the cross entropy from the entropy, performing a plurality of labeled images and a plurality of unlabeled images, and adding the obtained values to calculate an evaluation value;
The learning device learning a parameter used in the CNN process so as to minimize the evaluation value;
A learning method comprising: A program for learning a parameter for classifying an unlabeled image using a plurality of labeled images and a plurality of unlabeled images,
Generating a plurality of feature maps by CNN processing the image;
The total entropy obtained for each pixel for the plurality of generated feature maps is summed, and for a plurality of images generated from the labeled images, the cross entropy with the correct label attached to each pixel is further summed. Subtracting the cross entropy from the entropy is performed for a plurality of labeled images and a plurality of unlabeled images, and adding the obtained values to calculate an evaluation value;
Learning parameters used in the CNN process so as to minimize the evaluation value;
A program that executes

Images