JP2021022368A - Image recognition device and training device using neural network - Google Patents

Abstract

To incorporate human knowledge into a recognition results of a neural network that outputs an attention map that expresses a gaze area of an input image.SOLUTION: A processing unit 5 of an image recognition device 1 applies modification corresponding to a person's modification operation to an attention map 53 generated by a neural network 10 in which an input image 51 is input. The processing unit 5 then outputs a recognition result of the input image 51 generated by the neural network 10 based on the modified attention map 53 and the input image 51.SELECTED DRAWING: Figure 8

Description

The present invention relates to an image recognition device and a training device using a neural network.

Conventionally, in an image recognition technique using a neural network such as CNN (Convolutional Neural Network), a technique for generating an attention map expressing a gaze area at the time of inference by a neural network is known (for example, Non-Patent Document 1, 2).

Ramprasaath, R., S., Michael, C., Abhishek, D., Ramakrishna, V., Devi, P. and Dhruv, B .: Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization, International Conference on Computer Vision, pp. 618-626 (2017). Zhou, B., Khosla, A., Lapedriza, A., Oliva, A. and Torralba, A .: Learning Deep Features for Discriminative Localization, Computer Vision and Pattern Recognition, pp. 2921-2929 (2016).

However, according to the inventor's examination, when the neural network generates a recognition result and an attention map for a certain image, the recognition result and the gaze area in the attention map may not match. For example, when the recognition result is "smile" but the gaze area is hair, the recognition result and the gaze area in the attention map do not match. If there is a discrepancy between the recognition result and the gaze area in the attention map, it is a problem that leads to an error in the image recognition itself. And at present, there is no way to correct this discrepancy.

In view of the above points, it is an object of the present disclosure to incorporate human knowledge into the recognition function or learning of a neural network in an image recognition technique using a neural network that outputs an attention map.

According to one aspect of the present disclosure, the image recognition device includes an input unit (120) for inputting an image (51) into the neural network (10) and an attention map (53) representing the gaze area of the input image. ) Is generated by the neural network, based on the map correction unit (140) that corrects the generated attention map according to a human correction operation, the corrected attention map, and the image. When the neural network (10) generates the recognition result of the image, it includes an output unit (160) that outputs the generated recognition result.

In this way, the attention map 53 is modified by utilizing the knowledge of the person, so that the cognitive unit 14 attaches importance to the area intended by the person. As a result, image recognition can be performed according to the intention of the person. At this time, the parameters of the neural network 10 have not been changed. That is, it is possible to obtain a recognition result intended by a person without requiring re-learning of the neural network 10.

According to another aspect of the present disclosure, the training device is a neural network (10) trained to generate an attention map (53) representing the gaze area of the input image (51) and a recognition result of the image. The re-learning data set includes a reading unit (205) for reading out the above, and a training unit (250) for re-learning the neural network using the re-learning data set. Includes multiple teacher attention maps that are the correct values for the attention map when input to the neural network.

In this way, when re-learning the neural network that generates the attention map, the teacher attention map, which is the correct answer value of the attention map, is used. Since the teacher attention map is created based on human knowledge, it is possible to incorporate human knowledge into the learning of the neural network by doing so.

The reference numerals in parentheses attached to each component or the like indicate an example of the correspondence between the component or the like and the specific component or the like described in the embodiment described later.

It is a block diagram of an image recognition apparatus. It is a figure which shows the structure of a neural network. It is a figure which shows the structure of the attention part. It is a flowchart of the process executed by a processing unit. It is a figure which shows the display which the input image and the attention map before correction overlap. It is a figure which shows the display in which the input image and the attention map which deleted the gaze area overlap. It is a figure which shows the display in which the input image and the attention map to which the gaze area is added overlap. It is a figure which shows the relationship between the modified attention map and a neural network. It is a flowchart of the process executed by the processing part in 2nd Embodiment. It is a figure which shows the outline of relearning.

(First Embodiment)
Hereinafter, the first embodiment will be described. As shown in FIG. 1, the image recognition device 1 according to the present embodiment includes an operation device 2, a display device 3, a memory 4, and a processing unit 5.

The operation device 2 is a device that receives a human operation and outputs a signal corresponding to the received operation to the processing unit 5. The operation device 2 may be, for example, a mouse, a keyboard, a touch panel, or the like. The display device 3 is a device that displays an image on a person.

The memory 4 includes a RAM which is a rewritable volatile storage medium, a ROM which is a non-rewritable non-volatile storage medium, and a flash memory which is a rewritable non-volatile storage medium. RAM, ROM, and flash memory are non-transitional substantive storage media. The data of the trained neural network 10 is recorded in advance in the flash memory.

The processing unit 5 executes a program (not shown) stored in the ROM or the flash memory, and uses the RAM as a work area during the execution to realize various processes described later.

Here, the neural network 10 will be described. As shown in FIG. 3, the neural network 10 is a deep neural network including a feature extraction unit 11, an attention unit 12, a synthesis unit 13, and a cognitive unit 14.

The neural network 10 generates an attention map 53 when the input image 51 is input. The attention map 53 is data representing a gaze area at the time of inference of the neural network 10. That is, the attention map 53 is data for visual explanation explaining which region of the input image 51 is emphasized at the time of inference of the neural network 10.

Further, the neural network 10 outputs the classification result of the input image 51 based on the input image 51 and the attention map 53. The classification result of the input image 51 is a plurality of likelihoods corresponding to a plurality of classes (for example, Dalmatian, crayfish, finch, frog, etc.) corresponding to the recognition target of the image. Here, the number of classes is K.

The feature extraction unit 11 is a neural network having a plurality of layers. These plurality of layers include at least a plurality of convolution layers. Further, these plurality of layers may be components of a plurality of residual blocks, or may have a plurality of pooling layers and the like. Then, the feature extraction unit 11 generates the feature map 52 by propagating the information of the input input image 51 to these plurality of layers.

The feature map 52 is a map having K resolutions h × w corresponding to each of the K classes. h and w are arbitrary integers. Therefore, the number of channels in the feature map 52 is K. The resolution of the feature map 52 may be the same as the resolution of the input image 51, or may be lower than the resolution of the input image 51.

The feature extraction unit 11 may be composed of a portion of the baseline model starting from the input layer and before the first fully connected layer. As the baseline model, a model having a plurality of convolution layers and generating the likelihoods of a plurality of classes of the same type as the neural network 10 is selected. For example, as the baseline model, VGGNet shown in Non-Patent Document 3 may be used, ResNet shown in Non-Patent Document 4 may be used, or another CNN (Convolutional Neural Network) is used. May be good.

The attention unit 12 generates an attention map 53 from the feature map 52 generated by the feature extraction unit 11. The attention unit 12 is a neural network having a plurality of layers. As shown in FIG. 3, these plurality of layers have a first portion 12a having one or more convolution layers or one or more residual blocks, and a K × 1 × 1 convolution layer 12b in the subsequent stage of the first portion. .. Here, assuming that L, a, and b are arbitrary natural numbers, the L × a × b convolution layer means a convolution layer using an a × b kernel in each of the L channels.

The attention portion 12 has two K × 1 × 1 convolution layers 12c and a 1 × 1 × 1 convolution layer 12e that are branched in the subsequent stage of the convolution layer 12b. The attention portion 12 has a GAP (Global Average Pooling) layer 12d in the subsequent stage of the convolution layer 12c.

The information of the feature map 52 input to the attention unit 12 propagates through the first portion 12a, the convolution layer 12b, the convolution layer 12c, and the GAP layer 12d, and the output of the GAP layer 12d is input to the Softmax function, so that the neural network is used. Likelihoods of a plurality of classes of the same type as network 10 are generated as classification results. The classification result is a kind of recognition result.

Further, the information of the feature map 52 input to the attention unit 12 is propagated to the first portion 12a, the convolution layer 12b, and the convolution layer 12e, so that the attention map 53 is generated. By generating the attention map 53 through the convolution layer 12b instead of the fully connected layer, the information in the gaze area is propagated to the attention map 53 while being localized. Further, through the 1 × 1 × 1 convolution layer 12e, a 1-channel attention map 53 is generated as a weighted sum of the gaze areas corresponding to all the classes. Each value of the kernel of the convolution layer 12e may be 1 or other.

The resolution of each map of the feature map 52 and the resolution of the attention map 53 are the same. The attention portion 12 is configured so as to be so. In the attention map 53, a relatively high pixel value is given to the pixel corresponding to the gaze area, and a lower pixel value is given to the pixel not corresponding to the gaze area as compared with the gaze area. The value that each pixel value of the attention map 53 can take may be a binary value or a value in 256 steps. The higher the pixel value of a pixel, the higher the degree of attention at the position of that pixel.

The synthesizing unit 13 synthesizes the feature map 52 and the attention map 53. Specifically, the attention map 53 is multiplied by the map having the resolution h × w in each of the K channels in the feature map 52. The multiplication of the attention map 53 and the map having the resolution h × w is performed between pixels having the same position coordinates. Note that the composition may be multiplication, addition, or an operation consisting of a combination of addition and multiplication as described above. By this composition, a composition map 54 is obtained. The number of channels and the resolution of the composite map 54 are the same as those of the feature map 52.

The cognitive unit 14 outputs the likelihood of each class based on the synthetic map 54. The cognitive unit 14 is a neural network having a plurality of layers. These plurality of layers include at least a plurality of convolution layers. In addition, these plurality of layers include one or both of the fully connected layer and the GAP layer. Further, these plurality of layers may be components of a plurality of residual blocks, or may have a plurality of pooling layers. The cognitive unit 14 propagates the input information of the synthetic map 54 to these plurality of layers, and outputs the likelihood of each class as a classification result. The classification result is also a recognition result. The cognitive unit 14 may be composed of a portion of the above-mentioned baseline model from the portion immediately after the portion used by the attention unit 12 to the output layer.

The functions described above that the neural network 10, the feature extraction unit 11, the attention unit 12, the synthesis unit 13, and the cognitive unit 14 perform are actually performed by the processing unit 5 according to the structure and parameters of the neural network 10. It is realized by performing the processing.

The feature extraction unit 11, the attention unit 12, and the synthesis unit 13 are learned in advance by the error back-propagation method by supervised learning so as to realize the above functions. In learning, L = Latt + Lper is used as the learning error (also referred to as a loss function) L. Here, Latt is a learning error related to the classification result output by the attention unit 12, and Lper is a learning error related to the classification result output by the cognitive unit 14. Latt and Lper may be calculated by applying a combination of Softmax function and cross entropy to each classification result. The feature extraction unit 11 is learned by passing through the gradients of the attention unit 12 and the cognitive unit 14 in the error back propagation method.

Hereinafter, the image classification process of the processing unit 5 using the trained neural network 10 configured in this way will be described.

When a predetermined condition such as an execution start operation for the operation device 2 by a person is satisfied, the processing unit 5 starts the process shown in FIG. 4 defined in the predetermined program recorded in the memory 4. In this process, the processing unit 5 first reads the neural network 10 from the memory 4 in step 110.

Subsequently, in step 120, the input image 51 is acquired, and the input image 51 is input to the neural network 10. The input image 51 may be an image selected by a human operation on the operation device 2 or the like from a plurality of images recorded in the memory 4 in advance, or another device via a communication network (not shown). It may be an image received from.

When the input image 51 is input to the neural network 10, in the neural network 10, as described above, the feature extraction unit 11 generates the feature map 52 and the classification result from the input image 51, and the attention unit 12 draws attention from the feature map 52. Generate map 53.

The processing unit 5 acquires the attention map 53 generated in this way in step 130 following step 120. That is, the attention map 53 generated in the memory 4 by the neural network 10 is copied or moved to another area in the memory 4.

Subsequently, in step 140, the processing unit 5 modifies the acquired (that is, the copy destination or the moving destination) attention map 53 based on the modification operation for the human operating device 2. As a result, the attention map 53 is modified according to human knowledge.

Specifically, the processing unit 5 causes the display device 3 to display the attention map 53 and the pointer before modification. The pointer is an image that moves within the display range of the attention map 53 displayed on the display device 3 in response to a person's operation on the operation device 2. A person corrects the value of the position range that overlaps with the pointer in the displayed attention map 53 by performing a predetermined correction operation (for example, erasing operation, addition operation, etc.) on the operation device 2.

At this time, as shown in FIG. 5, the processing unit 5 responded to the above-mentioned correction operation in a state where the input image 51 was transparently aligned with the attention map 53 and superposed on the display device 3. The modification may be reflected in the attention map 53. At this time, if the resolutions of the input image 51 and the attention map 53 are different, the processing unit 5 lowers the resolution of the input image 51 so as to match the attention map 53, and then transparently superimposes the input image 51 on the attention map 53.

In FIG. 5, the input image 51 in which the Dalmatian is holding the soccer ball is transparently superimposed on the attention map 53.

In this attention map 53, the gaze area is in the area of the soccer ball. When the attention map 53 is input to the synthesis unit 13 as it is, and the composition map 54 which is the composition result of the attention map 53 and the feature map 52 is input to the first layer of the cognitive unit 14, the classification result generated by the cognitive unit 14 is generated. As a result, the likelihood of a soccer ball is the highest. That is, the neural network 10 recognizes the input image 51 as an image of a soccer ball.

However, if the person using the image recognition device 1 wants the input image 51 to be recognized as a Dalmatian image, the gaze area should be the area where the Dalmatian is present in this attention map 53.

Therefore, in such a case, a person uses the operating device 2 to correct the gaze area in the attention map 53. Specifically, first, a person erases the gaze area in the attention map 53 by using the operation device 2. For example, a person moves a pointer while pressing a predetermined erase button of the operating device 2 to scan the entire gaze area in the attention map 53. As a result, the processing unit 5 lowers the pixel value of the attention map 53 in the region scanned by the pointer while pressing the erase button to obtain a pixel value that does not serve as the gaze region, as shown in FIG.

After that, the person sets a new region to be the gaze region in the attention map 53 by using the operation device 2. For example, a person moves a pointer while pressing a predetermined additional button of the operating device 2 to scan the entire region to be the gaze region in the attention map 53. As a result, the processing unit 5 raises the pixel value of the attention map 53 in the area scanned by the pointer while pressing the add button, and sets the pixel value to be the gaze area as shown in FIG. In the example of FIG. 7, the new gaze area designated by the person is the Dalmatian face.

By displaying the input image 51 on the attention map 53 and displaying it on the display device 3 in this way, if a person can determine which part of the input image 51 should be the gaze area, the knowledge is efficient. It is often used to easily specify the gaze area in the attention map 53. By such processing in step 140, the attention map 53 acquired in step 130 is modified in the memory 4.

Subsequently, in step 150, the processing unit 5 inputs the attention map 53 corrected in the immediately preceding step 140 to the synthesis unit 13. Then, the synthesis unit 13 synthesizes the feature map 52 and the attention map 53 as described above to generate the synthesis map 54, and inputs the feature map 52 to the first layer of the cognitive unit 14. The cognitive unit 14 into which the synthetic map 54 is input generates a classification result based on the synthetic map 54 as described above. In this classification result, the likelihood of Dalmatian is the highest. That is, the neural network 10 recognizes the input image 51 as a Dalmatian image.

In step 160 following step 150, the processing unit 5 acquires and outputs the classification result generated by the cognitive unit 14 in this way. The output destination may be another device via a communication network (not shown), a memory 4, or a display device 3.

In this way, by modifying the attention map 53 by utilizing the knowledge of the person, the cognitive unit 14 is given a higher weight to the area intended by the person. As a result, image recognition can be performed according to the intention of the person. That is, the recognition result can be adjusted by using the attention map manually modified based on human knowledge.

At this time, the parameters of the neural network 10 have not been changed. That is, it is possible to obtain a recognition result intended by a person without requiring re-learning of the neural network 10.

For example, when a fundus image is input to the neural network 10 as an input image 51, the doctor corrects the gaze area of the attention map 53 by using the knowledge based on his own experience, so that the grade of the eye disease is classified as a class. The identification becomes more accurate. Thus, for example, in medical imaging, the function of this embodiment is useful.

As described above, as shown in FIG. 8, the processing unit 5 of the image recognition device 1 responds to a human correction operation with respect to the attention map 53 generated by the neural network 10 to which the input image 51 is input. Make the correction (step 140). Then, the processing unit 5 outputs the recognition result of the input image 51 generated by the neural network 10 based on the modified attention map 53 and the input image 51 (step 160).

In this way, the attention map 53 is modified by utilizing the knowledge of the person, so that the cognitive unit 14 attaches importance to the area intended by the person. As a result, image recognition can be performed according to the intention of the person. At this time, the parameters of the neural network 10 have not been changed. That is, it is possible to obtain a recognition result intended by a person without requiring re-learning of the neural network 10.

Further, the path through which the image information is propagated to generate the attention map 53 and the path through which the image information is propagated to generate the recognition result are partially shared (that is, the feature extraction unit 11). It is separated by other parts (that is, attention part 12 and cognitive part 14). Then, the synthesis unit 13 inputs the synthesis map 54 reflecting the modified attention map 53 to the side of the recognition unit 14 of the separated portion. As described above, since the input portion of the composite map 54 based on the modified attention map 53 is suitable for the structure of the neural network 10, the degree of improvement in the recognition result by the modified attention map 53 is improved. improves.

Further, the processing unit 5 transparently superimposes the input image 51 on the attention map 53 and displays it on the display device 3, and reflects the correction according to the correction operation of a person on the attention map 53. A person can relatively easily determine which part of the input image 51 should be the gaze area by looking at the input image 51. Therefore, by superimposing the input image 51 on the attention map 53 and displaying it on the display device 3, the person can easily specify the gaze area in the attention map 53 by visually and efficiently utilizing his / her knowledge. it can.

In the present embodiment, the processing unit 5 functions as an input unit by executing step 120, functions as a map correction unit by executing step 140, and functions as an output unit by executing step 160. To do.

(Second Embodiment)
Next, the second embodiment will be described. In the present embodiment, learning parameters such as weights and biases of the neural network 10 are corrected based on the corrected attention map according to the human correction operation. That is, the neural network 10 is relearned based on the modified attention map.

The hardware configuration of this embodiment is the same as that shown in FIG. 1 in the first embodiment. Further, the configuration of the trained neural network 10 stored in the memory 4 is the same as that of the first embodiment. The image recognition device 1 of the present embodiment corresponds to a training device.

One of the differences from the first embodiment of the present embodiment is that the processing unit 5 does not execute the processing of FIG. 4, but instead, the attention map 53 is not modified, and the classification corresponding to the input image 51 is performed. The result is to be generated by the neural network 10.

That is, the processing unit 5 first reads the neural network 10 from the memory 4 as in the first embodiment, then acquires the input image 51, and inputs the input image 51 to the neural network 10.

Then, in the neural network 10, the feature extraction unit 11 and the attention unit 12 function as in the first embodiment, so that the attention map 53 and the classification result are generated by the attention unit 12. The attention map 53 is input to the compositing unit 13 without being corrected by a person, that is, without being corrected. The compositing unit 13 generates a compositing map 54 by compositing the feature map 52 and the attention map 53 that has not been modified by a person. The cognitive unit 14 generates a classification result based on the synthetic map 54 as in the first embodiment. The processing unit 5 acquires and outputs this classification result in the same manner as in the first embodiment.

Then, in addition to the process of acquiring the classification result of the input image 51 from the input image 51 using the neural network 10 as described above, the processing unit 5 performs the process shown in FIG. 9 in order to relearn the neural network 10. Execute. By this re-learning, the neural network 10 is fine-tuned.

The processing unit 5 starts the processing of FIG. 9 based on the fact that a predetermined re-learning start operation is performed on the operating device 2. In this process, the processing unit 5 uses a data set for re-learning. The data set for re-learning has a plurality of groups (10, 100, or 100,000) including the image for learning and the teacher label.

The learning image is data input to the feature extraction unit 11 like the input image 51. The teacher label is data that is the correct answer value of the classification result output from the attention unit 12 and the cognitive unit 14 when the learning image of the same group is input to the feature extraction unit 11.

The data set for re-learning may be generated in advance and recorded in the non-volatile storage medium of the memory 4, or may be acquired from the data server via a communication network (not shown). Further, as the learning image and the teacher label of the data set for re-learning, the same learning data set used at the time of initial learning of the neural network 10 may be diverted, or the same as the learning data set may be used. It may be different.

In the process of FIG. 9, the processing unit 5 first executes the loop processing of steps 210 and 220 for each group included in the re-learning data set. In each loop processing, the processing unit 5 first inputs the learning image in the target group to the feature extraction unit 11 in step 210. Subsequently, in step 220, the attention map 53 and the classification result generated by the attention unit 12 based on the input learning image and the classification result generated by the cognitive unit 14 based on the learning image are acquired and acquired in the memory 4. Record in.

The method in which the neural network 10 outputs the attention map 53 and the two types of classification results based on the input learning image is the same as the above-mentioned method in which the learning image is replaced with the input image 51. After step 220, one loop process is completed.

When the loop processing is completed for the number of groups, the processing of the processing unit 5 proceeds to step 230. At this point, the attention map 53 and the classification result generated by the attention unit 12 and the classification result generated by the cognitive unit 14 are associated with each group in all the retraining data sets and stored in the memory. It is recorded in 4.

In step 230, the processing unit 5 extracts the group in which the erroneous recognition has occurred from the plurality of groups. The class with the highest likelihood in the classification result output by the cognitive unit 14 and the class indicated by the teacher label (that is, the class with the highest likelihood in the teacher label) are extracted as the occurrence of erroneous recognition. This is a group that did not do it. Alternatively, a group in which the class having the highest likelihood in the classification result output by the attention unit 12 and the class indicated by the teacher label do not match may be extracted as a misrecognition. Alternatively, both of them may be extracted. In most cases, there are multiple groups to be extracted.

Subsequently, in step 240, the attention map 53 recorded in the memory 4 corresponding to each of the groups extracted in the immediately preceding step 230 is modified based on human knowledge. Specifically, the attention map 53 is modified based on a person's modification operation on the operating device 2 by the same process as in step 140 of FIG. Then, the processing unit 5 stores the modified attention map 53 as a teacher attention map belonging to the group in the memory 4.

The teacher attention map created in this way is data that is the correct answer value of the attention map 53 output from the attention unit 12 when the learning images of the same group are input to the feature extraction unit 11. By this process, the teacher attention map is added to the data set for retraining.

Subsequently, in step 250, the processing unit 5 retrains the neural network 10 based on the two types of classification results, the attention map 53, and the relearning data set acquired in the process of FIG. 9 this time. As mentioned above, the retraining dataset includes a teacher attention map and a teacher label.

Specifically, as shown in FIG. 10, the weights of the attention unit 12 and the cognitive unit 14 are determined by the error backpropagation method, with the quantity L = Latt + Lper + Lmap consisting of the sum of the three learning errors Latt, Lper, and Lmap as the learning error. Learning parameters such as bias are updated. In FIG. 10, the output layer 14b of the cognitive unit 14 and the portion 14a in front of the output layer 14b of the cognitive unit 14 are shown. In this embodiment, learning parameters such as weights and biases of the feature extraction unit 11 are not updated.

Here, Latt sets an error between the classification result output by the attention unit 12 when the learning image 61 is input to the neural network 10 and the teacher label 60 belonging to the same group as the learning image 61. The amount shown.

Further, Lper determines an error between the classification result output by the feature extraction unit 11 when the learning image 61 is input to the neural network 10 and the teacher label 60 belonging to the same group as the learning image 61. The amount shown.

Further, Lmap determines an error between the attention map 53 output by the attention unit 12 when the learning image 61 is input to the neural network 10 and the teacher label 60 belonging to the same group as the learning image 61. The amount shown.

As the learning error Lmap, an L2 norm error may be adopted as shown in the following equation, or an error of another form may be adopted.
Lmap = γ × || M'-M || ₂
Here, M indicates the value of the attention map 53 output by the attention unit 12 when the learning image 61 is input to the neural network 10. M'indicates the value of the modified attention map corresponding to the same group as the learning image 61. By obtaining an error for each of these two attention map elements, the attention unit 12 is learned so as to output an attention map close to human knowledge.

Here, γ is a coefficient for adjusting the learning error Lmap. Lmap has a larger error value than Latt and Lper. Therefore, by multiplying γ by Lmap, the magnitudes of the three learning errors Lmap, Latt, and Lper can be adjusted. After step 250, the process of FIG. 9 ends and the relearned neural network 10 is recorded in memory 4.

In this way, fine tuning of the neural network 10 is performed based on the attention map modified based on human knowledge, so that the image recognition function of the neural network 10 is improved. That is, when the processing unit 5 inputs various input images 51 to the neural network 10 after fine tuning, the correct answer rate of the recognition result generated by the recognition unit 14 is improved.

As described above, the processing unit 5 retrains the neural network 10 using the data set for retraining (step 250). Then, the data set for re-learning includes a plurality of teacher attention maps.

In this way, when re-learning the neural network 10 that generates the attention map, the teacher attention map that is the correct answer value of the attention map is used. Since the teacher attention map is created based on human knowledge, it is possible to incorporate human knowledge into the learning of the neural network 10 by doing so.

Further, the processing unit 5 acquires a plurality of attention maps corresponding to the plurality of learning images by inputting the plurality of learning images into the neural network 10 (steps 210 and 220). Then, the processing unit 5 modifies the plurality of attention maps to obtain the teacher attention map according to the correction operation of the person (step 240).

In this way, the teacher attention map can be generated based on the correction operation performed by a person on the attention map generated by the neural network 10. Therefore, it is possible to more directly incorporate human knowledge into the learning of the neural network 10. Moreover, the correction operation is easier than when creating a teacher attention map from scratch.

Further, the plurality of teacher attention maps included in the data set for re-learning are only the learning images misrecognized by the neural network 10 before the re-learning. In this way, by using many teacher attention maps corresponding to misrecognized learning images for re-learning, re-learning can be performed with higher efficiency. This is because the attention map generated by inputting the erroneously recognized learning image is likely to have many errors in itself.

Further, the neural network 10 generates a recognition result of the input image 51 based on the input image 51 and the attention map 53. As described above, in the neural network 10 in which not only the input image 51 but also the attention map 53 is fed back as information for image recognition, the recognition result of the input image 51 and the attention map 53 are strongly related. Therefore, in such a neural network 10, the effect of re-learning using the teacher attention map contributes to the improvement of the recognition result of the input image 51 to a high degree.

Further, the processing unit 5 relearns the attention unit 12 without relearning the feature extraction unit 11. In this way, the portion of the neural network 10 that is strongly related to the generation of the attention map 53 is relearned, so that highly efficient fine tuning of the neural network 10 is realized.

In the present embodiment, the processing unit 5 functions as a reading unit by executing step 205, functions as a training unit by executing step 250, and serves as an acquisition unit by executing steps 210 and 220. It functions and functions as a map correction unit by executing step 240.

(Other embodiments)
The present invention is not limited to the above-described embodiment, and can be appropriately modified. Further, the above-described embodiments are not unrelated to each other, and can be appropriately combined unless the combination is clearly impossible. Further, in each of the above embodiments, the elements constituting the embodiment are not necessarily essential except when it is clearly stated that they are essential and when it is clearly considered to be essential in principle. Further, in each of the above embodiments, when numerical values such as the number, numerical values, amounts, and ranges of the constituent elements of the embodiment are mentioned, when it is clearly stated that they are particularly essential, and in principle, the number is clearly limited to a specific number. It is not limited to the specific number except when it is done. Further, when a plurality of values are exemplified for a certain quantity, it is also possible to adopt a value between the plurality of values unless otherwise specified or when it is clearly impossible in principle. ..

In addition, the present invention also allows the following modifications and equal range modifications for each of the above embodiments. In addition, the following modified examples can be independently selected to be applied or not applied to the above embodiment. That is, any combination of the following modifications can be applied to the above embodiment.

(Modification example 1)
The image recognition device 1 has a function of the first embodiment (that is, image recognition using an attention map modified based on human knowledge) and a function of the second embodiment (that is, modified based on human knowledge). It may have both functions (re-learning using an attention map).

(Modification 2)
In the above embodiment, the classification result is given as an example of the recognition result output by the attention unit 12 and the cognitive unit 14. However, the recognition result output by the attention unit 12 and the cognitive unit 14 is not limited to the classification result, but may be the result by regression. That is, the image recognition performed by the neural network 10 may be classification or regression.

(Modification 3)
In the first embodiment, the neural network 10 has a feature extraction unit 11, an attention unit 12, a synthesis unit 13, and a cognitive unit 14. However, the neural network for realizing image recognition using the attention map modified based on human knowledge is not limited to such a configuration. That is, in the case of a neural network that generates an attention map based on the input image and generates an image recognition result based on the image and the attention map, the image recognition function is improved by modifying the attention map. Can be done.

(Modification example 4)
In the second embodiment, the neural network 10 has a feature extraction unit 11, an attention unit 12, a synthesis unit 13, and a cognitive unit 14. However, it may be modified based on human knowledge. However, the neural network for realizing re-learning using the attention map modified based on human knowledge is not limited to such a configuration. That is, in the case of a neural network that generates an attention map and an image recognition result based on an input image, the image recognition function can be improved by re-learning using the modified attention map. For example, a neural network such as CAM (Class Activation Mapping) described in Non-Patent Document 2 may be relearned using an attention map modified based on human knowledge.

(Modification 5)
In the first embodiment, the processing unit 5 transparently superimposes the input image 51 on the attention map 53 and displays it on the display device 3, and reflects the correction according to the human correction operation on the attention map. ing. However, it is not always necessary to do this. For example, the processing unit 5 may reflect the correction according to the correction operation of a person on the attention map in a state where the input image 51 and the attention map 53 are displayed side by side on the display device 3 without overlapping. Further, for example, the processing unit 5 may reflect the correction according to the correction operation of a person on the attention map in a state where the attention map 53 is displayed on the display device 3 and the input image 51 is not displayed on the display device 3.

(Modification 6)
In the first and second embodiments, as a method of modifying the attention map, a method of changing only the values of some pixels in the attention map generated by the attention unit 12 and not changing the values of the remaining pixels is shown. ing. That is, a method of making changes to the attention map generated by the attention unit 12 is shown.

However, the method of modifying the attention map is not necessarily limited to such a method. For example, a new attention map may be created from scratch in addition to the attention map output by the attention unit 12 when the image is input to the neural network 10. In this case, in the first embodiment, the new attention map is input to the synthesis unit 13, and in the second embodiment, the new attention map becomes the teacher attention map.

As a method of creating a new attention map, for example, there are the following methods. First, a person determines the position range of the gaze area by looking at the image input to the neural network 10. Then, a person may operate a computer to create a new attention map that reflects the position range of the gaze area determined. This computer may be an image recognition device 1 or another device.

(Modification 7)
In the second embodiment, the teacher attention map used for the re-learning is only the teacher attention map corresponding to the learning image misrecognized by the neural network 10 before the re-learning. However, the teacher attention map used for relearning may include a teacher attention map corresponding to the learning image correctly recognized by the neural network 10 prior to relearning.

Even in that case, if the number of teacher attention maps corresponding to the misrecognized learning image is larger than the number of teacher attention maps corresponding to the correctly recognized learning image, the efficiency of re-learning can be improved. ..

Alternatively, the number of teacher attention maps corresponding to the misrecognized learning image may be less than the number of teacher attention maps corresponding to the correctly recognized learning image.

(Modification 8)
In the above embodiment, in the re-learning of the neural network 10, the feature extraction unit 11 is not re-learned, and only the attention unit 12 and the cognitive unit 14 are re-learned. The re-learning of the neural network 10 is not limited to this form. For example, the feature extraction unit 11 and the cognitive unit 14 may not be relearned, and only the attention unit 12 may be relearned. Further, for example, only the feature extraction unit 11 may be relearned, and the attention unit 12 and the cognitive unit 14 may not be relearned. Further, for example, the feature extraction unit 11 and the cognitive unit 14 may be relearned, and the attention unit 12 may not be relearned.

Further, a form in which the feature extraction unit 11, the attention unit 12, and the cognitive unit 14 are relearned is also allowed. In this case, the re-learning of the neural network 10 is not fine tuning.

(Modification 9)
In the above embodiment, the re-learning is performed by using the error back-propagation method using three learning errors of Lmap, Latt, and Lper. However, it is not necessary to use all of Lmap, Latt, and Lper. For example, only Lmap may be used.

1 ... image recognition device, 2 ... operation device, 3 ... display device, 4 ... memory, 5 ... processing unit, 10 ... neural network, 11 ... feature extraction unit, 12 ... attention unit, 13 ... synthesis unit, 14 ... cognitive unit , 51 ... Input image, 52 ... Feature map, 53 ... Attention map, 54 ... Composite map, 60 ... Teacher label, 61 ... Learning image

Claims (8)

An input unit (120) for inputting an image (51) into the neural network (10),
When the neural network generates an attention map (53) expressing the gaze area of the input image, the map correction unit (140) corrects the generated attention map according to a human correction operation. )When,
Image recognition including an output unit (160) that outputs the generated recognition result when the neural network (10) generates the recognition result of the image based on the modified attention map and the image. apparatus. The neural network includes a feature extraction unit (11), an attention unit (12), a synthesis unit (13), and a cognitive unit (14).
The feature extraction unit generates a feature map (52) including the features of the image by including the plurality of convolution layers and propagating the information of the image through the plurality of convolution layers.
The attention unit generates the attention map based on the feature map.
The compositing unit synthesizes the feature map and the modified attention map to generate a compositing map (54).
The image recognition device according to claim 1, wherein the recognition unit generates the recognition result based on the synthetic map. The map correction unit is characterized in that, in a state where the image is transparently superimposed on the attention map and displayed on the display device (3), the correction according to the correction operation is reflected in the attention map. The image recognition device according to claim 1 or 2. An attention map (53) expressing the gaze area of the input image (51) and a reading unit (205) for reading out a neural network (10) learned to generate a recognition result of the image, and
A training unit (250) for retraining the neural network using a data set for retraining is provided.
The data set for retraining is a training device including a plurality of teacher attention maps which are correct values of the attention maps when a plurality of images are input to the neural network. Acquisition units (210, 220) that acquire a plurality of attention maps corresponding to the plurality of learning images by inputting a plurality of learning images into the neural network.
The training device according to claim 4, further comprising a map correction unit (240) that modifies the plurality of attention maps to be a teacher attention map according to a person's correction operation. In the plurality of teacher attention maps included in the data set for re-learning, the teacher attention map corresponding to the learning image misrecognized by the neural network before the re-learning is performed by the neural network before the re-learning. The training device according to claim 4 or 5, which is more than a teacher attention map corresponding to a correctly recognized learning image. The training device according to any one of claims 4 to 6, wherein the neural network generates a recognition result of the image based on the image and the attention map. The neural network includes a feature extraction unit (11), an attention unit (12), a synthesis unit (13), and a cognitive unit (14).
The feature extraction unit generates a feature map (52) including the features of the image by including the plurality of convolution layers and propagating the information of the image through the plurality of convolution layers.
The attention unit generates the attention map based on the feature map.
The compositing unit synthesizes the feature map and the attention map to generate a compositing map (54).
The cognitive unit generates the recognition result based on the synthetic map.
The training device according to any one of claims 4 to 7, wherein the training unit relearns the attention unit without relearning the feature extraction unit.