JP2020008896A - Image identification apparatus, image identification method and program - Google Patents

Abstract

To improve an identification accuracy of an image by a convolutional neural network (CNN).SOLUTION: An image identification apparatus 100 includes a CNN identifier 11 that has an intermediate layer that is a layer other than an input layer which receives an input image and an output layer which outputs an identification result of the input image and that identifies the input image, feature map acquisition means that acquires feature map information for identifying the input image in the intermediate layer, activation map generation means that generates an activation map in which an activation state of the intermediate layer is visualized, edit means that refers to the activation map generated by the activation map generation means and edits the feature map acquired by the feature map acquisition means, and identification means that identifies the input image using the feature map edited by the edit means.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image identification device, an image identification method, and a program.

  Since a convolutional neural network (Convolutional Neural Network: CNN) exhibits high performance in image analysis, an apparatus for identifying an image using the CNN has been developed. And, technology for improving the accuracy of image identification by the CNN is also being developed. For example, Patent Literature 1 discloses a technique capable of improving identification accuracy when specialized for a specific identification target.

JP 2014-203135 A

  The technology disclosed in Patent Literature 1 enables specific identification by enabling the backpropagation (error propagation) method, which is a CNN learning method, to be performed even when an encoding process using a discontinuous function is performed. The accuracy of discrimination when specialized for the target is improved. However, in such a conventional technique, since the internal processing of the CNN is treated as a black box, there is room for improvement in the identification accuracy.

  SUMMARY An advantage of some aspects of the invention is to provide an image identification device, an image identification method, and a program capable of improving the accuracy of image identification by a CNN as compared with the related art.

In order to achieve the above object, an image identification device of the present invention comprises:
An input device to which an input image is input and an intermediate layer which is a layer other than an output layer from which an identification result of the input image is output, and an identifier for identifying the input image,
In the intermediate layer, a feature map obtaining means for obtaining a feature map for identifying the input image,
Activation map generation means for generating an activation map visualizing the activation state of the intermediate layer,
Editing means for editing the feature map obtained by the feature map obtaining means, with reference to the activation map generated by the activation map generating means,
Identification means for identifying the input image using the feature map edited by the editing means,
Is provided.

  ADVANTAGE OF THE INVENTION According to this invention, the identification accuracy of the image by CNN can be improved.

FIG. 2 is a diagram illustrating a functional configuration of the image identification device according to the first embodiment of the present invention. It is a figure explaining the outline of processing of a convolutional neural network (CNN). FIG. 9 is a diagram illustrating a specific example of a CNN convolution process and a pooling process. It is a figure explaining calculation of the output by CNN. FIG. 4 is a diagram illustrating an activation map generation method using simple averaging. FIG. 3 is a diagram illustrating an activation map generation method using a CAM. It is a figure explaining the activation map generation method by Grad-CAM. It is a figure explaining the activation map in the layer near the input side of CNN, and the layer near the output side. 5 is a flowchart of an image identification process according to the first embodiment. FIG. 4 is a diagram illustrating a specific example of the image identification processing according to the first embodiment. 13 is a flowchart of an image identification process according to a first modification. FIG. 14 is a diagram illustrating an example in which an input image and an activation map are displayed in an overlapping manner in the image identification processing according to Modification 1.

  Hereinafter, an image identification device and the like according to an embodiment of the present invention will be described with reference to the drawings. In the drawings, the same or corresponding parts have the same reference characters allotted.

(Embodiment 1)
The image identification device 100 according to the first embodiment of the present invention identifies an unknown image using a CNN identifier trained using a learning image. When identifying an unknown image, the image identification device 100 can improve the accuracy of image identification by deleting information of an area estimated to be unnecessary for image identification in the intermediate layer of the CNN identifier. . Such an image identification device 100 will be described below.

  As shown in FIG. 1, the image identification device 100 according to the first embodiment includes a control unit 10, a storage unit 20, an image input unit 31, an output unit 32, a communication unit 33, and an operation input unit 34.

  The control unit 10 is configured by a CPU (Central Processing Unit) or the like, and executes programs stored in the storage unit 20 to thereby control each unit (CNN discriminator 11, unnecessary area acquisition unit 12, unnecessary area deletion unit 13) described later. ).

  The storage unit 20 includes a ROM (Read Only Memory), a RAM (Random Access Memory), and the like, and stores a program executed by the CPU of the control unit 10 and necessary data.

  The image input unit 31 is a device for inputting image data for learning or image data to be identified (unknown). The control unit 10 acquires image data via the image input unit 31. As the image input unit 31, any device can be used as long as the control unit 10 can acquire image data. For example, when image data is stored in the storage unit 20 and the control unit 10 acquires the image data by reading out the storage unit 20, the storage unit 20 also serves as the image input unit 31. When the control unit 10 acquires image data from an external server or the like via the communication unit 33, the communication unit 33 also serves as the image input unit 31.

  The output unit 32 is a device for the control unit 10 to output a result of identifying an image input from the image input unit 31, an activation map described later, and the like. For example, the output unit 32 is a liquid crystal display or an organic EL (Electro-Luminescence) display. However, the image identification device 100 may include these displays as the output unit 32, or may include the output unit 32 as an interface for connecting an external display. When the image identification device 100 includes the output unit 32 as an interface, the identification result and the like are displayed on an external display connected via the output unit 32. The output unit 32 functions as an output unit.

  The communication unit 33 is a device (such as a network interface) for transmitting and receiving data to and from another external device (for example, a server that stores a database of image data). The control unit 10 can acquire image data via the communication unit 33.

  The operation input unit 34 is a device that receives a user's operation input to the image identification device 100, and is, for example, a keyboard, a mouse, a touch panel, or the like. The image identification device 100 receives an instruction or the like from a user via the operation input unit 34. The operation input unit 34 functions as an operation input unit.

  Next, the function of the control unit 10 will be described. The control unit 10 realizes the functions of the CNN discriminator 11, the unnecessary area acquisition unit 12, and the unnecessary area deletion unit 13.

  The CNN discriminator 11 is an image discriminator using a convolutional neural network (CNN). The control unit 10 also functions as a CNN discriminator 11 when the control unit 10 executes a program for realizing the discriminator by the CNN. The CNN discriminator 11 includes an input layer to which an input image is input via the image input unit 31, an output layer to which an identification result of the input image is output, an intermediate layer that is a layer other than the input layer and the output layer, And outputs the result of identifying the input image from the output layer. The outline of the process of image identification by the CNN will be described later.

  The unnecessary area obtaining unit 12 obtains an unnecessary area in the intermediate layer of the CNN included in the CNN discriminator 11 which is estimated not to be used when identifying an input image. The unnecessary area obtaining unit 12 functions as an unnecessary area obtaining unit.

  The unnecessary area deletion unit 13 deletes the information of the unnecessary area acquired by the unnecessary area acquisition unit 12 from the CNN intermediate layer of the CNN identifier 11. The unnecessary area deletion unit 13 functions as an unnecessary area deletion unit. Further, the process of “acquiring an area to be deleted and deleting the information of the area” is considered as a kind of editing processing. Therefore, the unnecessary area acquiring unit 12 and the unnecessary area deleting unit 13 constitute an editing unit. Is done.

  The functional configuration of the image identification device 100 has been described above. Next, an outline of the CNN will be described. CNN is a neural network that mimics the function of nerve cells in the human visual cortex, and its prototype is neocognitron. The CNN differs from a general forward-propagation type neural network in that not only a fully connected layer but also a convolutional layer and a pooling layer are included as intermediate layers. Features are extracted. Then, in the output layer, the identification result of the input image is stochastically expressed. A typical process outline of N class identification by the CNN will be described with reference to FIG.

  As shown in FIG. 2, in the N-class identification processing by the CNN, a convolution process (scanning of a filter) and a pooling process (scanning of a window) are performed on the input image 111 to calculate a feature map having a gradually smaller size. This is a process of finally obtaining an output 118. The layer storing the input image 111 is also called an input layer, and the layer storing the output 118 is also called an output layer. In the example illustrated in FIG. 2, the convolution processing filters 121, 123, 124, 125 and the pooling processing windows 122, 126 are gradually scanned in the vertical and horizontal directions by stride 2 on the input image 111. A small-sized feature map is calculated, and an output 118 is finally obtained. Note that “scanning by stride 2” means scanning by skipping one pixel or element constituting a feature map.

  A weighting factor is assigned to each element of the filters 121, 123, 124, and 125. By scanning on the input image or the feature map having the same number of channels in the plane direction, each attention of the input image or the feature map is obtained. At the point, the inner product result of the scalar values is output and a new feature map is obtained. Then, by applying a plurality of filters (n), filter maps of the number (n channels) can be obtained. Also, each time scanning is performed with stride 2, the size of the feature map becomes half the size both vertically and horizontally. As a result, more general feature extraction is performed (the filter size is relatively enlarged with respect to the feature map size) in the later stage.

  When scanning an input image or a feature map with a filter larger than 1 × 1, if the filter area is set so as not to protrude from the input image or the feature map, a point of interest at the time of scanning is set at an end of the input image or the feature map. Must be set to an inside point, and the output feature map is smaller than the size of the original input image or feature map. Therefore, padding 0 data or the like to the outside of the input image or the feature map by a necessary amount (for example, in the case of a 7 × 7 filter, three elements) according to the filter size is performed. Thereby, the outermost peripheral portion of the input image or the feature map can also be set as a point of interest.

  In addition, the filter output value (scalar value) is usually set to a negative value of 0 by applying an activation function ReLU (Rectified Linear Unit: y = max (x, 0)). However, in recent years, before applying the activation function ReLU, it has become common to perform Batch Normalization (a process of normalizing a filter output value so that an average becomes 0 and a variance becomes 1). This is because, when Batch Normalization is performed, it is possible to correct the bias of the degree of activity, and it is possible to accelerate learning. Note that the value of each element of the feature map is also referred to as the degree of activation, and an increase in the value is also referred to as activation.

  The convolution process using a filter and the pooling process using a window will be described with a specific example with reference to FIG. Here, the input image 110 is an image of 8 × 8 × 1 channel (black and white), the filter 120 is a filter of 3 × 3 × 1 channel, and the window 131 is a window of 3 × 3 size which returns the maximum value in the area. An example will be described. Scanning is performed by stride 2 both vertically and horizontally.

  When the filter 120 is scanned with the stride 2 on the input image 110 (0 is padded outside the input image), a feature map 130 is obtained. For example, when the point at the upper left of the input image 110 is set as the point of interest, the filter 120 is applied, and the five points that are protruding (padded) to the upper left, the point of interest, and the values of the right and lower points of the point of interest Is 0, the value remains at 0 even when the filter 120 is applied, and the value of the lower right point of the target point is 1 but the value of the lower right point of the filter 120 is 0. Go from the left end to the right, and if it reaches the right end, calculate the next lower row from the left end to the right end.) 0 = 0, and the filter output value becomes 0. Therefore, the value of the top left point of the feature map 130 is 0.

  Since scanning is performed with stride 2, the next point of interest is the third point from the left of the top row of the input image 110. When the filter 120 is applied to this point, the values of the three points (padded) protruding above, the point of interest, and the points on the left and right of the point of interest remain at 0, so that even when the filter 120 is applied, the value remains at 0. , In the same order as described above, 0 × 0 + 0 × 1 + 0 × 0 + 0 × 1 + 0 × (−4) + 0 × 1 + 1 × 0 + 1 × 1 + 1 × 0 = 1, and the filter output value becomes 1. Therefore, the value of the second point from the left of the top row of the feature map 130 is 1. For example, when the values of the second line from the top and the second point from the left of the feature map 130 are calculated in the same order as above, 1 × 0 + 1 × 1 + 1 × 0 + 1 × 1 + 2 × (−4) + 2 × 1 + 1 × 0 + 2 X1 + 3x0 = -2, but when the activation function ReLU is applied to -2, it becomes 0, and therefore it is 0 in FIG. (Note that, in FIG. 3, Batch Normalization is not performed for easy understanding.)

  Next, a specific example of the pooling process will be described with reference to FIG. When the Max Pooling window 131 is scanned with the stride 2 on the input image 110, a feature map 132 is obtained. For example, when the point at the upper left of the input image 110 is set as the point of interest, the window 131 is applied, and the five points protruding (padded) to the upper left, the point of interest, and the values of the right and lower points of the point of interest Is 0 and the value of the lower right point of the point of interest is 1, so the maximum value of these 9 points is 1 and the window output value is 1. Therefore, the value of the top left point of the feature map 132 is 1. Other points can be similarly obtained. Although not described in FIG. 3, the Average Pooling window 126 in FIG. 2 is a window that outputs an average value of scalar values of 49 points in a 7 × 7 window area.

  Returning to FIG. 2, the final layer (feature map 117) of the intermediate layer of the CNN and the output layer (output 118) are connected by a fully connected connection 127, and weighted addition is performed as in a normal neural network. The final layer of the intermediate layer of the CNN is also called an all-coupling layer because it is connected to the output layer by a full-coupling connection 127. In this example, since the N class is identified, the output 118 has N elements (or units), and the magnitude of the element represents the magnitude of the inference probability.

  In the CNN, the weight coefficient assigned to each connection of the all-connection connection 127 and the weight coefficients of the filters 121, 123, 124, and 125 can be acquired using learning data prepared in advance. More specifically, learning data is input as an input image, forward propagation described below is performed, a difference (error) between an output result and a correct answer (correct identification result of the input learning data) is obtained, and an error back propagation method is performed. To update the weighting coefficient in a direction to reduce the error. This operation is repeatedly performed while lowering the learning rate (the amount of update of the weighting coefficient in the backpropagation method) to converge the value of the weighting coefficient.

  After each weight coefficient of the CNN is trained with the learning data, the unknown image data is forward-propagated as input image data, so that the final output result is obtained as an inference value for identifying the input image.

  A specific example of the forward propagation process will be described with reference to FIG. In the example of FIG. 2, the input image 111 is a square having a size of 224 × 224 pixels, and one pixel includes three channels of RGB (Red, Green, Blue). The value of one channel (each of RGB) of each pixel is converted to a signed representation centered on 0 by subtracting the average value of the image database from an 8-bit integer absolute value representation (0 to 255) of a general pixel value representation. Shall be done.

  A feature map 112 is obtained by scanning the input image 111 with a filter 121 having a size of 7 × 7 × 3 channels in a vertical direction and a horizontal direction with a stride 2. In the above description with reference to FIG. 3, the input image 110 and the filter 120 each have one channel, but since the input image 111 and the filter 121 each have three channels, three channels obtained by performing the operation described in FIG. The value obtained by applying the activation function ReLU to the sum of the scalar values is the value of each point in the feature map 112. Since 64 filters 121 are prepared, 64 feature maps 112 (channels) are obtained. Since the feature map 112 is a value obtained by scanning the input image 111 having a size of 224 × 224 pixels both vertically and horizontally by stride 2, the size of the input image 111 in both the vertical and horizontal directions is １ ／ (112 × 112).

  A feature map 113 is obtained by scanning each channel of the feature map 112 with a 3 × 3 Max Pooling window 122 in the vertical and horizontal directions with stride 2 respectively. As described with reference to FIG. 3, the Max Pooling window 122 outputs the maximum value in this 3 × 3 area, and thus has a function of absorbing fluctuations in minute positions in the input image. Since the feature map 113 is a value obtained by scanning the feature map 112 having a size of 112 × 112 in both the vertical and horizontal directions by stride 2, the size of the feature map 112 in both the vertical and horizontal directions is × (56 × 56). Since the Max Pooling window 122 is scanned for each channel of the feature map 112 and the obtained output is used as data of the same channel of the feature map 13, the number of channels of the feature map 113 is the same as that of the feature map 112, and is 64. Channel.

  The feature map 113 is scanned by the filter 123 having a size of 3 × 3 × 64 channels in the vertical direction and the horizontal direction with the stride 2 to obtain the feature map 114 (in the same manner as the above-described feature map 112 is obtained). Is obtained. Since 128 filters 123 are prepared, 128 feature maps 114 (channels) are obtained. Since the feature map 114 is a value obtained by scanning the feature map 113 having a size of 56 × 56 in both the vertical and horizontal directions by the stride 2, the size of the feature map 113 in both the vertical and horizontal directions is １ ／ (28 × 28).

  The feature map 114 is scanned by a filter 124 having a size of 3 × 3 × 128 channels in the vertical direction and the horizontal direction in the stride 2 respectively, so that the feature map 114 is obtained (in the same manner as the above-described feature maps 112 and 114 are obtained). A map 115 is obtained. Since 256 filters 124 are provided, 256 feature maps 115 (channels) are obtained. Since the feature map 115 is a value obtained by scanning the feature map 114 having a size of 28 × 28 in both the vertical and horizontal directions by the stride 2, the size of the feature map 114 in both the vertical and horizontal directions is 1/2 (14 × 14).

  The feature map 115 is scanned by the filter 125 having a size of 3 × 3 × 256 channels with the stride 2 in the vertical direction and the horizontal direction, respectively (in the same manner as the above-described feature maps 112, 114, and 115 are obtained). 3.) A feature map 116 is obtained. Since 512 filters 125 are provided, 512 feature maps 116 (channels) are obtained. Since the feature map 116 is a value obtained by scanning the feature map 115 having a size of 14 × 14 in both the vertical and horizontal directions by the stride 2, the size of the feature map 115 in both the vertical and horizontal directions is 1/2 (7 × 7). As described above, by reducing the size of the feature map, the relative coverage area of the filter during the convolution process is expanded, so that more global and abstract features can be captured. In addition, by increasing the channel size (the number of filters), it becomes possible to cope with diversification of features due to combinations of local features.

  By scanning an average pooling window 126 having a size of 7 × 7 for each channel of the feature map 116, a feature map 117 is obtained. The average pooling window 126 outputs the average value in the 7 × 7 area. Since the size of each channel of the feature map 116 is 7 × 7, the average value of all elements of each channel is eventually equal to that of the feature map 117. Become. Therefore, the feature map 117 can be treated as a 512-dimensional vector (feature vector) composed of 512 (channel) scalar values. This operation is called Global Average Pooling because the total average is taken within each channel.

The feature map 117 is connected to the output 118 by a full connection connection 127, and the values of the N elements of the output 118 are obtained according to the weighting factors given to each connection. More specifically, as shown in FIG. 4, when each element of the feature map 117 (feature vector) is represented by h ₁ to h ₅₁₂ , a value A _i corresponding to each element of the output 118 is weighted to each element h _j . coefficient _{w i,} computed by taking the sum over the _j, it Softmax processing (the output value range of each element of the processing. the output 118 of applying a Softmax function using the calculation of _{y n} in FIG. 4 [0 , 1] to obtain a probabilistic expression, where the exp function serves to convert a value including a negative value into a monotonically increasing value from 0.) The output 118 has N values y _{1 to} y _N. Each of the N normalized values y _{1 to} y _N is an inference value (probabilistic expression) of the corresponding class. Briefly, if the element having the largest value among y _{1 to} y _N is y _i , the input image is identified as class i.

  The outline of the typical process of the N class identification by the CNN has been described above. By analyzing the state of each intermediate layer (each feature map) of the CNN described above, it is considered that a method of improving the identification accuracy by the CNN can be examined. However, since each feature map is composed of a large number of channels, it is difficult to grasp the state at a glance. Therefore, an attempt has been made to make it easy to interpret the internal state of the CNN by visualizing a large number of channels in the feature map.

  For example, if the value (scalar value) of the [x, y] element of the feature map is 0, the color of the pixel at the position corresponding to [x, y] is set to black, and as the value of the [x, y] element increases, [ [x, y], by changing the color of the pixel at the position corresponding to black → purple → blue → light blue → green → yellow green → yellow → orange → red etc. and displaying the activated state of the feature map ( It is possible to visualize which part of the element has a large value.

  However, in the drawings after FIG. 5 described below, the activated state of the feature map is represented by black and white. In these drawings, if the value (scalar value) of the [x, y] element of the feature map is 0, the color of the square at the position corresponding to [x, y] is white, and the value of the [x, y] element is large. The activation state of the feature map is visualized by shading the square color at the position corresponding to [x, y] as darker as possible.

  The visualization of the activation state of the feature map is referred to as an activation map here. Further, an area where the element (scalar value) has a large value in the feature map is referred to as an activated area (activated area). There are various methods for generating the activation map. A typical method will be described here.

  The first method is, as shown in FIG. 5, a method in which a feature map is simply added in the channel direction to obtain an activation map (after adding, the sum is divided by the number of channels to obtain a simple average). May be). FIG. 5 illustrates an example in which a 7 × 7 size feature map 116 is visualized when an input image 111 including a cat and a rabbit is input to the CNN discriminator 11. In FIG. 5, the activation map 140 is generated by adding the feature maps 116 for 512 channels. In FIG. 5, the feature map 116-1 indicates the first channel of the feature map 116, and the feature map 116-512 indicates the 512th channel of the feature map 116. In FIG. 5, the activation map 140 is generated by visualizing the characteristic map 116. However, the activation map may be generated in the same manner for other characteristic maps (for example, the characteristic maps 112 and 114). Can be. In this method, an activation map in which the reactions of all the classes are summed up is obtained, so that it is not possible to distinguish which class of the activation state is due to the reaction.

In the second method, as shown in FIG. 6, each channel of the feature map is assigned a weighting factor (wi, _{wi, i} ) of the all-connection connection 127 when calculating the output (yi) of the activation target class (i) _{. In this method} , the weighted and added values in _j ) are used as the activation map of the class i. This method is called CAM (Class Activation Mapping). FIG. 6 illustrates an example in which, when an input image 111 including a cat and a rabbit is input to the CNN discriminator 11, a class corresponding to the cat is set as a target class and a feature map 116 having a size of 7 × 7 is visualized. Is shown.

In FIG. 6, the weighting coefficient w _{i, j} (j is a channel number) for obtaining the output of class i of the fully connected connection 127 is multiplied by the j-th channel of the feature map 116, and this is added for 512 channels, The activation map 141 of the class i is generated. In FIG. 6, the feature map 116-1 indicates the first channel of the feature map 116, and the feature map 116-512 indicates the 512th channel of the feature map 116. In this method, an activation map is obtained for each class. However, due to the calculation principle, the output map is a feature map on the output side, and the feature map becomes a one-dimensional feature vector by Global Average Pooling, and the vector is fully connected to the output. This can be applied only in the configuration of

The third method, as shown in FIG. 7, is reversely propagate only activated target class output y _c, was determined by be regarded as the contribution of the output of the target class gradient magnitude, characterized This is a method in which a weighted coefficient (α _{c, k} ) of each channel of the map and added are weighted as an activation map of class c. This method is called Grad-CAM (Gradient-weighted Class Activation Mapping). FIG. 7 illustrates an example in which, when an input image 111 including a cat and a rabbit is input to the CNN discriminator 11, a 7 × 7 size feature map 116 is visualized using a class corresponding to the cat as a target class. Is shown. Since the feature map 116 has 512 channels, the feature map 116 corresponding to the k-th channel is represented by _Mk . Since M _k has a size of 7 × 7, elements of [i, j] of M _k (i and j are integers of 1 to 7) are represented by M _k [i, j].

In the Grad-CAM, first, an input image 111 is normally inferred by a CNN discriminator. And among the obtained output 118, the output of the activation target class _{(c) (y} c) only one, others _{(y n (n ≠ c)} ) to 0, the gradient (∂y _c / ∂M _k [I, j]) is calculated. Then, the average of the gradient is calculated for each channel, and is set as a weighting coefficient α _{c, k} . Note that Z in the expression of α _{c, k} in FIG. 7 is the number of elements of each channel of the feature map 116, and here, Z = 7 × 7 = 49. Then, the activation map 142 of the target class k is generated by multiplying the value M _k of the feature map by a weight coefficient α _{c, k} and adding the values. In this method, unlike the CAM, an activation map can be generated not only for the feature map 116 but also for other feature maps (for example, the feature maps 112 and 114) in a similar manner.

  In the above-described Grad-CAM, the contribution of the channel to each class can be made clear by averaging the gradient in the feature map for each channel. However, if it is not necessary to clarify the difference between the classes, it is also possible to visualize the contribution in pixel units by backpropagating the CNN to the input layer only for positive values without taking the average of the gradient. This method is referred to as Guided Backpropagation (indicated by the arrow from the output 118 in FIG. 7 to the Guided Backpropagation image 143).

  In Guided Backpropagation, the same level of resolution as the input image can be obtained, but the difference between the classes is not clear. Therefore, there is a method of visualizing the feature map by superimposing the output of the Grad-CAM on the result of Guided Backpropagation. This method is called Guided Grad-CAM, and it is possible to clearly distinguish feature points for each class and to visualize feature points with high resolution. FIG. 7 shows a state in which an image 143 of Guided Backpropagation is combined with an activation map 142 based on Grad-CAM to obtain an activation map 144 based on Guided Grad-CAM.

  The activation map generation method has been described above. When the activation map is generated in each intermediate layer, as shown in FIG. 8, in a layer near the input side of the CNN (an activation map 145 in which the feature map 112 is visualized), edge extraction is performed, and the layer approaches the output side of the CNN. (The activation map 146 in which the feature map 114 is visualized and the activation map 147 in which the feature map 116 is visualized), it can be understood that the feature of a more complicated large area is extracted.

  Each of the activation maps (activation maps 145, 146, and 147 obtained by visualizing the feature maps 112, 114, and 116) of FIG. 8 uses the Grad-CAM for the input image 111 in which a rabbit and a cat are captured. 8 shows an activation area of each layer when a class for identifying a cat is set as an activation target class. (However, since the size of the feature map decreases as approaching the output side, each activation map in FIG. 145, 146, and 147 resize and display each size to the input image size). Then, in the layer closest to the output side (the activation map 147 in which the feature map 116 is visualized), the portion corresponding to the position (lower left) where the class (cat) to be identified is shown in the input image 111 is activated. (That portion of the feature map has a large positive value).

  In the CNN, since the filter processing of the small area is repeated, features that are far apart on a plane are not integrated and evaluated unless they are at a later stage. Conversely, the locality of the features extracted in the intermediate layer is maintained to a certain degree. Then, the CNN actively and non-linearly truncates features not involved in discrimination (negative filter output) with the activation function ReLU, and locally integrates the remaining features (not related to discrimination) without being truncated. Operation is repeatedly performed. Therefore, by extracting the activation area at the subsequent stage, the global attention area of the CNN discriminator in the input image can be known.

  Next, the content of the image identification processing performed by the image identification device 100 will be described with reference to FIG. The image identification process is started when the user instructs the image identification device 100 to start the image identification process via the operation input unit 34.

  First, the control unit 10 of the image identification device 100 causes the CNN identifier 11 to learn from a large amount of learning image data (step S101). Step S101 may be performed before starting the image identification processing.

  Next, the control unit 10 inputs an unknown image to the CNN discriminator 11 via the image input unit 31 (Step S102). Step S102 is also called an image input step.

  Then, the control unit 10 extracts an activation area of the intermediate layer (for example, the feature map 116 in FIG. 2) of the CNN discriminator 11 to obtain an activation map (step S103). Step S103 includes a feature map obtaining step of obtaining a feature map, and an activation map generating step of generating an activation map from the obtained feature map. The control unit 10 functions as a feature map obtaining unit in the feature map obtaining step, and functions as an activation map generating unit in the activation map generating step.

  In step S103, for example, as described above, the control unit 10 calculates an activation map obtained by simply averaging the feature map in the channel direction. In this step, an activation map (for example, an image as shown in the activation map 147 of FIG. 8) may be presented to the user together with the input image (displayed on the output unit 32). If CAM or Grad-CAM is used as an activation area extraction method instead of simple averaging in the channel direction of the feature map, a feature map (activation map) for each class can be obtained. What is necessary is just to show the map (activation map) an average of all classes to the user.

  Next, the control unit 10 determines whether or not there is an unnecessary area in the activation map which is estimated not to be used for image identification (step S104). In this step, the control unit 10 may determine by obtaining from the user via the operation input unit 34 information on the presence / absence of an unnecessary area obtained by the user confirming the activation map presented in step S103. Then, whether the vicinity of the outer edge of the activation map (for example, the first and second regions from the end in all directions) is activated (the scalar value is a large value (for example, a positive value)) It may be determined whether or not there is an unnecessary area. Further, the determination may be made by a machine learning-based classifier created based on unnecessary area information or the like provided by the user. In this case, the classifier based on the machine learning functions as an automatic editing unit.

  If there is no unnecessary area in the activation map that is not estimated to be used for image identification (step S104; No), the control unit 10 performs image identification with the CNN identifier 11 as it is (step S107). The image identification processing ends.

  If there is an unnecessary area in the activation map that is estimated not to be used for image identification (step S104; Yes), the unnecessary area obtaining unit 12 determines the unnecessary area (position, size, shape, etc.). It is acquired (step S105). If the information on the presence or absence of the unnecessary area has been obtained from the user in step S104, the unnecessary area obtaining unit 12 obtains the unnecessary area from the user via the operation input unit 34. If it is determined in step S104 that there is an unnecessary area based on the fact that the vicinity of the outer edge of the activation map is activated, the unnecessary area acquisition unit 12 acquires the vicinity of the activated outer edge as an unnecessary area. . Step S105 is also called an unnecessary area acquisition step.

  Next, the unnecessary area deletion unit 13 sets the value of an element in the area corresponding to the unnecessary area acquired by the unnecessary area acquisition unit 12 to 0 in each channel of the feature map (Step S106). Step S106 is also called an unnecessary area deletion step. The process of “determining the presence / absence of an unnecessary area, acquiring the unnecessary area, and setting the value of an element in the area corresponding to the unnecessary area to 0” is considered as a kind of editing processing. The processing up to step S106 is also called an editing step. The control unit 10 also functions as an editing unit in this editing step.

  Then, the CNN discriminator 11 performs image identification using the feature map from which the unnecessary area deletion unit 13 has deleted the unnecessary area (step S107), and ends the image identification processing. Step S107 is also called an identification step. In step S107, the control unit 10 functions as an identification unit.

  The specific processing will be described with reference to FIG. As an example, assume that a ruler 1112 appears in the input image 111 in addition to the original identification target 1111. In this case, in the activation map 151 extracted in step S103, it is confirmed that there is an unnecessary area 1512 that is estimated not to be used for image identification, in addition to the necessary area 1511 corresponding to the original identification target 1111. it can.

  Therefore, the determination in step S104 is Yes, and the position and size of the unnecessary area 1512 are acquired in step S105. Then, in step S106, in each channel 152 of the feature map 116, the value of the area 1521 corresponding to the unnecessary area is deleted and becomes zero. After the value of the area 1521 corresponding to the unnecessary area is set to 0 for all the channels 152, the characteristic map 117 is calculated from the characteristic map 116 after the unnecessary area is deleted, and the output 118 is finally obtained. Then, the output 118 obtained here is considered to be an identification result that is not affected by other than the original identification target 1111 (such as a ruler 1112).

  Therefore, the image identification device 100 can prevent the influence from the object other than the original identification target by deleting the information of the area estimated to be unnecessary for the image identification, thereby improving the accuracy of the image identification by the CNN. Can be done.

(Modification 1)
In the first embodiment described above, the activation map may be presented to the user together with the input image in the extraction of the activation region of the intermediate layer in the image identification processing (FIG. 9) (step S103). However, it may be difficult for the user to determine whether or not there is an unnecessary area simply by presenting the activation map and the input image. Therefore, a description will be given of a first modification example in which the input image and the activation map are displayed translucently and superimposed so that it is easy to determine whether or not there is an unnecessary area.

  As shown in FIG. 11, the image identification processing of the first modification has the processing contents in which steps S111 and S112 are added between steps S103 and S104 of the image identification processing of the first embodiment (FIG. 9). I have.

  In step S111, the control unit 10 displays, on the output unit 32, an image 153 in which the input image 111 and the activation map 151 obtained in step S103 are translucently superimposed, as shown in FIG. As for the method of translucent superimposition, the input image 111 may be translucent and superimposed on the activation map 151, or the activation map 151 may be translucent and superimposed on the input image 111.

  In step S112, the control unit 10 receives selection of an unnecessary area from the user via the operation input unit 34. For example, on the image 153 displayed on the display, the user may select an unnecessary area by clicking a shaded square corresponding to the unnecessary area with a mouse.

  The processing other than steps S111 and S112 is the same as the image identification processing of the first embodiment described above, and thus the description is omitted. As described above, in the first modification, the activation map is displayed so as to overlap the input image, so that the user can easily select an unnecessary area estimated to be unnecessary for image identification.

(Modification 2)
In the above-described embodiment, in the determination of the presence or absence of the unnecessary area (step S104) and the acquisition of the unnecessary area (step S105) in the image identification processing (FIG. 9), information on the unnecessary area is obtained from the user, In some cases, the area near the outer edge is acquired as an unnecessary area. However, it is not limited to this. For example, the information of the unnecessary area may be acquired by the control unit 10 executing a program for performing image recognition separately from the CNN discriminator 11. Such an embodiment is referred to as a second modification. In the second modification, the control unit 10 executes a program for performing image recognition, so that the control unit 10 also functions as an image recognition unit.

  For example, when an image obtained by capturing an image of an affected part is input and the CNN classifier 11 for identifying a skin disease performs image identification, the control unit 10 executes a program for recognizing an image other than the skin disease (eg, ruler, hair, etc.). Thus, a region in which an image other than a skin disease is shown may be acquired as an unnecessary region, and the value of an element corresponding to the unnecessary region may be set to 0 in each channel of the feature map. By doing so, the image identification device according to Modification 2 can improve the identification accuracy without the user's operation of selecting an unnecessary area.

(Modification 3)
In the first embodiment described above, in the extraction of the activation region of the intermediate layer in the image identification processing (FIG. 9) (step S103), the average of all channels or all classes is calculated without specially treating a specific class. Generated activation map. However, it is not limited to this. If CAM or Grad-CAM is used to generate an activation map, an activation map for each class can be obtained. Therefore, instead of averaging the activation map for all classes, an activation map for each individual class is presented to the user. May be. Such an embodiment is referred to as a third modification.

  For example, when it is sufficient to identify only m classes out of N classes, m activation maps by CAM or Grad-CAM corresponding to the m classes to be identified, and an input image May be arranged on the display via the output unit 32 and presented to the user. Further, by displaying an s image (s = 1 to m, which can be appropriately selected by the user) out of the m activation maps and an image obtained by superimposing a translucent image on the input image, it is unnecessary. The determination of the presence or absence of a region and the selection of a region may be further facilitated.

(Modification 4)
Further, the activation map for each class obtained using CAM or Grad-CAM is set as the activation state of the class, and m activation states (activation maps for each class) of m classes to be identified are used. The averaged one may be used as the activation map. Such an embodiment is referred to as a fourth modification.

  For example, when identifying an image of an affected part of a patient with a skin disease, it is determined that the patient's disease is a specific disease (for example, any one of the three diseases of disease A, disease B, and disease C). The case is conceivable. In this case, among the N classes identified by the CNN identifier 11, only a part of the classes (any of the disease A, the disease B, and the disease C) needs to be identified.

  Therefore, in Modification 4, in such a case, the activation map for each class generated by the CAM or Grad-CAM in the extraction of the activation region of the intermediate layer in the image identification processing (FIG. 9) (step S103). Of the (activation states), only the activation map (activation state) of the class to be identified this time is averaged to obtain a new activation map. Since the activation map obtained by averaging only the classes to be identified does not include the information of the activation map (activation state) of the class not to be identified, the determination of the presence / absence of the unnecessary region in step S104 and the unnecessary region in step S105 Can be obtained with higher accuracy.

  Then, since the accuracy of selecting an unnecessary area that can be noise at the time of identification is increased, in the fourth modification, the accuracy of image identification by the CNN identifier 11 can be further improved.

  In the above-described first embodiment, the unnecessary area is acquired from the activation map in step S105 of the image identification processing (FIG. 9). Conversely, the unnecessary area estimated to be necessary for image identification is acquired. Is also good. In that case, in step S106 of the image identification processing, the values of the elements other than the necessary area of the feature map are set to 0. This is the same in the first modification, and in step S112 of the image identification processing (FIG. 11), the user may select a necessary area estimated to be necessary for image identification. In this case, if there is an area other than the necessary area (Step S104; Yes), the necessary area is acquired in Step S105, and the value of the element other than the necessary area in the feature map is set to 0 in Step 106.

  Further, in the above-described embodiment and the modified example, the control unit 10 executes the program for realizing the discriminator by the CNN, so that the control unit 10 also functions as the CNN discriminator 11, but is not limited thereto. Absent. The image identification device 100 may include a device that realizes the function of the CNN identifier 11 (for example, a dedicated IC (Integrated Circuit)) separately from the control unit 10.

  In the above-described Modifications 2 and 4, skin diseases are described as examples, but the present invention is not limited to the field of dermatology and can be widely applied in the field of image identification. For example, the present invention can be applied to identification of flowers, identification of micrographs of bacteria, and the like.

  Further, the above-described embodiments and modified examples can be appropriately combined. For example, by combining Modifications 1 and 4, it is possible to display an activation map, which is averaged by focusing on a specific (identification target) class, superimposed on an input image, and the user can display an unnecessary area. The selection can be made more accurately and easily.

  Note that each function of the image identification apparatus 100 can also be performed by a computer such as a normal PC (Personal Computer). Specifically, in the above-described embodiment, the description has been given assuming that the program of the image identification processing performed by the image identification device 100 is stored in the ROM of the storage unit 20 in advance. However, the program may be stored on a flexible disk, a compact disc read only memory (CD-ROM), a digital versatile disc (DVD), a magneto-optical disc (MO), a memory card, a universal bus (USB) memory such as a USB (Universal Serial) memory. A computer capable of realizing each of the above-described functions may be configured by storing the program in a recording medium, distributing the program, reading the program into the computer, and installing the program.

  As described above, the preferred embodiments of the present invention have been described, but the present invention is not limited to the specific embodiments, and the present invention includes the inventions described in the claims and equivalents thereof. It is. Hereinafter, the inventions described in the claims of the present application will be additionally described.

(Appendix 1)
An input device to which an input image is input and an intermediate layer which is a layer other than an output layer from which an identification result of the input image is output, and an identifier for identifying the input image,
In the intermediate layer, a feature map obtaining means for obtaining a feature map for identifying the input image,
Activation map generation means for generating an activation map visualizing the activation state of the intermediate layer,
Editing means for editing the feature map obtained by the feature map obtaining means, with reference to the activation map generated by the activation map generating means,
Identification means for identifying the input image using the feature map edited by the editing means,
An image identification device comprising:

(Appendix 2)
The activation map generation means generates the activation map using CAM or Grad-CAM for each class identified by the discriminator.
The image identification device according to attachment 1.

(Appendix 3)
When the editing unit edits the feature map, the editing unit further includes an output unit that displays an image obtained by superimposing the input image and the activation map.
3. The image identification device according to claim 1 or 2.

(Appendix 4)
In the intermediate layer, unnecessary area obtaining means for obtaining an unnecessary area estimated to be not used when identifying the input image,
In the intermediate layer, unnecessary area deletion means for deleting information of the unnecessary area acquired by the unnecessary area acquisition means,
Further comprising,
The image identification device according to any one of supplementary notes 1 to 3.

(Appendix 5)
Automatic editing means by a classifier previously learning the identification of the unnecessary area by machine learning is further provided,
The editing means edits the feature map using the automatic editing means,
The image identification device according to attachment 4.

(Appendix 6)
An input device to which an input image is input and an intermediate layer which is a layer other than an output layer from which an identification result of the input image is output, and an identifier for identifying the input image,
In the intermediate layer, unnecessary area obtaining means for obtaining an unnecessary area estimated to be not used when identifying the input image,
In the intermediate layer, unnecessary area deletion means for deleting information of the unnecessary area acquired by the unnecessary area acquisition means,
An image identification device comprising:

(Appendix 7)
The unnecessary area obtaining means obtains the unnecessary area in an intermediate layer immediately before a fully connected layer connected to the output layer,
The unnecessary area deletion unit deletes information of the unnecessary area obtained by the unnecessary area obtaining unit in the intermediate layer immediately before the all connected layers connected to the output layer,
7. The image identification device according to any one of supplementary notes 4 to 6.

(Appendix 8)
The unnecessary area obtaining unit generates an activation map that visualizes an activation state of the intermediate layer, and obtains the unnecessary area from the activation map.
8. The image identification device according to any one of supplementary notes 4 to 7.

(Appendix 9)
The unnecessary area acquisition unit generates the activation map using CAM or Grad-CAM for each class identified by the classifier, and acquires the unnecessary area from the activation map.
An image identification device according to attachment 8.

(Appendix 10)
The unnecessary area obtaining means obtains the activation state for each of the classes using CAM or Grad-CAM, and generates the activation map using only the activation state of the class to be identified in the input image. Generating and acquiring the unnecessary area from the activation map,
An image identification device according to attachment 9.

(Appendix 11)
Operation input means for receiving an operation input of a user;
Output means for displaying the activation map;
Further comprising
The unnecessary area acquiring unit acquires the area selected by the operation input unit as the unnecessary area after displaying the activation map on the output unit.
The image identification device according to any one of supplementary notes 8 to 10.

(Appendix 12)
The unnecessary area obtaining unit obtains, as the unnecessary area, a region selected by the operation input unit after displaying an image obtained by superimposing the input image and the activation map on the output unit.
An image identification device according to attachment 11.

(Appendix 13)
The unnecessary area obtaining means obtains an outer edge area of the activation map as the unnecessary area,
The image identification device according to any one of supplementary notes 8 to 10.

(Appendix 14)
Further comprising an image recognition means for recognizing an image other than the image identified by the classifier,
The unnecessary area acquisition unit acquires an area of an image recognized by the image recognition unit as the unnecessary area,
The image identification device according to any one of supplementary notes 8 to 10.

(Appendix 15)
The input image is an image of the affected part of the skin disease,
The unnecessary area obtaining means obtains an area other than the affected part as the unnecessary area in the intermediate layer of the discriminator.
15. The image identification device according to any one of supplementary notes 4 to 14.

(Appendix 16)
An image identification method for identifying the input image by an identifier having an input layer where an input image is input and an intermediate layer other than an output layer from which an identification result of the input image is output,
In the intermediate layer, a feature map obtaining step of obtaining a feature map for identifying the input image,
An activation map generation step of generating an activation map that visualizes the activation state of the intermediate layer,
An editing step of editing the feature map obtained in the feature map obtaining step with reference to the activation map generated in the activation map generating step;
An identification step of identifying the input image using the feature map edited by the editing step,
An image identification method including:

(Appendix 17)
A computer of an image identification device including an input layer in which an input image is input and a classifier having an intermediate layer that is a layer other than an output layer in which an identification result of the input image is output,
In the intermediate layer, a feature map obtaining step of obtaining a feature map for identifying the input image,
An activation map generating step of generating an activation map visualizing an activation state of the intermediate layer;
An editing step of editing the feature map obtained in the feature map obtaining step with reference to the activation map generated in the activation map generating step, and
An identification step of identifying the input image using the feature map edited by the editing step;
The program to execute.

DESCRIPTION OF SYMBOLS 10 ... Control part, 11 ... CNN discriminator, 12 ... Unnecessary area acquisition part, 13 ... Unnecessary area deletion part, 20 ... Storage part, 31 ... Image input part, 32 ... Output part, 33 ... Communication part, 34 ... Operation input Unit, 100 ... image identification device, 110, 111 ... input image, 112, 113, 114, 115, 116, 117, 130, 132 ... feature map, 118 ... output, 120, 121, 123, 124, 125 ... filter, 122, 126, 131: window, 127: all connection, 140, 141, 142, 144, 145, 146, 147, 151: activation map, 143, 153: image, 152: channel, 1111: identification target, 1112 ... ruler, 1511 ... necessary area, 1512 ... unnecessary area, 1521 ... area

Claims (17)

An input device to which an input image is input and an intermediate layer which is a layer other than an output layer from which an identification result of the input image is output, and an identifier for identifying the input image,
In the intermediate layer, a feature map obtaining means for obtaining a feature map for identifying the input image,
Activation map generation means for generating an activation map visualizing the activation state of the intermediate layer,
Editing means for editing the feature map obtained by the feature map obtaining means, with reference to the activation map generated by the activation map generating means,
Identification means for identifying the input image using the feature map edited by the editing means,
An image identification device comprising: The activation map generation means generates the activation map using CAM or Grad-CAM for each class identified by the discriminator.
The image identification device according to claim 1. When the editing unit edits the feature map, the editing unit further includes an output unit that displays an image obtained by superimposing the input image and the activation map.
The image identification device according to claim 1. In the intermediate layer, unnecessary area obtaining means for obtaining an unnecessary area estimated to be not used when identifying the input image,
In the intermediate layer, unnecessary area deletion means for deleting information of the unnecessary area acquired by the unnecessary area acquisition means,
Further comprising,
The image identification device according to claim 1. Automatic editing means by a classifier previously learning the identification of the unnecessary area by machine learning is further provided,
The editing means edits the feature map using the automatic editing means,
The image identification device according to claim 4. An input device to which an input image is input and an intermediate layer which is a layer other than an output layer from which an identification result of the input image is output, and an identifier for identifying the input image,
In the intermediate layer, unnecessary area obtaining means for obtaining an unnecessary area estimated to be not used when identifying the input image,
In the intermediate layer, unnecessary area deletion means for deleting information of the unnecessary area acquired by the unnecessary area acquisition means,
An image identification device comprising: The unnecessary area obtaining means obtains the unnecessary area in an intermediate layer immediately before a fully connected layer connected to the output layer,
The unnecessary area deletion unit deletes information of the unnecessary area obtained by the unnecessary area obtaining unit in the intermediate layer immediately before the all connected layers connected to the output layer,
The image identification device according to claim 4. The unnecessary area obtaining unit generates an activation map that visualizes an activation state of the intermediate layer, and obtains the unnecessary area from the activation map.
The image identification device according to claim 4. The unnecessary area acquisition unit generates the activation map using CAM or Grad-CAM for each class identified by the classifier, and acquires the unnecessary area from the activation map.
The image identification device according to claim 8. The unnecessary area obtaining means obtains the activation state for each of the classes using CAM or Grad-CAM, and generates the activation map using only the activation state of the class to be identified in the input image. Generating and acquiring the unnecessary area from the activation map,
The image identification device according to claim 9. Operation input means for receiving an operation input of a user;
Output means for displaying the activation map;
Further comprising
The unnecessary area acquiring unit acquires the area selected by the operation input unit as the unnecessary area after displaying the activation map on the output unit.
The image identification device according to claim 8. The unnecessary area obtaining unit obtains, as the unnecessary area, a region selected by the operation input unit after displaying an image obtained by superimposing the input image and the activation map on the output unit.
The image identification device according to claim 11. The unnecessary area obtaining means obtains an outer edge area of the activation map as the unnecessary area,
The image identification device according to claim 8. Further comprising an image recognition means for recognizing an image other than the image identified by the classifier,
The unnecessary area acquisition unit acquires an area of an image recognized by the image recognition unit as the unnecessary area,
The image identification device according to claim 8. The input image is an image of the affected part of the skin disease,
The unnecessary area obtaining means obtains an area other than the affected part as the unnecessary area in the intermediate layer of the discriminator.
The image identification device according to any one of claims 4 to 14. An image identification method for identifying the input image by an identifier having an input layer where an input image is input and an intermediate layer other than an output layer from which an identification result of the input image is output,
In the intermediate layer, a feature map obtaining step of obtaining a feature map for identifying the input image,
An activation map generation step of generating an activation map that visualizes the activation state of the intermediate layer,
An editing step of editing the feature map obtained in the feature map obtaining step with reference to the activation map generated in the activation map generating step;
An identification step of identifying the input image using the feature map edited by the editing step,
An image identification method including: A computer of an image identification device including an input layer in which an input image is input and a classifier having an intermediate layer that is a layer other than an output layer in which an identification result of the input image is output,
In the intermediate layer, a feature map obtaining step of obtaining a feature map for identifying the input image,
An activation map generating step of generating an activation map visualizing an activation state of the intermediate layer;
An editing step of editing the feature map obtained in the feature map obtaining step with reference to the activation map generated in the activation map generating step, and
An identification step of identifying the input image using the feature map edited by the editing step;
The program to execute.

Images