JP2020135438A - Basis presentation device, basis presentation method and basis presentation program - Google Patents

Abstract

To create a saliency map indicating its determination basis at low cost, even in a neural network handling variety of input data.SOLUTION: A basis presentation device 100 comprises: a synthesis unit 103 that synthesizes input data "x" and a saliency map 122 "m" to create synthetic data "y"; an inference unit 104 that creates inference data "z" as a result of inferencing using a first NN 112 from the input data "x" and the synthetic data "y" respectively; and an evaluation unit 105 that calculates an error between the inference data "z" of the input data "x" and the inference data "z" of the synthetic data "y" by means of a loss function "E1(-)", and makes the calculation result to be a loss score, and a map generation unit 102 updates a parameter of a second NN 121 by reflecting the loss score for the second NN 121 used in a previous time inference.SELECTED DRAWING: Figure 1

Description

The present invention relates to a rationale presentation device, a rationale presentation method, and a rationale presentation program.

In general, a multi-layer neural network having high performance in today's image recognition and the like is composed of a huge number of parameters and a complicated model. However, while this type of machine learning system shows excellent performance, there is a problem that it is difficult to interpret the judgment basis of the output of the neural network. In order to solve this problem, some methods for interpreting the judgment basis of the neural network have been proposed.

One of the methods for interpreting the judgment basis is to create a saliency map showing the contribution of the input to the output of the neural network.
The method described in Non-Patent Document 1 is one of the methods in which the gradients of the input and the output of the neural network are calculated, and the portion of the input having a large gradient has a large contribution to the output. The method using the gradient of the output and input has a problem that a lot of noise due to the gradient is added to the saliency map, so by preparing a large number of input with noise and taking the average value of the gradient. A device to remove noise has been added.

The method described in Patent Document 1 is one of the methods using the gradients of the input and output of the neural network, and calculates the degree of contribution by using the Taylor approximation formula of the neural network.

The method described in Non-Patent Document 2 is a method of obtaining a saliency map by learning, in which a part of the input is masked and input to a trained neural network, and the output value is remarkable so as to be far from the correct answer. By learning the sex map, it is possible to create a mask that hides the part where the contribution of input is high.

Special Table 2018-513507

Daniel Smilkov et al., "SmoothGrad: removing noise by adding noise", [online], 2017, [searched February 4, 2019], Internet <URL: https://arxiv.org/abs/1706.03825> Ruth Fong et al., "Interpretable Explanations of Black Boxes by Meaningful Perturbation", [online], January 10, 2018, [Search February 4, 2019], Internet <URL: https://arxiv.org/abs /1704.03296>

The method of creating a saliency map showing the judgment basis of a certain neural network needs to adjust the parameters according to the characteristics of the neural network and the input data. Therefore, the more neural networks and input data to be handled, the higher the cost for humans to prepare the parameters to be adjusted.

Therefore, the main object of the present invention is to create a saliency map showing the basis for judgment even in a neural network that handles various input data at low cost.

In order to solve the above problems, the rationale presentation device of the present invention has the following features.
The present invention creates a saliency map showing the characteristics of the input data as a result of inferring from the input data of the data set used for learning the first neural network by using the second neural network. A map generator that outputs as the basis of the neural network,
A compositing unit that synthesizes the input data and the prominence map to create composite data,
An inference unit that creates inference data as a result of inference from each of the input data and the synthetic data using the first neural network.
It has an evaluation unit that calculates the error between the inference data of the input data and the inference data of the composite data by a loss function and uses the calculation result as a loss score.
The map generation unit updates the parameters of the second neural network by reflecting the loss score on the second neural network used in the previous inference.

Further, the present invention is a ground presentation method executed by the ground presentation device and a ground presentation program for executing the ground presentation method.

According to the present invention, even in a neural network that handles various input data, a saliency map showing the judgment basis can be created at low cost.

It is a block diagram which shows the neural network system which concerns on Example 1 of this invention. An example of the processing of the synthesis part which synthesizes the input image data and the saliency map which concerns on Example 1 of this invention is shown. An example of a saliency map image in which the region of interest according to Example 1 of the present invention is too wide is shown. An example of a prominence map image in which the region of interest according to Example 1 of the present invention is too narrow is shown. It is a flowchart which shows the process of the basis presenting apparatus which concerns on Example 1 of this invention. It is a block diagram which shows the neural network system which concerns on Example 2 of this invention. It is explanatory drawing which shows the image data of the 1st region of interest which concerns on Example 2 of this invention. It is explanatory drawing which shows the image data of the 2nd attention area which concerns on Example 2 of this invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Each figure is only schematically shown to the extent that the present invention can be fully understood. Therefore, the present invention is not limited to the illustrated examples. Further, in the drawings to be referred to, the dimensions of the members constituting the present invention may be exaggerated for the sake of clarity. In each figure, common components and similar components are designated by the same reference numerals, and duplicate description thereof will be omitted.

[Example 1]
FIG. 1 is a configuration diagram showing a neural network system according to the first embodiment. The neural network system includes a machine learning device 110 and a rationale presentation device 100. Hereinafter, the neural network is abbreviated as NN (Neural Network).
In the machine learning device 110, the learning unit 111 machine-learns the first NN 112 with reference to the data set 113. The rationale presentation device 100 presents the rationale for determining the reasoning by the first NN 112 of the machine learning device 110. Hereinafter, it is assumed that the first NN112 is used for the inference processing represented by the function “f (・)”.

The control unit of the machine learning device 110 and the control unit of the rationale presentation device 100 are mainly composed of a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit) (not shown), respectively, and a ROM (Read Only Memory) and the like (not shown). By reading and executing a predetermined program from, various processes related to the neural network are performed. Further, the control unit has a storage unit including a RAM (Random Access Memory), a hard disk drive, a flash memory, and the like inside, and various information is stored in this storage unit.

The rationale presenting device 100 has an input unit 101, a map generation unit 102, a synthesis unit 103, an inference unit 104, and an evaluation unit 105 as processing units embodied by the control unit.
The rationale presenting device 100 has a second NN 121 and a saliency map 122 as data stored in the storage unit. Hereinafter, it is assumed that the second NN 121 is used for the inference processing represented by the function "g (・)".
From here on, after explaining each data (second NN 121 and saliency map 122) of the rationale presenting apparatus 100, each processing unit will be described in order with reference to FIG.

FIG. 2 shows an example of processing of the compositing unit 103 that synthesizes the input image data of the data set 113 and the saliency map 122.
As an example of the data set 113, the following is composed of a set of a dog image 301 input to the input layer of the first NN 112 and a determination result of "a dog is shown" output from the output layer of the first NN 112. Use teacher data.
One saliency map 122 "m" is generated corresponding to the input data "x" of the dataset 113, and the "m" and "x" are of the same size and dimension. In FIG. 2, as an example of the saliency map 122, the saliency map image 302 that visualizes the region where the dog's face is located is treated.
The region of interest 302a where the dog's face is located is an important region of the saliency map image 302 for obtaining the determination result that "the dog is captured" from the dog image 301. On the other hand, the black-painted area of the saliency map image 302 other than the attention area is referred to as a “mask area”.

The compositing unit 103 creates the compositing image 303 by superimposing the saliency map image 302 on the surface of the dog image 301. In the composite image 303, the dog's face shown in the attention region 303a and the mask region located outside the attention region 303a are visualized so as to be distinguishable.
Although the black-painted area is illustrated in FIG. 2 as the mask area of the saliency map image 302, the mask area may be set so as to blur the dog image 301 or mix noise. Good.

The main purpose of the rationale presentation device 100 is to create an appropriate saliency map 122 as judgment rationale data for the output of the first NN 112. For example, in the saliency map image 302 as shown in FIG. 2, the attention region 302a where the dog's face that contributes to the discrimination result is located is surely extracted, and the mask region of the other portion that does not contribute to the discrimination result is sufficiently widened. This is an example of an appropriate saliency map 122 set in.

On the other hand, in the saliency map image 311 of FIG. 3, since the region of interest 311a is too wide, the dog's face portion can be reliably extracted (reference numeral 312). However, since the mask area is too narrow and extra parts such as dog paws that do not contribute to the discrimination result are extracted (reference numeral 312a), the saliency map image 311 is an inappropriate saliency map 122.

Further, in the saliency map image 321 of FIG. 4, since the area of interest 321a is too narrow, only a part (nose and mouth) of the dog's face is extracted (reference numeral 322). Therefore, the saliency map image 321 is an inappropriate saliency map 122 because it is erroneously discriminated as another animal such as a raccoon dog similar to a dog if only a part of the face is set as the attention region 322a.
The examples of various saliency maps 122 have been described above.

Returning to FIG. 1, the second NN 121 is a neural network for outputting the saliency map 122 from the input data of the data set 113. The inference using the second NN121 is expressed by the formula "m = g (x)". Here, the second NN 121 is designed so that the range of m is 0 <m <1. For example, the normalization process may be inserted immediately before the output layer of the second NN 121.
Further, if it is effective for learning the saliency map 122, some rule may be given to the distribution of m within a range applicable to the backpropagation method. For example, an activation function in which the value of m is equal to or less than a certain threshold value may be added before the output of the second NN 121.

The internal structure of the second NN 121 is not particularly limited, and the structure according to the task can be freely selected. For example, a neural network in which fully connected layers are stacked may be used, or a convolutional neural network may be used. Convolutional neural networks are disclosed, for example, in the paper "U-Net: Convolutional Networks for Biomedical Image Segmentation" by Olaf Ronneberger et al.

FIG. 5 is a flowchart showing the processing of the basis presentation device 100. In this flowchart, the preparatory step (S101), the learning steps of the second NN121 (S102 to S107), and the learning end determination step (S108) of the second NN121 are executed in order. By repeating the learning process of the second NN 121 until the learning end determination is satisfied, the parameters of the second NN 121 are gradually improved.
Hereinafter, details of each process in the flowchart will be described.

The inference unit 104 acquires the first NN112 (learned model) as a preliminary preparation at the start of learning of the second NN121 (S101). The first NN112 can apply any neural network regardless of the internal structure.
The map generation unit 102 initializes the parameters of the second NN 121 at the start of learning of the second NN 121 (S102). The parameter of the second NN121 is, for example, the synapse weight “w” in the neural network. Since the parameter of the second NN 121 is not a fixed value but a variable value, the parameter update process is performed in the second and subsequent S102.

The input unit 101 acquires the input data “x” (dog image 301 in FIG. 2) from the data set 113 (S103). The input data "x" to be acquired is a part or all of the input data of the data set 113 used for the training of the first NN 112.
The format of the data set 113 is not particularly limited. Image data may be used, or time series data may be used. Further, the parameter of the first NN112 after learning is a fixed value, the update process is not performed, and the value of the partial differential is calculated by the error back propagation method.
The map generation unit 102 generates a saliency map 122 (saliency map image 302 in FIG. 2) by inference that the second NN 121 inputs the input data “x” (S104). The map generation unit 102 outputs the generated saliency map 122 to the user as the basis of the first NN 112.

The compositing unit 103 creates composite data “y” (composite image 303 in FIG. 2) by combining the input data “x” and the saliency map 122 (S105). This creation process is expressed by the formula "y = x · m = x · g (x)". That is, the synthesis unit 103 weights the input data “x” with the saliency map 122 “m”.

The inference unit 104 obtains the inference data "z" (for example, "a dog is shown") and notifies the evaluation unit 105 by inference that the synthetic data "y" is input to the first NN112 (trained model) (for example, "a dog is shown") S106). The inference process by the inference unit 104 is expressed by the formula "z = f (y) = f (x · m)".
By arranging the processing of the map generation unit 102 of S102 and S104 before the processing of the inference unit 104 of S106, the constructed first NN112 can be applied regardless of the internal structure of the neural network.

The evaluation unit 105 compares the inference data “z” with the correct value of the inference data by the loss function “E1 (・)” which is an evaluation function, and calculates the loss score as the comparison result (S107). The correct answer value of the inference data is the inference result when the input data "x" is input to the first NN 112. The loss function "E1 (・)" can be expressed by, for example, the following (Formula 1).

λ indicates a coefficient.

The first term on the right side of Equation 1 shows the result of comparison between the inference data "z" and the correct value of the inference data. Here, the function for comparison is the mean square error, but other loss functions may be used. For example, the average absolute value error may be used.
The second term on the right side of Equation 1 indicates a regularization term for appropriately reducing the feature portion (area of interest) of the saliency map 122. If the saliency maps 122 are all 1 (for example, as shown in FIG. 3, when there is almost no mask area), the loss score is always 0 (that is, it can be correctly determined as "a dog"). ). Therefore, the learning of the saliency map 122 may not proceed only for the first term on the right side of the equation 1.
Therefore, a regularization argument is required to reduce the value of the saliency map 122 “m”. According to the second term on the right side, the oversized region of interest in FIG. 3 can be converged to the region of interest of an appropriate size in FIG. Here, the square of m is used, but other regularization equations may be used.

When the learning end condition (predetermined condition) such as when the loss score becomes smaller than a certain threshold value is satisfied (Yes in S108), the evaluation unit 105 ends the learning. On the other hand, if the learning end condition is not satisfied (No in S108), the process returns to S102 and learning is continued. Another learning end condition may be when a certain number of learnings is completed.

In S102 again, the evaluation unit 105 instructs the map generation unit 102 to update the parameters of the second NN 121 based on the loss score by the error back propagation method similar to the learning of the normal neural network. As a result, the learning process of the parameters of the second NN 121 is performed. The instruction to the map generation unit 102 includes, for example, a value obtained by partially differentiating the loss score with the weight of the second NN 121 as the gradient described in Non-Patent Document 1.

According to the first embodiment described above, the map generation unit 102 creates the saliency map 122 using the second NN 121. The saliency map 122 can handle various input data "x" at once, and the influence of gradient noise is reduced.
Then, by feeding back (reflecting) the loss score from the evaluation unit 105 to the map generation unit 102, the machine learning of the second NN 121 is automatically performed without the user manually adjusting the parameters of the second NN 121. Therefore, the saliency map 122 and the second NN 121 can be created at low cost.

[Example 2]
FIG. 6 is a configuration diagram showing a neural network system according to a second embodiment.
The machine learning device 110 of the first embodiment shown in FIG. 1 and the machine learning device 110 of the second embodiment shown in FIG. 6 have the same configuration. In the basis presentation device 100 of the first embodiment shown in FIG. 1 and the basis presentation device 200 of the second embodiment shown in FIG. 6, parts having the same last two digits of the code correspond to each other. For example, the input unit 101 of FIG. 1 and the input unit 201 of FIG. 6 correspond to each other and have the same configuration.

The difference between Example 1 and Example 2 is the number of saliency maps. In Example 1, one saliency map 122 “m” corresponding to the region of interest of the entire image was created. On the other hand, the second embodiment is extended to create a saliency map 222 having a plurality of (N) elements for each of the plurality of areas of interest. The saliency map 222 is shown as the set "M = (m [1], m [2], ..., m [N])". m [1] indicates the first element of m.
Further, the saliency map 222 "M" is created for each input data "x" of the data set 113.

FIG. 7 is an explanatory diagram showing image data of a first region of interest among a plurality of regions of interest.
As shown by reference numeral 411, for example, when the first attention region is a dog's ear being photographed, the map generation unit 202 generates an image 412 including the attention region 412a as the first saliency map 222. To do. As a result of synthesizing the image 412 and the image 301 of FIG. 2 which is the input data, the compositing unit 203 obtains a composite image 413 including the region of interest 413a.
It should be noted that the description in the sentence “the ear is the region of interest” of reference numeral 411 is simplified in order to make it easy to understand that the saliency map 222 has N elements. When the user actually specifies the area of interest, it is not specified in such a sentence, but is specified by the feature specification parameter 223 described later.

FIG. 8 is an explanatory diagram showing image data of a second region of interest among the plurality of regions of interest.
As shown by reference numeral 421, for example, when the second attention region is a dog's eye being photographed, the map generation unit 202 generates an image 422 including the attention region 422a as the second saliency map 222. To do. As a result of synthesizing the image 422 and the image 301 of FIG. 2 which is the input data, the synthesizing unit 203 obtains a composite image 423 including the region of interest 423a.
So far, an example of generating two saliency maps 222 from two regions of interest has been described. It should be noted that the first NN112 does not directly determine the "dog's ear" or "dog's eye" and set it as the area of interest. In order to explain the region of interest determined by the first NN112 based on some criterion, the easily visualized information such as "dog's ear" and "dog's eye" is illustrated.

Returning to FIG. 6, the storage unit of the rationale presentation device 200 stores a feature specification parameter 223 that allows the user to specify what kind of N elements the saliency map 222 “M” is to be created. The feature specification parameter 223 includes, for example, the following data.
-Mesosphere number of the first NN112 (first mesosphere neuron, second mesosphere neuron ...). This number is used to allow the user to specify which neuron of interest region is to be adopted as an element of the saliency map 222.
The number of channels in the mesosphere of the first NN112 (N above). This number of channels is set by the map generation unit 202 as the number of outputs of the second NN221.

Hereinafter, each processing content of the second embodiment will be described with reference to the flowchart of FIG. 5 in the same manner as in the first embodiment.
The inference unit 204 acquires the first NN 112 at the start of learning (S101). Further, the inference unit 204 obtains the number of the mesosphere specified by the user by referring to the feature designation parameter 223.
At the start of learning of the second NN221, the map generation unit 202 initializes the parameters of the second NN221 according to the number of channels in the intermediate layer of the feature designation parameter 223 (S102). In S102 after the second time, the parameter update process is performed.
The input unit 201 acquires the input data “x” from the data set 113 as in the first embodiment (S103).

The map generation unit 202 generates the saliency map 222 “M” by inputting the input data “x” to the second NN221 (S104). This generation process is expressed by the formula "M = g (x)". Each element (i = 1 to N) of the saliency map 222 "M" is generated for each number of the intermediate layer acquired by the inference unit 204 in S101.

The synthesis unit 203 synthesizes the input data “x” and each element of the saliency map 222 “M” to synthesize the composite data “Y = (y [1], y [2],…, y [N]. ], Y [N + 1]) ”(S105). This synthesis process is expressed by the formula "y [i] = x · m [i], y [N + 1] = x". The [N + 1] th element of the composite data "Y" substitutes the input data "x" that is not composited as it is. This [N + 1] th element is used in the evaluation unit 205 to calculate the correct answer value.

The inference unit 204 causes the first NN 112 to input the composite data “Y” to perform inference, and sets the feature vector of the intermediate layer specified by the feature designation parameter 223 as the feature-specific inference data “Z”. (S106). The process of inferring the feature-specific inference data "Z" from the first NN112 is a function "F (・)". This inference process is expressed by the formula "Z [i] = F (y [i])". Then, the inference unit 204 evaluates the following two types of data Z [i] = (z [i] [1], ..., z [i] [N + 1]) as feature-specific inference data "Z". Notify 205.
-[Inference result data] When i = 1 to N, the inference data "z [i] [i]" corresponding to the i-th element of the saliency map 222 "M" is output.
・ [Correct value data] When i = N + 1, the inference data "Z [N + 1] = (z [N + 1] [1],…, z [N +" corresponding to the input data "x" 1] [N]) ”is output.

The evaluation unit 205 compares the inference result data of the feature-specific inference data “Z” with the correct answer value data by the loss function “E2 (・)” of the following mathematical formula 2 and calculates the loss score (S107).

Similar to the first embodiment, the first term on the right side of the equation 2 is a term indicating the comparison result between the inference result data and the correct answer value data, and the second term on the right side is the regularization for appropriately reducing the saliency map 222. Indicates a term.
Similar to the first embodiment, the evaluation unit 205 determines whether or not the learning end condition (predetermined condition) is satisfied based on the value of the loss score (S108), and when the learning end condition is not satisfied. The learning process (S102) of the above is executed.

In the second embodiment described above, the map generation unit 202 individually generates the saliency map 222 “M” having a plurality of elements according to the feature designation parameter 223 specified by the user. As a result, the saliency map 222 “M” for each feature (area of interest) desired by the user can be presented.
Further, with respect to the second NN221 used for generating the saliency map 222 “M”, the learning of the second NN221 is performed by feeding back the loss score which is the evaluation result of the evaluation unit 205 as in the first embodiment. ..
Therefore, since the user can analyze the high-precision saliency map 222 “M” created by the second NN221 improved by learning, the intermediate layer of the first NN112 can be interpreted in detail.

Although each embodiment of the present invention has been described above, the present invention is not limited to this, and can be carried out within a range that does not change the gist of the claims. For example, the following modification can be considered.

In each embodiment, the saliency maps 122 and 222 output by the second NN 121 and 221 have the same dimensions as the input data “x” of the data set 113. On the other hand, the second NN 121,221 may output one saliency map 122,222 and duplicate the saliency maps 122, 222 in the dimensional direction.
In each embodiment, the saliency maps 122 and 222 output by the second NN 121 and 221 have the same size as the input data “x” of the data set 113. On the other hand, the saliency maps 122 and 222 having a size smaller than the input data "x" may be output, and the output may be expanded so as to have the same size as the input data "x".
In each embodiment, the correct answer value data used by the evaluation units 105 and 205 uses the output of the first NN 112, but even if the output data (correct answer label) corresponding to the input data “x” of the data set 113 is used. Good.

In the second embodiment, the input unit 201 uses the data set 113 used for learning the first NN 112 as the input data “x”. On the other hand, the input unit 201 may use the intermediate layer data of the first NN 112 whose internal structure has been partially modified so that the error back propagation can be performed as the input data “x”. The map generation unit 202 can also create a saliency map 222 for the modified mesosphere data of the first NN 112.
In the second embodiment, the number of the intermediate layer of the first NN 112 is used as the feature designation parameter 223, but the number of the output layer may be used.
On the right side of the loss function "E2 (・)" of Example 2, a third term (regularization term) is provided so that the variance of the saliency map 222 becomes large for the purpose of improving the interpretability of the features for each channel. You may add it.

100,200 Rationale presentation device 101,201 Input unit 102,202 Map generation unit 103,203 Synthesis unit 104,204 Reasoning unit 105,205 Evaluation unit 110 Machine learning device 111 Learning unit 112 1st NN
113 Dataset 121,221 2nd NN
122,222 Severity map 223 Feature specification parameters

Claims (6)

As a result of inferring from the input data of the data set used for learning the first neural network using the second neural network, a saliency map showing the characteristics of the input data was created and the basis of the first neural network. The map generator that outputs as
A compositing unit that synthesizes the input data and the prominence map to create composite data,
An inference unit that creates inference data as a result of inference from each of the input data and the synthetic data using the first neural network.
It has an evaluation unit that calculates the error between the inference data of the input data and the inference data of the composite data by a loss function and uses the calculation result as a loss score.
The map generation unit is a rationale presenting device characterized in that the parameters of the second neural network are updated by reflecting the loss score on the second neural network used in the previous inference. The evaluation unit uses the loss function having a first term that the larger the error is, the higher the loss score is, and the second term that the larger the feature portion of the prominence map is, the higher the loss score is. The rationale presenting device according to claim 1, wherein the loss score is calculated. The rationale presenting device according to claim 1, wherein the map generation unit outputs one said saliency map corresponding to the input data. The rationale presenting device further has a storage unit of a feature designation parameter indicating a designated element from a plurality of elements showing the feature of the input data existing in the intermediate layer of the first neural network.
The rationale presenting device according to claim 1, wherein the map generation unit individually creates and outputs the saliency map for each element indicating a designated feature according to the feature designation parameter. The rationale presenting device has a map generation unit, a synthesis unit, an inference unit, and an evaluation unit.
The map generation unit creates a saliency map showing the characteristics of the input data as a result of inferring from the input data of the data set used for learning the first neural network using the second neural network. Output as the basis of the first neural network,
The synthesis unit creates composite data by synthesizing the input data and the saliency map.
The inference unit creates inference data as a result of inference from the input data and the synthetic data using the first neural network.
The evaluation unit calculates the error between the inference data of the input data and the inference data of the composite data by a loss function, and uses the calculation result as a loss score.
A rationale presentation method characterized in that the map generation unit updates the parameters of the second neural network by reflecting the loss score on the second neural network used in the previous inference. In the basis presentation device according to claim 1,
As a result of inferring from the input data of the data set used for learning the first neural network using the second neural network, a saliency map showing the characteristics of the input data was created and the basis of the first neural network. And the procedure to output as
A procedure for synthesizing the input data and the saliency map to create composite data, and
A procedure for creating inference data as a result of inference from the input data and the synthetic data using the first neural network, and
A procedure for calculating an error between the inference data of the input data and the inference data of the composite data by a loss function and using the calculation result as a loss score.
A rationale presentation program for executing a procedure for updating the parameters of the second neural network by reflecting the loss score on the second neural network used in the previous inference.