KR102157441B1 - Learning method for neural network using relevance propagation and service providing apparatus - Google Patents

Abstract

A neural network training method using relevance transfer comprises the following steps of: determining, by a training device, a first loss function by performing a forward transfer process using training data on a neural network; calculating, by the training device, a loss function for each layer based on a feature map of each layer and information on a backward relation information in a backward relation transfer process with respect to the neural network; determining, by the training device, a final loss function by summing a second loss function and the first loss function generated as a result of summing loss functions by layer; and updating, by the training device, the weight of each layer while performing a backward transfer process by using the final loss function.

Description

Neural network learning method and service device using relevance transcription {LEARNING METHOD FOR NEURAL NETWORK USING RELEVANCE PROPAGATION AND SERVICE PROVIDING APPARATUS}

The technology described below relates to a technique for learning an artificial neural network model.

Artificial neural networks are being applied to various industries based on their high accuracy. However, the operation of the artificial neural network model is a complex black-box model and does not provide information on specific predictions of the network. It is difficult to know by what criteria the artificial neural network makes decisions internally. Due to these characteristics, there is a limit that cannot be easily used in industries such as medical, autonomous driving, and military, where a basis for network judgment is essential. In order to overcome these limitations, studies are underway to interpret neural networks.

Sebastian Bach, Alexander Binder, Gregoire Montavon, Frederick Klauschen, Klaus-Robert Muller, Wojciech Samek, "On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation", PLoS one, 10(7): e0130140, 2015

The technique described below is intended to provide a new neural network learning method applying a technique for analyzing an artificial neural network. The technique described below is intended to provide a neural network learning method using the result of hierarchical relevance transcription.

In the neural network learning method using relevance transcription, the learning device determines a first loss function by performing a forward transcription process using training data for the neural network, and the learning device determines the characteristics of each layer in the backward relationship transcription process for the neural network. Computing a loss function for each layer based on the map and the backward relationship information, and determining a final loss function by summing the second loss function and the first loss function generated as a result of the learning device summing the loss functions for each layer And updating, by the learning device, a weight of each layer while performing a backward transfer process using the final loss function.

A service device using a neural network trained using relevance transcription includes an input device that receives input data, a storage device that stores a neural network model learned using relevance transcription, and inputs the input data into the neural network model, and the neural network model It includes a computing device that generates specific service information by using the output result. The neural network model is trained on the basis of the final loss function generated by using the loss function for each layer generated during the backward correlation transcription process.

The technique to be described below learns a neural network in consideration of a result of the decision information in the inner layer of the neural network to infer the neural network. Therefore, in the technique described below, the weight is updated so that the expression position of the feature map is specified while improving the performance for a given problem. Furthermore, the technique described below is a general-purpose technique applicable to a general deep learning model.

1 is an example of a DNN transcription process.
2 is an example of a DNN reverse transcription process.
3 is an example of a hierarchical relationship transcription process.
4 is an example of a DNN learning process.
5 is an example of a flow chart for a DNN learning process.
6 is another example of a flowchart for a DNN learning process.
7 is an example of a service device including a DNN.

The technology to be described below may be modified in various ways and may have various embodiments, and specific embodiments will be illustrated in the drawings and described in detail. However, this is not intended to limit the technology described below to a specific embodiment, it should be understood to include all changes, equivalents, and substitutes included in the spirit and scope of the technology described below.

Terms such as 1st, 2nd, A, B, etc. may be used to describe various components, but the components are not limited by the above terms, and only for the purpose of distinguishing one component from other components. Is only used. For example, a first component may be referred to as a second component, and similarly, a second component may be referred to as a first component without departing from the scope of the rights of the technology described below. The term and/or includes a combination of a plurality of related listed items or any of a plurality of related listed items.

In terms of the terms used in the present specification, expressions in the singular should be understood as including plural expressions unless clearly interpreted differently in context, and terms such as "includes" are specified features, numbers, steps, actions, and components. It is to be understood that the presence or addition of one or more other features or numbers, step-acting components, parts or combinations thereof is not meant to imply the presence of, parts, or combinations thereof.

Prior to the detailed description of the drawings, it is intended to clarify that the division of the constituent parts in the present specification is merely divided by the main function that each constituent part is responsible for. That is, two or more constituent parts to be described below may be combined into one constituent part, or one constituent part may be divided into two or more according to more subdivided functions. In addition, each of the constituent units to be described below may additionally perform some or all of the functions of other constituent units in addition to its own main function, and some of the main functions of each constituent unit are different. It goes without saying that it may be performed exclusively by.

In addition, in performing the method or operation method, each of the processes constituting the method may occur differently from the specified order unless a specific order is clearly stated in the context. That is, each process may occur in the same order as the specified order, may be performed substantially simultaneously, or may be performed in the reverse order.

The technique to be described below relates to a neural network learning method.

Neural networks or artificial neural networks are statistical learning algorithms that mimic the neural networks of living things. Various neural network models are being studied. Recently, deep learning networks (DNNs) are attracting attention.

DNN is an artificial neural network model consisting of several hidden layers between an input layer and an output layer. DNN can model complex non-linear relationships like general artificial neural networks. Various types of DNN models have been studied. For example, there are Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), Generative Adversarial Network (GAN), and Relation Networks (RL).

Learning methods for neural networks or DNNs are also being studied. The technique described below can be applied to a learning technique based on a backward propagation method. Weight control according to the reverse transcription method is a method used in most neural network models. Therefore, the technique described below is not limited to a specific neural network model, and can be applied to various neural network models having a reverse transcription-based learning method.

The neural network learning process includes forward propagation and backward propagation. The forward direction means the direction from the input layer to the output layer. Reverse warrior is also referred to simply as reverse warrior.

The neural network training process is first described based on the DNN.

1 is an example of a DNN transcription process. Neural networks are composed of multiple layers. Typically, a neural network is composed of an input layer, a plurality of hidden layers, and an output layer. In the initial learning phase, each layer of the neural network is initialized with a random weight.

이다. 여기서, f는 그 계층이 수행하는 함수,

및

는 각각 그 계층의 입력과 가중치를 의미한다. 신경망의 입력층 및 은닉층이 이와 같은 과정을 반복하고, 마지막으로 출력층은 최종적으로 나온 결과를 바탕으로 비용 함수(cost function)를 계산한다. 출력층은 결정 계층이라고 명명하기도 한다. In a neural network, each layer determines a feature map by calculating a given input data and a weight. The feature map created in one layer is used as an input to the next layer. For example, a feature map output from the first layer (input layer) in FIG. 1 is

to be. Where f is the function that the layer performs,

And

Represents the input and weight of the layer, respectively. The input layer and the hidden layer of the neural network repeat this process, and finally, the output layer calculates a cost function based on the final result. The output layer is sometimes referred to as the decision layer.

The cost function can be selected based on factors such as the type of learning (supervised learning, unsupervised learning, reinforcement learning, etc.), the type of neural network model, and the activation function. For example, when supervised learning is performed on a multi-classification problem, an activation function and a cost function are generally determined as a softmax function and a cross entropy function, respectively. The definition of each function is shown in FIG. 1.

The output layer outputs d ₁ , d ₂ ,..., d _K ,..., d _N. For d ₁ , d ₂ ,..., d _K ,..., d _N , the activation function applied values are p ₁ , p ₂ ,..., p _K ,..., p _N. It is assumed that p _K is the largest value after applying the activation function among the results of the output layer. p _K is the prediction made by the neural network. The cost function calculates the loss L between the judgment made by the neural network and the actual expected label.

2 is an example of a DNN reverse transcription process.

A neural network (DNN) is a structure in which many nonlinear activation functions are entangled through several layers. Therefore, optimizing the weights of a layer is a very difficult non-convex problem.

The weight of a general neural network is adjusted (updated) through stochastic gradient descent. The weight is updated by a method of converging to an appropriate value through the stochastic gradient descent method shown in FIG. 2. The weights of each layer are updated according to how much each weight affects the cost function determined at the last layer through the transcription.

The neural network is computed using a chain rule because the influence of the weights of the middle layer cannot be calculated directly because of its complex structure. 2 describes the reverse transcription process. In FIG. 2, the weight of the i-th layer is w _i , and the learning moment at which the weight is updated is t+1.

When the loss is calculated in the output layer (eg, n layer), the weight change amount for this value is calculated. The amount of change in weight for the loss of the layer immediately in front of the output layer (layer n-1) is the amount of change in weight for the loss from the output layer (layer n) and the weight of the corresponding layer (layer n-1) to the weight of the output layer (layer n). It can be found by serially multiplying the amount of change. Each layer's weight is updated according to this variation, and this variation is continuously transmitted to the previous layer. That is, the weight update proceeds from the output layer to the input layer. This process of updating weights is called reverse transcription. 2 shows a reverse transcription process and an equation required for calculation.

3 is an example of a layerwise relevance propagation (LRP) process.

Heatmap is a graphic technique that expresses an array of values in two dimensions in color. By designating different colors according to the size of the values, you can know the distribution of values in the array. In the field of artificial intelligence, the heat map can express the degree to which each pixel of the input image contributes to the judgment of the neural network. A representative method of generating heat maps in neural networks is the Layerwise Relevance Propagation (LRP) algorithm. LRP defines a signal transmitted in the reverse direction within a network as relevance, and proceeds with the assumption that this relevance is preserved at the same value even when passing through a layer.

을 얻을 수 있다.The relevance of the output layer of the neural network is calculated from the result itself after the transcription process. As in the reverse transcription process, the relevance is transmitted from the last layer, and the relevance is transmitted based on the learned weight of the passing layer and the weighted sum of the input feature map as shown in the equation of FIG. 3 while the total of the relevance is maintained. In the equation of FIG. 3, c _i denotes the number of kernels of the i-th layer, and function f is the same function as the operation function of the corresponding layer. w _i is the weight of the i-th layer. Through this process, it is possible to know the contribution of each pixel of the input feature map of each layer to the determination of the network. When passing through to the input layer, the heat map of the i-th layer that expresses the contribution of the pixels of the input image to the network determination with the same size as the input image.

Can be obtained.

Hereinafter, a neural network learning process to which the above-described algorithm for generating a heat map is applied will be described. 4 is an example of a DNN learning process. The learning process shown in FIG. 4 reflects loss information on a feature map of an intermediate layer along with conventional loss information in determining a cost function. The learning process is largely composed of three steps. The learning process consists of forward transcription, relevance transcription and reverse transcription. In other words, unlike the conventional technique, related transcription was added. The results of relevance transcription are used to determine the final loss information.

In FIG. 4, a feature map resulting from a layer through forward transcription is indicated as a ⁺ , and since LRP calculation is performed in the reverse direction of the network, the calculated relationship is indicated as a ⁻ .

(i) First, the loss L _T between the judgment of the problem handled by the neural network and the expected result is calculated through forward transcription.

(ii) Then, LRP is performed to calculate the relevance indicating the degree to which each of the layer-specific feature maps influences the decision of the network.

과 순방향 전사가 끝난 후 신경망이 내린 판단 정보가 전달된 계층의 관련성

간의 L₁ 손실

이다. Explaining based on the i-th layer of the neural network, the loss calculated in the i-th layer is a feature map created at the (i-1)-th layer, the input feature map of the layer.

And the relevance of the layer to which the judgment information made by the neural network is transmitted after the forward transcription is over

L ₁ loss of liver

to be.

은 실제 신경망의 입력이라고 정의한다. K개의 계층이 있는 신경망의 경우 계층마다 이 손실을 계산하여 나온 K개의 손실과 L_T를 모두 더한 L이 신경망의 최종적인 손실이 된다. At this time, the input feature map of the first layer

Is defined as the input of the actual neural network. In the case of a neural network with K layers, the final loss of the neural network is the sum of K losses and L _T obtained by calculating this loss for each layer.

(iii) When the LRP process is all over, it performs a reverse transcription process based on the final loss L, updates the weights of the neural network, and proceeds with learning.

The process described in FIG. 4 is defined using an equation as follows. The explanation is based on CNN.

라고 정의한다.

는 아래 수학식 1과 같이 정의된다. C_i는 i 번째 계층의 채널 크기이다.In the forward process, the set of forward activation maps for the ith layer is

Is defined as.

Is defined as in Equation 1 below. C _i is the channel size of the i-th layer.

이다. f는 해당 계층의 함수이다.C _i is the channel size of the i-th layer, and the convolutional kernel of each channel is

to be. f is a function of the layer.

라고 정의한다.

는 아래 수학식 2와 같이 정의된다.In the reverse process, the set of reverse relevance for the i-th layer is

Is defined as.

Is defined as in Equation 2 below.

와

의 비율로 재정규화된다. 도 4에 도시된 바와 같이

및

는 각각 i 번째 계층의 내측(inward) 액티베이션 맵과 관련성에 해당한다.At this time, the kernel

Wow

Are renormalized at the rate of. As shown in Figure 4

And

Each corresponds to the inward activation map and relevance of the i-th layer.

는 순방향 특징맵과 역방향 계층의 관련성을 이용한 손실 정보이다.In each layer, an average value obtained by summing a new loss function using forward information and backward information may be defined as in Equation 3 below. under

Is loss information using the relationship between the forward feature map and the reverse layer.

은 두 개의 입력 값(term)의 절대 차이를 연산하는 l₁ 손실 함수이다.K is the number of layers in the neural network.

Is an l ₁ loss function that calculates the absolute difference between two input terms.

The total loss function can be defined as in Equation 4 below.

는 신경망의 순방향 과정에서 산출되는 손실함수이다. 즉, 실제 라벨값과 크로스-엔트로피 손실 사이의 손실이다. 도 4에서 L_T로 표시한 값이다.

Is the loss function calculated in the forward process of the neural network. That is, it is the loss between the actual label value and the cross-entropy loss. It is a value expressed as L _T in FIG. 4.

Through this process, the loss function is newly defined. Therefore, the above-described process can be referred to as a normalization technique using relevance transcription.

Now, we can train the neural network using the final loss function L.

5 is an example of a flow chart for the DNN learning process 100. The neural network learning process is performed by a computing device or a computer device capable of processing input data. Since the computer device performs neural network learning, the computer device may be referred to as a learning device. For convenience of description, it is assumed that the computer device performs neural network learning.

The computer device receives the training data (110). The training data may be image data or the like. Initially, the weight of each layer may be set to a random value.

The computer device performs a forward transcription process using the training data (120). Each layer generates a feature map through the forward process. Through the forward process, the output layer outputs a judgment on a specific problem. And the output layer calculates the loss L _T by comparing the actual label value and the output judgment.

The computer device performs the hierarchical association transfer (LRP) described above (130). In this process, the L ₁ loss function for the forward feature map and the reverse relationship information is calculated in each layer. In addition, the computer device determines the final loss function L by summing all L ₁ loss functions of each layer and L _T (140). At this time, as described in Equation 3, the computer device may determine the final loss function L by summing all the L ₁ loss functions of each layer and summing the average value and L _T.

The computer device may update the weight of each layer by performing reverse transcription based on the final loss function L (150).

6 is another example of a flowchart for the DNN learning process 200.

When the relevance calculation process is performed at the initial stage of neural network learning, relevance information is computed from judgment information with low accuracy. In this case, there is a risk that inaccurate information is reflected in the weight learning, so that the weight learning of the neural network is not optimized. Accordingly, after learning is performed according to the conventional neural network learning method, after the accuracy of the neural network determination reaches a certain degree, the weight of the neural network may be updated according to the above-described method. 6 is an example of such a process.

The computer device receives training data (210). The training data may be image data or the like. Initially, the weight of each layer may be set to a random value.

The computer device performs a forward transcription process using the training data (220). Each layer generates a feature map through the forward process. Through the cruise process, the output layer outputs a judgment on a specific problem. And the output layer calculates the loss L _T1 by comparing the actual label value with the output judgment. The computer device may update the weight of each layer by performing reverse transcription based on the final loss function L _T1 (230).

The computer device determines whether the difference between the actual label value and the result of the current neural network output layer is greater than or equal to a predetermined reference value (240). When the learning accuracy is less than the reference value (NO of 240), the computer device may repeatedly perform a forward transfer and a reverse transfer process using the result.

When the learning accuracy is greater than or equal to the reference value (YES in 240), the computer device may perform hierarchical association transcription (250). At this point, it is assumed that the loss function determined by the output layer is L _Tk . In this process, the computer device calculates the L ₁ loss function for the forward feature map and the reverse relationship information in each layer.

The computer device determines a final loss function L by summing all L ₁ loss functions of each layer and L _Tk (250). In this case, as described in Equation 3, the computer device may determine the final loss function L by summing all the L ₁ loss functions of each layer and summing the average value and L _Tk .

The computer device may update the weight of each layer by performing reverse transcription based on the final loss function L (260).

7 is an example of a service device 300 including a DNN. The service device 300 refers to a device that provides a certain application service using a neural network model prepared by the aforementioned neural network learning method. The service device 300 may be a computer device that has learned a neural network. Alternatively, the service device 300 may be a device separate from the computer device that has learned the neural network.

The service device 300 includes a storage device 310, a memory 320, an operation device 330, an interface device 340, and a communication device 350.

The storage device 310 stores programs or codes for the operation of the service device 300. The storage device 310 stores the neural network model prepared by the above-described learning method. Furthermore, the storage device 310 may store a program or code for the aforementioned neural network learning process.

The memory 320 may temporarily store data and information generated during the operation of the service device 300.

The interface device 340 is a device that receives certain commands and data from the outside. The interface device 340 may receive certain information from a physically connected input device or a physical interface (keypad, touch panel, etc.). The interface device 340 may receive a neural network model, information for training a neural network model, and training data. The interface device 340 may receive input data for input to the neural network model.

The communication device 350 transmits and receives certain information through a wireless network. The communication device 350 may receive a neural network model, information for training a neural network model, and training data. The communication device 350 may receive input data for input to the neural network model. The communication device 350 may transmit a service result (a determination result) using a neural network model to an external object.

The interface device 340 and the communication device 350 may receive certain information and data from a user or an external object. Accordingly, the interface device 340 and the communication device 350 may be collectively referred to as an input device.

The computing device 330 controls the operation of the service device using programs or codes stored in the storage device 310. The computing device 330 may perform a certain determination process using the learned neural network model. The computing device 330 may input the input data into the learned neural network model to generate a certain determination result. For example, the computing device 330 may detect an object of an image, a determination result for a specific input, and a control command for a specific input.

Furthermore, the computing device 330 may update the neural network model by feeding back the calculated result. In this process, the computing device 330 may update the weight of the neural network model through the above-described learning process. The computing device 330 may be a device such as a processor, an AP, or a chip in which a program is embedded that processes data and processes certain operations.

In addition, the above-described DNN or neural network learning method may be implemented as a program (or application) including an executable algorithm that can be executed on a computer. The program may be provided by being stored in a non-transitory computer readable medium.

The non-transitory readable medium refers to a medium that stores data semi-permanently and can be read by a device, not a medium that stores data for a short moment, such as a register, cache, or memory. Specifically, the above-described various applications or programs may be provided by being stored in a non-transitory readable medium such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, and ROM.

The present embodiment and the accompanying drawings are merely illustrative of some of the technical ideas included in the above-described technology, and those skilled in the art will be able to easily within the scope of the technical ideas included in the specification and drawings of the above-described technology. It will be apparent that all of the modified examples and specific embodiments that can be inferred are included in the scope of the rights of the above-described technology.

Claims (10)

Determining, by the learning device, a first loss function by performing a forward transfer process using the training data on the neural network;
Calculating, by the learning device, a loss function for each layer based on a feature map of each layer and information on the backward relationship in a backward relationship transfer process with respect to the neural network;
Determining, by the learning device, a final loss function by summing a second loss function generated as a result of summing each layer loss function and the first loss function; And
And updating, by the learning device, a weight of each layer while performing a reverse transcription process using the final loss function. The method of claim 1,
The learning device further comprises initial learning of the neural network while performing a forward transcription process and a reverse transcription process using the learning data,
A neural network learning method using relevance transcription in which a process of calculating the loss function for each layer is performed when the determination result of the initially learned neural network has higher accuracy than a reference value. The method of claim 1,
The loss function for each layer of the i-th layer of the neural network is a loss function between a feature map generated in the i-1th layer and a backward relationship in the i-th layer. The method of claim 3,
A neural network learning method using correlation transcription in which the reverse relationship in the i-th layer is determined by the reverse activation map below.

(

Is the backward relationship of the i-th layer, C _i is the channel size of the i-th layer, w is the weight,

Is the forward activation map of the i-th layer) The method of claim 1,
The second loss function is a neural network learning method using association transcription, which is an average value of the loss function for each layer of the entire layer. A computer-readable recording medium in which a program for executing the neural network learning method using the association transcription according to any one of claims 1 to 5 is recorded on a computer. An input device receiving input data;
A storage device that stores a neural network model trained using relevance transcription; And
Including a computing device for inputting the input data into the neural network model and generating specific service information using a result output from the neural network model,
The neural network model is a service apparatus using a neural network trained using a relationship transcription learned based on a final loss function generated using a loss function for each layer generated in a backward relationship transcription process. The method of claim 7,
The final loss function is
A first loss function determined by performing a forward transcription process on the neural network model based on the training data and a backward relationship to the neural network model for each layer determined on the basis of the feature map of each layer and the backward relationship information in the transcription process A service device using a neural network learned by using relationship transcription, which is a loss function that is a summation of the second loss function generated using the loss function. The method of claim 7,
The loss function for each layer of the i-th layer of the neural network is a service device using a neural network learned using relationship transcription, which is a loss function between the feature map generated in the i-1th layer and the backward relationship in the i-th layer . The method of claim 9,
A service apparatus using a neural network learned by using a relationship transcription, in which the reverse relationship in the i-th layer is determined by a reverse activation map below.

(

Is the backward relationship of the i-th layer, C _i is the channel size of the i-th layer, w is the weight,

Is the forward activation map of the i-th layer)

Images