KR20230173840A - Device for training binary neural networks based on efficient gradient computations and method thereof - Google Patents

Abstract

The present invention relates to a lightweight neural network, and to a binary neural network learning method and device using efficient differential calculation, which efficiently calculates the differential value itself by approximating it and transfers the value to the weights to learn. According to an embodiment of the present invention, artificial intelligence neural network learning and execution can be performed in the environment of mobile devices or small devices that lack memory, power, or computing power.

Description

Binary neural network learning method and device using efficient differential calculation {DEVICE FOR TRAINING BINARY NEURAL NETWORKS BASED ON EFFICIENT GRADIENT COMPUTATIONS AND METHOD THEREOF}

The present invention relates to a lightweight neural network, and to a binary neural network learning method and device using efficient differential calculation, which efficiently calculates the differential value itself by approximating it and transfers the value to the weights to learn.

Artificial intelligence is a field of research that focuses on equipping machines with the ability to solve problems that require human intellectual ability. It has made great progress due to the following factors:

First, artificial intelligence models based on artificial neural networks (ANNs) enabled an efficient learning method (the so-called backpropagation algorithm) based on gradients.

Second, a system was established to generate and collect large-scale data using various devices.

Lastly, computing resources that can store large-scale data and process the large-scale calculations required for learning using artificial intelligence models with complex structures have become abundant. Based on these factors, the application of artificial intelligence is expanding to various fields that require complex problem solving, such as computer vision and natural language processing, and is attracting attention as a technology that brings about significant changes in industrial and social structures.

However, the complexity of the artificial neural network structure increases depending on the complexity of the problem to be solved, and in particular, an artificial neural network structure consisting of a large number of neurons and a large number of layers is required. These deep neural networks require a large amount of computation in both learning and inference, and consequently consume a lot of resources. As a result, there is a high possibility that general artificial intelligence models and learning methods based on deep neural networks will not operate smoothly in environments with constraints on battery, memory, and computing power, such as mobile devices and Internet of Things devices.

Accordingly, one of the representative ways to reduce the resources required for learning and inference of an artificial neural network or deep neural network is to solve the problem using a lightweight neural network, using a weight quantization technique that compresses the weights of neurons in the neural network and There is a technique for pruning weights. The quantization technique replaces 32-bit decimal operations by compressing weights with 32-bit float values into fewer bits, thereby reducing the storage capacity of each weight. In the extreme, it can be quantized to have values of -1 and 1. On the other hand, the pruning technique is a method of removing or pruning weights that are small and less important among the weights that make up the neural network, and as a result, the complexity and size of the model can be reduced.

Binary Neural Network (BNN), one of these lightweight neural networks, maximizes the complexity of the model by quantizing each weight (parameter) and activation result of the neural network to 1 bit, the smallest number of bits, so that it has a binary value. It is a shortened model. The weights and activation values of a binary neural network are made up of -1 and +1 (or 0 and 1), allowing calculations to be performed using much fewer resources, but the existing backpropagation learning technique cannot be applied due to the binarization process. . Specifically, the sign function is used to binarize the weights, and the derivative (gradient) of the sign function is 0 in most cases.

On the other hand, because modern artificial intelligence learning corrects learning errors through differentiation (gradient) of weights, there is a difficulty in applying existing learning techniques directly to binary neural networks. Accordingly, despite the improvement in energy efficiency due to the binarization process of the binary neural network, consideration is needed on how to efficiently calculate the differential (gradient) in order to use it for learning.

In research to solve this problem, methods of applying existing learning techniques by approximating the sign function with a differentiable function were mainly presented. Calculating the derivative (gradient) using a function that approximates the sign function has the advantage of being able to apply existing learning techniques without major problems, but has the disadvantage of not being able to accurately calculate the derivative (gradient) value of a binary neural network. This exists. On the other hand, the research methodology that uses a probabilistic binarization process instead of a sign function-based deterministic binarization process has the advantage of being able to calculate differential (gradient) values, but requires more complexity in the binarization process. Since the hardware implementation is not clear, active research has not yet been conducted.

1. Korean Patent Publication No. 10-2345409 “Processor and processor operation method for accelerating convolution operation in convolutional neural network” (Publication date: March 10, 2021)

Unlike the sign function approximation technique or the stochastic binarization method, the present invention provides a learning method for a binary neural network that efficiently calculates the differential (gradient) value itself by approximating it and transfers the value to the weights to learn.

The present invention provides a lightweight neural network that operates with few resources by using auxiliary variable values for approximating and calculating the derivative of a quantized value and binarizing the weight of the sign function as is.

According to one aspect of the present invention, a binary neural network learning device using efficient differential calculation is provided.

A binary neural network learning device using efficient differential calculation according to an embodiment of the present invention may include a binarization unit that performs a binarization step for binary neural network learning and a differential calculation unit that calculates the differential value required for binary neural network learning. there is.

According to another aspect of the present invention, a method for learning a binary neural network using efficient differential calculations and a computer program for executing the same are provided.

A binary neural network learning method using efficient differential calculation and a computer program executing the method according to an embodiment of the present invention may include the steps of binarizing weights, binarizing activation result values, and differentiating real value weights. there is.

According to an embodiment of the present invention, artificial intelligence neural network learning and execution can be performed in the environment of mobile devices or small devices that lack memory, power, or computing power.

In addition, according to an embodiment of the present invention, since binarization of a binary neural network is simple, the result of each layer multiplied by the weight can be binarized to replace the XNOR binary operation, thereby reducing computational complexity and making it effective.

Additionally, according to an embodiment of the present invention, artificial neural network learning is possible on devices with insufficient computing resources, thereby expanding the field of application of artificial intelligence.

Figure 1 shows an example of the structure of a binary neural network.
2 to 5 are diagrams illustrating a binary neural network learning device using efficient differential calculation according to an embodiment of the present invention.
Figure 6 is a diagram illustrating a binary neural network learning method using efficient differential calculation according to an embodiment of the present invention.
7 and 8 are exemplary diagrams of a binary neural network learning device using efficient differential calculation according to an embodiment of the present invention.
9 to 12 are diagrams illustrating experimental results of a binary neural network learning device using efficient differential calculation according to an embodiment of the present invention.

Since the present invention can make various changes and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail through detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all changes, equivalents, and substitutes included in the spirit and technical scope of the present invention. In describing the present invention, if it is determined that a detailed description of related known technologies may unnecessarily obscure the gist of the present invention, the detailed description will be omitted. Additionally, as used in this specification and claims, the singular expressions “a,” “a,” and “an” should generally be construed to mean “one or more,” unless otherwise specified.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, identical or corresponding components will be assigned the same drawing numbers and redundant description thereof will be omitted. Do this.

Figure 1 is an example of the structure of a binary neural network.

The binary neural network is a model that minimizes the complexity of the artificial intelligence model by quantizing each weight (parameter) and activation result value of the neural network to the smallest number of bits, 1 bit, to have a binary value. The weights and activation values of a binary neural network consist of -1 and 1 (or 0 and 1), allowing calculations to be performed using much fewer resources, but the existing backpropagation learning technique cannot be applied due to the binarization process. Specifically, the derivative (gradient) of the sign function used for binarization of weights is 0 in most cases. Since recent artificial intelligence learning corrects learning errors through differentiation (gradient) of weights, despite the improvement in energy efficiency due to the binarization process, differentiation-based learning techniques cannot be applied to binarized weights in binary neural networks. I can't. In addition, when calculating using a function that approximates the sign function, it is easy to calculate the differential (gradient) value, but since accurate binarization is not performed, computational efficiency cannot be obtained.

The present invention can implement a binary neural network that efficiently calculates the differential (gradient) value itself by approximating it and transfers the value to the weights to learn.

2 to 5 are diagrams for explaining a binary neural network learning device using efficient differential calculation according to an embodiment of the present invention.

Referring to FIG. 2, the binary neural network learning apparatus 10 using efficient differential calculation may include a binarization unit 100 and a differential calculation unit 200.

The binarization unit 100 may perform a binarization step for learning a binary neural network. The binarization unit 100 allows the overall calculation of the neural network to proceed through binary values.

The binarization unit 100 can binarize both the weight and the activation result of each layer. When both weights and activation results are binarized, all operations in a binary neural network are possible with XNOR, so the complexity of operations is also reduced, drastically reducing the energy consumption required for learning and execution.

Referring to FIG. 3, the binarization unit 100 may include a weight binarization unit 110 and an activation binarization unit 120.

The weight binarization unit 110 can binarize all weights of the neural network. The weight binarization unit 110 may binarize the weight using a sign function. For example, the weight binarization unit 110 may output +1 if the value of input x is greater than or equal to 0, and output -1 if the value of x is less than 0.

The weight binarization unit 110 calculates w ^b by binarizing the weight w of the real value as shown in [Equation 1].

를 계산하고, 이를 부호 함수(Sign)를 사용해 이진화 할 수 있다.The activation binarization unit 120 calculates the activation result of several layers using the binarized weight value and binarizes the activation result using the sign function. can do. The activation binarization unit 120 adds the product of the input values received from neurons of the previous layer and the binarized weights.

You can calculate and binarize it using the sign function (Sign).

Referring to FIG. 4, the binarization unit 100 uses a deterministic binarization method using a sign function, so constant binarization is always performed depending on the input value, and a 32-bit real number is reduced to 1 bit. Therefore, it can be implemented using approximately 32 times less memory. In detail, Figure 4(a) is the sign function for deterministic binarization, and Figure 4(b) is the differentiation (gradient) of the sign function.

The binarization unit 100 performs binarization of the weight and activation in the forward direction to obtain a binarized weight value w ^b for the real value weight w.

Referring again to FIG. 2, the differential calculation unit 200 can efficiently calculate the differential (gradient) value required for binary neural network learning.

The differential calculation unit 200 can calculate the differential value by approximating it using the auxiliary variable value to solve the problem in which the differential (gradient) value is 0 in most areas due to binarization through the sign function. For example, the differential calculation unit 200 may use the Straight Through Estimator (STE) technique.

The differential calculation unit 200 calculates the differential (gradient) value according to the chain rule as shown in [Equation 3].

의 값은 대부분 0의 값을 가지므로 유의미한 학습을 위해, Straight Through Estimator(STE) 기법에서는

의 값을 1로 가정하여, 미분(그래디언트) 값을 계산하도록 한다.In a binary neural network, the loss function ( L ) of the neural network is generally differentiated with respect to the binarized weight w ^b , but due to the nature of the sign function, this may cause a problem in that it has a value of 0 in most cases. In detail, due to deterministic binarization,

Most of the values of have a value of 0, so for meaningful learning, the Straight Through Estimator (STE) technique uses

Assuming the value of is 1, calculate the differential (gradient) value.

However, the Straight Through Estimator (STE) technique has a problem in that the error increases when the range difference between the binarized value and the actual real number value is large.

Referring to FIG. 5, the differential calculation unit 200 uses a method of limiting (clipping) real values to between -1 and +1 to prevent errors from increasing. Figure 5(a) is the differentiation (gradient) of the sign function, and Figure 5(b) is an approximation used in differential calculations based on the Straight Through Estimator (STE), which limits (clipping) real values to between -1 and 1 in the reverse process. It is a function.

The differential calculation unit 200 can learn the value itself by differentiating the loss function ( L ) in the reverse direction with respect to the real value weight w and use it as an auxiliary variable value.

Figure 6 is a diagram illustrating a binary neural network learning method using efficient differential calculation according to an embodiment of the present invention. Each process described below is a process performed by each functional unit constituting the binary neural network learning device using efficient differential calculation in each step, but for a concise and clear explanation of the present invention, the subject of each step is a binary neural network using efficient differential calculation. It will be collectively referred to as a network learning device.

The binary neural network learning device 10 using efficient differential calculation can randomly set the initial value w and perform binary neural network learning as shown in FIG. 6.

Referring to FIG. 6, in step S610, the binary neural network learning device 10 using efficient differential calculation binarizes the weights using real-valued weights and sign functions in each layer.

In step S620, the binary neural network learning device 10 using efficient differential calculation calculates and binarizes the binarized weights and the activation result values in the forward direction across multiple layers.

In step S630, the binary neural network learning device (10) using efficient differential calculation obtains the loss function and uses STE in the reverse direction to calculate the derivative (gradient) of the real value weight.

In step S640, the binary neural network learning device 10 using efficient differential calculation updates real value weights using gradient descent. For example, the learning speed can be improved by using gradient descent methods such as Adam, AdaGrad, and RMSProp.

In step S650, the binary neural network learning device 10 using efficient differential calculation ends learning when the differential (gradient) value is less than a certain value. The binary neural network learning device 10 using efficient differential calculation compares the differential (gradient) value with a certain value, returns to step S610, and performs this repeatedly until the differential (gradient) value becomes smaller than the certain value.

Figures 7 and 8 are exemplary diagrams of a binary neural network learning device using efficient differential calculation according to an embodiment of the present invention.

Referring to FIG. 7, the binary neural network learning device 10 using efficient differential calculation binarizes the weights using real-valued weights and sign functions, and calculates and binarizes the activation result values. Additionally, the binary neural network learning device 10 using efficient differential calculation can calculate a loss function by performing a binarization process in the forward direction across each layer.

Referring to FIG. 8, the binary neural network learning device 10 using efficient differential calculation can differentiate the loss function with respect to the real value weight based on STE in the reverse direction, learn the weight differential value, and use it as an auxiliary variable.

9 to 12 are diagrams illustrating experimental results of a neural network learning device using efficient differential calculation according to an embodiment of the present invention.

To check the performance of the neural network learning device (10) using efficient differential calculation, an experiment was conducted using a total of three data sets: ECG200, ECG5000, and ECGThorax. Each data set is electrocardiogram data, and these are the results of an experiment in which the binary neural network learning device 10 using efficient differential calculations learned a classification problem to classify cardiac abnormalities.

Figure 9 is an example diagram briefly expressing the binary neural network structure used in the experiment by the binary neural network learning device using efficient differential calculation according to an embodiment of the present invention.

Referring to FIG. 9, the binary neural network learning device 10 using efficient differential calculation can be composed of an input layer, an output layer, and two hidden layers. In the experiment, the binary neural network learning device 10 using efficient differential calculation varied the number of neurons in the two hidden layers depending on the data set. In detail, the binary neural network learning device 10 using efficient differential calculation set 1024 neurons for each layer in the ECG200 and ECG5000 data sets, and 4096 neurons in the ECGThorax data set.

The example in Figure 10 is an example of comparing a normal heartbeat and a heartbeat in ischemic heart disease based on the ECG200 data set, and is a standard for classifying heart rhythm abnormalities.

11 and 12 are examples of learning results of a binary neural network using efficient differential calculation according to an example of the present invention.

In the ECG200 data set environment, a binary neural network learning device (10) using efficient differential calculation conducted a binary classification experiment to classify which of 200 heartbeats was abnormal using the ECG200 data set. The binary neural network learning device (10) using efficient differential calculation was trained using 100 data sets as training data and 100 data sets as test data in the ECG200 data set.

Figure 11(a) is an example of the accuracy of a binary neural network learned by the binary neural network learning device 10 using efficient differential calculation with the ECG200 data set.

Figure 11(b) is an example of a learning curve for the ECG200 data set of a binary neural network learning device using efficient differential calculation.

Looking at the example of FIG. 11(b), it can be seen that the binary neural network learning device 10 using efficient differential calculation performs learning well, increasing accuracy and reducing loss at the beginning. The binary neural network learning device 10 using efficient differential calculation gradually increases the loss of the test set after about 50 training sessions, and shows overfitting. The test accuracy of the binary neural network learning device 10 using efficient differential calculation is about 93%.

Referring to FIG. 12, the binary neural network learning device 10 using efficient differential calculation operates well with the STE-based learning algorithm in the ECG5000 data set and the ECGThorax data set, similar to the ECG200 data set.

Figure 12(a) is an example of learning results for the ECG5000 data set performed by the binary neural network learning device 10 using efficient differential calculation. In the ECG5000 data set environment, the test accuracy of the binary neural network learning device 10 using efficient differential calculation is about 94.62%, and it can be seen that the accuracy weakens from the 100th learning.

In the ECG5000 data set environment, the binary neural network learning device (10) using efficient differential calculation tested the problem of classifying 5 classes into 1 normal beat and 4 abnormal beats with 500 learning data and 4500 test data.

Figure 12(b) is an example of learning results for the ECGThorax data set performed by the binary neural network learning device 10 using efficient differential calculation. In the ECGThorax data set environment, the test accuracy of the binary neural network learning device (10) using efficient differential calculation was about 9121%, and the performance did not deteriorate significantly until it was repeated 2000 times.

In the ECGThorax data set environment, the binary neural network learning device (10) using efficient differential calculation was tested on the problem of classifying the fetal heart rate into 42 classes, divided into 1800 learning data and 1965 test data.

DATA SET BNN(STE) (FULL-PRECISION) N.N. (OPTIMIZED) NN ECG5000 94.62% 94.62% 94.73% ECGTHORAX 91.21% 92.73% 94.68% ECG200 93% 94% 89.05%

[Table 1] shows the learning accuracy of the binary neural network learning device 10 using efficient differential calculation and the learning accuracy of the existing neural network (NN). In the case of a neural network (FULL-PRECISION NN) that did not go through any binarization process, it was commonly learned using a two-layer model of 4096 neurons, and in the case of OPTIMIZED NN, a conventional research model (start-of-the-line) with the best performance was used. -art). Referring to [Table 1], the binary neural network learning device 10 using efficient differential calculation can be applied to various data, and the structure of the model can also be varied by using each different model. there is. This means that the binary neural network learning device 10 using efficient differential calculations has various usability in various real-life healthcare-related applications based on mobile terminals or smart devices.

The binary neural network learning method using efficient differential calculation described above can be implemented as computer-readable code on a computer-readable medium. The computer-readable recording medium may be, for example, a removable recording medium (CD, DVD, Blu-ray disk, USB storage device, removable hard disk) or a fixed recording medium (ROM, RAM, computer-equipped hard disk). You can. The computer program recorded on the computer-readable recording medium can be transmitted to another computing device through a network such as the Internet, installed on the other computing device, and thus used on the other computing device.

In the above, even though all the components constituting the embodiment of the present invention have been described as being combined or operated in combination, the present invention is not necessarily limited to this embodiment. That is, within the scope of the purpose of the present invention, all of the components may be operated by selectively combining one or more of them.

Although operations are shown in the drawings in a specific order, it should not be understood that the operations must be performed in the specific order shown or sequential order or that all illustrated operations must be performed to obtain the desired results. In certain situations, multitasking and parallel processing may be advantageous. Moreover, the separation of the various components in the embodiments described above should not be construed as necessarily requiring such separation, and the program components and systems described may generally be integrated together into a single software product or packaged into multiple software products. You must understand that it exists.

So far, the present invention has been examined focusing on its embodiments. A person skilled in the art to which the present invention pertains will understand that the present invention may be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered from an illustrative rather than a restrictive perspective. The scope of the present invention is indicated in the claims rather than the foregoing description, and all differences within the equivalent scope should be construed as being included in the present invention.

10: Binary neural network learning device using efficient differential calculation
100: binarization unit
110: Weight binarization unit
120: Activation binarization unit
200: Differential calculation unit

Claims (6)

In a binary neural network learning device using efficient differential calculation,
A binarization unit that performs the binarization step for learning a binary neural network; and
A binary neural network learning device using efficient differential calculation, including a differential calculation unit that calculates the differential value required for learning the binary neural network.
According to paragraph 1,
The differential calculation unit
Calculate by approximating the differential value using the auxiliary variable value.
Binary neural network learning device using efficient differential calculation.
According to claim 1,
The binarization unit
A weight binarization unit that binarizes the weight and
Contains an activation binarization unit that binarizes the activation result value using a sign function.
Binary neural network learning device using efficient differential calculation.
In the learning method performed by a binary neural network learning device using efficient differential calculation,
Binarizing the weights;
Binarizing the activation result value; and
comprising differentiating real-valued weights.
Binary neural network learning method using efficient differential calculation.
According to paragraph 4,
The step of differentiating the real value weight is
Calculate by approximating the differential value using the auxiliary variable value.
Binary neural network learning method using efficient differential calculation.
A computer program recorded on a computer-readable recording medium that executes the binary neural network learning method using efficient differential calculation according to any one of claims 4 and 5.

Images