KR20220030108A - Method and system for training artificial neural network models - Google Patents

Abstract

The present disclosure relates to a method for training an artificial neural network model. The method for training the artificial neural network model comprises: a step of training a first artificial neural network model up to a preset epoch by a momentum-based gradient descent method, and determining a momentum value of the first artificial neural network model at the epoch; a step of setting a determined momentum value as an initial momentum value of a second artificial neural network model; and a step of updating a parameter value of the second artificial neural network model using the training data based on the initial momentum value, wherein the parameter comprises a plurality of weights and the momentum. Therefore, the present invention is capable of reducing errors.

Description

Artificial neural network model training method and system {METHOD AND SYSTEM FOR TRAINING ARTIFICIAL NEURAL NETWORK MODELS}

The present disclosure relates to a method and system for learning an artificial neural network model, and more particularly, to a method and system for learning an artificial neural network model and generating a quantized artificial neural network model.

An artificial neural network may refer to a computing device or a method or data structure performed by a computing device to implement interconnected sets of artificial neurons (or neuron models). In an artificial neural network, an artificial neuron included in an arbitrary layer may generate output data by performing operations such as multiplication of weights or application of an activation function to input data, and the output data is transmitted to an artificial neuron included in another layer. can be As an example of an artificial neural network, a deep neural network or deep learning model includes a plurality of layers, and nodes of each layer are trained according to a plurality of training data to recognize the characteristics of the training data. . Such an artificial neural network may be applied to various application fields such as classification, object detection, semantic segmentation, or style transfer.

In the related art, research on quantizing the weight and/or activation of the artificial neural network model is being actively conducted in order to reduce the complexity of the artificial neural network model and to speed up the processing of the artificial neural network model. Quantization of such an artificial neural network model aims to approximate a floating-point operation represented by a high number of bits of a full-precision model by operation of fixed-point data represented by a small number of bits. That is, it aims at transfer learning from a pre-trained full-precision model to a quantized counterpart.

However, approximation errors due to quantization are accumulated in the process of forward-propagation calculation, and accordingly, the performance of the artificial neural network model may be greatly degraded. In particular, in the case of a lightweight model, there is a problem in that it is difficult to use the weights of the pre-trained high-precision model as it is due to an initial statistical error due to quantization. To solve this problem, a method for simulating network quantization in the learning process may be used. For example, quantization-aware training (QAT) uses a straight-through estimator (STE) to simulate quantized inference in the forward propagation process and calculate the gradient in the backpropagation process. use. Although the QAT method can reduce the number of differences and outlier weight values in the range of weights, it is difficult to overcome the gradient approximation error caused by STE.

The present disclosure provides a method for learning an artificial neural network model, a computer program stored in a recording medium, and an apparatus (system) for solving the above problems.

The present disclosure may be implemented in various ways including a method, an apparatus (system), or a computer program stored in a readable storage medium.

According to an embodiment of the present disclosure, the artificial neural network model learning method performed by at least one processor learns the first artificial neural network model until a preset epoch by a momentum-based gradient descent method, Determining the momentum value of the first artificial neural network model in the epoch, setting the determined momentum value as the initial momentum value of the second artificial neural network model, and the second artificial neural network using training data based on the initial momentum value updating a parameter value of the model, wherein the parameter includes a plurality of weights and momentum.

There is provided a computer program stored in a computer-readable recording medium to execute the artificial neural network model learning method according to an embodiment of the present disclosure on a computer.

The artificial neural network model learning system according to an embodiment of the present disclosure includes at least one processor configured to execute at least one computer-readable program included in the communication module, the memory and the memory and is connected to the memory, and at least one The program learns the first artificial neural network model up to the preset epoch by the momentum-based gradient descent method, determines the momentum value of the first artificial neural network model at the epoch, and sets the determined momentum value to the initial value of the second artificial neural network model. Set as a momentum value and include instructions for updating a parameter value of the second artificial neural network model using training data based on the initial momentum value, and the parameter includes a plurality of weights and momentum.

In various embodiments of the present disclosure, by setting an appropriate initial momentum value of the QAT model, parameters may be induced to be updated in an appropriate learning direction, and erroneous gradient values may be ignored. Accordingly, it is possible to prevent an error due to quantization from being amplified as the epoch is repeated, and to reduce an error occurring when learning/updating parameters of the QAT model.

In various embodiments of the present disclosure, compared to the FP model, the quantized model requires low-cost and low-capacity resources in the process of generation, learning, and inference, and can execute the inference process at a faster speed.

In various embodiments of the present disclosure, the quantized model may have performance close to or more improved than that of the FP model, unlike a model obtained by quantizing the existing QAT model.

According to various embodiments of the present disclosure, it is possible to create and train a lightweight model having excellent performance for processing such as classification, object detection, semantic segmentation, or style transfer.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments of the present disclosure will be described with reference to the accompanying drawings described below, in which like reference numerals denote like elements, but are not limited thereto.
1 is an artificial neural network learning that generates an 8-bit integer (INT8: Integer 8-bit) artificial neural network model by executing quantization cognitive learning (QAT) on a high-precision (FP) artificial neural network model according to an embodiment of the present disclosure; It is a diagram showing an example of the process.
2 is a block diagram illustrating an internal configuration of an artificial neural network model learning system according to an embodiment of the present disclosure.
3 is a block diagram illustrating an internal configuration of a processor connected to a learning data DB according to an embodiment of the present disclosure.
4 is a flowchart illustrating a method for learning an artificial neural network model according to an embodiment of the present disclosure.
5 is a flowchart illustrating a method of updating a parameter value of a second artificial neural network model according to an embodiment of the present disclosure.
6 is histograms of weight gradient values in each layer for performance evaluation of an artificial neural network model created and learned according to embodiments of the present disclosure.
7 is loss function graphs (loss landscapes) for performance evaluation of an artificial neural network model created and trained according to embodiments of the present disclosure.
8 is an example of a performance evaluation result of an artificial neural network model before image style generated and learned according to embodiments of the present disclosure.
9 is a flowchart of an algorithm for performing a method for learning an artificial neural network model according to an embodiment of the present disclosure.

Hereinafter, specific contents for carrying out the present disclosure will be described in detail with reference to the accompanying drawings. However, in the following description, if there is a risk of unnecessarily obscuring the gist of the present disclosure, detailed descriptions of well-known functions or configurations will be omitted.

In the accompanying drawings, the same or corresponding components are assigned the same reference numerals. In addition, in the description of the embodiments below, overlapping description of the same or corresponding components may be omitted. However, even if descriptions regarding components are omitted, it is not intended that such components are not included in any embodiment.

Advantages and features of the disclosed embodiments, and methods of achieving them, will become apparent with reference to the embodiments described below in conjunction with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only the present embodiments allow the present disclosure to be complete, and the present disclosure provides those skilled in the art with the scope of the invention. It is provided for complete information only.

Terms used in this specification will be briefly described, and the disclosed embodiments will be described in detail. The terms used in this specification have been selected as currently widely used general terms as possible while considering the functions in the present disclosure, but these may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technology, and the like. In addition, in a specific case, there is a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the corresponding invention. Therefore, the terms used in the present disclosure should be defined based on the meaning of the term and the content throughout the present disclosure, rather than the simple name of the term.

References in the singular herein include plural expressions unless the context clearly dictates the singular. Also, the plural expression includes the singular expression unless the context clearly dictates the plural. In the entire specification, when a part includes a certain element, this means that other elements may be further included, rather than excluding other elements, unless otherwise stated.

In addition, the term 'module' or 'unit' used in the specification means a software or hardware component, and 'module' or 'unit' performs certain roles. However, 'module' or 'unit' is not meant to be limited to software or hardware. A 'module' or 'unit' may be configured to reside on an addressable storage medium or configured to reproduce one or more processors. Thus, as an example, a 'module' or 'unit' refers to components such as software components, object-oriented software components, class components, and task components, processes, functions, and properties. , procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, database, data structures, tables, arrays or at least one of variables. Components and 'modules' or 'units' are the functions provided therein that are combined into a smaller number of components and 'modules' or 'units' or additional components and 'modules' or 'units' can be further separated.

According to an embodiment of the present disclosure, a 'module' or a 'unit' may be implemented with a processor and a memory. 'Processor' should be construed broadly to include general purpose processors, central processing units (CPUs), microprocessors, digital signal processors (DSPs), controllers, microcontrollers, state machines, and the like. In some contexts, a 'processor' may refer to an application specific semiconductor (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), or the like. 'Processor' refers to a combination of processing devices, such as, for example, a combination of a DSP and a microprocessor, a combination of a plurality of microprocessors, a combination of one or more microprocessors in combination with a DSP core, or any other such configurations. You may. Also, 'memory' should be construed broadly to include any electronic component capable of storing electronic information. 'Memory' means random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erase-programmable read-only memory (EPROM); may refer to various types of processor-readable media, such as electrically erasable PROM (EEPROM), flash memory, magnetic or optical data storage, registers, and the like. A memory is said to be in electronic communication with the processor if the processor is capable of reading information from and/or writing information to the memory. A memory integrated in the processor is in electronic communication with the processor.

In the present disclosure, a 'full-precision or high-precision artificial neural network model' may refer to an artificial neural network model including parameters of a data type expressed by a large number of bits. For example, the high-precision neural network model may include an artificial neural network model composed of parameters (weight, momentum, kernel, activation, etc.) of a floating-point data type expressed in 32 bits or more bits. On the other hand, a 'low-precision artificial neural network model' may refer to an artificial neural network model including parameters of a data type expressed by a small number of bits. For example, the low-precision neural network model may include an artificial neural network model composed of parameters of a fixed-point or integer data type expressed in 8 bits.

In the present disclosure, 'quantization' may refer to a method of reducing the amount of memory in which the parameters of the artificial neural network are stored or the amount of computation for the parameters by reducing the number of bits of the data type of the parameters of the artificial neural network. For example, the quantization method of the artificial neural network includes a post-quantization method in which the data type of parameters of the corresponding model is converted into a data type expressed by a smaller number of bits after training of the high-precision artificial neural network model is completed; During the learning process of the artificial neural network, it may include quantization-aware training (QAT) that simulates an operation on at least some of the parameters with a small number of bit operations.

In the present disclosure, the 'error back-propagation learning' method is based on the difference (or error) between the output value of the input to the artificial neuron of the artificial neural network and the target value to be learned, parameters in the back-propagation learning process It may refer to a method of updating (eg, weight, momentum). In the error backpropagation learning method, 'gradient-descent' may be used as a method of determining weights so that the difference between the output value of the artificial neural network and the target value is minimized. When the parameters of the artificial neural network are updated according to the gradient descent method, the learning of the artificial neural network may stagnate at the local lowest point. In order to prevent this problem, momentum may be used as an element for maintaining the update direction of parameters in the previous epoch.

1 is an 8-bit integer type (INT8: Integer 8-bit) artificial neural network model 150 by executing quantization cognitive learning (QAT) on a high-precision (FP: Full-Precision) artificial neural network model according to an embodiment of the present disclosure. ) is a diagram showing an example of an artificial neural network learning process that generates

According to the illustrated artificial neural network learning process, at least some of initial parameter values for learning the QAT model 130 may be acquired through learning of the FP model 110 . In one embodiment, when the parameters of the QAT model 130 are optimized using the momentum-based gradient descent method, momentum among the initial parameters of the QAT model 130 can be obtained through learning of the FP model 110 . there is. For example, the momentum determined in the first constant period (eg, first epoch) of the parameter update of the FP model 110 may be used as the initial momentum of the QAT model 130 . In this way, by determining the initial momentum of the QAT model 130 from the momentum determined through learning up to the first predetermined period of the FP model 110, the QAT model 130 stagnates at a local minimum in the learning process. This can help to effectively search for a global minimum.

Specifically, the first model learning unit 120 may update the parameters of the FP model 110 by using the error backpropagation learning method. Here, the parameters of the FP model 110 may be expressed as a data type capable of performing a full-precision operation (eg, floating-point data expressed in 32 or more bits). Accordingly, high capacity and expensive memory and computational resources may be required to store and process parameters when generating and learning the FP model 110 and executing inference based on the model.

)을 결정할 수 있다.By learning the FP model 110 up to a preset epoch (eg, the first first epoch) in the first model learning unit 120, parameter values of the FP model 110 at the corresponding epoch can be determined. there is. For example, when the first model learning unit 120 optimizes parameters using the momentum-based gradient descent method, input training data to each layer of the FP model 110 in the first epoch to calculate an output value Executes the forward-propagation learning step, and updates the weight using a value (or gradient) obtained by partial differentiation of the difference (error) between the output value and the target value as a weight. -propagation) the learning step may be executed, which may include adding momentum to the weights along with the gradient so that the direction of updating each weight can be set to minimize the error of the FP model 110 Here, the momentum value may mean accumulating the traces of the gradient values calculated in the previous step (or the previous epoch). As such, the momentum value of the FP model 110 in the first epoch (

) can be determined.

을 QAT 모델(130)의 초기 모멘텀 값으로 설정하고, 이를 기초로, 학습 데이터를 이용하여 QAT 모델(130)을 학습/업데이트할 수 있다. 예를 들어, 제2 모델 학습부(140)는 모멘텀 기반의 경사하강법을 이용하여 QAT 모델(130)의 파라미터들을 최적화할 수 있다.The second model learning unit 140 may generate the QAT model 130 based on the momentum of the FP model 110 determined in this way, and may learn/update the generated QAT model 130 . For example, the second model learning unit 140 may perform the FP model 110 determined in the first epoch.

is set as the initial momentum value of the QAT model 130 , and based on this, the QAT model 130 may be trained/updated using the training data. For example, the second model learner 140 may optimize the parameters of the QAT model 130 by using the momentum-based gradient descent method.

In one embodiment, the second model learning unit 140 determines the predicted value by simulating the effect of quantization of parameters in the forward propagation learning process of the QAT model 130, and in the backpropagation learning process, the conventional method As shown, the QAT model 130 can be trained/updated by updating the parameter value expressed in the high-precision data type. For example, the second model learning unit 140 performs a forward propagation learning process by expressing at least some of the parameter values of the QAT model 130 as an 8-bit integer type, and the QAT model 130 expressed as a 32-bit real number type. ), a backpropagation learning process can be performed using the parameter values. Therefore, the parameter value of the QAT model 130 can be updated and stored in a 32-bit real number type, and the QAT model 130 is more than a model in which both forward propagation and back propagation learning processes are learned/updated in an 8-bit integer type. It can have high precision (or accuracy). To this end, the QAT model 130 additionally includes a fake quantization node in a path for transmitting input values to nodes included in each layer of the artificial neural network, and the second model learning unit 140 is In the forward propagation learning process, the input value is converted into a quantized 8-bit integer value through the most quantization node, so that the existing 32-bit real-type operation (for example, an operation including a parameter expressed in a 32-bit real-type) is performed. 8-bit integer arithmetic can be mimicked.

In an embodiment, the second model learning unit 140 may update a plurality of parameters of the QAT model 130 based on the gradient value in the backpropagation learning process. Specifically, the gradient value of at least one parameter among the plurality of parameters may be amplified, and the parameter value of the QAT model 130 may be updated based on the amplified gradient value. In this way, by updating the parameter values of the QAT model 130 by the second model learning unit 140, the gradient is calculated by the information including the error in the initial epoch of the QAT model 130, and It is possible to solve the problem of reducing the scope of the search area.

After the learning of the QAT model 130 is completed, the INT8 model 150 may be generated by quantizing the parameters of the QAT model 130 into a low-precision data type. For example, the INT8 model 150 may be generated by quantizing parameter values of the QAT model 130 of a 32-bit real number type into an 8-bit integer type. The INT8 model 150 generated in this way requires a lower cost and a lower capacity resource than the QAT model 130 , and an inference process using the corresponding model can be executed at a faster speed. In addition, the INT8 model 150 generated according to the above-described method is a model generated by quantizing after learning/updating an FP model including parameters expressed in a high-precision data type according to a post-quantization method. Compared to , the error due to quantization can be further reduced, and in some cases, the performance is further improved.

Although the first model learning unit 120 and the second model learning unit 140 are respectively illustrated in FIG. 1 , the present invention is not limited thereto, and the first model learning unit 120 and the second model learning unit 140 are one It can be implemented by being integrated into the learning unit of In addition, in FIG. 1 , the FP model 110 , the QAT model 130 , and the INT8 model 150 are respectively illustrated as separate artificial neural network models, but the present invention is not limited thereto, and parameter values included in one artificial neural network model structure. It may be implemented by modifying or changing their data types according to each learning step.

2 is a block diagram illustrating an internal configuration of an artificial neural network model learning system 200 according to an embodiment of the present disclosure. The artificial neural network model learning system 200 may include the at least one apparatus described above with reference to FIG. 1 or may perform the at least one method described above with reference to FIG. 1 . The artificial neural network model learning system 200, as shown, may include a communication module 210, a memory 220, and a processor 230, but is not limited thereto. The artificial neural network model learning system 200 is an artificial neural network that performs processing such as classification, object detection, semantic segmentation, or style transfer according to embodiments of the present disclosure. You can create, train, and update neural network models.

The communication module 210 may provide a configuration or function for the artificial neural network model learning system 200 to communicate with an external device or an external system (eg, a separate cloud system). For example, the artificial neural network model learning system 200 may receive training data and a program code for learning the artificial neural network model from an external system through the communication module 210 . Conversely, a control signal, command, or data provided under the control of the processor 230 of the artificial neural network model learning system 200 may be transmitted to an external device or an external system through the communication module 210 . For example, the artificial neural network model learning system 200 may provide parameters, program codes, and the like for the artificial neural network model that have been created and learned to an external device or an external system through the communication module 210 .

The memory 220 stores parameters of the artificial neural network model (eg, at least one of characteristics of inputs, weights, and characteristics of kernels constituting the artificial neural network) or a program in which an artificial neural network learning method is implemented. code can be saved. The memory 220 may be a volatile memory or a non-volatile memory. Alternatively, the memory 220 may include any non-transitory computer-readable recording medium. For example, the memory 220 is a non-volatile mass storage device such as random access memory (RAM), read only memory (ROM), disk drive, solid state drive (SSD), flash memory, etc. device) may be included.

The processor 230 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. The instructions may be provided to the processor 230 by the memory 220 or the communication module 210 . For example, the processor 230 may control the artificial neural network model learning system 200 by executing a program. The code of the program executed by the processor 230 may be stored in the memory 220 . The artificial neural network model learning system 200 may be connected to an external device (eg, a personal computer or a network) through the communication module 210 or an input/output device (not shown) and exchange data. According to an embodiment, the artificial neural network model learning system 200 is a CPU (Central Processing Unit) or GPU (Graphics Processing Unit), CNN accelerator, NPU (Neural Processing Unit) or VPU (Central Processing Unit) that processes operations related to the artificial neural network at high speed Vision Processing Unit) to control the dedicated processor. The artificial neural network model learning system 200 may employ a variety of hardware or may be employed in a variety of hardware according to design intent, and is not limited to embodiments of the illustrated components. When the above-described embodiments are applied when generating and learning/updating an artificial neural network model, memory can be saved and processing speed can be increased by reducing the amount of data or calculation required for processing of the artificial neural network, so the above-described embodiments are limited It may be suitable for an environment that uses resources or an embedded terminal. The artificial neural network model learning system 200 according to an embodiment can continuously develop the performance of the artificial neural network model by adjusting the direction of change of parameters in the backpropagation learning process so that errors can be minimized in the forward propagation learning process.

3 is a block diagram illustrating an internal configuration of the processor 230 connected to the learning data DB 310 according to an embodiment of the present disclosure. As shown, the processor 230 is, for example, connected to the training data DB 310 through a communication module, and the data input unit 320, the first artificial neural network model learning unit 330, the second artificial neural network model. It may include a learning unit 340 and a third artificial neural network model generation unit 350 . Here, the learning data DB 310 may include learning data used to generate and learn an artificial neural network model. For example, the learning data DB 310 may include images in which objects targeted for classification, recognition, or semantic segmentation are expressed, and images targeted for style transfer. The training data DB 310 may be included in the memory of the artificial neural network model learning system 200 . Alternatively, the training data DB 310 may be stored in an external system, an external device, or an external memory, and may be connected to the artificial neural network model learning system 200 through a communication module.

The data input unit 320 may receive training data from the training data DB 310 . Optionally or additionally, the data input unit 320 may perform pre-processing such as data type conversion, dimension reduction, normalization, etc. on the received training data. The training data received by the data input unit 320 may be used to generate and learn an artificial neural network model by the first artificial neural network model learning unit 330 and/or the second artificial neural network model learning unit 340 .

In an embodiment, the first artificial neural network model learning unit 330 and the second artificial neural network model learning unit 340 use the error backpropagation learning method to obtain parameters (eg, weights) of the artificial neural network model. You can learn/update. When the error backpropagation learning method is used, the first artificial neural network model learning unit 330 and the second artificial neural network model learning unit 340 are configured to determine optimal parameters for minimizing the output error using the momentum-based gradient descent method. is available. For example, the first artificial neural network model learning unit 330 and the second artificial neural network model learning unit 340 may learn/update the weights of the artificial neural network model based on Equation 1 below.

각각은 t번째 에포크에서의 가중치 값 및 t+1번째 에포크에서의 가중치 값을 나타내고,

는 그래디언트 값을 나타내고,

는 모멘텀 값을 나타내고,

는 모멘텀 값의 영향력을 조절하기 위한 상수를 나타낸다. 여기서, 모멘텀 값은 이전 단계(또는 이전 에포크)에서 계산된 그래디언트 값의 흔적(trace)을 누적한 것을 의미할 수 있으며,

는 인공신경망 모델 학습의 학습률을 제어하기 위한 상수를 나타낸다.in the above formula

each represents a weight value at the t-th epoch and a weight value at the t+1-th epoch,

represents the gradient value,

represents the momentum value,

denotes a constant for adjusting the influence of the momentum value. Here, the momentum value may mean the accumulation of traces of the gradient values calculated in the previous step (or the previous epoch),

denotes a constant for controlling the learning rate of artificial neural network model learning.

The first artificial neural network model learning unit 330 may learn/update parameters of the first artificial neural network model. Here, the parameter value of the first artificial neural network model may be expressed as a first data type. For example, the first artificial neural network model learning unit 330 may learn/update the FP model expressed by 32-bit real parameters.

The second artificial neural network model learning unit 340 may learn/update parameters of the second artificial neural network model. In an embodiment, the second artificial neural network model learning unit 340 performs a forward propagation learning process of the second artificial neural network model using the parameter value expressed in the second data type, and the parameter expressed in the first data type By performing a backpropagation learning process of the two artificial neural network models using the values, it is possible to learn/update parameters of the second artificial neural network model. Here, the second data type may be a data type expressed by a smaller number of bits than the first data type (eg, an 8-bit integer). For example, the second artificial neural network model learning unit 340 may update parameters of the second artificial neural network model through a quantization cognitive learning (QAT) method, and the second artificial neural network model may be a QAT model. In this case, the second artificial neural network model learning unit 340 simulates quantized inference in an 8-bit integer type during the forward propagation learning process of the second artificial neural network model, and STE (straight propagation) during the back propagation learning process through estimator) to compute the gradient value of the weight as a 32-bit real number and update the weight value. The QAT method used by the second artificial neural network model learning unit 340 may include any one of known QAT methods or may be a modified example of the method (Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko, "Quantization and training of neural networks for efcient integer-arithmetic-only inference" 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2704-2713 (see IEEE, 2018.).

According to the QAT method, due to quantization of some values of parameters, an error due to estimation may occur in calculating the gradient value. Errors may occur in updating. Since this error again causes an error in the gradient value calculated in the next epoch, the more repeating the learning of the artificial neural network model according to the QAT method, the more the error of the artificial neural network model can be amplified. The second artificial neural network model learning unit 340 may set an appropriate initial momentum value for the second artificial neural network model in order to prevent such error amplification. By setting an appropriate initial momentum value of the second artificial neural network model, parameters may be induced to be updated in an appropriate learning direction, and erroneous gradient values may be ignored. Accordingly, the quantization probability value of the weight determined according to the QAT method can be updated in the correct direction, and the error of the gradient value can be reduced.

값을 결정하고, 모멘텀

을 제2 인공신경망 모델 학습부(340)에 제공할 수 있다. 그 후, 제2 인공신경망 모델 학습부(340)는 제공받은 모멘텀 값을 제2 인공신경망 모델의 초기 모멘텀 값으로 설정하고, 학습 데이터를 이용하여 제2 인공신경망 모델의 파라미터들을 업데이트할 수 있다. 따라서, 제2 인공신경망 모델 학습부(340)는 제1 인공신경망 모델 학습부(330)로부터 FP 모델의 적절한 초기 모멘텀 값을 제공받아, QAT 모델의 학습 초기 단계의 불안정성을 제어함으로써 QAT 방법에 따라 QAT 모델의 파라미터들을 학습할 때 발생하는 오류를 감소시킬 수 있다.In one embodiment, in order for the second artificial neural network model learning unit 340 to set an appropriate initial momentum value for the second artificial neural network model, first, the first artificial neural network model is learned up to a preset epoch, and the preset epoch It is possible to determine the momentum value of the first artificial neural network model in . Also, the second artificial neural network model learning unit 340 may receive the momentum value of the first artificial neural network model determined as described above. For example, the first artificial neural network model learning unit 330 learns an FP model including 32-bit real parameters until the first epoch to obtain momentum.

Determine the value, momentum

may be provided to the second artificial neural network model learning unit 340 . Thereafter, the second artificial neural network model learning unit 340 may set the received momentum value as an initial momentum value of the second artificial neural network model, and update parameters of the second artificial neural network model using the training data. Accordingly, the second artificial neural network model learning unit 340 receives an appropriate initial momentum value of the FP model from the first artificial neural network model learning unit 330, and controls the instability of the initial stage of learning the QAT model according to the QAT method. It is possible to reduce errors that occur when learning the parameters of the QAT model.

In an embodiment, the second artificial neural network model learning unit 340 amplifies a gradient value for at least one weight among a plurality of weights of the second artificial neural network model, and based on the amplified gradient value, the second artificial neural network model parameters can be updated. For example, the second artificial neural network model learning unit 340 amplifies the gradient value in a direction corresponding to the current learning direction based on the sign of the gradient value for at least one weight among the plurality of weights, and the amplified gradient value Based on , parameters of the second artificial neural network model may be updated.

)에서 임의로 일부 가중치(

)를 추출하여, 추출한 일부 가중치에 대해서 그래디언트 값을 증폭하고, 나머지 가중치에 대해서는 증폭하지 않을 수 있다. 일 실시예에서, 제2 인공신경망 모델 학습부(340)는 아래 수학식 2 내지 4를 기초로 그래디언트 값을 증폭할 수 있다.The second artificial neural network model learning unit 340 includes a plurality of weights (

) at arbitrarily some weights (

), to amplify the gradient value for some of the extracted weights, and not to amplify the remaining weights. In an embodiment, the second artificial neural network model learning unit 340 may amplify the gradient value based on Equations 2 to 4 below.

는 복수의 가중치의 그래디언트 값들의 확률 분포를 기초로 결정되는 왜곡 값(distortion)을 나타내고,

은

의 부호(+ 혹은 -)를 나타내고,

는 급격한 그래디언트 증폭을 방지하기 위한 클램핑 계수(clamping factor)를 나타내고,

는 지수적 감쇄(exponential decay)하는 에포크의 지수(power of t)를 나타낼 수 있다. 여기서, 왜곡 값은 복수의 가중치의 그래디언트 값들의 라플라스(Laplace) 확률 분포를 기초로 결정될 수 있다. 따라서, 위 수학식들에 따르면

의 부호와

의 부호를 일치시키고,

에

를 더함으로써, 그래디언트 값을 현재 학습 방향으로 증폭할 수 있다.here,

represents a distortion value determined based on the probability distribution of gradient values of a plurality of weights,

silver

represents the sign (+ or -) of

represents a clamping factor to prevent abrupt gradient amplification,

may represent the power of t of an epoch with exponential decay. Here, the distortion value may be determined based on a Laplace probability distribution of gradient values of a plurality of weights. Therefore, according to the above equations

sign of and

match the signs of

to

By adding , the gradient value can be amplified in the current learning direction.

The third artificial neural network model generator 350 may generate a third artificial neural network model by quantizing parameters of the second artificial neural network model. For example, the third artificial neural network model may be generated by quantizing the 32-bit real-type parameters of the second artificial neural network model into 8-bit integer-type parameters. Since the third artificial neural network model created in this way is composed of 8-bit integer parameters, the size of the memory in which the quantized model is stored can be reduced and the speed of inference using the model can be increased, so that the environment using limited resources However, it can be applied to embedded terminals. In addition, the third artificial neural network model may have improved performance compared to the 8-bit integer artificial neural network model trained by the existing QAT method.

4 is a flowchart illustrating a method 400 for learning an artificial neural network model according to an embodiment of the present disclosure. As shown, the neural network model learning method 400 performed by at least one processor learns the first neural network model up to the preset epoch by the momentum-based gradient descent method, and the first neural network model at the epoch It may start with the step of determining the momentum value of the artificial neural network model ( S410 ). According to an embodiment, by learning the first neural network model until the first epoch, the momentum value of the first neural network model in the first epoch may be determined. The determined momentum value may be set as an initial momentum value of the second artificial neural network model (S420).

Thereafter, based on the initial momentum value, the parameter value of the second artificial neural network model may be updated using the training data ( S430 ). Here, the parameter may include a plurality of weights and momentum. In an embodiment, the processor performs a forward propagation learning process of the second artificial neural network model using the parameter value expressed in the second data type, and the second artificial neural network model using the parameter value expressed in the first data type. of the backpropagation learning process can be performed. The second data type may be expressed by a smaller number of bits compared to the first data type. For example, the second data type may be an 8-bit fixed-point data type, and the first data type may be a 32-bit floating-point data type.

In an embodiment, the third artificial neural network model may be generated by quantizing the parameter values of the updated second artificial neural network model. In an embodiment, the parameter value of the first artificial neural network model is expressed as a first data type, the parameter value of the second artificial neural network model is expressed as the first data type or the second data type, and the parameter value of the third artificial neural network model is expressed as The parameter value may be expressed as a second data type.

5 is a flowchart illustrating a method ( S430 ) of updating a parameter value of a second artificial neural network model according to an embodiment of the present disclosure. According to an embodiment, when updating the parameter values of the second artificial neural network model using the training data based on the initial momentum value in step S430 , the processor based on the gradient values for the plurality of weights. Therefore, the parameter value of the second artificial neural network model may be updated using the training data. Specifically, the processor may amplify the gradient value in a direction corresponding to the current learning direction based on the sign of the gradient value for at least one weight among the plurality of weights ( S510 ). Also, the processor may update the parameter value of the second artificial neural network model based on the amplified gradient value ( S520 ).

The artificial neural network model learning method described above in FIGS. 4 and 5 may be applied to creating and learning an artificial neural network model that performs processing such as classification, object detection, semantic segmentation, or style transfer.

FIG. 6 is histograms 610, 620, and 630 of weight gradient values in each layer for performance evaluation of an artificial neural network model created and learned according to embodiments of the present disclosure.

In order to evaluate the performance of the neural network model learning method and system according to the embodiments of the present disclosure, the MobileNetV2 model was trained up to the 10th epoch using the CIFAR-10 data set using the artificial neural network learning methods described above. The first histogram 610 represents the weight gradient values in each layer of the artificial neural network model learned by the conventional high-precision (FP) model learning method, and the second histogram 620 shows the momentum value of the FP model as the initial momentum value. It represents the weighted gradient values in each layer of the artificial neural network model learned by the conventional QAT method, which is not used as . In addition, the third histogram 630 uses the momentum value of the FP model determined in the first epoch as the initial momentum value, according to embodiments of the present disclosure, and the artificial neural network model learned by the QAT method for applying the amplification of the gradient value. represents the weight gradient value in each layer of . That is, the first histogram 610 may be a histogram of an artificial neural network model having a target performance targeted by the artificial neural network learning method and system of the present disclosure.

As shown in the second histogram 620 , according to the conventional QAT method, it can be confirmed that a problem in which the gradient value disappears due to a gradient approximation error due to STE occurs. On the other hand, the third histogram 630 shows that the distribution of weight gradient values in each layer of the artificial neural network model learned according to the embodiments of the present disclosure is similar to the distribution of the first histogram 610 . Accordingly, it can be confirmed that the artificial neural network model learned according to the embodiments of the present disclosure has similar performance to the artificial neural network model having the target performance.

7 is a loss function graph (loss landscapes) 710, 720, 730, and 740 for performance evaluation of an artificial neural network model created and trained according to embodiments of the present disclosure.

In order to evaluate the performance of the artificial neural network model learning method and system according to the embodiments of the present disclosure, a MobileNetV2 model was trained using the CIFAR-10 data set using the artificial neural network learning methods described above.

The first graph 710 shows a loss function graph of an artificial neural network model learned by the conventional FP learning method, and the second graph 720 does not use the momentum value of the FP model determined in the first epoch and without amplification of the gradient value. , shows the loss function graph of the artificial neural network model learned by the conventional QAT method. In addition, the third graph 730 shows the loss function graph of the artificial neural network model learned by the QAT method by using the momentum value of the FP model determined in the first epoch as the initial momentum value according to the embodiment of the present disclosure, 4 graph 740, according to embodiments of the present disclosure, using the momentum value of the FP model determined in the first epoch as the initial momentum value, and applying the amplification of the gradient value, the loss of the artificial neural network model learned by the QAT method Represents a function graph. That is, the first graph 710 may be a loss function graph of an artificial neural network model having a target performance targeted by the artificial neural network learning method and system of the present disclosure.

As shown, the second graph 720 appears as a flat surface, while the third graph 730 and the fourth graph 740 show a stable loss function graph similar to the first graph 710 . In particular, the fourth graph 740 includes a wider range of loss terrain, and through this, when learning the artificial neural network according to the QAT method applying the amplification of the gradient value, the optimal It can be seen that the search area of the local-minima can be expanded.

8 is an example of a performance evaluation result when an artificial neural network model created and learned according to embodiments of the present disclosure is applied to an image style transfer application field. The artificial neural network model learning method according to various embodiments of the present disclosure may be applied to create and learn an artificial neural network model that performs processing such as classification, object detection, semantic segmentation, or style transfer. 8 illustrates an input image of an artificial neural network model for style transfer learned according to embodiments of the present disclosure and an output image corresponding thereto.

By learning the Pix2Pix style previous model to which the minimax generation loss is applied, the robustness of the artificial neural network model training method according to the embodiments of the present disclosure for unstable learning loss can be evaluated. For layer fusion compatibility, the ResNet-based Pix2Pix model and the Adam optimizer to which the artificial neural network model learning method according to embodiments of the present disclosure is applied were used. Since discriminators are not used during the reasoning process using artificial neural networks, quantization is applied only to the generator of the model. It can be confirmed that the artificial neural network model learning method according to the embodiment of the present disclosure does not cause mode collapse, which is considered a failure signal of the mini-max-based generative model, and is suitable for a fuzzy learning condition. Accordingly, as shown, the artificial neural network model learning method according to an embodiment of the present disclosure may learn a Pix2Pix model capable of performing various image-image style transfers.

Meanwhile, Table 1 below shows the classification results (top 1 accuracy) for the ImageNet-1K data set of the classification artificial neural network model learned according to embodiments of the present disclosure.

Model Params MAdds FP training QAT
fine-tune StatAssist
Only StatAssist
GradBoost ResNet18 11.68M 7.25B 69.7 68.8 68.9 69.6 MobileNetV2 3.51M 1.19B 71.8 70.3 70.7 71.5 ShuffleNetV2 2.28M 0.57B 69.3 63.4 67.7 68.8

0.5ShuffleNetV2

0.5 1.36M 0.16B 58.2 44.8 56.8 57.3

In the table above, 'Params' indicates the number of parameters for each model, 'MAAdds' indicates the Multiply-Adds measured for 224 Х 224 inputs, and 'QAT Fine-tune' is the conventional It represents a quantized model trained or fine-tuned by the QAT method of Meanwhile, 'StatAssist Only' indicates a model learned by the QAT method using the momentum value of the FP model determined in the first epoch according to the embodiments of the present disclosure as the initial momentum value, and 'StatAssist GradBoost' is the According to embodiments, a model trained by the QAT method is shown using the momentum value of the FP model determined in the first epoch as the initial momentum value and applying the amplification of the gradient value.

The performance difference between 'QAT Fine-tune' and 'FP training' is different depending on the model structure. In particular, it can be seen that the performance difference is larger between the ShuffleNetV2 model and the ShuffleNetV2Х0.5 model. On the other hand, it can be seen that the performance difference between 'StatAssist GradBoost' and 'FP training' is 0.9% or less. Accordingly, the classification artificial neural network model learned by the artificial neural network model learning method according to the embodiments of the present disclosure has similar performance to the FP-trained model.

Table 2 below is an object detection result (mAP) for the PASCAL-VOC 2007 data set of the object detection artificial neural network model learned according to embodiments of the present disclosure.

Model Params MAdds FP training QAT
fine-tune StatAssist
Only StatAssist
GradBoost T-DSOD 2.17M 2.24B 71.5 71.4 71.9 72.0 SSD-mv2 2.95M 1.60B 71.0 70.8 71.1 71.3

)을

로 설정하고 총 180K 반복 중 120K 및 150K 반복에서 학습률을 0.1로 조정했다. SSD-mv2 모델의 경우, 초기 학습률은

로, 총 120K 반복을 사용하고, 80K 및 100K에서 학습률을 조정했다. 각 경우에 대해 배치 크기를 64로 설정하고, 성능 평가를 위해 각 모델의 모든 레이어를 융합하도록 감지기를 약간 수정했다.'MAdds' represents the Multiply-Adds values measured for 300 Х 300 inputs. For the T-DSOD model, the initial learning rate (

)second

and adjusted the learning rate to 0.1 at 120K and 150K iterations out of a total of 180K iterations. For the SSD-mv2 model, the initial learning rate is

, using a total of 120K iterations, and adjusting the learning rate at 80K and 100K. For each case, we set the batch size to 64, and slightly modified the detector to fuse all layers of each model for performance evaluation.

As shown in Table 2, compared to 'FP Training', 'QAT Fine-tune' showed a slight decrease in mAP, while 'StatAssist GradBoost' showed an increase in mAP. That is, the performance of 'QAT Fine-tune' cannot surpass that of 'FP training', but it can be confirmed that the performance of 'StatAssist GradBoost' is superior to that of 'FP training'.

Table 3 below shows the semantic segmentation results (mIOU) for the Cityscapes data set of the semantic segmentation artificial neural network model learned according to embodiments of the present disclosure.

Model Params MAdds FP training QAT
fine-tune StatAssist
Only StatAssist
GradBoost ESPNet 0.60M 16.8B 65.4 64.6 65.0 65.5 ESPNetV2 3.43M 27.2B 64.4 63.8 64.6 64.5 Mv3-LRASPP-Large 2.42M 7.21B 65.3 64.5 64.7 65.2 Mv3-LRASPP-Small 0.75M 2.22B 62.5 61.7 61.6 62.1 Mv3-LRASPP-Large-RE (Ours) 2.42M 7.21B 65.5 64.9 65.1 65.8 Mv3-LRASPP-Small-RE (Ours) 0.75M 2.22B 61.5 61.2 62.1 62.3

where 'MAdds' represents the Multiply-Adds measured for 768 Х768 inputs, and '-Large' and '-Small' are different for high and low resource use cases, respectively. It represents the model configuration, and '-RE' represents a model in which hard-swish activation is replaced with ReLU.

로 설정하고 폴리(poly) 학습률 일정과 함께 Nesterov-momentum SGD를 사용했다. 단일 NVIDIA P40 GPU에서, 모델에 적합하도록 768 Х 768로 무작위로 자른 기차 이미지를 이용하여 모델을 학습했다. 성능 평가는 풀 스케일 2048 Х 1024 이미지로 수행하였다.The initial learning rate for learning

, and Nesterov-momentum SGD with a poly learning rate schedule was used. On a single NVIDIA P40 GPU, the model was trained using train images randomly cropped to 768 Х 768 to fit the model. Performance evaluation was performed with full-scale 2048 Х 1024 images.

As shown in Table 3, compared to 'FP training', 'QAT Fine-tune' shows an average 0.65% mIOU decrease, but 'StatAssist GradBoost' shows an average 0.13% mIOU increase. Therefore, it can be confirmed that the performance of 'StatAssist GradBoost' is similar to that of 'FP training' or superior to that of 'FP training'.

9 is a flowchart of an algorithm 900 for performing an artificial neural network model learning method according to an embodiment of the present disclosure. The algorithm for executing the artificial neural network model learning method shown in FIG. 9 may be implemented using, for example, the PyTorch 1.4 quantization library (Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala, "Pytorch: An imperative style, high-performance deep learning library," In Advances in Neural Information Processing Systems 32, pages 8024-8035. See Curran Associates, Inc., 2019.).

The algorithm for performing the artificial neural network model learning method according to the present embodiment may begin with the step of preparing an FP model with the most quantization compatibility ( S910 ). For example, in the program code for learning the artificial neural network model, a pre-stored FP model may be called or declared. Thereafter, a learning workflow of the FP model may be generated ( S920 ). Also, the version of the optimizer that optimizes the parameters of the artificial neural network may be changed to a version that amplifies the gradient value when the weight value of the artificial neural network model is updated ( S930 ). Thereafter, the first momentum value may be determined by learning the FP model up to the first epoch ( S940 ). For example, the FP model may be trained up to the first epoch according to the learning workflow generated in step S920 .

Thereafter, layer fusion and the most quantized QAT model may be prepared (S950). Layer fusion of the QAT model may mean reducing computational delay by integrating convolution, regularization, and activation into one convolution operation in the reasoning process. For example, the QAT model may be prepared by setting the first momentum value of the FP model determined in step S940 as the initial momentum value.

The prepared QAT model may be trained by the QAT method (S960). At this time, since the version of the optimizer is changed to a version that amplifies the gradient value when the weight value of the artificial neural network model is updated in step S930, the gradient value for at least one weight among the plurality of weights of the QAT model is amplified and the parameter value of the QAT model may be updated based on the amplified gradient value. When the learning of the QAT model is completed, the trained model may be converted into a quantized version of INT8 (S970).

The above-described artificial neural network model learning method may be provided as a computer program stored in a computer-readable recording medium to be executed by a computer. The medium may be to continuously store a computer executable program, or to temporarily store it for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or several hardware combined, it is not limited to a medium directly connected to any computer system, and may exist distributed on a network. Examples of the medium include a hard disk, a magnetic medium such as a floppy disk and a magnetic tape, an optical recording medium such as CD-ROM and DVD, a magneto-optical medium such as a floppy disk, and those configured to store program instructions, including ROM, RAM, flash memory, and the like. In addition, examples of other media may include recording media or storage media managed by an app store that distributes applications, sites that supply or distribute various other software, and servers.

The method, operation, or techniques of this disclosure may be implemented by various means. For example, these techniques may be implemented in hardware, firmware, software, or a combination thereof. Those of ordinary skill in the art will appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design requirements imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementations should not be interpreted as causing a departure from the scope of the present disclosure.

In a hardware implementation, the processing units used to perform the techniques include one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs). ), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, and other electronic units designed to perform the functions described in this disclosure. , a computer, or a combination thereof.

Accordingly, the various illustrative logic blocks, modules, and circuits described in connection with this disclosure are suitable for use in general-purpose processors, DSPs, ASICs, FPGAs or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or the present disclosure. It may be implemented or performed in any combination of those designed to perform the functions described in A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, eg, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other configuration.

In firmware and/or software implementations, the techniques include random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), PROM ( on computer-readable media such as programmable read-only memory), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, compact disc (CD), magnetic or optical data storage devices, and the like. It may be implemented as stored instructions. The instructions may be executable by one or more processors, and may cause the processor(s) to perform certain aspects of the functionality described in this disclosure.

Although the embodiments described above have been described utilizing aspects of the presently disclosed subject matter in one or more standalone computer systems, the present disclosure is not so limited and may be implemented in connection with any computing environment, such as a network or distributed computing environment. . Still further, aspects of the subject matter in this disclosure may be implemented in a plurality of processing chips or devices, and storage may be similarly affected across the plurality of devices. Such devices may include PCs, network servers, and portable devices.

Although the present disclosure has been described in connection with some embodiments herein, various modifications and changes may be made without departing from the scope of the present disclosure that can be understood by those skilled in the art to which the present disclosure pertains. Further, such modifications and variations are intended to fall within the scope of the claims appended hereto.

110: FP model
120: first model learning unit
130: QAT model
140: second model learning unit
150: INT8 model
200: artificial neural network model learning system
210: communication module
220: memory
230: processor

Claims (15)

In the artificial neural network model learning method performed by at least one processor,
learning a first neural network model up to a preset epoch by a momentum-based gradient descent method, and determining a momentum value of the first neural network model at the epoch;
setting the determined momentum value as an initial momentum value of a second artificial neural network model; and
Based on the initial momentum value, updating the parameter value of the second artificial neural network model using training data,
The parameter includes a plurality of weights and momentum, an artificial neural network model learning method. According to claim 1,
The step of learning the first neural network model up to the preset epoch by the momentum-based descending gradient method and determining the momentum value of the first neural network model at the epoch includes:
Learning the first artificial neural network model up to a first epoch, and determining a momentum value of the first artificial neural network model in the first epoch. According to claim 1,
Quantizing the parameter values of the updated second artificial neural network model, further comprising the step of generating a third artificial neural network model. 4. The method of claim 3,
The parameter value of the first artificial neural network model is expressed as a first data type,
The parameter value of the second artificial neural network model is expressed as the first data type or the second data type,
The parameter value of the third artificial neural network model is expressed in the second data type,
The number of bits of the second data type is smaller than the number of bits of the first data type, an artificial neural network model learning method. 5. The method of claim 4,
The step of updating the parameter value of the second artificial neural network model using training data based on the initial momentum value includes:
performing a forward-propagation learning process of the second artificial neural network model using the parameter value expressed in the second data type; and
performing a backward-propagation learning process of the second artificial neural network model using the parameter value expressed in the first data type
Including, artificial neural network model learning method. According to claim 1,
The step of updating the parameter value of the second artificial neural network model using training data based on the initial momentum value includes:
and updating parameter values of the second artificial neural network model using training data based on gradient values for a plurality of weights of the second artificial neural network model. 7. The method of claim 6,
The updating of the parameter values of the second artificial neural network model using training data based on the gradient values of the plurality of weights of the second artificial neural network model includes:
amplifying the gradient value in a direction corresponding to a current learning direction based on the sign of the gradient value for at least one of the plurality of weights; and
updating a parameter value of the second artificial neural network model based on the amplified gradient value
Including, artificial neural network model learning method. A computer program stored in a computer-readable recording medium to execute the artificial neural network model learning method according to any one of claims 1 to 7 on a computer. As an artificial neural network model learning system,
communication module;
Memory; and
At least one processor coupled to the memory and configured to execute at least one computer-readable program included in the memory
including,
the at least one program,
By learning the first neural network model up to a preset epoch by the momentum-based gradient descent method, the momentum value of the first neural network model at the epoch is determined, and the determined momentum value is used as the initial value of the second neural network model. Set to a momentum value, and based on the initial momentum value, including instructions for updating the parameter value of the second artificial neural network model using training data,
The parameter comprises a plurality of weights and momentum, an artificial neural network model learning system. 10. The method of claim 9,
the at least one program,
and instructions for learning the first neural network model until a first epoch, and determining a momentum value of the first neural network model in the first epoch. 10. The method of claim 9,
the at least one program,
The artificial neural network model learning system further comprising instructions for generating a third artificial neural network model by quantizing the parameter values of the updated second artificial neural network model. 12. The method of claim 11,
The parameter value of the first artificial neural network model is expressed as a first data type, the parameter value of the second artificial neural network model is expressed as the first data type or the second data type, and the parameter of the third artificial neural network model A value is represented by the second data type, and the number of bits of the second data type is smaller than the number of bits of the first data type. 13. The method of claim 12,
the at least one program,
The forward propagation learning process of the second artificial neural network model is performed using the parameter value expressed in the second data type, and the back propagation of the second artificial neural network model is performed using the parameter value expressed in the first data type. An artificial neural network model learning system, comprising instructions for performing a learning process. 10. The method of claim 9,
the at least one program,
and instructions for updating parameter values of the second artificial neural network model using training data based on the gradient values for the plurality of weights of the second artificial neural network model. 15. The method of claim 14,
the at least one program,
and instructions for amplifying the gradient value in a direction corresponding to a current learning direction based on the sign of the gradient value for at least one weight among the plurality of weights.

Images