KR20240050628A - Apparatus and Method for Adjusting Quantization Precision for Artificial Neural Network - Google Patents

Abstract

The disclosed embodiment uses an artificial neural network to receive the number of bits of weights included in a plurality of layers of a neural network model that is a learning target, performs a neural network operation to estimate the number of quantization bits, and quantizes the weights of the neural network model according to the estimated number of quantization bits. , using an approximation function that approximates the modified quantization function transformed by the sum of the weight and the periodic function to a continuous function, the loss calculated during training of the neural network model is back-propagated to the artificial neural network, and the quantization bit is used when learning the weight of the neural network. We provide a quantization precision control device and method that can improve the performance of a neural network implemented on limited hardware resources by learning the number as a learnable parameter and obtaining the optimized number of quantization bits for each layer.

Description

Apparatus and Method for Adjusting Quantization Precision for Artificial Neural Network}

The disclosed embodiments relate to a method of controlling quantization precision for an artificial neural network, and to a method of controlling quantization precision for an artificial neural network that can optimize the number of quantization bits for each layer.

Recently, artificial neural networks have been actively applied in various fields such as object recognition, automatic conversation recognition, natural language processing, and music/acoustic signal processing.

Each of the multiple layers that make up the artificial neural network performs operations such as convolution with weights learned and set according to deep learning techniques on the input data and applies activation functions, thereby generating the required output data. can be created. And the output data of each layer can be transferred to other layers.

In this artificial neural network, the weights for each layer are quantized, and the higher the quantization level of the weights, the better the performance of the artificial neural network can be. However, artificial neural networks must be designed considering limited hardware resources. In addition, in order to reduce computational complexity and improve operation speed, the quantization level of the weight, that is, the number of bits of the weight, needs to be adjusted. In existing artificial neural networks, it is common for the weights of multiple layers to be quantized to have the same number of bits. However, while some layers among multiple layers in an artificial neural network greatly change the performance of the artificial neural network depending on the quantization level, some layers may have little effect on the performance of the artificial neural network even if the quantization level is variable. In an actual artificial neural network, the quantization sensitivity of each layer is different. Therefore, it is very inefficient to unify the quantization level so that the weights for all layers of an artificial neural network have the same number of bits.

Accordingly, recently, research has been actively conducted on the mixed-precision quantization method, which quantizes by independently setting the number of bits of the weight for each layer. Using a mixed-precision quantization method takes into account the quantization sensitivity of each layer and adjusts the quantization level of the weight for each layer, allowing the artificial neural network to operate more efficiently under limited hardware resources. However, it is very difficult to find the appropriate quantization level for each layer under limited hardware resources. In particular, when the memory capacity is small, the overall quantization level has no choice but to be set low. In this case, optimizing the quantization level for each layer also affects the adjustment of the quantization level of other layers, leading to a problem that the performance efficiency of the artificial neural network is greatly reduced. there is.

Korean Patent Publication No. 10-2018-0082344 (published on July 18, 2018)

The purpose of the disclosed embodiments is to provide a quantization precision control device and method that can improve the performance of a neural network by adjusting the number of quantization bits of the weight for each layer in limited hardware resources.

The purpose of the disclosed embodiments is to provide an apparatus and method for controlling quantization precision that can obtain an optimized number of quantization bits for each layer by learning the number of quantization bits as a learnable parameter when learning weights of a neural network.

A quantization precision control device according to an embodiment uses an artificial neural network to receive the number of bits of weights included in a plurality of layers of a neural network model that is a learning target, performs a neural network operation to estimate the number of quantization bits, and uses the neural network according to the estimated number of quantization bits. The weights of the model are quantized, and the loss calculated during training of the neural network model is back-propagated to the artificial neural network using an approximation function that approximates the transformed quantization function transformed by the sum of the weight and the periodic function into a continuous function. .

The approximate function may be obtained by replacing the periodic function of the modified quantization function with an approximate periodic function in the form of a sine function having the same period.

The modified quantization function may be calculated as the sum of a periodic function with a period and scale according to the number of quantization bits and the weight that is the input value of the quantization function.

The quantization precision control device changes the weight of the neural network model by quantizing the weight of the neural network model according to the number of quantization bits estimated when learning the neural network model, and applies learning data to the neural network model with the changed weight to output The loss can be calculated from the output data.

The quantization precision control device may backpropagate the total loss calculated as the sum of the neural network loss for learning the neural network model and the memory capacity limitation loss according to the number of quantization bits to the artificial neural network using the approximation function.

The quantization precision control device may backpropagate the neural network loss to the neural network model.

A method of controlling quantization precision according to an embodiment includes the steps of receiving the number of bits of weights included in a plurality of layers of a neural network model that is a learning target using an artificial neural network and performing a neural network operation to estimate the number of quantization bits; Quantizing weights of the neural network model according to the estimated number of quantization bits; And back-propagating the loss calculated during training of the neural network model to the artificial neural network using an approximation function that approximates the modified quantization function transformed by the sum of the weight and the periodic function to a continuous function.

Therefore, the apparatus and method for controlling quantization precision for an artificial neural network according to an embodiment learns the number of quantization bits as a learnable parameter when learning the weights of a neural network, and obtains an optimized number of quantization bits for each layer, thereby reducing hardware resources. The performance of neural networks implemented in can be improved.

Figure 1 is a graph showing the quantization of a quantization function according to the number of bits.
Figure 2 shows a schematic configuration of a quantization precision adjustment device according to an embodiment.
FIG. 3 is a diagram for explaining the operation of the modified quantization module of FIG. 2.
FIG. 4 is a diagram for explaining the operation of the approximate backpropagation module of FIG. 2.
Figure 5 shows a method for controlling quantization precision according to an embodiment.
FIG. 6 is a diagram for explaining a computing environment including a computing device according to an embodiment.

Hereinafter, specific embodiments of one embodiment will be described with reference to the drawings. The detailed description below is provided to provide a comprehensive understanding of the methods, devices and/or systems described herein. However, this is only an example and the present invention is not limited thereto.

In describing one embodiment, if it is determined that a detailed description of the known technology related to the present invention may unnecessarily obscure the gist of an embodiment, the detailed description will be omitted. In addition, the terms described below are terms defined in consideration of functions in the present invention, and may vary depending on the intention or custom of the user or operator. Therefore, the definition should be made based on the contents throughout this specification. The terminology used in the detailed description is intended to describe only one embodiment and should in no way be limiting. Unless explicitly stated otherwise, singular forms include plural meanings. In this description, expressions such as “comprising” or “including” are intended to indicate certain features, numbers, steps, operations, elements, parts or combinations thereof, and one or more than those described. It should not be construed to exclude the existence or possibility of any other characteristic, number, step, operation, element, or part or combination thereof. In addition, terms such as "... unit", "... unit", "module", and "block" used in the specification refer to a unit that processes at least one function or operation, which is hardware, software, or hardware. and software.

As mentioned above, a mixed-precision quantization method that adjusts the quantization precision for each layer has been used, and existing mixed-precision quantization methods include a rule-based approach and a NAS (Neural Architecture Search)-based method. The approach is representative. The rule-based approach measures the index value by constructing a separate function that measures the quantization sensitivity for each layer, and determines the number of quantization bits for each layer based on the measured index value. However, in the case of the rule-based approach, not only is optimization difficult because the index value is not a value for the actual performance of the neural network, but also the number of bits is determined based on the index value measured before quantization, so it is an index for the weight after quantization. The value may change. Additionally, there is a limitation that the process of determining the number of bits and the learning process of the neural network are independent of each other.

Meanwhile, in the NAS-based approach, the previously known NAS technique is used to control quantization precision, and the loss function according to the number of quantization bits for each layer in all possible combinations is analyzed to determine the optimal number of quantization bits. However, in the NAS approach-based method, the loss must be calculated by varying the number of quantization bits for each layer in all possible combinations, so a huge amount of search time is required to search for the optimal number of quantization bits. In addition, there is a limitation that the optimal number of bits may change depending on the data used in the actual neural network learning process.

As mentioned above, the existing quantization precision control method basically separates the process of searching for the number of quantization bits and the neural network learning process and is performed separately. And this is because it is not easy to obtain the optimal number of quantization bits through learning during the neural network learning process, depending on the characteristics of the quantization function.

Figure 1 is a graph showing the quantization of a quantization function according to the number of bits.

In Figure 1 (a) to (d), when the number of quantization bits (b) is 1 to 4 (b = 1 to 4), the quantization function (Q(x _n ;b)) represents the input value (x). The quantized result is shown, and the case where the lower bound (l) and upper bound (u) are 0 and 1, respectively (l = 0, u = 1) is shown.

Looking at the process by which the quantization function (Q(x _n ;b)) quantizes the input value (x) to the number of bits (b), first of all, the quantization function (Q(x _n ;b)) has an upper limit (u) and a lower limit ( Based on l), the input value (x) is normalized to a value within the specified range. Here, the normalized value (x _n ) is (where clip(·) is obtained as a clip function). Then, the quantization weight (2 ^b -1) according to the number of bits (b) is weighted (x _b = (2 ^b -1)x _n ) to the normalization value (x _n ), and the quantization weight (2 ^b -1) is The weighted normalized value (x _b ) is rounded (x _q = round(x _b )) with the rounding function (round(·)). And the scale is adjusted by dividing the rounded value (x _q ) by the quantization weight (2 ^b -1) ( ), the quantization value for the input value (x) ( ) to obtain.

This quantization function (Q(x _n ;b)) is a quantization value ( ), the quantization weight (2 ^b -1) to which the number of quantization bits (b) is applied is weighted to the normalization value (x _n ) and is then divided and erased again in the scale adjustment process. Therefore, the number of quantization bits (b) is the quantization function (Q(x _n ;b)) that converts the input value (x) to the quantization value ( ), it is only used internally and is not expressed externally. Because of this, the number of quantization bits (b) of the quantization function (Q(x _n ;b)) cannot be learned together during training of the artificial neural network. In other words, there is a problem that learning about the number of quantization bits (b) itself is impossible.

FIG. 2 shows a schematic configuration of a quantization precision adjustment device according to an embodiment, FIG. 3 is a diagram for explaining the operation of the modified quantization module of FIG. 2, and FIG. 4 is a diagram for explaining the operation of the approximate backpropagation module of FIG. 2. This is a drawing for

FIG. 2 shows a neural network learning system including a quantization precision adjustment device of an embodiment. As shown in FIG. 2, the neural network learning system may include a neural network model 11, a quantization precision adjustment device 12, and a learning module 16.

The neural network model 11 is an artificial neural network model to be learned, and may be composed of multiple layers. Here, the structure and purpose of the neural network model 11, and the number of layers included in the neural network model 11 and the size of each layer are not separately limited. Additionally, the number of weight bits in each layer may be different.

The quantization precision control device 12 is a device that adjusts the number of bits for the weight of each of the multiple layers of the neural network model 11. It estimates the optimal number of quantization bits (b) for the weight of each layer, and calculates the estimated number of bits. Based on the number of quantization bits (b), the weight for each layer is quantized and transmitted to the neural network model (11). The quantization precision adjustment device 12 can estimate the optimal number of quantization bits (b) in the process of updating the weight of each layer through learning, so it does not require a separate process to optimize the number of quantization bits (b). .

However, as mentioned above, considering the case of quantizing the weights using a general quantization function (Q(x _n ;b)), the number of quantization bits (b) within the quantization function (Q(x _n ;b)) The related item, quantization weight (2 ^b -1), is erased, so the learning itself cannot be performed. In order to overcome this limitation, the quantization precision control device 12 performs quantization using a modified quantization function that modifies the quantization function (Q(x _n ;b)), so that the number of quantization bits (b) is adjusted for artificial neural network learning. Make it a parameter that can be learned together.

In addition, the quantization function (Q(x _n ;b)) is basically a discontinuous, discrete function and a non-differentiable function, as shown in (a) to (d) of FIG. 1, and this also applies to the modified quantization function described later. . And in the case of non-differentiable functions, the loss cannot be back-propagated, so learning is not performed normally. In order to solve this problem, the quantization precision adjustment device 12 of the embodiment additionally utilizes an approximation function that approximates the modified quantization function to enable loss backpropagation, so that the number of quantization bits (b) of the neural network model 11 During training, it can be updated along with the weights through learning.

The quantization precision adjustment device 12 may include a quantization bit number estimation module 13, a quantization module 14, and an approximate backpropagation module 15.

The quantization bit number estimation module 13 receives the number of bits for the weights for each layer included in the neural network model 11 as input and performs a neural network operation to optimize the number of quantization bits for the weights for each layer. Estimate (b). The quantization bit number estimation module 13 is implemented as a learnable artificial neural network, and learning is performed simultaneously with the training of the neural network model 11, so that the optimal number of quantization bits (b) can be estimated.

The quantization module 14 receives the weight (x) for each layer as an input value, and the weight for each layer applied based on the number of quantization bits (b) estimated by the quantization bit number estimation module 13 is the number of quantization bits ( Weight with the number of bits according to b) ( ) is quantized. At this time, the quantization module 14 uses a modified quantization function in which the quantization function (Q(x _n ;b)) is modified as shown in Equation 1 so that the quantization bit number estimation module 13 can be learned together with the neural network model 11. The weight (x) can be quantized using .

Here, g(x;b) is a periodic function used to transform the quantization function.

Hereinafter, the periodic function (g(x;b)) of Equation 1 will be described with reference to FIG. 3. In Figure 3, (a) shows the operation of the quantization function (Q(x;2)) when the number of bits (b) is 2 (b = 2), and (c) shows the operation of the quantization function (Q(x;2)) when the number of bits (b) is 3. In the case (b = 3), (b) and (d) represent the operation of the quantization function (Q(x;3)), and (b) and (d) are the periodic functions ( It represents the operation of g(x;b)).

Referring to (a) and (c) of FIG. 3, looking again at the operation of the quantization function (Q(x _n ;b)), the quantization function (Q(x _n ;b)) is substantially the input value, the weight ( Period according to the number of quantization bits (b) for x) ( ) can be seen as quantizing the weight (x) by repeating every time and subtracting a value that decreases linearly in proportion to the weight (x). At this time, the cycle can be viewed as being repeated by the rounding function (round(·)), and the standard value for dividing the cycle is the cycle ( ) to 1/2 the value of It can be seen as

According to Equation 1, the periodic function (g(x;b)) is calculated by subtracting the weight (x) from the quantization function (Q(x _n ;b)) (Q(x _n ;b) - x) As shown in (b) and (d) of FIG. 3, it has a function form that repeats periodically and linearly decreases, and can be defined as Equation 2.

In Equation 2, the weight (x) is only the first cycle section. It only shows cases included in the range of , and as shown in Figures 3 (b) and (d), the periodic function (g(x;b)) is the upper limit (u) and lower limit (l) at which the quantization function performs quantization. It can be applied within the scope of.

And as mentioned above, the periodic function (g(x;b)) has a period ( ), so the first period section ( ), other cycle sections except It can be calculated as And the periodic function (g(x;b)), depending on the number of quantization bits (b), has a maximum value of the period ( ) is a function that is scaled by the ratio.

Therefore, according to Equations 1 and 2, the quantization function (Q(x _n ;b)) can be transformed into the sum (x + g(x;b)) of the weight (x) and the periodic function (g(x;b)). Even though the weight (x) is the same quantization value ( ) is quantized. Accordingly, the quantization module 14 uses the modified quantization function (x + g(x;b)) to set the weight (x) of each layer of the neural network model 11 to the quantization weight ( ) is quantized and transmitted.

However, here, the quantization module 13 quantizes the weight (x) using the modified quantization function (x + g(x;b)), and then the approximate backpropagation module 15 obtains it from the learning module 16. The purpose is to apply an approximate quantization function to the loss so that it can be back-propagated to each layer of the neural network model (11) and the quantization bit number estimation module (13). As shown in Equation 1, the quantization function (Q( quantized _weight ( ) and the quantization weight (quantized by the modified quantization function (x + g(x;b))) ) has the same value. Accordingly, the quantization module 13 may perform quantization using not only the modified quantization function (x + g(x;b)) but also the existing quantization function (Q(x _n ;b)).

The neural network model (11) uses the quantization weight ( ) is approved and the weight (x) of each of the multiple layers is replaced and set, and the set quantization weight ( ), performs a neural network operation on the input data and outputs the output data.

The learning module 16 receives the output data of the neural network model 11, calculates the loss for the applied output data based on a designated loss function, and backpropagates it to the neural network model 11, thereby creating the neural network model 11. Let them learn. That is, the weight of each layer of the neural network model (11) ( ) is updated.

However, since the quantization function (Q(x _n ;b)) and the modified quantization function (x + g(x;b)) are both discontinuous and non-differentiable functions, the loss obtained in the learning module 16 is normally quantized. It cannot be back-propagated to the bit number estimation module 13.

Accordingly, the quantization precision control device 12 is provided with an approximate back-propagation module 15, and the approximate back-propagation module 15 approximates the modified quantization function (x + g(x;b)) with a differentiable approximation function to reduce the loss. Allow this to be back-propagated. In particular, the approximate backpropagation module 15 approximates the periodic function (g(x;b)) with discontinuity in the modified quantization function (x + g(x;b)) as a sine function with continuity as shown in Equation 3. Construct the approximate function (x + h(x;b)) using the approximate periodic function (g(x;b) ≒ h(x;b)), and construct the approximate function (x + h(x;b)) Backpropagate the loss using .

Here, the approximate periodic function (h(x;b)) is similar to the periodic function (g(x;b)). It has a period of and depending on the number of quantization bits (b), the maximum value is the period ( ) is a function that is scaled by the ratio.

That is, the approximate function (x + h(x;b)) is an approximate periodic function in the form of a sine function with the same period as the periodic function (g(x;b)) in the modified quantization function (x + g(x;b)). It can be obtained by substituting (g(x;b) ≒ h(x;b)).

In Figure 4 (a) and (c), like (b) and (d) in Figure 3, the periodic function (g(x) when the number of quantization bits (b) is 2 and 3 (b = 2, 3) ;b)) and its approximate periodic function (h(x;b)), and (b) and (d) of Figures 4 represent the periodic function (g(x;b)) of (a) and (c) It shows the result of applying a weight (x) to the approximate periodic function (h(x;b)).

As shown in Figure 4, if the approximate periodic function (h(x;b)) is applied instead of the periodic function (g(x;b)) in the modified quantization function, a certain level of error may occur, but the differentiation Since it is converted into a possible functional form, lossy backpropagation is possible using the differential value of the approximate periodic function (h(x;b)).

However, since the quantization bit number estimation module 13 is also an artificial neural network, it must be learned along with the learning of the neural network model 11. To this end, the learning module 16 additionally applies a loss for training the quantization bit number estimation module 13. Must be acquired.

The learning module 16 may first obtain the neural network loss (L _gt ) obtained to train the neural network model 11. The neural network loss (L _gt ) can be calculated using a loss function preset according to the neural network model 11, and the neural network loss (L _gt ) is also used for learning the quantization bit number estimation module 13. However, the learning module 16 additionally calculates the memory capacity limitation loss for the quantization bit number estimation module 13.

In many cases, the neural network model 11 is not implemented as dedicated hardware, but as software that runs on a general-purpose device that includes a general-purpose processor and memory. Therefore, most neural network models 11 can be viewed as consisting of program code and weights that are actually stored in memory and executed by a processor. Therefore, the number of bits of the weight, that is, the number of quantization bits (b), must be limited by memory capacity. Therefore, the learning module 16 can further calculate the memory capacity limitation loss by considering the memory capacity of the device to which the neural network model 11 is applied.

The learning module 16 can calculate the weight memory capacity limit loss (L _w ) according to the number of quantization bits (b) of the weight according to Equation 4.

Here, p _i is the weight, i is the weight identifier, b _i ^w is the number of bits of the weight (p _i ), and b _t ^w is the weight target number of bits considering memory capacity. And the weight target number of bits (b _t ^w ) can be set as the average value of the number of weight bits that can be set according to the memory capacity that can be utilized to store the weight of the neural network.

Accordingly, the learning module 16 calculates the loss for training the quantization bit number estimation module 13 according to Equation 5, transfers the calculated loss to the approximate back-propagation module 15, and back-propagates using the approximation function. can do.

Here, λ _w is the loss weight for the weight memory capacity limit loss (L _w ).

However, as mentioned above, each of the plurality of layers of the neural network model 11 has a plurality of weights and not only performs a designated weighting operation such as a convolution operation on the input data, but also performs a designated activation function. Accordingly, an activation operation is performed to output features, and the output features are stored in memory so that they can be applied as input to other layers. The activation function is generally implemented to perform an operation such as ay + c on the weighted operation output (y), and at this time, the characteristics are determined according to the number of bits of the activation weight (a) weighted for the weighted operation output (y). It can be seen that the number of bits of a character is variable.

Therefore, the learning module 16 not only calculates the weight memory capacity limit loss (L _w ), which represents the memory capacity limit loss for the weights, but also calculates the activation weight (a ), the feature memory capacity limit loss (L _a ) can be additionally calculated.

The feature memory capacity limit loss (L _a ) can be calculated using Equation 6.

Here, a _i is the activation weight, i is the activation weight identifier, b _i ^a is the number of bits of the activation weight (a _i ), and b _t ^a is the activation weight target number of bits considering memory capacity. And the activation weight target number of bits (b _t ^a ), similar to the weight target number of bits (b _t ^w ), can be set to the average value of the number of weight bits that can be set according to the memory capacity that can be utilized to store the activation weight.

Accordingly, the learning module 16 may additionally apply the feature memory capacity limitation loss (L _a ) to calculate and backpropagate the loss for training the quantization bit number estimation module 13 according to Equation 7.

Here, λ _a is the loss weight for the feature memory capacity limitation loss (L _a ).

The learning module 16 may terminate learning and not backpropagate the loss when the total loss calculated according to Equation 6 or 7 becomes less than or equal to the reference loss or when the number of learning repetitions exceeds the reference number.

In FIG. 2, the quantization precision control device 12 and the learning module 16 are components required only when learning the neural network model 11, and the neural network model 11 is not used after learning is completed. That is, only the neural network model 11 that has completed learning is applied to a separately required device. However, in some cases, the quantization bit number estimation module 13 may be included in the neural network model 11. In other words, it may be implemented as a neural network integrated into the neural network model 11 rather than as a separate neural network from the neural network model 11.

As a result, the quantization precision adjustment device 12 according to the embodiment receives the number of bits of the weights of multiple layers of the neural network model 11 as input when learning the neural network model 11, and optimizes the weight for each layer. Quantize the weight by estimating the number of quantization bits. In particular, by estimating the optimal number of quantization bits through learning together when learning the neural network model 11, the neural network model 11 shows the best possible performance even when applied to devices with limited hardware resources such as memory capacity. You can do it.

In the illustrated embodiment, each component may have different functions and capabilities in addition to those described below, and may include additional components other than those described below. Additionally, in one embodiment, each configuration may be implemented using one or more physically separate devices, one or more processors, or a combination of one or more processors and software, and, unlike the example shown, may be implemented in specific operations. It may not be clearly distinguished.

Additionally, the quantization precision adjustment device shown in FIG. 2 may be implemented within a logic circuit using hardware, firmware, software, or a combination thereof, and may also be implemented using a general-purpose or special-purpose computer. The device may be implemented using hardwired devices, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc. Additionally, the device may be implemented as a System on Chip (SoC) including one or more processors and a controller.

In addition, the quantization precision control device may be mounted on a computing device or server equipped with hardware elements in the form of software, hardware, or a combination thereof. A computing device or server includes all or part of a communication device such as a communication modem for communicating with various devices or wired and wireless communication networks, a memory for storing data to execute a program, and a microprocessor for executing a program to perform calculations and commands. It can refer to a variety of devices, including:

Figure 5 shows a method for controlling quantization precision according to an embodiment.

When explaining the quantization precision control method of FIG. 5 with reference to FIGS. 2 to 4, first, the quantization bit number estimation module 13 inputs the currently set number of bits to the weights of multiple layers of the neural network model 11 to be learned. The optimal number of quantization bits (b) for the weight of each layer is estimated by performing a neural network operation (21). Here, the quantization bit number estimation module 13 may be implemented as an artificial neural network separate from the neural network model 11 that is the learning target.

Then, the weights of the neural network model (11) are quantized according to the estimated number of quantization bits (b) (22). At this time, the weight may be quantized according to a general quantization function (Q(x _n ;b)), but the period ( ) can also be quantized according to a modified quantization function (x + g(x;b)) expressed as the sum of a periodic function (g(x;b)) repeated as ).

If the weight of the neural network model (11) is quantized according to the number of quantization bits (b), the quantized weight ( ) is stored as the weight of the neural network model (11) (23).

When the weight of the neural network model 11 is changed to a quantized weight, training data prepared for learning the neural network model 11 is input into the neural network model 11 to start learning (24). When the learning data is the applied neural network model 11, a neural network operation is performed and the output data is output, the learning module 16 calculates the loss (25). The learning module 16 may first calculate the neural network loss (L _gt ) in a preset manner for the learning data. As an example, the neural network loss (L _gt ) can be calculated based on the difference between the output data and the ground truth set by matching the training data. In addition, the learning module 16 further calculates the memory capacity limitation loss for training the quantization bit number estimation module 13, which estimates the quantization bit number b. Here, the memory capacity limitation loss includes the weight memory capacity limitation loss (L _w ) according to the number of quantization bits (b) of the weight, and additionally includes the feature memory capacity limitation loss (L _a ) according to the activation weight (a). You can.

And the learning module 16 determines whether the total loss calculated as the sum of the neural network loss (L _gt ) and the memory capacity limit loss is less than or equal to the specified reference loss (26). Alternatively, it may be determined whether the number of repetitive learning exceeds the standard number. If the total loss is less than the standard loss or the number of learning exceeds the standard number, learning is determined to be completed and the final weight of each layer of the neural network model (11) set according to the currently estimated number of quantization bits (b) is determined. It is determined by the number of weight bits (28).

However, if the total loss exceeds the reference loss or the learning number is less than the reference number, the calculated loss is back-propagated and the neural network model 11 and the quantization bit number estimation module 13 are trained together (27). Here, only the neural network loss (L _gt ) can be back-propagated to the neural network model (11), and the quantization bit number estimation module (13) can be trained by back-propagating the sum of the neural network loss (L _gt ) and memory capacity limitation loss. However, since the quantization function (Q(x _n ;b)) or the modified quantization function (x + g(x;b)) are both discontinuous and non-differentiable, it is difficult to backpropagate the calculated loss. Accordingly, in the embodiment, the periodic function (g(x;b)), which is a discontinuous function in the modified quantization function (x + g(x;b)), is converted into a continuous sine function so that the gradient represented by the differential value of the loss can be back-propagated. Based on the approximate periodic function (g(x;b) ≒ h(x;b)) approximated in the form, the transformed quantization function (x + g(x;b)) is converted to the approximate function (x + h(x;b) )) to enable the loss to be back-propagated.

Then, in order to estimate the number of quantization bits (b) again, the currently estimated number of quantization bits (b), which is set as the number of weight bits, is applied again as an input to the quantization bit number estimation module 13 (21).

In FIG. 5, each process is described as being sequentially executed, but this is only an illustrative explanation, and those skilled in the art can change the order shown in FIG. 5 and execute it without departing from the essential characteristics of the embodiments of the present invention. Alternatively, it can be applied through various modifications and modifications by executing one or more processes in parallel or adding other processes.

FIG. 6 is a diagram for explaining a computing environment including a computing device according to an embodiment.

In the illustrated embodiment, each component may have different functions and capabilities in addition to those described below, and may include additional components in addition to those not described below. The illustrated computing environment 30 may include a computing device 31 to perform the quantization precision adjustment method shown in FIG. 6 . In one embodiment, computing device 31 may be one or more components included in the quantization precision adjustment device shown in FIG. 4 .

Computing device 31 includes at least one processor 32, a computer-readable storage medium 33, and a communication bus 35. Processor 32 may cause computing device 31 to operate according to the above-mentioned example embodiments. For example, the processor 32 may execute one or more programs 34 stored in the computer-readable storage medium 33. The one or more programs 34 may include one or more computer-executable instructions, which, when executed by the processor 32, cause the computing device 31 to perform operations according to an example embodiment. It can be configured to perform these.

Communication bus 35 interconnects various other components of computing device 31, including processor 32 and computer-readable storage media 33.

Computing device 31 may also include one or more input/output interfaces 36 and one or more communication interfaces 37 that provide an interface for one or more input/output devices 38 . The input/output interface 36 and communication interface 37 are connected to the communication bus 35. Input/output device 38 may be coupled to other components of computing device 31 through input/output interface 36. Exemplary input/output devices 38 include, but are not limited to, a pointing device (such as a mouse or trackpad), a keyboard, a touch input device (such as a touchpad or touch screen), a voice or sound input device, various types of sensor devices, and/or imaging devices. It may include input devices and/or output devices such as display devices, printers, speakers, and/or network cards. The exemplary input/output device 38 is a component constituting the computing device 31 and may be included within the computing device 31. The exemplary input/output device 38 may be a separate device distinct from the computing device 31 and may be included in the computing device 31. It may be connected.

Although the present invention has been described in detail through representative embodiments above, those skilled in the art will understand that various modifications and other equivalent embodiments are possible therefrom. Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the attached claims.

11: Neural network model 12: Quantization precision adjuster
13: Quantization bit number estimation module 14: Quantization module
15: Approximate backpropagation module 16: Learning module

Claims (20)

One or more processors; and a memory storing one or more programs executed by the one or more processors,
The processor is
Using an artificial neural network, the number of bits of weights included in multiple layers of the neural network model that is the subject of learning is received and the number of quantization bits is estimated by performing neural network calculations,
Quantizing the weights of the neural network model according to the estimated number of quantization bits,
A quantization precision control device that backpropagates the loss calculated during training of the neural network model to the artificial neural network using an approximation function that approximates the modified quantization function transformed by the sum of the weight and the periodic function to a continuous function. The method of claim 1, wherein the approximate function is
A quantization precision control device obtained by replacing the periodic function of the modified quantization function with an approximate periodic function in the form of a sine function having the same period. The method of claim 1, wherein the modified quantization function is
A quantization precision control device that calculates the weight, which is an input value of the quantization function, as the sum of a periodic function with a period and scale according to the number of quantization bits. The method of claim 1, wherein the periodic function is
math equation

(where g(x;b) is a periodic function and weight(x) is the first periodic interval If it is included in the range of, b is calculated as the number of quantization bits,
In other cycle sections It is calculated as,
The period of the periodic function (g(x;b)) is Phosphorus quantization precision control device. The method of claim 1, wherein the approximate function is
In the modified quantization function (x + g(x;b)) (where x is the weight and b is the number of quantization bits), the periodic function (g(x;b)) is expressed as Equation

A quantization precision control device obtained by replacing with an approximate periodic function (h(x;b)) approximated by . The method of claim 1, wherein the processor
Changing the weights of the neural network model by quantizing them according to the number of quantization bits estimated when learning the neural network model,
A quantization precision control device that applies learning data to the neural network model with changed weights and calculates a loss calculated from the output data. The method of claim 1, wherein the processor
A quantization precision control device that backpropagates a total loss calculated as the sum of a neural network loss for training the neural network model and a memory capacity limitation loss according to the number of quantization bits to the artificial neural network using the approximation function. The method of claim 7, wherein the memory capacity limit loss is
Includes weight memory loss for the number of bits of the weight, and the weight memory loss is expressed in the equation

(Here, p _i is the weight, i is the weight identifier, b _i ^w is the number of bits of the weight (p _i ), and b _t ^w is the weight target number of bits considering memory capacity. And the weight target number of bits (b _t ^w ) can be set as the average value of the number of weight bits that can be set depending on the memory capacity available to store the weights of the neural network.)
Quantization precision adjustment device calculated according to . The method of claim 8, wherein the memory capacity limit loss is
It further includes a feature memory capacity limitation loss according to the activation weight (a) of the activation function in each layer of the neural network model, and the memory capacity limitation loss is expressed in Equation

(Here, a _i is the activation weight, i is the activation weight identifier, b _i ^a is the number of bits of the activation weight (a _i ), and b _t ^a is the number of bits of the activation weight target considering memory capacity. And the activation weight target The number of bits (b _t ^a ) can be set to the average ^value of the number _of weight bits that can be set according to the memory capacity that can be utilized to store the activation weight, similar to the weight target number of bits (b t w ).)
Quantization precision adjustment device calculated according to . The method of claim 7, wherein the processor
A quantization precision control device that backpropagates the neural network loss to the neural network model. 1. A method performed by a computing device having one or more processors and a memory that stores one or more programs to be executed by the one or more processors, comprising:
Estimating the number of quantization bits by receiving the number of bits of weights included in a plurality of layers of a neural network model to be learned using an artificial neural network and performing a neural network operation;
Quantizing weights of the neural network model according to the estimated number of quantization bits; and
Quantization precision adjustment comprising the step of backpropagating the loss calculated during training of the neural network model to the artificial neural network using an approximation function that approximates the modified quantization function transformed by the sum of the weight and the periodic function to a continuous function. method. The method of claim 11, wherein the approximation function is
A method of controlling quantization precision obtained by replacing the periodic function of the modified quantization function with an approximate periodic function in the form of a sine function having the same period. The method of claim 11, wherein the modified quantization function is
A method of controlling quantization precision in which the weight, which is the input value of the quantization function, is calculated as the sum of a periodic function with a period and scale according to the number of quantization bits. The method of claim 11, wherein the periodic function is
math equation

(where g(x;b) is a periodic function and weight(x) is the first periodic interval If it is included in the range of, b is calculated as the number of quantization bits,
In other cycle sections It is calculated as,
The period of the periodic function (g(x;b)) is Quantization precision control method. The method of claim 11, wherein the approximation function is
In the modified quantization function (x + g(x;b)) (where x is the weight and b is the number of quantization bits), the periodic function (g(x;b)) is expressed as Equation

Quantization precision control method obtained by substituting with an approximate periodic function (h(x;b)) approximated by . The method of claim 11, wherein the step of backpropagating
Changing the weights of the neural network model by quantizing them according to the number of quantization bits estimated when learning the neural network model,
A quantization precision control method that applies learning data to the neural network model with changed weights, calculates the loss calculated from the output data, and backpropagates it. The method of claim 11, wherein the step of backpropagating
A quantization precision control method in which the total loss calculated as the sum of the neural network loss for training the neural network model and the memory capacity limitation loss according to the number of quantization bits is back-propagated to the artificial neural network using the approximation function. The method of claim 17, wherein the memory capacity limit loss is
It includes a weight memory loss for the number of bits of the weight, and the weight memory loss is expressed by the equation

(Here, p _i is the weight, i is the weight identifier, b _i ^w is the number of bits of the weight (p _i ), and b _t ^w is the weight target number of bits considering memory capacity. And the weight target number of bits (b _t ^w ) can be set as the average value of the number of weight bits that can be set depending on the memory capacity available to store the weights of the neural network.)
Quantization precision control method calculated according to. The method of claim 18, wherein the memory capacity limit loss is
It further includes a feature memory capacity limitation loss according to the activation weight (a) of the activation function in each layer of the neural network model, and the memory capacity limitation loss is expressed in Equation

(Here, a _i is the activation weight, i is the activation weight identifier, b _i ^a is the number of bits of the activation weight (a _i ), and b _t ^a is the number of bits of the activation weight target considering memory capacity. And the activation weight target The number of bits (b _t ^a ) can be set to the average ^value of the number _of weight bits that can be set according to the memory capacity that can be utilized to store the activation weight, similar to the weight target number of bits (b t w ).)
Quantization precision control method calculated according to . The method of claim 17, wherein the step of backpropagating
A quantization precision control method for back-propagating the neural network loss to the neural network model.