KR20210093648A - Method and apparatus for processing weight of artificial neural network - Google Patents

Abstract

The present disclosure relates to a method and an electronic device for processing weights of an artificial neural network. The method for processing weights of the artificial neural network includes the steps of: quantizing the layers in the artificial neural network and weights related to connection strength between the layers; normalizing the quantized weights using a weight normalization function for adjusting the distribution of the weights; identifying the type of normalized weight; and encoding at least a portion of the weight based on the identified type of weight.

Description

Method and apparatus for processing weights in artificial neural networks {METHOD AND APPARATUS FOR PROCESSING WEIGHT OF ARTIFICIAL NEURAL NETWORK}

The present disclosure relates to a method and apparatus for processing weights in an artificial neural network. More particularly, it relates to a method and apparatus capable of expressing weights by quantizing the weights of an artificial neural network.

An artificial neural network may refer to a computing device or a method performed by a computing device to implement interconnected sets of artificial neurons. As an embodiment of the artificial neural network, a deep neural network or deep learning may have a multi-layer structure, and each of the layers may be learned according to a plurality of data.

Recently, as the development of artificial neural network technology has been activated, quantization technology has been actively studied as a method to reduce the amount of computation and improve power efficiency at the same time in the field of artificial intelligence. Although the quantization technology of an artificial neural network can provide advantages of high performance and low power consumption, low-precision-based low-bit operation according to the quantization technology has a problem in that the accuracy of the artificial intelligence neural network is deteriorated.

Accordingly, there is a demand for technology development for effectively compressing the artificial neural network and improving the accuracy of the artificial neural network by performing a low-bit operation through a quantization technique in the artificial neural network.

Korea Patent Publication No. 2019-0034985

According to an embodiment, a method and an electronic device for processing a weight of an artificial neural network may be provided.

Also, according to an embodiment, a method and an electronic device for encoding at least a portion of a weight of a quantized artificial neural network may be provided.

According to an embodiment of the present disclosure for achieving the above-described technical problem, the method comprising: quantizing layers in an artificial neural network and weights related to connection strength between the layers; normalizing the quantized weights using a weight normalization function for adjusting the distribution of the weights; identifying the type of normalized weight; and encoding at least a portion of the weight based on the identified type of weight. A method of processing weights of an artificial neural network including

In addition, according to another embodiment of the present disclosure for solving the above technical problem, a memory for storing one or more instructions; and at least one processor executing the one or more instructions. including, wherein the at least one processor quantizes the weights related to the layers in the artificial neural network and the connection strength between the layers by executing the one or more instructions, and uses the quantized weights to distribute the weights. A weight of an artificial neural network, which normalizes using a weight normalization function for adjusting , identifies a type of the normalized weight, and encodes at least a portion of the weight based on the identified type of weight An electronic device for processing may be provided.

In addition, according to another embodiment of the present disclosure for solving the above technical problem, the method comprising: quantizing layers in an artificial neural network and weights related to connection strength between the layers; normalizing the quantized weights using a weight normalization function for adjusting the distribution of the weights; identifying the type of normalized weight; and encoding at least a portion of the weight based on the identified type of weight. A computer-readable recording medium recording a program for executing the method on a computer, including a computer-readable recording medium, may be provided.

1 is a diagram illustrating a process in which an electronic device processes a weight of an artificial neural network, according to an embodiment.
2 is a flowchart of a method of processing a weight of an artificial neural network according to an embodiment.
3 is a diagram for describing a specific method of quantizing a weight of an artificial neural network according to an embodiment.
4 is a diagram for describing a specific method of normalizing a weight of an artificial neural network according to an embodiment.
5 is a diagram illustrating a weight normalization function for adjusting a distribution of weights according to an embodiment.
6 is a diagram for describing in detail a method of identifying a weight type according to an embodiment.
7 is a diagram for describing in detail a method of encoding at least a portion of a weight according to an exemplary embodiment.
8 is a diagram for describing a method of encoding at least a portion of a weight by an electronic device, according to an exemplary embodiment.
9 is a flowchart of a method of processing, by an electronic device, a weight of an artificial neural network according to another embodiment.
10 is a diagram for explaining different quantized weights according to the types of weights identified by an electronic device according to an embodiment.
11 is a block diagram of an electronic device that processes a weight of an artificial neural network according to an embodiment.
12 is a block diagram of a server capable of communicating with an electronic device according to an exemplary embodiment.
13 is a diagram for explaining the performance of an artificial neural network used by an electronic device, according to an embodiment.
14 is a diagram for explaining the performance of an artificial neural network used by an electronic device according to another embodiment.
15 is a diagram for describing an operation of an artificial neural network in an electronic device according to an embodiment.
16 is a diagram for describing an operation of an artificial neural network in an electronic device according to an embodiment.
17 is a diagram illustrating a weight normalization function for adjusting a distribution of weights according to another embodiment.

Terms used in this specification will be briefly described, and the present disclosure will be described in detail.

The terms used in the present disclosure are selected as currently widely used general terms as possible while considering the functions in the present disclosure, which may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technology, and the like. In addition, in specific cases, there are also terms arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the corresponding invention. Therefore, the terms used in the present disclosure should be defined based on the meaning of the term and the contents of the present disclosure, rather than the simple name of the term.

When a part "includes" a certain component throughout the specification, this means that other components may be further included, rather than excluding other components, unless otherwise stated. In addition, terms such as "...unit" and "module" described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software, or a combination of hardware and software. .

Hereinafter, with reference to the accompanying drawings, embodiments of the present disclosure will be described in detail so that those of ordinary skill in the art to which the present disclosure pertains can easily implement them. However, the present disclosure may be implemented in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present disclosure in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

1 is a diagram illustrating a process in which an electronic device processes a weight of an artificial neural network, according to an embodiment.

According to an embodiment, the electronic device 1000 may include an artificial neural network 102 and process weights of the artificial neural network. For example, the electronic device 1000 may quantize the weights of the artificial neural network and encode at least a portion of the quantized weights.

According to an embodiment, the electronic device 1000 includes a smart phone, a tablet PC, a PC, a smart TV, a mobile phone, a media player, a server, an AI program loaded with an AI program for processing the weights of the artificial neural network and including a voice recognition function; It may be, but is not limited to, a micro server or other mobile or non-mobile computing device.

According to an embodiment, an artificial neural network used by the electronic device 1000 may refer to a computing system focusing on a biological neural network. Unlike classical algorithms that perform tasks according to predefined conditions, artificial neural networks can learn to perform tasks by considering a large number of samples. An artificial neural network may have a structure in which artificial neurons are connected, and a connection between neurons may be referred to as a synapse. A neuron may process a received signal and transmit the processed signal to another neuron through a synapse. The output of a neuron may be referred to as activation, and a neuron and/or synapse may have a weight that can be varied, and the influence of a signal processed by the neuron may increase or decrease depending on the weight. .

For example, the artificial neural network may be composed of a plurality of neural network layers. Each of the plurality of neural network layers has a plurality of weight values (weights, 104), and a neural network operation is performed through an operation between an operation result of a previous layer and a plurality of weights. The plurality of weights of the plurality of neural network layers may be optimized by the learning result of the artificial neural network.

For example, a plurality of weights may be modified and updated so that a loss value or a cost value obtained from the artificial intelligence model during the learning process is reduced or minimized. The artificial neural network according to the present disclosure may include a deep neural network (DNN), for example, a Convolutional Neural Network (CNN), a Deep Neural Network (DNN), a Recurrent Neural Network (RNN), Restricted RBM (RBM). Boltzmann Machine), Deep Belief Network (DBN), Bidirectional Recurrent Deep Neural Network (BRDNN), or Deep Q-Networks, but is not limited to the above-described example. Hereinafter, for convenience, a case in which the artificial neural network according to the present disclosure is a deep neural network will be described as an example.

According to an embodiment, the electronic device 1000 may quantize the weights of the artificial neural network (S112) and normalize the distribution of the quantized weights (S114).

According to an embodiment, in the quantization performed by the electronic device 1000 according to the present disclosure, the number of input values is smaller than the number of input values, for example, mapping a real number to an integer through rounding. It may refer to a process of mapping to values of . According to another embodiment, the quantization performed by the electronic device 1000 may refer to a process of converting a neural network based on a floating point operation into a neural network based on a fixed point operation. Quantization according to the present disclosure may include other algorithms for converting input values into values of simplified units.

For example, the electronic device 1000 quantizes the weights of the artificial neural network (S112) and normalizes the weights of the artificial neural network (S114) in the quantization process, so that some weights having large weights are converted into weights having small weights. By normalizing, it is possible to reduce the high computational complexity required to calculate the weights of the artificial neural network. For example, referring to S114 of FIG. 1 , the electronic device 1000 may perform centralized quantization on the weights by normalizing the quantized weights while quantizing the weights.

Also, the electronic device 1000 may classify the weights quantized in S114 into preset types. For example, the electronic device 1000 may identify the type of the weights as the first type of weight 122 or the second type of weight 124 by comparing the absolute value magnitudes of the quantized weights with a preset threshold. there is. The electronic device 1000 may encode at least a portion of the quantized weights based on the identified type of weights.

The electronic device 1000 according to the present disclosure identifies the type of the distributed weight through normalization of the weight in the quantization process, and displays the weight using both values and indicators according to the identified type of weight. Therefore, it is possible to effectively compress the artificial neural network and improve the accuracy of the artificial neural network. A detailed method for the electronic device 1000 to encode a weight based on the type of weight will be described in detail with reference to FIGS. 6 to 8 , which will be described later.

2 is a flowchart of a method of processing a weight of an artificial neural network according to an embodiment.

In S210 , the electronic device 1000 may quantize the layers in the artificial neural network and weights related to connection strength between the layers. For example, the electronic device 1000 determines a first quantization function based on the size of the pre-trained weights in the artificial neural network, and minimizes the first loss function of the artificial neural network including weights quantized according to the determined quantization function. The weights can be quantized so that

More specifically, the electronic device 1000 may input input data to an artificial neural network including quantized weights, and determine a first loss function relating to a difference between an output value of the artificial neural network corresponding to the input data and a correct answer value. there is. The electronic device 1000 may quantize the weights such that the first loss function of the artificial neural network including the quantized weights is minimized. A detailed method for the electronic device 1000 to quantize the weights will be described in more detail with reference to FIG. 3 .

In S220, the electronic device 1000 may normalize the quantized weight by using the weight normalization function. For example, the electronic device 1000 may normalize the quantized weights using a weight normalization function determined according to a preset constraint among regression methodologies for avoiding overfitting. In FIG. 2 , the electronic device 1000 quantizes the weights and then normalizes the quantized weights, but is not limited thereto, and the electronic device 1000 quantizes the weights and sets the weights in advance. Normalization using a regularization function determined according to a constraint may be performed together. A detailed method for the electronic device 1000 to normalize the weights will be described in detail with reference to FIGS. 4 to 5 to be described later.

In S230 , the electronic device 1000 may identify the type of normalized weight. For example, the electronic device 1000 modifies the weights of the artificial neural network so that the first loss function of the artificial neural network including the quantized weights and the second loss function of the artificial neural network generated by normalizing the quantized weights are minimized ( modify) and refresh (refine). The electronic device 1000 compares the weight of the artificial neural network in which the first loss function and the second loss function are minimized with a preset threshold, and converts the weights into the first type of weight or the second type of weight based on the comparison result. can be classified. A detailed method for the electronic device 1000 to identify the types of normalized weights will be described in detail with reference to FIG. 6 .

In S240 , the electronic device 1000 may encode at least a portion of the weight based on the identified type of weight. According to an embodiment, the electronic device 1000 may encode all or part of the weights. A detailed method for the electronic device 1000 to encode a weight based on the type of the weight will be described in detail with reference to FIGS. 7 to 8 .

Also, according to an embodiment, although not shown in FIG. 2 , before step S210 , the electronic device 1000 in the artificial neural network based on pre-trained weights before the weights in the artificial neural network are quantized. Weights may be initialized, and weights in the initialized artificial neural network may be quantized.

For example, the electronic device 1000 may quantize the output of the activation function in the artificial neural network by limiting the range of activated values using ReLU1 as the activation function. For example, the output value of the quantized activation function may be expressed as in Equation 1 below.

Here, a is the output value of the activation function ReLU1, and the activation function Relu1 outputs 0 when the input variable x is less than 0, and outputs the input variable as it is when the input variable is located between 0 and 1. When it is greater than 1, it may correspond to a clipping function that outputs 1. Also, the activation function ReLU1 may correspond to a min function that takes an output value of max(0, x) as an input. Also, a _q represents the output value of the quantized activation function, Q _{a is} a quantization function for quantizing the output value of the activation function, and a variable k representing the number of steps between 0 and 1 and the output a of the activation function are input. It can be defined using a round function (eg, a rounding function). The electronic device 1000 according to the present disclosure further improves the accuracy of the artificial neural network by initializing the artificial neural network using the weights of the FP (Full Precision) model pre-trained through the Relu1 activation function before the weights of the artificial neural network are quantized. can do it

3 is a diagram for describing a specific method of quantizing a weight of an artificial neural network according to an embodiment.

In S320 , the electronic device 1000 may determine the first quantization function based on the size of the pre-learned weight in the artificial neural network. According to an embodiment, the electronic device 1000 may quantize the weights using a first quantization function determined according to Equation 2 below.

Here, w _q is a quantized weight, Qw is the first quantization function, w _c clipped weight, rng is a fixed range of weights, the maximum absolute value of the pre-learned weights, and s is the bit of the weights of the second type The number of bits derived from a number, the round function is a rounding function, and the clip function outputs -rng if the weight input to the clip function is less than ??rng, and if the input weight is greater than ??rng and less than rng This function outputs the value of the input weight as it is, and outputs rng if the input weight is greater than rng. According to an embodiment, the rng value may be determined in a pre-training step of the artificial neural network, and may be fixed in a re-training step to minimize quantization loss.

That is, as shown in Equation 2, according to an embodiment, the first quantization function is the output value of the clipping function that clips the initialized weights based on the size of the weights previously learned in the artificial neural network and the bits of the weights. It can be a round function that takes a number as an input. The quantization function according to the above equation may be expressed again as in Equation 3 below by inputting the clip function to the quantization function.

Here, w _q is the quantized weight, Qw is the first quantization function, M _wp is the maximum absolute value of the pre-trained weights, and clip is the clip function. If it is less than M _{wp , - M wp} is output, and the input weight is ?? Is greater than M _wp, M _wp is and output as the value of the weighted input if less than, the input weighting is a function that outputs a larger M _wp than M _wp, s _t is quantized states needed to represent the weight for the first type The number s _sl represents the number of quantization states required to represent the second type of weight.

The electronic device 1000 according to the present disclosure may prevent quantization misquantization according to the round function by clipping using the clip function Wc before quantizing the weights.

In S340 , the electronic device 1000 may quantize the weights to minimize the first loss function of the artificial neural network including the weights quantized according to the first quantization function. For example, the electronic device 1000 may quantize the weights using the first quantization function determined based on Equation 2, and then modify and update the weights of the artificial neural network using the quantized weights. After applying the input data to the artificial neural network including the quantized weight, the electronic device 1000 determines a first loss function related to a difference between an output value of the artificial neural network with respect to the input data and a correct answer value, and the determined first loss We can quantize the weights so that the function is minimized. According to an embodiment, the first loss function may include a mean square error or a cross entropy error.

4 is a diagram for describing a specific method of normalizing a weight of an artificial neural network according to an embodiment.

In S420 , the electronic device 1000 may determine the first weight normalization function based on the partial L1 constraint and the L2 constraint. According to an embodiment, the first weight normalization function used by the electronic device 1000 may be determined based on Equation 4 below.

및

는 하이퍼 파라미터들로써 제1 가중치 정규화 함수의 정규화(또는 규제화)의 강도(intensity)를 조절하기 위한 변수이고, pL1은 과적합(over fitting)을 막기 위한 회귀 방법론 중 하나로써 부분 L1 제약 조건(part of L1, pL1)을 따르는 정규화 함수이고, L2는 L2 제약 조건 (Ridge)을 따르는 정규화 함수이며, Mwp는 인공 신경망 내 가중치들이 양자화 되기 전 사전 학습된 가중치들의 최대 절대 값을 나타낸다. L2 및 pL1 제약 조건에 따른 가중치의 손실은 후술하는 도 5를 참조하여 더 구체적으로 설명하기로 한다.Here, WR is a first weight normalization function with a weight w, a first hyperparameter, and a second hyperparameter as inputs. also,

and

is a variable for adjusting the intensity of regularization (or regulation) of the first weight regularization function as hyperparameters, and pL1 is a partial L1 constraint as one of the regression methodologies to prevent overfitting. L1, pL1) is a regularization function, L2 is a regularization function conforming to the L2 constraint (Ridge), and Mwp represents the maximum absolute value of pre-trained weights before the weights in the artificial neural network are quantized. The weight loss according to the L2 and pL1 constraints will be described in more detail with reference to FIG. 5 to be described later.

In S440 , the electronic device 1000 may normalize the weights to minimize the second loss function of the artificial neural network including the weights normalized according to the first weight normalization function. For example, the electronic device 1000 normalizes the quantized weights using the first weight normalization function determined based on Equation 4, and then applies input data to the artificial neural network including the normalized weights. , a second loss function relating to a difference between an output value of the artificial neural network with respect to the input data and a correct answer value may be determined, and the weights may be normalized so that the determined second loss function is minimized.

5 is a diagram illustrating a weight normalization function for adjusting a distribution of weights according to an embodiment.

Referring to chart 502 of FIG. 5 , a loss curve 506 of the regularization function according to the L2 constraint and a loss curve 508 of the regularization function according to the partial L1 constraint pL1 are shown. A dotted line in chart 502 indicates a threshold for distinguishing a weight of a first type and a weight of a second type. A normalization function conforming to the partial L1 constraint pL1 may be determined based on Equation 5 below.

Here, pL1 is a regularization function that conforms to the partial L1 constraint. If the absolute value of the weight w is greater than the threshold, the difference value from the absolute value of the weight to the threshold is the output value, and the absolute value of the weight w is smaller than the threshold. In this case, it is a normalization function that outputs a value of 0. In addition, Mwp may be the maximum absolute value of the pre-learned weights, s _t may indicate the number of bits required to express the weight of the first type, and s _sl may indicate the number of bits required to express the weight of the second type.

Referring to the chart 512 of FIG. 5 , a loss curve of the first weight normalization function used by the electronic device 1000 is illustrated. According to an embodiment, the first weight normalization function may be defined by adding a regularization function conforming to the L2 constraint and a regularization function conforming to the partial L1 constraint pL1. Referring to the chart 512 of FIG. 5 , the first weight normalization function is defined as the sum of the regularization function conforming to the L2 constraint and the regularization function conforming to the partial L1 constraint (pL1), so a loss curve with a steeper slope near the threshold can indicate

Referring to chart 522 of FIG. 5 , the slope of the first weight normalization function according to the present disclosure is shown. Since the electronic device 1000 normalizes the weights by using the first weight normalization function indicating the slope of the chart 522 of FIG. 5, by performing normalization of a greater strength to weights above a threshold (regulation), a significant number of It can be transformed so that the weights are between the thresholds.

6 is a diagram for describing in detail a method of identifying a weight type according to an embodiment.

In S620, the electronic device 1000 compares the absolute value of the normalized weight in the quantization process and a preset threshold. In S640 , the electronic device 1000 identifies the normalized weight as one of -1, 0, or 1 or a first type of weight indicating an arbitrary value between -1 and 1 based on the comparison result, or Among them, a weight other than the first type of weight may be identified as the second type of weight. An operation in which the electronic device 1000 identifies the type of weight may be defined based on Equation 4 below.

Here, mask(w) is a mask function for identifying the type of weight, and when the absolute value of the normalized weight in the quantization process is smaller than a preset threshold, the weight is classified as a ternary weight (TW), and the weight is When the absolute value of is smaller than a preset threshold, it is a function that classifies it as a sparse-large weight. According to an embodiment, the ternary weight (TW) indicates a type of weight indicating a range between -1 and 1 or a weight indicating one of -1, 0, and 1, and a large weight (SLW) is a weight among the weights, - The type of weight representing a value less than 1 or greater than 1 may be indicated. Hereinafter, for convenience, the ternary weight TW is defined as the first type of weight and the large weight SLW is defined as the second type of weight.

7 is a diagram for describing in detail a method of encoding at least a portion of a weight according to an exemplary embodiment.

In S720 , the electronic device 1000 converts the first type of weight into binary values respectively corresponding to quantization states that can be expressed by 2 bits. For example, when the normalized weight is identified as the first type of weight in the quantization process, the electronic device 1000 may binarize the first type of weight based on a value of 0 or 1. In more detail, the electronic device 1000 may represent the weight as a binary value using 01 when the weight w is 1, 00 when the weight is 0, and 11 when the weight is -1.

In S740 , the electronic device 1000 encodes the weight of the second type by using a quantization state that does not correspond to the weight of the first type that is converted into the binary value of the weight of the second type. An operation in which the electronic device 1000 converts a first type of weight into a binary value and an operation of encoding a second type of weight that is not the first type of weight among the weights will be described in detail with reference to FIG. 8 , which will be described later. to be explained as

8 is a diagram for describing a method of encoding at least a portion of a weight by an electronic device, according to an exemplary embodiment.

The electronic device 1000 according to the present disclosure stores the weight of the first type as a value by converting it into a binary value, and the weight of the second type is in a separate quantization state that does not correspond to the weight of the first type. A weight may be represented using an indicator corresponding to . That is, the electronic device 1000 according to the present disclosure expresses the weights of the first type as values and the weights of the second type as indicators, thereby expressing the weights in the artificial neural network as a hybrid weight representation. method) can be used.

First, referring to the embodiment 802 shown in FIG. 8 , the electronic device 1000 identifies 1, 0, and -1 as the first type of weight, and 3, 2, -2, and 3 as the second type of weight. The description will be made assuming that -3 is identified. That is, the number of bits b _t of the weights of the first type is 2, the number of quantization states s _t of the weights of the first type is 3, the number of bits b _sl of the weights of the second type is 2, and the weight of the second type It is assumed that the number of quantization states s _{sl is 4.}

The electronic device 1000 may identify -1, 0, or 1 by quantizing weights in which the absolute values of the weights of the artificial neural network are smaller than the threshold 1. The electronic device 1000 may represent the first type of weights 812 (eg, -1, 0, 1) using quantization states 806 having a quantization level number of 4. The quantization states in which the number of quantization states is 4 may mean quantization states that can be expressed with 2 bits. For example, the electronic device 1000 sets 1 to a first quantization state (01), 0 to a second quantization state (00), and -1 to a third quantization state ( 11) can be mapped to each.

Also, the electronic device 1000 may encode the weight of the second type by using the fourth quantization state (00, 808), which is a quantization state that is not used to represent the weight of the first type. For example, the electronic device 1000 uses a state value 10 corresponding to a fourth quantization state 808 that is not used to convert a weight of the first type into a binary value, and performs a weight value 818 of the second type. Among them, a weight of 3 may be encoded as 1001, a weight of 2 as 1000, a weight of -2 as 1010, and a weight of -3 as indices such as 1011 may be encoded. According to the present disclosure, the indicators used by the electronic device 1000 to encode the weights of the second type may respectively correspond to quantization states of the weights of the second type.

As shown in embodiment 802 of FIG. 8 , the electronic device 1000 according to the present disclosure converts a weight of a first type into a binary value and stores it, but converts a weight of a second type into a binary value of the first type By encoding using a quantization state that is not used to represent the weight of , the accuracy of the artificial neural network can be further improved.

Referring to the embodiment 822 shown in FIG. 8 , the electronic device 1000 identifies 1, 0, and -1 as the first type of weight, and 5, 4, 3, 2, - as the second type of weight. It is assumed that 2, -3, -4, and -5 are identified. That is, the number of bits b _t of the weights of the first type is 2, the number of quantization states s _t of the weights of the first type is 3, which is the same as in the embodiment 802, but the number of bits b _sl of the weights of the second type is 3 and the quantization state number s _sl of the weights of the second type will be described on the assumption that 8 is 8.

Referring to the embodiment 822 of FIG. 8 , the operation of the electronic device 1000 converting the weight of the first type into a binary value may correspond to the embodiment 802 . That is, the electronic device 1000 may represent the first type of weights 832 (eg, -1, 0, 1) using quantization states 826 having a quantization level number of 4 . . For example, the electronic device 1000 sets 1 to a first quantization state (01), 0 to a second quantization state (00), and -1 to a third quantization state ( 11) can be mapped to each. Also, the electronic device 1000 may encode the weight of the second type by using the fourth quantization state (00) (828), which is a quantization state that is not used to represent the weight of the first type.

For example, the electronic device 1000 uses a state value 10 corresponding to the fourth quantization state 828 that is not used to convert the weight of the first type into a binary value to obtain a weight 838 of the second type. Medium, weight 5 to 10011, weight 4 to 10010, weight 3 to 10001, weight 2 to 10000, weight -2 to 10100, weight -3 to 10101, weight -4 to 10110, weight -5 can be encoded with indicators such as 10111.

Accordingly, since the electronic device 1000 according to the present disclosure normalizes the weights of the second type in the process of quantizing the weights of the artificial neural network, and uses a significant number of the weights by changing them to the weights of the first type, the weights of the artificial neural network In addition to efficiently performing the operation, the weight of the first type is expressed as a binary value, and the weight of the second type corresponds to the quantization state (808, 828) which is not used to represent the weight of the first type converted into a binary value. It is possible to improve the accuracy of the artificial neural network by encoding it using an indicator.

9 is a flowchart of a method of processing, by an electronic device, a weight of an artificial neural network according to another embodiment.

In S910 , the electronic device 1000 may primarily identify the type of weight. Since S910 may correspond to S230 of FIG. 2 , a detailed description thereof will be omitted.

In S920 , the electronic device 1000 may determine a second quantization function that is different from the first quantization function based on the first identified weight type. For example, the electronic device 1000 identifies a weight type, determines different quantization functions according to the identified weight types, and applies different quantization functions differently for each type of weight, thereby quantizing the weights quantized in step S210 . can be re-quantized. The electronic device 1000 may determine different quantization functions according to the type of weight based on Equation 7 below.

Here, mask(w) is a mask function defined in Equation 6 in order for the electronic device 1000 to primarily identify the type of weight, Q _W is quantization functions generated for each type of the identified weight, s _t is the number of quantization states of the weights of the first type, s _sl is the number of quantization states of the weights of the second type, and thresh is the threshold determined based on the maximum size of the absolute value of the weights of the artificial neural network in the pre-learning step. indicates.

For example, when the weight identified in S910 is the first type of weight (eg, ternary weight, TW), the electronic device 1000 outputs ??thresh if the weight is less than -thresh, and the weight is ??thresh and thresh, the weight is output as it is, and when the weight is greater than thresh, a clip function that outputs thresh, thresh, and a second quantization using a value of quantization states of weights of the second type + 1 function can be determined.

Also, when the weight identified in S910 is the second type of weight (eg, large weight, SLW), the electronic device 1000 outputs ??M _{wp when the weight is smaller than -M wp} , and the weight is ??M output as the weight if it has a value of _wp and M _wp, and if the weight is greater than M _wp, the clip function for outputting the M _wp, M _wp and the number of quantization states of the weights of the first type s _t and second type It is possible to determine a first quantization function that takes as an input the sum of the quantization state number ssl of the weights of . In Equation 7, when the weight identified by the electronic device 1000 is the second type of weight, the first quantization function used may correspond to the first quantization function defined based on Equation 3 above.

In S930, the electronic device 1000 may re-quantize the weights by selectively applying the first quantization function and the second quantization function according to the type of the weights identified as the first. That is, as expressed in Equation 7 above, since the electronic device 1000 quantizes the weights by selectively applying the first quantization function or the second quantization function according to the type of the weights, the electronic device calculates the weights in S930. The re-quantization operation may correspond to a selective quantization operation of weights.

For example, when the primarily identified weight type corresponds to the first weight type, the electronic device 1000 may re-quantize the weight using the second quantization function. More specifically, the electronic device 1000 may quantize the weights such that the third loss function of the artificial neural network including the weights quantized according to the second quantization function is minimized. For example, the electronic device 1000 quantizes the weights using the second quantization function, then uses the quantized weights to correct and update the weights of the artificial neural network, and input data to the artificial neural network including the quantized weights. After applying , a loss function related to the difference between the output value of the artificial neural network with respect to the input data and the correct value is determined, and the weights are quantized so that the determined loss function is minimized.

Also, when the first weight type is the second weight type, the electronic device 1000 may re-quantize the weight using the first quantization function. More specifically, the electronic device 1000 may quantize the weights such that the loss function of the artificial neural network including the weights quantized according to the first quantization function is minimized. For example, the electronic device 1000 quantizes the weights using the first quantization function, then corrects and updates the weights of the artificial neural network using the quantized weights, and inputs data to the artificial neural network including the quantized weights. After applying , a loss function related to the difference between the output value of the artificial neural network with respect to the input data and the correct answer value may be determined, and the weights may be quantized so that the determined loss function is minimized.

After quantizing the weights in step S210, the electronic device 1000 according to the present disclosure fine tunes the values of the weights of the artificial neural network learned in the previous learning step by quantizing the weights again based on the types of the quantized weights. ), and as a result, the performance of artificial neural networks can be improved.

In S940 , the electronic device 1000 may re-normalize the re-quantized weight using a second weight normalization function corresponding to the L2 constraint. For example, the electronic device 1000 may determine the second weight normalization function by removing the normalization function corresponding to the pL1 constraint shown in the chart 502 of FIG. 5 from the first weight normalization function. After the electronic device 1000 performs step S220 of FIG. 2 , the weights of the artificial neural network are intensively distributed below the threshold. Accordingly, the electronic device 1000 according to the present disclosure may improve the accuracy of weight identification by renormalizing the weights using the second regularization function determined by removing the influence of the regularization function according to the pL1 constraint from the first regularization function. can

According to an embodiment, the electronic device 1000 may renormalize the re-quantized (eg, selective quantized) weights to minimize the fourth loss function of the artificial neural network including the re-normalized weights according to the second weight normalization function. there is. In more detail, the electronic device 1000 re-normalizes the normalized weights according to the second weight normalization function, applies the input data to the artificial neural network including the normalized weights, and then applies the input data to the artificial neural network for the input data. The weight may be renormalized by determining a fourth loss function related to the difference between the output value and the correct answer value, and modifying and updating the weight so that the determined fourth loss function is minimized.

In S950, the electronic device 1000 may identify the type of the renormalized weight. Since the operation of the electronic device 1000 to identify the renormalized weight type may correspond to the operations S620 to S640 in which the electronic device 1000 identifies the weight type of FIG. 6 , a detailed description thereof will be omitted.

In S960, the electronic device 1000 may encode at least a portion of the renormalized weights based on the type of the renormalized weights. Since the operation of the electronic device 1000 to encode at least a portion of the weight based on the type of the renormalized weight may correspond to the operation of the electronic device 1000 to encode at least a portion of the weight in FIGS. 7 to 8 . A detailed description will be omitted.

In addition, according to an embodiment, although not shown in FIG. 9 , after step S960 , the electronic device 1000 coded the weight of the second type using the weights of the first type converted into binary values and indices. A step of storing all of the values in the memory of the electronic device 1000 may be further performed.

10 is a diagram for explaining different quantized weights according to the types of weights identified by an electronic device according to an embodiment.

Referring to FIG. 10 , clipped weights 1004 and 1014 and quantized weights 1006 and 1016 are shown according to the type of weight identified by the electronic device 1000 . According to an embodiment, when the type of the weight identified by the electronic device 1000 is the weight 1002 of the first type, the weights 1004 clipped according to the clipping function clip of Equation 7 have a weight range of - It converges to a value smaller than a predetermined threshold value than -1 in a place spaced apart from 1 by a preset interval, increases linearly between -1 and 1, and a value greater than 1 by a predetermined threshold value in a range where the weight range is greater than 1 can converge on In addition, it can be observed that the weights 1006 quantized according to the second quantization function with the clipping function as an input converge to -1, 0, or 1 as the first type of weight.

Also, according to an embodiment, when the type of the weight identified by the electronic device 1000 is the second type of weight 1012 , the weights 1014 clipped according to the clipping function clip of Equation 7 are the weight range. converges to the value of -3 in the range smaller than -3, increases linearly between -3 and 3, and may converge to 3 in the range where the weight range is larger than 3. In addition, it can be observed that the weights 1016 quantized according to the first quantization function to which the clipping function is input converge to -3, -2, 2, or 3 as the second type of weight.

11 is a block diagram of an electronic device that processes a weight of an artificial neural network according to an embodiment.

As shown in FIG. 11 , the electronic device for processing the weights of the artificial neural network may include a processor 1400 and a memory 1402 . However, not all illustrated components are essential components, and the electronic device 1000 may be implemented with more components than the illustrated components, and the electronic device 1000 may be implemented with fewer components. may be According to an embodiment, the electronic device 1000 may further include a communication unit (not shown) in addition to the processor 1400 and the memory 1402 .

The processor 1400 generally controls the overall operation of the electronic device 1000 .

According to an embodiment, the processor 1400 according to the present disclosure executes programs stored in the memory 1402 to perform the functions of the electronic device 1000 described in FIGS. 1 to 10 . In addition, the processor 1400 may be composed of one or a plurality of processors, and the one or more processors are a general-purpose processor such as a CPU, an AP, a digital signal processor (DSP), etc., a graphics-only processor such as a GPU, or artificial intelligence (AI). ) may be a dedicated processor. According to an embodiment, when the processor 1400 includes a general-purpose processor, an artificial intelligence processor, and a graphics-only processor, the artificial intelligence processor may be implemented as a general-purpose processor or a separate chip from the graphics-only processor.

According to an embodiment, when the processor 1400 is implemented as a plurality of processors or graphics-only processors or artificial intelligence-only processors, at least some of the plurality of processors or graphics-only processors or artificial intelligence-only processors include the electronic device 1000 and It may be mounted on another electronic device connected to the electronic device 1000 or a server.

For example, the processor 1400 quantizes the weights in the artificial neural network stored in the electronic device by executing the programs stored in the memory 1402 into the first type weight or the second type weight, and in the quantization process, the first Two types of weights can be sparse, the weights of the first type are expressed as values, and the weights of the second type are encoded using an indicator, so that the weights in the artificial neural network are quickly processed and the accuracy of the artificial neural network is improved. can do.

According to an embodiment, the processor 1400 quantizes the layers in the artificial neural network and weights related to the connection strength between the layers, and uses the quantized weights as a weight normalization function to adjust the distribution of the weights. may be used for normalization, the type of the normalized weight may be identified, and at least a portion of the weight may be encoded based on the identified type of the weight.

Also, the processor 1400 may initialize the weights in the artificial neural network based on pre-trained weights before the weights in the artificial neural network are quantized, and quantize the initialized weights in the artificial neural network. .

In addition, the processor 1400 determines a first quantization function based on the size of the pre-trained weights in the artificial neural network, and the first loss function of the artificial neural network including weights quantized according to the determined first quantization function. The weights can be quantized to minimize them.

In addition, the processor 1400 determines a first weight normalization function based on the partial L1 constraint and the L2 constraint, and a second loss function of the artificial neural network including weights normalized according to the determined first weight normalization function. The weights may be normalized to minimize .

In addition, the processor 1400 compares the absolute value of the weight with a preset threshold, and based on the comparison result, sets the normalized weight to a first type of weight indicating one of -1, 0, or 1 or the weight. Among them, a weight other than the first type of weight may be identified as the second type of weight.

In addition, the processor 1400 converts the weight of the first type into binary values corresponding to quantization states that can be expressed with two bits, and converts the weight of the second type into the weight of the first type converted to the binary value. The weight of the second type may be encoded using the non-corresponding quantization state.

Also, the processor 1400 determines a second quantization function different from the first quantization function based on the identified type of weight, and the first quantization function and the second quantization function according to the type of the identified weight. By selectively applying , the weights may be re-quantized, and the re-quantized weights may be re-normalized using a second weight normalization function corresponding to the L2 constraint.

The communication unit (not shown) may include one or more components that allow the electronic device 1000 to communicate with another device (not shown) and the server 2000 . The other device (not shown) may be a computing device such as the electronic device 1000 or a sensing device, but is not limited thereto. For example, the communication unit (not shown) may include a short-range communication unit and a mobile communication unit.

Short-range wireless communication unit, Bluetooth communication unit, BLE (Bluetooth Low Energy) communication unit, near field communication unit (Near Field Communication unit), WLAN (Wi-Fi) communication unit, Zigbee communication unit, infrared (IrDA, infrared) It may include a data association) communication unit, a Wi-Fi Direct (WFD) communication unit, an ultra wideband (UWB) communication unit, and the like, but is not limited thereto. The mobile communication unit transmits/receives a radio signal to and from at least one of a base station, an external terminal, and a server on a mobile communication network.

According to an embodiment, the communication unit (not shown) may transmit, under the control of the processor, weight values in the artificial neural network, the value of the loss function of the artificial neural network including the weights, the loss slope value, and the like, and is modified from the server. It is also possible to receive the calculated weight values, the value of the loss function, the gradient value, and the like.

The memory 1402 may store a program for processing and controlling the processor 1400 , and may also store data input to or output from the electronic device 1000 . In addition, the memory 1402 includes information on weights related to the layers constituting the artificial neural network, the nodes included in the layers, and the connection strength of the layers, and the weight normalization function, the quantization function, the type of the identified weight, and the identified weight values. Depending on the type of weight, encoded weight values may be stored. That is, when the weights in the artificial neural network are modified and updated, the memory 1402 may further store information about the modified and updated weights.

The memory 1402 may include a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (eg, SD or XD memory), and a RAM. (RAM, Random Access Memory) SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk , may include at least one type of storage medium among optical disks.

12 is a block diagram of a server capable of communicating with an electronic device according to an exemplary embodiment.

12 is a block diagram of a server providing a voice assistant service according to an embodiment.

According to an embodiment, the server 2000 may include a communication unit 2100 , a database 2200 , and a processor 2300 .

The communication unit 2100 may correspond to the communication unit (not shown) of the electronic device 1000 described above. For example, the communication unit 2100 may receive, from the electronic device 1000 , information about layers of the artificial neural network and nodes included in the layers, or weight values related to connection strength of layers in the neural network.

The database 2200 may correspond to the memory 1402 of the electronic device shown in FIG. 11 . For example, the database 2200 may store a program for processing and controlling the processor 2300 , and may also store data input to or output from the electronic device 1000 . In addition, the database 2200 includes information on weights related to the layers constituting the artificial neural network, the nodes included in the layers, and the connection strength of the layers, and the type and identification of the weight normalization function, the quantization function, and the identified weight. The encoded weight values may be stored according to the type of the weighted weight. Also, when the weights in the artificial neural network are modified and updated, the database 2200 may further store information about the modified and updated weights.

The processor 2300 typically controls the overall operation of the server 2000 . For example, the processor 2300 may control the DB 2200 and the communication unit 2100 in general by executing programs stored in the DB 2200 of the server 2000 . Also, the processor 2300 may perform some of the operations of the electronic device 1000 in FIGS. 1 to 11 by executing programs stored in the DB 2100 .

For example, the processor 2300 quantizes the received weights by receiving the initialized weights of the artificial neural network stored in the electronic device 1000, normalizes the weights in the quantization process, identifies the type of normalized weight, At least a portion of the weight may be encoded according to the identified type of weight.

13 is a diagram for explaining the performance of an artificial neural network used by an electronic device, according to an embodiment.

Referring to FIG. 13 , after the electronic device 1000 trains the artificial neural network based on data of the ImageNet dataset, the model name 1322 , the bit width 1324 , and the number of bits required to quantize the output value of the activation function 1326 ), the second hyperparameter 1328, the ratio of SLW weights that are the second type of weights 1332, the average bit length 1332 encoded as the electronic device encodes the weights, and the quantization and normalization method 1336. Accuracy of different artificial neural networks is shown.

According to an embodiment, the electronic device 1000 is a deep learning-based artificial neural network, based on ResNet-18, has a bit width of 2/3, the number of bits required to quantize the output value of the activation function is 4, and the SLW weight is present at 0.54%, the average coded bit length is 2.016, different quantization functions are applied according to the identified weight type, and when only selective quantization is performed in the quantization process, the accuracy of 57.57% is shown. Neural networks can be trained.

According to another embodiment, the artificial neural network taught by the electronic device 1000 is a deep learning-based artificial neural network based on ResNet-18, the bit width is 2/4, and the activation function is The number of bits required to quantize the output value is 4, the SLW weight is 0.269%, the average coded bit length is 2.011, and only selective quantization (SQ) is performed by applying different quantization functions according to the identified weight type. When performed, it can represent an accuracy of 59.00%.

14 is a diagram for explaining the performance of an artificial neural network used by an electronic device according to another embodiment.

Referring to FIG. 14 , in FIG. 14 , a model name 1422 of an artificial neural network learned by the electronic device 1000 , a quantum plan 1424 , a second hyper parameter 1425 , and a weight normalization function used are determined. The accuracy of the artificial neural network according to the type of constraint 1428 is shown.

According to an embodiment, in the electronic device 1000, a quantization plan (eg, bit width, 1424) is 2/4, a second hyper parameter is set to 1, a regularization function conforming to an L2 constraint, and a partial L1 constraint When training the AlexNet model using all of the regularization functions that follow (pL1), the artificial neural network can show an accuracy of 76.88%.

15 is a diagram for describing an operation of an artificial neural network in an electronic device according to an embodiment.

A detailed operation of the artificial neural network used by the electronic device 1000 will be described with reference to FIG. 15 , according to an embodiment. For example, when the artificial neural network used by the electronic device 1000 according to the present disclosure is a neural network of a ResNet structure, before the ResNet add operation, from each block of ResNet, the output of the last convolutional layers is from the maximum absolute value of the convolutional layers. It can be scaled by the derived scale factor r.

According to the method of the electronic device 1000 processing weights according to the present disclosure, the weights in the artificial neural network and the output of the activation function have the same interval (w _q= i _w ??I _w , a _q =i _a ??I). _a ) can be quantized. where I _w and i _a are fixed float intervals, and i _w and i _a are integer variables. Accordingly, the convolutional layer and the fully connected layer in the artificial neural network may be inferred by the integer operation by Equation 8 below.

where a _q is the output of the quantized activation function, w _q is the quantized weight, I _w and i _a are fixed float intervals, and i _w and i _a are integer variables. For example, batch normalization (BN), the ReLU1 activation function 1504, and the quantization function Qa(??) for quantizing the output of the activation function are integrated or compressed by integer comparators. can be The function for quantizing the output of the activation function, the boundary of the activation function, and coefficients of batch normalization may be fixed in order to be pre-ordered.

However, ResNet has an identity that prevents data streams from maintaining the same quantization intervals due to layer-wise intervals. Accordingly, to solve this problem, the electronic device 1000 may scale the input of the add operation based on the scale factor r defined by Equation 9 below. The scaling factor r may be determined based on the maximum absolute value of the blk1/shortcut convolutional layer 1506 and the maximum absolute value of the last convolutional layer blk1/conv-b 1512 . According to an embodiment, since the electronic device 1000 uses a log function and a floor function, the scaling factor r may always have a range from 0.5 to 1, and as a result, the author device 1000 performs the mapping operation of ResNet. can do it effectively. If the scaling factor r is 0, nothing is added to the add operation, and if the scaling factor r is greater than 1, the shortcut may be perturbed by a large residual.

Here, r is a scaling factor, and m _short and m _convb may represent the maximum absolute value of the blk1/shortcut convolutional layer 1506 and the maximum absolute value of the last convolutional layer blk1/conv-b 1512 , respectively.

Using Equation (10), the electronic device 1000 can adjust the distance between the j-th shortcut and the k-th convolutional layer by adjusting the scaling factor r, and only two shift operations (multiplication of 2) are used. You can add input. According to an embodiment, the electronic device 1000 may not quantize the first convolutional layer in the artificial neural network. If the first convolutional layer in the artificial neural network is not quantized, to prevent the ResNet precision output from being mixed after the add operation, the first residual block does not have a skip connection even if the input shape of the block is the same as the output shape of the first layer. may not be

16 is a diagram for describing an operation of an artificial neural network in an electronic device according to an embodiment.

Referring to FIG. 16 , a logic circuit used by an electronic device to process a hybrid weight representation (HWR) during inference time of an artificial neural network is shown. Referring to FIG. 16 , the logic circuit used by the electronic device is largely a block 1616 for receiving a weight identified as a ternary weight (TW) among the input reception node 1612 and weight 1614 of the previous layer, the weight Of 1614, a block 1618 for receiving a weight identified as a large weight (SLW), a Sign asgmt block 1622 for converting a sign, and a MUTT block 1624 for multiplying an input with a weight, a Sign asgmt block 1622 and a switch block 1626 for selecting the output of the MUTT block 1624, and a sigma block 1628 for accumulating the product sum of the input and the weight.

According to an embodiment, the electronic device may use an XNPR gate to utilize a ternary weight (TW) and a binary weight. When the input value or weight of the artificial neural network is 0, the multiplication operation is omitted, and when it is not 0, the XNOR operation may be applied. Referring to FIG. 16 , when the ternary weight is 012 or 112 and the input value is not 0, only a sign conversion operation may be performed by the sign asgmt block 1622 for converting a sign. However, when the ternary weight is 102, only integer multiplication can be performed by the MUTT block 1624 . The result value of sign conversion or multiplication may be the output value of one neuron.

17 is a diagram illustrating a weight normalization function for adjusting a distribution of weights according to another embodiment.

According to an embodiment, as described with reference to FIG. 5 , the electronic device 1000 uses a first weight normalization function determined by adding a regularization function conforming to the L2 constraint and a regularization function conforming to the partial L1 constraint pL1. weights can be normalized. However, according to another embodiment, the electronic device 1000 may normalize the weights by using a third weight normalization function different from the first weight normalization function. For example, referring to the chart 1702 of FIG. 17 , the loss curve 1706 of the regularization function conforming to the L2 constraint and the loss curve of the regularization function conforming to the part of exponential lasso (PeL1) constraint ( 1708) is shown. A dotted line in the chart 1702 may indicate a threshold for distinguishing the first type of weight and the second type of weight.

Also, referring to the chart 1712 of FIG. 17 , a loss song of the third weight normalization function used by the electronic device 1000 is illustrated. The third weight normalization function may be defined by adding a regularization function conforming to the L2 constraint and a regularization function conforming to a part of exponential lasso (PeL1) constraint. Referring to the chart 1712 of FIG. 17 , the third weight normalization function is defined as the sum of the regularization function conforming to the L2 constraint and the regularization function conforming to the part of exponential lasso (PeL1) constraint, so that it is steeper near the threshold. A loss curve with a slope can be represented. Referring to chart 1722 of FIG. 17 , the slope of the third weight normalization function according to the present disclosure is shown.

According to an embodiment, the electronic device 1000 normalizes the weights using a third weight normalization function indicating a slope as shown in the chart 1722 of FIG. 17 , thereby more effectively penalizing the weights greater than or equal to a threshold. can do. The electronic device 1000 according to the present disclosure primarily identifies a type of weight, determines different quantization functions according to the identified type of weight, and applies different quantization functions to each type of weight. performance can be further improved.

The method according to an embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the present disclosure, or may be known and available to those skilled in the art of computer software.

In addition, according to the embodiment, a computer program apparatus including a recording medium storing a program for performing another method may be provided. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

Although the embodiment of the present disclosure has been described in detail above, the scope of the present disclosure is not limited thereto, and various modifications and improved forms of the present disclosure are also provided by those skilled in the art using the basic concept of the present disclosure as defined in the following claims. belong to the scope of the right.

Claims (20)

In the method of processing the weight of an artificial neural network (Neural Network),
quantizing a weight of layers in the artificial neural network and a connection strength between the layers;
normalizing the quantized weights using a weight normalization function for adjusting the distribution of the weights;
identifying the type of normalized weight; and
encoding at least a portion of the weight based on the identified type of weight; A method comprising The method of claim 1, wherein the method
initializing the weights in the artificial neural network based on pre-trained weights before the weights in the artificial neural network are quantized; further comprising,
The method of claim 1, wherein the quantizing is quantizing the weights in the initialized artificial neural network. The method of claim 2, wherein the quantizing comprises:
determining a first quantization function based on the size of pre-trained weights in the artificial neural network; and
quantizing the weights to minimize a first loss function of the artificial neural network including weights quantized according to the determined first quantization function; A method comprising 4. The method of claim 3, wherein the normalizing comprises:
determining a first weight normalization function based on the partial L1 constraint and the L2 constraint; and
normalizing the weights to minimize a second loss function of the artificial neural network including weights normalized according to the determined first weight normalization function; A method comprising The method of claim 1, wherein the identifying comprises:
comparing the absolute value of the weight with a preset threshold; and
Based on the comparison result, a first type of weight representing one of -1, 0, or 1 as the normalized weight or a weight other than the first type of weight among the weights is identified as a second type of weight step; A method comprising The method of claim 5, wherein the encoding comprises:
converting the weight of the first type into binary values respectively corresponding to quantization states that can be expressed by two bits; and
encoding the weight of the second type by using a quantization state that does not correspond to the weight of the first type converted into the binary value; A method comprising 5. The method of claim 4, wherein the method
determining a second quantization function different from the first quantization function based on the identified type of weight;
re-quantizing the weight by selectively applying the first quantization function and the second quantization function according to the identified type of weight; and
re-normalizing the re-quantized weights using a second weight normalization function corresponding to the L2 constraint; A method further comprising: The method of claim 7, wherein the re-quantization comprises:
re-quantizing the weights to minimize a third loss function of the artificial neural network including weights quantized according to the first quantization function or the second quantization function; A method comprising The method of claim 7, wherein the re-normalizing
re-normalizing the re-quantized weights to minimize a fourth loss function of the artificial neural network including re-normalized weights according to the determined second weight normalization function; A method comprising 8. The method of claim 7, wherein the method
identifying a type of the renormalized weight; further comprising,
The encoding method, characterized in that encoding at least a part of the renormalized weight based on the type of the renormalized weight. The method of claim 1, wherein the method
storing an uncoded weight among the encoded weights and the weights; A method further comprising: 9. The method of claim 8, wherein the first quantization function is
The method according to claim 1, wherein the round function is a round function inputting an output value of a clipping function for clipping the initialized weights based on the size of the pre-trained weights in the artificial neural network and the number of bits of the weights. In the electronic device for processing the weight of the artificial neural network (Neural Network),
a memory storing one or more instructions; and
at least one processor executing the one or more instructions; including,
The at least one processor by executing the one or more instructions,
quantizing the weights related to the layers in the artificial neural network and the connection strength between the layers,
Normalizing the quantized weights using a weight normalization function for adjusting the distribution of the weights,
identify the type of the normalized weight;
An electronic device that encodes at least a portion of the weight based on the identified type of weight. 14. The method of claim 13, wherein the at least one processor comprises:
Before the weights in the artificial neural network are quantized, the weights in the artificial neural network are initialized based on pre-trained weights,
The electronic device, characterized in that the weights in the initialized artificial neural network are quantized. 15. The method of claim 14, wherein the at least one processor comprises:
Determine a first quantization function based on the size of the pre-trained weight in the artificial neural network,
quantizing the weights to minimize a first loss function of the artificial neural network including weights quantized according to the determined first quantization function. 16. The method of claim 15, wherein the at least one processor comprises:
determine a first weight normalization function based on the partial L1 constraint and the L2 constraint;
and normalizing the weight to minimize a second loss function of the artificial neural network including weights normalized according to the determined first weight normalization function. 14. The method of claim 13, wherein the at least one processor comprises:
comparing the absolute value of the weight with a preset threshold,
Based on the comparison result, a first type of weight representing one of -1, 0, or 1 as the normalized weight or a weight other than the first type of weight among the weights is identified as a second type of weight , electronic devices. 18. The method of claim 17, wherein the at least one processor comprises:
converting the weight of the first type into binary values corresponding to quantization states that can be expressed with 2 bits,
and encoding the weight of the second type by using a quantization state that does not correspond to the weight of the first type converted into the binary value. 17. The method of claim 16, wherein the at least one processor comprises:
determine a second quantization function different from the first quantization function based on the identified type of weight;
re-quantizes the weight by selectively applying the first quantization function and the second quantization function according to the identified type of weight;
and re-normalizing the re-quantized weight using a second weight normalization function corresponding to the L2 constraint. quantizing a weight of layers in an artificial neural network and a connection strength between the layers;
normalizing the quantized weights using a weight normalization function for adjusting the distribution of the weights;
identifying the type of normalized weight; and
encoding at least a portion of the weight based on the identified type of weight; A computer-readable recording medium recording a program for executing the method on a computer, comprising a.

Images