KR102645267B1 - Quantization method for transformer encoder layer based on the sensitivity of the parameter and apparatus thereof - Google Patents

Abstract

A method of performing quantization in a neural network including a plurality of transformer encoder layers, dividing the plurality of transformer encoder layers into groups including at least one transformer encoder layer based on sensitivity of a parameter and dividing. It includes the step of applying the quantization method determined for each group.

Description

Method and apparatus for quantizing a plurality of transformer encoder layers based on the sensitivity of parameters {QUANTIZATION METHOD FOR TRANSFORMER ENCODER LAYER BASED ON THE SENSITIVITY OF THE PARAMETER AND APPARATUS THEREOF}

Embodiments disclosed in this specification relate to a method and device for quantizing parameters of a transformer-based network, and more specifically, to a method and device for providing a method and device for quantizing parameters based on sensitivity.

With the advent of Bidirectional Encoder Representations from Transformers (BERT) in the field of artificial intelligence, huge models began to appear in the field of natural language processing. BERT is a Transformer-based model that enables pre-training and fine-tuning of large models in natural language processing, similar to computer vision, and provides excellent performance in a variety of problems. showed it

Vert is a widely used model in natural language processing, but the model itself uses many parameters, the model is very large, and requires a long inference time. Therefore, techniques for compressing butts have become necessary.

There are several methods of butt compression. Among them, quantization is one way to compress butts. Quantization is the use of low-precision numbers to represent a model. Low-precision numbers can save a lot of storage space and speed up inference.

It can be said that the conventional method of existing BERT quantization is not the optimal method because it does not consider the sensitivity of parameters. When quantizing a butt using the existing quantization method, two problems arise because the sensitivity of the parameters is not considered. First, the sensitive parameters of the butt are compressed too much, resulting in a decrease in accuracy. Second, because the non-sensitive parameters of the butt are not optimally compressed, a problem occurs in which the butt is not optimally compressed.

Therefore, technology to solve the above-mentioned problems has become necessary.

Meanwhile, the above-described background technology is technical information that the inventor possessed for deriving the present invention or acquired in the process of deriving the present invention, and cannot necessarily be said to be known technology disclosed to the general public before filing the application for the present invention. .

The purpose of the embodiments disclosed herein is to present a method and apparatus for quantizing a plurality of transformer encoder layers based on the sensitivity of parameters.

According to one embodiment as a technical means for achieving the above-mentioned technical problem, a method of performing quantization in a neural network including a plurality of transformer encoder layers includes at least one transformer encoder layer based on the sensitivity of the parameter. It may include dividing into groups including the transformer encoder layer and applying a quantization method determined to each divided group.

According to another embodiment, an apparatus for performing quantization in a neural network including a plurality of transformer neural network layers includes a storage unit storing a program for performing quantization and a control unit including at least one processor, The control unit may divide the plurality of transformer encoder layers into groups including at least one transformer encoder layer based on parameter sensitivity and apply a quantization method determined for each divided group.

According to another embodiment, a computer-readable recording medium records a program for executing a method of performing quantization in a neural network including a plurality of transformer encoder layers in a computer, wherein the plurality of transformer encoder layers are stored based on the sensitivity of parameters. The quantization method performed in the quantization device includes dividing the plurality of transformer encoder layers into groups including at least one transformer encoder layer based on parameter sensitivity and applying a quantization method determined for each divided group. May include steps.

According to another embodiment, it is performed by an apparatus for quantizing a plurality of transformer encoder layers based on the sensitivity of a parameter, and is a computer program stored in a recording medium to perform the quantization method, and quantizing the plurality of transformers based on the sensitivity of the parameter. A quantization method performed in an apparatus for quantizing an encoder layer includes dividing the plurality of transformer encoder layers into groups including at least one transformer encoder layer based on sensitivity of a parameter, and a quantization method determined for each divided group. It may include the step of applying.

According to any of the above-mentioned problem solving methods, much less capacity than the default may be consumed when storing the final model because quantization is performed based on the sensitivity of the parameters.

In addition, by quantizing the sensitive part to 8 bits and the less sensitive part to 1 bit, it is possible to provide a quantization method that is small in size but has no loss of accuracy.

In addition, to improve the performance of the 1-bit part, three learning methods were proposed to further improve performance.

Additionally, for fast inference speed, XNOR-COUNT operation is applied to the 1-bit part, and FP 16 GEMM is applied to the 8-bit index part to enable high-speed calculation.

The effects that can be obtained from the disclosed embodiments are not limited to the effects mentioned above, and other effects not mentioned are clear to those skilled in the art to which the disclosed embodiments belong from the description below. It will be understandable.

Figure 1 shows an example diagram for explaining a process of quantizing a plurality of transformer encoder layers based on the sensitivity of parameters.
FIG. 2 is a block diagram illustrating the configuration of an apparatus for quantizing a plurality of transformer encoder layers based on sensitivity of parameters according to an embodiment.
Figure 3 is a flowchart illustrating a method of quantizing a plurality of transformer encoder layers based on the sensitivity of parameters according to an embodiment.
Figure 4 is an example diagram to explain 8-bit index quantization.
Figure 5 is an example diagram to explain the process of applying LWFT.

Below, various embodiments will be described in detail with reference to the attached drawings. The embodiments described below may be modified and implemented in various different forms. In order to more clearly explain the characteristics of the embodiments, detailed descriptions of matters widely known to those skilled in the art to which the following embodiments belong have been omitted. In addition, in the drawings, parts that are not related to the description of the embodiments are omitted, and similar parts are given similar reference numerals throughout the specification.

Throughout the specification, when a configuration is said to be “connected” to another configuration, this includes not only cases where it is “directly connected,” but also cases where it is “connected with another configuration in between.” In addition, when a configuration “includes” a configuration, this means that other configurations may be further included rather than excluding other configurations, unless specifically stated to the contrary.

However, before explaining this, we first define the meaning of the terms used below.

Hereinafter, a 'neural network' may be composed of an input layer, at least one hidden layer, and an output layer, and each layer may be composed of at least one 'node'. And the nodes of each layer can form a connection relationship with the nodes of the next layer.

And 'parameter' is a value that determines the intensity of reflection of the data input to the layer when transferring the data input to the node of each layer of the neural network to the next layer. For example, weight, kernel parameter Or it may be Activation.

And ‘data’ is the value input at each layer stage of the neural network.

Bidirectional Encoder Representations from Transformers (BERT) is a transformer-based neural network. At this time, the transformer neural network follows the encoder-decoder structure of seq2seq and is a model implemented only with attention without using RNN. BERT includes multiple transformer encoder layers, and each transformer encoder layer is an MLM (Masked Language Model) that masks some words in a sentence and then predicts them, and an NSP that predicts the next sentence. It is pre-trained by Next Sentence Prediction, and then retrained (fine-tuned) for specific tasks. At this time, specific tasks may include language inference and question answering.

According to one embodiment, BERT-base may include a WordPiece embedding layer and 12 transformer encoder layers, and each transformer encoder layer includes a self-attention layer. and a Feed Forward Network (FFN). At this time, the wordpiece embedding layer performs embedding on the input.

In general, each self-attention layer and feedforward network determine the final output through operations between the embedding of the natural language received as input and matrices including Key, Value, and Query matrices. .

Generally, the output of the feed forward network used in BERT is as shown in [Equation 1].

[Equation 1]

FFN(x)= max(0,xW1+b1)W2+b2

Here, x means input, w1, w2 mean weight, and b1, b2 mean bias. max(c,d) means the larger value between c and d.

Quantization is one way to compress BERT. Quantization is the use of low-precision numbers to represent a model. Low-precision numbers can save a lot of storage space and speed up inference. In general, BERT parameters, including all weights and activations, use the FP32 (32-bit Floating Point) format. That is, all parameters of a general BERT are expressed as 32-bit floating point numbers, and parameters of a quantized model can be expressed as low-precision numbers of 8, 4, 2, or 1 bit.

Hereinafter, embodiments will be described in detail with reference to the attached drawings.

Figure 1 shows an example diagram for explaining a process of quantizing a plurality of transformer encoder layers based on the sensitivity of parameters.

Referring to FIG. 1, a process of quantizing a first neural network 10 including a plurality of transformer encoder layers into a second neural network 20 is shown. According to one embodiment, the first neural network 10 may be pre-trained BERT. Among the transformer encoder layers, some transformer encoder layers are quantized into MP encoder layers, and the remaining transformer encoder layers are quantized into 8-bit encoder layers. For example, among N transformer encoders, M transformer encoder layers are quantized into MP encoder layers, and N-M transformer encoder layers are quantized into 8-bit encoder layers.

The 8-bit encoder layer may include a feedforward network and a self-attention layer each quantized by applying 8-bit index quantization, and the MP encoder layer may include a feedforward network quantized by applying 1-bit quantization and an 8-bit index quantization (8 -bit Index Quantization) can be applied to include a quantized self-attention layer. Preferably, the embedding layer can also be quantized by applying 8-bit index quantization.

A specific method of quantizing a plurality of transformer encoder layers based on parameter sensitivity will be described in detail below with reference to other drawings.

FIG. 2 is a block diagram illustrating the configuration of an apparatus 200 for quantizing a plurality of transformer encoder layers based on sensitivity of parameters according to an embodiment.

Referring to FIG. 2, the device 200 for quantizing a plurality of transformer encoder layers based on the sensitivity of parameters according to an embodiment may include an input/output unit 210, a storage unit 220, and a control unit 230. there is.

The input/output unit 210 according to an embodiment includes an input device for receiving input from a user, and information such as the status of the device 200 for quantizing a plurality of transformer encoder layers based on the result of performing a task or the sensitivity of parameters. It may include an output device for displaying. For example, the input/output unit 210 may include an input unit for receiving a data processing command and an output unit for outputting a result processed according to the received command. According to one embodiment, the input/output unit 210 may include a user input means such as a keyboard, mouse, or touch panel, and an output means such as a monitor or speaker.

Meanwhile, the storage unit 220 may store data for quantizing a plurality of transformer encoder layers based on parameter sensitivity. For example, data required for quantization to quantize a neural network can be stored. Additionally, various data or programs required to quantize a plurality of transformer encoder layers based on parameter sensitivity can be stored.

Additionally, the control unit 230 controls the overall operation of the device 200 for quantizing a plurality of transformer encoder layers based on the sensitivity of parameters, and may include a processor such as a CPU. In particular, the control unit 230 may execute a program stored in the storage unit 220 or read data and quantize a plurality of transformer encoder layers based on the sensitivity of the parameter. A specific method by which the control unit 230 quantizes a plurality of transformer encoder layers based on parameter sensitivity will be described in detail below with reference to other drawings.

Figure 3 is a flowchart illustrating a method of quantizing a plurality of transformer encoder layers based on the sensitivity of parameters according to an embodiment.

Referring to FIG. 3, the control unit 230 divides the plurality of transformer encoder layers into groups including at least one transformer encoder layer based on parameter sensitivity (S310). After step S310, the control unit 230 applies the quantization method determined for each divided group (S320).

When quantizing a butt using the existing quantization method, two problems arise because the sensitivity of the parameters is not considered. First, the sensitive parameters of the butt are compressed too much, resulting in a decrease in accuracy. Second, because the non-sensitive parameters of the butt are not optimally compressed, a problem occurs in which the butt is not optimally compressed. To solve this problem, considering the sensitivity of the parameters, part of the butt is quantized by applying 8-bit index quantization, and the remaining part is quantized by applying 1-bit quantization. Meanwhile, the LayerNorm layer and bias, which account for a very small portion of the overall model size, may not be quantized.

Because the self-attention layer plays a critical role in improving the accuracy of the model by calculating the relationship between input word embeddings, the sensitivity of the parameters of the self-attention layer is higher than the sensitivity of the parameters of the feed forward network. In addition, the transformer encoder layer close to the input plays a critical role in improving the accuracy of the model by extracting important low-dimensional features from the input. Therefore, the sensitivity of the parameters of the transformer encoder layer increases as the layer closer to the initial input increases. is high.

Since the sensitivity of the parameters of the transformer encoder layer is higher in layers closer to the initial input, and the sensitivity of the parameters of the self-attention layer is higher than that of the feed forward network, the MP encoder layer can be placed close to the final output layer, The 8-bit encoder layer can be placed close to the first input layer.

In relation to this, the control unit 230 divides the plurality of transformer encoder layers into a first group close to the initial input and a second group close to the final output.

The control unit 230 may divide a preset number of layers into a first group and the remaining layers into a second group in the order closest to the initial input. For example, if there are a total of 5 layers, the control unit 230 may divide the 2 layers close to the initial input into the first group and the remaining 3 layers into the second group.

The control unit 230 applies 8-bit index quantization to the first group close to the initial input and quantizes it with an 8-bit encoder layer including a quantized feed forward network and self-attention layer, and 1-bit quantization to the second group, which is the remaining layer. By applying a quantized feed forward network and 8-bit index quantization, it is quantized into an MP encoder layer including a quantized self-attention layer.

Figure 4 is an example diagram to explain 8-bit index quantization.

Referring to FIG. 4, 8-bit Index Quantization is performed through quantization, de-quantization, and training steps.

In forward propagation, the control unit 230 sequentially applies quantization and de-quantization to the weight matrix using an 8-bit index. Through this, the size and accuracy of the model can be minimized. At this time, de-quantization means expressing the number in bit format before quantization.

In the present invention, an 8-bit index means an INT8 index format expressed as one of 256 integers in the range of [-128,127].

For example, in each layer, the control unit 230 first quantizes the FP32 format weight matrix into an INT8 index format matrix, and dequantizes the INT8 index format matrix into an FP32 format weight matrix.

8-bit index quantization and de-quantization are performed by equation (2) and equation (3).

[Equation 2]

[Equation 3]

here, is the quantized weight matrix with the 8-bit index of the lth layer, is the weight matrix expressed in FP32 format of the lth layer, silver The minimum weight value at, is the quantization weight scale of the lth layer, is the quantization scale, is the minimum value of a number that can be expressed by quantization, means the inverse-quantized weight matrix of the lth layer. Floor(x) is a function that outputs the maximum integer less than x. especially , can be derived from equations (4), (5), and (6).

[Equation 4]

[Equation 5]

[Equation 6]

Here, b is the number of quantization bits, is the maximum value of a number that can be expressed by quantization, Is means the maximum weight value. For example, in the 8-bit index quantization method, b is 8, is 127, has a value of -128.

For example, in Burt's first transformer encoder layer, the feed forward network has a weight matrix dimension of 768*3072, which means it has 768*3072 parameters.

Referring to FIG. 4, the control unit 230 quantizes the 4*4 weight matrices into INT8 format according to Equation 2. That is, the control unit 230 quantizes all weights of the matrix expressed in FP32 format into an INT8 index format expressed as one of 256 integers in the range of [-128,127]. And after that, the control unit 230 de-quantizes the weight matrix quantized in INT8 format back into FP32 format according to Equation 3. After de-quantization is performed, the control unit 230 performs the forward propagation process performed by the existing BERT.

In backward propagation, the control unit 230 can learn weights using a standard gradient descent method. At this time, since the weight cannot be learned using the differential value of the Floor function used in quantization, the control unit 230 uses an 8-bit clip function instead of the Floor function. At this time, the 8-bit clip function is a function that approximates the Floor function. That is, in reverse propagation, the control unit 230 can learn the weight using an 8-bit clip function.

From Equation 7 used in standard gradient descent, the 8-bit clip function is derived from Equation 8.

[Equation 7]

[Equation 8]

After learning is completed, the control unit 230 may store the maximum and minimum values of the quantized weight matrix and the maximum and minimum values of the weight matrix in INT8 format. Through this, the size of the model can be reduced to 1/4.

Below, a 1-bit quantization method, which is an embodiment of the present invention, will be described.

1-bit quantization refers to quantization that expresses weight and activation as two numbers. Unlike 8-bit index quantization, in which only the index is compressed while the information is intact (intact), 1-bit quantization may result in significant information loss.

In forward propagation, the control unit 230 can quantize all weights and activations to -1 or +1 using the sign function.

The sign function used in 1-bit quantization is as shown in Equation 9.

[Equation 9]

here, and means activation and weight in FP32 format, respectively. means quantized activation and weight, respectively.

In backward propagation, the derivative of the sign function has 0 in almost all regions, so standard gradient descent cannot be used directly in the learning phase. Therefore, in reverse propagation, the control unit 230 learns the weight using a 1-bit clip function. This is derived from Equation 10 and Equation 11.

[Equation 10]

[Equation 11]

Here, L refers to the loss function of the model, means learning rate, means a weight matrix quantized into 2 bits. by the clip function If exists in the area of [-1,1], it has the value of 1, otherwise it has the value of 0.

is derived from Equation 12 by the chain rule.

[Equation 12]

here, means the activation matrix in FP32 format, means the simplified mid term of activation matrix, represents the differentiation of the LayerNorm layer and the GeLU activation function.

Additionally, in order to reduce the accuracy loss caused by 1-bit quantization, in the learning step, the control unit 230 can perform absolute binary weight regularization (ABWR), which learns each absolute value of the parameter to be close to 1. there is.

95% of the weights of pre-trained BERT are in the range [-0.07,0.07], It is far from This is one of the main reasons why accuracy drops significantly when applying 1-bit quantization.

In relation to this, the control unit 230 may introduce a regularizer so that the weight of the 32FP format approaches +1 in the learning stage. ABWR's intuition is to minimize the loss of accuracy when applying 1-bit quantization by initially training the absolute value of the weight to be close to +1.

The absolute binary weight normalization is derived from Equation 13.

[Equation 13]

here, means ABWR term. In the learning phase, the loss function By adding , absolute binary weight normalization can be performed to minimize the decrease in accuracy of 1-bit quantization.

In relation to this, the control unit 230 may add the ABWR term to the loss function so that the weight of the 32FP format approaches +1 in the learning stage.

In addition, in order to prevent forgetting the previously learned knowledge learned in the FP32 format, the control unit 230 performs a binarization feature priority learning (Prioritized) function that allows the weights in the FP32 format to learn the binary characteristics of the input before performing 1-bit quantization. Training, PT) can be performed. Binarization feature priority learning allows a neural network expressed in FP32 format to learn the features of the binarized input, then quantize the weight with 1-bit quantization, and then learn the binarized feature on the 1-bit quantized weight. says

General 1-bit quantization learns the weights by directly quantizing the activation and weights. On the other hand, PT first learns FP32 format weights and then applies 1-bit quantization, and in this case, activation maintains the 1-bit quantized state from the beginning. In other words, PT first learns a neural network in FP32 format from the binary features of the input to better understand the binary features of the input, and then applies 1-bit quantization to the neural network.

Unlike the existing 1-bit quantization, PT adds the process of learning FP32 format weights, allowing the binary features of the input to be better learned in the weights.

Additionally, in the learning stage of 1-bit quantization, the control unit 230 can apply layer-wise re-learning (Layer-Wise Fine-Tuning, LWFT). Through this, the problem of accuracy degradation due to knowledge loss that occurs when too many layers are quantized by 1 bit at once can be solved.

LWFT works as follows. The control unit 230 first quantizes one layer among the plurality of transformer encoder layers into an MP encoder layer, and then sequentially quantizes the next layer into an MP encoder layer. At this time, no parameters of the plurality of transformer encoder layers are frozen. The intuition of LWFT is not to learn the final quantized neural network at once, but to place the MP encoder layers one by one. Through this, it is possible to effectively avoid inability caused by learning the final quantized neural network at once.

Figure 5 is an example diagram to explain the process of applying LWFT.

Referring to Figure 5, it can be seen that 8-bit index quantization is performed in the self-attention layer of the 8-bit encoder and MP encoder layer. In addition, it can be seen that the process of quantizing a second MP encoder layer by a neural network that has completed quantization with one MP encoder layer is shown. The control unit 230 can complete quantization of the MP encoder layer for each layer. That is, the control unit 230 can sequentially complete 1-bit quantization for each layer of the MP encoder layer that needs to perform 1-bit quantization.

In relation to this, the control unit 230 may sequentially apply 1-bit quantization for each layer of the second group of feed forward networks that are close to the final output.

Additionally, to increase the inference speed, the control unit 230 can apply XNOR-Count GEMM and FP 16 GEMM to the 1-bit and 8-bit portions of the neural network. In this case, GEMM (GEneral Matrix Multiplication) refers to matrix multiplication.

In the 1-bit quantization part, XNOR-Count GEMM can be used, replacing the general GEMM. XNOR-Count GEMM is a method used for fast matrix multiplication in matrices quantized to 1 bit. XNOR-Count GEMM can only be used in binarized neural networks, and requires much less computational cost than existing GEMM.

XNOR-Count GEMM consists of Equation 14.

[Equation 14]

here, represents the i row j column element of the activation matrix of the l+1th layer, refers to the i row of the activation matrix of the lth layer, means the j column of the lth weight matrix.

The XNOR-Count GEMM first performs an ) is a method of finding the value of.

The control unit 230 can replace the existing FP32 format matrix multiplication with XNOR-Count GEMM.

Relatedly, it is difficult for the widely used NVIDIA GPU and deep learning framework to store each weight using 1 bit. To solve this problem, the control unit 230 stores 32 binary weights in FP32 format numbers and calculates 32 binary weights at once. That is, the control unit 230 encodes the input matrix and weight matrix into a 1-bit small matrix during each matrix multiplication process. Afterwards, the control unit 230 can apply XNOR-Count GEMM to the encoded small matrix.

For example, if the dimensions of the input matrix and weight matrix are (100,3072) and (3072, 768), the control unit 230 encodes the input matrix and weight matrix as (100, 96) and (96, 768) and 32 XNOR-Count GEMM is applied to each weight. Because the weight matrix is already stored in encoded format after learning, there is no need to encode the weight matrix in the inference step.

For the 8-bit index quantization part, the control unit 230 can use FP16 GEMM to replace FP32 GEMM. The reason for choosing inter-matrix multiplication in the FP 16 format is that the accuracy of the neural network is higher and the inference speed is faster than that of the FP32 format. At this time, FP16 GEMM means performing a general GEMM using the FP16 format.

In the inference step, the control unit 230 dequantizes the INT8 index format matrix into an FP32 format weight matrix according to Equation 3, and then quantizes the FP32 format weight matrix into a 16 format weight matrix. Afterwards, the control unit 230 can process all tasks in FP16 format.

Unlike the learning process, there is no need to quantize and de-quantize the weight matrix in every forward propagation because the weight matrix is already stored in INT8 format. Additionally, the control unit 230 may use the FP16 format for the bias and LayerNOrm layers to maintain the same accuracy and high inference speed.

In addition, the terms “…unit” and “…module” used in the specification refer to a unit that processes at least one function or operation, which may be implemented as hardware, software, or a combination of hardware and software.

The term '~unit' used in the above embodiments refers to software or hardware components such as FPGA (field programmable gate array) or ASIC, and the '~unit' performs certain roles. However, '~part' is not limited to software or hardware. The '~ part' may be configured to reside in an addressable storage medium and may be configured to reproduce on one or more processors. Therefore, as an example, '~ part' refers to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, and procedures. , subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables.

The functions provided within the components and 'parts' may be combined into a smaller number of components and 'parts' or may be separated from additional components and 'parts'.

In addition, the components and 'parts' may be implemented to regenerate one or more CPUs within the device or secure multimedia card.

The quantization method according to the embodiments described with reference to FIGS. 3 to 5 may also be implemented in the form of a computer-readable medium that stores instructions and data executable by a computer. At this time, instructions and data can be stored in the form of program code, and when executed by a processor, they can generate a certain program module and perform a certain operation. Additionally, computer-readable media can be any available media that can be accessed by a computer and includes both volatile and non-volatile media, removable and non-removable media. Additionally, computer-readable media may be computer recording media, which are volatile and non-volatile implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. It can include both volatile, removable and non-removable media. For example, computer recording media may be magnetic storage media such as HDDs and SSDs, optical recording media such as CDs, DVDs, and Blu-ray discs, or memory included in servers accessible through a network.

Additionally, the quantization method according to the embodiments described with reference to FIGS. 3 to 5 may be implemented as a computer program (or computer program product) including instructions executable by a computer. A computer program includes programmable machine instructions processed by a processor and may be implemented in a high-level programming language, object-oriented programming language, assembly language, or machine language. . Additionally, the computer program may be recorded on a tangible computer-readable recording medium (eg, memory, hard disk, magnetic/optical medium, or solid-state drive (SSD)).

Therefore, the quantization method according to the embodiments described with reference to FIGS. 3 to 5 can be implemented by executing the above-described computer program by a computing device. The computing device may include at least some of a processor, memory, a storage device, a high-speed interface connected to the memory and a high-speed expansion port, and a low-speed interface connected to a low-speed bus and a storage device. Each of these components is connected to one another using various buses and may be mounted on a common motherboard or in some other suitable manner.

Here, the processor can process instructions within the computing device, such as displaying graphical information to provide a graphic user interface (GUI) on an external input or output device, such as a display connected to a high-speed interface. These may include instructions stored in memory or a storage device. In other embodiments, multiple processors and/or multiple buses may be utilized along with multiple memories and memory types as appropriate. Additionally, the processor may be implemented as a chipset consisting of chips including multiple independent analog and/or digital processors.

Memory also stores information within a computing device. In one example, memory may be comprised of volatile memory units or sets thereof. As another example, memory may consist of non-volatile memory units or sets thereof. The memory may also be another type of computer-readable medium, such as a magnetic or optical disk.

And the storage device can provide a large amount of storage space to the computing device. A storage device may be a computer-readable medium or a configuration that includes such media, and may include, for example, devices or other components within a storage area network (SAN), such as a floppy disk device, a hard disk device, an optical disk device, Or it may be a tape device, flash memory, or other similar semiconductor memory device or device array.

The above-described embodiments are for illustrative purposes, and those skilled in the art will recognize that the above-described embodiments can be easily modified into other specific forms without changing the technical idea or essential features of the above-described embodiments. You will understand. Therefore, the above-described embodiments should be understood in all respects as illustrative and not restrictive. For example, each component described as single may be implemented in a distributed manner, and similarly, components described as distributed may also be implemented in a combined form.

The scope sought to be protected through this specification is indicated by the patent claims described later rather than the detailed description above, and should be interpreted to include the meaning and scope of the claims and all changes or modified forms derived from the equivalent concept. .

200: Device for quantizing a plurality of transformer encoder layers based on sensitivity of parameters 210: Input/output unit
220: storage unit 230: control unit

Claims (18)

A method of performing quantization in a neural network including a plurality of transformer encoder layers by a device including a control unit and quantizing the plurality of transformer encoder layers,
dividing, by the control unit, the plurality of transformer encoder layers into groups including at least one transformer encoder layer based on sensitivity of a parameter; and
It includes applying, by the control unit, a quantization method determined for each divided group,
The parameter represents a value that determines the intensity of reflection of data input to each layer of a model with a transformer-based neural network,
The sensitivity of the parameter corresponds to the change in accuracy of the model due to compression of the parameter,
The step of dividing the plurality of transformer encoder layers into groups including at least one transformer encoder layer includes a preset number of layers as a first group and the remaining layers as a second group in order from the first input layer. Divide,
In the step of applying the quantization method determined for each divided group, the first group set from the initial input layer is quantized with an 8-bit encoder layer including a feed forward network and a self-attention layer, each quantized by applying 8-bit index quantization. And, the second group set from the final output layer is quantized by an MP encoder layer including a feed forward network quantized by applying 1-bit quantization and a self-attention layer quantized by applying 8-bit index quantization. delete delete According to paragraph 1,
The 8-bit index quantization is,
sequentially applying quantization and inverse-quantization using an 8-bit index in forward propagation; and
A method comprising training weights using an 8-bit clip function in back propagation. According to paragraph 1,
The 1-bit quantization is,
Quantizing using a sign function in forward propagation; and
A method comprising training weights using a 1-bit clip function in back propagation. According to clause 5,
The 1-bit quantization is,
The method further includes performing absolute binary weight regularization to learn each absolute value of the parameter to be close to 1. According to clause 5,
The 1-bit quantization is,
Before performing the 1-bit quantization, the method further comprises performing binarization feature priority training (Prioritized Training) such that weights in FP32 format learn the binary characteristics of the input. According to clause 5,
The 1-bit quantization is,
The method further includes applying layer-wise fine-tuning in the learning step. In a device that performs quantization in a neural network including a plurality of transformer encoder layers,
a storage unit in which a program for performing quantization is stored; and
A control unit including at least one processor,
The control unit,
Divide the plurality of transformer encoder layers into groups including at least one transformer encoder layer based on sensitivity of parameters,
The quantization method determined for each divided group is applied,
The parameter represents a value that determines the intensity of reflection of data input to each layer of a model with a transformer-based neural network,
The sensitivity of the parameter corresponds to the change in accuracy of the model due to compression of the parameter,
The control unit divides the plurality of transformer encoder layers in order from the first input layer, with a preset number of layers into a first group and the remaining layers into a second group,
The control unit quantizes the first group set from the initial input layer into an 8-bit encoder layer including a feed forward network and a self-attention layer each quantized by applying 8-bit index quantization, and the second group set from the final output layer. The group quantizes the device with a feedforward network that is quantized by applying 1-bit quantization and an MP encoder layer that includes a self-attention layer that is quantized by applying 8-bit index quantization. delete delete According to clause 9,
The control unit,
The 8-bit index quantization is performed,
In forward propagation, quantization and inverse-quantization are applied sequentially using an 8-bit index,
A device that trains weights using an 8-bit clip function in backpropagation. According to clause 9,
The control unit,
The above 1-bit quantization is performed,
In forward propagation, quantize using the sign function,
Apparatus for learning weights using a 1-bit clip function in backpropagation According to clause 13,
The control unit,
The above 1-bit quantization is performed,
A device that performs absolute binary weight regularization, which learns each absolute value of a parameter to be close to 1. According to clause 13,
The control unit,
The above 1-bit quantization is performed,
Before performing the 1-bit quantization, a device that performs binarization feature priority learning (Prioritized Training) such that weights in FP32 format learn the binary characteristics of the input. According to clause 13,
The control unit,
The above 1-bit quantization is performed,
A device that applies layer-wise relearning (Layer-Wise Fine-Tuning) in the learning stage. A computer-readable recording medium on which a program for executing the method according to claim 1 is recorded on a computer. A computer program stored in a medium for performing the method according to claim 1, which is performed by an apparatus for quantizing a plurality of transformer encoder layers based on the sensitivity of parameters,
The parameter represents a value that determines the intensity of reflection of data input to each layer of a model with a transformer-based neural network,
The computer program of claim 1, wherein the sensitivity of the parameter corresponds to a change in accuracy of the model due to compression of the parameter.

Images