KR20220069225A - Knowledge distillation method for compressing transformer neural network and apparatus thereof - Google Patents

Abstract

A method for training a student network including at least one transformer neural network by using knowledge distillation in a teacher network including the at least one transformer neural network includes; pre-training the student network in a pretraining with teacher's predictions scheme of initializing the student network based on a result value of the teacher network previously trained with learning data; and training the student network by performing the knowledge distillation in the teacher network.

Description

KNOWLEDGE DISTILLATION METHOD FOR COMPRESSING TRANSFORMER NEURAL NETWORK AND APPARATUS THEREOF

Embodiments disclosed herein relate to a knowledge distillation method and apparatus for lightening a transformer-based network, and more particularly, to a method and apparatus for providing a knowledge distillation method and apparatus with increased parameter efficiency and accuracy.

With the advent of Bidirectional Encoder Representations from Transformers (BERT) in the field of artificial intelligence, huge models began to appear in the field of natural language processing. BERT is a Transformer-based model that enables pre-training and fine-tuning of huge models in natural language processing just like computer vision, and has excellent performance in various problems. showed

However, the disadvantage of BERT is that the model is very large, and BERT-base uses about 110 million parameters, so a very large amount of memory is required to use BERT.

Therefore, model compression of BERT was required, and the present invention relates to a Transformer-based model widely used in the field of natural language processing, in particular, Model Compression for BERT. can

While maintaining the performance of the BERT, the need for technology to reduce the weight of the model emerged.

One of them, the Knowledge Distillation (KD) method, trains the student network by transferring the generalization ability of a larger teacher network to a generally much smaller student network.

However, the existing knowledge distillation used to lighten the model of BERT had two problems.

First, the size of the student network was absolutely small and the model complexity was insufficient.

Second, due to the absence of an initial guide in the student network, it was difficult for the student network to faithfully imitate the performance of the teacher network.

Therefore, there is a need for a technique for solving the above-mentioned problems.

On the other hand, the above-mentioned background art is technical information that the inventor possessed for the purpose of derivation of the present invention or acquired during the derivation process of the present invention, and it cannot be said that it is necessarily known technology disclosed to the general public before the filing of the present invention. .

Embodiments disclosed herein have an object to present a knowledge distillation method and apparatus in order to lighten a transformer-based network.

According to an embodiment, as a technical means for achieving the above-described technical task, a transformer neural network is formed using knowledge distillation in a teacher network including at least one transformer neural network. A method for learning a student network including at least one is a PTP (Pretraining with Teacher's Predictions) method of initializing the student network based on a result value of the teacher network that has been previously learned by learning data. pre-training the network; and performing a knowledge distillation process on the teacher network to learn the student network.

According to another embodiment, a student network including at least one transformer neural network using knowledge distillation in a teacher network including at least one transformer neural network The knowledge distillation apparatus for learning includes: a storage unit in which a program for performing knowledge distillation is stored; and a control unit including at least one processor, wherein the control unit initializes the student network based on a result value of the teacher network that has been previously learned by learning data in a PTP (Pretraining with Teacher's Predictions) method. By pre-training the student network and performing a knowledge distillation process on the teacher network, the student network may be trained.

According to another embodiment, as a computer-readable recording medium recording a program for executing a knowledge distillation method in a computer, the knowledge distillation method performed in the knowledge distillation apparatus includes at least one transformer neural network In the method for learning a student network including at least one transformer neural network using knowledge distillation in a teacher network, the result value of the teacher network previously learned by learning data pre-training the student network in a PTP (Pretraining with Teacher's Predictions) method of initializing the student network based on and performing a knowledge distillation process on the teacher network to learn the student network.

According to another embodiment, as a computer program stored in a recording medium to perform the knowledge distillation method and performed by the knowledge distillation apparatus, the knowledge distillation method performed in the knowledge distillation apparatus comprises at least a transformer neural network. A method for learning a student network including at least one transformer neural network using knowledge distillation in a teacher network including one, wherein the teacher has been previously learned by learning data pre-training the student network in a PTP (Pretraining with Teacher's Predictions) method for initializing the student network based on a result value of the network; and performing a knowledge distillation process on the teacher network to learn the student network.

According to any one of the above-described task solving means, the learning performance of the student network may be improved by increasing the complexity of the student network.

In addition, by presenting an initialization task of the student network that did not exist before, the learning performance of the student network may be improved. Therefore, despite the weight reduction of the network, it is possible to maintain the learning performance of the teacher network.

Effects obtainable in the disclosed embodiments are not limited to the above-mentioned effects, and other effects not mentioned are clear to those of ordinary skill in the art to which the embodiments disclosed from the description below belong. can be understood clearly.

1 shows an exemplary diagram for explaining a process of learning a DNN using knowledge distillation.
2 is a block diagram illustrating the configuration of a knowledge distillation apparatus according to an embodiment.
3 is a flowchart illustrating a method of performing knowledge distillation for learning a student network according to an embodiment.
4 is an exemplary diagram of applying the PTP method to a student network according to an embodiment.
5 is an exemplary diagram of a PTP label assignment method according to an embodiment.
6 is a flowchart illustrating a method of performing knowledge distillation for learning a student network according to an embodiment.
7 is an exemplary diagram of applying the SPS scheme to a student network according to an embodiment.

Hereinafter, various embodiments will be described in detail with reference to the accompanying drawings. The embodiments described below may be modified and implemented in various different forms. In order to more clearly describe the characteristics of the embodiments, detailed descriptions of matters widely known to those of ordinary skill in the art to which the following embodiments belong are omitted. In addition, in the drawings, parts irrelevant to the description of the embodiments are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a component is said to be “connected” with another component, it includes not only a case of 'directly connected' but also a case of 'connected with another component interposed therebetween'. In addition, when a component "includes" a component, it means that other components may be further included, rather than excluding other components, unless otherwise stated.

According to an embodiment of the present invention, knowledge distillation (KD) is used to learn a deep neural network (DNN). Knowledge distillation refers to a method of training a student network by transferring the generalization ability of a larger teacher network to a generally smaller student network. The teacher network provides hard-target information and soft-target information to the student network 30 so that the student network can learn to generalize similarly.

In the present invention, the neural network trained using the knowledge distillation method is a transformer-based neural network, and according to an embodiment, may be Bidirectional Encoder Representations from Transformers (BERT). At this time, the transformer follows the encoder-decoder structure of seq2seq, and is a model implemented only with attention without using RNN.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

1 shows an exemplary diagram for explaining a process of learning a DNN using knowledge distillation.

Referring to FIG. 1, the student model is converted into an SPS student model by applying the Shuffled Parameter Sharing (SPS) method, and the SPS student model is pre-trained through the PTP (Pretraining with Teacher's Predicitons) method, and PTP In this way, the pre-trained SPS student network 30 is trained to imitate the teacher network using knowledge distillation. Specific methods of the SPS method and the PTP method applied to the student network 30 will be described in detail with reference to other drawings below.

2 is a block diagram illustrating the configuration of the knowledge distillation apparatus 200 according to an embodiment.

Referring to FIG. 2 , the knowledge distillation apparatus 200 according to an embodiment may include an input/output unit 210 , a storage unit 220 , and a control unit 230 .

The input/output unit 210 according to an embodiment may include an input device for receiving an input from a user, and an output device for displaying information such as a result of a job or a state of the knowledge distillation device 200 . For example, the input/output unit 210 may include an input unit for receiving a data processing command and an output unit for outputting a result processed according to the received command. According to an embodiment, the input/output unit 210 may include a user input unit such as a keyboard, mouse, or touch panel, and an output unit such as a monitor or speaker.

Meanwhile, the storage 220 may store data for learning the student network 30 by performing knowledge distillation. For example, it is possible to store learning data for learning a neural network (teacher network 20, student network 30), and output data of the teacher network 20 required for knowledge distillation for learning the student network 30 can be saved In addition, various data or programs necessary for knowledge distillation for learning the student network 30 may be stored.

In addition, the controller 230 controls the overall operation of the neural network system 200 and may include a processor such as a CPU. In particular, the controller 230 may execute a program stored in the storage 220 or read data to perform knowledge distillation for learning the student network 30 . A specific method for the controller 230 to perform knowledge distillation for learning the student network 30 will be described in detail below with reference to other drawings.

Hereinafter, a specific method of performing knowledge distillation for learning the student network 30 will be described.

3 is a flowchart illustrating a method of performing knowledge distillation for learning the student network 30 according to an embodiment.

Referring to FIG. 3 , the controller 230 pre-trains the student network 30 in the PTP method ( S310 ). After step S310 , the controller 230 performs a knowledge distillation process on the teacher network 20 to learn the student network 30 ( S320 ).

Pretraining with Teacher's Predictions (PTP) method

Here, the PTP (Pretraining with Teacher's Predicitons) method is a method of initializing the student model before knowledge distillation. It can be seen as a 'specialized' initialization method for a kind of knowledge distillation. By the PTP method, the student network 30 may acquire additional knowledge about the teacher network 20 as well as a downstream task.

4 is an exemplary diagram of applying the PTP method according to an embodiment to the student network 30 .

Referring to FIG. 4, the PTP method is performed as follows. The controller 230 inputs the learning data 10 to the teacher network 20 and collects the result values of the teacher network 20 . Thereafter, the controller 230 assigns a PTP label to the learning data 10 based on a confidence defined based on the result value of the teacher network 20 . The controller 230 inputs the learning data 10 to the student network 30 and trains the student network 30 to predict the PTP label corresponding to the input. The controller 230 uses the learned student network 30 to predict the PTP label as an initial state of the knowledge distillation process.

Confidence is defined as follows. The largest value among the result values of the teacher network 20 is assumed to be confidence.

5 is an exemplary diagram of a PTP label assignment method according to an embodiment.

Specifically, looking at the PTP label assignment method shown in FIG. 5 , the control unit first divides the teacher network 20 into whether the teacher network 20 corrects the correct answer for the input data, and the control unit sets the confidence level to t (hyperparameter) in each case. A total of 4 types of labels are given to the input data by dividing them according to whether they are abnormal or not. In this case, the hyperparameter t preferably has a value between 0.5 and 1. For example, assuming that t is 0.6, the maximum value of the result value of the teacher network 20 for the input x is 0.7, and the teacher network 20 gives the correct answer to the input x, the controller 230 Since the maximum value of the result of the teacher network 20 for the input x is greater than t, it is possible to confidently assign a correct label to the input x.

Through the PTP method, the student network 30 can obtain additional information about the teacher network 20 in advance, and since the knowledge distillation process gives the knowledge of the teacher network 20 to the student network 30 , the teacher network 20 . The student network 30 , which has pre-trained the information on , can more efficiently receive the knowledge of the teacher network 20 .

A formula for the control unit 230 to pretrain the student network 30 using the PTP label is as shown in Equation 1 above.

[Equation 1]

는 학생 네트워크의 파라미터를 의미하고,

는 학습 데이터를 의미하고,

는 학생 네트워크의 출력값,

는 소프트맥스(softmax) 함수를 의미하고,

는 PTP 라벨을 의미하고, 첨자 s는 학생 네트워크를 의미하고,

는 크로스엔트로피 손실(cross-entropy loss)을 의미할 수 있다. here,

is the parameter of the student network,

is the training data,

is the output of the student network,

is a softmax function,

means PTP label, subscript s means student network,

may mean cross-entropy loss.

를 지식증류 시 학생 네트워크(30)의 초기 상태에 사용할 수 있다. 즉, 제어부는 학생 네트워크(30)의 파라미터가

인 상태로 지식증류 프로세스를 시작한다.When the pre-learning of the student network 30 in the PTP method is completed, the control unit

can be used for the initial state of the student network 30 during knowledge distillation. That is, the control unit determines that the parameters of the student network 30 are

Start the knowledge distillation process in the phosphorus state.

Referring to FIG. 6 , the process of pre-training the student network 30 in the PTP method in step S310 may be divided into steps S601 to S602 again.

The controller 230 may apply the SPS method to the layer of the student network 30 (S601).

SPS (Shuffled Parameter Sharing) method

Here, the SPS method is a method of increasing the model complexity of the student network 30 without increasing the number of parameters.

The SPS method is performed as follows. The control unit 230 causes the student network 30 to include a plurality of layer pairs that share parameters, and the control unit divides the layer pair in which the parameters are shared into two groups in the step of learning the student network 30, The first layer group uses parameters interchangeably with each other, and the second layer group uses the shared parameters as they are. In this case, the learning step includes a pretraining step and a learning step during knowledge distillation.

7 is an exemplary diagram of applying the SPS method to the student network 30 according to an embodiment.

Referring to FIG. 7 , it can be seen that there are a total of three layers in the student network 30 . The controller 230 adds a layer so that the student network 30 includes a plurality of layer pairs sharing parameters. In this case, the number of layers to be added does not have to be exactly the same as that of the existing layers. For example, when there are a total of 6 layers in the student network 30, only 3 layers sharing parameters may be added.

Referring to FIG. 7 , it can be seen that the lower layer 1 and the upper layer 1 share parameters with each other, and similarly, the layer 2 and the upper layer 2 and the lower layer 3 and the upper layer 3 also share the parameters with each other. The control unit divides the upper layers 1,2,3 into the first group and the bottom layers 1,2,3 into the second group, and then sets the first layer group with parameters Q (Query), K ( key), and the second layer group uses the parameters Q and K as they are.

Through this, the student network 30 may obtain a higher complexity and thus the performance (accuracy) of learning may be improved.

After step S601, the control unit 230 pre-trains the SPS student network in the PTP method (S602).

The control unit 230 applies the SPS method to the existing student network 30 and converts it to the SPS student network 30 , and pre-trains the SPS student network 30 with the PTP method. The student network 30 to which the SPS method and the PTP method are applied has significantly higher performance (accuracy) of learning by knowledge distillation than the existing student network 30 . At this time, since the SPS method and the PTP method are independent of each other, when they are applied together, the effects are combined and a greater performance improvement can be obtained.

After step S310 , the controller 230 performs a knowledge distillation process on the teacher network 20 to learn the student network 30 ( S320 ).

Before proceeding with the knowledge distillation process, the teacher network must be sufficiently trained with the learning data (10). For example, the teacher network may be a 12-layer Bert-base model.

The parameters of the learned teacher network are as in <Equation 2>.

[Equation 2]

는 선생 네트워크의 파라미터를 의미하고,

는 소프트맥스 함수(softmax function)를 의미하고,

는 학습데이터(10)를 의미하고,

는 선생 네트워크의 결과값을 의미하고,

는 참 라벨(true labels)을 의미하고,

는 크로스엔트로피 손실(cross-entropy loss)을 의미할 수 있다. <수학식 2>는 손실 함수(Loss function)을 나타내는 <수학식 4>에서 사용될 수 있다.here,

is the parameter of the teacher network,

means a softmax function,

means the learning data (10),

is the result of the teacher network,

means true labels,

may mean cross-entropy loss. <Equation 2> may be used in <Equation 4> representing a loss function.

Equations used for the progress of the knowledge distillation process are <Equation 3> and <Equation 4>.

[Equation 3]

는 네트워크의 출력값을 의미할 수 있다. <수학식 3>는 손실 함수(Loss function)을 나타내는 <수학식 4>에서 교사네트워크의 결과값을 소프트화하기 위해 사용될 수 있다.For knowledge distillation, the result value of the network is expressed as a soft value by <Equation 2>. T may mean the temperature of the sptmax function that softly adjusts the output value of the teacher network 20,

may mean an output value of the network. <Equation 3> may be used to soften the result value of the teacher network in <Equation 4> representing a loss function.

[Equation 4]

은 지식증류하여 학생 네트워크를 학습시키기 위해 필요한 손실함수(Loss function)을 의미하고,

와

는 각각 교사 네트워크(20)와 학생 네크워크(32)의 결과값을 의미하고,

는 페이션트 지식증류(patient KD)에서 사용되는 특정 레이어의 지수를 의미하고,

는 학생 네트워크의 파라미터를 의미하고,

는 학습데이터(10)를 의미하고,

는 소프트맥스 함수(softmax function)를 의미하고,

는 참 라벨(true labels)을 의미하고,

는 크로스엔트로피 손실(cross-entropy loss)을 의미하고,

는 쿨백-라이블러 발산손실(Kullback-Leibler divergence loss)을 나타내고,

는 k번째 레이어의 아웃풋 로짓(output logits)을 의미할 수 있고,

와

는 하이퍼파라미터를 의미할 수 있다.

와

는 각각 수학식<2> 및 <3>에 의해 정의된다.

is the loss function required to learn the student network by distillation of knowledge,

Wow

means the result values of the teacher network 20 and the student network 32, respectively,

is the index of a specific layer used in patient KD,

is the parameter of the student network,

means the learning data (10),

means a softmax function,

means true labels,

means cross-entropy loss,

represents the Kullback-Leibler divergence loss,

may mean the output logits of the k-th layer,

Wow

may mean a hyperparameter.

Wow

are defined by Equations <2> and <3>, respectively.

The control unit 230 learns the student network 30 through the loss function of Equation (4).

In addition, the terms “…unit” and “…module” described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software or a combination of hardware and software.

The term '~ unit' used in the above embodiments means software or hardware components such as field programmable gate array (FPGA) or ASIC, and '~ unit' performs certain roles. However, '-part' is not limited to software or hardware. '~' may be configured to reside on an addressable storage medium or may be configured to refresh one or more processors. Accordingly, as an example, '~' indicates components such as software components, object-oriented software components, class components, and task components, and processes, functions, properties, and procedures. , subroutines, segments of program patent code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables.

The functions provided in the components and '~ units' may be combined into a smaller number of elements and '~ units' or separated from additional components and '~ units'.

In addition, components and '~ units' may be implemented to play one or more CPUs in a device or secure multimedia card.

The knowledge distillation method according to the embodiments described with reference to FIGS. 3 to 7 may also be implemented in the form of a computer-readable medium for storing instructions and data executable by a computer. In this case, the instructions and data may be stored in the form of program codes, and when executed by the processor, a predetermined program module may be generated to perform a predetermined operation. In addition, computer-readable media can be any available media that can be accessed by a computer, and includes both volatile and nonvolatile media, removable and non-removable media. In addition, the computer-readable medium may be a computer recording medium, which is a volatile and non-volatile and non-volatile storage medium implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. It may include both volatile, removable and non-removable media. For example, the computer recording medium may be a magnetic storage medium such as HDD and SSD, an optical recording medium such as CD, DVD, and Blu-ray disc, or a memory included in a server accessible through a network.

In addition, the knowledge distillation method according to the embodiments described with reference to FIGS. 3 to 7 may be implemented as a computer program (or computer program product) including instructions executable by a computer. The computer program includes programmable machine instructions processed by a processor, and may be implemented in a high-level programming language, an object-oriented programming language, an assembly language, or a machine language. . In addition, the computer program may be recorded in a tangible computer-readable recording medium (eg, a memory, a hard disk, a magnetic/optical medium, or a solid-state drive (SSD), etc.).

Accordingly, the knowledge distillation method according to the embodiments described with reference to FIGS. 3 to 7 may be implemented by executing the above-described computer program by a computing device. The computing device may include at least a portion of a processor, a memory, a storage device, a high-speed interface connected to the memory and the high-speed expansion port, and a low-speed interface connected to the low-speed bus and the storage device. Each of these components is connected to each other using various buses, and may be mounted on a common motherboard or in any other suitable manner.

Here, the processor may process a command within the computing device, such as, for example, to display graphic information for providing a graphic user interface (GUI) on an external input or output device, such as a display connected to a high-speed interface. Examples are instructions stored in memory or a storage device. In other embodiments, multiple processors and/or multiple buses may be used with multiple memories and types of memory as appropriate. In addition, the processor may be implemented as a chipset formed by chips including a plurality of independent analog and/or digital processors.

Memory also stores information within the computing device. As an example, the memory may be configured as a volatile memory unit or a set thereof. As another example, the memory may be configured as a non-volatile memory unit or a set thereof. The memory may also be another form of computer readable medium, such as, for example, a magnetic or optical disk.

In addition, the storage device may provide a large-capacity storage space to the computing device. A storage device may be a computer-readable medium or a component comprising such a medium, and may include, for example, devices or other components within a storage area network (SAN), a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory, or other semiconductor memory device or device array similar thereto.

The above-described embodiments are for illustration, and those of ordinary skill in the art to which the above-described embodiments pertain can easily transform into other specific forms without changing the technical idea or essential features of the above-described embodiments. You will understand. Therefore, it should be understood that the above-described embodiments are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a dispersed form, and likewise components described as distributed may be implemented in a combined form.

The scope to be protected through this specification is indicated by the claims described below rather than the above detailed description, and should be construed to include all changes or modifications derived from the meaning and scope of the claims and their equivalents. .

10: learning data 20: teacher network
30: student network 200: knowledge distillation device
210: input/output unit 220: storage unit
230: control unit

Claims (10)

In a method of training a student network including at least one transformer neural network using knowledge distillation in a teacher network including at least one transformer neural network,
pre-training the student network in a PTP (Pretraining with Teacher's Predictions) method of initializing the student network based on a result value of the teacher network learned in advance by learning data; and
and performing a knowledge distillation process on the teacher network to train the student network. According to claim 1,
The step of pre-training the student network,
inputting the learning data into the teacher network, and collecting result values of the teacher network;
assigning a PTP label to the learning data based on a confidence defined based on a result value of the teacher network; and
The method further comprising: inputting the learning data into the student network to train the student network to predict a PTP label corresponding to the input. 3. The method of claim 2,
The confidence is
The method, which means the largest value among the output values of the teacher model. According to claim 1,
The step of pre-learning the student network,
causing the student network to include a plurality of layer pairs sharing a parameter; and
Among the layer pairs in which the parameters are shared, the first layer swaps the shared parameters of the query parameter and the key parameter, and the second layer keeps the query parameter and the key parameter as it is. A method comprising the step of applying the SPS (Shuffled Parameter Sharing) method used to the student network. In a knowledge distillation apparatus for learning a student network including at least one transformer neural network using knowledge distillation in a teacher network including at least one transformer neural network in,
a storage unit in which a program for performing knowledge distillation is stored; and
A control unit including at least one processor,
The control unit is
Pre-training the student network in a PTP (Pretraining with Teacher's Predictions) method that initializes the student network based on the result value of the teacher network learned in advance by learning data,
An apparatus for learning the student network by performing a knowledge distillation process on the teacher network. 7. The method of claim 6,
The control unit is
input the learning data into the teacher network, and collect the results of the teacher network,
A PTP label is given to the learning data based on a confidence defined based on the result value of the teacher network,
An apparatus for learning the student network to predict the PTP label corresponding to the input by inputting the learning data into the student network. 8. The method of claim 7,
The confidence is
The device, which means the largest value among the output values of the teacher model. 7. The method of claim 6,
The control unit is
let the student network include a plurality of layer pairs sharing a parameter,
Among the layer pairs in which the parameters are shared, the first layer exchanges the shared parameters of the query parameter and the key parameter, and the second layer uses the query parameter and the key parameter as it is. A device that applies a Shuffled Parameter Sharing (SPS) method to the student network. A computer-readable recording medium in which a program for executing the method according to claim 1 is recorded on a computer. It is performed by the knowledge distillation device, A computer program stored in a medium for performing the method according to claim 1.

Images