KR102061935B1 - Knowledge Transfer Method Using Deep Neural Network and Apparatus Therefor - Google Patents

Abstract

Disclosed are a method and apparatus for transferring information using a deep neural network. Information transfer method according to an embodiment of the present invention comprises the steps of defining the information flow between the layers of the first deep neural network that has already been learned; And learning a second deep neural network to be trained using the defined information flow, wherein the defining step defines a flow between characteristics between two layers as the information flow, wherein the flow is two It can be defined as a matrix using dot products representing the directionality between vectors.

Description

Information transfer method using deep neural network and its device {Knowledge Transfer Method Using Deep Neural Network and Apparatus Therefor}

The present invention relates to a technology for transferring information, and more particularly, to an information transfer method and apparatus for transmitting information to a new deep neural network using an information flow corresponding to a correlation between layers constituting a deep neural network. It is about.

Deep Neural Networks (DNNs) have high performance in a wide variety of applications. You will design the DNN for a specific task and learn about it. Most learned DNNs are only used for reasoning. However, this does not mean that you have used up all the skills of a DNN that has already been learned.

One way to more fully use the capabilities of already learned networks is to extract useful information from the network and pass it on to the new network. This is known as knowledge transfer technology, and many existing researchers have studied this technology.

As an example of the conventional information transfer technology, the knowledge of the network has been learned to soften the output features of the already learned network by using a technique called dark knowledge to create a new network as it is. .

Another example of a conventional information transfer technique is to allow the new network to mimic the nature of a teaser network by giving a loss that not only makes the last output characteristic but also the middle characteristic the same. It was.

 However, it can be very difficult to have the student DNN, the existing technique, make exactly the characteristics that the teaser DNN generates, which is simply a mimic of the teaser DNN. .

Thus, there is a need for a new way of transferring information.

Embodiments of the present invention provide an information transfer method and apparatus capable of transferring (or transferring) information to a new deep neural network using an information flow corresponding to a correlation between layers constituting the deep neural network.

Information transfer method according to an embodiment of the present invention comprises the steps of defining the information flow between the layers of the first deep neural network that has already been learned; And training a second deep neural network to be trained using the information flow defined above.

The defining step defines the flow between the properties between the two layers as the information flow, and the flow may be defined as a matrix using a dot product representing the direction between the two vectors.

The training may include learning the information flow of the second deep neural network such that the information flow of the second deep neural network is the same as the information flow of the first deep neural network.

The learning may include learning the information flow of the second deep neural network using Euclidean loss such that the information flow of the second deep neural network is the same as the information flow of the first deep neural network.

The learning may be performed by training a second deep neural network by performing a classification task using a classification loss in which an initial weight is used as a weight of a second deep neural network learned from the information flow.

An apparatus for transferring information according to an embodiment of the present invention includes: a definer defining an information flow between layers of a first deep neural network that has been learned; And a learning unit learning a second deep neural network to be trained using the defined information flow.

The definition unit defines the flow between the properties between the two layers as the information flow, and the flow may be defined as a matrix using a dot product representing the direction between two vectors.

The learner may learn the information flow of the second deep neural network so that the information flow of the second deep neural network is the same as the information flow of the first deep neural network.

The learner may learn the information flow of the second deep neural network using Euclidean loss such that the information flow of the second deep neural network is the same as the information flow of the first deep neural network.

The learner may learn the second deep neural network by performing a classification task using a classification loss in which the weight of the second deep neural network learned from the information flow is used as an initial weight.

According to embodiments of the present invention, information may be transferred (or transferred) to a new deep neural network by using an information flow corresponding to a correlation between layers constituting the deep neural network.

According to embodiments of the present invention, it is less restrictive and free than existing methods, and has an excellent effect compared to the existing methods in terms of performance in compression.

According to embodiments of the present invention, a smart device that can improve learning speed in various fields such as image recognition, object recognition, discrimination, and error detection using a deep neural network, and has limited memory and computational power, as compared to a computer, for example You can expect higher performance than existing methods in smart phones, smart watches and smart watch.

1 is a flowchart illustrating an operation of a method for transferring information according to an embodiment of the present invention.
2 illustrates an example diagram for describing a process of defining an information flow.
Figure 3 shows an example for explaining the process of learning the DNN using the information flow.
4 shows an exemplary view of the results for the method of the present invention.
Figure 5 shows an exemplary view for explaining a method according to an embodiment of the present invention.
6 shows a configuration of an information transfer apparatus according to an embodiment of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the present invention is not limited or limited by the embodiments. Also, like reference numerals in the drawings denote like elements.

Embodiments of the present invention define a correlation between layers constituting a deep neural network as an information flow, and use the defined information flow to transfer (or transfer) information to a new deep neural network, for example, a student DNN. That's the point.

Herein, the present invention provides a correlation between the layers constituting the first DNN pre-learned, for example, the teaching DNN, that is, the information flow between the layers constituting the second DNN to learn the flow of information. 2 Can train DNN.

1 is a flowchart illustrating an operation of a method for transferring information according to an embodiment of the present invention.

Referring to FIG. 1, the information transfer method according to an embodiment of the present invention defines a flow of information between layers constituting a first deep neural network (DNN) that is already learned, for example, a teaching DNN (S110).

Here, step S110 may define a flow between the properties between the two layers in the layers constituting the learned DNN, and the flow may be defined by calculating the inner product between the properties of the two layers. Also, the flow may be defined as a matrix using a dot product representing the direction between two vectors for the properties of two layers.

For example, as illustrated in FIG. 2, step S110 is a matrix representing the flow of information of the teaching DNN using a feature map between inputs and outputs of the layers constituting the teaching DNN, for example, Gram Matrix or You can define the flow of information in the Flow of Solving a Problem (FSP) Matrix. The information flow or matrix defined in this way can be used to learn new DNNs to learn, for example, student DNNs.

로 나타낼 수 있으며,

는 한 레이어의 i번째 채널 특성을 의미하고,

는 또 다른 레이어의 j번째 채널 특성을 의미할 수 있다.For example, suppose a matrix representing the flow between two layers is G.

Can be represented by

Is the i-th channel characteristic of one layer,

May mean the j-th channel characteristic of another layer.

In the present invention, the flow of information from the input space of the teach-in DNN to the output space becomes an important characteristic to quickly learn the student DNN, and the flow of information between the layers constituting the teach-in DNN is provided. It can also be defined as a correlation.

If the flow of information of the teaching DNN is defined in step S110, a second DNN, for example, a student DNN, is trained using the information flow defined as described above (S120).

Here, step S120 may learn the information flow of the second DNN using the information flow of the first DNN so that the information flow of the second DNN becomes the same as the information flow of the first DNN.

In the present invention, step S120 can learn the second DNN using two levels of loss, and can learn the second DNN using two losses of Euclidean loss and classification loss.

를 사용할 수 있다.That is, as shown in Fig. 3, step S120 is an example of a student DNN information flow, the matrix (G ^S), for an information flow example defined by the Teacher DNN, Euclidean so like all the rest of the matrix (G ^T) Ross ( Euclidean loss can be used to learn the information flow of the teaching DNN. For example, assuming that the matrix of the pre-learned teacher DNN is G1 and the matrix of the student DNN is G2, Euclidean loss is made to equalize the two matrices.

Can be used.

The classification loss may be used to train the student DNN learned by Euclid Ross. That is, step S120 performs the classification task using the classification loss by using the weight of the student DNN learned by Euclidean as the initial weight, thereby making the final learning of the student DNN, and thereby quickly transferring information from the teacher DNN to the student DNN. can do.

The present invention can improve the performance of the DNN by using an information transfer technique together with a fast optimization technique.

Figure 4 shows an example of the results for the method of the present invention, showing the results when using the data set of CIFAR-10.

Here, the data set is composed of 10 classes, and has a total of 50,000 training sets and a full set of test sets, and the Teach Network is a 26-storey residual network. The network may be used, and the student network may use the same structure as the teacher network.

The matrix in FIG. 4 was extracted from a total of three parts of which the spatial size of the characteristic was reduced. As shown in FIG. 4, when the information transfer method of the present invention was used, the matrix was learned faster than the conventional technique. It can be seen that it converges about three times faster.

Table 1 below shows the performance and ensembles of each of the 26-layer residual networks turned to 64000 iterations, 21000 iterations, and 21000 iterations after learning using the invention (iteration 21000). It showed performance.

Accuracy1 Accuracy2 Accuracy3 Ensemble Original 91.61 91.56 92.09 93.48 Original with 1/3 iter 90.47 90.83 90.62 92.6 Proposed method 92.28 92.24 92.07 93.26

As shown in Table 1, in the case of the original method, when the number of iterations decreased by about one-third from 64000 iterations to 21000 iterations, the performance was reduced by about 1.5%. We know that we're still doing the same, even though we've only run the ship's iteration. This indicates that the initial weight using the matrix of information flow is a good initial point in the present invention.

Table 2 below shows the performance improvement when the information extracted from the network of the present invention is delivered to the shallow network, and the teaching network is a residual network using 26 layers, and the student network has 8 layers. The results used are shown. Here, the matrix may be extracted from three parts in which the spatial size of the characteristic is reduced.

Accuracy Original 87.9 Romero 88.34 Proposed 88.62

As can be seen from Table 2, the method according to the present invention (proposed) can be seen that the performance is increased by about 0.7% than when using the conventional techniques (original).

As described above, the method according to the present invention improves performance compared to the conventional method, the network can be learned quickly, and can be useful for both a task requiring an ensemble of several networks and a task requiring a small network. Can be.

The present invention will be described in more detail as follows.

The method for transferring information according to the present invention is to define important information of a first deep neural network, for example, a teaching DNN, and to convey distilled knowledge to another DNN, for example, a student DNN.

In the present invention Distill  Information

The DNN creates interlayer characteristics. Higher layer properties are close to useful properties that can perform key tasks. If the input of the DNN is the question and the output is the answer, then you can think of the characteristics that are generated in the middle of the DNN as intermediate results. In this sense, information transfer techniques can be seen as simply mimicking the intermediate results of the teacher DNN. have. However, in the case of DNN, there are various ways to solve the problem of generating output from input. In this sense, mimicking the generated characteristics of the teaching DNN can be a limitation for the student DNN.

In humans, the teacher explains the process of solving a problem, and the student learns the process of solving a problem. The student DNN does not need to learn the intermediate output when a specific question is entered, but can learn how to solve the problem when faced with a particular type of question. In this way the present invention can provide a better method than teaching an intermediate result.

Distill  Mathematical representation of information

는 두 레이어들의 특성들에 의해 생성된다. 선택된 레이어들 중 하나가 특성 맵

을 생성한다고 하면 다른 선택된 레이어는 특성 맵

를 생성한다. 여기서, h, w, m은 각각 채널의 높이, 폭과 수를 의미할 수 있다.The present invention may define a relationship between two intermediate results. In the case of the DNN, the relationship can be considered mathematically by the direction between the properties of the two layers. In the present invention, the flow (or flow) of the solution processor may be represented by an FSP matrix. FSP Matrix

Is generated by the properties of the two layers. One of the selected layers is the property map

If you create a different layer

Create Here, h, w and m may mean the height, width and number of the channels, respectively.

는 아래 <수학식 1>과 같이 계산될 수 있다.Then, the FSP matrix

May be calculated as in Equation 1 below.

[Equation 1]

Here, x and W may mean the input image and the weight of the DNN, respectively. In the present invention, it can be described using a residual network having 8, 26, 36 layers trained by the CIFAR-10 dataset. For a CIFAR-10 dataset whose spatial size varies, the residual network may have three points, and the present invention may select several points for generating an FSP matrix as shown in FIG.

FSP  Ross of the matrix

, 학생 네트워크에 의해 생성되는 n FSP 행렬을

라 가정하면, 본 발명은 같은 공간 사이즈를 가지는 티쳐 네트워크와 학생 네트워크 간 FSP 행렬의 쌍

을 고려한다. 각 쌍에 대한 비용 함수로 제곱된 L2 표준을 이용하며, 디스틸드 정보 이전에 대한 비용 함수는 아래 수학식 2와 같이 정의될 수 있다.In order to assist the student network, the present invention transfers or transmits the tilt information of the teaching network to the student network. As described above, the tilt information may be represented in the form of an FSP matrix including information on the flow of a solution process. have. N FSP matrices generated by the

, The n FSP matrix generated by the student network

In the present invention, the present invention provides a pair of FSP matrices between a student network and a student network having the same spatial size.

Consider. Using the L2 standard squared as the cost function for each pair, the cost function for the transfer of detiled information may be defined as in Equation 2 below.

[Equation 2]

는 각 로스 텀에 대한 웨이트를 의미하고, N은 데이터 포인트들의 수를 의미할 수 있다.here.

Denotes the weight for each loss term, and N may denote the number of data points.

를 사용한다.The present invention assumes that the entire loss term is equally important. In other words, the same

Use

learning Procedure

The information transfer method according to the present invention uses the de-tilt information generated by the teach network. In the present invention, two conditions are defined in order to clarify what a teaching network is.

First, the teaching network must be trained by a preset dataset. At this time, the data set for training the teacher network may be the same as or different from the data set learned by the student network. The teaching network can use a dataset different from the student network's dataset for transfer learning tasks.

Second, the teaching network may be deeper or larger than the student network. In the detailed description of the present invention, the teaching network is described as having the same depth or deeper than the student network.

The learning procedure involves two stages of training. First, the present invention minimizes the _loss function L _FSP to make the FSP matrix of the student network the same as the FSP matrix of the teacher network. In the first phase, the student network is trained by the main task loss. In the present invention, the soft task cross entropy loss Lori may be used as the main task loss.

That is, the learning procedure of the present invention learns the FSP matrix, which is the first step, by using Equation 3 below, and, for the student network where the FSP matrix is learned in the first step, uses Equation 4 below. Train the second step, the original task.

<Equation 3>

Here, Ws and Wt may refer to weights of the student network and weights of the teacher network.

<Equation 4>

A deep residual network may create an ensemble structure through a shortcut connection, and the shortcut connection may perform training of a deeper network.

FIG. 5 illustrates an exemplary view for explaining a method according to an embodiment of the present invention. Although the teaching network of 32 layers and the student network of 14 layers have been described as an example, the number of layers of the teaching network and the number of layers of the student network are illustrated. Can be changed.

As shown in FIG. 5, the FSP matrix can be extracted from three sections maintaining the same spatial size, in the first step the student network and the teacher network train the student network so that the distance between the FSP matrices is minimal, The pre-trained weights of the student DNN are used as initial weights in the second stage. The second stage represents the normal training procedure.

That is, the deep residual network includes three sections and selects two layers to create an FSP matrix, and there is no limit to the method of selecting the two layers. For example, you can select the first and last layer to create an FSP matrix. Since the FSP matrix is generated by two layer properties having the same spatial size, the max pooling layer can be used to make the same spatial size when the two layer properties have different sizes.

The size of the teaching DNN is larger than the size of the student DNN, and the student DNN can be constructed by simply reducing the number of remaining modules in the teaching DNN. Thus, the student DNN may use smaller parameters than the teacher DNN.

5 illustrates a configuration of an apparatus for transferring information according to an embodiment of the present invention, and illustrates a configuration of an apparatus for performing the method of FIGS. 1 to 4 described above.

Referring to FIG. 5, an apparatus 500 for transferring information according to an embodiment of the present invention includes a definer 510 and a learner 520.

The definition unit 510 defines a flow of information between layers of the first deep neural network that is already learned, for example, the TDN.

Here, the definition unit 510 defines a flow between the characteristics between the two layers as an information flow, and the flow may be defined as a matrix using a dot product representing the direction between the two vectors.

The learner 520 learns a second deep neural network, for example, a student DNN, to be trained by using the information flow of the teach-in DNN defined by the definer 510.

Here, the learner 520 may learn the information flow of the second deep neural network so that the information flow of the second deep neural network becomes the same as the information flow of the first deep neural network.

For example, the learner 520 may learn the information flow of the second deep neural network using Euclidean loss so that the information flow of the second deep neural network is the same as the information flow of the first deep neural network. The second deep neural network may be trained by performing a classification task using a classification loss in which the weight of the second deep neural network learned as the information flow is used as an initial weight.

Although the description of the apparatus of FIG. 5 is omitted, the apparatus of FIG. 5 may include all the contents described with reference to FIGS. 1 to 4, and may include all functions or contents related to information transfer. It is obvious to those skilled in the art.

The system or apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the systems, devices, and components described in the embodiments may include, for example, processors, controllers, arithmetic logic units (ALUs), digital signal processors, microcomputers, field programmable arrays (FPAs). ), A programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process and generate data in response to the execution of the software. For the convenience of understanding, a processing device may be described as one being used, but a person skilled in the art will appreciate that the processing device includes a plurality of processing elements and / or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as parallel processors.

The software may include a computer program, code, instructions, or a combination of one or more of the above, and configure the processing device to operate as desired, or process it independently or collectively. You can command the device. Software and / or data may be any type of machine, component, physical device, virtual equipment, computer storage medium or device in order to be interpreted by or to provide instructions or data to the processing device. Or may be permanently or temporarily embodied in a signal wave to be transmitted. The software may be distributed over networked computer systems so that they may be stored or executed in a distributed manner. Software and data may be stored on one or more computer readable recording media.

The method according to the embodiments may be embodied in the form of program instructions that may be executed by various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the media may be those specially designed and constructed for the purposes of the embodiments, or they may be of the kind well-known and available to those having skill in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks, such as floppy disks. Magneto-optical media, and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

Although the embodiments have been described by the limited embodiments and the drawings as described above, various modifications and variations are possible to those skilled in the art from the above description. For example, the described techniques may be performed in a different order than the described method, and / or components of the described systems, structures, devices, circuits, etc. may be combined or combined in a different form than the described method, or other components. Or even if replaced or substituted by equivalents, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and equivalents to the claims are within the scope of the claims that follow.

Claims (10)

Defining an information flow between layers of the first deep neural network already learned in the definition unit; And
Learning a second deep neural network to learn by using the information flow defined in the learning unit
Including;
The defining step
The flow between the properties between two layers is defined as the information flow,
The flow is
A method of transferring information defined by matrices using dot products representing the direction between two vectors.
delete delete Defining an information flow between layers of the first deep neural network already learned in the definition unit; And
Learning a second deep neural network to learn by using the information flow defined in the learning unit
Including;
The learning step
And information flow of the second deep neural network using Euclidean loss so that the information flow of the second deep neural network is the same as the information flow of the first deep neural network.
The method of claim 4, wherein
The learning step
And training the second deep neural network by performing a classification task using a classification loss in which the weight of the second deep neural network learned by the information flow is used as an initial weight.
A definition unit defining an information flow between layers of the first deep neural network that has already been learned; And
Learning unit for learning a second deep neural network to learn using the information flow defined above
Including;
The definition part
The flow between the properties between two layers is defined as the information flow,
The flow is
Information transfer device defined as a matrix using dot products representing the direction between two vectors.
delete delete A definition unit defining an information flow between layers of the first deep neural network that has already been learned; And
Learning unit for learning a second deep neural network to learn using the information flow defined above
Including;
The learning unit
And information flow of the second deep neural network using Euclidean loss so that the information flow of the second deep neural network becomes the same as the information flow of the first deep neural network.
The method of claim 9,
The learning unit
And training the second deep neural network by performing a classification task using a classification loss in which the weight of the second deep neural network learned by the information flow is used as an initial weight.

Images