KR20180096469A - Knowledge Transfer Method Using Deep Neural Network and Apparatus Therefor - Google Patents

Abstract

An information transfer method using a deep neural network and an apparatus thereof are disclosed. The information transfer method according to an embodiment of the present invention includes the steps of: defining an information flow between layers of a first deep neural network already learned; and learning a second deep neural network to be learned using the defined information flow. In the defining step, a flow between characteristics between two layers is defined as the information flow and the flow is defined as a matrix using an inner product indicating directionality between two vectors. Accordingly, the present invention can transfer information to a new deep neural network.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an information transfer method using a deep neural network,

More particularly, the present invention relates to an information transfer method capable of transferring information to a new deep neural network using an information flow corresponding to a correlation between layers constituting a deep neural network, .

DNN (Deep Neural Network) has high performance in a wide variety of fields. We design DNNs for specific tasks and learn them. Most learned DNNs are used only for reasoning. However, this can not be said to have used all of the learned DNN capabilities.

One way to get more out of the already learned capabilities of the network is to extract useful information from the network and deliver it to the new network. This is called knowledge transfer technology, and many existing researchers have studied this technology.

An example of a conventional information transfer technique is to soften an output feature of an already learned network using a technique called Dark knowledge and to learn a new network as it is .

Another example of a conventional information transfer technique is to allow a new network to mimic the characteristics of a teacher network by providing a loss that makes the characteristics in the middle as well as the last- Respectively.

 However, it can be very difficult to make this existing technique, student DNN, to exactly match the characteristics generated by the teacher DNN, which can simply be mimicking the teach DNN .

Therefore, there is a need for a new information transfer technology.

Embodiments of the present invention provide an information transfer method and apparatus that can transfer (or transfer) information to a new deep neural network using an information flow corresponding to a correlation between layers constituting a deep neural network.

The information transfer method according to an embodiment of the present invention includes: defining an information flow between layers of a first deep neural network already learned; And learning a second deep neural network to be learned using the defined information flow.

The defining step defines a flow between the properties between the two layers as the information flow, and the flow can be defined as a matrix using an inner product representing the directionality between the two vectors.

The learning step may learn the information flow of the second deep neural network such that the information flow of the second deep neural network is the same as the information flow of the first deep neural network.

The learning step may learn the information flow of the second deep neural network using Euclidean loss so that the information flow of the second deep neural network is the same as the information flow of the first deep neural network.

And the learning step may perform the classification task using the classification loss with the weight of the second deep neural network for which the information flow is learned as an initial weight, thereby learning the second deep neural network.

The information transfer apparatus according to an embodiment of the present invention includes a definition unit that defines an information flow between layers of a first deep neural network that has been already learned; And a learning unit for learning a second deep neural network to be learned using the defined information flow.

The definition defines a flow between properties between two layers as the information flow, and the flow can be defined as a matrix using an inner product representing the directionality between the two vectors.

The learning unit may learn the information flow of the second deep neural network such that the information flow of the second deep neural network is the same as the information flow of the first deep neural network.

The learning unit may learn the information flow of the second deep neural network using an Euclidean loss so that the information flow of the second deep neural network becomes the same as the information flow of the first deep neural network.

The learning unit may learn the second deep neural network by performing a classification task using a classification loss whose initial weight is the weight of the second deep neural network for which the information flow is learned.

According to embodiments of the present invention, it is possible to transfer (or transfer) information to a new deep neural network using an information flow corresponding to a correlation between layers constituting the deep neural network.

According to the embodiments of the present invention, there is less limit and freedom than the existing methods, and the compression method is superior in performance to the conventional method.

According to embodiments of the present invention, learning speed can be improved in various fields such as image recognition, object recognition, discrimination and error detection using a deep neural network, and a smart device having a limited memory and computational power compared to a computer, , Smart phones, smart watches, and so on.

1 is a flowchart illustrating an operation of an information transfer method according to an embodiment of the present invention.
FIG. 2 shows an example of a process for defining an information flow.
3 illustrates an example of a process of learning DNN using an information flow.
Figure 4 shows an example of the results for the method of the present invention.
5 illustrates an exemplary view for explaining a method according to an embodiment of the present invention.
6 illustrates a configuration of an information transfer apparatus according to an embodiment of the present invention.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to or limited by the embodiments. In addition, the same reference numerals shown in the drawings denote the same members.

Embodiments of the present invention define a correlation between layers making up a deep neural network as an information flow and transfer (or transfer) information to a new deep neural network, e.g., a student DNN, using the information flow thus defined That is the point.

Here, the present invention can be applied to the case where the first DNN learned beforehand, for example, the correlation between the layers constituting the teach DNN, that is, the information flow between the layers constituting the second DNN, 2 DNN can be learned.

1 is a flowchart illustrating an operation of an information transfer method according to an embodiment of the present invention.

Referring to FIG. 1, an information transfer method according to an embodiment of the present invention defines an information flow between layers constituting a learned first DNN, for example, a teach DNN (S110).

Here, step S110 may define the flow between the characteristics between the two layers in the layers constituting the learned DNN, and the flow may be defined by calculating the dot product between the characteristics of the two layers. The flow may also be defined as a matrix using an inner product representing the directionality between two vectors for the properties of the two layers.

For example, in step S110, as shown in FIG. 2, a matrix indicating the information flow of the teach DNN using a feature map between the input and the output of the layer constituting the teach DNN, for example, a Gram Matrix Flow of Solving a Problem (FSP) Matrix allows you to define information flow. These defined information flows or matrices can be used to learn new DNNs to learn, for example student DNNs (transfer the distilled knowledge).

로 나타낼 수 있으며,

는 한 레이어의 i번째 채널 특성을 의미하고,

는 또 다른 레이어의 j번째 채널 특성을 의미할 수 있다.For example, assuming that the matrix representing the flow between two layers is G,

Lt; / RTI >

Denotes an i-th channel characteristic of one layer,

May mean the jth channel characteristic of another layer.

In the present invention, the flow of information from the input space to the output space of the teacher DNN becomes an important characteristic that can quickly learn the student DNN, and this information flow is a function between the layers constituting the teacher DNN It can also be defined as a correlation.

If the information flow of the teach DNN is defined in step S110, the second DNN, for example, the student DNN to be learned using the thus-defined information flow is learned (S120).

Here, in step S120, the information flow of the second DNN can be learned using the information flow of the first DNN so that the information flow of the second DNN is the same as the information flow of the first DNN.

In step S120 of the present invention, the second DNN may be learned by using the two-step loss, and the second DNN may be learned by using two losses of Euclidean loss and classification loss.

를 사용할 수 있다.That is, as shown in Fig. 3, step S120 is an example of a student DNN information flow, the matrix (G ^S), for an information flow example defined by the Teacher DNN, Euclidean so like all the rest of the matrix (G ^T) Ross ( Euclidean loss) can be used to learn information DNN's information flow. For example, assuming that the matrix of the learned learned DNN is G1, and the matrix of the student DNN is G2, Euclidean loss is set so that the two matrices are equal to each other.

Can be used.

Then, the student DNN learned by Euclidean loss can be learned using the classification loss. That is, in step S120, the class DN of the student DNN learned by Euclidean loss is used as the initial weight, and the classification task using the classification loss is performed to finally learn the student DNN, thereby rapidly transferring information from the teacher DNN to the student DNN can do.

The present invention can improve the performance of DNN using information transfer technology together with fast optimization technique.

Figure 4 shows an example of the results of the method of the present invention and shows the results when using a dataset of CIFAR-10.

Here, the data set is composed of 10 classes, and has a total of 50,000 training sets and a full test set. The teach network is a 26-layer residual network, Network, and the student network can use the same structure as the Teacher Network.

As shown in FIG. 4, when the information transfer method of the present invention is used, the matrix in FIG. 4 is extracted from three parts in which the spatial size of the characteristic is reduced. And it converges about three times faster.

Table 1 below shows the performance and ensemble performance of a twenty-two iteration of twenty-four (24) iterations of the residual network after each iteration, Performance.

Accuracy1 Accuracy2 Accuracy3 Ensemble Original 91.61 91.56 92.09 93.48 Original with 1/3 iter 90.47 90.83 90.62 92.6 Proposed method 92.28 92.24 92.07 93.26

As shown in Table 1, when the number of iterations is reduced by 1/3 times from the original number of 64000 iterations to about 21000 iterations, the performance is reduced by about 1.5%. On the other hand, We know that we can see the existing performance even though we have only rotated the ship's iteration. This indicates that the initial weight using the matrix of the information flow in the present invention is a good initial point.

Table 2 below shows the performance enhancement when information extracted from the network of the present invention is transferred to a shallow network in comparison with the existing method. The teach network is a residual network using 26 layers, and the student network has eight layers The results are shown in Fig. Here, the matrix can be extracted from a total of three parts where the spatial size of the characteristic is reduced.

Accuracy Original 87.9 Romero 88.34 Proposed 88.62

As can be seen from Table 2, the proposed method according to the present invention shows a performance increase of about 0.7% as compared with learning using the general technique (original).

As described above, the method according to the present invention is improved in performance compared to the conventional method, the network can be learned quickly, and it is useful for both a task requiring an ensemble of various networks and a task requiring a small network .

Hereinafter, the present invention will be described in more detail.

The information transfer method according to the present invention is to define important information of a first deep neural network, for example, a teach DNN, and to transmit distilled knowledge to another DNN, for example, a student DNN.

In the present invention Distilled  Information

DNN creates inter-layer properties. The upper layer properties are close to useful properties that can perform major tasks. If the inputs of the DNN are questioned and the output is the answer, the intermediate generated properties of the DNN can be conceived, and in this respect, the information transfer technique can be seen as simply imitating the intermediate result of the student DNN have. However, in the case of DNN, there are various ways to solve the problem of generating output from the input, and in this sense imitating the generated characteristics of the teach DNN can be a constraint on the student DNN.

In the case of a person, the teacher explains the process of solving the problem, and the student learns the process of solving the problem. Student DNN does not need to learn the intermediate output when a specific question is entered, but it can learn how to solve the problem when a specific type of question is encountered. In this way, the present invention can provide a better method than teaching intermediate results.

Distilled  Mathematical representation of information

는 두 레이어들의 특성들에 의해 생성된다. 선택된 레이어들 중 하나가 특성 맵

을 생성한다고 하면 다른 선택된 레이어는 특성 맵

를 생성한다. 여기서, h, w, m은 각각 채널의 높이, 폭과 수를 의미할 수 있다.The present invention can define the relationship between two intermediate results. In the case of DNN, the relationship can be mathematically considered by the orientation between the properties of the two layers. In the present invention, the flow (or flow) of the solution processor may be represented by an FSP matrix. FSP Matrix

Is generated by the properties of the two layers. One of the selected layers is a property map

, Then the other selected layer will have a property map

. Here, h, w, and m may denote the height, width, and number of the channel, respectively.

는 아래 <수학식 1>과 같이 계산될 수 있다.Then, the FSP matrix

Can be calculated as Equation (1) below.

[Equation 1]

Where x and W may denote the input image and weights of the DNN, respectively. In the present invention, a residual network having 8, 26, and 36 layers trained by a CIFAR-10 data set can be used. For a CIFAR-10 dataset with varying spatial sizes, the residual network may have three points, and the present invention may select multiple points to generate the FSP matrix as shown in FIG.

FSP  Ross of the Matrix

, 학생 네트워크에 의해 생성되는 n FSP 행렬을

라 가정하면, 본 발명은 같은 공간 사이즈를 가지는 티쳐 네트워크와 학생 네트워크 간 FSP 행렬의 쌍

을 고려한다. 각 쌍에 대한 비용 함수로 제곱된 L2 표준을 이용하며, 디스틸드 정보 이전에 대한 비용 함수는 아래 수학식 2와 같이 정의될 수 있다.To assist the student network, the present invention relocates or conveys the desk tilt information of the teach network to the student network, and as described above, the desk tilt information can be represented in the form of an FSP matrix containing information about the flow of the solution process have. The n FSP matrix generated by the texture network is denoted by

, The n FSP matrix generated by the student network

, The present invention provides a pair of FSP matrices between a teacher network and a student network having the same spatial size

. The L2 standard squared as a cost function for each pair is used, and the cost function for the prior art distinction information can be defined as shown in Equation 2 below.

&Quot; (2) &quot;

는 각 로스 텀에 대한 웨이트를 의미하고, N은 데이터 포인트들의 수를 의미할 수 있다.here.

Means the weight for each of the terms, and N can mean the number of data points.

를 사용한다.The present invention assumes that the entire roster is equally important. That is,

Lt; / RTI &gt;

learning Procedure

The information transfer method according to the present invention uses distilled information generated by a teach network. In order to clearly explain what the teacher network is in the present invention, the following two conditions are defined.

First, the teach network must be pre-trained by a preset data set. At this time, the dataset that trains the teacher network may be the same as or different from the dataset learned by the student network. The Teacher Network can use a data set that is different from the student network's data set for transition learning tasks.

Second, a teacher network may be deeper or worse than a student network, and in the detailed description of the present invention, the teacher network is described as being the same or deeper in depth than the student network.

The learning procedure includes two phases of training. First, the present invention minimizes the _loss function L _FSP to make the FSP matrix of the student network equal to the FSP matrix of the teach network. The student network after the first step is trained by the main task loss in the second step. In the present invention, the soft max cross entropy loss Lori can be used as a main task loss.

That is, the learning procedure of the present invention learns the FSP matrix as a first step by using Equation (3) below, and uses the following Equation (4) for the student network in which the FSP matrix is learned in the first step The second step, the original task, is trained.

&Quot; (3) &quot;

Here, Ws and Wt can denote the weights of the student network and the teach network.

&Quot; (4) &quot;

Deep residual networks can create an ensemble structure through a shortcut connection, and a shortcut connection can perform deeper network training.

FIG. 5 is a diagram illustrating an exemplary method for explaining a method according to an embodiment of the present invention. For example, a teacher network of 32 layers and a student network of 14 layers are illustrated. However, Can be changed.

As shown in FIG. 5, the FSP matrix can be extracted from three sections maintaining the same spatial size, and in the first step, the student network is trained such that the distance between the student network and the teach network is minimized, The pre-trained weights of student DNN are used as initial weights in the second step. The second step represents the normal training procedure.

That is, the deep residual network includes three sections and selects two layers to create the FSP matrix. There is no restriction on how to select the two layers. For example, you can select the first and last layers to create an FSP matrix. Because the FSP matrix is created by two layer properties with the same spatial size, the max pooling layer can be used to create the same spatial size if the two layer properties are of different sizes.

The size of the teacher DNN is larger than the size of the student DNN, and the student DNN can be configured by simply reducing the number of remaining modules in the teacher DNN. Thus, the student DNN may use parameters that are smaller than the teacher DNN.

FIG. 5 shows a configuration of an information transfer apparatus according to an embodiment of the present invention, and shows a configuration of an apparatus for performing the methods of FIGS. 1 to 4 described above.

Referring to FIG. 5, an information transfer apparatus 500 according to an embodiment of the present invention includes a definition unit 510 and a learning unit 520.

The definition unit 510 defines an information flow between the layers of the first deep neural network, for example, the learned DNN, which has been already learned.

Here, the definition unit 510 defines a flow between characteristics between two layers as an information flow, and a flow can be defined as a matrix using an inner product indicating a directionality between two vectors.

The learning unit 520 learns the second deep neural network, for example, the student DNN, which is to be learned by using the information flow of the teacher DNN defined by the defining unit 510. [

Here, the learning unit 520 can learn the information flow of the second deep neural network such that the information flow of the second deep neural network is the same as the information flow of the first deep neural network.

For example, the learning unit 520 can learn the information flow of the second deep neural network using the Euclidean loss so that the information flow of the second deep neural network becomes the same as the information flow of the first deep neural network, The second deep neural network can be learned by performing a classification task using a classification loss whose initial weight is the weight of the second deep neural network for which the information flow is learned.

Although the description is omitted in the apparatus of FIG. 5, it is to be understood that the apparatus of FIG. 5 may include all of the features described in FIGS. 1 through 4, It is obvious to those skilled in the art.

The system or apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the systems, devices, and components described in the embodiments may be implemented in various forms such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array ), A programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For ease of understanding, the processing apparatus may be described as being used singly, but those skilled in the art will recognize that the processing apparatus may have a plurality of processing elements and / As shown in FIG. For example, the processing unit may comprise a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of the foregoing, and may be configured to configure the processing device to operate as desired or to process it collectively or collectively Device can be commanded. The software and / or data may be in the form of any type of machine, component, physical device, virtual equipment, computer storage media, or device , Or may be permanently or temporarily embodied in a transmitted signal wave. The software may be distributed over a networked computer system and stored or executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

The method according to embodiments may be implemented in the form of a program instruction that may be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI &gt; or equivalents, even if it is replaced or replaced.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (10)

Defining an information flow between layers of the first deep neural network already learned; And
Learning the second deep neural network to be learned by using the defined information flow
/ RTI &gt;
The method according to claim 1,
The step of defining
The flow between the properties between the two layers is defined as the information flow,
The flow
Wherein the matrix is defined as a matrix using an inner product representing a directionality between two vectors.
The method according to claim 1,
The step of learning
Wherein the information flow of the second deep neural network is learned so that the information flow of the second deep neural network becomes the same as the information flow of the first deep neural network.
The method of claim 3,
The step of learning
Wherein the information flow of the second deep neural network is learned using Euclidean loss such that the information flow of the second deep neural network is the same as the information flow of the first deep neural network.
5. The method of claim 4,
The step of learning
Wherein the second deep neural network is learned by performing a classification task using a classification loss whose initial weight is a weight of a second deep neural network for which the information flow is learned.
A defining unit that defines an information flow between layers of the first deep neural network already learned; And
A learning section for learning a second deep neural network to be learned using the defined information flow;
Lt; / RTI &gt;
6. The method of claim 5,
The definition part
The flow between the properties between the two layers is defined as the information flow,
The flow
Wherein the matrix is defined as a matrix using an inner product indicating a directionality between two vectors.
The method according to claim 6,
The learning unit
And the information flow of the second deep neural network is learned such that the information flow of the second deep neural network becomes the same as the information flow of the first deep neural network.
9. The method of claim 8,
The learning unit
Wherein the information flow of the second deep neural network is learned using Euclidean loss such that the information flow of the second deep neural network is the same as the information flow of the first deep neural network.
10. The method of claim 9,
The learning unit
Wherein the second deep neural network learns the second deep neural network by performing a classification task using a classification loss whose initial weight is a weight of a second deep neural network for which the information flow is learned.

Images