KR102102772B1 - Electronic apparatus and method for generating trained model thereof - Google Patents

Abstract

Disclosed is a method for generating a learning model. The method for generating a learning model includes receiving learning data, generating an asymmetric multi-task feature network having a parameter matrix of a learning model that allows the transfer of asymmetric knowledge between tasks and a feedback matrix for feedback connection from a task to a feature. , Calculating a parameter matrix of the asymmetric multi-task feature network using the input learning data so that the predetermined objective function is minimized, and using the calculated parameter matrix as a parameter of the generated asymmetric multi-task feature network. And-generating a task feature learning model.

Description

ELECTRONIC APPARATUS AND METHOD FOR GENERATING TRAINED MODEL THEREOF}

The present disclosure relates to an electronic device and a method for generating a learning model, and more particularly, to an electronic device and a method for generating a learning model capable of task learning in a state in which negative transfer is reduced by enhancing asymmetric knowledge transfer.

One area of machine learning is multi-task learning, which utilizes the transfer of knowledge between tasks using the association of different tasks.

There is a learning task grouping and overlap in multi-task learning (GO-MTL) model that allows common tasks to be shared and selected by learning common features as a representative form of multitask learning. This GO-MTL model can be seen as a nonlinear transformation added to a neural network with a single hidden layer.

However, the GO-MTL model exhibits poor performance than when it is independently trained, when the difference in difficulty between tasks is severe, and the incorrectly learned parameter information for a specific task is transferred to another task. This is called negative transfer and is an important task to be solved in multitask learning.

Recently, asymmetric multi-task learning (AMTL) model has been used to solve the above-mentioned negative transition.

In the AMTL model, unlike GO-MTL, each task does not go through a common feature and utilizes direct knowledge transfer from other task parameters. For example, the task of predicting a zebra (normal horse + stripes) was derived to be close to the linear combination of the parameters of the task predicting the normal horse and the parameter of the task predicting the tiger (stripes).

This method enables asymmetric knowledge transfer between tasks, unlike common feature learning. Even if each task is related to each other, the problem of the negative transfer of existing feature sharing based models by inducing the transfer of knowledge only from the easy task to the difficult task and preventing the transfer of knowledge from the difficult task to other tasks Will solve it.

However, the AMTL model is useful for solving negative transitions, but has the following problems.

First, AMTL does not assume an explicit feature, so it was not appropriate to apply it directly to a deep neural network.

Second, when the number of tasks increases, the memory required for learning and its time cost are not efficient because the size of the transition matrix between each task to be carried in the AMTL increases quadratically.

Third, rather than direct knowledge transfer between tasks, the common feature-based model better reflects the real case, and thus more consistent with human intuition.

Therefore, a new model that can be applied to deep neural networks while solving negative transitions is required.

Accordingly, an object of the present disclosure is to provide an electronic device capable of task learning in a state in which a negative transfer is reduced by strengthening an asymmetric knowledge transfer and a method for generating a learning model.

The method for generating a learning model of the present disclosure for achieving the above object includes receiving a training data, a parameter matrix of a learning model that allows the transfer of asymmetric knowledge between tasks, and a feedback matrix for linking feedback from tasks to features. Generating an asymmetric multi-task feature network having, calculating a parameter matrix of the asymmetric multi-task feature network using the input training data such that a predetermined objective function is minimized, and calculating the calculated parameter matrix. And using the generated asymmetric multi-task feature network as a parameter to generate an asymmetric multi-task feature learning model.

In this case, the preset objective function includes a loss function for the 'learning model that allows the transfer of asymmetric knowledge between tasks', an autoencoder term using the feedback matrix and deriving a nonlinear combination of task parameters and a parameter attenuation normalization term. It can contain.

Meanwhile, the preset objective function may be the following equation.

는 다중 레이어로 구성되는 비대칭 다중-태스크 특징 네트워크의 행렬이고,

은 상기 비대칭 다중-태스크 특징 네트워크의 마지막 레이어에 대한 가중치 행렬이고,

는 상기 피드백 행렬,

는

벡터의 t번째 행,

는 태스크 t에 대한 싱글 태스크 학습의 평균 유효성 손실이고,

,

는 비선형 함수, α,

,λ 각각은 각 항의 비중을 조절하기 위한 모델 파라미터이다. here,

Is a matrix of asymmetric multi-task feature networks composed of multiple layers,

Is a weighting matrix for the last layer of the asymmetric multi-task feature network,

Is the feedback matrix,

The

T row of vector,

Is the average effectiveness loss of single-task learning for task t,

,

Is a nonlinear function, α,

Each of, λ is a model parameter for controlling the specific gravity of each term.

Meanwhile, the asymmetric multi-task feature network may have a plurality of hidden layers.

Meanwhile, the electronic device according to an embodiment of the present disclosure has an asymmetric multiple having a memory in which learning data is stored, and a parameter matrix of a learning model that allows transfer of asymmetric knowledge between tasks, and a feedback matrix for feedback connection from a task to a feature. -Create a task feature network, calculate a parameter matrix of the asymmetric multi-task feature network using the stored learning data so that a predetermined objective function is minimized, and generate the calculated parameter matrix as the generated asymmetric multi-task feature And a processor that generates an asymmetric multi-task feature learning model using the parameters of the network.

In this case, the preset objective function includes a loss function for the 'learning model that allows the transfer of asymmetric knowledge between tasks', an autoencoder term using the feedback matrix and deriving a nonlinear combination of task parameters and a parameter attenuation normalization term. It can contain.

Meanwhile, the preset objective function may be the following equation.

는 다중 레이어로 구성되는 비대칭 다중-태스크 특징 네트워크의 행렬이고,

은 상기 비대칭 다중-태스크 특징 네트워크의 마지막 레이어에 대한 가중치 행렬이고,

는 상기 피드백 행렬,

는

벡터의 t번째 행,

는 태스크 t에 대한 싱글 태스크 학습의 평균 유효성 손실이고,

,

는 비선형 함수, α,

,λ 각각은 각 항의 비중을 조절하기 위한 모델 파라미터이다. here,

Is a matrix of asymmetric multi-task feature networks composed of multiple layers,

Is a weighting matrix for the last layer of the asymmetric multi-task feature network,

Is the feedback matrix,

The

T row of vector,

Is the average effectiveness loss of single-task learning for task t,

,

Is a nonlinear function, α,

Each of, λ is a model parameter for controlling the specific gravity of each term.

Meanwhile, the asymmetric multi-task feature network may have a plurality of hidden layers.

On the other hand, in a computer-readable recording medium including a program for executing a method for generating a learning model in an electronic device of the present disclosure, the method for generating a learning model includes receiving learning data and allowing asymmetric knowledge transfer between tasks. Generating an asymmetric multi-task feature network having a parameter matrix of the training model and a feedback matrix for feedback connection from task to feature, using the input training data to minimize a predetermined objective function to the asymmetric multi-task Calculating a parameter matrix of a feature network, and using the calculated parameter matrix as a parameter of the generated asymmetric multi-task feature network to generate an asymmetric multi-task feature learning model.

As described above, according to various embodiments of the present disclosure, a negative transition caused by a symmetrical effect of each task in a learning process for a feature can be solved, and it can be applied to a deep neural network.

1 is a block diagram showing a simple configuration of an electronic device according to an embodiment of the present disclosure;
2 is a block diagram showing a specific configuration of an electronic device according to an embodiment of the present disclosure;
3 is a view for explaining an asymmetric multi-text feature learning method according to the present disclosure;
4 is a diagram illustrating feedback reconstruction according to an embodiment of the present disclosure,
5 is a diagram showing a Deep-AMTFL according to an embodiment of the present disclosure,
6 is a view showing a layer of the AMTFL model viewed from an arbitrary base network,
7 is a diagram showing features and parameters learned with a composite data set;
8 is a view showing the results of quantitative evaluation of various learning models,
9 is a view showing a reduction in errors per task compared to STL,
10 is a diagram showing error reduction and learning time according to an increased number of tasks,
11 is a diagram showing experimental results of two models applied to a real data set;
12 is a diagram showing quantitative evaluation in a plurality of in-depth models,
13 is a view showing a result of a task unit for an AWA data set,
14 is a diagram illustrating a method for generating a learning model according to an embodiment of the present disclosure.

Terms used in this specification will be briefly described, and the present disclosure will be described in detail.

Terms used in the embodiments of the present disclosure, while considering the functions in the present disclosure, general terms that are currently widely used are selected, but this may vary depending on the intention or precedent of a person skilled in the art or the appearance of new technologies. . In addition, in certain cases, some terms are arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the corresponding disclosure. Therefore, the terms used in the present disclosure should be defined based on the meaning of the terms and the contents of the present disclosure, not simply the names of the terms.

Embodiments of the present disclosure may apply various transformations and may have various embodiments, and specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope of the specific embodiments, it should be understood to include all transformations, equivalents, or substitutes included in the disclosed spirit and scope of technology. In the description of the embodiments, when it is determined that the detailed description of the related known technology may obscure the subject matter, the detailed description is omitted.

Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms are only used to distinguish one component from other components.

Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, "includes." Or "composed." Terms such as intended to designate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, one or more other features or numbers, steps, operation, component, part, or It should be understood that the possibility of the presence or addition of these combinations is not excluded in advance.

In the exemplary embodiment of the present disclosure, the 'module' or the 'unit' performs at least one function or operation, and may be implemented by hardware or software, or a combination of hardware and software. In addition, a plurality of 'modules' or a plurality of 'units' may be integrated into at least one module except for a 'module' or 'unit', which needs to be implemented with specific hardware, to be implemented with at least one processor.

Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art to which the present disclosure pertains can easily implement them. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in order to clearly describe the present disclosure in the drawings, parts irrelevant to the description are omitted, and like reference numerals are assigned to similar parts throughout the specification.

Hereinafter, the present disclosure will be described in more detail with reference to the drawings.

1 is a block diagram illustrating a simple configuration of an electronic device according to an embodiment of the present disclosure.

Referring to FIG. 1, the electronic device 100 may include a memory 110 and a processor 120. Here, the electronic device 100 may be a PC, a notebook PC, or a server capable of data calculation.

The memory 110 may store learning data for training the learning model, and may store data for classification or recognition using the learning model.

The memory 110 may store a learning model that allows asymmetric knowledge transfer between tasks. Here, the learning model may be asymmetric multi-task learning (AMTL). Meanwhile, the learning model may be referred to as a network.

In addition, the memory 110 may generate an asymmetric multi-task feature learning model (ie, asymmetric multi-task feature learning) generated by the processor 120. The generated AMTFL model may have one hidden layer, , May be plural.

Also, the memory 110 may store a program necessary to perform learning model optimization.

The memory 110 is a storage medium and an external storage medium in the electronic device 100, for example, a removable disk including a USB memory, a storage medium connected to a host, a web server through a network, or the like. Can be implemented.

The processor 120 performs control for each component in the electronic device 100. Specifically, when a boot command is input from the user, the processor 120 may boot using an operating system stored in the memory 110.

The processor 120 may receive various parameters necessary for generating an asymmetric multi-task feature network through the manipulation input 140 through the manipulation input 140. Here, various parameters received may be hyperparameters or the like.

Upon receiving various information, the processor 120 may generate an asymmetric multi-task feature network having a parameter matrix of a learning model that allows the transfer of asymmetric knowledge between tasks and a feedback matrix for feedback connection from task to feature. Specifically, as shown in FIG. 4, a neural network having a feedback connection can be generated.

Here, the learning model that allows the transfer of asymmetric knowledge between tasks is a learning model based on the AMTL model.

The feedback matrix is for feedback connection from the task to the feature, and may be a feedback connection matrix for the feature space z.

Then, the processor 120 calculates a parameter matrix of the asymmetric multi-task feature network using the stored learning data so that the predetermined objective function is minimized. At this time, the processor 120 may use the stochastic gradient descent method to minimize the objective function.

Here, the objective function may include a loss function for the 'learning model allowing asymmetric knowledge transfer between tasks', an autoencoder term using a feedback matrix, and deriving a nonlinear combination of task parameters, and a parameter attenuation normalization term. It can be expressed as 7. Details of the objective function will be described later with reference to FIGS. 3 and 4.

Then, the processor 120 uses the calculated parameter matrix as a parameter of the generated asymmetric multi-task feature network to generate an asymmetric multi-task feature learning model.

In addition, the processor 120 may perform various processes such as vision recognition, speech recognition, and natural language processing using the generated learning model. Specifically, if the learning model is related to image classification, the processor 120 may classify the input image using the generated learning model and the input image.

As described above, the electronic device 100 according to the present disclosure can solve negative transitions caused by symmetrical effects of each task in the process of learning about features. Also, since the electronic device 100 according to the present disclosure generates a learning model in which negative transitions are solved, it is possible to generate a learning model similar to a real model.

Meanwhile, in the above, only a simple configuration constituting the electronic device has been illustrated and described, but in the implementation, various configurations may be additionally provided. This will be described below with reference to FIG. 2.

2 is a block diagram illustrating a specific configuration of an electronic device according to an embodiment of the present disclosure.

Referring to FIG. 2, the electronic device 100 may include a memory 110, a processor 120, a communication unit 130, a display 140, and an operation input unit 150.

Operations of the memory 110 and the processor 120 have been described with reference to FIG. 1, and duplicate descriptions are omitted.

The communication unit 130 may be connected to another electronic device and receive learning data from the other electronic device.

In addition, the communication unit 130 may receive information for processing using a learning model, and may transmit the processing results to other electronic devices. For example, if the corresponding learning model is a model for classifying an image, the communication unit 130 may receive an image to be classified from another electronic device and transmit information about the classification result to another electronic device.

The communication unit 130 is formed to connect the electronic device 100 with an external device, and is connected to a terminal device through a local area network (LAN) and an internet network, as well as a universal serial bus (USB). It is also possible to connect through a port or a wireless communication (eg, WiFi 802.11a / b / g / n, NFC, Bluetooth) port.

The display 140 displays various information provided by the electronic device 100. Specifically, the display 140 may display a user interface window for selecting various functions provided by the electronic device 100. Specifically, the corresponding user interface window may include an item for receiving a parameter necessary for generating a learning model.

The display 140 may be a monitor such as an LCD, CRT, OLED, or the like, and may be implemented as a touch screen capable of simultaneously performing the functions of the manipulation input unit 150 to be described later.

In addition, the display 140 may display information about test results using a learning model. For example, if the corresponding learning model is a model for classifying images, the display 140 may display a classification result for the input image.

The manipulation input unit 150 may receive various parameters necessary for generating a learning model from a user.

The manipulation input unit 150 may be implemented with a plurality of buttons, keyboard, mouse, or the like, and may also be implemented with a touch screen capable of simultaneously performing the functions of the display 140 described above.

On the other hand, in the illustration and description of FIGS. 1 and 2, although it has been described that the electronic device 100 includes only one processor, the electronic device may include a plurality of processors, and a general CPU as well as a GPU may be utilized. have.

Hereinafter, a learning method capable of solving the negative transfer problem will be described in detail.

Multi-task learning jointly trains to improve the generalization performance of multiple task predictors and allows multiple kinds of knowledge transfer at the same time. One of the important tasks in multi-task learning is a negative transition problem that accounts for the situation where an accurate predictor for an easy task is negatively affected by an inaccurate predictor of a more difficult task.

Asymmetric multi-task learning (AMTL), a recently introduced method, solves this negative transition problem by allowing asymmetric knowledge transfer between tasks through parameter normalization between tasks. In particular, AMTL learns directed graphs that determine the amount of knowledge transfer between tasks by specifying that task parameters for each task are also represented by sparse combinations of parameters for other tasks.

However, the delivery model between tasks is limited in many ways. First, in most cases, the task is somewhat relevant, but it has no strict causality. Therefore, it is more natural to assume that they are generated from a common tentative base set, rather than taking into account that they are generated from a set of related tasks, as assumed in the cross-task transfer model.

For example, when one task predicts whether a person has fatty liver and the other predicts diabetes, given the patient's health record, the two tasks are correlated, but not a clear cause and effect relationship. . Rather, it may be caused by common factors such as obesity.

In addition, AMTL does not scale well as the number of tasks in AMTL increases because the graph of knowledge transfer between tasks increases secondarily, so it tends to be inefficient and excessive when there are a large number of tasks such as large-scale classification. Sparsity helps reduce the number of parameters, but does not reduce the inherent complexity of the problem.

Finally, the transition model between tasks stores knowledge in the trained model parameters and their relationship graphs. However, sometimes it can be useful to save what you learn in an explicit expression format that can be used for other tasks, such as transfer learning.

Accordingly, the present disclosure is based on a multi-task feature learning approach aimed at learning a latent feature, which is one of the most common ways to share knowledge between tasks in a multi-task learning framework, and asymmetric knowledge transfer. It aims to strengthen and prevent negative metastasis. In particular, the task predictor with high reliability has a greater influence on shared function learning, while reducing the influence of the task predictor with low reliability so that there is little or no contribution to feature learning.

For this operation, the present disclosure intends to present an efficient model that can solve a problem of negative transition while being a common feature-based model. This model is called asymmetric multi-task feature learning (AMTFL) in that it has the advantages of AMTL and learns features at the same time.

The AMTFL model will be described below with reference to FIG. 3.

3 is a diagram for explaining an asymmetric multi-text feature learning method according to the present disclosure.

Referring to FIG. 3, it is assumed that there is a task for determining tigers, zebras, and hyenas, respectively, and that tasks share a common feature of 'stripes'.

At this time, the existing common feature-based multi-task learning model (a) treats tasks having different difficulties in the same way, and it is easy for a difficult hyena task to transmit wrong information to the striped feature. Specifically, hyenas may have objects without streaks, and they are often blurry.

Therefore, as the conventional AMTL model does not assume an explicit feature, an asymmetric knowledge transfer occurs from an easy task to a difficult task as shown in FIG. 3 (b). Accordingly, hyenas simply transfer knowledge, so they can effectively deal with negative transfer problems, but they do not constitute common features (stripes), so learning cannot be considered efficient.

To solve this problem, the AMTEL of the present disclosure uses a feature called stripes as shown in FIG. 3 (c) in common for all tasks, but additionally infers stripes again using the results learned from the tiger, zebra, which are easy tasks. Establish a feedback connection.

This additional feedback linkage creates a strong circular dependency between the easier task and the striped feature, thereby relatively weakening the effect of difficult tasks such as hyena on the striped feature.

This is explained by substituting this into the human perspective.

A person first learns the concept of stripes through various visual information, and then infers it back through easy concepts such as tigers and zebras. Then, the idea of stripes previously learned is reinforced through this inference.

Therefore, even if a difficult task, such as a hyena, refers to the notion of stripes, it does not exert much influence on features during learning. In terms of the asymmetric knowledge transfer direction, knowledge transfer such as [(tiger, zebra)-> stripes-> hyena] occurs, which can be seen as a knowledge transfer between tasks similar to the existing AMTL model. Of course, the model of the present disclosure is different in that it is based on a common feature like GO-MTL.

The AMTFL model according to the present disclosure is that the top layer of the network includes an original feedforward connection and an additional weight matrix for feedback connection, so that it naturally extends to a function learned in a deep neural network. And it allows asymmetric transmission from each task predictor to the bottom layer. Through this, the AMTFL model has the advantage of utilizing the state-of-the-art deep neural network model to benefit from the recent development of deep learning.

Hereinafter, the content of learning the asymmetric multi-task feature of the present disclosure will be described.

This asymmetric multi-task feature model is based on the basic idea of the AMTL model. The AMTL model is characterized by the fact that each task parameter is derived close to a linear combination of other task parameters, as described above.

로 표현한다. 여기서, T는 총 태스크의 개수,

는 태스크

의 데이터(인스턴스) 개수,

는 특징 차원 (feature dimension)을 의미한다. 그리고 X는 데이터 인스턴스들의 세트를 나타내고, yt는 대응하는 라벨을 나타낸다. First, given data

Expressed as Where T is the total number of tasks,

The task

Data (instances) count,

Means a feature dimension. And X represents a set of data instances, and yt represents a corresponding label.

On the other hand, the goal of multi-task learning is to train the learning model using all tasks (T) as in the general learning goal, as shown in Equation 1 below.

은 학습 데이터의 교차 엔트로피 손실(손실 함수),

는 태스크 t에 대한 모델 파라미터이고,

는

로 정의되는 열 방향 연결 행렬이다. 그리고 패널티(

)는 모델 파라미터(W)에 대한 특정 사전 가정을 강화하는 정규화 항이다. here

Is the cross-entropy loss of the training data (loss function),

Is the model parameter for task t,

The

It is a column-directed connection matrix defined by. And penalty (

) Is a normalization term that reinforces certain prior assumptions for the model parameter (W).

는

로 분해될 수 있다. One of the common assumptions is that there is a common set of tentative criteria across tasks. With this assumption, the matrix

The

Can be decomposed into

는 k로 표현되는 잠재적 기저 행렬이고,

는 기저 행렬과 선형적으로 결합하기 위한 계수 행렬이다. 따라서, 정규화 항

을 이용하면 멀티 태스크 학습은 다음과 같은 수학식 2로 표현될 수 있다. here

Is the potential basis matrix expressed in k,

Is a coefficient matrix for linearly combining the base matrix. Therefore, the normalization term

Using, multi-task learning may be expressed by Equation 2 below.

는 태스크 t에 대한 기저 가중치를 나타내는

의 t 번째 열 벡터, 즉

이다. 다른 관점에서,

은 특징 변환으로 생각할 수 있는바, 각 인스턴스 i에 대한 변화된 특징(

)을 이용하여 각 태스크 t에 대한 학습 파라미터

로 간주 할 수 있다. here

Denotes the basis weight for task t

T th column vector, i.e.

to be. From another point of view,

Can be thought of as a feature transformation, the changed feature for each instance i

) To learn the training parameters for each task t

Can be considered as

가 정규화되고,

가 희귀하도록

을 강화할 수 있다. As a special case of Equation 2, for example, in a Go-MTL model, element-wisely

Is normalized,

To be rare

Can strengthen.

은 직교되고,

는 (2,1) - 놈(norm)으로 정규화되도록 제한될 수 있다. As another example, in Equation 3

Is orthogonal,

(2,1)-can be limited to normalize to a norm.

가 다른 태스크 파라미터

의 희소한 조합으로 재구성된다는 가정에 기초한 멀티 태스크 학습의 사례이다.On the other hand, task relevance may be used without assuming that there is a common set of potential bases. AMTL is a parameter for each task

Different task parameters

It is an example of multitask learning based on the assumption that it is reconstructed with a rare combination of.

에서

의 가중치

는

로 재구성될 수 있으며, 이는 태스크

에서 t 로의 지식 전달의 양으로 해석될 수 있는 것으로 이해될 수 있다. 행렬

에는 대칭 제약이 없기 때문에보다 신뢰성 있는 태스크에서 덜 신뢰성 있는 태스크로 비대칭 지식 전달 방향을 배울 수 있게 된다. 이러한 점을 고려하여, AMTL을 다음과 같은 수학식 4를 통해 멀티 태스크 학습 문제를 풀 수 있다. In other words

in

Weight of

The

Can be reconfigured as a task

It can be understood that it can be interpreted as the amount of knowledge transfer from to t. procession

Because there is no symmetry constraint in, you can learn asymmetric knowledge transfer direction from more reliable task to less reliable task. Considering this, AMTL can solve the multi-task learning problem through Equation 4 below.

와

는 각 항의 비중을 조절하기 위한 모델 파라미터이다. 그리고,

는 각

의 연결(concatenation)이고,

는 태스크 간 전이행렬,

벡터는

의 각 행이다.

Wow

Is a model parameter for controlling the specific gravity of each term. And,

Each

Is the concatenation of

Is a transition matrix between tasks,

Vector is

Is each row.

의 대각 요소들은 모두

이다. 즉,

의 각 요소는 태스크

가

를 제외한 다른 태스크에 얼마나 많은 지식을 전이했는지를 나타내게 된다.At this time

All of the diagonal elements of

to be. In other words,

Each element of the task

end

It indicates how much knowledge has been transferred to other tasks except.

항의 역할이 각 태스크 파라미터가 다른 태스크 파라미터의 선형 결합으로 유도되도록 하는 것이다. In Equation 4 to be described above

The role of the term is to make each task parameter lead to a linear combination of the other task parameters.

On the other hand, Equation (3) has the drawback that a serious negative transition from a noisy and difficult task to a clean / easy task occurs because all tasks equally contribute to the potential base construction.

In addition, Equation 4 cannot be extended to suit the number of tasks, and has the disadvantage of not learning explicit features.

Therefore, hereinafter, an effective method of overcoming the limitations of the above-described two transition approaches and applying asymmetric knowledge transfer to a deep neural network while preventing negative transfer will be described with reference to FIG. 4.

4 is a diagram illustrating feedback reconstruction according to an embodiment of the present disclosure.

Referring to FIG. 4, L is a trained base matrix, and LS is a trained model parameter.

을 얻는다. 여기서

는 비선형 변환(구현시에는 recified linear unit(RELU)가 활용될 수 있다)이다. First, the base feature is reconstructed using the autoencoder framework. Specifically, new features using the same L as feature transformation

Get here

Is a non-linear transformation (a recified linear unit (RELU) can be used in implementation).

이다. 여기서 마지막 층의

는 실제 예측에 대한 비선형성(예 : 분류를 위한 softmax 또는 sigmoid 함수, 회귀(regression)의 경우 하이퍼 볼릭 탄젠트 함수(hyperbolic tangent function) 사용)이다. 이 재구성 항을 사용하여 다중 태스크 특징 학습 공식을 수학식 5와 같이 표현할 수 있다. And to get the model output, we learn st on the transformed features. The model A is then output to reconstruct the potential feature by introducing transform A back into the feature space. In other words,

to be. Here's the last floor

Is the nonlinearity of the actual prediction (e.g., a softmax or sigmoid function for classification, or a hyperbolic tangent function for regression). Using this reconstruction term, the multi-task feature learning formula can be expressed as Equation (5).

이라 표현한다. 이때,

은 각

에 곱해져 특징을 형성하는 잠재적 기저행렬이며,

는 임의의 비선형 함수를 나타낸다. 그리고

의 요소 개수를

라고 할 때,

는 태스크에서 특징으로의 피드백 연결을 위한 행렬이며, AMTFL 모델에서 가장 중요한 변수가 된다.

의 각 행 벡터

는 AMTL에서의

와 유사한 역할을 하며, 각 태스크

가 특징요소들에 얼마나 많은 지식을 전이했는지를 나타낸다.

는 네트워크 가장 마지막 부분에서 실제 예측을 위해 쓰이는 함수 (예를 들어 sigmoid, 혹은 softmax 함수)를 나타낸다.

는 각 항들의 비중을 조절하기 위한 모델 파라미터이다. AMTFL, unlike AMTL, assumes an explicit feature, so

It is expressed as At this time,

Silver angle

Is a potential base matrix multiplied by to form a feature,

Denotes any nonlinear function. And

The number of elements in

When I say,

Is a matrix for connecting feedback from task to feature, and is the most important variable in the AMTFL model.

Each row of vector

In the amtl

Plays a similar role to each task

Indicates how much knowledge has been transferred to the feature elements.

Denotes a function (eg sigmoid, or softmax function) used for actual prediction at the end of the network.

Is a model parameter for adjusting the specific gravity of each term.

Limits A, which adds to task difficulty for asymmetric learning transfer from task to feature. Therefore, an easy and reliable task predictor of latent features contributes more. The asymmetric multi-task feature learning model considering this point can be expressed as Equation 6 below.

If the above expression is replaced with an expression for multiple layers, Equation 7 is given.

는 레이어 L-1에서

로 표현되는 학습된 심층의 행렬이고,

은 마지막 레이어에 대한 가중치 행렬이고,

는 특징 공간 z에 대한 피드백 연결의 행렬이다. 여기서,

는

벡터의 t번째 행이다. here,

In layer L-1

Is a trained deep matrix represented by

Is the weight matrix for the last layer,

Is a matrix of feedback connections to feature space z. here,

The

It is the t-th row of the vector.

에 대한 유효 희소성은

이고, 여기서

는 태스크 t에 대한 싱글 태스크 학습의 평균 유효성 손실이고, 표시를 단순화하기 위하여

이다.

Effective scarcity for

And here

Is the average effectiveness loss of single-task learning for task t, to simplify the presentation

to be.

하나만으로는 오버피팅을 줄이기 위하여 실제 위험을 포착하기에는 불충분할 수 있다. 여기서 만약 몇 개의 태스크가 낮은 리스트를 갖는 경우, 그것은 낮은 희소성 및 잠재적 기저 구축에 더 많은 기여를 하게 된다. Note that

One alone may not be enough to capture the actual risk to reduce overfitting. Here, if a few tasks have a low list, it contributes more to low scarcity and potential base construction.

는 일반적인 손실 함수 일 수 있다. 즉, 일반적인 인스턴스(회귀 태스크에서는 스퀘어드 손실인

, 분류 태스크에서는 로직 손실인

)을 사용할 수 있다. 여기서

는 시그모이드 함수이다. Loss function in the previous equation

Can be a general loss function. In other words, a typical instance (squared loss in regression tasks

In the classification task, logic loss

) Can be used. here

Is a sigmoid function.

5 is a diagram showing a Deep-AMTFL according to an embodiment of the present disclosure.

Referring to Fig. 5, the gray scale of the last layer represents the amount of actual risk measurement, the arrow in the upper direction represents the feedback connection, and the arrow in the lower direction represents the feed forward step.

FIG. 6 is a diagram showing layers of an AMTFL model viewed from an arbitrary base network.

Referring to FIG. 6, the solid line and the dashed-dotted line represent the feed forward connection, and the dotted line and the elements represent feedback connection and knowledge transfer. The thickness of the line represents the amount of knowledge propagated to the feature part by different amounts of reliability and feedback connections.

Hereinafter, the effect of the deep asymmetric multi-task feature learning method according to the present disclosure will be described with reference to FIGS. 7 to 11.

Specifically, experimental results using a synthetic data set and experimental results using a real data set will be described.

First, the learning model used in the experiment using the synthetic data set will be described below.

The first is STL. STL is a linear single task learning model, a ridge regression model for regression analysis, a logistic regression model for classification, and a softmax regression model for multiclass classification.

The second is GO-MTL. GO-MTL is a feature-sharing MTL model, and different task predictors share a common tentative criterion.

The third is AMTL. AMTL is an asymmetric multi-task learning model, as described above, and performs knowledge transfer between tasks through parameter-based normalization.

The fourth is NN. NN is a simple feed forward neural network with a single hidden layer.

The fifth is multi-task NN. Multi-task NN is similar to NN, but it adjusts task loss by dividing the loss for each task by Nt. In this model, l1 normalization was applied to the last fully connected layer.

The sixth is AMTFL. Asymmetric multi-task feature learning according to the present disclosure.

7 is a diagram illustrating features and parameters learned with a composite data set.

First, in order to obtain the image of FIG. 7, six 30-dimensional real base matrices were first generated.

하기 용이하도록, 12개 태스크에 대한 파라미터를 생성하고, 노이즈 레벨(클린, 노이즈)에 기초하여 두 개의 그룹을 생성하기 위하여

를 구분하였다. 클린 태스크는

의 가우시안 노이즈 레벨을 갖고, 노이즈한 태스크는

갖도록 하였다. 각 노이즈한 태스크는

중에서 선택되고, 각 클린 태스크 파라미터는 선형적으로

에 결합하기 위하여 4개의 기저

의 2개의 출력으로 조합되었다. 따라서, 기저

는 클린 및 노이즈한 태스크 각각에 오버랩되었으며, 다른 기저는 각 그룹에서 독점적으로 사용되었다. First, Gaussian noise

To make it easy to create parameters for 12 tasks, and to create two groups based on the noise level (clean, noise)

Separated. Clean Task

With a Gaussian noise level of

To have. Each noisy task

Is selected from, and each clean task parameter is linearly

4 bases to join on

It was combined into two outputs. Therefore, the basis

Was overlapped with each of the clean and noisy tasks, and the other base was used exclusively in each group.

Then, four random learning / evaluation / task splits were generated for each group ({50/50/100 of clean task) and {25/25/100} of noise task.

정규화를 적용하였다. 그리고 AMTFL의 히든 레이어(hidden layer)에서 비선형성을 제거하였으며, 모든 하이퍼 파라미터는 개별적인 유효 세트에서 찾았다. 그리고 AMTL을 위하여 데이터 세트 특징에 기초하여 음수 아닌 제약 조건을 제거하였다. And, for easy comparison with AMTFL, all basic models were implemented with neural networks, and all models were modeled for better reconstruction of L.

Normalization was applied. In addition, nonlinearity was removed from the hidden layer of AMTFL, and all hyperparameters were found in individual valid sets. And for AMTL, non-negative constraints were removed based on data set characteristics.

Referring to FIG. 7, the first image 710 represents an actual base matrix, and the second image 720 represents an actual model parameter.

And, the third image 730 is a base matrix generated using the present disclosure, and the fourth image 740 represents model parameters generated using the present disclosure.

In addition, the fifth image 750 represents a base matrix generated using Go-MTL.

And the sixth image 760 is a model parameter generated using Go-MTL.

And the seventh image 770 is a model parameter result generated by learning in the AMTL, and the sixth image 780 represents a task transfer function matrix.

When the third image 730 and the fifth image 750 of FIG. 7 are compared, the base matrix 730 according to the present disclosure is actually the base matrix 710 than the base matrix 750 learned using Go-MTL. ).

In addition, when comparing the fourth image 740, the sixth image 760, and the seventh image 760, the actual model parameter 760 using GO-MTL and the model parameter 770 learned from AMTL It can be seen that the trained model parameter 740 according to the present disclosure is similar to the real model parameter 720 and contains the least amount of noise. This is because the transfer of knowledge from task to feature significantly helps learning.

Also, referring to the eighth image 780, it can be seen that there is no part where knowledge is transferred from a difficult task (tasks 7 to 12) to an easy task (tasks 1 to 6).

8 is a diagram showing the results of quantitative evaluation of various learning models.

Referring to FIG. 8, it can be confirmed that AMTFL according to the present disclosure is superior to conventional methods. Specifically, AMTFL according to the present disclosure has a relatively low error for both clean and noisy tasks, but it can be confirmed that GO-MTL has a higher error than STL for noisy tasks. This is judged to be due to a negative transition from a noisy task.

And even when compared to AMTL, it can be seen that the AMTFL according to the present disclosure is excellent. Specifically, even if it is assumed that data of each task is generated from the same set of bases, AMTL is difficult to find a meaningful connection between tasks in a specific composite data set.

9 is a diagram showing an error reduction per task compared to an STL.

Referring to FIG. 9, it can be seen that AMTFL effectively prevents negative metastasis and a greater improvement than AMTL while GO-MTL suffers from negative metastasis.

10 is a diagram showing error reduction and learning time according to an increased number of tasks.

Referring to FIG. 10, it can be confirmed that AMTFL according to the present disclosure is superior to AMTL in terms of error reduction and learning time. In particular, when the number of tasks increases, it can be confirmed that such excellence is obvious.

Hereinafter, the learning model and the actual data set used in the experiment using the actual data set will be described.

1) AWA-Attr: A set of classification data consisting of 30475 images, and the task predicts 85 binary features for each image describing a single animal.

2) MNIST: A standard data set for classification consisting of 60000 images for learning and 10000 images for tasks with 28x28 images displayed in decimal.

3) School: This is a set of regression data predicting the test scores of 15362 students in 139 schools. Each school's test score prediction is considered a task.

4) ImageNet-Room: A subset of the ImageNet data set, which divides 14140 images into 20 indoor scene classes.

11 is a diagram showing experimental results of two models applied to an actual data set.

Referring to FIG. 11, it can be confirmed that the AMTFL according to the present disclosure has better performance than the conventional method for most data sets.

Hereinafter, four in-depth models and experimental results are described.

1) DNN: general deep neural network with softmax loss without feedback connection.

2) Multitasking DNN: DNN with softmax loss for each task divided by Nt (number of positive instances for each class).

3) Deep-AMTL: Same as Multitask DNN, but the asymmetric multitask learning goal is to replace the original softmax loss.

4) Deep-AMTFL: Deep asymmetric multitask feature learning model of the present disclosure.

For in-depth model experiments, CIFAR-100, AWA, and ImageNet-Small were used as data sets.

And the Caffe framework was used to implement the existing model and the deep model of the present disclosure. In the case of DNN and multitasking DNN, the model was trained from the beginning, and the rest of the models were created in the Multitask DNN for quick learning.

에 대해 각 태스크에 대한 실제 리스크의 프록시로서

를 사용하였다. 구체적으로, 유효성 손실을 사용은 모델을 두 번 학습해야 하기 때문에 실용적이지 않기 때문이다. 유사하게 원칙적으로

도 교차 검증을 통하여 찾아야 하나,

가 충분히 크다는 불균형에 기초하여 수정된

을 사용하였다. 또한, 수학식 7에서의

과

와 같은 파라미터를 찾기 위하여 광범위한 하이퍼파라미터 검색을 사용하지 않고, 합리적인 숫자의 값을 설정하였다. Deep-AMTFL's task confidence term

As a proxy of the actual risk for each task

Was used. Specifically, using loss of effectiveness is not practical because you have to train the model twice. Similarly, in principle

Should be found through cross-validation,

Corrected based on the imbalance that is large enough

Was used. Also, in equation (7)

and

In order to find such parameters, a wide number of hyperparameters were not used, and reasonable numeric values were set.

12 is a diagram showing quantitative evaluation in a plurality of in-depth models.

Referring to FIG. 12, it can be seen that the Deep-AMTFL of the present disclosure surpasses the performance of other models including Deep-AMTL showing the effect of asymmetric multi-task feature learning in the deep learning framework.

13 shows the results of task units for the AWA data set.

Referring to FIG. 13, it can be confirmed that AMTFL has better performance than the existing CNN in 41 of 50 classes and does not deteriorate except for a few classes.

These results were mostly attributed to suppression of negative metastasis.

14 is a diagram illustrating a method for generating a learning model according to an embodiment of the present disclosure.

Learning data is input (S1410). Specifically, various data sets necessary for learning may be input, and the corresponding learning data may be pre-entered and stored at the time of implementation.

In addition, an asymmetric multi-task feature network having a parameter matrix of a learning model allowing asymmetric knowledge transfer between tasks and a feedback matrix for feedback connection from the task to the feature is generated (S1420). Specifically, as shown in FIG. 4, a neural network having a feedback connection can be generated.

Then, the parameter matrix of the asymmetric multi-task feature network is calculated using the input learning data so that the predetermined objective function is minimized (S1430). Specifically, a parametric matrix of the network may be calculated by minimizing an objective function such as Equation 7 using a stochastic gradient descent method.

An asymmetric multi-task feature learning model is generated using the calculated parameter matrix as a parameter of the generated asymmetric multi-task feature network (S1440).

Therefore, the learning model generation method according to the present embodiment can solve the negative transition caused by the symmetrical influence of each task in the learning process for the feature. In addition, the learning model generation method according to the present disclosure generates a learning model in which negative transitions are solved, so that a learning model similar to an actual model can be generated. The method for generating a learning model as shown in FIG. 14 may be executed on an electronic device having the configuration of FIG. 1 or 2, and may be executed on an electronic device having other configurations.

In addition, the learning model generation method as described above may be implemented as a program including an executable algorithm that can be executed on a computer, and the above-described program is stored and provided in a non-transitory computer readable medium. Can be.

A non-transitory readable medium means a medium that stores data semi-permanently and that can be read by a device, rather than a medium that stores data for a short time, such as registers, caches, and memory. Specifically, programs for performing the various methods described above may be stored and provided in a non-transitory readable medium such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

In addition, although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and the technical field to which the present disclosure belongs without departing from the gist of the present disclosure claimed in the claims. In addition, various modifications can be carried out by a person having ordinary knowledge, and these modifications should not be individually understood from the technical idea or prospect of the present disclosure.

100: electronic device 110: memory
120: processor 130: communication interface
140: display 150: operation input

Claims (9)

A method for generating a learning model in an electronic device,
Receiving learning data from the electronic device;
Generating, in the electronic device, an asymmetric multi-task feature network having a parameter matrix of a learning model allowing asymmetric knowledge transfer between tasks and a feedback matrix for feedback-to-feature feedback connection;
Calculating, in the electronic device, a parameter matrix of the asymmetric multi-task feature network using the input learning data so that a predetermined objective function is minimized; And
And generating, by the electronic device, an asymmetric multi-task feature learning model using the calculated parameter matrix as a parameter of the generated asymmetric multi-task feature network. According to claim 1,
The predetermined objective function,
A learning model generation method comprising a loss function for the 'learning model allowing asymmetric knowledge transfer between tasks', an autoencoder term using the feedback matrix and deriving a nonlinear combination of task parameters and a parameter attenuation normalization term. According to claim 1,
The predetermined objective function,
The following formula, how to create a learning model.

here,

Is a matrix of asymmetric multi-task feature networks composed of multiple layers,

Is a weighting matrix for the last layer of the asymmetric multi-task feature network,

Is the feedback matrix,

The

T row of vector,

Is the average effectiveness loss of single-task learning for task t,

,

Is a nonlinear function, α,

Each of, λ is a model parameter for controlling the specific gravity of each term. According to claim 1,
The asymmetric multi-task feature network is a method for generating a learning model having multiple hidden layers. In the electronic device,
A memory in which learning data is stored; And
Create an asymmetric multi-task feature network with a parameter matrix of the learning model that allows the transfer of asymmetric knowledge between tasks and a feedback matrix for feedback from task to feature, and use the stored training data to minimize a predetermined objective function And a processor for generating a parameter matrix of the asymmetric multi-task feature network, and using the calculated parameter matrix as a parameter of the generated asymmetric multi-task feature network to generate an asymmetric multi-task feature learning model. Electronic devices. The method of claim 5,
The predetermined objective function,
An electronic device comprising a loss function for the 'learning model allowing asymmetric knowledge transfer between tasks', an autoencoder term using the feedback matrix and deriving a nonlinear combination of task parameters and a parameter attenuation normalization term. The method of claim 5,
The predetermined objective function,
The following formula is an electronic device.

here,

Is a matrix of asymmetric multi-task feature networks composed of multiple layers,

Is a weighting matrix for the last layer of the asymmetric multi-task feature network,

Is the feedback matrix,

The

T row of vector,

Is the average effectiveness loss of single-task learning for task t,

,

Is a nonlinear function, α,

Each of, λ is a model parameter for controlling the specific gravity of each term. The method of claim 5,
The asymmetric multi-task feature network is an electronic device having a plurality of hidden layers. A computer readable recording medium comprising a program for executing a method for generating a learning model in an electronic device, the computer readable recording medium comprising:
The learning model generation method,
Receiving learning data;
Generating an asymmetric multi-task feature network having a parameter matrix of a learning model that allows the transfer of asymmetric knowledge between tasks and a feedback matrix for feedback connection from task to feature;
Calculating a parameter matrix of the asymmetric multi-task feature network using the input learning data so that a predetermined objective function is minimized; And
And using the calculated parameter matrix as a parameter of the generated asymmetric multi-task feature network to generate an asymmetric multi-task feature learning model.

Images