KR20210134153A - Method and apparatus for deep knowledge transfer via prerequisite learning - Google Patents

Abstract

Disclosed are a method and device for a prior learning-based transfer learning. The method for performing prior learning-based transfer learning of a convolutional deep neural network according to one embodiment may comprise: a step of performing a prior learning in a source network; a step of performing a loss-based learning for a target task in a target network; a step of performing a soft label-based self-supervised learning for an auxiliary task to transfer-receive the data priorly learned from the source network; and a step of updating a learning parameter for the target task based on the learning for the auxiliary task. Therefore, the present invention is capable of having an advantage for which the learned prior knowledge can be transmitted naturally in a learning process.

Description

Prior learning-based transfer learning method and apparatus

The present invention relates to a prior learning-based transfer learning method and apparatus. More specifically, the prior learning-based transfer learning method according to an embodiment is a method and apparatus for enabling knowledge transfer between structurally complete heterogeneous networks because knowledge transfer for all target tasks is possible only with learning information of the source task. it's about

Transfer learning is the most representative method that can increase generalization efficiency by learning using a deep neural network with an insufficient amount of training data or a short learning time. Representatively, in the case of ImageNet data-based transfer learning using a CNN (convolutional neural network), the weight generated by learning data for image classification is used as an initial value for the target task to detect objects or semantic segmentation. It is used as a feature encoder, and has been very successfully applied in various computer image systems. However, in the case of transfer learning that uses a pretrained weight as an initializer, it is assumed that the pretrained weight has a global prior to learning, but in fact, prior knowledge beyond setting an appropriate initial value Questions have been raised as to whether it is involved in transfer learning.

Furthermore, in the case of transfer learning using a prior learning network, structural dependency is premised on sharing the fundamentally the same network structure. For this reason, transfer learning using pretrained weights provides a good initial value and is helpful for fast learning or learning using small data, but a question arises that it has no effect on knowledge transfer.

Another representative method of transfer learning using CNNs is the knowledge distillation method. The knowledge distillation method is a method in which a pre-trained teacher network distills knowledge hidden in inference results or learned features in various ways during learning of the student network. The knowledge distillation method is largely divided into a loss function-based knowledge distillation method that transmits dark knowledge through loss, or a method of transferring knowledge by increasing the similarity between features extracted at a specific stage of CNNs.

Transfer learning through this knowledge distillation has the advantage of being able to learn a student network that shows generalization performance comparable to that of the teacher network using fewer parameters, but has a data dependency (dataset) that the learning data of the teacher network and the student network must be the same. dependency) is assumed.

Therefore, there is a need for a deep learning learning methodology that is free from the structural dependence of the network and at the same time free from the data dependence by focusing on the knowledge transfer or transfer method between intelligent life forms that exist in nature.

Laid-open Patent Publication No. 10-2019-0138238, 2019.12.12.

The problem to be solved by the present invention is It is to propose a learning methodology that can transfer knowledge without additional supervision from other tasks. Prior learning-based transfer learning method and apparatus according to an embodiment is self-supervision as an auxiliary task for a target task to be learned using a source network previously learned from other tasks and a structure that can improve generalization performance by transferring previously learned knowledge through auxiliary tasks to the learning process of the target task is disclosed.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

A method for performing transfer learning based on prior learning of a convolutional deep neural network according to an embodiment of the present invention for solving the above-described problem includes: performing prior learning in a source network; performing loss-based learning on a target task in a target network; performing soft-label-based self-supervised learning on an auxiliary task to transfer previously learned data from the source network; and updating the learning parameter for the target task based on the learning for the auxiliary task.

In addition, the learning of the auxiliary task comprises configuring a multi-task network that simultaneously updates the loss of the target task and the loss of the auxiliary task so as to receive dark knowledge about the data learned from the source network. can induce

In addition, the auxiliary task includes a transition module consisting of a convolution block,

Additional features for the auxiliary task may be extracted through the knowledge transfer module.

In addition, the total loss function of the multi-task network including the target task and the auxiliary task is expressed as the following equation,

는 타깃태스크의 손실,

는 보조태스크의 손실,

는 미리 학습된 소스 네트워크에 대한 소프트맥스 출력을 의미할 수 있다.where i is the placement of ith on the training data,

is the loss of the target task,

is the loss of auxiliary tasks,

may mean a softmax output for the pre-trained source network.

In addition, at least one of cross entropy, focal, and knowledge distillation methods may be used for learning the auxiliary task.

In addition, the source network will be composed of at least one or more multi-task, the method.

Also, the source network and the target network may have different modalities or different problem domains of tasks.

Prior learning-based transfer learning apparatus according to another disclosure of the present invention, one or more processors; and one or more memories storing instructions that, when executed by the one or more processors, cause the one or more processors to perform an operation, wherein the operation performed by the processor includes: an operation for performing prior learning in a source network; an operation for performing loss-based learning on a target task in a target network; an operation for performing soft label-based self-supervised learning on an auxiliary task in order to transfer the data previously learned from the source network; and an operation of updating the learning parameter for the target task based on the learning for the auxiliary task.

According to an embodiment of the present invention, unlike the conventional transfer learning, there is an advantage that prior knowledge learned from various tasks can be naturally transmitted in the learning process because it is not limited by the network structure to be learned.

In addition, generalization performance can be improved in most datasets through task transfer learning between classification tasks. For example, the performance of the motion classification network can be improved by utilizing the prior learning network learned in the image classification task. Also, it is possible to improve the efficiency of knowledge transfer by adding a CNN-based transfer module for efficient task transfer learning.

Effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a diagram schematically illustrating a prior learning-based transfer learning method according to an embodiment.
2 is a flowchart illustrating a prior learning-based transfer learning method according to an embodiment.
3 is a diagram for explaining various embodiments of a transfer learning method based on prior learning according to an embodiment.
4 is a diagram conceptually illustrating a transition module according to an embodiment.
5 to 10 are diagrams for explaining the effect of the prior learning-based transfer learning method according to an embodiment.
11 is a block diagram schematically illustrating an internal configuration of an apparatus for transfer learning based on prior learning according to an embodiment.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only these embodiments allow the disclosure of the present invention to be complete, and those of ordinary skill in the art to which the present invention pertains. It is provided to fully understand the scope of the present invention to those skilled in the art, and the present invention is only defined by the scope of the claims.

The terminology used herein is for the purpose of describing the embodiments and is not intended to limit the present invention. As used herein, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, “comprises” and/or “comprising” does not exclude the presence or addition of one or more other components in addition to the stated components. Like reference numerals refer to like elements throughout, and "and/or" includes each and every combination of one or more of the recited elements. Although "first", "second", etc. are used to describe various elements, these elements are not limited by these terms, of course. These terms are only used to distinguish one component from another. Accordingly, it goes without saying that the first component mentioned below may be the second component within the spirit of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used herein will have the meaning commonly understood by those of ordinary skill in the art to which this invention belongs. In addition, terms defined in a commonly used dictionary are not to be interpreted ideally or excessively unless specifically defined explicitly.

1 is a diagram schematically illustrating a prior learning-based transfer learning method according to an embodiment.

Referring to FIG. 1 , the prior learning-based transfer learning method according to an embodiment has a basic structure of pre-task learning using an input 101 as a source network 103 . The target network 102 receives the soft label-based self-guidance of the source network 103 to reflect the knowledge of the source network 103 to the weight update while learning through the loss of the target task and to the previously learned data. Induces them to receive dark knowledge about Here, the dark knowledge of the source network 103 may be transmitted through the co-task loss 106 of the target network 102 to affect generalization performance. In addition, the auxiliary task may include a transfer module 105 composed of a convolutional block, and extract additional features for the auxiliary task through the knowledge transfer module to improve the learning ability through the loss of the auxiliary task 106 . have.

Therefore, the prior learning-based transfer learning method according to an embodiment does not depend on the structure of the network, unlike the existing transfer learning that uses the existing previously learned weight as a simple initial value. Therefore, even in a network learned in any structure, knowledge transfer for all target tasks is possible as long as there is learning information (the number of classes) of the source task to be transmitted. For example, knowledge transfer may be possible between structurally complete heterogeneous networks, such as knowledge transfer from 2D-CNN to 3D-CNN.

In addition, the transfer learning method based on prior learning according to an embodiment can achieve transfer learning by utilizing a source network learned from a completely different type of data, unlike a data-dependent transfer learning method such as a knowledge distillation method. Therefore, knowledge transfer may be possible even in heterogeneous datasets such as image and video datasets.

On the other hand, for convenience of explanation, the difference between the transfer learning method based on prior learning based transfer learning and the conventional transfer learning method according to an embodiment will be summarized once more below and will be described in more detail with reference to FIGS. 2 to 11 . do.

(1) Difference from the method of using pretrained initializer

As the most commonly used method in transfer learning using deep neural networks, models with high performance on large-scale training data have emerged, and are being utilized in most image recognition fields. Since there is a belief that low-level feature information is shared in most image recognition problems, fine-tuning is often applied by updating only a specific upper layer when learning a new task. In general, it can be used in the same network structure, and in many image recognition applications, only the feature encoder part of the entire network structure is used as a source network for prior learning. Transfer learning using ImageNet has been widely used until now, but recently, taking object detection and key point detection as an example, the pre-learned initial values do not have comprehensive prior knowledge, but are generalized by using good initial values. It has been found to help with ability. That is, there are experimental results that the method using the pre-learned initial value helps to have a short learning curve or is efficient in learning a small amount of data. In the case of transfer learning based on prior learning according to an embodiment, the source network learned in advance is used, but unlike transfer learning using the initial value learned in advance, inference information of the source network learned in advance is used as a soft label. There is a difference.

(2) Difference from knowledge distillation technique

In the case of the knowledge distillation method, a pre-trained teacher network with a large number of parameters performs knowledge transfer in the learning process of a student network with a relatively small number of parameters, thereby improving the generalization performance of the student network or generalizing it more than the teacher network. It can train a high-performing student network. For effective transfer learning using the knowledge distillation technique, the relationship between inferred information is utilized or the dataset is redefined as a subclass of the class to be learned and used for knowledge distillation. In addition, high recognition performance is achieved by constructing the same structure of teacher and student networks and creating an ensemble model of student networks trained by knowledge distillation with different initial values. The prior learning-based transfer learning method according to an embodiment was also inspired by the knowledge distillation technique using a loss, but in that it transfers new knowledge through the self-learning technique and performs transfer learning through the auxiliary task, it is There is a difference from the knowledge distillation method of

(3) Difference from auxiliary learning method

The auxiliary learning method is a type of multi-task learning, and the multi-task learning has the goal of achieving all of the plurality of tasks that the deep neural network wants to perform with high performance, but in the prior learning-based transfer learning method according to an embodiment, There is a difference in that the auxiliary task aims to improve the performance of the main task. Recently, by analyzing the cosine similarity of the slope of the loss occurring in the main task and the auxiliary task, the cause of the influence of the auxiliary task on learning is also analyzed. The transfer learning method based on prior learning according to an embodiment also has a basic structure of the learning methodology similar to the meta-learning method using auxiliary tasks, but unlike learning using general auxiliary tasks, additional guidance is not required, and a part of the network is not required. do not share Also, there is a difference in that updates to the source network are not made during training.

(4) Differences from self-supervised learning methods.

In a typical self-learning technique, pretext task unsupervised learning is performed to obtain a good weight as a pre-learned initial value. However, the prior learning-based transfer learning method according to an embodiment also has similarities in that the learning of the auxiliary task is performed by using only the soft label of the source network without requiring an additional label when learning the auxiliary task.

Knowledge obtained in transfer learning according to an embodiment is not obtained by unsupervised learning, and it is different from self-learning in that target task learning requires meta information such as the number of classes used in previous learning. But

(5) Difference from domain adaptation or expansion method.

In the case of transfer learning of prior learning according to an embodiment, it has a characteristic similar to domain adaptation or expansion in that it improves generalization performance to new tasks with the previously learned network. However, unlike the general domain adaptation method, there is a big difference in that information learned from the previously learned domain is used for new network learning without sharing parameter information about the source network previously learned. Additionally, since any update process for the source network is not included, the update is performed regardless of the performance of the source domain, and there is no change in the inference performance of the source domain.

2 is a flowchart illustrating a prior learning-based transfer learning method according to an embodiment.

Referring to FIG. 2 , in step S210, the transfer learning method according to an embodiment may perform prior learning in a source network.

Here, the source network may be a network of a domain completely different from the target network.

In step S220, in the transfer learning method according to an embodiment, loss-based learning is performed on the target task in the target network.

In step S230, the transfer learning method according to an embodiment performs soft-label-based self-supervised learning on the auxiliary task in order to receive the data previously learned from the source network. For example, by configuring a multi-task network that simultaneously updates the loss of the target task and the loss of the auxiliary task, it is possible to induce the transmission of dark knowledge about data learned from the source network.

The auxiliary task includes a transition module composed of a convolution block,

Through the knowledge transfer module, additional features for auxiliary tasks can be extracted. Therefore, the performance of the auxiliary task can be improved through the transition module.

In step S240, the transfer learning method according to an embodiment may update the learning parameter for the target task based on the learning for the auxiliary task. For example, loss-based learning for a target task and loss-based learning for an auxiliary task may be performed simultaneously.

For example, the transfer learning method may obtain feature information of the source network and update the gradient by correcting the gradient in a direction that can help the generalization performance of the target network.

3 is a diagram for explaining various embodiments of a transfer learning method based on prior learning according to an embodiment.

Figure 3 (a) is for a detailed description of the transfer learning method in a single source network, Input (301), Target (302), Source (303), Target loss (304), Transfer module (305) and Auxiliary The loss 306 may correspond to the input 101, the target network 102, the source network 103, the target task loss 104, the transition module 105 and the auxiliary task loss 106 described above in FIG. 1, respectively. have.

로 나타내면, 이때 는 입력,

은 타깃 네트워크의 파라미터,

은 타깃 데이터셋을

는 학습하고자 하는 태스크의 종류를 나타낸다.

는 주된 태스크를 해결하기 위해 학습 도중 타깃 태스크 와 보조 태스크의 손실을 통해 동시에 업데이트 될 수 있다.In the prior learning-based transfer learning method according to an embodiment, a multi-task network for auxiliary task learning

If expressed as, in this case is the input,

is the parameter of the target network,

is the target dataset

indicates the type of task to be learned.

can be updated simultaneously through the loss of the target task and the auxiliary task during learning to solve the main task.

는 입력 를 받아 타깃 네트워크에 지식을 전달해줄 소스 네트워크를 나타내며

는 소스 데이터셋(ImageNet, Places, 등.)

에 대해 소스 태스크

에 의해 학습된 파라미터를 의미한다.

는 학습 도중에 업데이트 되지 않는다. 주된 태스크 학습을 위한 다중 태스크 네트워크 h_t는 컨볼루션 블록의 최상위 레이어까지 공유한다. 이때 타깃 태스크의 손실 연산을 위한 브랜치의 마지막 특징 레이어는

을 출력하게 되며, 보조 태스크 손실 연산을 위한 브랜치는 마지막 특징 레이어

으로 출력 된다. 이때 보조 태스크에 대한 추가적인 특징 추출을 위해 컨볼루션 블록으로 구성된 전이 모듈을 가질 수 있다

represents the source network that will receive the input and transfer the knowledge to the target network.

is the source dataset (ImageNet, Places, etc.)

About the source task

It means the parameters learned by

is not updated during training. The multi-task network h _t for main task learning shares up to the top layer of the convolution block. At this time, the last feature layer of the branch for the loss calculation of the target task is

, and the branch for the auxiliary task loss calculation is the last feature layer.

is output as In this case, it may have a transition module composed of convolution blocks for additional feature extraction for auxiliary tasks.

For example, FIG. 4 is a diagram conceptually illustrating a transition module according to an embodiment.

의 예시를 보여주고 있다. 또한, 도4의 (c)는 멀티플 소스 네트워크를 이용할때의 활용되는 전이모듈(402)의 예시를 도시하고 있다.Fig. 4 (a) shows the structure of a multi-task network using assisted learning without a transition module, and Fig. 4 (b) is a multi-task network including a transition module 401.

shows an example of Also, FIG. 4C shows an example of the transition module 402 utilized when using a multiple source network.

에 속한 에 대해

으로 주어지며 보조 태스크에 대한 지상 진실 레이블은 h_s의 소프트맥스 출력인

으로 주어진다. Referring back to Fig. 3 (a), the ground truth label of the target task for the total loss calculation of the multi-task network is

about belonging to

given by , and the ground truth label for the auxiliary task is the softmax output of _{h s .}

is given as

At this time, the total loss function for the single task transition based on the multi-task network is given by the following Equation 1:

[Equation 1]

배치를 의미하며

는 미리 학습된 소스 네트워크에 대한 소프트맥스 출력으로 얻어지며 미리 학습된 데이터셋에 대한 암흑 지식을 소프트 레이블로써 전달하게 된다. 예를 들면, 효율적인 보조 태스크 학습을 위해 [수학식2]와 같이 보조 태스크에 대한 교차 엔트로피(

), 포칼 (

) 및 지식증류 방법(

) 등의 총 세 가지 손실 방법이 적용될 수 있다.where i is the training data

means to place

is obtained as a softmax output for the pre-trained source network and delivers the dark knowledge of the pre-trained dataset as a soft label. For example, for efficient auxiliary task learning, cross entropy (

), pokal (

) and knowledge distillation method (

), etc., a total of three loss methods can be applied.

[Equation 2]

는 학습 도중 네트워크로부터 얻어진 최종 특징 출력이며

는 온도(temperature) 파라미터로 레이블 소프트닝(label softening) 강도를 조절한다.

는 소프트 맥스 함수를 의미하고, KL은 서로 다른 분포에 대한 KL 다이버전스(divergence)를 의미한다. 수학식 2의 세 가지 손실 연산 기법들은 모든 학습 시나리오에서 적용될 수 있으며. 이때, k + 1번째 반복에서 기울기에 대한 업데이트는 다음의 수학식 3과 같다.here,

is the final feature output obtained from the network during training.

controls the label softening intensity with a temperature parameter.

denotes a soft max function, and KL denotes KL divergence for different distributions. The three loss calculation techniques of Equation 2 can be applied in all learning scenarios. In this case, the update of the gradient in the k + 1st iteration is expressed by Equation 3 below.

[Equation 3]

Here, η means the learning rate.

On the other hand, although Figs. 1 and 3 (a) show a transfer learning method in a single source network for convenience of explanation, it is not limited to this configuration and as shown in Fig. 3 (b), a multi-source task Knowledge may be transferred from the network 313 . In this case, the transition module 315 may exist as many as the number corresponding to each source task. That is, if knowledge transfer is performed through a loss function in a single task transfer setting, _{there is no reason why the source network h s} to perform knowledge transfer should be limited to a single task.

Therefore, the prior learning-based transfer learning method according to an embodiment can improve the structure by increasing the types of source tasks for transfer learning of the target task as shown in [Equation 4].

[Equation 4]

In this case, if there is no memory burden as a source task index for SM knowledge transfer, multi-task transfer learning can be theoretically performed from a total of M source tasks. Like the single task transfer learning, the gradient update rule is as follows [Equation 5].

[Equation 5]

와

의 형태를 명시하지 않는 동일한 문제 도메인의 태스크를 가정한 전이 학습 이었다. On the other hand, the transition for single or multiple tasks described above is

Wow

It was transfer learning assuming a task in the same problem domain that does not specify the form of .

와

의 모달리티가 다르거나 태스크의 문제 도메인이 다른 경우에도 전이 학습이 가능하다. 이 경우 입력 데이터의 특성이 다르고 학습을 위한 네트워크의 구조에도 큰 차이가 있으므로 전이 학습을 위한 준비 과정으로 데이터 변환이나 전처리를 수반할 수 있다. Transfer learning and apparatus based on prior learning according to an embodiment go one step further

Wow

Transfer learning is possible even when the modalities of the tasks are different or the problem domains of the tasks are different. In this case, since the characteristics of the input data are different and there is a big difference in the structure of the network for learning, data transformation or preprocessing may be involved as a preparation process for transfer learning.

3(c) shows transfer learning when the target network 322 and the source network 323 have different modalities, and FIG. 3(d) shows the case where the domains of the target network 332 and the source network 333 are different. Conceptually illustrates the method.

More specifically, in the case of transfer learning having different problem domains, the total loss function can be defined as in Equation 6 below.

[Equation 6]

가 3 차원 텐서로 정의되어 질 수 있다. 이때 전이 학습을 위한 사전 학습 네트워크를2D-CNN을 활용한 영상 인식 문제 T_s를 통해 얻는다면

에 대해 입력이 가능한 2차원 행렬로 변환하는 함수로 정의되어야 한다. Here, the data conversion function f _s refers to a function that transforms the form of data to match the source task so that cognitive information can be inferred and delivered to the task in the source domain. For example, if T _t is an action recognition problem using 3D-CNN, input

can be defined as a three-dimensional tensor. At this time, if the pre-learning network for transfer learning is obtained through _{the image recognition problem T s using 2D-CNN,}

It should be defined as a function that transforms into a two-dimensional matrix that can be input.

On the other hand, transfer learning to another problem domain may also perform knowledge transfer in a multi-source task as shown in Equation (7).

[Equation 7]

That is, referring to FIG. 4C , it can be defined in the same way as single or multi-task transfer learning for gradient update.

5 to 10 are diagrams for explaining the effect of the prior learning-based transfer learning method according to an embodiment.

5 is a summary of an experiment setting for verification of a transfer learning technique according to an embodiment.

Referring to FIG. 5 , four experimental scenarios were constructed to verify the knowledge transfer technique through prior learning. The first scenario ( FIGS. 6 and 7 ) is the setup of transfer learning between networks with a single source task in the same problem domain. For this, the previously learned network learned for classifying 2D images can be used for learning for classifying 2D images. (2D cls → 2D cls).

The second (FIG. 8) is transfer learning (multiple 2D cls to 2D cls) for classifying two-dimensional images using a multi-source network, in which a target network for learning a target task has two or more auxiliary tasks.

The third (FIG. 9) and fourth (FIG. 10) are knowledge transfer scenarios using source networks with different problem domains, and experiments were performed in two settings.

The third is target task learning with different objectives, but transfer learning to domains within the same image recognition category (2D cls to 2D multi-cls), where the targeted task helps general image classification networks to solve multi-class classification problems is when you receive

The fourth is a scenario of knowledge transfer using heterogeneous networks and problem domains. In learning an action recognition network using 3D-CNN, it was verified that the information learned during 2D image recognition can be used for knowledge transfer (2D). cls to 3D action-cls).

First, referring to FIG. 6 , a learning setup for performing transfer learning between two-dimensional image classification problems in the first experiment is shown. According to an embodiment, a network pre-trained by ImageNet and Place365 dataset was used as the source network. As a data set of the corresponding target task, CIFAR10, CIFAR100, and STL10, which have relatively small data volumes, along with large data sets ImageNet and Places365 were used. As the pre-trained source network, the ResNet50 network provided by torchvision was used. ResNet18 was utilized for the network for the target task.

Referring to FIG. 7 , it shows performance changes when scratch learning in a 2D image classification problem and transfer learning method (PreLeKT) according to an embodiment are applied. In transfer learning between 2D image classification problems, it was found that generalization performance was improved in all datasets when the transfer learning method (PreLeKT) disclosed in the present invention was used. In addition, it shows the performance change when the transfer module is used to efficiently transform the feature for learning auxiliary tasks. Of particular note, the SLT10 data set with a small percentage of training data showed the greatest improvement in generalization performance. It can be seen that when the amount of guidance is small, knowledge transfer using transfer learning has a greater effect.

Next, in the second experiment, multiple source networks were used to apply the multi-source task-based transfer learning method (PreLeKT) to the 2D image classification problem. As an example, the Res-Net50 model trained on the ImageNet and Places365 datasets was used as a source task at the same time. As a dataset for target task learning, Exp. The same datasets as set 1 were used, and the training details are shown in FIG. 6 .

Accordingly, FIG. 8 shows the performance learned from scratch on a specific dataset and the performance using single task-based transfer learning and multi-source task-based transfer learning (PreLeKT) in the learning process. At the same time, the learning results using the transfer module for each transfer learning (PreLeKT) are also shown. When transfer learning (PreLeKT) is applied to a 2D image classification problem by utilizing a multi-source task, it can be seen that the generalization performance is improved in almost all datasets than when a single-source task network is utilized.

Next, in the third experiment, transfer learning was performed between a 2D image classification problem and a 2D multi-class image classification problem to confirm whether transfer learning according to an embodiment is also effective for knowledge transfer between different problem domains. For this purpose, a multi-class image classification problem using the PASCAL VOC dataset was used as the target task. At this time, the target loss function for solving multi-class image classification was used as binary cross entropy, and the loss function for the source task was configured the same as in the first and second experiments. Learning settings for the target task and the source task are shown in FIG. 6 .

9 shows the performance learned from scratch for multi-class image classification and performance changes when the transfer learning method according to an embodiment is used in the learning process. When transfer learning is performed according to an embodiment, it can be confirmed that there is a very large performance improvement in transfer learning between different problem domains using a single source. In addition, it was confirmed that the performance improvement was larger than transfer learning between the same problem domains. In addition, it was confirmed that there was an additional performance improvement when transfer learning from a multi-source network was applied in the same way as in the setting of the second experiment.

Meanwhile, in the third experiment, even if there is a difference between the problem domains, since the modality of the input data of the source task and the input data of the target task is the same, the performance improvement of knowledge transfer may be relatively natural. Therefore, in the fourth experiment, the 2D image classification network was used as the source network for the experimental design of knowledge transfer in a more extended meaning, and the target task was set as an action recognition problem using 3D-CNNs.

On the other hand, in order to utilize the dark knowledge of the source network, the transformation function

의 정의가 필요하다. 일 실시 예에 따른 전이학습 방법의 주요 기술적 기여가 좋은 fs를 정의하는데 있지 않기 단순하게 fs의 역할을 3 차원 비디오 클립에서 중앙에 있는 프레임을 추출하는 것으로 한정하였다. 또한, 액션 인식 문제 학습을 위해 3D ResNet 기반의 CNNs 모델을 활용하였다. 보조 태스크를 추론을 돕기 위한 전이 모듈은 2D-CNN 의 경우와 유사하게 4 레이어의 3D 컨볼루션 블록으로 구성하였다. 3D ResNet 훈련을 위한 학습 설정은 도 6의 소스 네트워크의 학습 설정과 같이 표기되어 있다.

needs a definition of Since the main technical contribution of the transfer learning method according to an embodiment is not in defining a good fs, the role of fs is simply limited to extracting a frame at the center from a 3D video clip. In addition, a 3D ResNet-based CNNs model was used to learn the action recognition problem. The transition module to help infer the auxiliary task is composed of 4 layers of 3D convolution blocks similar to the case of 2D-CNN. The learning setting for 3D ResNet training is marked as the learning setting of the source network of FIG. 6 .

10 shows 3D-ResNet performance learned from scratch on the UCF-101 dataset and performance changes when the transfer learning method (PreLeKT) according to an embodiment is used in the learning process. Referring to FIG. 10 , it is shown that the auxiliary task of the transfer learning method (PreLeKT) has an effect of preventing overfitting during model learning of 3D-CNNs. At the same time, it contains the experimental results of whether transfer learning can be performed even when fine-tuning is performed with pre-learned weights for 3D ResNet. According to an embodiment, in the use of PreLeKT between heterogeneous problem domains, in the case of a video dataset such as Kinetics-400, in the case of a video dataset such as Kinetics-400, there may be other performance improvement effects depending on the definition of fs according to the definition of fs. and modality are different, so it was possible to give the effect of improving performance even when using pre-learned weights. That is, even in the case of 3D ResNet in which the pre-trained initial value is set, it was confirmed that some, but not all models, showed improved recognition results by transfer learning (PreLeKT) according to an embodiment.

11 is a block diagram schematically illustrating an internal configuration of an apparatus 100 for transfer learning based on prior learning according to an embodiment.

The prior learning-based transfer learning apparatus 100 is a device capable of performing a function to be described later, and may include, for example, a server computer, a personal computer, and the like. In an embodiment, the prior learning-based transfer learning apparatus 100 may include one or more processors 110 and/or one or more memories 120 .

According to an embodiment, the operation performed by the one or more processors 110 may include an operation for performing prior learning in a source network; an operation for performing loss-based learning on a target task in a target network; an operation for performing soft label-based self-supervised learning on an auxiliary task in order to transfer the data previously learned from the source network; and an operation of updating the learning parameter for the target task based on the learning for the auxiliary task.

In one embodiment, at least one of these components of the transfer learning apparatus 100 based on prior learning may be omitted, or other components may be added to the transfer learning apparatus 100 based on prior learning. In addition, additionally (additionally) or alternatively (alternatively), some of the components may be implemented as integrated, or may be implemented as a singular or a plurality of entities. At least some of the internal and external components of the prior learning-based transfer learning apparatus 100 include a bus, a general purpose input/output (GPIO), a serial peripheral interface (SPI), or a mobile industry processor interface (MIPI), etc. They may be connected to each other through the interface to transmit and receive data and/or signals.

The one or more memories 120 may store various data. Data stored in the memory 120 is data obtained, processed, or used by at least one component of the prior learning-based transfer learning apparatus 100 , and may include software (eg, a program). Memory 120 may include volatile and/or non-volatile memory. The one or more memories 120 may store instructions that, when executed by the one or more processors 130 , cause the one or more processors 110 to perform an operation. In an embodiment, the one or more memories 120 may store personalization information for one or more users and/or recommendation information for one or more products. In the present disclosure, programs or commands are software stored in the memory 120 , and have various functions so that an operating system, an application, and/or an application for controlling the resources of the prior learning-based transfer learning apparatus 100 can utilize the resources of the apparatus. may include middleware that provides

The one or more processors 110 may control at least one component of the prior learning-based transfer learning apparatus 100 connected to the processor 130 by driving software (eg, a program, a command). In addition, the processor 130 may perform various operations, processing, data generation, processing, etc. related to the present disclosure. In addition, the processor 110 may load data or the like from the memory 120 or store it in the memory 120 .

In an embodiment, the prior learning-based transfer learning apparatus 100 may further include a communication interface (not shown). The communication interface may perform wireless or wired communication between the prior learning-based transfer learning apparatus 100 and another server or other external device. For example, the communication interface is eMBB (enhanced Mobile Broadband), URLLC (Ultra Reliable Low-Latency Communications), MMTC (Massive Machine Type Communications), LTE (long-term evolution), LTE-A (LTE Advance), UMTS ( Universal Mobile Telecommunications System), GSM (Global System for Mobile communications), CDMA (code division multiple access), WCDMA (wideband CDMA), WiBro (Wireless Broadband), WiFi (wireless fidelity), Bluetooth (Bluetooth), NFC (near field) communication), a global positioning system (GPS), or a global navigation satellite system (GNSS) may perform wireless communication.

Various embodiments of the prior learning-based transfer learning apparatus 100 according to the present disclosure may be combined with each other. Each embodiment may be combined according to the number of cases, and an embodiment of the prior learning-based transfer learning apparatus 100 made by combining also falls within the scope of the present disclosure. In addition, the internal/external components of the preceding learning-based transfer learning apparatus 100 according to the present disclosure may be added, changed, replaced, or deleted according to embodiments. In addition, the internal/external components of the preceding learning-based transfer learning apparatus 100 may be implemented as hardware components.

Meanwhile, the method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions may include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine language codes such as those generated by a compiler.

So far, the present invention has been looked at with respect to preferred embodiments thereof. Those of ordinary skill in the art to which the present invention pertains will understand that the present invention can be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments are to be considered in an illustrative rather than a restrictive sense. The scope of the present invention is indicated in the claims rather than the foregoing description, and all differences within the scope equivalent thereto should be construed as being included in the present invention.

101: input
102: target network
103: source network
104: target task lost
105: transition module
106: Loss of auxiliary task

Claims (10)

In a method for performing transfer learning based on prior learning of a convolutional deep neural network,
performing prior learning in the source network;
performing loss-based learning on a target task in a target network;
performing soft-label-based self-supervised learning on an auxiliary task to transfer previously learned data from the source network; and
and updating a learning parameter for the target task based on learning for the auxiliary task. According to claim 1,
The step of learning for the auxiliary task is,
The method of constructing a multi-task network that simultaneously updates the loss of the target task and the loss of the auxiliary task, so as to induce the transmission of dark knowledge about the data learned from the source network. According to claim 1,
The auxiliary task includes a transition module composed of a convolution block,
and additional features for auxiliary tasks are extracted via the transition module. The method of claim 1,
The total loss function of the multi-task network including the target task and the auxiliary task is expressed by the following equation,

where i is the i _th batch for the training data,

is the loss of the target task,

is the loss of auxiliary tasks,

means the softmax output for the pre-trained source network. The method according to claim 1, wherein at least one of cross entropy, focal and knowledge distillation methods is used for learning the auxiliary task. The method of claim 1, wherein the source network is configured with at least one multi-task. The method according to claim 1, wherein the source network and the target network have different modalities or different problem domains of tasks. A computer program stored in a computer-readable recording medium in combination with a computer, which is hardware, to perform the preceding learning-based transfer learning method of claims 1 to 7. one or more processors; and one or more memories storing instructions that, when executed by the one or more processors, cause the one or more processors to perform an operation, wherein the operation performed by the processor comprises:
an operation for performing prior learning in the source network;
an operation for performing loss-based learning on a target task in a target network;
an operation for performing soft-label-based self-supervised learning on an auxiliary task in order to transfer previously learned data from the source network; and
and an operation for updating a learning parameter for the target task based on learning for the auxiliary task. [10] The apparatus of claim 9, wherein the source network and the target network have different modalities or different problem domains of tasks.

Images