KR20190138238A - Deep Blind Transfer Learning - Google Patents

Abstract

How can we transfer a semi-supervised model trained in a source domain to a target domain without transferring any instances within a source data set? In conventional transfer learning, there is no limitation on utilization of source and target data. However, if the size of the source data set is too large to enter a memory or the source data set is personal, we cannot utilize source data while training target data. In the paper, we add a completely blind constraint to a corresponding model, which means that source or target data cannot be used while training each of a target or a source model. The present invention provides a simple and novel semi-supervised transfer learning method utilized for allowing a model learned in a source domain and an aligning neural network to learn a better model in a target domain. The aligning neural network maps a target feature space to a source feature space so that a supervised model learned from classified data of a source domain can be easily used in a target domain. Thus, the most advanced performance in partial-supervised transfer learning is achieved.

Description

Deep Blind Transfer Learning

Artificial intelligence systems for realizing human intelligence are currently used in various technical fields. In general, artificial intelligence systems, unlike conventional rule-based smart systems, have the feature that machines learn and decide on their own and become smarter. As the user uses the artificial intelligence system, the recognition rate of the artificial intelligence system is improved, and the artificial intelligence system can better understand the user's preferences or interests. Therefore, existing rule-based smart systems are being replaced by deep learning-based AI systems.

Artificial intelligence technologies include elementary technologies that use machine learning (eg, deep learning) and machine learning.

Machine learning refers to an algorithmic technique in which a machine classifies and learns characteristics of input data. Elemental technology refers to the technology of simulating the cognitive or deterministic functions of the human brain using machine learning algorithms such as deep learning, and can be divided into the areas of language understanding, visual understanding, reasoning, prediction, and knowledge representation.

Artificial intelligence technology (AI) can be applied to various fields. Language understanding refers to the technology of recognizing, applying and processing human verbal / written languages and includes natural language processing, machine translation, conversation systems, question and answer and speech recognition, and speech synthesis. Visual understanding refers to the technology of recognizing and processing objects from a human perspective, and includes object recognition, object tracking, image retrieval, human recognition, scene understanding, spatial understanding, and image enhancement techniques. Inference / prediction refers to the technique of determining information and performing logical reasoning and prediction, and includes knowledge / probability based inference, optimization prediction, preference based planning and recommendation. Knowledge representation refers to the technology of processing human experience information into automated knowledge data and includes knowledge composition (generation / classification data) and knowledge management (data utilization). The motion control refers to a technology for controlling the automatic driving of the vehicle and the motion of the robot, and includes motion control (navigation, collision, driving) and operation control (motion control).

Reference

[1] S. J. pan and Q. Yang, "A survey on transfer learning," IEEE Transactions on knowledge and data engineering, vol, 22, no. 10, pp. 1345-1359, 2010

[2] K. Weiss, T. M Khoshgoftaar, and D. Wang, "A survey of transfer learning," Journal of Big Data, vol. 3, no. 1, p. 9, 2016.

[3] J. Quionero-Candela, M. Sugiyama, A. Schwaighofer, and N. D. Lawrence, Dataset shift in machine learning. The MIT Press, 2009.

[4] Y. Aytar and A. Zisserman, "Tabula rasa: Model transfer for object category detection," in Computer Vision (ICCV), 2011 IEEE International Congerence on. IEEE 2011, pp. 2252-2259.

[5] R. Girshick, J. Donahue, T. Darrell, and J. Malik, "Rich feature hierarchies for accurate object detection and semantic segmentation," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2014, pp. 580-587.

[6] J. Long. E Shelhamer, and T. Darrell, "Fully convolutional networks for semantic segmentation," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 3431-3440.

[7] S. Ren, K. He, R. Girshick, and J. Sun, "Faster r-cnn: Towards real-time object detection with region proposal networks," in Advances in neural information processing systems, 2015, pp. 91-99.

[8] R. Collobert, J. Weston, L. Bottou, M. Karen, K. Kavukcuoglu, and P. Kuksa, "Natural language processing (almost) from scratch," Journal of machine Learning Reserch, Vol. 12, no. Aug, pp. 2493-2537, 2011.

[9] X. Glorot, A. Bordes, and Y. Bengio, "Domain adaptation for large-scale sentiment classification: A deep learning approach," in Proceedings of the 28th international conference on machine learning (ICML-11), 2011, pp. 513-520.

[10] M. Chen, Z. Xu, K. Q. Weinberger, and F. Sha, "Marginalized denoising autoencoders for domain adaptation," in Proceedings of the 29th International Coference on International Conference on Machine Learning. Omnipress, 2012, pp. 1627-1634.

[11] M Xiao and Y. Guo, "Domain adaptation for sequence labeling tasks with a probabilistic language adaptation model," in Proceedings of the 30th International Conference on Machine Learning Volume 28. JMLR. org, 2013, pp. 1-293.

[12] E. Tzeng, J. Hoffman, N. Zhang, K, Saenko, and T. Darrell, "Deep domain confusion: Maximizing for domain invariance," arXiv preprint arXiv: 1412.3474, 2014.

[13] E. Tzeng, J. Hoffman, T. Darrel, and K, Saenko, "Simultaneous deep transfer across domains and tasks," in Computer Vision (ICCV), 2015 IEEE International Conference on. IEEE 2015, pp. 4068-4076.

[14] E. Tzeng, J. Hoffman, K. Saenko, and T. Darrel, "Adversarial discriminative domain adaptation," in Computer Vision and Pattern Recognition (CVPR), vol. 1, no. 2, 2017, p.4

[15] M. Long, Y. Cao, J. Wang, and M. Jordan, "Learning transferable features with deep adaptation networks." International Conference on Machine Learning, 2015, pp. 97-105.

[16] M. Long, Y. Cao, J. Wang, and M. Jordan, "Unsupervised domain adaptation with residual transfer networks," in Advances in neural information processing systems, 2016, pp. 136-144.

[17]-, "Deep transfer learning with joint adaptation networks," International Conference on Machine Learning, 2017, pp. 2208-2217.

[18] Y. Ganin and V. Lempitsky, "Unsupervised domain adaptation by backpropagation," International Conference on Machine Learning, 2015, pp. 1180-1189.

[19] Y. Ganin, E. Ustinova, H. Marchand, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky, "Domain-adversarial training of neural networks," The Joural of Machine. Learning Reserach, vol. 17, no. 1, pp. 2096-2030, 2016.

[20] K. Bousmalis, G. Trigeorgis, N. Silberman, D. Krishnan, and D. Erhan, "Domain separation networks," in Advances in neural information processing systems, 2016, pp. 343-351.

[21] X. Zhang, F. X. Yu. S.-F. Chang and S, Wang, "Deep transfer networks: Unsupervised domain adaptation," arXiv preprint arXiv: 1503.00591, 2015.

[22] F. Ma, D. Meng, Q. Xie, Z. Li, and X, Dong, "Self-paced co-training," in International Conference on Machine Learning, 2017, pp. 2275-2284.

[23] K. Saito, Y. Ushiku, and T. Harada, "Asymmetric tri-training for unsupervised domain adaptation," arXiv preprint arXiv: 1702.08400, 2017.

[24] A. Kumar, P. Sattigeri, and T. Fletcher, "Semi-supervised learning with in Advances in neural information processing systems,

[25] Z. Dai, Z. Yang, F. Yang, W. W. Cohen, and R. R. Salakhutdinov, "Good semi-supervised learning that requires a bad gan," in Advances in neural information processing systems, 2017, pp. 6513-6523.

[26] T. Miyato, S-i. Maeda, M. Koyama, and S. Ishii, "Virtual adversarial training: a regularization method for supervised and semi-supervised learning: arXiv preprint arXiv: 1704.03976, 2017.

[27] C. N. d. Santos, K. Wadhawan, and B. Zhou, "Learning loss functions for semi-supervised learning via discriminative adversarial network," arXiv preprint arXiv: 1707.02198, 2017.

[28] L. Duan, I. W. Tsang, D. Xu, and T.-S. Chua, "Domain adaptation from multiple sources via auxiliary classifiers," in Proceedings of the 26th Annual International Conference on Machine Learning. ACM, 2009, pp. 289-296.

[29] H. Daume III, A. Kurnar, and A. Saha, "Frustratingly easy semi-supervised domain adaptation," in Proceedings of the 2010 Workshop on Domain Adaptation for Natural Language Processing. Association for Computational Linguistics, 2010, pp. 53-59

[30] J. Donahue, J. Hoffman, E. Rodner, K. Saenko, and T. Darrell, "semi-supervised domain adaptation with instance constraints," in Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on. IEEE, 2013, pp. 668-675

[31] T. Yao, Y. Pan, C.-W. Ngo H. Li, and T. Mei, "Semi-supervised domain adaptation with subspace learning for visual recognition," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 2142-2150.

[32] N. Papernot, M. Abadi U. Erlingsson, I. Goodfellow, and K. Talwar, "Semi-supervised knowledge transfer for deep learning form private training data." arXiv preprint arXiv: 1610.05755, 2016.

[33] S. Ao, X. Li, and C. X. Ling, "Fast generalized distillation for semi-supervised domain adaptation," in AAAL, 2017, pp. 1719-1725.

[34] M. Chen, K. Q. Weinberger, and J. Blitzer, "Co-training for domain adaptation," in Advances in neural information processing systems, 2011, pp. 2456-2464.

[35] M. Rohrbach, S. Ebert, and B. Schiele, "Transfer learning in a transductive setting," in Advances in neural information processing systems, 2013, pp. 46-54.

[36] D.-H. Lee, "Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks," in Workshop on Challenges in Representation Learning, ICML, vol. 3, 2013, p.2.

[37] S. Khamis and C. H. Lampert, "Coconut: Co-classification with output space regularization." in BMVC, 2014.

[38] O. Sener, H. O. Song, A. Saxena, and S. Savarese, "Learning transferable representations for unsupervised domain adaptation," in Advances in Neural Information Processing Systems, 2016, pp. 2110-2118.

[39] N. N. Pise and P. Kulkarni, "A survey of semi-supervised learning methods," in Computational Intelligence and Security, 2008. CIS'08. International Conference on, vol. 2. IEEE, 2008, pp. 30-34.

[40] V. J. Prakash and D. L. Nithya, "A survey on semi-supervised learning techniques," arXiv preprint arXiv: 1402.4645, 2014.

DETAILED DESCRIPTION The present disclosure relates to deep blind transfer learning (DBTL) that solves a feature distribution shift problem by aligning source and target feature spaces.

1 is a diagram of the structure and learning of DBTL. In step 1, an auto encoder is learned through an unlabeled source instance. In step 2, a stack of classifiers and encoders is learned through a labeled source instance. Then, after the model is transferred to the target domain, in step 3, the autoencoder is fine-tuned through the unlabeled target instance. In step 4, we use an unlabeled instance to map the target feature space to the source feature space and train the aligner. Finally, in step 5, we train the entire stack of encoders, classifiers, and sorters on the labeled target instance.

I. Introduction

How can you transfer a trained deep learning model from source to target under the constraints of data visibility? The latest improved performance of deep neural networks (DNNs) requires large amounts of labeled data. However, in most cases, there is more unlabeled data than labeled data, and it is expensive to label the unlabeled data. Thus, the present disclosure may additionally use relevant data that has a slightly different distribution while using a small amount of labeled data. Since acquisition of unlabeled data is relatively inexpensive, the present disclosure may utilize unlabeled data to train a deep circulatory neural network (DNN).

Previous learning methods [1, 2] that leveraged information from rich labeled data for other source domains have received a lot of attention in the field of machine learning (machine learning). Unlike other learning methods, previous learning applies knowledge learning from the source domain to the relevant target domain. Since the source and target domains may have different distributions or play different roles, it is difficult to transfer appropriate information from the target domain to the source domain to aid in prediction [3].

In a practical straight-forward deep transfer [4, 5, 6, 7, 8, 9, 10, 11], some low-level parameters of source-based feedforward networks Variables can be copied and fine-tuned to the target data. The learning process of the source domain is strictly separated from the learning process of the target domain. In contrast, recent deep transfer learning methods [12, 13, 14, 15, 16, 17, 18, 19, 20, 21] have been used to minimize feature distribution discrepancies between source and target domains. Train the data together.

However, some previous learning applications cannot use source datasets in target data processes because of the large volume of ineffective source data or the sensitivity of source data to privacy. In the present disclosure, a learning program before blind is proposed in the previous learning. In the learning program before the blind, the learning process for the source data and the target data set is strictly separated. Under blind constraints knowledge from the source domain can only be transferred once to the source domain. The target data cannot be utilized during the learning of the source data, and vice versa. Blind constraints are not clear and have not been studied in previous studies. Under blind constraints, it is impossible to empirically calculate or optimize the distribution inconsistency in the learning process, and it becomes more difficult to reduce the change in the data distribution.

In the present disclosure, both source and target domains are classified into full and partial semi-supervised transfer learning depending on whether labeled instances and unlabeled instances are included. For pre-supervised learning, the learning process can be even more complicated by sequentially tweaking unsupervised models in the source / target data set. Multiple fine-tuning through different data or objects can erase previous learning and result in loss of information. In addition, the source and target feature spaces can be distinguished and the source classifier is incompatible with the target feature space. Sequential fine tuning approaches cannot guarantee good performance in the final model of the target domain.

The present disclosure proposes a deep blind transfer learning (DBTL) for aligning source and target feature spaces to solve a feature distribution shift problem. DBTL extends a simple refinement approach with simple alignment of the cyclic neural network to map target features into the source feature space. Specifically, the mapping of source and target features can be learned individually in an unsupervised manner in each domain. Thereafter, the target target feature is mapped to the source feature to learn another circulatory neural network by mapping a source feature that is not labeled with the target feature to the target feature. Due to the feature space alignment, the classifier is learned in the source domain to which the target data is applied. Fine tuning the entire structure in the target labeled data set produces the final model. In this process, sequential refinement of a single structure over multiple data or objective functions can satisfy blind constraints without eliminating previous learning. All information in the source / target labeling / target unlabeled data can be preserved to improve the final model. Experimental results on the method of this disclosure show excellent performance.

II. Introduction

Automatic encoding consists of two parts: an encoder and a decoder. The encoder aims to map the input data to the hidden layer to obtain a new representation in low dimensional space. The decoder attempts to reconstruct the original input using a hidden representation by minimizing the error between the original input and the reconstructed output.

를 m개의 샘플,

차원에 대한 입력 데이터 셋으로 정의한다. 인코더와 디코더는 다음과 같이 형성된다.In the present disclosure

M samples,

Defined as the input data set for the dimension. The encoder and decoder are formed as follows.

은 은닉 표현이고,

는 원래의 특징

에 대한 재구성된 출력이다. 모델 매개변수

는 가중치 행렬

,

및 편향

,

을 포함한다.

Is a hidden expression,

Original features

Reconstructed output for. Model parameters

Is the weight matrix

,

And deflection

,

It includes.

를 학습하는 것이다. The goal of the automatic encoder is to minimize model reconstruction errors for a given data set X and objective function.

To learn.

Where R (W, W ') is used as the regularization.

III. How to suggest

This section describes in detail the deep blind transfer learning method, which is the blind full semi-supervised transfer learning method of the present disclosure.

A. Overview

로 정의하고, 타겟 도메인에 대한 라벨링되지 않은 샘플을

로 정의한다. 소스 도메인에 대한 라벨링된 샘플을

로 정의하고, 타겟 도메인에 대한 라벨링된 샘플을

로 정의한다. In the method of learning before full semi-supervised, there are four types of data (labeled / unlabeled, source / target). Unlabeled Samples for Source Domain

To be unlabeled samples of your target domain

Defined as Labeled samples for source domains

, And labeled samples for your target domain

Defined as

과

에 대한 접근 없이

과

를 바탕으로 처음 학습된다 그 후, 학습된 모델은 전의되고,

과

에 대한 접근 없이

과

를 통해 학습된다. 블라인드 제약하에서는 두 가지 과제(challenge)가 있다. 첫 번째로 소스 및 타겟 데이터가 학습과정에서 함께 사용되지 않으므로, 샘플링 접근으로부터 특징 분포 차이를 계산하기 힘들다. 두 번째로 소스 데이터에 대해 학습된 특징 표현과 분류자를 모두 활용하는 것은 학습된 특징 표현이 소스 및 타겟 도메인간에서 다르기 때문에 어렵다. 본 개시는 이러한 어려움을 해결하기 위해 딥 블라인드 전의 학습 모델을 제안한다. 소스와 타겟 특징 공간 간의 불일치를 최소화 하는 대신 위 모델에서는 타겟 특징 공간을 소스 특징 공간과 정렬하는 방법을 학습한다. 타겟 도메인에 대해서만 맵핑을 학습 함으로써, 블라인드 제약이 만족된다. 또한, 타겟 자동인코더의 맨 위에 타겟에서 소스 특징으로의 맵핑은 소스 특징 공간에서 학습된 분류자가 타겟 데이터를 계속 예측할 수 있도록 한다.The blind semi-supervised transfer learning process constrains the model learning process as follows. The above model

and

Without access to

and

Is trained first based on the trained model.

and

Without access to

and

Are learned through. There are two challenges under blind constraints. First, since the source and target data are not used together in the learning process, it is difficult to calculate the feature distribution difference from the sampling approach. Second, utilizing both learned feature representations and classifiers on source data is difficult because the learned feature representations differ between the source and target domains. The present disclosure proposes a learning model before deep blinds to solve this difficulty. Instead of minimizing discrepancies between the source and target feature spaces, the model learns how to align the target feature spaces with the source feature spaces. By learning the mapping only for the target domain, blind constraints are satisfied. In addition, the mapping from target to source feature on top of the target autoencoder allows the classifier learned in the source feature space to continue to predict the target data.

B. DBTL

The model according to the present disclosure is composed of three neural network structures as shown on the left side of FIG. 1. The first neural network is trained with an autoencoder through unlabeled data to learn the structure of the feature space using the model according to the present disclosure. The second neural network is learned by mapping the output of the target autoencoder to the output of the source autoencoder. The third neural network is trained in a general supervisory fashion with labeled data. The first, second, and third neural networks are defined as encoders, aligners, and classifiers, respectively. In order to satisfy the blind constraint, the next unlabeled source / target data and the labeled source / target data are sequentially learned.

 Step 1: The encoder learns about unlabeled source data using the autoencoder reconstruction loss.

 Step 2: Train the classifier on top of the previous encoder using the finely tuned labeled target data.

 Step 3: The encoder is fine-tuned again on unlabeled target data due to the reconstruction loss of the standard autoencoder. In this step, an encoder with fine-tuned parameters is called a target encoder.

 Step 4: The sorter learns to map the target encoding to the source encoding of unlabeled target data. At this stage, only the sorter (green part) is trained.

 Step 5: The stack of encoders, sorters, and classifiers is refined in the target labeled data.

Source and target encoders are defined as feature spaces encoded by the source and target encoders, respectively. It aims to learn the aligner to align the target feature space with the source feature space.

C. Model properties

Blind constraints, use only source or target datasets for all learning phases. In the training phase, in the first and second stages, the model is trained on the source data set, and no target data set is required. In contrast, steps 3, 4, and 5 utilize the target data set without accessing the source data set. The only information passed between steps 2 and 3 is the source auto-encoding and source classifier. This clear separation in the learning process satisfies the blind constraint.

 Forecast of performance. The target auto encoding of step 3 provides another feature mapped through the source auto encoding fine tuned in step 2. Because of these differences, the source classifier learned in step 2 does not provide consistent performance for features from target auto encoding. In the third stage, the predictive performance of the source classifier is degraded while learning the structure of the target data set. Therefore, the sorter is introduced in step 4. Learn the sorter to map the encoding of the source autoencoder to the encoding of the target autoencoder. The stack of target autoencoder and alignment neural network maps data of the target domain into the source feature space. The source classifier then provides good prediction performance in the ordered feature space.

IV. Experiment result

In this section, extensive experiments verify the performance of the DBTL. This disclosure attempts to answer the following questions.

Q1. (Expectation of performance) How does DBTL perform blind semi-supervised transfer learning?

Q2. (Use unlabeled data in quasi-supervised learning) Do you effectively use unlabeled data when the labeled data is sparse?

Q3. (The effect of sorter learning) Does the sorter's benefit come from sorting features rather than placing more layers?

A. Experiment setup

Dataset. In the present disclosure, the experiment is performed through five actual data of a multidimensional structure such as HEPMASS, HIGGS, SUSY, and Sensorless Gas. Three data sets (HEPMASS, HIGGS and SUSY) are sensor observations of particle collision experiments. The labeling here is the actual generation of interest and the binary. Another data set (Sensorless Gas) is labeling, which is a sensor observation of the type of gas in the diagnosis and chemical reaction of a motor condition. The statistics of the data are shown in Table 1. All data sets are available in the UCL machine learning (machine learning) repository.

Table 1

According to the data set statistics, all data sets are available in the UCL machine learning archive. URL: HTTP: ARCHIVE.ICS.UCI.EDU/ML

Data processing . To perform a full semi-supervised transfer experiment, you need a data set with source / target, labeled / unlabeled instances. Split the above data into source / target and specify labeling or no labeling as follows:

 Step 1: First normalize the feature range. If the feature contains a negative value, it can be normalized to the range -1 to 1. If it does not contain a negative value, it can be normalized in the range of 0 to 1.

Step 2: Using the K-means clustering algorithm with K set to 10, 10% of the total data is selected as the target data. Target data is sampled in clusters of sufficient size.

Step 3: Sample 1% of the total data as source data and sample target test data from the partition. In the present disclosure, set is set as learning data.

Step 4: The present disclosure randomly selects 90% of the source and target training data and deletes the labeling to obtain data that is not labeled with the training data.

 In K-means clustering, instances in each cluster are closer to each other than instances in other clusters. Since the target data instance is sampled in the same cluster, the source and target data have different intended feature distributions. The final data rate is labeled source: unlabeled source: test source: labeled target: unlabeled target = 80.1: 8.9: 1: 8.1: 0.9: 1. to be.

 B. Prediction of Performance

According to an embodiment of the present disclosure, the predictive performance of the DBTL is set as the performance for blind full semi-supervised transfer learning. The method according to the present disclosure uses all four types of training data. Compare the methods according to the present disclosure with four other reference methods that use only a portion of the data.

One) T-L: One neural network is trained only with labeled target datasets.

2) ST-L: A single neural network was pretrained with a labeled source data set and fine tuned to a labeled target data set. This is a simple pre-learning approach that utilizes a labeled source data set.

3) T-LU: A stack of encoders and neural network classifiers (STACKs) where the encoder is trained on a target data set that is not labeled with an automatic encoder and the entire network is further trained on a labeled target data set. This is a simple semi-supervised learning approach that takes advantage of unlabeled target data sets.

4) ST-LU: The stack of encoders and neural network classifiers is learned like the DBTL method except that there is no sorter. This is a natural approach that combines quasi-supervised learning with advocacy.

 In order to compare on the same ground, the structure of the encoder, the aligner and the classifier is fixed to 80_40_20_10, 100_50_100 and 00_300_300_300_300, respectively. In the optimization process for each DBTL step, we use adam optimizer with a learning rate of 0.001 and a batch size of 100. Learning continues until the performance of the test data set converges. The potential benefit of increasing the size of the layer is analyzed in part IV_D.

Accuracy predictions for the target test set are summarized in Table 2. Final performance is reported with the accuracy of multi-class data sets (Gas, Sensorless) and binary labeled data sets (HEPMASS, HIGGS, SUSY). For all data sets, DBTL showed the best predictive performance in the natural approach of ST-LU. Compared with naive supervised learning, DBTL improved performance by 6% in multi-class datasets and 3-12% in binary labeled datasets. In addition, the application of previous and quasi-supervised learning has improved predictive performance over natural learning. The previous learning model showed a performance increase of 1-8%, while the semi-supervised learning model showed a performance increase of 1-3%.

TABLE 2

DBTL predictive performance. We showed the performance of DBTL compared with the four basic methods in the blind pre-supervised learning problem. The method DBTL according to the present disclosure achieved the best prediction performance for all five databases.

C. Leverage unlabeled data from semi-supervised learning

In this disclosure, we examined the effects of semi-supervised learning in DBTL.

D. Effect of Sort Learning

In the present disclosure, the effect of sorter learning of DBTL was tested.

V. related research

Learning deep neural network (DNN) requires huge data sets. However, labeled job data sets are generally lacking. Learning a deep neural network requires a huge amount of data sets, but it also requires a set of labeled or unlabeled data sets from other domains.

 The previous training [1, 2] aims to transfer the machine learning model from the learned data to different domains with different distributions. Leverage generalizable knowledge gained from the source domain to learn better models in the target domain. In the field of computer vision [4, 5, 6, 7] and natural language processing [8, 9, 10, 11], there have been a lot of studies applying previous learning to obtain the most advanced performance.

 Existing methods for deep transfer learning frequently minimize feature distribution discrepancies between source and target domains [12, 13, 14, 15, 16, 17, 18, 19, 20]. This method uses source and target data together in the learning process to calculate and optimize feature distribution inconsistencies. Other studies of previous learning have learned and assigned pseudo-lables by unlabeled and target data, respectively [34, 35, 36, 37, 38, 22, 23]. This method iteratively fine tunes the model in the source and target data to provide better pseudo-labeling.

 Semi-supervised learning [39, 40] seeks to utilize both labeled and unlabeled data for better performance with labeling predictions. Unlabeled data is used to find the structure in the feature space structure or to normalize the model with over-fitting on small labeled data. The most recent study [24, 25, 26, 27] used adversarial learning to learn the structure in feature space. Some studies have studied the problem of semi-supervision adaptation [28, 29, 30, 31]. The study also used both source and target data to measure inconsistencies in the data distribution.

In the present disclosure, the visibility of the source and target data sets is explicitly limited in the learning process. As we know, the isolation of source data from target data in the learning process of semi-supervised learning has only recently been considered [32, 33]. In this study, we provide similar labeling of target domains so that trained classifiers can learn from classifiers. However, even with ensembling [32] or distillation [33], these models can be trained with incorrectly labeled data. The work of this disclosure utilizes all source / target, labeled / unlabeled data in the training of the final model and provides a principled approach to satisfying blind constraints. In addition, the task of this disclosure is to first address the issue of pre-supervision learning. The differences between the problem settings are summarized in Table 3.

TABLE 3

Summary of differences between problem settings. The method according to the present disclosure utilizes all other types of databases and meets blind constraints.

VI. conclusion

 The present disclosure proposes a new deep transfer learning method under full semi-supervision with visibility constraints on source data during target learning. The present disclosure carefully sets the learning process to maximize the utilization of all data and to meet blind constraints. Because of blind constraints, previous approaches to minimizing or repeating learning for pseudo-labeling are not applicable. In order to take advantage of classifiers learned in the source domain, the target feature is mapped to the source feature region through automatic encoding. In this way, both information from unsupervised learning of target data and information from supervised learning of source data are utilized. It demonstrates an effective approach with state-of-the-art performance in full pre-supervised learning of real-world data under blind constraints.

Claims (6)

In the Deep Blind Transfer Learning method of the artificial intelligence neural network model,
Learning an encoder through unlabeled source data;
Learning a classifier and the encoder through labeled source data;
Fine-tuneing the encoder via unlabeled target data;
Learning an alien from the unlabeled target data; And
Learning the entire stack of classifier, sorter, and encoder from labeled target data;
Learning method comprising a. The method of claim 1,
Learning an encoder through the source data,
A learning method comprising learning an encoder through unlabeled source data based on a reconstruction loss of an autoencoder. The method of claim 1,
The classifier is trained in a common supervised fashion based on the labeled data. The method of claim 1,
Learning the sorter,
And learning by mapping a target feature space to a source feature space. The method of claim 1,
The encoder and classifier are not trained while the sorter is trained. The method of claim 1,
The learning method characterized in that the source data and the target data are not used at the same time in each learning step.

Images