JP7277645B2 - Information processing device, information processing method and program - Google Patents

Description

The present invention relates to an information processing device, an information processing method, and a program, and more particularly to a technique for machine learning a learning model.

In supervised machine learning, a learning model is subjected to machine learning using a teacher data set consisting of teacher samples (specimens) and teacher labels so that the relationship between the teacher samples and labels is reflected in the learning model. By applying a trained model obtained by such machine learning to unlabeled unknown samples in the inference phase, desired processing results such as image recognition and classification can be obtained.

If a trained model trained using a certain teacher data set is applied as it is to a task targeting a data set whose domain is different from that of the teacher data set, the accuracy of inference will be degraded. Here, the domain refers to the type, range, distribution, etc. of data in a data set.
Therefore, there is a technology that makes it possible to apply the trained model to the target domain by further machine learning the trained model using samples of the target domain, which is the same domain as the target dataset. It is called Transfer Learning.

One of this transfer learning is domain adaptation. In this domain adaptation, it is assumed that the distribution of samples is different between the domain of the training dataset and the domain of the target dataset. A learning model is trained so as to approximate the distribution of a certain target dataset domain (target domain).

US Pat. No. 6,200,000 discloses a domain adaptation approach in the task of classifying vehicle images captured by cameras.
Specifically, in the labeling system of US Pat. to train a boosted classifier such that the misalignment between domains is learned. This utilizes images collected by early-introduced cameras and cameras installed in other locations to classify vehicle images.

JP 2016-58079 A

By the way, in domain adaptation, the teacher samples included in the source domain are labeled, but the target data samples included in the target domain are not necessarily labeled. Domain adaptation when the samples in the target domain are unlabeled is also called unsupervised domain adaptation.
In any case, domain adaptation assumes that the target domain contains samples corresponding to all labels attached to samples belonging to the source domain.

However, it is not always possible to prepare samples corresponding to all labels of the source domain as samples of the target domain. As described above, if the target domain lacks samples corresponding to some labels attached to samples belonging to the source domain, the technique of Patent Document 1, when the source domain is domain-adapted to the target domain, The sample distribution of the source domain cannot be sufficiently close to the sample distribution of the target domain.
Therefore, there is a risk that the accuracy of domain adaptation to the trained model will decrease, and that the accuracy of various processes for reasoning using the trained model will also decrease.

The present invention has been made to solve the above problems, and its object is to obtain highly accurate processing results even when the samples in the target domain do not sufficiently correspond to the samples in the source domain. An object of the present invention is to provide an information processing device, an information processing method, and a program.

In order to solve the above problems, one aspect of the information processing apparatus according to the present invention provides a first class sample and a second class sample included in a source domain, and the first class sample included in a target domain. a feature extracting unit for extracting features from each of the samples of the target based on the distribution of the samples of the first class included in the target domain in the feature space of the features extracted by the feature extracting unit a pseudo sample generation unit that generates pseudo samples of the second class of the domain; a data conversion unit that converts data by machine learning so as to approximate the distribution of the samples of the first class and the pseudo samples of the second class included in the target domain.

The pseudo sample generator estimates a first reliability of a distribution of samples of the first class included in the target domain in the feature space, and based on the estimated gradient of the first reliability , may generate the pseudo-samples.

The pseudo sample generation unit may generate the pseudo samples in regions of the feature space where the estimated first reliability is low.

The pseudo sample generator estimates a second confidence of a distribution of samples of the second class included in the source domain in the feature space, and based on the estimated second confidence gradient , may generate the pseudo-samples.

The pseudo-sample generation unit combines the first reliability and the second reliability, and generates the generated pseudo-sample based on the combined reliability in the feature space. It may be resampled to distribute it in regions with higher confidence.

The pseudo sample generator generates the pseudo samples based on a distance between a distribution of the first class samples and a distribution of the second class samples included in the source domain in the feature space. You can

Further comprising a domain classifier for classifying samples into either the source domain or the target domain, the pseudo sample generation unit classifying the samples classified into the target domain by the domain classifier into the source domain The domain classifier may be trained to give higher weight to classified samples.

A class classifier that classifies samples into either the first class or the second class, wherein the pseudo sample generator causes the class classifier to classify the samples into the second class as , the classifier may be trained to give higher weight to samples classified into the first class.

The data conversion unit may machine-learn at least one of the domain classifier and the class classifier so that cross-entropy loss in the feature space calculated using a first loss function becomes smaller. .

The data transformation unit may perform machine learning so that the Wasserstein distance between the source domain and the target domain in the feature space calculated using a second loss function becomes smaller.

One aspect of the information processing method according to the present invention is an information processing method executed by an information processing apparatus, wherein a first class sample and a second class sample included in a source domain and a sample of a second class included in a target domain extracting features from the samples of the first class, respectively; generating the pseudo-samples of the second class; and, in the feature space, the distribution of the first class samples and the second class samples contained in the source domain contained in the target domain. and transforming data by machine learning to approximate the distribution of the samples of the first class and the pseudo-samples of the second class.

One aspect of the information processing program according to the present invention is an information processing program for causing a computer to execute information processing, wherein the program stores, in the computer, first class samples and second class samples included in a source domain. A feature extraction process for extracting features from each of the samples of the class of and the samples of the first class included in the target domain; a pseudo-sample generation process for generating pseudo-samples of the second class of the target domain based on the distribution of the first-class samples contained in the source domain in the feature space; Data for transforming the distribution of the class samples and the second class samples by machine learning so as to approach the distribution of the first class samples and the pseudo samples of the second class included in the target domain. It is for executing processing including conversion processing.

According to the present invention, highly accurate processing results can be obtained even if the samples in the target domain do not sufficiently correspond to the samples in the source domain.
The objects, aspects and effects of the present invention described above and the objects, aspects and effects of the present invention not described above can be understood by a person skilled in the art to carry out the following invention by referring to the accompanying drawings and the description of the claims. can be understood from the form of

FIG. 1 is a block diagram showing an example of the functional configuration of a learning device according to an embodiment of the invention. FIG. 2 is a diagram for explaining the asymmetry (domain shift) of the sample distributions of the source domain and the target domain according to this embodiment. FIG. 3 is a diagram for explaining the source domain and the target domain that are domain-adapted through the pseudo-sample generation process executed by the learning apparatus according to this embodiment. FIG. 4 is a conceptual diagram showing an example of a module configuration of a learning model and a schematic processing procedure when the learning device according to the present embodiment is implemented in machine learning. FIG. 5 is a flowchart showing an example of a detailed processing procedure of the pseudo sample generation process executed by the pseudo sample generation unit of the learning device according to the present embodiment. FIG. 6 is a schematic diagram illustrating an example of a procedure for generating pseudo-samples of the target domain from samples of the target domain and the source domain in the feature space. FIG. 7 is a schematic diagram illustrating another example of the procedure for generating pseudo samples of the target domain from samples of the target domain and the source domain on the feature space. FIG. 8 is a block diagram showing an example of the hardware configuration of the learning device according to this embodiment.

Embodiments for carrying out the present invention will be described in detail below with reference to the accompanying drawings. Among the constituent elements disclosed below, those having the same functions are denoted by the same reference numerals, and descriptions thereof are omitted. The embodiments disclosed below are examples of means for realizing the present invention, and should be appropriately modified or changed according to the configuration of the device to which the present invention is applied and various conditions. is not limited to the embodiment of Also, not all combinations of features described in the present embodiment are essential for the solution means of the present invention.

The learning device according to the present embodiment extracts the features of the source domain samples and the target domain samples, respectively, and selects a plurality of labels (classes) assigned to the source domain samples that are not sufficiently included in the target domain. Label (class) samples are generated as pseudo-samples in regions of the target domain in the feature space, and the generated pseudo-samples are complemented to the samples of the target domain.
The learning device according to the present embodiment also performs machine learning so that the source domain is domain-adapted to the target domain complemented with the pseudo samples.

Below, an example in which the present embodiment is applied to, for example, recognition and classification of images will be described, but the present embodiment is not limited to this, and any Applicable to any kind of data or domain.

<Functional configuration of learning device>
FIG. 1 is a block diagram showing an example of the functional configuration of a learning device 1 according to this embodiment.
A learning device 1 shown in FIG.
The study device 1 may be communicably connected to a client device (not shown) configured by a PC (Personal Computer) or the like via a network. In this case, the learning output device 1 is mounted on a server, and the client device may provide a user interface when the learning device 1 inputs and outputs information to and from the outside. You may provide a part or all of 15.

The data acquisition unit 11 acquires samples of the source domain from the source data set 2 and samples of the target domain from the target data set 3, respectively. supply to

The source data set 2 is composed of a non-volatile storage device such as a HDD (Hard Disk Drive) or SSD (Solid State Drive), and stores samples (samples) belonging to the source domain of the domain adaptation source. A sample belonging to the source domain is called a source sample. A source sample is teacher data for pre-learning a learning model, and each source sample is labeled with a class indicating correct classification.

Like the source data set 2, the target data set 3 is composed of a non-volatile storage device such as an HDD or SSD, and stores samples belonging to the target domain to which the domain is applied. A sample belonging to the target domain is called a target sample. A target sample is a sample belonging to the same domain as the data to be processed by the task to which the learning model is to be applied. Each target sample may be labeled with a class, but is not necessarily labeled with a class. may
Note that a domain is an area to which data generated from a certain probability distribution belongs. For example, a domain is configured by attributes such as data type, range, and distribution.

The data acquisition unit 11 may acquire the source samples and the target samples by reading the source samples and the target samples previously stored in the source data set 2 and the target data set 3, or acquire the source samples and the target samples. It may be received via the communication I/F from the same or different stored device.

The data acquisition unit 11 also receives input of various parameters necessary for executing domain-adaptive machine learning processing in the learning device 1 . The data acquisition unit 11 may receive input of various parameters via a user interface of a client device communicably connected to the learning device 1 .

The feature extraction unit 12 extracts features of each source sample from the source samples supplied from the data acquisition unit 11 .
The feature extraction unit 12 also extracts features of each target sample from the target samples supplied from the data acquisition unit 11 .
The feature extraction unit 12 supplies the extracted features of the source samples and the features of the target samples to the pseudo sample generation unit 13 .

Based on the features of the source samples and the features of the target samples supplied from the feature extraction unit 12, the pseudo sample generation unit 13 generates target samples of classes that do not appear or are missing in the target samples of the target domain. , generated as pseudo-samples.

In this embodiment, the pseudo sample generation unit 13 maps the features of the source samples and the features of the target samples supplied from the feature extraction unit 12 onto the feature space, and estimates the reliability of the distribution of the target samples on the feature space. and augment the target domain's target samples by generating pseudo-samples based on the estimated confidence. The pseudo sample generator 13 may further estimate the reliability of the source sample distribution on the feature space and generate pseudo samples based on the estimated reliability.
The details of the pseudo sample generation processing executed by the pseudo sample generation unit 13 will be described later with reference to FIG.

The data transformation unit 14 performs data transformation so that the distribution of the features of the source samples of the source domain supplied from the feature extraction unit 12 matches the distribution of the features of the target samples of the target domain. That is, the data conversion unit 14 receives the features of the source sample and the features of the target sample, and performs domain adaptation to convert teacher data to be learned by the learning model from data of the source domain to data of the target domain.

In the present embodiment, the data conversion unit 14 complements the target domain with the pseudo samples generated by the pseudo sample generation unit 13, inputs the features of the target samples in the target domain in which the pseudo samples are complemented, and converts the source domain to the target. Perform domain adaptation to the domain.
The data conversion unit 14 uses the converted teacher data (learning data) to machine-learn the parameter values of the domain adaptation function in the learning model.

The inference unit 15 outputs various processing results for the input data using a learned learning model to which domain adaptation has been applied by the data conversion unit 14 .
In the machine-learned learning model that has been trained in this way, the target samples generated as pseudo samples in the target domain complement the target samples of the non-appearing classes, so the decrease in accuracy in various inference processes is effectively reduced. prevented.

<Domain adaptation and pseudo sample generation>
FIG. 2 is a diagram for explaining the asymmetry (domain shift) of the sample distributions of the source domain and the target domain according to this embodiment.
FIG. 2(a) shows the distribution of feature values P _s (x) of source samples belonging to the source domain and the distribution of feature values P _t (x) of target samples belonging to the target domain. As shown in FIG. 2(a), the distribution of the feature values of the target sample does not match the distribution of the feature values of the source samples, and the distribution of the feature values of the source samples is positively covariate with the distribution of the feature values of the source samples. (covariate) shifted (P _s (x)≠P _t (x)).

FIG. 2(b) shows the distribution by class of source samples belonging to the source domain (P _s (y)) and the distribution by class of target samples belonging to the target domain (P _t (y)). In FIG. 2(b), it is assumed that both the source domain and the target domain have two classes (-1, +1).
As shown in FIG. 2(b), in the source domain, the number of source samples labeled with class (-1) and the number of source samples labeled with class (+1) are almost the same. On the other hand, in the target domain, the number of target samples labeled with class (-1) is approximately the same as the number of source samples with class (-1), whereas the number of target samples labeled with class (+1) is significantly lower than the number of source samples for class (+1), and the target domain also undergoes a shift in class distribution (P _s (y=+1)≠P _t (y=+1)).

As a non-limiting example, domain adaptation is applied to the learning model to obtain a learning model that classifies the input image into either a dog image or a cat image, with the source domain being an illustration image and the target domain being an actual image. Consider the case of
The source sample of the source domain includes both a dog illustration image (P _s (y=−1)) depicting a dog and a cat illustration image (P _s (y=+1)) depicting a cat, Each source sample is labeled with either the dog class (-1) or the cat class (+1).
On the other hand, as shown in FIG. 2(b), the target samples of the target domain are almost all actual dog images (P _t (y=−1)) in which a dog is imaged, and a cat in which a cat is imaged. It is assumed that no or only a very small amount of the photographed image (P _t (y=+1)) was prepared.

In the inference phase that actually uses the learning model, that is, the phase of the task of classifying the input images, it is assumed that not only real images of dogs but also real images of cats will be input. There is a demand for high-precision discrimination from real-world images.
However, as shown in FIG. 2(b), since the target domain lacks the actual cat image (P _t (y=+1)), even if the domain adaptation is applied to the learning model, the actual cat image Therefore, the accuracy of classifying the image by distinguishing it from the actual image of the dog decreases.

On the other hand, in the present embodiment, when the learning model is domain-adapted, a real cat image (P _t (y=+1)) that is missing in the target domain is generated as a pseudo sample, and the generated pseudo sample Complement the target domain with .

FIG. 3 is a diagram for explaining the source domain and target domain that are domain-adapted through the pseudo-sample generation processing executed by the learning device 1 according to this embodiment.
FIG. 3A shows the distribution of the feature values (P _s (x)) of the source samples belonging to the source domain after domain adaptation and the distribution of the feature values (P _t (x)) of the target samples belonging to the target domain. indicates As shown in FIG. 3(a), the distribution of the feature values of the target samples approximately matches the distribution of the feature values of the source samples (P _s (x)≈P _t (x)).

FIG. 3(b) shows the distribution (P _s (y)) of class (+1) of source samples belonging to the source domain and the distribution (P _t (y)) of class (+1) of target samples belonging to the target domain. and
Since the target domain was interpolated with pseudo-samples generated for class (+1) during domain adaptation, the number of source samples labeled to class (+1) in the source domain and the target The number of target samples labeled with class (+1) in the domain is almost the same (P _s (y=+1)≈P _t (y=+1)).
As shown in FIGS. 3(a) and 3(b), according to the present embodiment, not only the shift in feature value distribution but also the shift between classes that can occur between the source and target domains is eliminated. obtain.

<Module configuration of learning model for machine learning>
FIG. 4 is a conceptual diagram showing an example of a module configuration and a schematic processing procedure when the learning device 1 according to this embodiment is implemented in a machine learning model.
Referring to FIG. 4 , learning device 1 may be configured from feature extraction modules 121 and 122 , encoding module 141 , pseudo sample generation module 13 , classifier module 142 and data conversion module 143 . Among the modules shown in FIG. 4, the encoding module 141, the pseudo-sample generation module 13, the classifier module 142, and the data conversion module 143 constitute the domain adaptation module 14 following the feature extraction modules 121 and 122. FIG.

FIG. 4 illustrates an example of training a learning model that recognizes and classifies an input image.
Note that the feature extraction modules 121 and 122 in FIG. 142 and the domain application module 14 including the data conversion module 143 respectively correspond to the data conversion unit 14 of the learning device 1 .

The feature extraction module 121 takes as input the source images of the source domain, extracts features from each source image, and outputs the features of the source images.
The feature extraction module 122 takes as input the target images of the target domain, extracts features from each target image, and outputs the features of the target image.
Note that when the learning model is learned, the feature extraction modules 121 and 122 may be executed in parallel, or one of the feature extraction modules 121 and 122 may be executed first and then the other.

The feature extraction modules 121, 122 for extracting image features of the source image and the target image may be configured by, for example, a Convolutional Neural Network (CNN).
The feature extraction modules 121, 122 further apply algorithms of data augmentation to the source and target images to center the object to be analyzed (e.g., human) in the images at an appropriate scale. It may be positioned or the background may be removed.
The feature extraction modules 121, 122 further apply an attention mechanism, e.g., Attention Branch Network (ABN), to generate and optimize attention maps in the images from the source and target images. , may weight the extracted image features.

The encoding module 141 of the domain adaptation module 14 encodes the source image features and the target image features output by the feature extraction module into a common feature space.
Here, it is assumed that the source domain includes both the positive class feature vector z _s ⁺ and the negative class feature vector z _s ⁻ (z _s ⁺ , z _s ^- ∈ R ^d ). On the other hand, it is assumed that the target domain contains only negative class feature vectors z _t ⁻ (z _t ⁻ ∈R ^d ). That is, the positive class is a class that has not appeared (unobserved) in the target domain. Assume that these feature vectors input to the encoding module 141 are d-dimensional feature vectors.

The encoding module 141 learns the parameters of the feature space that are domain invariant and may be implemented in a Fully Connected Layer as a learnable mapping function G, for example.
Encoding module 141 outputs encoded feature vectors ̂z _s ⁻ , ̂z _s ⁺ , ̂z _t ⁻ . Let these encoded feature vectors be m-dimensional (m<d) feature vectors (̂zεR ^m ).

Pseudo-sample generation module 13 takes as input the encoded feature vectors ̂z _s ⁻ , ̂z _s ⁺ , ̂z _t ⁻ mapped to a common feature space to generate positive ) Generate class pseudo-samples, and consider the generated positive-class pseudo-samples as the positive-class feature vector ̂z _t ⁺ to impute the samples of the target domain.
In FIG. 4, the pseudo sample generation module 13 executes the pseudo sample generation processing after the encoding module 141 encodes the feature vectors. It may be performed before the encoding of feature vectors by module 141 .
Details of the pseudo sample generation processing by the pseudo sample generation module will be described later with reference to FIG.

At S5, the classifier module 142 (discriminator) of the domain adaptation module classifies the encoded feature vector ̂z (̂zεR ^m ).
The classifier module 142 includes a domain classifier (C_d) that classifies the input encoded feature vector into either a source domain or a target domain, and a classifier that classifies the input encoded feature vector into either a positive class or a negative class. and a class classifier (C_c) for classifying into one class.

The classifier module 142 may be implemented, for example, in a Fully Connected Layer as a learnable mapping function C, which performs classification by mapping ^Rm to ^Rc . In the above domain classifier and class classifier, c=2.
The classifier module 142 uses a loss function Lc to minimize the loss shown in Equation 1 below, such that the binary cross entropy loss between domains and classes is smaller. Machine learning can maintain classification performance.

（式１）

(Formula 1)

は、ｉ番目のソースサンプルのバイナリラベルを示し、

は、指標関数である。なお、分類器モジュールは、上記式１で、バイナリクロスエントロピー損失に替えて、二乗誤差等、他の損失を算出してもよい。 here,

denotes the binary label of the i-th source sample, and

is the index function. Note that the classifier module may calculate other losses, such as squared error, instead of the binary cross-entropy loss in Equation 1 above.

The data transformation module 143 of the domain adaptation module 14 transforms the encoded feature vector representation ̂z (̂z∈R ^m ) such that data discrepancy between the source and target domains is minimized. , into a real number z (zεR). That is, the data transformation module 143 is a module (domain critical) that evaluates domain adaptation.
Data transformation module 143 may be implemented in a fully connected layer as a learnable transformation function F, for example.

Specifically, the data transformation module 143 converts the encoded source domain feature vectors ̂z _s ⁻ , ̂z _s ⁺ and the encoded target domain feature vectors ̂z _t ⁻ and pseudo-sample positive ^Given _the ^class _feature ^vectors ^ ^z _t ⁺ _{and the} The coded feature vectors in the source domain are domain-adapted to the coded feature vectors in the target domain by estimating the distance in the common feature space and machine learning to minimize this distance.
This distance may be, for example, the Wasserstein distance as the distance between probability distributions on the metric space, but the data conversion module 143 may use other distances.
The data transformation module 143 uses a loss function Lw to minimize the loss shown in Equation 2 below, for example, so that the distance between the sample distributions between the source domain and the target domain is less lossy. Perform domain adaptation by performing machine learning.

（式２）

(Formula 2)

where _ns denotes the number of positive and negative class samples in the source domain, and _nt denotes the number of positive and negative class samples in the target domain.
In this embodiment, since the encoded feature vector of the pseudo-positive samples generated by the pseudo-sample generation module 13 is added to the target domain, the data conversion module 143 performs the encoding of the pseudo-positive samples added to the target domain. Feature vectors can be used to perform domain adaptation with high accuracy.

Note that the classifier module 142 and the data conversion module 143 may be executed in parallel when the learning model is subjected to machine learning. You may Learning by the classifier module 142 and learning by the data transformation module 143 may be performed as adversarial learning.

The domain adaptation module 14 that trains the learning model performs machine learning so that each parameter of the mapping function G, the mapping function C, and the transformation function F is optimized so that the total loss of the loss function is minimized. Repeat. This learns the parameters of a common feature space that is domain-invariant such that positive and negative sample distributions in the feature space of the source domain are translated into positive and negative sample distributions in the feature space of the target domain with high accuracy. Domain adapted.
Note that the module configuration of the learning model shown in FIG. 4 is an example, and the learning device 1 according to the present embodiment may use other feature extraction and domain adaptation methods.

<Detailed processing procedure of pseudo sample generation processing>
FIG. 5 is a flowchart showing an example of a detailed processing procedure of the pseudo sample generation process executed by the pseudo sample generation unit 13 of the learning device 1 according to this embodiment.
Each step in FIG. 5 is implemented by the CPU reading and executing a program stored in a storage device such as an HDD of the learning device 1 . Also, at least part of the flowchart shown in FIG. 5 may be realized by hardware. When implemented by hardware, a dedicated circuit may be automatically generated on an FPGA (Field Programmable Gate Array) from a program for implementing each step by using a predetermined compiler, for example. Also, a Gate Array circuit may be formed in the same manner as the FPGA and implemented as hardware. Alternatively, it may be realized by an ASIC (Application Specific Integrated Circuit).

In S51, the pseudo sample generator 13 of the learning device 1 pre-learns a domain classifier that classifies samples into either the source domain or the target domain. The domain classifier is trained to give higher weight to samples classified into the target domain than to samples classified into the source domain.
In S51, the pseudo sample generator 13 may further pre-learn a class classifier that classifies samples into either positive class or negative class. The classifier is trained to give higher weight to samples classified into the positive class than to samples classified into the negative class.

In S52, the pseudo sample generator 13 of the learning device 1 estimates the reliability of the negative class samples of the target domain from the distribution of the negative class samples of the target domain on the feature space.
Specifically, the pseudo sample generation unit 13 estimates the mean vector and covariance matrix of the distribution of the negative class samples of the target domain on the feature space, and calculates the distribution of the negative class samples of the target domain. The negative class sample probability value for the distribution is estimated as the confidence score of the negative class sample for the target domain. Here, the distribution of negative class samples can be regarded as a Gaussian distribution (normal distribution).

In S53, the pseudo-sample generation unit 13 of the learning device 1 generates pseudo-positive class samples in the region of the target domain on the feature space.
Assuming that the reliability score of the negative class sample of the target domain on the feature space estimated in S52 is p(D _t ⁻ |x ⁺ ), the reliability of the pseudo positive class sample of the target domain on the feature space is can be estimated as Equation 3 below.
p(D _t ⁺ |x ⁺ )=1−p(D _t ⁻ |x ⁺ ) (equation 3)

Specifically, the pseudo sample generating unit 13 uniformly distributes the target domain around the region where the reliability of the negative class samples of the target domain is low, based on the gradient on the feature space of the reliability of the negative class samples of the target domain. Generate a pseudo-positive class sample.

The pseudo-sample generator 13 may generate pseudo-positive class samples in regions of the target domain determined based on the mean and standard deviation of the inter-class distances of the positive class samples and the negative class samples in the source domain.
That is, the inter-class distance of positive class samples and negative class samples in the source domain can be considered equal to the inter-class distance of positive class samples and negative class samples in the target domain. Therefore, the pseudo sample generation unit 13 may generate the pseudo positive class samples of the target domain in a region separated by the interclass distance from the region where the negative class samples of the target domain are distributed.

The pseudo sample generator 13 may also generate the same number of pseudo positive class samples in the region of the target domain as the number of positive class samples in the source domain (N _t ⁺ =N _s ⁺ ). A domain classifier (C_d) classifies the generated pseudo-positive class samples into the target domain.

In S54, the pseudo sample generation unit 13 of the learning device 1 estimates the mean vector and covariance matrix of the distribution of the positive class samples of the source domain on the feature space, and the positive class sample probability for the distribution of the positive class samples of the source domain. Estimate the value as the confidence of the positive class samples in the source domain. Here, the distribution of positive class samples can also be regarded as a Gaussian distribution (normal distribution). Similar to S53, the pseudo sample generation unit 13 uniformly generates pseudo samples of the target domain around the region where the reliability of the positive class samples of the source domain is low based on the gradient of the reliability of the positive class samples of the source domain on the feature space. A positive class sample may be generated.
The class (content) classifier (C_c) may use the confidence p(D _s ⁺ |x ⁺ ) of the positive class samples of the source domain to update the confidence of the pseudo-positive class samples.

In S55, the pseudo sample generation unit 13 of the learning device 1 uses the domain classifier and class classifier trained in S51 to calculate the reliability of the negative class samples of the target domain and the reliability of the positive class samples of the source domain. Combine to update sample weights. The confidence of the negative class samples of the target domain is transformed into the confidence of the positive class samples of the target domain as shown in Equation 3.
Here, samples that fall into the positive class (y=+1) have a higher weight. It is also assumed that the distribution of positive samples in the source domain (D _s ⁺ ) and the distribution of positive samples in the target domain (D _t ⁺ ) are conditionally independent, as shown in Equation 4 below.
p(D _s ⁺ , D _t ⁺ |x ⁺ )=p(D _s ⁺ |x ⁺ )p(D _t ⁺ |x ⁺ ) (Formula 4)

In S56, the pseudo sample generator 13 of the learning device 1 calculates the parameter distribution of the samples given higher weights in S55, and resamples the pseudo positive class samples of the target domain on the feature space.
Specifically, the pseudo-sample generation unit 13 uses the reliability as a weight to increase (up-sample) the number of samples in the area of pseudo-positive class samples with higher reliability, and pseudo-positive class samples with lower reliability. Down-sample the number of samples in the region of class samples.

In S57, the pseudo sample generator 13 of the learning device 1 repeats the processes from S51 to S56 until a predetermined convergence condition is reached.
As a convergence condition, for example, when the processing of S54 to S55 is bypassed and the information of the positive class samples of the source domain is not used, the processing shown in FIG. .
Alternatively, when using the information of the positive class samples of the source domain, the convergence condition may be set by the number of iterations. A convergence condition may be set such that the distance between the distributions and the distance between the distributions of the positive class samples and the negative class samples of the target domain converge within a predetermined threshold. Also, instead of the distance between sample distributions, the distance between sample confidences may be used. Here, Jensen-Shannon divergence, for example, can be used as the distance.

FIG. 6 is a schematic diagram illustrating an example of a procedure for generating pseudo-positive class samples of the target domain from samples of the target domain and source domain on the feature space.
Referring to FIG. 6A, in the feature space, the left side shows the region of the source domain and the right side shows the region of the target domain. A vertical line 61 indicates the boundary between the source and target domains on the feature space defined by the domain classifier.
The region of the source domain contains the distribution of the negative class samples denoted by (-) and above the distribution of the negative class samples the distribution of the positive class samples denoted by (+). On the other hand, the region of the target domain contains the distribution of negative class samples indicated by (-), but the distribution of positive class samples does not appear.

Referring to FIG. 6(b), the pseudo-sample generator 13 of the learning device 1 generates pseudo-positive class samples in the region of the target domain on the feature space. In the region of the target domain on the right side of the boundary 61, the distribution of negative class samples indicated by (-) indicates that the confidence of the negative class samples of the target domain is high.
Pseudo-sample generator 13 generates pseudo-positive class samples of the target domain, since the further away from the distribution of the negative class samples indicated by (-) of the target domain, the lower the reliability of the negative class samples of the target domain is. is determined to be an area with high reliability, and a plurality of areas 63 to 67 of pseudo-positive class samples are uniformly generated around the distribution of negative class samples indicated by (-) of the target domain (S53 in FIG. 5). .

Referring to FIG. 6(c), the pseudo-sample generator 13 of the learning device 1 resamples pseudo-positive class samples in the region of the target domain on the feature space. A horizontal line 62 indicates the boundary between the positive and negative classes defined by the classifier.

The pseudo sample generating unit 13 generates the positive class sample distribution indicated by (+) in the source domain among the plurality of pseudo positive class sample regions 63 to 67 generated in the target domain in FIG. 6(b). Areas 63 with closer distances are determined to be areas with high confidence of pseudo-positive class samples and are given higher weights.
On the other hand, among the regions 63 to 67 of the plurality of pseudo positive class samples, the regions 64 to 67, which are farther from the distribution of the positive class samples indicated by (+) in the source domain, represent the reliability of the pseudo positive class samples. is considered to be a low region and is given a lower weight. Also, regions 65-67 of pseudo-positive class samples below horizontal line 62 are determined by the classifier to be regions of the negative class, and thus are given a lower weight than regions of pseudo-positive class samples above horizontal line 62. may be added, and the pseudo-positive class samples may be deleted.
The pseudo sample generator 13 may finally generate pseudo positive class samples in the region 63 of pseudo positive class samples for which higher reliability is calculated.

FIG. 7 is a schematic diagram illustrating another example of the procedure for generating pseudo-positive class samples of the target domain from samples of the target domain and the source domain on the feature space.
Referring to FIG. 7A, in the feature space, the left side shows the region of the source domain and the right side shows the region of the target domain. A vertical line 71 indicates the boundary between the source and target domains on the feature space defined by the domain classifier.
The region of the source domain contains the distribution of the negative class samples denoted by (-) and above the distribution of the negative class samples the distribution of the positive class samples denoted by (+). On the other hand, the region of the target domain contains the distribution of negative class samples indicated by (-), but the distribution of positive class samples does not appear. However, unlike FIG. 6(a), the distribution of the negative class samples indicated by (-) in the target domain is (+) across the boundary 71 from the distribution of the negative class samples indicated by (-) in the source domain. are adjacent by the distribution of the positive class samples denoted by .

Referring to FIG. 7(b), the pseudo-sample generating unit 13 of the learning device 1 generates pseudo-positive class samples in the region of the target domain on the feature space. In the region of the target domain to the right of the boundary 71, the distribution of negative class samples indicated by (-) indicates that the confidence of the negative class samples of the target domain is high.
Pseudo-sample generator 13 generates pseudo-positive class samples of the target domain, since the further away from the distribution of the negative class samples indicated by (-) of the target domain, the lower the reliability of the negative class samples of the target domain is. is determined to be an area with high reliability, and a plurality of areas 73 to 77 of pseudo-positive class samples are uniformly generated around the distribution of the negative class samples indicated by (-) of the target domain (S53 in FIG. 5). .

Referring to FIG. 7(c), the pseudo-sample generator 13 of the learning device 1 resamples pseudo-positive class samples in the region of the target domain on the feature space. A diagonal line 72 indicates the boundary between the positive and negative classes defined by the classifier.

The pseudo sample generation unit 13 generates the positive class sample distribution indicated by (+) in the source domain among the plurality of pseudo positive class sample regions 73 to 77 generated in the target domain in FIG. 7(b). Regions 73 with closer distances are determined to be regions with high confidence of pseudo-positive class samples and are given higher weights.
On the other hand, among the regions 73 to 77 of the plurality of pseudo positive class samples, the regions 74 to 77 that are farther from the distribution of the positive class samples indicated by (+) in the source domain are the confidence of the pseudo positive class samples. is considered to be a low region and is given a lower weight. Also, since the regions 75-77 of pseudo-positive class samples below the diagonal line 72 are determined to be negative class regions by the classifier, they are given a lower weight than the region of pseudo-positive class samples above the diagonal line 72. may be added, and the pseudo-positive class samples may be deleted.
The pseudo sample generator 13 may finally generate pseudo positive class samples in the region 73 of pseudo positive class samples for which higher reliability is calculated.

<Hardware configuration of learning device>
FIG. 8 is a diagram showing a non-limiting example of the hardware configuration of the learning device 1 according to this embodiment.
The learning device 1 according to this embodiment can be implemented on any computer, mobile device or any other processing platform, single or multiple.
With reference to FIG. 8, an example in which the learning device 1 is implemented in a single computer is shown, but the learning device 1 according to this embodiment may be implemented in a computer system including a plurality of computers. . A plurality of computers may be interconnectably connected by a wired or wireless network.

As shown in FIG. 8 , the learning device 1 may include a CPU 81 , a ROM 82 , a RAM 83 , an HDD 84 , an input section 85 , a display section 86 , a communication I/F 87 and a system bus 88 . The learning device 1 may also comprise an external memory.
A CPU (Central Processing Unit) 81 comprehensively controls the operation of the learning device 1, and controls each component (82 to 87) via a system bus 88, which is a data transmission line.
The learning device 1 may also include a GPU (Graphics Processing Unit). The GPU has a higher computational capability than the CPU 81, and by operating a plurality of or a large number of GPUs in parallel, particularly for applications such as image processing using machine learning as in the present embodiment, higher processing performance I will provide a. A GPU typically includes a processor and shared memory. Each processor obtains data from a high-speed shared memory and executes a common program to perform a large amount of the same kind of computation at high speed.

A ROM (Read Only Memory) 82 is a non-volatile memory that stores control programs and the like necessary for the CPU 81 to execute processing. The program may be stored in a non-volatile memory such as a HDD (Hard Disk Drive) 84 or an SSD (Solid State Drive) or an external memory such as a removable storage medium (not shown).
A RAM (Random Access Memory) 83 is a volatile memory and functions as a main memory, a work area, and the like for the CPU 81 . That is, the CPU 81 loads necessary programs and the like from the ROM 82 to the RAM 83 when executing processing, and executes the programs and the like to realize various functional operations.

The HDD 84 stores, for example, various data and information necessary for the CPU 81 to perform processing using programs. The HDD 84 also stores various data, information, and the like obtained by the CPU 81 performing processing using programs and the like, for example.
The input unit 85 is composed of a pointing device such as a keyboard and a mouse.
The display unit 86 is configured by a monitor such as a liquid crystal display (LCD). The display unit 86 is a GUI (Graphical User Interface), which is a user interface for inputting instructions to the parameter adjustment device 1 regarding various parameters used in the abnormal scene detection process, communication parameters used in communication with other devices, and the like. ) may be provided.

Communication I/F 87 is an interface that controls communication between study device 1 and an external device.
A communication I/F 87 provides an interface with a network and performs communication with an external device via the network. Various data, various parameters, etc. are transmitted/received to/from an external device via the communication I/F 87 . In this embodiment, the communication I/F 87 may perform communication via a wired LAN (Local Area Network) conforming to a communication standard such as Ethernet (registered trademark) or a dedicated line. However, the network that can be used in this embodiment is not limited to this, and may be configured as a wireless network. This wireless network includes a wireless PAN (Personal Area Network) such as Bluetooth (registered trademark), ZigBee (registered trademark), and UWB (Ultra Wide Band). It also includes a wireless LAN (Local Area Network) such as Wi-Fi (Wireless Fidelity) (registered trademark) and a wireless MAN (Metropolitan Area Network) such as WiMAX (registered trademark). Furthermore, wireless WANs (Wide Area Networks) such as LTE/3G, 4G, and 5G are included. It should be noted that the network connects each device so as to be able to communicate with each other, and the communication standard, scale, and configuration are not limited to those described above.

At least some of the functions of the elements of the learning device 1 shown in FIG. 1 can be realized by the CPU 81 executing a program. However, at least part of the functions of the elements of the learning device 1 shown in FIG. 1 may operate as dedicated hardware. In this case, the dedicated hardware operates under the control of the CPU 81 .

As described above, according to the present embodiment, the learning device extracts the features of the samples of the source domain and the samples of the target domain, respectively, and classifies the classes in the target domain among the plurality of classes labeled with the samples of the source domain. Samples of the under-represented classes are generated as quasi-samples in regions of the target domain in the feature space, and the generated quasi-samples are complemented to the samples of the target domain.
The learning device according to the present embodiment also performs machine learning so that the source domain is domain-adapted to the target domain complemented with the pseudo samples.

Therefore, highly accurate processing results can be obtained even if the samples of the source domain do not sufficiently correspond to the samples of the target domain.
For example, in the task of detecting an anomalous scene that appears only very infrequently in a video, pseudo samples of the anomalous scene to be filtered are generated in the target domain, and the generated pseudo samples are used to detect the target domain. , the asymmetry between the classes of the source and target domains is resolved.
As a result, domain adaptation can be achieved with high accuracy, contributing to improved availability of machine learning models.

It should be noted that although specific embodiments are described above, the embodiments are merely examples and are not intended to limit the scope of the invention. The apparatus and methods described herein may be embodied in forms other than those described above. Also, appropriate omissions, substitutions, and modifications may be made to the above-described embodiments without departing from the scope of the invention. Forms with such omissions, substitutions and modifications are included in the scope of what is described in the claims and their equivalents, and belong to the technical scope of the present invention.

REFERENCE SIGNS LIST 1 learning device 2 source data set 3 target data set 11 data acquisition unit 12 feature extraction unit 13 pseudo sample generation unit 14 data conversion unit 15 inference unit 81 CPU , 82... ROM, 83... RAM, 84... HDD, 85... Input unit, 86... Display unit, 87... Communication I/F, 88... Bus, 121... Source feature extraction module, 122... Target feature extraction module, 141... Encoder (Encoding Module) 142 Classifier 143 Data Conversion Module

Claims (14)

the first class of samples of the target domain based on the features extracted from the first class of samples and the second class of samples of the source domain, and the first class of samples of the target domain; a pseudo-sample generator of class 2, which generates pseudo-samples used for domain adaptation from the source domain to the target domain and complements the target domain with the generated pseudo-samples. An information processing device characterized by: 2. The information processing apparatus according to claim 1, wherein the pseudo sample generation unit generates the pseudo samples based on distribution of samples in the feature space of the extracted features. 3. The information processing apparatus according to claim 2, wherein the pseudo sample generator generates the pseudo samples based on a distribution of samples of the first class included in the target domain in the feature space. . The pseudo sample generator estimates a first reliability of a distribution of samples of the first class included in the target domain in the feature space, and based on the estimated gradient of the first reliability , to generate the pseudo samples. The information processing apparatus according to claim 4, wherein the pseudo sample generation unit generates the pseudo samples in a region of the feature space where the estimated first reliability is low. The pseudo sample generator estimates a second confidence of a distribution of samples of the second class included in the source domain in the feature space, and based on the estimated second confidence gradient , to generate the pseudo samples. The pseudo-sample generation unit combines the first reliability and the second reliability, and generates the generated pseudo-sample based on the combined reliability in the feature space. 7. The information processing apparatus according to claim 6, wherein resampling is performed so as to distribute in an area with higher reliability. The pseudo sample generator generates the pseudo samples based on a distance between a distribution of the first class samples and a distribution of the second class samples included in the source domain in the feature space. The information processing apparatus according to any one of claims 2 to 7, characterized by: further comprising a domain classifier that classifies samples into one of the source domain and the target domain;
The pseudo sample generator trains the domain classifier so that the domain classifier gives higher weight to samples classified into the target domain than samples classified into the source domain. The information processing apparatus according to any one of claims 2 to 8. further comprising a classifier that classifies samples into one of the first class and the second class;
The pseudo sample generation unit learns the class classifier so that the class classifier gives higher weight to samples classified into the second class than samples classified into the first class. The information processing apparatus according to claim 9, characterized by: further comprising a machine learning unit that performs machine learning so that the source domain is domain-adapted to the target domain complemented with the pseudo samples of the second class generated by the pseudo sample generation unit;
The machine learning unit performs machine learning so that at least one of the domain classifier and the class classifier has a smaller cross-entropy loss in the feature space, which is calculated using a first loss function. 11. The information processing apparatus according to claim 10. 3. The machine learning unit performs machine learning so that a Wasserstein distance between the source domain and the target domain in the feature space calculated using a second loss function becomes smaller. 12. The information processing device according to 11. An information processing method executed by an information processing device,
the first class of samples of the target domain based on the features extracted from the first class of samples and the second class of samples of the source domain, and the first class of samples of the target domain; generating pseudo-samples of class 2, which pseudo-samples are used for domain adaptation from said source domain to said target domain;
and complementing the target domain with the generated pseudo-samples. An information processing program for causing a computer to execute information processing, the program causing the computer to:
the first class of samples of the target domain based on the features extracted from the first class of samples and the second class of samples of the source domain, and the first class of samples of the target domain; pseudo-samples of class 2, said pseudo-sample generating process for generating pseudo-samples used for domain adaptation from said source domain to said target domain, and complementing said target domain with said generated pseudo-samples. An information processing program characterized in that it is for executing

Images