CN111914929B - Zero sample learning method - Google Patents

Abstract

The present invention provides a zero-sample learning method, which recognizes never-before-seen data categories by performing knowledge transfer from visible category samples to invisible category samples, which mainly includes the following steps: acquiring a training feature data set; building a generative-based network , noise autoencoder, regression network and zero-shot learning model of discriminant network; train generative network and noise autoencoder; train discriminant network; obtain the overall objective function, and iterate to achieve the purpose of algorithm optimization. Through knowledge transfer, the invention fuses two semantic features of attribute and word vector, conducts training under the confrontation mechanism to minimize the distribution difference between real samples and generated samples, and maps visual features to semantic features through regression network, which effectively solves the problem. The problem of neighborhood transfer of model prediction results can identify difficult-to-label examples and reduce the cost of identification.

Description

zero-shot learning method

technical field

The invention relates to a zero-sample learning method, which belongs to the field of pattern recognition.

Background technique

With the development of deep learning, the performance of both computer vision and machine learning methods has been greatly improved, and deep learning models have achieved surprising success in the field of image classification, even comparable to human recognition ability. However, humans have a natural advantage in recognizing novel objects, which humans have only heard of or seen a few times before, or new objects that they have never encountered before. The most fundamental reason for the difference between the two is that deep models rely on fully supervised learning. Training neural networks therefore requires a large amount of annotated data, and in fact, due to the tens of thousands of species in nature, collecting and annotating visual data is cumbersome and expensive. This leads to a new task that can identify unseen class samples by transferring knowledge from visible class samples to unseen class samples to solve the problem of image annotation.

Zero-shot learning is currently receiving more and more attention. In zero-shot learning, it is usually assumed that the set of visible and unseen classes is disjoint. A part of the samples in the feature space are labeled, these samples are called visible class samples, and only the visual instances of the visible class samples are used to train the model. There are also some unlabeled sample instances in the feature space, and these sample categories are called invisible category samples. The feature space is composed of vectors extracted by the samples through the neural network, and each sample belongs to a category. In order to establish the connection between visible class samples and unseen class samples, semantic features are usually introduced from zero-shot learning. In zero-shot learning, attributes are the most commonly used semantic features, but manually annotating visual features for each semantic attribute is a time-consuming and laborious invention. Natural language processing technology uses some semantic features of replaceable attributes (such as word vector, glove) to directly obtain text information from Wikipedia articles, but such semantic features are rough and invisible, so their performance is higher than that of attribute features. poor.

In view of this, it is indeed necessary to propose a zero-shot learning method to solve the above problems.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a zero-sample learning method for recognizing difficult-to-label samples and reducing the recognition cost.

In order to achieve the above purpose, the present invention provides a zero-sample learning method, which is used for knowledge transfer from visible class samples to invisible class samples to identify data classes that have never been seen before, which mainly includes the following steps:

Step 1, obtaining a training feature data set, the training feature data set includes visible class samples, wherein the visible class samples include labels, real visual features and semantic features;

Step 2. Build a zero-sample learning model based on a generative network, a noise autoencoder, a regression network, and a discriminant network, and initialize the generative network, noise autoencoder, regression network, and discriminant network in the zero-shot learning model;

Step 3, train the generation network and the noise autoencoder to generate the first visual feature and the second visual feature respectively, and fuse the first visual feature and the second visual feature into pseudo visual features according to different weights;

Step 4, training the discriminant network to classify the pseudo visual features and the real visual features, and optimize the generation network and the discriminant network through the confrontation mechanism;

Step 5, training a regression network, using the pseudo-visual features as input, to map the pseudo-visual features to semantic features;

Step 6. Add the loss functions of the generative network, the noise autoencoder, the regression network and the discriminant network to obtain the total objective function, and iterate to achieve the purpose of algorithm optimization.

Optionally, in step 1, the labels include quantity labels and category labels, and the semantic features include word vectors and attribute features.

Optionally, in step 2, a feedforward neural network is used for data transfer among the generating network, the noise autoencoder, the regression network and the discriminant network.

Optionally, in step 3, the generating network generates the first visual feature through attribute features and Gaussian random noise; the noise autoencoder generates the second visual feature through word vectors, latent variables and Gaussian random noise.

Optionally, in step 3, the formula for fusing the first visual feature and the second visual feature into a pseudo visual feature is:

是第二视觉特征表示，第一视觉特征和第二视觉特征两个部分的权重之和为1。where x _f is the pseudo visual feature, λ is the corresponding weight, x ₁ is the first visual feature representation,

is the second visual feature representation, and the sum of the weights of the first visual feature and the second visual feature is 1.

Optionally, in step 4, the confrontation mechanism can be expressed as:

α～U(0,1)。where x is the real visual feature, x _f =G(a,w,z),

α～U(0,1).

Optionally, in step 4, the pseudo visual features and the real visual features are both constrained by the least squares loss formula, and the least squares loss formula is:

where x is the real visual feature and x _f is the pseudo visual feature.

Optionally, the overall objective function in step 6 is:

L=L _WGAN +L ₁ +λ ₂ *L _R ,

where _λ2 is a hyperparameter that assigns weights to different parts.

Optionally, in step 6, use Adam as an optimizer to perform algorithm optimization.

Optionally, step 7 is also included, using the generating network trained in step 3 to generate real visual features of samples of invisible categories, and classifying them to test the total objective function in step 6.

The beneficial effects of the present invention are as follows: the present invention integrates two semantic features of attribute and word vector through knowledge transfer, conducts training under the confrontation mechanism to minimize the distribution difference between real samples and generated samples, and maps visual features to Semantic features can effectively solve the problem of neighborhood migration of model prediction results, and can identify samples that are difficult to label, while reducing the cost of identification.

Description of drawings

FIG. 1 is a schematic flowchart of the zero-sample learning method of the present invention.

FIG. 2 is a flow chart of generating a first visual feature in the zero-sample learning method of the present invention.

FIG. 3 is a flow chart of generating a second visual feature in the zero-sample learning method of the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

As shown in Fig. 1, the present invention discloses a zero-sample learning method, which is used for knowledge transfer from visible class samples to invisible class samples to identify data classes that have never been seen before, which mainly includes the following steps:

Step 1, obtaining a training feature data set, the training feature data set includes visible class samples, wherein the visible class samples include labels, real visual features and semantic features;

Step 2. Build a zero-sample learning model based on a generative network, a noise autoencoder, a regression network, and a discriminant network, and initialize the generative network, noise autoencoder, regression network, and discriminant network in the zero-shot learning model;

Step 3, train the generation network and the noise autoencoder to generate the first visual feature and the second visual feature respectively, and fuse the first visual feature and the second visual feature into pseudo visual features according to different weights;

Step 4, training the discriminant network to classify the pseudo visual features and the real visual features, and optimize the generation network and the discriminant network through the confrontation mechanism;

Step 5, training a regression network, using the pseudo-visual features as input, to map the pseudo-visual features to semantic features;

Step 6. Add the loss functions of the generative network, the noise autoencoder, the regression network and the discriminant network to obtain the total objective function, and iterate to achieve the purpose of algorithm optimization.

Steps 1 to 6 will be described in detail below.

In step 1, the training feature dataset is a 2048-dimensional visual feature extracted by a deep convolutional neural network, which is a vector group; the labels include quantity labels and category labels, and the semantic features include word vectors and attribute features. The training feature dataset shows excellent performance under the 2048-dimensional visual function of the top-level pooling unit of the deep convolutional neural network model RstNet101. For the AwA1 and AwA2 databases, in addition to using visual features as semantic features, each category is also represented by using word vectors, where the dimension of each category word vector is 1000. Specifically, it uses natural language processing technology to extract word vectors from each category in a large language corpus. As for attribute features, through continuous-valued semantic attributes, the dimensions are shown in Table 1 below.

Table 1

In step 2, the generative network, noisy autoencoder, regression network and discriminative network use feedforward neural network for data transfer.

Among them, the noise autoencoder has two hidden fully connected layers, 1200 and 600 units respectively, which are implemented by 2048 and 4096 units of hidden fully connected layers; while the discriminant network is only implemented by one hidden fully connected layer of 512 units; The regression network has only one hidden layer, consisting of 600 units. All noise dimensions in the present invention are 100, and pseudo-visual features can be synthesized from two different semantic features respectively, while using regression network and discriminative network to perform semantic reasoning and related constraints.

In step 3, the purpose of the generative network is to learn the probability distribution of the data points in order to collect samples from it for the data augmentation mechanism. As one of the most promising generative models, generative adversarial models have been widely studied.

并且将属性特征a_i∈R^q作为输入，然后输出第一视觉特征。一旦生成网络学习到基于可见类别样本生成第一视觉特征，便可以嵌入属性特征以生成任何看不见的视觉特征类。生成网络可以通过以下优化函数来学习：As shown in Figure 2, given training data D ^tr of the seen classes, it aims to learn a generative network G: Z × C → X, Gaussian random noise

And take the attribute feature a _i ∈ R ^q as input, and then output the first visual feature. Once the generative network has learned to generate the first visual feature based on the visible class samples, attribute features can be embedded to generate any unseen visual feature class. The generative network can be learned by the following optimization function:

where x ₁ =G(z, a _i ) is the i-th generated first visual feature representation in the visual space, with the corresponding visual feature a _i and noise z.

The present invention also uses the noise self-encoder to obtain the second visual feature by using the word vector as a semantic feature. The word vector is another kind of semantic information, and describes the invisible category samples from another angle and supplements the attribute features, and uses the word vector as another kind of semantic information. WAE expands to conditional WAE.

上的各向同性高斯先验分布。具体来说，潜在空间Z中引入了一个判别网络，目标是将从Z:

采样的“假”点与从Q(Z|X)采样的“真”点区分开。其中，解码器用于解码词向量w_i∈R^p和潜在变量Q(Z|X)，然后生成第二视觉特征，其中，附条件的WAE的损失函数定义如下：As shown in Figure 3, given specific conditional information (word vectors), the noisy autoencoder is used to generate a probability distribution Q(Z|X), where Q is the distribution of the latent space and _PZ is Z:

An isotropic Gaussian prior distribution on . Specifically, a discriminative network is introduced into the latent space Z, with the goal of changing from Z:

The "false" points sampled are distinguished from the "true" points sampled from Q(Z|X). Among them, the decoder is used to decode the word vector w _i ∈ R ^p and the latent variable Q(Z|X), and then generate the second visual feature, where the loss function of the conditional WAE is defined as follows:

是视觉空间中第i个生成的第二视觉特征表示，具有相应的词向量w_i。同时选择D_Z(Qz,P_Z)＝D_JS(Q_Z,P_Z)并使用对抗训练进行评估，λ＞0是一个超参数。in,

is the i-th generated second visual feature representation in the visual space, with the corresponding word vector w _i . Also choose D _Z (Qz, P _Z ) = D _JS (Q _Z , P _Z ) and use adversarial training for evaluation, λ>0 is a hyperparameter.

The present invention chooses to fuse the first visual feature and the second visual feature. On the one hand, attribute features contain more effective semantic information than word vectors, based on human experience in learning to recognize new objects. On the other hand, compared with real visual features, the generated pseudo visual features are a large amount of invalid information, which can be removed through feature fusion to maintain the validity of the information.

Based on the above knowledge, it is necessary to assign different weights to the pseudo-visual features generated by attribute features and word vectors. The following is the formula for fusing the first and second visual features into pseudo-visual features:

是第二视觉特征表示，第一视觉特征和第二视觉特征两个部分的权重之和为1。where x _f is the pseudo visual feature, λ is the corresponding weight, x ₁ is the first visual feature representation,

is the second visual feature representation, and the sum of the weights of the first visual feature and the second visual feature is 1.

In step 4, a discriminant network is used to classify the pseudo visual features and the real visual features. The pseudo-visual features can deceive the discriminative network as successfully as possible. As the discriminative network continues to improve its discriminative ability, the generation network and the discriminative network can be optimized through an adversarial mechanism, and the quality of the generated pseudo-visual features is also continuously improved. The present invention uses the improved WGAN for adversarial training, and the adversarial process of training the discriminant network can be expressed as:

α～U(0,1)；等式中的前两个项近似于Wasserstein距离，而第三个项是梯度罚项，强制执行D的梯度沿直线具有单位范数。where x is the real visual feature, x _f =G(a,w,z),

α ~ U(0,1); the first two terms in the equation approximate the Wasserstein distance, while the third term is a gradient penalty term that enforces that the gradient of D has a unit norm along the line.

Although adversarial mechanisms enable generative networks to generate realistic visual features and similar distributions, it is not enough to ensure that fused pseudo-visual features are effective. Therefore, we pass the least squares loss formula for the fused pseudo visual features and real visual features to constrain their distribution:

In step 5, the regression network takes the pseudo-visual features as input, and then converts the pseudo-visual features into semantic features. The generative network and the regression network together form a dual learning framework, so they can learn from each other. In the present invention, the main task is to generate visual features conditioned on class embeddings, while the dual task is to return visual features to the corresponding class semantic space.

The real visual feature sampled from the training feature dataset is x, and the second is the fused pseudo visual feature x _f . With the paired training data (x,a), we can train the regression network with a supervised loss:

In step 6, the loss functions of the generative network, the noise autoencoder, the regression network and the discriminant network are added to obtain the total objective function:

L=L _WGAN +L ₁ +λ ₂ *L _R ,

where _λ2 is a hyperparameter that assigns weights to different parts.

By choosing Adam as the optimizer and setting the parameters β1 and _β2 to ₍ 0.9, 0.999). First train the discriminant network and optimize its parameters, then fix the discriminant network parameters, set the learning rate of the discriminant network to 0.00001; train the generative network, the regression network and optimize its parameters, set the learning rate of the generative network and the regression network to 0.0001, and zero-sample learning All modules in the model were trained with a batch size of 128 by training on each dataset for 1000 epochs, saving model parameters every 10 epochs, and then evaluating on the test set.

Preferably, the present invention further includes step 7, specifically: using the generating network trained in step 3 to generate real visual features of samples of invisible categories, and classifying them to test the overall objective function described in step 6.

After training the model, in order to predict the labels of unseen class samples, a new sample can be first generated for each unseen class sample, and a new classifier can be trained based on the new set of samples containing both visible and unseen class samples. These synthetic samples are then combined with other samples in the training data, after which any new classifier can be trained based on this new dataset of samples containing both visible and invisible classes.

Further, the present invention is compared with 15 methods, including: SSE, LATEM, ALE, DEVISE, SJE, ESZSL, SYNC, SAE, DEM, RelationNet, PSR-ZSL, SP-AEN, CAPD, CVAE, GDAN.

For reasonable comparison with other benchmarks, a simple 1-NN classifier can be applied for testing. By comparing GDFN with state-of-the-art generalized zero-shot learning, the results are shown in Table 2 below.

Table 2 Results of generalized zero-shot learning methods on four benchmark datasets

It can be seen from the results that the present invention achieves good results on the generalized zero-shot learning dataset.

For the CUB dataset, the present invention achieves good results on unseen classes and the highest accuracy on visible classes. This shows that the present invention performs well in terms of harmonic mean, which again shows that the present invention maintains a good balance of predicted images between visible and invisible classes, and the present invention shows better performance compared to previous models .

For AwA2 data, the present invention outperforms state-of-the-art methods such as SP-AEN and PSR-ZSL in unseen class precision and harmonic mean, and shows higher accuracy on unseen class samples.

For the SUN data set, the present invention has high accuracy in the recognition of both visible and invisible class samples, and has obvious progress in the recognition and classification of visible class samples.

For the aPY dataset, the similarity between the attribute variances of irrelevant training images and test images is much smaller than for other datasets, indicating that it is difficult to synthesize and classify unseen classes. Although the prior art has relatively low accuracy for unseen class recognition, good results can be achieved when testing this dataset using the present invention. The prior art has higher accuracy for the identification of visible class samples, while the present invention has higher accuracy for visible class samples, and achieves a balance between visible and invisible classes, providing accurate aPY harmonics Average precision.

To sum up, the present invention uses knowledge transfer, fuses two semantic features of attribute and word vector, conducts training under an adversarial mechanism to minimize the distribution difference between real samples and generated samples, and maps visual features to semantic features through a regression network. , which effectively solves the problem of neighborhood migration of model prediction results, and can identify samples that are difficult to label, while reducing the cost of identification.

The above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be modified or equivalently replaced. Without departing from the spirit and scope of the technical solutions of the present invention.

Claims (10)

1. A zero sample learning method for knowledge migration from visible class samples to invisible class samples to identify never seen data classes, comprising essentially the steps of:

step 1, obtaining a training feature data set, wherein the training feature data set comprises visible class samples, and the visible class samples comprise labels, real visual features and semantic features;

step 2, building a zero sample learning model based on the generation network GAN, the noise self-encoder WAE, the regression network and the discrimination network, and initializing the generation network, the noise self-encoder, the regression network and the discrimination network in the zero sample learning model;

step 3, training a generating network and a noise self-encoder to respectively generate a first visual feature and a second visual feature, and fusing the first visual feature and the second visual feature into a pseudo visual feature according to different weights;

step 4, training a discrimination network to classify the pseudo visual features and the real visual features, and optimizing and generating the network and the discrimination network through a countermeasure mechanism;

step 5, training a regression network, and taking the pseudo-visual features as input so as to map the pseudo-visual features to semantic features;

and 6, adding the loss functions of the generation network, the noise self-encoder, the regression network and the discrimination network to obtain a total objective function, and iterating to achieve the purpose of algorithm optimization.

2. The zero-sample learning method according to claim 1, characterized in that: in step 1, the labels include quantity labels and category labels, and the semantic features include word vectors and attribute features.

3. The zero-sample learning method according to claim 1, characterized in that: in step 2, a feedforward neural network is used for data transmission among the generation network, the noise self-encoder, the regression network and the discrimination network.

4. The zero-sample learning method according to claim 2, characterized in that: in step 3, the generation network generates a first visual feature through an attribute feature and Gaussian random noise; the noise autoencoder generates a second visual feature by a word vector, a latent variable, and gaussian random noise.

5. The zero-sample learning method according to claim 1, wherein in step 3, the formula for fusing the first visual feature and the second visual feature into the pseudo visual feature is as follows:

wherein x is _f Is a pseudo-visual feature, λ is the corresponding weight, x ₁ Is a representation of the first visual characteristic,

is a second visual feature representation, the sum of the weights of the two parts of the first visual feature and the second visual feature being 1.

6. The zero-sample learning method according to claim 5, wherein in step 4, the countermeasure can be expressed as:

where x is the true visual feature, x _f ＝G(a,w,z)，

α～U(0,1)。

7. The zero-sample learning method of claim 6, characterized in that: in step 4, the pseudo visual features and the real visual features are both constrained in distribution by a least square loss formula, wherein the least square loss formula is as follows:

where x is the true visual feature, x _f Is a pseudo-visual feature.

8. The zero-sample learning method of claim 7, wherein: the total objective function in step 6 is:

L＝L _WGAN +L ₁ +λ ₂ *L _R ,

wherein λ is ₂ Is a hyper-parameter that assigns weights in different parts.

9. The zero-sample learning method according to claim 1, characterized in that: in step 6, algorithm optimization is performed using Adam as an optimizer.

10. The zero-sample learning method according to claim 1, characterized in that: and 7, using the generation network trained in the step 3 for generating the real visual features of the invisible class samples, and classifying the invisible class samples to test the total objective function in the step 6.

Images