CN113657455A - Semi-supervised learning method based on triple network and labeling consistency regularization - Google Patents

Abstract

The invention discloses a semi-supervised learning method based on triple network and label consistency regularization, comprising the following steps: step 1, inputting an image data set and its corresponding label set; Perform preprocessing; step 3, build and train a deep network of adaptive vision mechanism to extract the depth features of the image data set; step 4, build a twin network, use the depth features of labeled data and unlabeled data to obtain forward propagation Results and pseudo-labels; Step 5: Use the forward propagation results and pseudo-labels to construct a loss function for training labeled data and unlabeled data, and perform semi-supervised training on the twin network; the present invention constructs a triple network for data sets with insufficient labeled data. For training, first build a generative adversarial network with an adaptive vision mechanism to perform unsupervised learning on image data sets for more effective feature extraction, which can eliminate the difference between heterogeneous labeled data and unlabeled data feature extraction; then, based on label consistency The principle of establishing and training a twin network can eliminate the difference between the characteristics of the same labeled data and unlabeled data, reduce the number of network training parameters, and more effectively use unlabeled data for semi-supervised learning.

Description

A Semi-Supervised Learning Method Based on Triple Network and Label Consistency Regularization

technical field

The invention belongs to the technical field of deep learning, and in particular relates to a semi-supervised learning method based on triple network and label consistency regularization.

Background technique

In recent years, the deep learning method has made rapid development in the field of artificial intelligence. Deep learning technology is a branch of machine learning, which is an algorithm that uses artificial neural network as the architecture to perform representation learning on data. The application of deep learning methods usually uses a large amount of labeled data to learn fully supervised models, and has achieved good application results. However, such fully supervised learning is expensive and time-consuming, and data labeling must be done manually by researchers with relevant domain expertise. Due to the characteristics of high intra-class diversity and high inter-class similarity in some image datasets, it is difficult to accurately label the data.

Therefore, for different deep learning tasks in practical scenarios, due to the diversity of data sources, usually only a subset of the training set has labels, and the rest of the data does not have labels. This happens in a variety of tasks, especially for image multi-classification tasks. In the case of insufficient annotation and supervision information, the model cannot be adequately fitted, resulting in differences in extracting the features of labeled data and unlabeled data.

Data labeling has always been a key research area of computer vision and artificial intelligence. In order to improve the efficiency of deep learning models, it is necessary to eliminate the differences in model feature extraction by insufficiently fitted models, and to study semi-supervised multi-classification techniques for labeling consistency.

There is a difference between the existing technology when extracting labeled data and unlabeled data. Therefore, a semi-supervised learning method with consistent labeling is urgently needed, which can fully learn the unlabeled data in the data set and label the depth of insufficient data for subsequent actual scenes. Facilitates learning multi-classification tasks.

SUMMARY OF THE INVENTION

The technical problem to be solved by the present invention is to provide a semi-supervised learning method based on triple network and label consistency regularization, which is simple in structure and reasonable in design, aiming at the deficiencies in the prior art.

Considering the fact that the actual scene has the characteristics of incomplete information, the acquired data is generally lack of labeling, resulting in a serious lack of supervision information, resulting in differences between the extraction of labeled data and unlabeled data, limiting the training effect and generalization of deep learning network classification. ability. In order to solve the above-mentioned technical problems, the technical solution adopted in the present invention is: a semi-supervised learning method based on triple network and label consistency regularization, which is characterized in that: it includes the following steps:

Step 1. Input the image dataset and its corresponding label set:

Step 101. Input image data set V, specifically, V={v ₁ ,...v _i ,...v _l }, divided into labeling data X={x ₁ ,...x _f ,... x _n } and unlabeled data U={u ₁ ,...u _j ,... _um }, where v _i represents the i-th image sample data, 1≤i≤l, 1≤f≤n, 1≤j≤m, and l=m+n, n, m and l are all positive integers;

Step 102: Input the label set corresponding to the image set V, the labels of the labeling data X={x ₁ ,...x _i ,...x _n } are p={p ₁ ,...p _i ,... p _n }, the unlabeled data U={u ₁ , . . . u _j , . . . _um } has no labels.

Step 2: Preprocess the labeled and unlabeled datasets:

Step 201 , performing data enhancement on the image labeled data X and the unlabeled data U, wherein, for the labeled data, a single enhancement is performed to obtain the enhanced data X′. For unlabeled data, perform K random enhancements to obtain the enhanced data U';

Step 202: Mix the data X' and U', and arrange them randomly to obtain a data combination W, wherein the label of the enhanced data is consistent with the original label.

Step 3. Build and train a deep network of adaptive vision mechanism to extract the deep features of the image dataset:

Step 301: Build a generative adversarial network G, which is divided into a data generator and a discriminator.

Step 302 , using a self-convolution layer in the generative adversarial network, and setting an adaptive convolution kernel generating function based on the principles of spatial specificity and frequency domain independence. According to the input image features, output the convolution kernel with the same size as the feature map, control the scaling ratio to adjust the parameter amount, and scale the feature map channel.

Step 303: Delete the labels of the labeled data in the image set V, and use all the unlabeled image sets V to perform unsupervised learning on the generative adversarial network G, so that the pseudo data features generated by the generator are close to the real image features, and self-convolution is used. layer to enhance the feature representation capability of the discriminator.

Step 304: Use the trained discriminator G _d of the generated adversarial network G as the feature extractor F _d for extracting the target image annotation data X={x ₁ ,...x _f ,...x _n } and no The depth features x _labeled =F _d (x _f ) and x _unlabeled =F _d (u _j ) of the labeled data U={u ₁ ,...u _j ,... _um }.

Step 4. Build a twin network, and use the depth features of labeled data and unlabeled data to obtain forward propagation results and pseudo-labels:

Step 401, construct two shallow classification networks Net ₁ and Net ₂ , as twin networks, input data combination W;

其中，p_d1和p_d2为Net₁与Net₂的组合预测，w为超参数；Step 402: For the labeled data, input the enhanced data X' and the corresponding label p to obtain the depth feature x _labeled =F _d (X'), use the twin network for prediction, and the forward propagation result is:

Among them, p _d1 and p _d2 are the combined predictions of Net ₁ and Net ₂ , and w is a hyperparameter;

其中，

与

为孪生网络对无标注数据的预测，θ为网络训练参数。Step 403: For the unlabeled data, input the enhanced data U' to obtain the depth feature x _unlabeled =F _d (U'), use the twin network for prediction, and use the output weighted average as the forward propagation result p _n ,

in,

and

is the prediction of the Siamese network on unlabeled data, and θ is the network training parameter.

T是锐化参数，K是增强次数，P(U；θ) 是网络对每个类别的预测概率。Step 404: Sharpen the prediction of the unlabeled data to obtain a pseudo-label q. Among them, the sharpening operation is specifically

T is the sharpening parameter, K is the number of enhancements, and P(U; θ) is the predicted probability of the network for each class.

其中，

为网络Net₁锐化的伪标签，

为网络Net₂锐化的伪标签，λ服从根据实际数据集设置的概率分布。Step 405: Perform label fusion on the pseudo-label q predicted by the Siamese network. Specifically, the fused pseudo-label is:

in,

Pseudo-labels sharpened for network Net ₁ ,

The pseudo-labels sharpened for the network Net ₂ , λ obey the probability distribution set according to the actual dataset.

Step 5. Use the forward propagation results and pseudo-labels to construct a loss function for training labeled data and unlabeled data, and perform semi-supervised training on the Siamese network:

Step 501: Establish a semi-supervised labeling consistency regularization loss function, calculate the regular term for the difference between labeled data and unlabeled data according to each category, and eliminate the difference between labeled data and unlabeled data in the same category, as shown below:

Among them, num is the number of categories, x _labeled and x _unlabeled are the depth features of image labeled data and unlabeled data, and class-k is the kth category;

Step 502: For the enhanced labeled data X', establish a loss function as follows:

Step 503: For the enhanced unlabeled data U', establish a loss function as follows:

是交叉熵函数，x,p是增强的已标注数据和标签，u,q是增强的未标注数据和伪标签。Among them, |X′| is equal to the number of samples per batch, |U′| is equal to K times the number of samples per batch,

is the cross-entropy function, x, p are the augmented labeled data and labels, and u, q are the augmented unlabeled data and pseudo-labels.

Step 504, the overall loss function L is the weight of the three, as follows:

L=L _X +λ _U L _U +β _U Loss _{semi-supervised}

Among them, λ _U and β _U are hyperparameters. Using the overall loss function L, after continuous iteration, the trained twin network model is classified and tested.

Compared with the prior art, the present invention has the following advantages:

1. The structure of the present invention is simple, the design is reasonable, and the realization, use and operation are convenient.

2. The present invention uses a generative adversarial network using an adaptive vision mechanism to perform unsupervised learning on the data set, and extracts the depth features of the data set from the trained model, which can effectively eliminate the difference between the feature extraction of labeled data and unlabeled data between different categories. The feature extraction and selection are more robust, protect the integrity of image information, and improve the performance of semi-supervised multi-classification.

3. Based on the idea of labeling consistency, the present invention uses a twin network for semi-supervised learning, which can effectively eliminate the difference in feature discrimination between labeled data and unlabeled data between the same categories, and avoid different classification results caused by feature differences, and The amount of training parameters is relatively small and has higher validity and correctness.

In conclusion, the present invention has simple structure and reasonable design. The invention constructs a triple network to train data sets with insufficient labeled data. First, a generative adversarial network with an adaptive vision mechanism is established, and unsupervised learning is performed on the image data set for more effective feature extraction. The difference in feature extraction of labeled data; then, based on the principle of labeling consistency, a twin network is established and trained, which can eliminate the difference in feature discrimination between the same labeled data and unlabeled data, reduce the number of network training parameters, and more effectively use unlabeled data for semi-supervision study.

The technical solutions of the present invention will be further described in detail below through the accompanying drawings and embodiments.

Description of drawings

FIG. 1 is a flow chart of the method of the present invention.

Detailed ways

The method of the present invention will be described in further detail below with reference to the accompanying drawings and the embodiments of the present invention.

It should be noted that the embodiments in the present application and the features of the embodiments may be combined with each other in the case of no conflict. The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

It should be noted that the terminology used herein is for the purpose of describing specific embodiments only, and is not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly dictates otherwise, the singular is intended to include the plural as well, furthermore, it is to be understood that when the terms "comprising" and/or "including" are used in this specification, it indicates that There are features, steps, operations, devices, components and/or combinations thereof.

It should be noted that the terms "first", "second", etc. in the description and claims of the present application and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific sequence or sequence. It is to be understood that data so used may be interchanged under appropriate circumstances such that the embodiments of the application described herein can, for example, be practiced in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "having" and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those expressly listed Rather, those steps or units may include other steps or units not expressly listed or inherent to these processes, methods, products or devices.

For ease of description, spatially relative terms, such as "on", "over", "on the surface", "above", etc., may be used herein to describe what is shown in the figures. The spatial positional relationship of one device or feature shown to other devices or features. It should be understood that spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if the device in the figures is turned over, elements described as "above" or "over" other devices or features would then be oriented "below" or "over" the other devices or features under other devices or constructions". Thus, the exemplary term "above" can encompass both an orientation of "above" and "below." The device may also be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptions used herein interpreted accordingly.

As shown in Figure 1, the present invention comprises the following steps:

Step 1. Input the image dataset and its corresponding label set:

Step 101. Input image data set V, specifically, V={v ₁ ,...v _i ,...v _l }, divided into labeling data X={x ₁ ,...x _f ,... x _n } and unlabeled data U={u ₁ ,...u _j ,... _um }, where v _i represents the i-th image sample data, 1≤i≤l, 1≤f≤n, 1≤j≤m, and l=m+n, n, m and l are all positive integers;

Step 102: Input the label set corresponding to the image set V, the labels of the labeling data X={x ₁ ,...x _i ,...x _n } are p={p ₁ ,...p _i ,... p _n }, the unlabeled data U={u ₁ , . . . u _j , . . . _um } has no labels.

Step 2: Preprocess the labeled and unlabeled datasets:

Step 201 , performing data enhancement on the image labeled data X and the unlabeled data U, wherein, for the labeled data, a single enhancement is performed to obtain the enhanced data X′. For unlabeled data, perform K random enhancements to obtain the enhanced data U';

Step 202: Mix the data X' and U', and arrange them randomly to obtain a data combination W, wherein the label of the enhanced data is consistent with the original label.

Step 3. Build and train a deep network of adaptive vision mechanism to extract the deep features of the image dataset:

Step 301: Build a generative adversarial network G, which is divided into a data generator and a discriminator, wherein the generative adversarial network uses DCGAN, specifically Resnet-18.

Step 302 , using a self-convolution layer in the generative adversarial network, and setting an adaptive convolution kernel generating function based on the principles of spatial specificity and frequency domain independence. According to the input image features, output the convolution kernel with the same size as the feature map, control the scaling ratio to adjust the parameter amount, use the 1x1 convolution kernel to scale the feature map channel, and get the feature map, where the number of output feature channels is (Z *Z*Gs), Z is the size of the subsequent self-convolution kernel, and Gs represents the number of groups of self-convolution operations.

Step 303: Delete the labels of the labeled data in the image set V, and use all the unlabeled image sets V to perform unsupervised learning on the generative adversarial network G, so that the pseudo data features generated by the generator are close to the real image features, and self-convolution is used. layer to enhance the feature representation capability of the discriminator.

Step 304: Remove the fully connected layer from the trained discriminator G _d of the generated adversarial network G, and retain the convolution layer as the feature extractor F _d for extracting the target image annotation data X={x ₁ ,...x _f ,...x _n } and the depth features of unlabeled data U={u ₁ ,...u _j ,...u _m } x _labeled =F _d (x _f ) and x _unlabeled =F _d (u _j ).

Step 4. Build a twin network, and use the depth features of labeled data and unlabeled data to obtain forward propagation results and pseudo-labels:

Step 401, construct two shallow classification networks Net ₁ and Net ₂ , as twin networks, input data combination W, wherein the shallow classification network uses VGG-11;

其中，p_d1和p_d2为Net₁与Net₂的组合预测，w为超参数；Step 402: For the labeled data, input the enhanced data X' and the corresponding label p to obtain the depth feature x _labeled =F _d (X'), use the twin network for prediction, and the forward propagation result is:

Among them, p _d1 and p _d2 are the combined predictions of Net ₁ and Net ₂ , and w is a hyperparameter;

其中，

与

为孪生网络对无标注数据的预测，θ为网络训练参数。Step 403: For the unlabeled data, input the enhanced data U' to obtain the depth feature x _unlabeled =F _d (U'), use the twin network for prediction, and use the output weighted average as the forward propagation result

in,

and

is the prediction of the Siamese network on unlabeled data, and θ is the network training parameter.

T是锐化参数，K是增强次数，P(U；θ) 是网络对每个类别的预测概率。Step 404: Sharpen the prediction of the unlabeled data to obtain a pseudo-label q. Among them, the sharpening operation is specifically

T is the sharpening parameter, K is the number of enhancements, and P(U; θ) is the predicted probability of the network for each class.

其中，

为网络Net₁锐化的伪标签，

为网络Net₂锐化的伪标签，λ～Beta(α，α)，α根据实际数据集设置服从的概率分布。Step 405: Perform label fusion on the pseudo-label q predicted by the Siamese network. Specifically, the fused pseudo-label is:

in,

Pseudo-labels sharpened for network Net ₁ ,

Pseudo-labels sharpened for the network Net ₂ , λ ~ Beta(α, α), α sets the probability distribution obeyed according to the actual dataset.

Step 5. Use the forward propagation results and pseudo-labels to construct a loss function for training labeled data and unlabeled data, and perform semi-supervised training on the Siamese network:

Step 501: Establish a semi-supervised labeling consistency regularization loss function, calculate the regular term for the difference between labeled data and unlabeled data according to each category, and eliminate the difference between labeled data and unlabeled data in the same category, as shown below:

Among them, num is the number of categories, x _labeled and x _unlabeled are the depth features of image labeled data and unlabeled data, and class-k is the kth category;

Step 502: For the enhanced labeled data X', establish a loss function as follows:

Step 503: For the enhanced unlabeled data U', establish a loss function as follows:

是交叉熵函数，x，p是增强的已标注数据和标签，u，q是增强的未标注数据和伪标签。Among them, |X′| is equal to the number of samples per batch, |U′| is equal to K times the number of samples per batch,

is the cross-entropy function, x, p are the augmented labeled data and labels, and u, q are the augmented unlabeled data and pseudo-labels.

Step 504, the overall loss function L is the weight of the three, as follows:

L=L _X +λ _U L _U +β _U Loss _{semi-sup ervised}

Among them, λ _U and β _U are hyperparameters. Using the overall loss function L, after continuous iteration, the trained twin network model is classified and tested.

The above are only the embodiments of the present invention and do not limit the present invention. Any simple modifications, changes and equivalent structural changes made to the above embodiments according to the technical essence of the present invention still belong to the technical solutions of the present invention. within the scope of protection.

Claims (1)

1. A semi-supervised learning method based on a triple network and labeling consistency regularization is characterized by comprising the following steps:

step one, inputting an image data set and a corresponding label set:

step 101, an image data set V is input, in particular V ═ V₁,...v_i,...v_lDividing the data into marked data X ═ X₁,...x_f,...x_nAnd un-annotated data U ═ U₁,...u_j,...u_mIn which v is_iRepresenting ith image sample data, i is more than or equal to 1 and less than or equal to l, f is more than or equal to 1 and less than or equal to n, j is more than or equal to 1 and less than or equal to m, l is m + n, and n, m and l are positive integers;

step 102, inputting a label set corresponding to the image set V, wherein the label data X is { X ═ X }₁,...x_i,...x_nThe label of is p ═ p₁,...p_i,...p_nU ═ U }, no-mark data U ═ U₁,...u_j,...u_mThere is no label.

Step two, preprocessing the labeled and unlabeled data sets:

step 201, performing data enhancement on the image labeled data X and the unlabeled data U, wherein the labeled data is subjected to single enhancement to obtain enhanced data X'. For the data without the label, performing random enhancement for K times to obtain enhanced data U';

step 202, mixing the data X 'and the data U', and randomly arranging to obtain a data combination W, wherein the label of the enhanced data is consistent with the original label.

Step three, constructing and training a depth network of the self-adaptive visual mechanism, and extracting the depth characteristics of the image data set:

step 301, constructing and generating the countermeasure network G, which is divided into a data generator and a discriminator.

Step 302, using a self-convolution layer in the generation of the countermeasure network, and setting a self-adaptive convolution kernel generation function based on the principles of spatial specificity and frequency domain independence. And outputting a convolution kernel with the same size as the feature map according to the input image features, controlling the scaling ratio to adjust the parameter quantity, and scaling the feature map channel.

And 303, deleting labels of the labeled data in the image set V, carrying out unsupervised learning on the countermeasure network G by using all the image sets V without labels, enabling the pseudo data characteristics generated by the generator to be close to real image characteristics, and enhancing the characteristic representation capability of the discriminator by using the self-convolution layer.

Step 304, generating the trained discriminator G of the countermeasure network G_dAs a feature extractor F_dAnd is used for extracting target image labeling data X ═ X₁,...x_f,...x_nAnd un-annotated data U ═ U₁,...u_j,...u_mDepth feature x of_labeled＝F_d(x_f) And x_unlabeled＝F_d(u_j)。

Step four, constructing a twin network, and acquiring a forward propagation result and a pseudo label by using the depth characteristics of the marked data and the unmarked data:

step 401, constructing two shallow classification networks Net₁And Net₂As a twin network, a data combination W is input;

step 402, inputting the enhanced data X' and the corresponding label p for the labeled data to obtain the depth feature X_labeled＝F_d(X'), prediction is carried out by using twin network, and forward propagation results are

Wherein p is_d1And p_d2Is Net₁And Net₂W is a hyperparameter;

step 403, inputting the enhanced data U' for the unmarked data to obtain the depth feature x_unlabeled＝F_d(U') predicting using twin networks, and taking the weighted average of the outputs as the forward propagation result p_n，

Wherein,

and

and theta is a network training parameter for the prediction of the twin network on the unmarked data.

And step 404, sharpening the prediction of the label-free data to obtain a pseudo label q. Wherein the sharpening operation is specifically

T is the sharpening parameter, K is the number of enhancements, and P (U; θ) is the prediction probability of the network for each class.

And 405, performing label fusion on the pseudo label q predicted by the twin network. Specifically, the fused pseudo tag is:

wherein,

for the network Net₁A pseudo-label of the sharpening is generated,

for the network Net₂The sharpened pseudo-label, λ, obeys a probability distribution set from the actual data set.

And fifthly, constructing loss functions of training labeled data and unlabeled data by using the forward propagation result and the pseudo label, and performing semi-supervised training on the twin network:

step 501, establishing a semi-supervised labeling consistency regularization loss function, calculating a regularization term of the difference between labeled data and unlabeled data according to each category, and eliminating the difference between labeled data of the same category and unlabeled data, as follows:

where num is the number of categories, x_labeled、x_unlabeledMarking the depth features of the image marking data and the image unmarked data, wherein class-k is the kth class;

step 502, for the enhanced labeled data X', establishing a loss function as follows:

step 503, for the enhanced unmarked data U', establishing a loss function as follows:

wherein, | X '| equals to the number of samples in each batch, | U' | equals to K times the number of samples in each batch,

is a cross entropy function, x, p are enhanced labeled data and labels, and u, q are enhanced unlabeled data and pseudo labels.

Step 504, the overall loss function L is a weighting of the three, as follows:

L＝L_X+λ_UL_U+β_ULoss_{semi-supervised}

wherein λ is_U、β_UIs a hyper-parameter. And carrying out classification test on the trained twin network model by using the overall loss function L through continuous iteration.

Images