CN113657561A - Semi-supervised night image classification method based on multi-task decoupling learning - Google Patents

Abstract

The invention discloses a semi-supervised nighttime image classification method based on multi-task decoupling learning. The daytime labeled samples and the nighttime unlabeled samples are input into a feature extraction network together, and the feature vector extracted from the daytime samples is input into a classification network head , using the cross-entropy loss function for supervision; the feature vector extracted from the night samples is firstly input into the classification network head to obtain pseudo-labels, and then constructs positive and negative sample pairs according to the pseudo-labels and then inputs them into the self-supervised network head, and uses the angle contrast loss function for supervision training; After the multi-task training of the model is completed, a small number of labeled samples in the night data set are input into the feature extraction network and the classification network head, and the iterative self-distillation learning is performed, and finally the night data set can be effectively classified.

Description

A semi-supervised nighttime image classification method based on multi-task decoupling learning

technical field

The invention relates to multi-task learning in the technical field of computer vision recognition, in particular to a semi-supervised night image classification method based on multi-task decoupling learning.

Background technique

Domain transfer is an urgent problem in computer vision, in which the definition of the problem is that the tasks of the source and target domains are the same, and the data are different but related. The core task of this type of learning is to solve the problem of differences in the data distributions of the two domains. Current general image recognition algorithms are trained on supervised datasets and have achieved high performance on images with similar distributions. However, when migrating to images from other target domains, the performance tends to drop dramatically, which is caused by the difference in data distribution between the source and target domains. For example, when a network trained on a daytime dataset predicts nighttime images, the recognition performance tends to drop dramatically.

As we all know, there are a large number of open-source daytime image classification datasets, such as PASCAL VOC, but labeled nighttime image classification datasets are very scarce. Therefore, we hope to train the network with a dataset of daytime images and enable the network to efficiently transfer to nighttime image classification, thereby improving the performance of nighttime image classification.

Self-supervised learning mainly uses auxiliary tasks to mine its own supervision information from large-scale unsupervised data, and trains the network through this constructed supervision information, thereby learning valuable representations for downstream tasks. This learning method is proven to capture discriminative features of images and is an effective solution for tasks lacking labeled data. Self-supervised learning on a large number of unlabeled nighttime images enables the network to learn the feature distribution of nighttime images, thereby improving the accuracy of nighttime image classification.

Therefore, by decoupling the task of nighttime image classification into a supervised classification task of daytime images and a self-supervised task of nighttime images, and performing multi-task learning on the two tasks, the model can not only have the ability to extract various discriminative features , and can also adapt to the data distribution of nighttime images. However, in multi-task learning, there is a competitive relationship between each task. How to make the two tasks promote each other instead of restricting each other requires the design of an effective loss function.

Knowledge distillation has become a hot topic in recent years. Knowledge distillation achieves knowledge transfer by introducing soft targets related to the teacher network as part of the loss to induce the training of the student network. The definition of self-distillation is to learn from oneself to induce the training of the next generation network with soft goals related to oneself. This approach usually enhances the robustness of the network and avoids overfitting, so it can be applied to further improve the performance of the model on nighttime images.

SUMMARY OF THE INVENTION

In order to solve the deficiencies of the prior art and achieve the purpose of improving the performance of image recognition at night, the present invention adopts the following technical solutions:

A semi-supervised nighttime image classification method based on multi-task decoupling learning, including the following steps:

S1, construct a labeled daytime image classification dataset D; construct a nighttime image classification dataset A, in which only some samples of the nighttime images have labels, and the rest of the samples have no class labels;

S2, the labeled samples in the daytime image data set and the unlabeled samples in the nighttime image data set are input into the feature extraction network together, and the daytime image feature vector and the nighttime image feature vector are output; the feature extraction network is a deep residual convolution network. ;

S3, access a multi-task learning network after the feature extraction network layer, the network consists of a supervised classification network head and a self-supervised network head;

损失监督训练；对于夜间图像 特征向量，通过同一分类网络头预测其类别作为伪标签，并根据伪标签构造夜间图像正负 样本对；分类网络头由一个全局平均池化层和全连接层构成； S4, for the daytime image feature vector, through the classification network head

Loss-supervised training; for nighttime image feature vectors, the same classification network head predicts its category as a pseudo-label, and constructs nighttime image positive and negative sample pairs according to the pseudo-label; the classification network head consists of a global average pooling layer and a fully connected layer;

指导特征空间的学习，使正样本相 似，负样本有效区分； S5, the self-supervised network head normalizes the positive and negative samples of the night image according to the weight parameters of the classification network head to obtain the normalized feature vector, and uses the contrast loss

Guide the learning of the feature space, so that the positive samples are similar and the negative samples are effectively distinguished;

S6, performing co-supervised training on the loss supervision training and the contrast loss;

损失监督训练，使分类网络头适应 夜间图像的特征分布；进入自蒸馏学习阶段，进行多次迭代更新，利用前一次

损失监督训 练的分类预测结果作为软目标，与真实标签一同参与监督； S7, input the labeled samples in the night image data set into the trained feature extraction network and the classification network head, fix the weight of the feature extraction network, and carry out the classification network head.

Loss-supervised training to adapt the classification network head to the feature distribution of night images; enter the self-distillation learning stage, perform multiple iterative updates, and use the previous

The classification prediction result of loss supervision training is used as a soft target to participate in supervision together with the real label;

S8, in the inference stage, input the nighttime image to be tested into the trained feature extraction network and classification network head, and output the image classification result.

Further, in the S4, the daytime image feature vector is input into the classification network head, the daytime sample category is output, and the cross-entropy loss function is used to supervise:

表示第i个样本的类别预测概率值。 Among them, N represents the total number of labeled samples in the daytime image dataset, y ⁱ represents the true label of the ith sample,

Represents the class prediction probability value of the ith sample.

Further, in S4, the nighttime image feature vector is input into the classification network head for calculation to obtain the predicted pseudo-label, and the nighttime image positive and negative sample pairs { k , k ₊ , k _- } _m , k ₊ are constructed according to the pseudo-label is the positive sample of k , which belongs to the same label as k , k _- is the negative sample of k , which belongs to a different label from k , and m represents the number of sample pairs.

Further, in the S5, the angle normalization is performed on the positive and negative sample feature pairs:

Among them, x represents the input feature vector, ||x|| represents the modulo length of the feature vector x, y represents the label to which the vector x belongs, W _y represents the parameter of the yth row of the fully connected layer in the classification network head; Perform angle normalization calculation on each sample eigenvector in { k , k ₊ , k _- } _m to obtain normalized eigenvectors {Λ k , Λ k ₊ , Λ k _- } _m :

Λ k =Λ( k ,W,y)

Λ k ₊ =Λ( k ₊ ,W,y)

Λ k _- =Λ( k _- ,W,y).

Further, in the S5, the contrast loss is used to guide the learning of the feature space, so that the positive samples are similar and the negative samples are effectively distinguished, and the following loss function is used:

表示相似度函数。 Among them, y ^k , y ^k +, y ^k - represent the true labels of samples k , k ₊ , k _- in a sample pair, respectively, 𝜂 is a hyperparameter, which represents the minimum distance threshold between samples of different classes,

represents the similarity function.

Further, the cosine similarity function is used to compare the similarity of the normalized feature vectors {Λ k , Λ k ₊ , Λ k _- } _m :

为1，负 样本的相似度

为-1。 Among them, A _i and B _i represent the components of the vectors A and B, respectively, and the similarity of the positive samples

is 1, the similarity of negative samples

is -1.

Further, the total loss function of the S6 is:

When the training epoch reaches the specified number of times, stop training.

Further, in the described S7, the samples with labels in the nighttime image data set are input into the trained feature extraction network and the classification network head, the weight of the fixed feature extraction network is fixed, and the cross entropy loss function is used to supervise the classification network head:

表示第i个样本的类别预测概率值。 Among them, N' represents the total number of labeled samples in the night image dataset, y ⁱ represents the true label of the ith sample,

Represents the class prediction probability value of the ith sample.

损 失监督训练的分类预测结果作为软目标

，与真实标签y一同参与监督： Further, in the S7, enter the self-distillation learning stage, perform multiple iterative updates, and use the previous

Loss-supervised training classification predictions as soft targets

, participate in supervision together with the ground truth label y :

Among them, λ represents the proportion of soft target loss, and after several iterations of updates, the self-distillation training is completed.

A semi-supervised nighttime image classification method based on multi-task decoupling learning, the image to be tested is input into the trained feature extraction network and classification network head, and the image classification result is output.

The advantages and beneficial effects of the present invention are:

The present invention first proposes to combine multi-task learning and knowledge distillation to enable night image classification, and use unlabeled images at night for self-supervised learning, so that the network can adaptively learn the feature distribution of night images while learning the features of daytime image categories. ; Carry out self-supervised learning through angle normalization loss function to reduce the competition between self-supervised loss and supervised loss; Through self-distillation method, using a small amount of labeled data at night for distillation learning can avoid network overfitting to the target domain and lose the generalization ability, while appropriately further adapting the model to the nighttime data.

Description of drawings

Fig. 1 is the schematic flow chart of the method of the present invention.

FIG. 2 is a schematic diagram of a multi-task decoupling learning stage in the present invention.

FIG. 3 is an example diagram of a positive and negative sample pair in the present invention.

Figure 4 is a schematic diagram of the self-distillation learning stage in the present invention.

Detailed ways

The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present invention, but not to limit the present invention.

By combining the self-supervised learning of nighttime data and the supervised learning of daytime data, the invention trains a feature extraction network with domain self-adaptation ability, and further self-distills learning of the image recognition network through a small number of labeled samples in the nighttime data set, The classification network head is migrated to the nighttime data distribution features, thereby improving the nighttime image recognition performance.

As shown in Figure 1 and Figure 2, a semi-supervised nighttime image classification method based on multi-task decoupling learning of the present invention includes the following steps:

Step 1: Build a labeled daytime image classification dataset, build a nighttime image classification dataset, in which only a small number of nighttime samples are labeled. This example uses 12 categories in the open source dataset Exclusively Dark (ExDARK), which are bicycles, boats, bottles, buses, cars, cats, chairs, cups, dogs, motorcycles, people and tables. For the above 12 categories, 800 corresponding images were selected from the COCO public data set, respectively, as the daytime image classification data set D. In addition, the ExDARK dataset is divided into 3 parts: 400 images are extracted from each category to construct an unsupervised nighttime image dataset A; 10 images are extracted from each category as a small number of labeled nighttime image datasets T; the last remaining images are used as the night image classification performance validation set V to evaluate the effectiveness of the algorithm;

的输入尺寸，并使用随机裁剪、水平翻转的图像增强技术来扩增样本 多样性。每次输入的白天图像样本batch size为32，夜间图像样本batch_size为32，采用8 卡GPU并行训练； Step 2: Input the labeled image samples in the daytime data set D and the unlabeled image samples in the nighttime data set A into the feature extraction network together, and output the feature vector of each sample data. The feature extraction network is a deep residual convolutional network. In this embodiment, a ResNet50 network is used, and a feature vector with a dimension of 2048 is output in the conv5_x layer. The network uses the

, and use randomly cropped, horizontally flipped image augmentation techniques to amplify sample diversity. The batch size of daytime image samples input each time is 32, and the batch_size of nighttime image samples is 32, and 8-card GPU is used for parallel training;

Step 3, after the feature extraction network layer is connected to a multi-task decoupling learning network, the network consists of a supervised classification network head and a self-supervised network head;

Step 4: Construct the classification network head, which consists of a global average pooling layer and a fully connected layer. This embodiment uses the average_pool layer and a fully connected layer with a dimension of [2048, 12], where 12 is the number of output categories;

如下： Step 4.1: Input the feature vector extracted by the daytime sample in step 2 into the classification network head, select the category corresponding to the highest probability as the category prediction result of the feature point, and use the cross entropy loss function to supervise, the calculation formula

as follows:

表示第i个样本的类别预测概 率值； N represents the total number of samples, y ⁱ represents the true label of the ith sample,

Represents the category prediction probability value of the ith sample;

Step 4.2: Input the feature vector extracted from the night sample in step 2 into the classification network head, obtain the pseudo-label of the sample, and construct a positive and negative sample pair { k , k ₊ , k _- } _m according to the pseudo-label: k ₊ is k Positive samples, that is, belong to the same label as k ; k _- is a negative sample of k , that is, belong to different labels from k , and m represents the number of sample pairs. The specific construction method is as follows: in the 32 nighttime sample vectors, first randomly select a category C1, and randomly pair the samples in this category to obtain a set of positive sample pairs C1{…}, and randomly select 1 from other categories. The samples are combined with the positive sample pairs in C1{…} to obtain multiple positive and negative sample pairs; then a category C2 is selected from the remaining other categories, and the above operations are repeated until 16 positive and negative sample pairs are obtained. For the extreme case of less than 16, that is, all samples are from the same category, this time there is no input from the self-supervised network. So m takes the value 16 in most cases. FIG. 3 is an example of a positive and negative sample pair in this embodiment;

Step 5: Construct the self-supervised network head: Input the positive and negative sample pairs { k , k ₊ , k _- } _m obtained in step 3.2 and the weight parameter W of the classification network head into the self-supervised network head, and firstly normalize the angle of the sample features , its calculation formula is as follows:

x represents the input feature vector, ||x|| represents the modulo length of the feature vector x, y represents the label to which the vector x belongs, and W _y represents the parameter of the yth row of the fully connected layer in the classification network head. Angle normalization can alleviate the competitive relationship between additional tasks and main tasks in multi-task learning tasks, that is, reduce the negative impact of self-supervised tasks on supervised tasks;

Perform angle normalization calculation on the positive and negative samples for each sample feature vector in { k , k ₊ , k _- } _m to obtain the normalized feature vector {Λ k , Λ k ₊ , Λ k _- } _m :

Λ k =Λ( k ,W,y)

Λ k ₊ =Λ( k ₊ ,W,y)

Λ k _- =Λ( k _- ,W,y)

计算公式如下： Step 5.1: Use the cosine similarity function to compare the similarity of {Λ k , Λ k ₊ , Λ k _- } _m , and the similarity function

Calculated as follows:

应为1，负样本 的相似度

应为-1； A _i , B _i represent the components of vectors A and B, respectively, where the similarity of positive samples

Should be 1, the similarity of negative samples

should be -1;

计算公式如下： Step 5.2: Use the contrast loss to guide the learning of the feature space, so that the positive samples are similar and the negative samples are effectively distinguished, and its loss function

Calculated as follows:

y ^k , y ^k +, y ^k - represent the true labels of samples k , k ₊ , k _- in a sample pair, respectively, 𝜂 is a hyperparameter, indicating that the distance between samples of different classes should exceed this value;

Step 6: Use the loss functions of Step 4.1 and Step 5.2 to jointly supervise the training of the feature extraction network and the multi-task decoupling learning network. The total loss function is:

In this embodiment, the SGD optimizer is used, and its initial learning rate is 0.01, and when the training epoch reaches 70, the learning rate is reduced to 0.001. When the training epoch reaches 100 times, stop training;

Step 7: Input a small number of labeled samples in the night dataset into the trained feature extraction network and classification network head, fix the weights of the feature extraction network, and use the cross-entropy loss function to further supervise the training of the classification network head, so that the classification network head adapts The data distribution of nighttime image features is calculated as follows:

表示第i个样本的类别预测概 率值； N' represents the total number of samples, y ⁱ represents the true label of the ith sample,

Represents the category prediction probability value of the ith sample;

，与真实标签y一同参与监督，其损失函数

的计算公式如下： Step 7.1: As shown in Figure 4, in the self-distillation learning stage, the previous classification prediction result is used as a soft target

, participate in supervision together with the real label y, and its loss function

The calculation formula is as follows:

对网络进行反向传播，学习率为0.005，,通过批量梯度下降法不断更新网络参数； λ represents the proportion of soft target loss. In this embodiment, when λ=0.5, the model has the best performance. based on loss function

Backpropagating the network, the learning rate is 0.005, and the network parameters are continuously updated through the batch gradient descent method;

Step 7.2: Repeat step 6.1. After 10 iterations of updating, the difference between the two losses before and after the model is less than 0.1, and the training of the self-distillation network is completed;

Step 8: In the inference stage, the night image to be tested is input into the feature extraction network and the classification network head, and the image classification result is output. The training and inference stages of this example are implemented on the GPU server GEFORCE RTX 2080 Ti.

The invention decouples the task of night image classification into a supervised classification task of daytime images and a self-supervised task of nighttime images, performs multi-task learning and trains a feature extraction network with domain self-adaptive ability, and uses a small amount of labels at night to train a feature extraction network. The sample performs further self-distillation learning on the image recognition network, so that the representation learned by the classification network head is transferred to the nighttime image features, thereby improving the nighttime image recognition performance. The classification performance of the verification data set V used in this embodiment reaches 83.8% based on the ResNet50 network, and the algorithm of the present invention can make the classification performance reach 89.2%, which is 5.4% higher than the baseline accuracy rate, which fully reflects the present invention practical benefits and application value.

The above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it can still be described in the foregoing embodiments. The technical solutions of the present invention are modified, or some or all of the technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims (9)

1. A semi-supervised night image classification method based on multitask decoupling learning is characterized by comprising the following steps:

s1, constructing a daytime image classification data set and a nighttime image classification data set, wherein the daytime image classification data set is sample images with class labels, and only part of the sample images in the nighttime image classification data set are labeled;

s2, inputting the sample images with labels in the daytime image classification data set and the sample images without labels in the nighttime image classification data set into a feature extraction network together, and outputting a daytime image feature vector and a nighttime image feature vector;

s3, accessing a multi-task decoupling learning network after the feature extraction network layer, wherein the network is composed of a supervised classification network head and an automatic supervision network head;

s4, for the feature vector of the daytime image, the step is carried out through a classification network head

Loss supervision training; for the feature vector of the night image, predicting the category of the feature vector as a pseudo label through a classified network head, and constructing a pair of positive and negative samples of the night image according to the pseudo label;

s5, the self-monitoring network head normalizes the positive and negative sample pairs of the night image according to the weight parameter of the classification network head to obtain the normalized feature vector, and adopts the contrast loss

Guiding the learning of the feature space to enable the positive samples to be similar and the negative samples to be effectively distinguished;

s6, the classification network head and the self-supervision network head are subjected to multi-task training;

s7, inputting the sample with the label in the night image data set, and finishing trainingThe weight of the characteristic extraction network is fixed and the operation is carried out by the classification network head

Loss supervision training, so that the classification network head adapts to the feature distribution of the images at night; entering into self-distillation learning stage, performing multiple iteration update on the weight parameters of the classification network head, and utilizing the previous time

Taking the classification prediction result of the loss supervision training as a soft target, and participating in supervision together with a real label;

and S8, in the reasoning stage, inputting the night image to be detected into the trained feature extraction network and classification network head, and outputting an image classification result.

2. The semi-supervised night image classification method based on multi-task decoupled learning as claimed in claim 1, wherein in S4, the feature vectors of the daytime images are input into a classification network header, the predicted sample classes are output, and the monitoring is performed through a cross entropy loss function:

wherein,Nrepresenting the total number of samples tagged in the daytime dataset,y ⁱ is shown asiThe authenticity of the label of the individual specimen,

is shown asiThe class of samples predicts a probability value.

3. The method for semi-supervised night image classification based on multi-task decoupling learning as claimed in claim 1, wherein in S4, the night image feature vector is input into the classification network head for calculation to obtain the predicted pseudo-feature vectorA label, and constructing a positive and negative sample pairing according to the pseudo labelk,k ₊,k _-} _m ，k ₊Is composed ofkA positive sample of (1), andkbelonging to the same label, and the label is a single label,k _-is composed ofkA negative sample ofkBelonging to different labels, and belonging to different labels,mindicating the number of sample pairs.

4. The semi-supervised nighttime image classification method based on multi-task decoupled learning of claim 3, wherein the positive and negative sample feature pairs are angle-normalized in the step S5:

wherein x represents the input feature vector, | | x | | | represents the modular length of the feature vector x, y represents the label to which the vector x belongs, and W represents the label to which the vector x belongs_yA parameter indicating the y-th row of the fully connected layer in the classified network header; checking positive and negative samplesk,k ₊,k _-} _m Carrying out angle normalization calculation on each sample feature vector to obtain normalized feature vector { Lambdak,Λk ₊,Λk _-} _m ：

Λk=Λ(k,W,y)

Λk ₊=Λ(k ₊,W,y)

Λk _-=Λ(k _-,W,y)。

5. The semi-supervised night image classification method based on multi-task decoupling learning as claimed in claim 4, wherein in the step S5, the following contrast loss function is adopted:

wherein, y^k,y^k+,y^kRespectively representing samples of a sample pairk,k ₊,k _-The real label of (a) is,𝜂is a hyper-parameter, representing a minimum threshold value for the distance between samples of different classes,

representing a similarity function.

6. The semi-supervised night image classification method based on multi-task decoupling learning as claimed in claim 5, wherein the normalized eigenvector { Lambda ] is subjected to cosine similarity functionk,Λk ₊,Λk _-} _m And (3) carrying out similarity comparison:

wherein A is _i 、B _i Representing the components of vectors A and B, respectively, with the similarity of the positive samples

1, similarity of negative examples

Is-1.

7. The semi-supervised night image classification method based on multi-task decoupled learning of claim 1, wherein the total loss function of S6 is as follows:

and stopping training when the training reaches the specified times.

8. The semi-supervised night image classification method based on multi-task decoupling learning of claim 1, wherein in S7, the samples with labels in the night image data set are input to the trained feature extraction network and classification network header, the weights of the feature extraction network are fixed, and the classification network header is supervised by using the cross entropy loss function:

wherein,N’indicating the total number of samples with labels in the night image dataset,y ⁱis shown asiThe authenticity of the label of the individual specimen,

is shown asiThe class of samples predicts a probability value.

9. The semi-supervised night image classification method based on multi-task decoupling learning as claimed in claim 1, wherein in step S7, a self-distillation learning stage is entered, and multiple iterative updates are performed, using a previous iteration

Class prediction of loss supervised training as soft target

And a genuine labelyAnd (3) participating in supervision together:

wherein, lambda represents the proportion of the soft target loss, and the self-distillation training is completed after repeated iteration updating.

Images