CN117237703A - A zero-sample day and night domain adaptive training method and image classification method based on spectrum analysis - Google Patents

Abstract

The invention discloses a zero sample day-night domain self-adaptive training method based on spectrum analysis, which comprises the following steps: receiving a daytime image training set; acquiring an original training feature diagram according to the daytime image training set; determining an enhanced feature map according to the original training feature map; determining a supervision contrast loss function according to the original training feature map and the enhancement feature map; calculating an output descriptor according to the original training feature map; determining a cross entropy loss function according to the enhancement feature map and the output descriptor; and updating network parameters according to the supervision comparison loss function and the cross entropy loss function to obtain the self-adaptive classification model. The invention also provides a zero-sample day-night domain self-adaptive image classification method based on spectrum analysis, which can remove the influence of generalization irrelevant factors and retain generalization relevant object characteristics so as to improve the adaptation performance of the model in a night domain.

Description

A zero-sample day and night domain adaptive training method and image classification based on spectrum analysis class method

Technical field

The invention belongs to the technical field of deep image recognition, and specifically relates to a zero-sample day and night domain adaptive training method and an image classification method based on spectrum analysis.

Background technique

In practical applications, depth image recognition models are easily affected by illumination changes. For example, when a model trained in daytime scenes is applied to nighttime scenes, the performance of the model usually drops significantly due to the impact of domain shift. Therefore, zero-shot day and night domain adaptation remains an important research area. The goal of domain adaptation is to train the model on the source domain data set so that the model performs well on different but similar target domain data sets. Domain adaptation can save expensive data collection and annotation costs. Existing deep domain adaptation methods can be mainly divided into two categories: methods based on differential optimization and methods based on adversarial learning. The zero-shot day-night domain adaptation task aims to generalize models trained only on daytime data well to the night domain. Compared with traditional domain adaptation, the zero-shot day and night domain adaptation task is more challenging because it cannot access the target domain (night scene) data during the training process. Existing zero-shot day and night domain adaptation methods focus on extracting color-invariant representations to mitigate the low-light impact of night scenes, while ignoring the impact of other factors (e.g., texture and style) on generalization ability, resulting in poor performance of the model. Poor generalization performance leads to poor adaptability of the model in the night domain.

Contents of the invention

In order to solve the above problems existing in the prior art, the present invention provides a zero-sample day and night domain adaptive training method and image classification method based on spectrum analysis. The technical problems to be solved by the present invention are achieved through the following technical solutions:

A zero-sample day and night domain adaptive training method based on spectrum analysis, including the following steps:

Receive daytime image training set;

Obtain original training feature maps according to the daytime image training set;

Determine enhanced feature maps based on the original training feature maps;

Determine a supervised contrast loss function based on the original training feature map and the enhanced feature map;

Calculate an output descriptor based on the original training feature map;

determining a cross-entropy loss function based on the enhanced feature map and the output descriptor;

The network parameters are trained and updated according to the supervised contrast loss function and the cross-entropy loss function to obtain an adaptive classification model.

In one embodiment of the present invention, determining the enhanced feature map based on the original training feature map includes:

Determine the feature map spectrum characteristics according to the original training feature map;

Determine the high-frequency component of the spectrum according to the spectrum characteristics of the characteristic map;

Determine a high-pass feature map according to the high-frequency component of the spectrum;

An enhanced feature map is determined based on the original training feature map and the high-pass feature map.

In one embodiment of the present invention, determining the feature map spectrum characteristics based on the original training feature map includes:

Fourier transform is performed on the original training feature map to obtain the spectrum characteristics of the feature map.

In one embodiment of the present invention, determining the high-frequency component of the spectrum based on the spectrum characteristics of the feature map includes:

The low-frequency component of the spectrum feature of the feature map is removed through a high-pass filter to obtain the high-frequency component of the spectrum.

In one embodiment of the present invention, determining a high-pass feature map based on the high-frequency component of the spectrum includes:

Perform inverse Fourier transform on the high-frequency component of the spectrum to obtain a high-pass feature map.

In one embodiment of the present invention, determining the enhanced feature map based on the original training feature map and the high-pass feature map includes:

Perform batch splicing processing on the original training feature map and the high-pass feature map to obtain a spliced feature map;

The spliced feature map is input into the residual network for conversion, and an enhanced feature map is output.

In one embodiment of the present invention, calculating the output descriptor based on the original training feature map includes:

Determine the dimensionality reduction feature map according to the original training feature map;

Determine a prototype set according to the dimensionality reduction feature map;

Output descriptors are computed based on the set of prototypes.

In one embodiment of the present invention, the expression of the supervised contrast loss function is:

Among them, y _i and y _j represent the labels of anchor sample i and sample j respectively, and B represents the batch size. Represents the number of samples labeled y _j in the batch, l _i≠j and/> Represents a similar index function, s _{i, j} = v _i ^T v _j /||v _i ||||v _j || Represents the cosine similarity between anchor sample i and sample j, v _i and v _j respectively Represents the high-level feature vectors of anchor sample i and sample j, t represents the temperature hyperparameter; the original training feature map is used as the anchor sample, the enhanced feature map is used as a positive sample, and the feature map of a different category from the current input is used as a negative sample .

In one embodiment of the present invention, the expression of the high-pass filter is:

Among them, α and β represent the cut-off threshold for removing low-frequency components, and (u, v) represents the coordinates of the center spectrum.

A second aspect of the embodiment of the present invention provides a zero-sample day and night domain adaptive image classification method based on spectrum analysis, which includes the following steps:

The pre-trained feature extractor extracts features from the data set to be classified and outputs the feature map to be classified;

The prototype compensation module after inputting the feature map to be classified calculates the prototype, and calculates the output descriptor to be classified based on the prototype;

The feature map to be classified and the output descriptor to be classified are spliced, and the spliced feature map to be classified is output;

The spliced feature map to be classified is input into the trained classification network, and the classification result is output;

The pre-trained feature extractor, the trained prototype compensation module and the trained classification network constitute an adaptive classification model trained by the training method provided in the first aspect of the embodiment of the present invention.

Beneficial effects of the present invention:

The present invention reduces the influence of low-frequency components (for example, factors such as texture, style and color of the image) by comparing the original training feature map and the enhanced feature map, which is beneficial to extracting generalized representation. By calculating the prototype from the underlying feature map, it promotes the high-level to contain rich semantic information, which is conducive to enhancing the generalization ability of the model. When night images cannot be accessed during the training process, the classification and detection of the model in the night domain are improved. ability.

The present invention will be further described in detail below with reference to the accompanying drawings and examples.

Description of drawings

Figure 1 is a schematic flow chart of a zero-sample day and night domain adaptive training method based on spectrum analysis provided by an embodiment of the present invention;

Figure 2 is an overall network framework diagram of a zero-sample day and night domain adaptive training method based on spectrum analysis provided by an embodiment of the present invention;

Figure 3 is a schematic diagram of the knowledge filtering network of a zero-sample day and night domain adaptive training method based on spectrum analysis provided by an embodiment of the present invention;

Figure 4 is a visual image of the night domain detection results of the image classification method of the present invention and the prior art;

Figure 5a is a schematic diagram of the feature visualization results of the image classification method of the present invention and the existing technology using t-SNE on the CODaN data set;

Figure 5b is a schematic diagram of the feature visualization results of the image classification method of the present invention and the existing technology using t-SNE on the ShapeNet data set;

Figure 6 is a schematic diagram of the visualization results of the feature map of the CODaN data set using the image classification method of the present invention and the prior art.

Detailed ways

The present invention will be described in further detail below with reference to specific examples, but the implementation of the present invention is not limited thereto.

Embodiment 1

As shown in Figure 1, Figure 2 and Figure 3, this embodiment provides a zero-sample day and night domain adaptive training method based on spectrum analysis, including the following steps:

Step 10: Receive the daytime image training set.

Step 20: Obtain the original training feature map based on the daytime image training set.

In this step, the input daytime image training set is extracted by the pre-trained feature extractor, and the original training feature map is output. Among them, H, W and C respectively represent the height, width and number of channels of the original training feature map. In order to reduce the influence of generalization-irrelevant factors and extract generalized representations, frequency transformation needs to be used to remove the low-frequency components of the extracted original training feature maps.

Step 30: Determine the enhanced feature map based on the original training feature map. Specifically, the enhanced feature map is determined based on the original training feature map. Both this step and step 40 are performed through the comparison filtering module.

Specifically, step 30 includes step 31-step 34:

Step 31: Determine the spectral characteristics of the feature map based on the original training feature map. Specifically, in this step, Fourier transform is performed on the original training feature map F _b to obtain the spectrum characteristics of the feature map. The spectrum characteristics of the obtained feature map can be expressed as follows:

in, Represents the i-th feature map of the original training feature map, i=1,...,C,/> represents the Fourier transform, and (u, v) represents the coordinates of the center spectrum.

Step 32: Determine the high-frequency component of the spectrum based on the spectrum characteristics of the feature map. Specifically, the low-frequency component of the spectrum feature of the feature map is removed through a high-pass filter to obtain the high-frequency component of the spectrum.

In this step, after Fourier transformation, the low-frequency components of the image contain domain-specific information, and the high-frequency components contain domain-invariant information. Next, in order to reduce the impact of low-frequency components, define a high-pass filter to remove low-frequency components. The high-pass filter is:

Among them, α and β represent the cut-off threshold for removing low-frequency components. Then, the high-pass filter removes the low-frequency components of the feature map spectrum features by element-wise multiplication with the feature map spectrum features, and obtains the output spectrum high-frequency components:

Among them, ⊙ represents the element-wise product, Represents the high-frequency component of the spectrum output by the high-pass filter.

Step 33: Determine the high-pass feature map based on the high-frequency components of the spectrum. Specifically, the inverse Fourier transform is performed on the high-frequency components of the spectrum to obtain the high-pass feature map:

in, is the filtered high-pass feature map.

After passing through the high-pass filter, the output feature map contains less generalization-irrelevant background information than the original input feature map, and contains richer object-related content, which is conducive to enhancing the model's ability to extract robust features and improving the model's ability to extract robust features. generalization performance.

Step 34: Determine the enhanced feature map based on the original training feature map and the high-pass feature map. This step includes step 341 and step 342:

Step 341: Perform batch splicing processing on the original training feature map and the high-pass feature map to obtain a spliced feature map.

Step 342: Input the spliced feature map into the residual network for conversion, and output the enhanced feature map F.

F=Φ([F _a , F _b ] _B );

Among them, the residual network Φ consists of four residual network blocks, which are used to transform the connection results, [,] _B represents the splicing operation of the batch size dimension.

Step 40: Determine the supervised contrast loss function based on the original training feature map and the enhanced feature map.

In order to encourage the filtered features to contain more object-related information, supervised contrastive learning (Prannay Khosla et al. "Supervised contrastive learning". In: Advances in neuralinformation processing systems 33(2020), pp.18661-18673 .) to further enhance the filtered features. Specifically, the initially extracted features of the current input and the corresponding filtered features are used as anchor points and positive samples respectively, and the features of different categories from the current input are used as negative samples, and the supervised contrast loss function can be determined:

Among them, y _i and y _j represent the labels of anchor sample i and sample j respectively, and B represents the batch size. Indicates the number of samples labeled y _j in the batch. l _i≠j and/> is a similar indicator function, for example, if i≠j, l _i≠j ∈{0, 1}=1, otherwise l _i≠j ∈{0, 1}=0. s _{i, j} = v _i ^T v _j /||v _i ||||v _j || represents the cosine similarity between anchor sample i and sample j, where v _i and v _j represent anchor points respectively. High-level feature vectors of sample i and sample j. t is the temperature hyperparameter, set to 0.07. The original training feature map is used as the anchor sample, the enhanced feature map is used as the positive sample, and the feature map of a different category from the current input is used as the negative sample.

The supervised contrast loss is used to narrow the semantic gap between the original features and the corresponding filtered features, promote the filtered features to contain rich object-related information, and help alleviate the impact of low light in night scenes.

Step 50: Calculate the output descriptor based on the original training feature map. After obtaining the original training feature map, this step is performed in parallel with steps 30 and 40.

After processing by the comparison filtering module (after processing in steps 30 and 40), the extracted features contain rich object-related information, which improves the generalization ability of the model. However, the comparison filtering module may filter out some discriminative content that is crucial for identifying the corresponding objects. To this end, multiple prototypes can be calculated from the low-level feature map through the prototype compensation module to enhance the semantic information of high-level features. Specifically, step 50 is performed by the prototype compensation module, including steps 51 to 53:

Step 51: Determine the dimensionality reduction feature map based on the original training feature map.

Step 52: Determine the prototype set based on the dimensionality reduction feature map.

Step 53: Calculate the output descriptor according to the prototype set.

Specifically, first use the maximum pooling operation to perform the original training feature map Perform dimensionality reduction to obtain the dimensionality reduction feature map/> Then the dimensionally reduced feature map F _bm is input into the prototype compensation module to calculate a set of prototypes (Aming Wu et al. "Universal-prototype enhancing for few-shot object detection". In: Proceedings of the IEEE/CVF International Conference onComputer Vision. 2021, pp.9567-9576.). These prototypes form a prototype set/> Among them, p _i is the prototype and D is the number of prototypes. Next, descriptors representing image-level information are calculated based on the prototype set P:

in, and/> is the convolution parameter,/> Represents the output descriptor, /> Represents a residual operation that assigns visual features to corresponding prototypes.

Here, the comparison filtering module and the prototype compensation module form a knowledge filtering network.

Step 60: Determine the cross-entropy loss function based on the enhanced feature map and the output descriptor.

In this step, specifically, F and The splicing results are input into the classification network to obtain the predicted probability, and the cross-entropy loss function is calculated.

in, Represents the predicted probability:

Ψ consists of two 1×1 convolutional layers with ReLU activation function for aligned dimensions. Yes/> plastic surgery results. /> and/> are the parameters of the fully connected layer. [,] _C represents the splicing operation of the channel dimension. Through the splicing operation, the descriptor containing the underlying discriminant information/> Fusion into feature F enhances the discriminative ability of F. Cls represents the classification network, which is a fully connected layer, where N is the number of categories in the daytime image training set. By using the prototype compensation module, the model's ability to extract domain-invariant features can be further improved.

Step 70: Train according to the supervised contrast loss function and the cross-entropy loss function to update the network parameters. After the training is completed, an adaptive classification model is obtained.

Specifically, training is performed according to the supervised contrast loss function and the cross-entropy loss function to update the parameters of the contrast filtering module, the prototype compensation module and the classification network. After the training is completed, the trained contrast filtering module, the trained prototype compensation module and the trained Classification network. The trained prototype compensation module, the trained classification network and the pre-trained feature extractor constitute an adaptive classification model.

Specifically, the overall training objective (the overall loss function ) by the cross entropy loss function/> Comparing loss function with supervision/> composition.

The loss function of the model is:

Among them, λ represents the hyperparameter, and setting λ to 1 can balance the two loss terms.

The training method of this embodiment can effectively improve the model's image classification and target detection capabilities in the night domain. The contrast filtering module can effectively remove the influence of irrelevant factors, reduce the influence of low-frequency components (for example, style), and extract domain-invariant features. The prototype compensation module helps the high-level to contain rich semantic information and improve the generalization ability of the model.

It should be noted that the contrast filter module and prototype compensation module need to be used simultaneously during training. The structure of the network model during the training process includes: pre-trained feature extractor, contrast filter module, prototype compensation module and classification network. However, in the actual reasoning process, by discarding the contrast filter module and retaining the prototype compensation module, the model is generalized to the unknown target domain to classify images. That is, after the training is completed, the trained contrast filter module will be removed. If the network is used as an adaptive classification model, the adaptive classification model obtained after training includes: pre-trained feature extractor, trained prototype compensation module and trained classification network.

Embodiment 2

This embodiment provides a zero-sample day and night domain adaptive image classification method based on spectrum analysis, which is applied to the adaptive classification model trained in Embodiment 1. The adaptive classification model includes: a pre-trained feature extractor and a trained prototype. Compensation module and trained classification network;

Specifically, a zero-sample day and night domain adaptive image classification method based on spectrum analysis includes the following steps:

Step 1: The pre-trained feature extractor extracts features from the data set to be classified and outputs the feature map to be classified.

Step 2: The feature map to be classified is input into the trained prototype compensation module to calculate the prototype, and the output descriptor to be classified is obtained based on the prototype calculation.

Step 3: Splice the feature map to be classified and the output descriptor to be classified, and output the spliced feature map to be classified;

Step 4: The feature map to be classified and spliced is input into the trained classification network for classification, and the classification result is output.

As shown in Figure 4, compared with the results of Faster R-CNN (shown in the first row), the image classification method of the present invention (shown in the second row) accurately detects objects in nighttime images.

Figure 5a is a schematic diagram of the results of feature visualization using t-SNE on the CODaN data set by the present invention and the prior art CIConv. Figure 5b is a schematic diagram of the feature visualization results of the present invention and the prior art CIConv using t-SNE on the ShapeNet data set. Schematic diagram of the results of feature visualization by SNE, in which the colored points represent different object categories. It can be seen that the method of the present invention can extract more discriminative features, thereby significantly improving the classification performance.

As shown in Figure 6, compared with the feature map from CIConv (shown in the second column), the features extracted by the method of the present invention (shown in the third column) contain more object-related information and less background information, which helps improve generalization performance.

In addition, the terms “first” and “second” are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the quantity of indicated technical features. Therefore, features defined as "first" and "second" may explicitly or implicitly include one or more of these features. In the description of the present invention, "plurality" means two or more than two, unless otherwise explicitly and specifically limited.

In the description of this specification, reference to the terms "one embodiment," "some embodiments," "an example," "specific examples," or "some examples" or the like means that specific features are described in connection with the embodiment or example. , structures, materials or features are included in at least one embodiment or example of the invention. In this specification, the schematic expressions of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the specific features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, those skilled in the art may join and combine the different embodiments or examples described in this specification.

The above content is a further detailed description of the present invention in combination with specific preferred embodiments, and it cannot be concluded that the specific implementation of the present invention is limited to these descriptions. For those of ordinary skill in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, and all of them should be regarded as belonging to the protection scope of the present invention.

Claims (10)

1. A zero sample day-night domain self-adaptive training method based on spectrum analysis is characterized by comprising the following steps:

receiving a daytime image training set;

acquiring an original training feature map according to the daytime image training set;

determining an enhanced feature map according to the original training feature map;

determining a supervision contrast loss function according to the original training feature map and the enhancement feature map;

calculating an output descriptor according to the original training feature map;

determining a cross entropy loss function from the enhancement feature map and the output descriptor;

and training and updating network parameters according to the supervision comparison loss function and the cross entropy loss function to obtain a self-adaptive classification model.

2. The spectrum analysis-based zero-sample day-night domain adaptive training method according to claim 1, wherein said determining an enhanced feature map from said original training feature map comprises:

determining the spectrum characteristics of the feature map according to the original training feature map;

determining a spectrum high-frequency component according to the spectrum characteristics of the characteristic diagram;

determining a high-pass feature map according to the frequency spectrum high-frequency component;

and determining an enhancement feature map according to the original training feature map and the high-pass feature map.

3. The spectrum analysis-based zero-sample day-night domain adaptive training method according to claim 2, wherein said determining feature map spectrum features from the original training feature map comprises:

and carrying out Fourier transform on the original training feature map to obtain the spectrum features of the feature map.

4. The spectrum analysis-based zero-sample day-night domain adaptive training method according to claim 2, wherein said determining spectral high-frequency components according to the spectral features of the feature map comprises:

and removing the low-frequency component of the spectral characteristics of the characteristic map through a high-pass filter to obtain a spectral high-frequency component.

5. The spectrum analysis-based zero-sample day-night domain adaptive training method according to claim 2, wherein the determining a high-pass feature map according to the spectrum high-frequency component comprises:

and carrying out inverse Fourier transform on the frequency spectrum high-frequency component to obtain a high-pass characteristic diagram.

6. The spectral analysis-based zero-sample day-night domain adaptive training method of claim 5, wherein determining an enhanced feature map from the original training feature map and the high-pass feature map comprises:

performing batch splicing processing on the original training feature map and the high-pass feature map to obtain a spliced feature map;

and inputting the spliced characteristic map into a residual error network for conversion, and outputting an enhanced characteristic map.

7. The spectrum analysis-based zero-sample day-night domain adaptive training method according to claim 1, wherein said calculating an output descriptor from said original training feature map comprises:

determining a dimension reduction feature map according to the original training feature map;

determining a prototype set according to the dimension reduction feature map;

and calculating an output descriptor according to the prototype set.

8. The spectrum analysis-based zero-sample day-night domain adaptive training method according to claim 1, wherein the expression of the supervision contrast loss function is:

wherein y is _i And y _j Labels representing anchor samples i and samples j, respectively, B represents the batch size,indicating the label in the batch as y _j Number of samples of (1) _i≠j And->Representing similar index functions s _i，j ＝v _i ^T v _j /||v _i ||||v _j The i represents cosine similarity between anchor sample i and sample j, v _i And v _j The high-level characteristic vectors of the anchor point sample i and the anchor point sample j are respectively represented, and t represents a temperature super-parameter; the original trainingThe training feature map is used as an anchor point sample, the enhancement feature map is used as a positive sample, and a feature map of a different category from the current input is used as a negative sample.

9. The spectrum analysis-based zero-sample day-night domain adaptive training method according to claim 1, wherein the expression of the high-pass filter is:

where α and β denote cut-off thresholds for removing low frequency components, and (u, v) denote coordinates of the center spectrum.

10. A zero sample day-night domain self-adaptive image classification method based on spectrum analysis is characterized by comprising the following steps:

the pre-trained feature extractor performs feature extraction on the data set to be classified and outputs a feature map to be classified;

the feature map to be classified is input into a prototype compensation module after training to calculate a prototype, and an output descriptor to be classified is obtained according to the prototype calculation;

the feature images to be classified and the output descriptors to be classified are spliced, and the spliced feature images to be classified are output;

inputting the spliced characteristic diagrams to be classified into a trained classification network, and outputting classification results;

the pre-trained feature extractor, the trained prototype compensation module and the trained classification network form the training method according to any one of claims 1-9, and an adaptive classification model is obtained by training.