CN117370594A - Distributed difference self-adaptive image retrieval method based on space-frequency interaction - Google Patents

Abstract

The invention relates to a distributed difference self-adaptive image retrieval method based on space-frequency interaction, which comprises the steps of firstly obtaining an original image for training, and obtaining strong and weak transformation images through data enhancement; then, constructing a deep hash network; the strong transformation image and the weak transformation image are respectively input into a student model and a teacher model to obtain hash quantization codes, and then self-distillation difference quantization loss, hash agent loss and binary cross entropy loss are obtained; then, a distribution migration module is constructed, and the distribution center and the discrete degree of the hash quantization codes extracted by the student model are utilized to migrate the hash quantization codes extracted by the teacher model, so that distribution migration loss is obtained; constructing a frequency component extraction module, extracting frequency domain information of the hash quantization codes through fast Fourier transformation, and extracting frequency components of the hash quantization codes through arc tangent transformation, so as to obtain frequency component loss; and finally, constructing a target optimization function based on all losses, training a student model and a teacher model, and using the trained student model or teacher model for image retrieval. By fully quantizing the difference information between hash codes, the retrieval performance is improved.

Description

Distribution difference adaptive image retrieval method based on space-frequency interaction

Technical field

The invention belongs to the field of image retrieval in information retrieval, and is specifically a distribution difference adaptive image retrieval method based on space-frequency interaction.

Background technique

Image retrieval is a technology that searches for and retrieves matching images through the image to be queried. The main purpose is to match the image that is most semantically related to the image to be queried from a large image database. Image retrieval has wide application value and plays a key role in many fields, including image search engines, medical image analysis, security monitoring, etc. It can help people access and manage image data more easily, thereby improving work efficiency and user experience. Shorten the decision-making process. As the size of the database gradually increases, traversing and searching for the images required in the database requires more manpower and time resources. The images in the database and the image to be queried are represented by semantic features, thereby converting the search problem of the image to be queried in the database into a similarity judgment problem between semantic features, which can greatly improve the retrieval efficiency.

The hash algorithm has significant advantages in speed and storage, so it is widely used in large-scale image retrieval. Hash algorithms are divided into two categories: traditional hashing and deep hashing algorithms. Early hashing algorithms are mostly traditional hashing algorithms, implemented based on image features. The features of the image are extracted through manually designed convolution kernels, and the similarity between features is used. The size of the image is used to determine the matching image in the database and use it as the retrieval result. Compared with early retrieval methods that relied on manually input metadata and tags, traditional hashing algorithms are easier to implement. However, they are limited by manually designed convolution kernels and model depth, and the generated hash codes only contain a small amount of semantic information. With the development of deep learning technology, tremendous progress has been made in the field of image retrieval. With the help of the more powerful representation learning ability of deep neural networks, image retrieval algorithms based on deep learning can obtain feature codes containing more high-level semantic information. Then, in order to achieve faster retrieval, the features extracted by deep neural networks are compressed into In Hamming space, the calculation of similarity between discrete feature quantization codes is converted into the calculation of Hamming distance of binary hash codes.

Currently, distance quantification methods between hash codes include tuple loss and center coding loss. Tuple loss includes pairwise loss and triplet loss. Treat a pair of images as a group, convert the extracted image features into codes, and use the distance between codes as a loss. Since the relationship between the two samples is only similar and dissimilar, although the pair-value loss will make Similar images are close, and dissimilar images are far away. The difference in magnitude between the two will also lead to the problem of imbalance of positive and negative samples. At the same time, it is impossible to obtain intra-class and inter-class relationships. In addition, calculate the difference between any set of images. Encoding distance also has a huge time overhead. Although triplet loss can alleviate the problem of imbalance between positive and negative samples to a certain extent, due to the problem of the number of samples within and between classes, the trained model has a certain bias and cannot obtain the relationship between classes. The class center loss constructs the class center by definition or through clustering in advance, and converts the encoding loss between image pairs into the distance between the image encoding and the class center encoding. Compared with tuple loss, class center loss does not require pairwise calculation of the distance between all samples, which greatly reduces training time. At the same time, due to the relationship between class centers, the learned encoding also has a certain class relationship. .

Existing deep learning-based image retrieval methods focus more on how to better quantify the differences between image encodings, but how to more fully utilize the intra-class and inter-class relationships between images for encoding so that the encoding can more fully solve the problem. Coupling category information and exploiting distribution differences between categories are also important to improve the performance of image retrieval.

Contents of the invention

In view of the shortcomings of the existing technology, the technical problem to be solved by the present invention is to provide a distribution difference adaptive image retrieval method based on space-frequency interaction.

The present invention solves the technical problems and adopts the following technical solutions:

A distribution difference adaptive image retrieval method based on space-frequency interaction, characterized in that the method includes the following steps:

The first step: obtain the original image for training, perform data enhancement on the original image, and obtain a strong transformation image and a weak transformation image;

Step 2: Construct a deep hash network; input the strong and weak transformed images into the student model and teacher model of the deep hash network respectively to obtain the hash quantization code extracted by the student model and the hash quantization code extracted by the teacher model; Based on the hash quantization encoding extracted by the student model and the teacher, the self-distillation difference quantization loss L _Sdh , the hash proxy loss L _HP and the binary cross-entropy loss L _bce-Q are obtained;

L _Sdh = 1-cos (H _T , H _S ) (1)

L _HP =H(y,Softmax(P _T /T)) (4)

In the formula, H _T and H _S represent the hash quantization codes extracted by the teacher model and the student model, P _T is the proxy sample, T represents the temperature scale hyperparameter, and H(·) represents the difference between the real category label and the predicted category label of the image. The quantified error between , y represents the real category label sequence of the image, Indicates that the value of hash encoding is 1, /> Represents the maximum likelihood estimate of the k-th hash code, H _k represents the k-th bit of the hash quantization code H _T , and K represents the coding length;

The third step is to build a distribution migration module. The distribution migration module uses the distribution center and discrete degree of the hash quantization code extracted by the student model to migrate the hash quantization code extracted by the teacher model, and obtains the hash quantization code after distribution migration. Quantify the difference between the hash quantization coding after distribution migration and the hash quantization coding extracted by the teacher model to obtain the distribution migration loss; the distribution migration loss L _DIT is expressed as:

L _DIT =1-cos( _HT , _{HT_S} ) (12)

In the formula, _{HT_S} is the hash quantization code after range-constrained distribution migration;

Step 4: Construct a frequency component extraction module. The frequency component extraction module extracts the frequency domain information of hash quantization coding through fast Fourier transform, and then extracts the frequency component of hash quantization coding through arctangent transformation;

Where, The spatial coordinates of the hash quantization code, x(h, w) represents the value of the hash quantization code at the spatial domain coordinates (h, w), H and W represent the length and width of the hash quantization code;

Among them, PH represents the frequency component of hash quantization coding, R(x′)(u, v) is the real part of hash quantization coding at frequency domain coordinates (u, v), I(x′)(u, v) ) is the imaginary part of the hash quantization code at the frequency domain coordinates (u, v);

The frequency component loss L _ph is expressed as:

L _ph =1-cos (PH _T , PH _S ) (15)

Among them, PH _T represents the frequency component of the hash quantization code _HT , and PH _S is the frequency component of the hash quantization code _HS ;

Step 5: Construct the objective optimization function, train the student model and teacher model, and measure the training loss through the objective optimization function; the objective optimization function is:

Among them, N _B is the total number of samples, λ ₁ , λ ₂ , λ ₃ , and λ ₄ are all weights;

Input the image to be queried into the trained student model or teacher model, and output the retrieved image.

Compared with the prior art, the advantages and beneficial effects of the present invention are:

1. The current image retrieval method based on the self-distillation model transforms the image through data enhancement to change the distribution of the image, and guides the generation of the hash code by quantifying the hash code difference between the image pairs. However, when the two When the distribution difference between images is too large, directly quantifying the hash coding between the two will lead to the problem that the difference information generated by data enhancement cannot be fully utilized. The difference information cannot be fully quantified, which will affect the retrieval performance and reduce the retrieval efficiency. Accuracy, so a distribution migration module is designed. The distribution migration module uses the distribution center and discrete degree of the hash quantization code extracted by the student model to migrate the hash quantization code extracted by the teacher model, and obtains the hash quantization code after distribution migration; By calculating the similarity between the hash quantization code after distribution migration and the hash quantization code extracted by the student model, we can assist in quantifying the difference between the hash quantization code extracted by the teacher model and the student model, so as to make full use of The distribution difference information generated by data enhancement improves retrieval performance.

2. The information transformation of images is often more reflected in the frequency domain space. When quantizing image coding, the current deep hash network only considers the coding quantization in the spatial domain, without considering the relative transformation of the image itself. Therefore, The present invention designs a frequency component extraction module to analyze the frequency components of hash quantization coding. By capturing the relative changes caused by image transformation, the generated image coding can contain higher-level semantic information with more obvious differences. The frequency component extraction module first converts the image coding in the spatial domain into the frequency domain form through fast Fourier transform, and then performs phase analysis through arctangent transformation, and captures the relative changes in the frequency domain by quantizing the phase difference between codes, thereby making it more precise. Fully quantify the differential information produced by data augmentation.

3. The present invention conducted experiments on the single-label data set ImageNet and the multi-label data set MS COCO, NUS-WIDE, and NUS-WIDE_M data sets. Experimental results show that compared with the currently popular image retrieval model DHD, the present invention has improved performance when the coding length is 32, 48, and 64 on a single-label data set with a simpler distribution. When the coding length is 16 Comparable results were obtained. For multi-label data sets, various improvements have been achieved under different encoding lengths. The results show that the method of the present invention has better adaptability and performance in solving the relative changes caused by the self-distillation data enhancement form, and has achieved better results. Good search results.

Description of the drawings

Figure 1 is a schematic structural diagram of the deep hash network of the present invention in the training stage;

Figure 2 is a schematic diagram of the distributed migration module of the present invention;

Figure 3 is a schematic diagram of the frequency component extraction module of the present invention.

Detailed ways

The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific implementation modes, but this does not limit the scope of protection of the present application.

The present invention is a distribution difference adaptive image retrieval method based on space-frequency interaction (referred to as method, see Figures 1 to 3), which includes the following steps:

The first step: obtain the original image, and several original images form a data set;

This embodiment selects four data sets: MS COCO, ImageNet, NUS-WIDE, and NUS-WIDE_M. The ImageNet data set is a single-label data set, and the other three are multi-label data sets. Each image is processed into 256*256 Pixels; each data set is divided into database, training set and test set. The ImageNet data set contains 100 categories, of which the database, training set and test set contain 128503, 13000 and 5000 images respectively; NUS-WIDE and NUS-WIDE The -M data set contains 21 categories, and the database, training set, and test set contain 149,736, 10,500, and 2,100 images respectively; the MS COCO data set contains 80 categories, and the database, training set, and test set contain 117,218, and 10,000 images respectively. , 5000 images.

The original image is data enhanced through random cropping, horizontal flipping, Gaussian blur, brightness, contrast, saturation transformation and other operations, and the strength of the overall image data enhancement is characterized in the form of probability, and strong transformation images and weak transformation images are obtained; in practice In the application, the image to be queried may be transformed, and the image transformation in the actual application process can also be simulated through data enhancement.

Step 2: Build a deep hash network;

Deep-Hash-Distillation (DHD) uses twin network as the basic framework, including student model and teacher model, parameter sharing between student model and teacher model, both student model and teacher model include feature extraction network and coding There are two parts of the generation network. The feature extraction network usually uses ResNet50 or AlexNet network. The features extracted by the feature extraction network are input into the encoding generation network, and the encoding generation network generates hash quantization encoding; the encoding generation network includes a fully connected layer and a layer normalization layer. and tanh activation function. The tanh activation function constrains the value of the hash quantization code obtained by the layer normalization layer to the range of [-1, 1]; input the strong and weak transformed images into the student model and the teacher model respectively to extract their respective Hash quantization coding. For distillation models, fixed single branches can effectively improve the performance of the model and assist the generation of codes by quantifying the distribution difference information between the two. The DHD network transforms the distribution between images by strong and weak Difference realizes the self-distillation difference quantification loss between the student model and the teacher model, calculated through cosine similarity, the formula is as follows:

L _Sdh = 1-cos (H _T , H _S ) (1)

H _T =tanh(h _T ) (2)

H _S =tanh(h _S ) (3)

In the formula, L _Sdh is the self-distillation difference quantization loss between the student model and the teacher model, h _T and h _S represent the hash quantization codes output by the layer normalization layer of the teacher model and the student model, and H _T and H _S represent The hash quantization encoding extracted by the teacher model and the student model, tanh(·) represents the tanh activation function;

In the coding difference quantification stage, the hash quantized code H _T and the proxy sample P _T are judged for similarity. The proxy sample P _T represents the center of the image sample. The calculation formula of the hash proxy loss is as follows:

L _HP =H(y,Softmax(P _T /T)) (4)

Among them, T is the temperature scale hyperparameter, H(·) represents the quantified error between the real category label of the image and the predicted category label. The predicted category label is obtained through the Softmax function, and y represents the real category label sequence of the image;

The deep hashing algorithm reduces the quantization error by minimizing the distance between the hash code and the binary target through regression. Since hash quantization coding quantizes the coding of each bit separately, coding quantization is regarded as binary classification, and the coding result of each bit is predicted by the Gaussian distribution estimator g(h). The formula is as follows:

Among them, m and σ are the mean and standard deviation of the Gaussian distribution estimator 9(h), and the value of m is +1 or -1. When When , m takes +1, when/> When , m takes -1;

To sum up, the calculation formula of binary cross entropy (BCE) loss is as follows:

in, Indicates that the value of hash encoding is 1, /> Represents the maximum likelihood estimate of the k-th hash code, H _k represents the k-th bit of the hash quantization code H _T , and K represents the coding length;

The third step is to construct a distribution migration module to quantify the distribution migration loss; the distribution migration module uses the distribution center and discrete degree of the hash quantization code extracted by the student model to migrate the hash quantization code extracted by the teacher model, and obtains the distribution migration Hash quantization coding, by quantifying the difference between the hash quantization coding after distribution migration and the hash quantization coding extracted by the teacher model, the distribution migration loss is obtained;

The existing deep hash network only considers the transformation difference caused by data enhancement of the image, quantifies the difference between the transformed images through the teacher model and the student model, and only directly quantifies the difference between the hash quantization codes of the teacher model and the student model. Difference information is used to assist the deep hash network to generate hash quantization coding of images; however, when there is a large difference in distribution between strong and weak transformation images, the hash quantization coding obtained by directly quantizing strong and weak transformation images cannot be fully utilized. The distribution difference information between strong and weakly transformed images, so how to make full use of the distribution difference between strong and weakly transformed images due to data enhancement, make full use of the intra-class and inter-class relationships between images, so that the network can construct a network with Hash encoding of more image difference information is critical to improving image retrieval performance. Therefore, in the quantized code generation stage, the present invention uses the distribution information transformation module (Distribution Information Transformation Block) to guide the hash quantized code generated by the teacher model through the distribution information of the hash quantized code extracted by the student model, and quantizes the hash quantized code extracted by the teacher model. The code is migrated to obtain the hash quantized code after distribution migration. By quantifying the distribution difference between the hash quantization code after distribution migration and the hash quantization code extracted by the teacher model, it is used to assist in extracting the hash quantization code and the hash quantization code of the student model. The hash of the teacher model quantifies the difference between encodings, thereby more fully utilizing the distribution difference information caused by data augmentation transformation, and the DIT-Block module allows the distribution difference caused by data augmentation to receive more attention.

The input of the DIT-Block module includes the hash quantization codes extracted by the teacher model and the student model. Since the tanh activation function constrains the range of the hash quantization codes, the hash quantization codes after the range constraints produce a certain amount of information loss. Therefore, the hash quantization codes h _T and h _S output by the layer normalization layer are used as the input of the DIT-Block module; assuming that the mean and variance of the hash quantization code represent the distribution center and discreteness of the code respectively, first, calculate the hash The mean and variance of the quantized codes h _T and h _S are used to guide the migration of the hash quantized codes h _T through the distribution center and discrete degree of the hash quantized codes h _S to obtain the hash quantized codes after distribution migration. Hash quantization coding serves as a constraint guidance term that quantifies the distribution difference between the hash quantization coding extracted by the teacher model and the hash quantization coding extracted by the student model;

In the formula, h _{T_S} represents the hash quantization code after distribution migration, μ(.) represents the mean value of the hash quantization code, that is, the distribution center; σ(.) represents the variance of the hash quantization code, that is, the degree of discreteness; x _hw represents The value of the hash quantization code at coordinates (h, w), H and W represent the length and width of the hash quantization code, ∈ is the offset value;

The distribution migration of the hash quantization code h _T is implemented through the DIT-block module, and the hash quantization code h _{T_S} after the distribution migration is obtained; the range constraint of the hash quantization code h _{T_S} after the distribution migration is obtained by using the tanh activation function. Hash quantization code _{HT_S} ; for the difference quantification between hash quantization code _{HT_S} and _HT , that is, distribution migration loss, it is judged by cosine similarity, the formula is as follows:

L _DIT =1-cos( _HT , _{HT_S} ) (12)

Step 4: Construct a frequency component extraction module to quantify frequency component loss;

Data enhancement causes a certain transformation in the image. Current image retrieval networks usually only consider the difference in spatial domain information between codes when performing coding quantization, without considering the frequency domain information of coding, resulting in the inability to fully utilize the relative changes caused by image transformation. The information obtains the relative difference change between the image pairs. Therefore, the present invention extracts the frequency domain information of the encoding through the frequency component extraction module (Frequency Component Extraction Block), and pays attention to the hash quantization encoding H _T extracted by the teacher model and the hash extracted by the student model. The difference in frequency domain information between _HS quantization codes is used to obtain the impact of the relative transformation relationship caused by data enhancement on the image in the hash coding process, thereby improving the accuracy of retrieval;

The FCE-Block module extracts the frequency domain information of hash quantization coding through fast Fourier transform, and converts the image coding representation in the spatial domain into the representation of frequency domain space through fast Fourier transform;

Where, The spatial coordinate of the hash quantization code, x(h, w) represents the value of the hash quantization code at the spatial coordinate (h, w), Represents the frequency domain coordinates of hash quantization coding;

The original representation is converted from the spatial domain to the frequency domain through fast Fourier transform, and then phase analysis is performed through arctangent transform to extract the frequency component;

Among them, PH represents the frequency component of the hash quantization code, R(x′)(u, v) is the real part of the hash quantization code x at the frequency domain coordinate (u, v), I(x′)(u, v) is the imaginary part of hash quantization encoding x at frequency domain coordinates (u, v);

The hash quantization codes _HT and _HS pass through the frequency component extraction module to obtain the frequency components PH _T and PH _S. The similarity between the frequency components is quantified through cosine similarity, thereby focusing on the hash quantization code extracted by the teacher model and the student The frequency domain information difference between the hash quantization codes extracted by the model can more fully utilize the relative transformation relationship produced by the data enhancement transformation on the image; the frequency component loss calculation formula is:

L _ph =1-cos (PH _T , PH _S ) (15)

Among them, L _ph is the frequency component loss. The closer the loss value is to 1, the more similar or relevant the two frequency components are;

Step 5: Construct the objective optimization function;

Combined hash proxy loss L _HP , self-distillation difference quantization loss L _Sdh , binary cross-entropy loss L _bce-Q , distribution transfer loss L _DIT and frequency component loss L _ph , the objective optimization function is obtained:

Among them, N _B is the total number of samples, λ ₁ , λ ₂ , λ ₃ , and λ ₄ are all weights. In this embodiment, λ ₁ and λ ₂ take 0.1. When the feature extraction network uses ResNet50, λ ₃ takes 1. When the feature When the extraction network uses the AlexNet network, λ ₃ is taken to be 0.7; when the frequency domain difference of the images in the data set is small, for different encoding lengths, the balance between frequency domain and spatial domain quantization is achieved by adjusting λ ₄ , so that the retrieval effect is optimized. ;

Use the data set to train the deep hash network, and use small batches of mini-batch samples for training. Samples x _i represents the input image, /> Indicates the label corresponding to the input image; the original image of the data set is enhanced through data to obtain a strong transformation image and a weak transformation image. The strong and weak transformation images are input into the student model and the teacher model respectively, and the training loss is measured through the objective optimization function until the loss converges. , obtain the trained student model and teacher model;

Input the image to be queried into the trained student model or teacher model, output the retrieved image, and complete the image retrieval.

The parts not described in the present invention are applicable to the existing technology.

Claims (4)

1. A distribution difference adaptive image retrieval method based on space-frequency interaction, characterized in that the method includes the following steps: The first step: obtain the original image for training, perform data enhancement on the original image, and obtain a strong transformation image and a weak transformation image; Step 2: Construct a deep hash network; input the strong and weak transformed images into the student model and teacher model of the deep hash network respectively to obtain the hash quantization code extracted by the student model and the hash quantization code extracted by the teacher model; Based on the hash quantization encoding extracted by the student model and the teacher, the self-distillation difference quantization loss L _Sdh , the hash proxy loss L _HP and the binary cross-entropy loss L _bce-Q are obtained; L _sdh ＝1-cos(H _T ,H _S ) (1) L _HP =H(y,Softmax(P _T /T)) (4) In the formula, H _T and H _S represent the hash quantization codes extracted by the teacher model and the student model, P _T is the proxy sample, T represents the temperature scale hyperparameter, and H(·) represents the difference between the real category label and the predicted category label of the image. The quantified error between , y represents the real category label sequence of the image, Indicates that the value of hash encoding is 1, /> Represents the maximum likelihood estimate of the k-th hash code, H _k represents the k-th bit of the hash quantization code H _T , and K represents the coding length; The third step is to build a distribution migration module. The distribution migration module uses the distribution center and discrete degree of the hash quantization code extracted by the student model to migrate the hash quantization code extracted by the teacher model, and obtains the hash quantization code after distribution migration. Quantify the difference between the hash quantization coding after distribution migration and the hash quantization coding extracted by the teacher model to obtain the distribution migration loss; the distribution migration loss L _DIT is expressed as: L _DIT =1-cos(H _T ,H _{T_S} ) (12) In the formula, _{HT_S} is the hash quantization code after range-constrained distribution migration; Step 4: Construct a frequency component extraction module. The frequency component extraction module extracts the frequency domain information of hash quantization coding through fast Fourier transform, and then extracts the frequency component of hash quantization coding through arctangent transformation; Where, The spatial coordinates of the hash quantization code, x(h,w) represents the value of the hash quantization code at the spatial domain coordinates (h, w), H and W represent the length and width of the hash quantization code; Among them, PH represents the frequency component of hash quantization coding, R(x′)(u,v) is the real part of hash quantization coding at frequency domain coordinates (u,v), I(x′)(u,v) ) is the imaginary part of the hash quantization code at the frequency domain coordinates (u, v); The frequency component loss L _ph is expressed as: l _ph ＝1-cos(PH _T ,PH _S ) (15) Among them, PH _T represents the frequency component of the hash quantization code _HT , and PH _S is the frequency component of the hash quantization code _HS ; Step 5: Construct the objective optimization function, train the student model and teacher model, and measure the training loss through the objective optimization function; the objective optimization function is: Among them, N _B is the total number of samples, λ ₁ , λ ₂ , λ ₃ , and λ ₄ are all weights; Input the image to be queried into the trained student model or teacher model, and output the retrieved image. 2. The distribution difference adaptive image retrieval method based on space-frequency interaction according to claim 1, characterized in that, in the third step, the hash quantization code H _{T_S} is the hash quantization code h _{T_S} after distribution migration through tanh The activation function is obtained, and the hash quantization code h _{T_S} after distribution migration is expressed as: Among them, μ(.) represents the mean value of hash quantization coding, which represents the distribution center of hash quantization coding; σ(.) represents the variance of hash quantization coding, which represents the degree of discreteness of hash quantization coding; h _T is the teacher model's Hash quantization encoding of layer normalization layer output. 3. The distributed difference adaptive image retrieval method based on space-frequency interaction according to claim 1 or 2, characterized in that both the student model and the teacher model include a feature extraction network and a code generation network, and the feature extraction network adopts ResNet50. Or AlexNet network, the encoding generation network includes a fully connected layer, a layer normalization layer and a tanh activation function. 4. The distribution difference adaptive image retrieval method based on space-frequency interaction according to claim 3, characterized in that in the first step, data enhancement includes random cropping, horizontal flipping, Gaussian blur and brightness, contrast, and saturation transformation. .