CN114610941A - Cultural relic image retrieval system based on comparison learning - Google Patents

Abstract

The invention discloses a cultural relic image retrieval system based on contrast learning, which is characterized in that firstly, a supervised contrast learning algorithm is used for training a model to obtain a good feature extractor, so that extracted features can accurately represent semantic information contained in an image, then, the similarity between feature representations is calculated for retrieval, and the accuracy of a retrieval result is further improved through average query expansion and database end feature enhancement. The retrieval system of the invention trains the network through the supervised contrast learning algorithm to obtain the feature extractor, can extract the image to obtain effective and discriminant feature representation, and further improves the retrieval accuracy through average query expansion and data characteristic enhancement of the data end.

Description

Image retrieval system of cultural relics based on contrastive learning

technical field

The invention relates to the feature extraction and contrast matching technology of cultural relic image data, and more particularly, to an image retrieval system for cultural relic data based on a contrast learning algorithm.

Background technique

The review work in each link of the transaction and circulation of folk cultural relics relies too much on empirical analysis and judgment with the naked eye, and there are problems such as complicated processes and low efficiency, which has also spawned the need for computers to automatically retrieve images of cultural relics. Image retrieval aims to build an index between a query image and an image database, and output images in the database that match or are similar to the query image according to some measure. Based on the current situation of large amount of image data and high retrieval demand, it is urgent to propose a high-fidelity digital information acquisition technology that adapts to the diversity of folk cultural relics and the complexity of the scene, and to design the extraction method and comparison and matching method for key feature information of cultural relics data.

SUMMARY OF THE INVENTION

In order to solve the problems in the prior art, the present invention provides a cultural relic image retrieval system based on contrastive learning, which solves the problems of low retrieval accuracy and efficiency and high requirements for computing power in the prior art.

The technical scheme of the present invention is:

A cultural relic image retrieval system based on comparative learning, including a feature extractor and a retrieval module, the feature extractor includes preprocessing and feature extraction, the retrieval module includes sorting and indexing, similarity calculation; , perform preprocessing and feature extraction on it to obtain the corresponding feature vector, and simultaneously perform preprocessing and feature extraction on all images in the image database to obtain the corresponding image feature library, and then use the retrieval module to calculate the feature vector of the query image and the image The similarity between the feature vectors in the feature library is used to sort and index the images in the image database, and the image matching the query image is obtained as the final retrieval result.

The feature extractor network is trained using a supervised contrastive learning algorithm; the contrastive learning model employs two branches that are fully symmetrical and parameter-sharing, each branch includes data augmentation, an encoder network, and a projection network, where the encoder network and the projection network The network constitutes a feature extractor; for any image x, it forms two enhanced views x _i and x _j through two different data enhancement methods; since the upper and lower branches are completely symmetrical, x _i in the upper branch first passes through the encoder The network is converted to the corresponding feature representation _hi = f _θ (x _i ); then the nonlinear transformation structure-projection network maps the feature representation to the final feature representation _zi = g _θ ( _hi ); similarly, the lower branch The enhanced view of is subjected to two nonlinear transformations to obtain the final feature representation z _j =g _θ (f _θ (x _j )).

其中，

和

是同一个样本经两种随机的数据增强方式得到的数据对，数据增强过程中的标签信息始终不会改变；对于有监督对比学习，一个样本对应着多个正样本，即Batch内与其标签信息相同的样本作为正样本，而与其标签信息不同的样本作为负样本，这样可以有效利用已知的标签信息进行监督学习，从而实现同类别的样本在表示空间中更加接近，而不同类别的样本在表示空间中相互远离，提高特征表示的判别能力；因此，有监督对比学习的损失函数定义为：The network training is: randomly sample N samples to form a Batch, denoted as {x _k , y _k } _{k = 1, 2, ..., N} , y _k is the label of x _k , and 2N can be obtained through data enhancement samples

in,

and

It is a data pair obtained by the same sample through two random data enhancement methods, and the label information will never change during the data enhancement process; for supervised comparative learning, one sample corresponds to multiple positive samples, that is, the batch and its label information The same sample is used as a positive sample, and a sample with different label information is used as a negative sample, which can effectively use the known label information for supervised learning, so that the samples of the same category are closer in the representation space, while the samples of different categories are The representation space is far away from each other to improve the discriminative ability of feature representation; therefore, the loss function of supervised contrastive learning is defined as:

表示Batch中与样本z_i具有相同标签信息的样本总数；通过优化式(4)中的损失函数对网络进行训练，将训练好的编码器网络与投影网络作为特征提取器对查询图像和图像数据库中的图像进行特征提取。Among them, 1 _i≠j ∈{0, 1} is the indicator function, if and only if i≠j, take 1, otherwise take 0; τ>0 is the temperature parameter; z _{j(i) represents} the positive sample of zi, z _i ·z _j(i) represents the inner product operation between vectors;

Represents the total number of samples in the Batch that have the same label information as the sample _zi ; the network is trained by optimizing the loss function in equation (4), and the trained encoder network and projection network are used as feature extractors to query images and image databases. feature extraction from the images.

The similarity calculation function between the eigenvectors adopts the dot product after regularization of the eigenvectors L2 or the cosine similarity between the eigenvectors:

Among them, _zi and _zi represent a one-dimensional vector, and ||·|| ₂ represents the L2 norm of the vector.

During the indexing and sorting process, average query expansion and database-side feature enhancement are used to further improve the accuracy of retrieval results.

The average query expansion is to first sort the images in the database according to the similarity between the feature vector of the original query Q ₀ and the feature vector in the feature database, return the first m (m<50) results, and then analyze the original query Q ₀ and m results are averaged to form a new query Q _avg , and the new query is used to generate the final retrieval result;

Among them, z ₀ is the feature vector of the original query, and _zi is the feature vector of the ith result.

The database-side feature enhancement replaces the original image by the combination of the image in the database and its similar images, aiming to improve the quality of the image representation by using the features of the image neighborhood; first, the feature vectors in the image feature library are calculated in pairs. Similarity, for any image, the K nearest image features are added, or the sum is weighted according to the ranking of the features:

where r is the rank of image features and k is the total number of close images considered.

Beneficial effects:

The cultural relic image retrieval system based on contrastive learning proposed by the present invention trains the network through a supervised contrastive learning algorithm to obtain a feature extractor, which can extract an effective and discriminative feature representation from the image, and expands the data with the average query. The end data feature enhancement is used to further improve the retrieval accuracy. The user inputs an image of a cultural relic as a query, and the retrieval system can accurately retrieve and return results (one or multiple sorted) matching the query image in the image database. Both quantitative and qualitative results obtained on the common image dataset cifar10 demonstrate the effectiveness of the retrieval system.

Description of drawings

Fig. 1 Cultural relic image retrieval system based on contrastive learning;

Figure 2 compares the learning model;

Figure 3 Quantitative and qualitative results of the system.

Detailed ways

The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

The cultural relic image retrieval system based on comparative learning of the present invention is shown in Figure 1. An image to be retrieved is input, and preprocessed such as scale transformation, random flip, etc., to extract the corresponding query image features. All images are preprocessed and feature extracted to obtain the corresponding image feature library, then calculate the similarity between the image features and all the features in the image feature library, and use the similarity to sort and index the images in the image database to get The images (one or more sorted images) matching the query image are used as the final retrieval result. The following is a detailed introduction to the training of the feature extractor and the implementation of the retrieval module:

1. Feature Extractor

The feature extractor needs to perform feature extraction on the query image and the images in the image database to effectively represent the image information. The key to retrieval accuracy is the retrieval system. Here the feature extractor is trained using a supervised contrastive learning algorithm.

The core idea of contrastive learning is to pull in the distance between samples and positive samples, and at the same time pull the distance between samples and their negative samples. In the supervised contrastive learning algorithm, the label information in the data set is used as supervision, and each sample corresponds to multiple positive samples and negative samples. Training the feature extractor can make it generate a discriminative feature representation, which is beneficial to image retrieval. realization of the task. The contrastive learning model is shown in Figure 2. It adopts two branches that are completely symmetrical and share parameters. Each branch includes data enhancement, encoder network and projection network. The encoder network and projection network form a feature extractor.

For any image x, it forms two augmented views x _i and x _j through two different data augmentation methods. Since the upper and lower branches are completely symmetrical, taking the above branch as an example, x _i is first converted into a corresponding feature representation hi =f _θ ( _{xi )} through an encoder network (usually using ResNet as the model structure). Then the nonlinear transformation structure-projection network (consisting of [FC->BN->ReLU->FC] two-layer MLP) maps the feature representation to the final feature representation _zi =g _θ (h _i ). Similarly, the enhanced view of the lower branch undergoes two nonlinear transformations to obtain the final feature representation z _j =g _θ (f _θ (x _j )). The purpose of contrastive learning is to make the distance between positive samples in the representation space closer, while the distance between negative samples is farther.

其中，

和

是同一个样本经两种随机的数据增强方式得到的数据对，数据增强过程中的标签信息始终不会改变。若不考虑类别的监督信息，数据对

互为正样本，而

与Batch中除

外的其他任意2N-2个样本都互为负样本。此时为自监督的对比学习算法，其损失函数定义为：During network training, randomly sample N samples to form a Batch, denoted as {x _k , y _k } _{k = 1, 2, ..., N} , y _k is the label of x _k , and 2N samples can be obtained through data enhancement

in,

and

It is a data pair obtained by the same sample through two random data enhancement methods, and the label information in the data enhancement process will never change. If the supervisory information of the category is not considered, the data

are positive samples of each other, and

Except in Batch

Any other 2N-2 samples are negative samples of each other. At this time, it is a self-supervised contrastive learning algorithm, and its loss function is defined as:

Among them, 1 _i≠k ∈{0,1} is the indicator function, if and only if i≠k, take 1, otherwise take 0; τ>0 is the temperature parameter; z _{j(i) represents} the positive sample of zi, z _i ·z _j(i) represents an inner product operation between vectors. It can be seen that the numerator part of the loss function encourages the higher the similarity between the sample and the positive sample, the better, that is, the closer the distance in the representation space, the better; the denominator part encourages the lower the similarity between the sample and the negative sample, the better , that is, the farther the distance in the representation space, the better.

It can be seen that the loss function of self-supervised contrastive learning treats each sample as a separate category, and cannot handle the situation where there are labels in the dataset, that is, it is known that multiple samples belong to the same category. For supervised contrastive learning, one sample corresponds to multiple positive samples, that is, the samples with the same label information in the batch are regarded as positive samples, and the samples with different label information are regarded as negative samples, which can effectively use the known label information to carry out Supervised learning, so that samples of the same category are closer in the representation space, while samples of different categories are far away from each other in the representation space, improving the discriminative ability of feature representation. Therefore, the loss function for supervised contrastive learning is defined as:

表示Batch中与样本z_i具有相同标签信息的样本总数。通过优化式(4)中的损失函数对网络进行训练，将训练好的编码器网络与投影网络作为特征提取器对查询图像和图像数据库中的图像进行特征提取。in,

Indicates the total number of samples in the Batch that have the same label information as the sample _zi . The network is trained by optimizing the loss function in equation (4), and the trained encoder network and projection network are used as feature extractors to extract features from the query image and the images in the image database.

2. Retrieval module

The feature extractor is used to extract the features of all the images in the database to obtain the corresponding image feature library. When retrieving, input a query image, and perform feature extraction on it to obtain the corresponding feature vector. Afterwards, through the retrieval module, the similarity between the feature vector of the query image and the feature vector in the graphic feature library is calculated, and the indexing and sorting are performed according to the similarity, and the sorted images are output as the final result (the number is set manually).

The similarity calculation function between eigenvectors generally adopts the dot product after regularization of eigenvectors L2 or the cosine similarity between eigenvectors:

During the indexing and sorting process, average query expansion and database-side feature enhancement are used to further improve the accuracy of retrieval results. The average query expansion is to first sort the images in the database according to the similarity between the feature vector of the original query Q ₀ and the feature vector in the feature database, and return the first m (m < 50) results, and then compare the original query Q0 and m The results are averaged to form a new query Q _avg , and the new query is used to generate the final retrieval result.

Among them, z ₀ is the feature vector of the original query, and _zi is the feature vector of the ith result. Database-side feature enhancement replaces the original image with a combination of images in the database and its adjacent images, aiming to improve the quality of image representation by utilizing the features of image neighborhoods. First, the similarity is calculated for the feature vectors in the image feature library pairwise. For any image, the K nearest image features are added, or the sum is weighted according to the ranking of the features:

where r is the rank of image features and k is the total number of close images considered.

In the proposed cultural relic image retrieval system based on contrastive learning, the encoder network in the feature extractor adopts the ResNet50 network architecture, the similarity between image feature vectors is calculated using cosine similarity, and the retrieval module performs average query expansion and database-side feature enhancement. , select the top five results of feature similarity for calculation. First, the feature extractor is trained based on the supervised contrastive learning algorithm, and the trained feature extractor is used to extract the features of the images in the image database to obtain the feature database. When the user queries, input a query image into the system, the system will The query feature is obtained by preprocessing and feature extraction by the feature extractor, and then the similarity between the query feature and the feature in the feature database is measured, and the feature similarity is used to achieve sorting and indexing, and finally an image matching the query image is output (a or sorted multiple) to the user. The retrieval system can achieve fast and effective retrieval. Table 1 shows its retrieval accuracy on the common image dataset cifar10, and Figure 3 shows the retrieval results with 10 outputs.

Table 1

Precision@1(%) Precision@10(%) Map@all(%) Cifar10 98.0 100 98.1

The technical solutions disclosed and proposed in the present invention can be realized by those skilled in the art by referring to the content of this article and appropriately changing the conditions, routes and other links. The methods and technical routes described herein can be modified or recombined without departing from the content, spirit and scope of the present invention to achieve the final preparation technology. It should be particularly pointed out that all similar substitutions and modifications apparent to those skilled in the art are deemed to be included in the spirit, scope and content of the present invention. Matters not covered by the present invention belong to the known technology.

Claims (7)

1. a cultural relic image retrieval system based on contrast learning, is characterized in that, comprises feature extractor and retrieval module, described feature extractor comprises preprocessing and feature extraction, and described retrieval module comprises sorting and index, similarity calculation; Input the image to be retrieved, perform preprocessing and feature extraction on it to obtain the corresponding feature vector, and simultaneously perform preprocessing and feature extraction on all images in the image database to obtain the corresponding image feature library, and then calculate the query through the retrieval module. The similarity between the feature vector of the image and the feature vector in the image feature library is used to sort and index the images in the image database, and the image matching the query image is obtained as the final retrieval result. 2. the cultural relic image retrieval system based on contrastive learning according to claim 1, is characterized in that, using supervised contrastive learning algorithm to carry out network training to feature extractor; Contrastive learning model adopts two branches of complete symmetry and parameter sharing, Each branch includes data enhancement, encoder network and projection network, wherein the encoder network and projection network form a feature extractor; for any image x, it forms two enhanced views x _i through two different data enhancement methods With x _j ; since the upper and lower branches are completely symmetrical, x _i in the upper branch is first converted into the corresponding feature representation h _i =f _θ (x _i ) through the encoder network; then the nonlinear transformation structure-projection network will feature representation The mapping is the final feature representation z _i =g _θ ( _hi ); similarly, the enhanced view of the lower branch undergoes two nonlinear transformations to obtain the final feature representation z _j =g _θ (f _θ (x _j )). 3. The cultural relic image retrieval system based on contrast learning according to claim 2, wherein the network training is: randomly sampling N samples to form a Batch, denoted as {x _k , y _k } _{k=1,2 , ..., N} , y _k are the labels of x _k , 2N samples can be obtained by data augmentation

in,

and

It is the data pair obtained by the same sample through two random data enhancement methods, and the label information in the data enhancement process will never change; For supervised contrastive learning, one sample corresponds to multiple positive samples, that is, the samples with the same label information in the batch are regarded as positive samples, and the samples with different label information are regarded as negative samples, which can effectively use the known label information for supervision. Learning, so that the samples of the same category are closer in the representation space, while the samples of different categories are far away from each other in the representation space, which improves the discriminative ability of feature representation; therefore, the loss function of supervised contrastive learning is defined as:

Among them, 1 _i≠j ∈{0, 1} is the indicator function, if and only if i≠j, take 1, otherwise take 0; τ>0 is the temperature parameter; z _{j(i) represents} the positive sample of zi, z _i ·z _j(i) represents the inner product operation between vectors;

Represents the total number of samples in the Batch that have the same label information as the sample _zi ; the network is trained by optimizing the loss function in equation (4), and the trained encoder network and projection network are used as feature extractors to query images and image databases. feature extraction from the images. 4. the cultural relic image retrieval system based on contrast learning according to claim 1, is characterized in that, the similarity calculation function between described feature vectors adopts the dot product after feature vector L2 regularization or the cosine similarity between feature vectors Spend:

Among them, z _i and z _j represent a one-dimensional vector, and ||·|| ₂ represents the L2 norm of the vector. 5 . The cultural relic image retrieval system based on contrastive learning according to claim 1 , wherein in the indexing and sorting process, average query expansion and database-side feature enhancement are used to further improve the accuracy of retrieval results. 6 . 6. the cultural relic image retrieval system based on contrast learning according to claim 5, is characterized in that, described average query expansion namely first according to the similarity between the feature vector in the feature vector of original query Q ₀ and the feature vector in the feature library to the database. The images are sorted, the first m (m<50) results are returned, and then the original query Q ₀ and the m results are averaged to form a new query Q _avg , and the new query is used to generate the final retrieval result;

Among them, z ₀ is the feature vector of the original query, and _zi is the feature vector of the ith result. 7. The cultural relic image retrieval system based on contrastive learning according to claim 5, is characterized in that, described database end feature enhancement replaces original image by the combination of image in database and its adjacent image, is intended to utilize image neighborhood feature to improve the quality of image representation; first, calculate the similarity of the feature vectors in the image feature library pairwise, for any image, add its nearest K image features, or find the feature based on the ranking of the features. and weighted:

where r is the rank of image features and k is the total number of close images considered.

Images