CN109871454B - A Robust Discrete Supervised Cross-media Hashing Retrieval Method - Google Patents

Abstract

The invention discloses a robust discrete-supervised cross-media hash retrieval method. By learning a robust similarity matrix between two samples, the semantic association between heterogeneous samples can be mined, and content-based cross-media retrieval can be realized by using the method. , the method includes the following steps: establishing an image and text data set, and extracting visual and text features from the image and text samples in the data set respectively; using the class labels, image and text features of the samples to construct a similarity matrix between two samples, And use the low rank of the similarity matrix between the two samples and the sparse characteristics of the sample noise to learn a robust similarity matrix between the two samples; and then use the robust similarity matrix between the two samples to learn a more discriminative Ha Greek code; applied to the hash function Norm regular term constraints, to learn a more robust hash function; a discrete iterative optimization algorithm is proposed to directly obtain the discrete solution of the hash code; the method of the present invention learns a robust similarity matrix between pairs of samples. Effectively resist the noise that may exist in the sample, thereby greatly improving the performance of multimedia retrieval.

Description

A Robust Discrete Supervised Cross-media Hashing Retrieval Method

Technical field:

The invention relates to a robust discrete-supervised cross-modal hash retrieval method, which belongs to the fields of multimedia retrieval and machine learning.

Background technique:

In recent years, a large amount of data is generated on the Internet every day, which brings great challenges to multimedia retrieval tasks. How to efficiently and effectively find similar samples has become an urgent need. The hashing method maps the samples from the original feature space to the Hamming space by learning a set of hash functions. Due to its fast calculation speed and saving storage space in large-scale applications, it has attracted great attention from researchers. The storage cost of the hash code is much lower than that of the original feature, and the similarity between samples can be quickly calculated by using the XOR operation in the Hamming space. Hashing methods have been extensively studied, but most studies only focus on one modality, however samples with the same semantics on the Internet can usually be represented as multiple modalities, which leads to a heterogeneous semantic gap between different modalities . For example, images can be represented by visual and corresponding textual features. In addition, when a user submits a query sample to a search engine, the user prefers that the search engine returns similar samples in multiple modalities. Therefore, cross-media retrieval has attracted more and more attention. The goal of cross-media hashing methods is to map heterogeneous samples into a shared Hamming space and preserve the similar structure of samples in this space. Specifically, for similar heterogeneous samples, the Hamming distance is smaller in the shared Hamming space, and vice versa. According to whether class labels are used during training, cross-media hashing methods can generally be divided into two categories: unsupervised and supervised methods. The former usually learns hash codes by maintaining the intra-modal and inter-modal similarity of samples, while the latter can further combine class labels to learn more discriminative hash codes. Recent work has shown that incorporating class labels of samples can improve retrieval performance.

Although many supervised cross-modal hashing methods have been proposed and achieved satisfactory results, there are still some issues that need to be further addressed. First, in the real world, samples may contain noise. However, most supervised cross-modal hashing methods only use the class labels of the training data to construct the similarity matrix between two samples, without considering the noise in the samples, such as: outliers. Obviously, these noise samples will seriously damage the structure of the similarity matrix between pairs of samples, thereby misleading the learning of hash codes and resulting in reduced retrieval performance. Secondly, the discrete constraints of hash codes make mixed integer optimization problems usually difficult to solve. Most methods first relax the discrete constraints of hash codes to obtain continuous solutions, and then quantize to generate hash codes. However, quantization leads to loss of information, making hash codes less discriminative.

Invention content:

The purpose of the present invention is to overcome the deficiencies of the above-mentioned prior art and provide a hash code with better learning performance, improve the performance of the algorithm, better resist noise, improve the distinguishing ability of the hash code, and be suitable for real-world applications. A Robust Discrete Supervised Cross-Modal Hashing Retrieval Method for Cross-Media Retrieval of Web Data.

The object of the present invention can be achieved by providing the following measures: a robust discrete supervised cross-modal hash retrieval method, characterized in that the method comprises the following steps:

Step 1: Collect image and text sample pairs containing class labels to form a one-to-one image-text cross-modal retrieval image-text dataset;

Step 2: Extract features from the image and text modal samples respectively, and remove the mean value from the features of the image and text modal samples, so that the mean value of the feature data of the two modal samples is 0;

Step 3: Randomly divide all sample pairs in the data set into training set and test set;

Step 4: Use the class label of the sample pair in the training set, the sample features of the image and the text mode to construct the similarity matrix between the two samples, and use the low-rank characteristics of the similarity matrix between the two samples and the sparse characteristics of the noise samples , to learn a robust similarity matrix between two samples; the feature of the training sample pair is set to X, X={X ⁽¹⁾ , X ⁽²⁾ }, where X ⁽¹⁾ represents the sample of the image modality in the training set feature, X ⁽²⁾ represents the sample feature of the text modality in the training set, where d ₁ and d ₂ represent the dimensions of image and text modality sample features respectively, N represents the number of image or text modality samples in the training set, and the class label of the sample pair is denoted by L, c represents the number of sample categories, l _i ∈ {0, 1} ^c , if l _ij = 1, it means that the i-th sample belongs to the j-th class; otherwise, if l _ij = 0, it means that the i-th sample does not belong to the j-th class; learning the objective function of a robust pairwise similarity matrix includes the following steps:

(1) Use the sample features of the image modality to calculate the similarity matrix between two samples based on the image modality features, which is defined as follows:

Where ||·|| _F represents the Frobenius norm, S ⁽¹⁾ represents the similarity matrix between two samples of the image modality, Indicates the similarity between the i-th image sample and the j-th image sample, σ ₁ is the scale parameter;

(2) Using the sample features of the text modality to calculate the similarity matrix between two samples based on the text modality features, the definition is as follows:

Among them, S ⁽²⁾ represents the similarity matrix between two samples of the text modality, Indicates the similarity between the i-th text sample and the j-th text sample, and σ ₂ is a scale parameter;

(3) Use the class labels of the sample pairs to calculate the similarity matrix between two samples based on the class labels, which is defined as follows:

where S ⁽³⁾ represents the pairwise similarity matrix of samples to labels, Indicates the similarity between the i-th sample pair label and the j-th sample pair label;

(4) The objective function for learning a robust pairwise similarity matrix is defined as follows:

stS ⁽ⁱ⁾ = S+||E ⁽ⁱ⁾ || ₀

where S represents the learned pairwise similarity matrix between robust samples, E ⁽ⁱ⁾ represents the noise in the i-th pairwise similarity matrix, rank( ) represents the rank of the matrix, and ||·|| ₀ represents l ₀ norm;

(5) Since the objective function in (4) above has constraints of discrete low rank and l ₀ norm, it is difficult to solve the problem directly. These two constraints can be relaxed to obtain an approximate solution to the problem, so the above formula can be rewritten for

stS ⁽ⁱ⁾ = S+||E ⁽ⁱ⁾ || ₁

where ||·|| _* represents the nuclear norm, ||·|| ₁ represents the l ₁ norm,

(6) Use the augmented Lagrangian multiplier method to solve this problem, and obtain a robust similarity matrix between two samples;

Step 5: Construct the objective function, which specifically includes the following steps:

(1) Maintain the similarity based on the similarity matrix between robust pairs of samples in the Hamming space, and since the image and text samples have the same class labels, their distance should be as small as possible, so the objective function of hash code learning is defined as follows :

Where k represents the length of the hash code, _B1 is the hash code of the image modality sample, _B2 is the hash code of the text modality sample, and λ is the weight parameter;

(2) Use the linear map as the hash function, and use the l _2,1 norm as the regular item to constrain the learning of image and text modal hash functions to enhance its ability to resist noise, so each modal hash function learns The objective function of is defined as follows:

Among them, W ₁ and W ₂ represent the hash functions of the image modality and the text modality respectively, and Reg(·) represents the regular term to prevent overfitting, here β _i and μ are weight parameters;

(3) Adding the objective function of hash code and hash function learning is the objective function of this method, which is defined as follows:

Where β _i is the weight parameter;

Step 6: Since the objective function contains discrete constraints of multiple unknown variables and hash codes, the objective function is difficult to solve, but it can be found through observation that when other variables are fixed to solve one of the variables, it is a convex optimization problem, so An iterative optimization algorithm can be used to solve the problem, and the solution process includes the following steps:

(1) Fix W ₁ , W ₂ and B ₂ , and solve for B ₁ :

Remove the constant term, the objective function can be written as:

Since B ₁ is discrete, it is difficult to solve the problem directly. Here, it can be solved sample by sample. Let b _1i represent the i-th column of B ₁ , and b _2j represent the j-th column of B _2. The objective function can be written as:

This problem is still difficult to solve directly. Here, the cyclic coordinate gradient descent method is used to solve it bit by bit. Let b _1im represent the mth bit of b _1i , Indicates that b _1i is a vector composed of bits other than the mth bit, then b _1im can be obtained by the following formula:

Repeat the above formula until the hash codes of all image modality samples are solved;

(2) Fix W ₁ , W ₂ and B ₁ , solve for B ₂ :

Similar to solving B ₁ , we can get

Repeat the above formula until the hash codes of all text modal samples are solved;

(3) Fix W ₂ , B ₁ and B ₂ , and solve for W ₁ :

Remove the constant term, the objective function can be written as:

There is a closed solution to this problem

where _D1 is a diagonal matrix,

(4) Fix W ₁ , B ₁ and B ₂ , and solve for W ₂ :

Similar to solving W ₁ , there is a closed solution for W ₂

where _D2 is a diagonal matrix,

(5) Repeat (1)-(4) until the algorithm converges or reaches the maximum number of iterations;

Step 7: The user enters the query sample, extracts its features, and removes the average value of the extracted features;

Step 8: Use the learned hash function to generate the hash code of the query sample:

Step 9: Calculate the Hamming distance between the query sample and the heterogeneous samples in the target (training) set, and arrange the Hamming distance in ascending order, and the samples corresponding to the first r Hamming distances are the retrieval results.

Compared with the prior art, the present invention can produce the following positive effects: the method of the present invention learns a robust similarity matrix between two samples by integrating the features of class labels, images and text modalities into a framework, so that the learning performance is better The hash code of the algorithm improves the performance of the algorithm; it is proposed to apply the l _2,1 norm as a regular term to constrain the learning of the hash function to better resist noise; a discrete optimization algorithm is proposed, which can directly obtain the discrete hash code , which improves the distinguishing ability of hash codes, and the invention is suitable for cross-media retrieval of actual network data.

Description of drawings:

Fig. 1 is a flow chart of the robust discrete supervised cross-modal hash retrieval method of the present invention.

Detailed ways:

In order to make the technical solution of the present invention clearer, the present invention will be described in further detail below in conjunction with specific embodiments, rather than limiting its protection scope.

Embodiment: a kind of robust discrete supervised cross-modal hash retrieval method, it comprises the following steps:

Step 1: Collect image and text sample pairs containing class labels to form a one-to-one image-text cross-modal retrieval image-text dataset;

The second step: extract the features of the image and text, where the image modality sample is represented by a 150-dimensional texture feature, and the text modality sample is represented by a 500-dimensional BOW (Bag Of Words) feature, and the features are averaged to make the two The mean value of the characteristic data of the modal sample is 0;

Step 3: Randomly divide all sample pairs in the data set into training set and test set;

Step 4: Use the class label of the sample pair in the training set, the sample features of the image and the text mode to construct the similarity matrix between the two samples, and use the low-rank characteristics of the similarity matrix between the two samples and the sparse characteristics of the noise samples , to learn a robust similarity matrix between two samples; the feature of the training sample pair is set to X, X={X ⁽¹⁾ , X ⁽²⁾ }, where X ⁽¹⁾ represents the sample of the image modality in the training set feature, X ⁽²⁾ represents the sample feature of the text modality in the training set, where d ₁ and d ₂ represent the dimensions of image and text modality sample features respectively, N represents the number of image or text modality samples in the training set, and the class label of the sample pair is denoted by L, c represents the number of sample categories, l _i ∈ {0, 1} ^c , if l _ij = 1, it means that the i-th sample belongs to the j-th class; otherwise, if l _ij = 0, it means that the i-th sample does not belong to the j-th class; here d ₁ =150, d ₂ =500;

Learning the objective function of a robust pairwise similarity matrix includes the following steps:

(1) Use the sample features of the image modality to calculate the similarity matrix between two samples based on the image modality features, which is defined as follows:

Where ||·|| _F represents the Frobenius norm, S ⁽¹⁾ represents the similarity matrix between two samples of the image modality,

Indicates the similarity between the i-th image sample and the j-th image sample, σ ₁ is a scale parameter, where σ ₁ =0.8;

(2) Using the sample features of the text modality to calculate the similarity matrix between two samples based on the text modality features, the definition is as follows:

Among them, S ⁽²⁾ represents the similarity matrix between two samples of the text modality, Indicates the similarity between the i-th text sample and the j-th text sample, σ ₂ is a scale parameter, where σ ₂ =0.3;

(3) Use the class labels of the sample pairs to calculate the similarity matrix between two samples based on the class labels, which is defined as follows:

where S ⁽³⁾ represents the pairwise similarity matrix of samples to labels, Indicates the similarity between the i-th sample pair label and the j-th sample pair label;

(4) The objective function for learning a robust pairwise similarity matrix is defined as follows:

stS ⁽ⁱ⁾ = S+||E ⁽ⁱ⁾ || ₀

where S represents the learned pairwise similarity matrix between robust samples, E ⁽ⁱ⁾ represents the noise in the i-th pairwise similarity matrix, rank( ) represents the rank of the matrix, and ||·|| ₀ represents l ₀ norm;

(5) Since the objective function in (4) above has constraints of discrete low rank and l ₀ norm, it is difficult to solve the problem directly. These two constraints can be relaxed to obtain an approximate solution to the problem, so the above formula can be rewritten for

stS ⁽ⁱ⁾ = S+||E ⁽ⁱ⁾ || ₁

where ||·|| _* represents the nuclear norm, ||·|| ₁ represents the l ₁ norm,

(6) Use the augmented Lagrangian multiplier method to solve this problem, and obtain a robust similarity matrix between two samples;

Step 5: Construct the objective function, which specifically includes the following steps:

(1) Maintain the similarity based on the similarity matrix between robust pairs of samples in the Hamming space, and since the image and text samples have the same class labels, their distance should be as small as possible, so the objective function of hash code learning is defined as follows :

Wherein k represents the length of the hash code, B ₁ is the hash code of the image modality sample, B ₂ is the hash code of the text modality sample, and λ is a weight parameter, where λ=1;

(2) Use the linear map as the hash function, and use the l _2,1 norm as the regular item to constrain the learning of image and text modal hash functions to enhance its ability to resist noise, so each modal hash function learns The objective function of is defined as follows:

Among them, W ₁ and W ₂ represent the hash functions of the image modality and the text modality respectively, and Reg(·) represents the regular term to prevent overfitting, here β _i and μ are weight parameters, where β ₁ =10, β ₂ =10, μ=0.1:

(3) Adding the objective function of hash code and hash function learning is the objective function of this method, which is defined as follows:

Step 6: Since the objective function contains discrete constraints of multiple unknown variables and hash codes, the objective function is difficult to solve, but it can be found through observation that when other variables are fixed to solve one of the variables, it is a convex optimization problem, so An iterative optimization algorithm can be used to solve the problem, and the solution process includes the following steps:

(1) Fix W ₁ , W ₂ and B ₂ , and solve for B ₁ :

Remove the constant term, the objective function can be written as:

Since B ₁ is discrete, it is difficult to solve the problem directly. Here, it can be solved sample by sample. Let b _1i represent the i-th column of B ₁ , and b _2j represent the j-th column of B _2. The objective function can be written as:

This problem is still difficult to solve directly. Here, the cyclic coordinate gradient descent method is used to solve it bit by bit. Let b _1im represent the mth bit of b _1i , Indicates that b _1i is a vector composed of bits other than the mth bit, then b _1im can be obtained by the following formula:

Repeat the above formula until the hash codes of all image modality samples are solved;

(2) Fix W ₁ , W ₂ and B ₁ , solve for B ₂ :

Similar to solving B ₁ , we can get

Repeat the above formula until the hash codes of all text modal samples are solved;

(3) Fix W ₂ , B ₁ and B ₂ , and solve for W ₁ :

Remove the constant term, the objective function can be written as:

There is a closed solution to this problem

where _D1 is a diagonal matrix,

(4) Fix W ₁ , B ₁ and B ₂ , and solve for W ₂ :

Similar to solving W ₁ , there is a closed solution for W ₂

where _D2 is a diagonal matrix,

(5) Repeat (1)-(4), and end if the absolute value of the error of the last two iterations is less than 0.01 or the number of iterations is greater than 20;

Step 7: The user enters the query sample, extracts its features, and removes the average value of the extracted features;

Step 8: Use the learned hash function to generate the hash code of the query sample:

Step 9: Calculate the Hamming distance between the query sample and the heterogeneous samples in the target (training) set, and arrange the Hamming distance in ascending order. The samples corresponding to the first r Hamming distances are the retrieval results, where r=100.

In order to verify the effectiveness of the present invention, this embodiment takes the public data set Mirflickr25K as an example. This data set contains 20015 image-text pairs, and all sample pairs can be divided into 24 categories; 15011 (75%) sample pairs are randomly selected to form The training set, and the remaining 5004 (25%) sample pairs constitute the test set; image modality samples are represented by 150-dimensional texture features, text modality samples are represented by 500-dimensional BOW (Bag Of Words) features, and the feature To remove the mean value, the mean value of the feature data of the two modal samples is 0; in order to objectively evaluate the retrieval performance of the method of the present invention, the average accuracy rate (MeanAverage Precision, MAP) is used as the evaluation standard, and on the Mirflickr25K data set, different hash codes The MAP results of long p are shown in Table 1.

Table 1 MAP results on the Mirflickr25K dataset

p=16 p=32 p=64 p=96 image retrieval text 0.6718 0.6785 0.6843 0.6918 Text Retrieval Image 0.6813 0.6953 0.6977 0.7045

It should be understood that the parts not described in detail in this specification belong to the prior art. The above description of the preferred embodiments is more detailed, but it should not be considered as limiting the protection scope of the patent of the present invention.

Claims (1)

1. A robust discrete supervision cross-media hash retrieval method, characterized in that the method comprises the steps: Step 1: Collect image and text sample pairs containing class labels to form a one-to-one image-text cross-modal retrieval image-text dataset; Step 2: Extract features from the image and text modal samples respectively, and remove the mean value from the features of the image and text modal samples, so that the mean value of the feature data of the two modal samples is 0; Step 3: Randomly divide all sample pairs in the data set into training set and test set; Step 4: Use the class label of the sample pair in the training set, the sample features of the image and the text mode to construct the similarity matrix between the two samples, and use the low-rank characteristics of the similarity matrix between the two samples and the sparse characteristics of the noise samples , to learn a robust similarity matrix between two samples; the feature of the training sample pair is set to X, X={X ⁽¹⁾ , X ⁽²⁾ }, where X ⁽¹⁾ represents the sample of the image modality in the training set feature, X ⁽²⁾ represents the sample feature of the text modality in the training set, where d ₁ and d ₂ represent the dimensions of image and text modality sample features respectively, N represents the number of image or text modality samples in the training set, and the class label of the sample pair is denoted by L, c represents the number of sample categories, l _i ∈ {0, 1} ^c , if l _ij = 1, it means that the i-th sample belongs to the j-th class; otherwise, if l _ij = 0, it means that the i-th sample does not belong to the j-th class; learning the objective function of a robust pairwise similarity matrix includes the following steps: (1) Use the sample features of the image modality to calculate the similarity matrix between two samples based on the image modality features, which is defined as follows: Where ||·|| _F represents the Frobenius norm, S ⁽¹⁾ represents the similarity matrix between two samples of the image modality, Indicates the similarity between the i-th image sample and the j-th image sample, σ ₁ is the scale parameter; (2) Using the sample features of the text modality to calculate the similarity matrix between two samples based on the text modality features, the definition is as follows: Among them, S ⁽²⁾ represents the similarity matrix between two samples of the text modality, Indicates the similarity between the i-th text sample and the j-th text sample, and σ ₂ is a scale parameter; (3) Use the class labels of the sample pairs to calculate the similarity matrix between two samples based on the class labels, which is defined as follows: where S ⁽³⁾ represents the pairwise similarity matrix of samples to labels, Indicates the similarity between the i-th sample pair label and the j-th sample pair label; (4) The objective function for learning a robust pairwise similarity matrix is defined as follows: stS ⁽ⁱ⁾ = S+||E ⁽ⁱ⁾ || ₀ where S represents the learned pairwise similarity matrix between robust samples, E ⁽ⁱ⁾ represents the noise in the i-th pairwise similarity matrix, rank( ) represents the rank of the matrix, and ||·|| ₀ represents l ₀ norm; (5) The objective function in (4) above has constraints of discrete low rank and l ₀ norm, the above formula can be rewritten as stS ⁽ⁱ⁾ = S+||E ⁽ⁱ⁾ || ₁ where ||·|| _* represents the nuclear norm, ||·|| ₁ represents the l ₁ norm, (6) Use the augmented Lagrangian multiplier method to solve this problem, and obtain a robust similarity matrix between two samples; Step 5: Construct the objective function, which specifically includes the following steps: (1) In the Hamming space, the similarity based on the similarity matrix between two samples is maintained, and the objective function of hash code learning is defined as follows: Where k represents the length of the hash code, _B1 is the hash code of the image modality sample, _B2 is the hash code of the text modality sample, and λ is the weight parameter; (2) Use the linear map as the hash function, and use the _l2,1 norm as the regular item to constrain the learning of image and text modal hash functions. The objective function of each modal hash function learning is defined as follows: Among them, W ₁ and W ₂ represent the hash functions of the image modality and the text modality respectively, and Reg(·) represents the regular term to prevent overfitting, here β _i and μ are weight parameters; (3) Adding the objective function of hash code and hash function learning is the objective function of this method, which is defined as follows: Where β _i is the weight parameter; Step 6: The objective function is solved using an iterative optimization algorithm. The solution process includes the following steps: (1) Fix W ₁ , W ₂ and B ₂ , and solve for B ₁ : Remove the constant term, the objective function can be written as: Here it can be solved sample by sample, let b _1i represent the i-th column of B ₁ , b _2j represent the j-th column of B ₂ , and the objective function can be written as: Here, the cyclic coordinate gradient descent method is used to solve bit by bit, let b _1im represent the mth bit of b _1i , Indicates that b _1i is a vector composed of bits other than the mth bit, then b _1im can be obtained by the following formula: Repeat the above formula until the hash codes of all image modality samples are solved; (2) Fix W ₁ , W ₂ and B ₁ , solve for B ₂ : The same method as solving B ₁ , we can get Repeat the above formula until the hash codes of all text modal samples are solved; (3) Fix W ₂ , B ₁ and B ₂ , and solve for W ₁ : Remove the constant term, the objective function can be written as: There is a closed solution to this problem where _D1 is a diagonal matrix, (4) Fix W ₁ , B ₁ and B ₂ , and solve for W ₂ : Same as the method for solving W ₁ , W ₂ has a closed solution where _D2 is a diagonal matrix, (5) Repeat (1)-(4) until the algorithm converges or reaches the maximum number of iterations; Step 7: The user enters the query sample, extracts its features, and removes the average value of the extracted features; Step 8: Use the learned hash function to generate the hash code of the query sample: Step 9: Calculate the Hamming distance between the query sample and the heterogeneous samples in the training set, and arrange the Hamming distance in ascending order, and the samples corresponding to the first r Hamming distances are the retrieval results.