CN111737507A - A Single-modal Image Hash Retrieval Method - Google Patents

Abstract

The invention particularly relates to a single-mode image hash retrieval method. The single-mode image Hash retrieval method comprises four parts of image preprocessing, image feature extraction, attention image output and Hash retrieval model generation. According to the single-mode image Hash retrieval method, semantic information in a picture mode is extracted through an attention mechanism, the quality of Hash function generated by a Hash retrieval model is improved, meanwhile, the retrieval precision among a plurality of label data is enhanced through a multi-stage semantic supervision mode, the most matched item is located in front of the final retrieval result, and therefore the retrieval efficiency is greatly improved.

Description

A Single-modal Image Hash Retrieval Method

technical field

The invention relates to the technical field of image retrieval, in particular to a single-modal image hash retrieval method.

Background technique

With the advancement of science and technology, the rapid development of Internet technology, and the rapid technological updates, there has been a big explosion of image and video data. Conventional image retrieval technologies include two retrieval methods: Text-based Image Retrieval (TBIR) and Content-based Image Retrieval (CBIR). Among them, the text-based image retrieval technology uses text description to describe the characteristics of the image, such as the author, age, genre, size, etc. of the painting; content-based image retrieval technology is the content semantics of the image, such as the color, Image retrieval technology for analysis and retrieval of textures, layouts, etc. At present, content-based image retrieval technology has become the mainstream image retrieval method.

Image hash retrieval technology aims to search the existing data set to find the image data that meets the requirements. Because hash codes have the advantages of small storage data and fast retrieval speed, hash retrieval is widely used in retrieval tasks. Existing image hash retrieval techniques can be divided into deep model retrieval techniques and non-deep model retrieval techniques. The traditional method is generally to use a deep network to extract image features, and use a fully connected network to convert the samples into hash codes and store them in the database at cross entropy loss according to the extracted features.

Since in the real environment, an image contains a lot of rich information, there are often multiple classes of information, and the traditional information for one class is often not accurate enough, redundant information in the background in the image and areas worthy of attention. Information occupies the same place in the hash learning process. However, most of the existing hash retrieval models only focus on the information in the image worthy of attention, and cannot make full use of all the image information.

Based on the above problems, the present invention proposes a single-modal image hash retrieval method.

SUMMARY OF THE INVENTION

In order to make up for the defects of the prior art, the present invention provides a simple and efficient single-modal image hash retrieval method.

The present invention is achieved through the following technical solutions:

A single-modal image hash retrieval method, characterized in that it includes four parts: image preprocessing, image feature extraction, outputting attention images and generating a hash retrieval model;

Firstly, the rich semantic information in the multi-label data is maintained by defining a multi-level semantic similarity relationship matrix. At the same time, the Attention mechanism is used to spontaneously find the key attention areas in the image, and the mask of the same size as the image representation is generated by learning to extract the image modalities. Semantic information in the auxiliary hash retrieval model to obtain a higher quality hash function.

The single-modal image hash retrieval method of the present invention, the specific implementation steps are as follows:

The first step is to obtain the original image of the training set, and input the images corresponding to different residual networks;

The second step is to input the training samples into the hash retrieval model, and optimize the parameters of the hash retrieval model by minimizing the loss function;

The third step is to fix the model, obtain the corresponding hash codes of all samples through the hash retrieval model, and store them in the input library for use;

The fourth step, when using the hash retrieval model for retrieval tasks, only need to input any modal sample of the picture into the model to generate the corresponding hash code of the modal, and then search for the nearest Hamming distance in the hash code database of another modal. The N hash codes (customized according to requirements) are returned, and the corresponding samples can be returned.

In the second step, an iterative optimization method is used to optimize the model parameters, that is, one parameter is fixed and another parameter is optimized.

In the second step, optimizing the hash retrieval model includes the following steps:

(1) Generate a similarity matrix S with multi-level semantics;

(2) extracting the feature of the image modality, obtaining the image modality feature P _i , and classifying the image, and outputting the attention image;

(3) Dot multiplication of the obtained feature image and the attention image to obtain the feature representation F _{i of the picture modality and the feature representation F j of} the text modality;

(4) Use the loss function to iteratively optimize the hash retrieval model, and finally obtain the optimized hash retrieval model.

In the step (1), the similarity matrix S with multi-level semantics is expressed as:

Among them, |C _i | and |C _j | represent the number of categories of sample i and sample j, respectively, D(i,j) represents the number of categories shared by the two samples; the similarity composed of sample i and sample j The matrix S _ij ∈ [0,1], thus ensuring that the generated S matrix has greater discrimination.

In the step (2), the Resnet101 network is used for extraction, and the image modal feature P _i is obtained; at the same time, the Resnet01 network is used, the fully connected layer is removed, the average pooling layer is added, and the output is sample category data, and the image is classified. , the last layer adds the Attention mechanism, the output is an attention image, and the focus area is activated.

In the described step (3), the obtained feature image and the attention image are dot multiplied, and the obtained result is used as an input to enter the fully connected layer to obtain the feature representation F _i of the picture mode; the BOW ( Bag of words) means input to the fully connected layer to obtain the feature representation F _j of the text mode.

In the step (4), the loss function is expressed as:

Among them, S _ij is the similarity matrix composed of sample i and sample j, σ is a hyperparameter used to balance the penalty term and data loss term, F ^T _i is the transpose of the feature representation of the image modality, and F _j is the text The feature representation of the modality, L ₂ is the common quantization loss, and L ₃ is the bit balance loss.

The beneficial effects of the present invention are: the single-modal image hash retrieval method extracts the semantic information in the image mode through the attention mechanism, improves the quality of the hash function generated by the hash retrieval model, and at the same time uses the multi-level semantic supervision method. , which enhances the retrieval accuracy between data with multiple tags, so that the most matching item is located in front of the final retrieval result, thus greatly improving retrieval efficiency.

Description of drawings

In order to illustrate the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are For some embodiments of the present invention, for those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative efforts.

FIG. 1 is a schematic diagram of a single-modal image hash retrieval method according to the present invention.

Detailed ways

In order to make those skilled in the art better understand the technical solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some, but not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

The single-modal image hash retrieval method includes four parts: image preprocessing, image feature extraction, output attention image and generation of hash retrieval model;

Firstly, the rich semantic information in the multi-label data is maintained by defining a multi-level semantic similarity relationship matrix. At the same time, the Attention mechanism is used to spontaneously find the key attention areas in the image, and the mask with the same size as the image representation is generated by learning to extract the image modalities. Semantic information in the auxiliary hash retrieval model to obtain a higher quality hash function.

The specific implementation steps of the single-modal image hash retrieval method are as follows:

The first step is to obtain the original image of the training set, and input the images corresponding to different residual networks;

The second step is to input the training samples into the hash retrieval model, and optimize the parameters of the hash retrieval model by minimizing the loss function;

The third step is to fix the model, obtain the corresponding hash codes of all samples through the hash retrieval model, and store them in the input library for use;

The fourth step, when using the hash retrieval model for retrieval tasks, only need to input any modal sample of the picture into the model to generate the corresponding hash code of the modal, and then search for the nearest Hamming distance in the hash code database of another modal. The N hash codes (customized according to requirements) are returned, and the corresponding samples can be returned.

In the second step, an iterative optimization method is used to optimize the model parameters, that is, one parameter is fixed and another parameter is optimized.

In the second step, optimizing the hash retrieval model includes the following steps:

(1) Generate a similarity matrix S with multi-level semantics;

(2) extracting the feature of the image modality, obtaining the image modality feature P _i , and classifying the image, and outputting the attention image;

(3) Dot multiplication of the obtained feature image and the attention image to obtain the feature representation F _{i of the picture modality and the feature representation F j of} the text modality;

(4) Use the loss function to iteratively optimize the hash retrieval model, and finally obtain the optimized hash retrieval model.

In the step (1), the similarity matrix S with multi-level semantics is expressed as:

Among them, |C _i | and |C _j | represent the number of categories of sample i and sample j, respectively, D(i,j) represents the number of categories shared by the two samples; the similarity composed of sample i and sample j The matrix S _ij ∈ [0,1], thus ensuring that the generated S matrix has greater discrimination.

In the step (2), the Resnet101 network is used for extraction, and the image modal feature P _i is obtained; at the same time, the Resnet01 network is used, the fully connected layer is removed, the average pooling layer is added, and the output is sample category data, and the image is classified. , the last layer adds the Attention mechanism, the output is an attention image, and the focus area is activated.

In the described step (3), the obtained feature image and the attention image are dot multiplied, and the obtained result is used as an input to enter the fully connected layer to obtain the feature representation F _i of the picture mode; the BOW ( Bag of words) means input to the fully connected layer to obtain the feature representation F _j of the text mode.

In the step (4), the loss function is expressed as:

Among them, S _ij is the similarity matrix composed of sample i and sample j, σ is a hyperparameter used to balance the penalty term and data loss term, F ^T _i is the transpose of the feature representation of the image modality, and F _j is the text The feature representation of the modality, L ₂ is the common quantization loss, and L ₃ is the bit balance loss.

Compared with the current prior art, the single-modal image hash retrieval method has the following characteristics:

First, the focus of attention is separated from redundant data, and the key areas are highlighted, thereby improving the retrieval efficiency;

In the previous image retrieval, most of the existing hash retrieval models for image information could not be fully utilized. The background and redundant information in the image and the information of the area worthy of attention occupy the same position in the hash learning process. The method separates background and redundant information from areas worthy of key attention, which can greatly improve retrieval efficiency.

Second, through the Attention mechanism, the hash retrieval model can focus on more discriminative regions, which improves the quality of the hash function generated by the model;

In recent years, the Attention mechanism has been widely used in computer vision, and has achieved good results. The Attention mechanism is used in image recognition, and can spontaneously find the places in the image that need to be focused, that is, by learning to generate a Mask with the same size as the image representation, for the area of interest, the corresponding position of the Mask has a higher area of attention, and the Attention mechanism is integrated. into the hash retrieval method to make it more interpretable and improve retrieval efficiency.

Third, the multi-level semantic supervision method is used to enhance the retrieval accuracy of data with multiple labels, so that the most matching item is located in front of the final retrieval result.

The above-mentioned embodiment is only one of the specific embodiments of the present invention, and the usual changes and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included in the protection scope of the present invention.

Claims (8)

1. a unimodal image hash retrieval method is characterized in that: comprise image preprocessing, image feature extraction, output attention image and generate hash retrieval model four parts; Firstly, the rich semantic information in the multi-label data is maintained by defining a multi-level semantic similarity relationship matrix. At the same time, the Attention mechanism is used to spontaneously find the key attention areas in the image, and the mask with the same size as the image representation is generated by learning to extract the image modalities. Semantic information in the auxiliary hash retrieval model to obtain a higher quality hash function. 2. The single-modality image hash retrieval method according to claim 1, wherein the specific implementation steps are as follows: The first step is to obtain the original image of the training set, and input the images corresponding to different residual networks; The second step is to input the training samples into the hash retrieval model, and optimize the parameters of the hash retrieval model by minimizing the loss function; The third step is to fix the model, obtain the corresponding hash codes of all samples through the hash retrieval model, and store them in the input library for use; The fourth step, when using the hash retrieval model for retrieval tasks, only need to input any modal sample of the picture into the model to generate the corresponding hash code of the modal, and then search for the nearest Hamming distance in the hash code database of another modal. The n hash codes of , and the corresponding samples can be returned. 3 . The single-modality image hash retrieval method according to claim 2 , wherein in the second step, an iterative optimization method is used to optimize model parameters, that is, one parameter is fixed and another parameter is optimized. 4 . 4. The single-modality image hash retrieval method according to claim 3, wherein: in the second step, optimizing the hash retrieval model comprises the following steps: (1) Generate a similarity matrix S with multi-level semantics; (2) extracting the feature of the image modality, obtaining the image modality feature P _i , and classifying the image, and outputting the attention image; (3) Dot multiplication of the obtained feature image and the attention image to obtain the feature representation F _{i of the picture modality and the feature representation F j of} the text modality; (4) Use the loss function to iteratively optimize the hash retrieval model, and finally obtain the optimized hash retrieval model. 5. The single-modal image hash retrieval method according to claim 4, wherein: in the step (1), the similarity matrix S with multi-level semantics is expressed as:

Among them, |C _i | and |C _j | represent the number of categories of sample i and sample j, respectively, D(i,j) represents the number of categories shared by the two samples; the similarity composed of sample i and sample j The matrix S _ij ∈ [0,1], thus ensuring that the generated S matrix has greater discrimination. 6. single-modal image hash retrieval method according to claim 4, is characterized in that: in described step (2), adopt Resnet101 network to extract, and obtain image modal characteristic P _i ; Adopt Resnet01 network simultaneously, The fully connected layer is removed, the average pooling layer is added, the output is sample category data, and the image is classified. 7. The single-modal image hash retrieval method according to claim 6, wherein: in the step (3), the obtained feature image and the attention image are dot-multiplied, and the obtained result is used as Input into the fully connected layer to obtain the feature representation F _i of the picture modality; input the BOW representation of the text modality into the fully connected layer to obtain the feature representation _Fj of the text modality. 8. The single-modal image hash retrieval method according to claim 4, 5, 6 or 7, characterized in that: in the step (4), the loss function is expressed as:

Among them, S _ij is the similarity matrix composed of sample i and sample j, σ is a hyperparameter used to balance the penalty term and data loss term, F ^T _i is the transpose of the feature representation of the image modality, and F _j is the text The feature representation of the modality, L ₂ is the common quantization loss, and L ₃ is the bit balance loss.

Images