CN112650869B - Image retrieval reordering method and device, electronic equipment and storage medium - Google Patents
Image retrieval reordering method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN112650869B CN112650869B CN202011535678.1A CN202011535678A CN112650869B CN 112650869 B CN112650869 B CN 112650869B CN 202011535678 A CN202011535678 A CN 202011535678A CN 112650869 B CN112650869 B CN 112650869B
- Authority
- CN
- China
- Prior art keywords
- image
- candidate
- target
- determining
- feature vector
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/53—Querying
- G06F16/538—Presentation of query results
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Abstract
The invention provides an image retrieval reordering method, an image retrieval reordering device, electronic equipment and a storage medium, wherein target examples possibly contained in each candidate image can be preliminarily excavated by extracting interest areas in each candidate image and determining foreground characteristics corresponding to the interest areas in each candidate image, so that the influence of background noise on reordering results is reduced; by introducing the correlation between the image features of the target image and the foreground features corresponding to the interest areas in the candidate images, the query feature vectors of the target image can be determined by effectively utilizing the foreground features; through the determined query feature vector, each candidate image is reordered, so that the reordering precision is improved, and the reordering result is more accurate.
Description
Technical Field
The present invention relates to the field of image retrieval reordering technology, and in particular, to an image retrieval reordering method and apparatus, an electronic device, and a storage medium.
Background
The image retrieval reordering refers to secondary ordering of a plurality of images in the front row in the initial ordering result to obtain a new ordering result, so that the image retrieval performance is improved. From the perspective of a user, the image retrieval reordering can improve the user experience; for a retrieval system, image retrieval reordering can effectively improve image retrieval performance.
The existing image retrieval and reordering methods generally include re-ordering methods such as extended query, spatial verification and the like, for example, retrieval and reordering are performed by using multi-modal features of images; enhancing the image characteristics, and retrieving and reordering by utilizing the enhanced characteristics; performing retrieval reordering by performing correlation verification of spatial feature distribution on local features of the image; carrying out retrieval reordering through the characteristic correlation weight; and carrying out retrieval reordering by an active verification mode.
However, the above image retrieval reordering methods all rely on global features or local features of the image, for example, although the feature vector output by the convolutional neural network contains image semantic information, the content distribution and structure of the image are lost after pooling or full-link processing, which affects the ordering result. The local features are adopted for reordering, so that the reordering is sensitive to noise and greatly influenced by the target environment and posture.
Disclosure of Invention
The invention provides an image retrieval reordering method, an image retrieval reordering device, electronic equipment and a storage medium, which are used for overcoming the defects in the prior art.
The invention provides an image retrieval reordering method, which comprises the following steps:
extracting interest areas in the candidate images based on the image features of the target image and the image features of the candidate images, and determining foreground features corresponding to the interest areas in the candidate images;
determining a query feature vector of the target image based on the correlation between the image features of the target image and foreground features corresponding to the interest areas in the candidate images;
and reordering the candidate images based on the query feature vector.
According to the image retrieval reordering method provided by the invention, the extracting of the interest region in each candidate image based on the image features of the target image and the image features of each candidate image specifically comprises:
weighting a set feature vector containing the image features of the target image and the image features of each candidate image in a channel dimension to obtain a weighted feature vector;
determining a basis vector of the weighted feature vector based on a statistical method, and determining a projection of the weighted feature vector on the basis vector;
based on the projections, regions of interest in the candidate images are determined.
According to the image retrieval reordering method provided by the present invention, weighting the set feature vector including the image feature of the target image and the image feature of each candidate image in the channel dimension to obtain the weighted feature vector, the method further includes:
calculating the difference value of the initial query feature vector of the target image and the global feature vector of any candidate image, and determining the gradient value of the difference value which is reversely propagated to the target feature map of the target image and the candidate feature map of any candidate image;
determining a mean value of the gradient values in a channel dimension, and determining a weight vector for weighting the image features of the candidate images in the set feature vector in the channel dimension based on the mean value.
According to the image retrieval reordering method provided by the invention, the determining of the region of interest in each candidate image based on the projection specifically comprises:
determining projection components corresponding to candidate images in the projection;
and if the projection component corresponding to any candidate image is larger than 0, determining that the interest area in any candidate image is the projection area of the projection component corresponding to any candidate image.
According to the image retrieval reordering method provided by the invention, the query feature vector of the target image is determined based on the correlation between the image feature of the target image and the foreground feature corresponding to the interest region in each candidate image, and the method specifically comprises the following steps:
determining comprehensive relevance response characteristics of the foreground characteristics corresponding to the interest areas in the candidate images to the image characteristics of the target image based on the relevance of the foreground characteristics corresponding to the interest areas in the candidate images to the image characteristics of the target image;
and determining an activation response characteristic of the target image based on an activation function, and determining a query feature vector of the target image based on the comprehensive relevance response characteristic and the activation response characteristic.
According to the image retrieval reordering method provided by the invention, the determining the query feature vector of the target image based on the comprehensive relevance response feature and the activation response feature specifically comprises:
weighting a target feature map of the target image by taking the sum of the comprehensive correlation response feature and the activation response feature as a weight;
and determining a query feature vector of the target image based on the weighted target feature map.
According to the image retrieval reordering method provided by the invention, the extracting of the interest region in each candidate image based on the image features of the target image and the image features of each candidate image further comprises:
and determining each candidate image in the image base library based on the correlation between the initial query feature vector of the target image and the global image feature vector of each base library image in the image base library.
The invention also provides an image retrieval reordering device, comprising: the device comprises a foreground characteristic determining module, a query characteristic vector determining module and a reordering module. Wherein the content of the first and second substances,
the foreground characteristic determining module is used for extracting interest areas in the candidate images based on the image characteristics of the target image and the image characteristics of the candidate images and determining foreground characteristics corresponding to the interest areas in the candidate images;
the query feature vector determination module is used for determining a query feature vector of the target image based on the correlation between the image feature of the target image and the foreground feature corresponding to the interest region in each candidate image;
and the reordering module is used for reordering the candidate images based on the query feature vector.
The invention also provides an electronic device, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor executes the computer program to realize the steps of the image retrieval and reordering method.
The present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the image retrieval reordering method as described in any of the above.
The invention provides an image retrieval reordering method, an image retrieval reordering device, electronic equipment and a storage medium, wherein firstly, based on the image characteristics of a target image and the image characteristics of each candidate image, interest areas in each candidate image are extracted, and foreground characteristics corresponding to the interest areas in each candidate image are determined; then determining query feature vectors of the target image based on the correlation between the image features of the target image and foreground features corresponding to the interest areas in the candidate images; and finally, reordering the candidate images based on the query feature vector. In the embodiment of the invention, by extracting the interest areas in the candidate images and determining the foreground characteristics corresponding to the interest areas in the candidate images, target examples possibly contained in the candidate images can be initially excavated, and the influence of background noise on the reordering result is reduced; by introducing the correlation between the image characteristics of the target image and the foreground characteristics corresponding to the interest areas in the candidate images, the query characteristic vectors of the target image can be determined by effectively utilizing the foreground characteristics; through the determined query feature vector, each candidate image is reordered, so that the reordering precision is improved, and the reordering result is more accurate.
Drawings
In order to more clearly illustrate the technical solutions of the present invention or the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 is a schematic flow chart of a method for retrieving and reordering images according to the present invention;
FIG. 2 is a schematic view of a complete flow chart of the image retrieval and reordering method provided by the present invention;
FIG. 3 is a schematic structural diagram of an image retrieving and reordering apparatus according to the present invention;
fig. 4 is a schematic structural diagram of an electronic device provided in the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
With the rapid development of multimedia technology and network technology, the variety and amount of mass digital image data are increasing. How to quickly and accurately retrieve image data contents required by all users from some large image databases and present retrieval results to existing users in an accurate order is one of the hot problems in the research of various image retrieval systems. Currently, the correlation index based on the image content features is widely applied in the image retrieval field. As one of the important parts of image content retrieval, the ranking optimization algorithm determines the final result of the retrieval seen by the user. Therefore, the performance of the ranking algorithm plays an important role in the performance of image retrieval. In brief, reordering is the secondary ordering of the top-ranked images in the original ranking list to obtain a new ranking list. Reordering can screen discriminative features for query, thus improving retrieval accuracy. The image retrieval method especially aims at the problems that the image instance retrieval method is often confronted with the problems of disordered retrieval target backgrounds, intra-class difference and inter-class similarity of target instances and the like, and the retrieval effect is influenced. Since image retrieval reordering is a process of correcting an initial ordering result, image features are generally fused and then similarity is recalculated or supplementary information is added for secondary verification.
In an image retrieval task, a reordering method comprises reordering methods such as extended query and spatial verification, and specifically comprises the following steps:
(1) And (3) utilizing the multi-modal features to conduct retrieval reordering:
because the single-mode characteristics are difficult to completely express the image correlation and distinctiveness, some reordering methods combine reordering by using color, outline shape and texture information, or combine text keywords to reorder. The general method is to adjust and update the initial ranking according to the correlation of the supplementary features after the initial ranking is performed based on the retrieval features, or to directly use the combined features to retrieve images in front of the ranking to obtain a new ranking relation. The search efficiency is greatly influenced due to the need of additionally calculating and storing different modal characteristics.
(2) And (3) retrieving and reordering after feature enhancement:
the method excavates the relevance of the features in the search library and the relevance of the query image and the search image to update the features, so as to obtain a more accurate sequencing relation; the principle of the extended Query is to update the image features of the top rank in the library by combination as new Query features and retrieve and rearrange, wherein a typical Average extended Query (AQE) method uses the Average features of the top rank images to issue new queries. Another approach, DBA, updates only the features in the database, updating each feature in the library to a weighted sum of the original feature and its K nearest neighbor feature.
Such methods mine global features of an image, but simply by simple combination of global features, lack efficient mining of the relevance of features and efficient utilization of local features and target information.
(3) And (3) space verification:
the retrieval based on the Local features requires encoding the Local features to obtain encoded retrieval features, which are encoded in common ways such as Bag-of-Words model (BoW) encoding and related Local Aggregated descriptor Vectors (VLAD) and fishery algorithms (FV), and the features lose position distribution information of the Local features after encoding. In order to effectively utilize the local feature position information, when the retrieval and the reordering are carried out, the spatial consistency of the local feature distribution is solved by matching the local features, and the verification and the ordering are carried out according to the correlation of the spatial feature distribution.
The method needs to match local features, has high calculation complexity, and in addition, the local features and the space consistency thereof cannot effectively represent the consistency of image semantic information, so that the problem of large target posture and environment change is difficult to solve.
(4) Characteristic correlation weight:
the method redefines the measurement mode of the characteristic correlation, for example, the characteristic distance calculation is changed according to the characteristic prior information, and different distance calculation modes are adopted to mine the characteristic correlation information. Typical methods such as K-receiprocal calculate K-receiprocal nearest neighbor by using Jaccard distance and Mahalanobis distance, and supplement K nearest neighbor correlation information of images in a library as distance measurement to obtain more accurate sequencing results. The method does not carry out mining and processing on local characteristic information and noise as an extended query method, and completely depends on query global characteristics for sequencing.
(5) Active verification reordering:
the method needs the user to select the retrieval result and then update the retrieval sequence according to the interaction result, and needs an additional interaction process due to the combination of the subjective prior information of the user.
However, the image retrieval reordering method in the prior art has the following defects:
1) And the local information and the global information of the image are independently relied to carry out reordering, and the information is not fully utilized. Most of the existing reordering methods rely on global features or local features of images independently, for example, although feature vectors output by a convolutional neural network contain image semantic information, the content distribution and structure of the images are lost after pooling or full-link processing, and the ordering result is influenced. The local features are adopted for reordering, so that the reordering is sensitive to noise and greatly influenced by the target environment and posture.
2) The image correlation information mining is insufficient, and the influence of intra-class difference and inter-class similarity of the images is large. The existing method such as an extended query reordering method only simply recombines and updates the features, lacks of distinguishing and utilizing foreground and background information, and cannot effectively avoid the influence on background noise, shielding and retrieval negative cases.
In summary, the image retrieval reordering methods in the prior art all rely on global features or local features of an image, for example, although a feature vector output by a convolutional neural network contains semantic information of the image, the content distribution and structure of the image are lost after pooling or full-link processing, and the ordering result is affected. The local features are adopted for reordering, so that the reordering is sensitive to noise and greatly influenced by the target environment and posture. Therefore, the embodiment of the invention provides an image retrieval reordering method to solve the technical problems in the prior art.
Fig. 1 is a schematic flow chart of an image retrieval and reordering method according to an embodiment of the present invention, as shown in fig. 1, the method includes:
s1, extracting interest areas in the candidate images based on the image features of the target image and the image features of the candidate images, and determining foreground features corresponding to the interest areas in the candidate images;
s2, determining a query feature vector of the target image based on the correlation between the image features of the target image and foreground features corresponding to the interest areas in the candidate images;
and S3, reordering the candidate images based on the query feature vector.
Specifically, in the image retrieval reordering method provided in the embodiment of the present invention, the execution main body is a server, and may be specifically a local server or a cloud server, and the local server may be a computer, a tablet computer, or a smart phone, which is not specifically limited in the embodiment of the present invention. The application scene is the sequencing display after the image retrieval, namely, the target image is given, and after all the images of the image base required by the user are retrieved from the image base, the retrieval result needs to be presented to the user in an accurate sequence. The operation objects are all bottom library images in the preliminary ranking result obtained by preliminary ranking the retrieval result, namely all candidate images. The operational action is to reorder the candidate images.
Step S1 is performed first. Each candidate image refers to each base image in the preliminary ranking result obtained after the search result is preliminarily ranked, the number of the candidate images can be selected according to needs, and specifically, the first N base images in the preliminary ranking result can be selected as the candidate images, that is, the number of the candidate images is N, and N is greater than or equal to 1.
The image feature of the target image q may refer to a local feature in the target image q, and may be represented by a local feature vector. All image features of the target image q may form a target feature map Fq, which includes a plurality of local feature vectors. The target feature map Fq may be specifically output by inputting the target image q to the trained convolutional neural network model. The convolutional neural network model may be specifically constructed based on a twin convolutional neural network, which is not specifically limited in the embodiment of the present invention. The size of the target feature map Fq may be h × w, h is the height of the target feature map Fq, w is the width of the target feature map Fq, and h × w represents the number of image features of the target image q, that is, the number of local feature vectors included in the target feature map Fq. The dimension of each local feature vector may be the c dimension, i.e. each local feature vector has a channel number c. For any candidate image g (1 ≦ g ≦ N), the image feature of the candidate image g may refer to a local feature in the candidate image g, and may be represented by a local feature vector. All image features of the candidate image g may constitute a candidate feature map Fg, which includes a plurality of local feature vectors. The candidate feature map Fg may be specifically output by inputting the candidate image g to the trained convolutional neural network model. The size of the candidate feature map Fg may be h × w, h being the height of the candidate feature map Fg, w being the width of the target candidate feature map Fg, and h × w indicating the number of local feature vectors included in the candidate feature map Fg. The dimension of each local feature vector may be the c dimension, i.e. each local feature vector has a channel number c.
According to the image characteristics of the target image q and the image characteristics of the candidate image g, the interested region in the candidate image g can be extracted. The region of interest in the candidate image g is a foreground region of the candidate image g, and may specifically be a region that may include a target example, where the target example is content commonly included in the target image q and the candidate image g. The extraction of the interest region in the candidate image g can be obtained through a trained convolutional neural network model. And (5) sequentially taking values of g from 1 to N, and repeating the process to extract the interest areas in the candidate images. It should be noted that the region of interest is not included in all the candidate images, and therefore the region of interest in each candidate image may be empty.
After the interest areas in the candidate images are extracted, foreground features corresponding to the interest areas in the candidate images can be further determined. For the candidate image g, the foreground feature of the candidate image g may be a global feature for characterizing a region of interest in the candidate image g. The foreground features may be represented by a foreground feature vector, which is a global feature vector used to identify the region of interest of the candidate image g.
Then step S2 is performed. Since the image features of the target image q can be represented by the local feature vectors, and the foreground features corresponding to the region of interest in the candidate image g can be represented by the foreground feature vectors, the similarity between the image features of the target image q and the foreground features corresponding to the region of interest in the candidate image g can be represented by the similarity between the local feature vectors of the target image q and the foreground feature vectors. The similarity may be determined by an inner product of two vectors obtained by calculation, or may be determined in other manners, which is not specifically limited in the embodiment of the present invention.
And according to the correlation between the image features of the target image and the foreground features corresponding to the interest areas in the candidate images, the query feature vector of the target image can be determined again. When the query feature vector of the target image is re-determined, specifically, the weight of a target feature map formed by the image features of the target image can be determined according to the correlation between the image features of the target image and the foreground features corresponding to the interest areas in the candidate images, then the target feature map is weighted by the weight, and the global image features of the target image are obtained through pooling, so that the query feature vector of the target image is obtained. That is, the query feature vector of the target image is actually a global image feature that identifies the target image.
Finally, step S3 is performed. And reordering the candidate images according to the query feature vector. The process is consistent with the process of primarily sorting images in the image base, and the difference is only that the adopted query feature vector is different, the query feature vector adopted in the step S3 is determined through the step S1 and the step S2, and the query feature vector adopted in the primary sorting is the global image feature directly determined according to the target feature map of the target image. For example, the global image feature of each candidate image may be determined and represented by a global feature vector, then the similarity between the query feature vector and each global feature vector is calculated, and finally all candidate images are sorted in the order of the corresponding similarity from high to low. This process is a reordering process, and the result of the ordering is the result of the reordering.
The image retrieval reordering method provided by the embodiment of the invention comprises the steps of firstly extracting interest areas in each candidate image based on the image characteristics of a target image and the image characteristics of each candidate image, and determining foreground characteristics corresponding to the interest areas in each candidate image; then determining query feature vectors of the target image based on the correlation between the image features of the target image and foreground features corresponding to the interest areas in the candidate images; and finally, reordering the candidate images based on the query feature vector. In the embodiment of the invention, by extracting the interest areas in the candidate images and determining the foreground characteristics corresponding to the interest areas in the candidate images, target examples possibly contained in the candidate images can be initially excavated, and the influence of background noise on the reordering result is reduced; by introducing the correlation between the image features of the target image and the foreground features corresponding to the interest areas in the candidate images, the query feature vectors of the target image can be determined by effectively utilizing the foreground features; through the determined query feature vector, each candidate image is reordered, so that the reordering precision is improved, and the reordering result is more accurate.
On the basis of the foregoing embodiment, the image retrieval reordering method provided in the embodiments of the present invention is that extracting regions of interest in each candidate image based on the image features of the target image and the image features of each candidate image, and specifically includes:
weighting a set feature vector containing the image features of the target image and the image features of each candidate image in a channel dimension to obtain a weighted feature vector;
determining a basis vector of the weighted feature vector based on a statistical method, and determining a projection of the weighted feature vector on the basis vector;
based on the projections, regions of interest in the candidate images are determined.
Specifically, when the interest areas in the candidate images are extracted according to the image features of the target image and the image features of the candidate images, the interest areas in the candidate images can be extracted in batch at the same time, and the batch extraction of the interest areas can be realized through the trained convolutional neural network model.
First, the target image and each candidate image may be made up of an image set, i.e., the image setThe image processing device comprises a target image and N candidate images, wherein the number of the candidate images is N + 1. Inputting an image set into a trained convolutional neural network model, extracting image features of each image in the image set by using the convolutional neural network model, obtaining a target feature map after extracting the image features of the target image, obtaining N candidate feature maps after extracting the image features of the N candidate images, wherein the image features contained in the target feature map and the image features contained in the N candidate feature maps can form a set feature vector D of the image set set I.e. the set of feature vectors D set The image feature of the target image and the image features of the N candidate images are included. Set feature vector D set The shape of (c) can be expressed as (N +1, h, w, c), wherein N +1 represents the number of images in the image set, that is, the number of feature maps obtained after the image set passes through a convolutional neural network model, w and h represent the width and height of the feature maps respectively, and c represents the number of channels of the feature maps.
In order to enhance the feature distinguishing information and the foreground related information, the set feature vectors are weighted in the channel dimension, that is, the image features of the N +1 feature maps on each channel are weighted, the weight values corresponding to the image features at different positions in different feature maps of the same channel may be the same or different, and may be specifically determined according to the needs, which is not specifically limited in the embodiment of the present invention. Set feature vector D set Corresponding to a weight vector set as W ∈ R c The weight vector set W comprises N +1 weight vectors W with the length of c n (N is more than or equal to 1 and less than or equal to N + 1). Set feature vector D set In (D) set Any one of D set,n May include image features in the nth feature map, D set,n The corresponding weight vector is W n Weight vector W n And representing the weight value corresponding to each image feature in the nth feature map. If the weighted eigenvector obtained after weighting the set eigenvector in the channel dimension is D w Then, there are:
D w,n ＝D set，n *W n
wherein D is w,n To weightThe feature vector is D w The nth term in (b) corresponds to the weighted image feature in the nth feature map. I.e. D w Each item D of w,n Is D set Each item D set,n Corresponding weight W n Multiplication results in channel dimensions.
Determination of weighted feature vectors D by statistical methods w And determining a weighted feature vector D w Projection onto the basis vectors. The statistical method may specifically include a Principal Component Analysis (PCA), a Linear Discriminant Analysis (LDA), a Weighted Principal Component Analysis (WPCA), and the like, which is not specifically limited in the embodiments of the present invention. Due to D w Is passing through pair D set The weighting is carried out on the channel dimension, so that the image features of different channels are adjusted by the weight vector, the image features of the candidate image are reserved at the parts, close to the image features of the target image, of the corresponding channels, the parts, close to the features, in the candidate image are strengthened, namely the parts, close to the features, in the candidate image are the interested areas, namely the foreground areas in the candidate image, and the basis vector obtained by principal component analysis is closer to foreground feature expression.
To ensure the efficiency and accuracy of the base vector determination based on statistical methods, the weighted feature vector D may be first weighted w Performing a pretreatment of w Transforming the shape of the model into a tensor of (N + 1) × h × w, c) from the shape of (N +1, h, w, c) and normalizing the tensor to obtain D rw ，D rw Representing a feature vector having (N + 1) × h × w lengths c. Then determining D according to statistical method rw The basis vector of (2). Taking the statistical method as the PCA as an example, the eigenvector corresponding to the maximum eigenvalue can be specifically taken as D rw Is (xi ∈ R) c ) The length of the basis vector is c.
D rw Is also D w Is used to represent the tensor of the basis vector of (a). Finally determine D w Projection on its basis vector, i.e. determining D rw Projection P on basis vector xi set ，P set Is ((N + 1) × h × w, 1). P set In particular toComprises the following steps:
P set ＝D rw *ξ
will P set Transform back to weighted eigenvector D w I.e. the shape is converted from ((N + 1) × h × w, 1) to ((N + 1), h, w, 1). Wherein ((N + 1), h, w, 1) can be further represented as ((N + 1), h, w). Will P set After transforming the shape, N +1 projection components can be obtained:
P set ＝{P 1 ，P 2 ，…，P n ,…，P N+1 }
wherein, P n Is P set The nth projection component represents a two-dimensional matrix of shape (h, w), i.e., P n The composite material contains h × w elements. P n Corresponding to the nth candidate image.
Determination of D rw Projection P on basis vector xi set Then, can be based on the projection P set Determining the region of interest in each candidate image, i.e. according to P set And determining the interest area in the nth candidate image according to the nth projection component. According to P set The positive and negative values of the nth projection component can judge the characteristics of positive correlation, irrelevance and negative correlation with the base vector xi, and further determine the interest area in the nth candidate image. Wherein the region of interest in the candidate image may be a region in the candidate image corresponding to a feature positively correlated to the basis vector.
In the embodiment of the invention, when the interest area in each candidate image is extracted, the set feature vector containing the image features of the target image and the image features of each candidate image is weighted in the channel dimension to obtain the weighted feature vector, so that the feature distinguishing information and the foreground related information of each candidate image can be enhanced; based on a statistical method, determining a basis vector of the weighted feature vector, determining the projection of the weighted feature vector on the basis vector, and determining the basis vector to perform re-projection by combining the local features of the target image and the bottom library image, so that the extraction of an interested region, namely a foreground region can be realized.
On the basis of the foregoing embodiment, the image retrieval and reordering method provided in the embodiment of the present invention is a method for weighting a set feature vector formed by image features of a target image and image features of candidate images according to a channel dimension to obtain a weighted feature vector, where the method further includes:
calculating the difference value of the initial query feature vector of the target image and the global feature vector of any candidate image, and determining the gradient value of the difference value which is reversely propagated to the target feature map of the target image and the candidate feature map of any candidate image;
determining a mean value of the gradient values in a channel dimension, and determining a weight vector for weighting the image features of the candidate images in the set feature vector in the channel dimension based on the mean value.
Specifically, before obtaining the weighted feature vector, the weight vector corresponding to each channel needs to be determined, so as to realize that the set feature vector is weighted according to the channel dimension. I.e. it is necessary to determine the weight vectors W contained in the set of weight vectors W i 。
Weight vector W corresponding to each channel in target characteristic diagram of target image q q Can be expressed as a vector with each component having a value of 1, and a weight vector W q The length of (c) is the number of channels c:
W q ＝{1，1，…，1}
weight vector W corresponding to each channel in candidate feature map of candidate image g g Can be expressed as:
in the calculation of
In the method, an initial query feature vector of the target image and a global feature vector of each candidate image can be determined according to the image features of the target image and the image features of each candidate image. The initial query feature vector refers to a global feature vector of the target image, and specifically may be obtained by pooling image features of the target image, and any candidate imageThe global feature vector may be obtained by pooling image features of the target image. Specifically, a target feature map formed by image features of the target image and candidate feature maps formed by image features of the candidate images may be input to the pooling layer of the trained convolutional neural network model, and the pooling layer outputs corresponding global feature vectors.
Then, calculating the difference value between the initial query feature vector and the global feature vector of the candidate image g, reversely propagating the difference value to the target feature map of the target image q and the candidate feature map of the candidate image g, and determining the gradient value in the process of reverse propagation. And determining the mean value of each gradient value in the channel dimension, and taking the mean value in each channel as the corresponding weight value of the channel.
On the basis, in order to ensure that the weight value is between [0,1], the obtained average value can be normalized through a Sigmoid function, that is, the weight value calculation formula is as follows:
where k denotes the kth channel and Z denotes the number of features of channel k, i.e. Z = h × w, q k -g k Is the difference between the initial query feature vector and the global feature vector of the candidate image g at the k-th channel,
representing candidate feature maps F g Image feature at position (i, j) under the top k-th channel. Here, the position (i, j) represents the candidate feature map F g The value range of i is [1,h ] at the position of the ith row and the jth column under the kth channel]And j has a value range of [1,w]。
In the embodiment of the invention, when the weight vector used for weighting the image features of each candidate image in the set feature vector in the channel dimension is calculated, the gradient mean value of the feature map reversely propagated by the difference value of the initial query feature vector and the global feature vector of each candidate image is adopted as the corresponding weight value, and when the basis vector is determined based on a statistical method, the distinctiveness of different feature components can be further improved. And moreover, the global features and the local features of the image are utilized simultaneously, the utilization rate of the features is improved, and the reordering result is more accurate.
On the basis of the foregoing embodiment, the image retrieval reordering method provided in the embodiment of the present invention determines, based on the projection, a region of interest in each candidate image, specifically including:
determining projection components corresponding to candidate images in the projection;
and if the projection component corresponding to any candidate image is larger than 0, determining that the interest area in any candidate image is the projection area of the projection component corresponding to any candidate image.
Specifically, when determining the region of interest in each candidate image, the embodiment of the present invention may directly determine by determining the positive and negative of the projection component corresponding to each candidate image. First, a projection component, i.e., P, corresponding to each candidate image in the projection is determined n . Then, the positive and negative of each projection component are determined. For P n If P is n >0, then consider the nth projection component to be positively correlated with the base vector, i.e., P n The corresponding projection area is the region of interest. Note that, P is due to n Is a two-dimensional matrix, thus P n >0 means that each element in the matrix is greater than 0, i.e., the matrix is a positive definite matrix. Otherwise, if P n If not all elements in the image are greater than 0 but 0 or less than 0 exists, the candidate image corresponding to the nth projection component is considered as a noise image. Therefore, the noise image can be detected from each candidate image according to the method provided in the embodiment of the present invention. On the basis, P can be further limited n When the number of elements specified in (b) is less than 0, the candidate image corresponding to the nth projection component is considered as a noise image, and the detection standard of the noise image can be reduced.
In the embodiment of the invention, the interesting areas in the candidate images are determined by judging the positive and negative of the projection components, so that the influence of background noise on the reordering result can be further reduced.
On the basis of the foregoing embodiment, the image retrieval reordering method provided in the embodiment of the present invention determines the query feature vector of the target image based on the correlation between the image feature of the target image and the foreground feature of each candidate image, and specifically includes:
determining comprehensive relevance response characteristics of the foreground characteristics corresponding to the interest areas in the candidate images to the image characteristics of the target image based on the relevance of the foreground characteristics corresponding to the interest areas in the candidate images to the image characteristics of the target image;
based on an activation function, determining an activation response feature of the target image, and based on the comprehensive correlation response feature and the activation response feature, determining a query feature vector of the target image.
Specifically, in this embodiment of the present invention, the correlation between the foreground feature corresponding to the region of interest in any candidate image and each image feature of the target image may be a correlation between a foreground feature vector corresponding to the region of interest in any candidate image and a local feature vector at each position in the target feature map of the target image. For the candidate image g, the correlation between the foreground feature vector corresponding to the region of interest in the candidate image g and the local feature vector at each position in the target feature map Fq of the target image q can be represented by calculating the inner product between the two feature vectors. Namely, the method comprises the following steps:
wherein, F pool,g Representing the corresponding foreground feature vector of the region of interest in the candidate image g, F q,ij Target feature map F representing target feature map q q Local feature vector at intermediate position (i, j), c gij Is shown as F pool,g And F q,ij The inner product between the two is used for characterizing the correlation between the two, namely for characterizing the correlation between the base vector xi and the candidate feature map Fg of the candidate image g.
Traversal targetFeature map F q Repeatedly executing the above process at each position (i, j) to obtain a correlation graph c of the foreground feature vector corresponding to the region of interest in the candidate image g and the target feature graph Fq g ，c g Is a two-dimensional matrix with the shape of (h, w), and the element in the two-dimensional matrix is c gij The number of elements in the two-dimensional matrix is the number of channels c.
According to the correlation c between the foreground characteristics corresponding to the interest areas in the candidate images and each image characteristic of the target image gij Determining comprehensive correlation response characteristic C of foreground characteristic corresponding to the interest area in each candidate image to the image characteristic of the target image g . Integrated correlation response characteristic C q The method is used for representing the overall correlation between the foreground features corresponding to the interest areas in all candidate images and the image features of the target image, and specifically, the method can be obtained by fusing the correlation graphs corresponding to all candidate images, namely:
finally, for the target feature map Fq of the target image q, the activation response of each position on the target feature map Fq can be obtained through an activation function, and then the activation responses are summed in the channel dimension to obtain a two-dimensional activation response map Fa, where the shape of the activation response map Fa is (h, w), the activation response map Fa is a set formed by activation response features, and the number of elements in the activation response map Fa is the channel number c. Response characteristics C according to comprehensive correlation q And activating the response graph Fa, a query feature vector of the target image can be determined, which can be understood as being based on C q And Fa the process of re-determining the query feature vector.
In the embodiment of the invention, the foreground characteristics of the candidate images are determined after the interesting regions are extracted, the corresponding correlations of different candidate images are fused according to the correlation between the foreground characteristics of each candidate image and the image characteristics of the target image, the re-determination of the query characteristic vector is realized by combining the activation response characteristics of the target image, the target example region characteristics contained in the target image are reserved as far as possible, the foreground characteristics of the candidate images can provide positive feedback help for the re-determination of the query characteristic vector, the re-determined query characteristic vector is more accurate, the final re-ordering result is more accurate, and the re-ordering precision is higher.
On the basis of the foregoing embodiment, the image retrieval reordering method provided in the embodiment of the present invention, where the determining a query feature vector of the target image based on the comprehensive relevance response feature and the activation response feature specifically includes:
weighting a target feature map of the target image using a sum of the integrated relevance response feature and the activation response feature as a weight;
and determining a query feature vector of the target image based on the weighted target feature map.
Specifically, in the embodiment of the present invention, when determining the query feature vector of the target image, the sum of the comprehensive relevance response feature and the activation response feature may be used as a weight to weight the target feature map of the target image, that is, the comprehensive relevance response feature C is first weighted q And summing with the activation response graph Fa, and weighting the target feature graph of the target image by taking the summation result as the weight of the target feature graph of the target image. Sequentially pooling and normalizing the weighted target characteristic graph to obtain a query characteristic vector F of the target image qw . Namely, the method comprises the following steps:
F qw ＝norm(pool(F q *(F a +C q )))
wherein norm is normalized, pool is pooled, F q Is a target feature map.
On the basis of the foregoing embodiment, the image retrieval reordering method provided in the embodiment of the present invention specifically includes:
pooling image features in the region of interest in any candidate image to obtain foreground features corresponding to the region of interest in any candidate image.
Specifically, the determination of the foreground features may be implemented by pooling, and the image features in the region of interest in any candidate image may be input to a pooling layer of the trained convolutional neural network model, and the foreground features corresponding to the region of interest in any candidate image may be output by the pooling layer.
For example, all image features in the region of interest in the candidate image g are denoted as D region,g The shape is (s, c), s represents the set feature vector D set Image feature D of the middle candidate image g set，g Corresponds to P g Area of positive region, D region，g Obtaining foreground characteristics F corresponding to the interest area in the candidate image g through pooling pool，g ∈R c 。
On the basis of the foregoing embodiment, the image retrieval reordering method provided in the embodiment of the present invention extracts a region of interest in each candidate image based on the image features of the target image and the image features of each candidate image, and before the method, the method further includes:
and determining each candidate image in the image base library based on the correlation between the initial query feature vector of the target image and the global image feature vector of each base library image in the image base library.
Specifically, in the embodiment of the present invention, before extracting an interest region in each candidate image, each candidate image needs to be determined, and the determining method includes: firstly, determining an initial query feature vector of a target image, where the initial query feature vector is a global feature vector of the target image, and both the initial query feature vector of the target image and the global image feature vector of each image in an image base can be obtained by pooling, which is specifically referred to the above embodiment, but is not specifically limited in the embodiment of the present invention.
According to the relevance ranking between the initial query feature vector of the target image and the global image feature vector of each base image in the image base, each candidate image in the image base, namely the first N base images in the relevance ranking result, can be determined. In the embodiment of the present invention, the image content with high default feature correlation is more similar.
The embodiment of the invention provides a determination method of each candidate image, so that the reordering scheme of each candidate image can be smoothly carried out.
Fig. 2 is a schematic view of a complete flow of the image retrieving and reordering method according to the embodiment of the present invention. As shown in fig. 2, the method includes:
acquiring a target image q and an image base containing a plurality of base images;
respectively inputting a target image q and images in each image base to the trained convolutional neural network model to respectively obtain a target characteristic diagram and an initial query characteristic vector of the target image q, the first N candidate images in the image base obtained by preliminary sequencing, a candidate characteristic diagram and a global characteristic vector of each candidate image g;
calculating a weight vector set W based on the initial query feature vector and the global feature vector of each candidate image;
weighting the target characteristic diagram of the target image q and the candidate characteristic diagram of each candidate image g in channel dimensions based on the weight vector set W to obtain weighted characteristic vectors;
determining a basis vector of the weighted feature vector through a statistical method, and determining the projection of the weighted feature vector on the basis vector;
each candidate image is subjected to characteristic response, the characteristic region with positive response corresponds to the interested region in each candidate image, and the foreground characteristic of the interested region is determined through pooling operation;
calculating the correlation between the image characteristics of the target image and the foreground characteristics corresponding to the interest areas in the candidate images, determining the comprehensive correlation response characteristics of the foreground characteristics corresponding to the interest areas in the candidate images to the image characteristics of the target image based on the correlation, and determining the weight of the target characteristic image by combining the activation response characteristics of the target image;
weighting, pooling and normalizing the target feature map to obtain a query feature vector of the target image;
and reordering the candidate images according to the query feature vector.
In summary, the image retrieval reordering method provided in the embodiment of the present invention is actually a two-stage feature optimization method, that is, after the target image and the candidate image are weighted and feature transformed by the determined basis vector, a feature region with high correlation between the candidate image and the target image is calculated by projection, so as to achieve target instance positioning of the candidate image. And pooling an interest area in the candidate image to obtain a foreground characteristic, performing correlation calculation with the target image to obtain a comprehensive correlation response characteristic, fusing an activation response characteristic of the target image, keeping a target example area characteristic of the target image as much as possible, and improving reordering precision. And moreover, global features and local features of the image are utilized simultaneously, extraction of the region of interest and determination of foreground features of the region of interest are realized in a mode of extracting basis vectors by combining the local features of the target image and the candidate image for projection, before the basis vectors are determined by a statistical method, the weights of feature channels are calculated by utilizing return gradients through the global feature correlation of the target image and the candidate image, and the distinctiveness of different feature components is further improved. Compared with the existing method for searching based on the global features, the method for searching based on the global features directly processes and fuses the global features, and in the embodiment of the invention, based on the assumption that the target image and the Top-N base image, namely the candidate image have certain correlation information or common prospect, the method provides the joint processing of the local features of the target image and the candidate image, calculates the region of interest through the projection of the basis vector, and reduces the influence caused by background noise.
As shown in fig. 3, on the basis of the above embodiment, an embodiment of the present invention provides an image retrieval reordering apparatus, including: a foreground feature determination module 31, a query feature vector determination module 32 and a reordering module 33. Wherein the content of the first and second substances,
the foreground characteristic determining module 31 is configured to extract an interest region in each candidate image based on the image characteristics of the target image and the image characteristics of each candidate image, and determine a foreground characteristic corresponding to the interest region in each candidate image;
the query feature vector determination module 32 is configured to determine a query feature vector of the target image based on a correlation between the image feature of the target image and a foreground feature corresponding to the region of interest in each candidate image;
the reordering module 33 is configured to reorder the candidate images based on the query feature vector.
On the basis of the foregoing embodiment, in the image retrieval and reordering device provided in the embodiment of the present invention, the foreground characteristic determining module is specifically configured to:
weighting a set feature vector containing the image features of the target image and the image features of each candidate image in a channel dimension to obtain a weighted feature vector;
determining a basis vector of the weighted feature vector based on a statistical method, and determining a projection of the weighted feature vector on the basis vector;
based on the projections, regions of interest in the candidate images are determined.
On the basis of the foregoing embodiment, the image retrieval reordering apparatus provided in the embodiment of the present invention further includes a weight vector determination module, configured to:
calculating the difference value of the initial query feature vector of the target image and the global feature vector of any candidate image, and determining the gradient value of the difference value which is reversely propagated to the target feature map of the target image and the candidate feature map of any candidate image;
determining a mean of the gradient values in a channel dimension, and determining a weight vector for weighting image features of candidate images in the set feature vector in the channel dimension based on the mean.
On the basis of the foregoing embodiment, in the image retrieval and reordering device provided in the embodiment of the present invention, the foreground characteristic determining module is specifically configured to:
determining projection components corresponding to candidate images in the projection;
and if the projection component corresponding to any candidate image is larger than 0, determining that the interest area in any candidate image is the projection area of the projection component corresponding to any candidate image.
On the basis of the foregoing embodiment, in the image retrieval and reordering device provided in the embodiment of the present invention, the query feature vector determination module is specifically configured to:
determining comprehensive relevance response characteristics of the foreground characteristics corresponding to the interest areas in the candidate images to the image characteristics of the target image based on the relevance of the foreground characteristics corresponding to the interest areas in the candidate images to the image characteristics of the target image;
and determining an activation response characteristic of the target image based on an activation function, and determining a query feature vector of the target image based on the comprehensive relevance response characteristic and the activation response characteristic.
On the basis of the foregoing embodiment, in the image retrieval and reordering device provided in the embodiment of the present invention, the query feature vector determination module is specifically configured to:
weighting a target feature map of the target image by taking the sum of the comprehensive correlation response feature and the activation response feature as a weight;
and determining a query feature vector of the target image based on the weighted target feature map.
On the basis of the foregoing embodiment, the image retrieval reordering device provided in the embodiment of the present invention further includes a preliminary sorting module, configured to:
and determining each candidate image in the image base library based on the correlation between the initial query feature vector of the target image and the global image feature vector of each base library image in the image base library.
Specifically, the functions of the modules in the image retrieval and reordering device provided in the embodiment of the present invention correspond to the operation flows of the steps in the embodiments of the methods one to one, and the implementation effects are also consistent, for which specific reference is made to the embodiments above, which are not described again in the embodiments of the present invention.
Fig. 4 illustrates a physical structure diagram of an electronic device, which may include, as shown in fig. 4: a processor (processor) 410, a communication Interface (Communications Interface) 420, a memory (memory) 430 and a communication bus 440, wherein the processor 410, the communication Interface 420 and the memory 430 are in communication with each other via the communication bus 440. The processor 410 may call logic instructions in the memory 430 to perform the image retrieval reordering method in the above embodiments, the method comprising: extracting interest areas in the candidate images based on the image features of the target image and the image features of the candidate images, and determining foreground features corresponding to the interest areas in the candidate images; determining a query feature vector of the target image based on the correlation between the image features of the target image and foreground features corresponding to the interest areas in the candidate images; and reordering the candidate images based on the query feature vector.
In addition, the logic instructions in the memory 430 may be implemented in the form of software functional units and stored in a computer readable storage medium when the software functional units are sold or used as independent products. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk, and various media capable of storing program codes.
In another aspect, the present invention also provides a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions, which when executed by a computer, the computer is capable of executing the image retrieval reordering method in the above embodiments, the method comprising: extracting interest areas in the candidate images based on the image characteristics of the target image and the image characteristics of the candidate images, and determining foreground characteristics corresponding to the interest areas in the candidate images; determining a query feature vector of the target image based on the correlation between the image features of the target image and foreground features corresponding to the interest areas in the candidate images; and reordering the candidate images based on the query feature vector.
In yet another aspect, the present invention further provides a non-transitory computer-readable storage medium, on which a computer program is stored, the computer program being implemented by a processor to execute the image retrieval reordering method in the above embodiments, the method including: extracting interest areas in the candidate images based on the image features of the target image and the image features of the candidate images, and determining foreground features corresponding to the interest areas in the candidate images; determining a query feature vector of the target image based on the correlation between the image features of the target image and foreground features corresponding to the interest areas in the candidate images; and reordering the candidate images based on the query feature vector.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. An image retrieval reordering method, comprising:
extracting interest areas in the candidate images based on the image features of the target images and the image features of the candidate images through the trained convolutional neural network model, and determining foreground features corresponding to the interest areas in the candidate images;
determining a query feature vector of the target image based on the correlation between the image features of the target image and foreground features corresponding to the interest areas in the candidate images; the determination process of the query feature vector comprises the following steps: determining the weight of a target feature map formed by the image features of the target image according to the correlation between the image features of the target image and the foreground features corresponding to the interest areas in the candidate images, then weighting the target feature map according to the weight of the target feature map, and pooling to obtain the query feature vector;
and reordering the candidate images based on the query feature vector.
2. The image retrieval reordering method according to claim 1, wherein the extracting a region of interest in each candidate image based on the image features of the target image and the image features of each candidate image specifically comprises:
weighting a set characteristic vector containing the image characteristics of the target image and the image characteristics of each candidate image in a channel dimension to obtain a weighted characteristic vector;
determining a basis vector of the weighted feature vector based on a statistical method, and determining a projection of the weighted feature vector on the basis vector;
based on the projections, regions of interest in the candidate images are determined.
3. The method of claim 2, wherein the weighting the feature vector of the set including the image feature of the target image and the image feature of each candidate image in the channel dimension to obtain the weighted feature vector further comprises:
calculating the difference value of the initial query feature vector of the target image and the global feature vector of any candidate image, and determining the gradient value of the difference value which is reversely propagated to the target feature map of the target image and the candidate feature map of any candidate image;
determining a mean value of the gradient values in a channel dimension, and determining a weight vector for weighting the image features of the candidate images in the set feature vector in the channel dimension based on the mean value.
4. The method for reordering image retrieval according to claim 2, wherein the determining the region of interest in each candidate image based on the projection comprises:
determining projection components corresponding to candidate images in the projection;
and if the projection component corresponding to any candidate image is larger than 0, determining that the interest area in any candidate image is the projection area of the projection component corresponding to any candidate image.
5. The method of reordering image retrieval according to claim 1, wherein the determining the query feature vector of the target image based on the correlation between the image feature of the target image and the foreground feature corresponding to the region of interest in each candidate image specifically comprises:
determining comprehensive relevance response characteristics of the foreground characteristics corresponding to the interest areas in the candidate images to the image characteristics of the target image based on the relevance of the foreground characteristics corresponding to the interest areas in the candidate images and each image characteristic of the target image;
and determining an activation response characteristic of the target image based on an activation function, and determining a query feature vector of the target image based on the comprehensive relevance response characteristic and the activation response characteristic.
6. The method for reordering image retrieval according to claim 5, wherein the determining the query feature vector of the target image based on the comprehensive relevance response feature and the activation response feature comprises:
weighting a target feature map of the target image by taking the sum of the comprehensive correlation response feature and the activation response feature as a weight;
and determining a query feature vector of the target image based on the weighted target feature map.
7. The image retrieval and reordering method of any one of claims 1 to 6, wherein the extracting a region of interest in each candidate image based on the image features of the target image and the image features of each candidate image further comprises:
and determining each candidate image in the image base library based on the correlation between the initial query feature vector of the target image and the global image feature vector of each base library image in the image base library.
8. An image retrieval reordering apparatus comprising:
the foreground characteristic determining module is used for extracting interest areas in the candidate images based on the image characteristics of the target images and the image characteristics of the candidate images through the trained convolutional neural network model, and determining foreground characteristics corresponding to the interest areas in the candidate images;
the query feature vector determining module is used for determining a query feature vector of the target image based on the correlation between the image feature of the target image and the foreground feature corresponding to the interest region in each candidate image; the determination process of the query feature vector comprises the following steps: determining the weight of a target feature map formed by the image features of the target image according to the correlation between the image features of the target image and the foreground features corresponding to the interest areas in the candidate images, then weighting the target feature map according to the weight of the target feature map, and pooling to obtain the query feature vector;
and the reordering module is used for reordering the candidate images based on the query feature vector.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the image retrieval reordering method according to any one of claims 1 to 7 when executing the program.
10. A non-transitory computer readable storage medium, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the steps of the image retrieval reordering method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011535678.1A CN112650869B (en) | 2020-12-23 | 2020-12-23 | Image retrieval reordering method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011535678.1A CN112650869B (en) | 2020-12-23 | 2020-12-23 | Image retrieval reordering method and device, electronic equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112650869A CN112650869A (en) | 2021-04-13 |
| CN112650869B true CN112650869B (en) | 2022-12-06 |
Family
ID=75359317
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011535678.1A Active CN112650869B (en) | 2020-12-23 | 2020-12-23 | Image retrieval reordering method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112650869B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114491131B (en) * | 2022-01-24 | 2023-04-18 | 北京至简墨奇科技有限公司 | Method and device for reordering candidate images and electronic equipment |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| FR2802003A1 (en) * | 1999-12-03 | 2001-06-08 | Canon Kk | Image searching method from database in computer, involves comparing two data items indicating region of interest in new image and representing visual content characteristic of example image |
| CN102750347A (en) * | 2012-06-08 | 2012-10-24 | 天津大学 | Method for reordering image or video search |
| CN107273836A (en) * | 2017-06-07 | 2017-10-20 | 深圳市深网视界科技有限公司 | A kind of pedestrian detection recognition methods, device, model and medium |
| CN107506429A (en) * | 2017-08-22 | 2017-12-22 | 北京联合大学 | A kind of image rearrangement sequence method integrated based on marking area and similitude |
| CN107679250A (en) * | 2017-11-01 | 2018-02-09 | 浙江工业大学 | A kind of multitask layered image search method based on depth own coding convolutional neural networks |
| CN109033172A (en) * | 2018-06-21 | 2018-12-18 | 西安理工大学 | A kind of image search method of deep learning and approximate target positioning |
| CN109934258A (en) * | 2019-01-30 | 2019-06-25 | 西安理工大学 | The image search method of characteristic weighing and Regional Integration |
| US10824942B1 (en) * | 2017-04-10 | 2020-11-03 | A9.Com, Inc. | Visual similarity and attribute manipulation using deep neural networks |
-
2020
- 2020-12-23 CN CN202011535678.1A patent/CN112650869B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| FR2802003A1 (en) * | 1999-12-03 | 2001-06-08 | Canon Kk | Image searching method from database in computer, involves comparing two data items indicating region of interest in new image and representing visual content characteristic of example image |
| CN102750347A (en) * | 2012-06-08 | 2012-10-24 | 天津大学 | Method for reordering image or video search |
| US10824942B1 (en) * | 2017-04-10 | 2020-11-03 | A9.Com, Inc. | Visual similarity and attribute manipulation using deep neural networks |
| CN107273836A (en) * | 2017-06-07 | 2017-10-20 | 深圳市深网视界科技有限公司 | A kind of pedestrian detection recognition methods, device, model and medium |
| CN107506429A (en) * | 2017-08-22 | 2017-12-22 | 北京联合大学 | A kind of image rearrangement sequence method integrated based on marking area and similitude |
| CN107679250A (en) * | 2017-11-01 | 2018-02-09 | 浙江工业大学 | A kind of multitask layered image search method based on depth own coding convolutional neural networks |
| CN109033172A (en) * | 2018-06-21 | 2018-12-18 | 西安理工大学 | A kind of image search method of deep learning and approximate target positioning |
| CN109934258A (en) * | 2019-01-30 | 2019-06-25 | 西安理工大学 | The image search method of characteristic weighing and Regional Integration |
Non-Patent Citations (5)
| Title |
|---|
| Adaptive Query Re-ranking Based on ImageGraph for Image Retrieval;Haonan Fan etc.;《2018 IEEE International Conference on Big Data (Big Data)》;20190124;第4593-4599页 * |
| Discriminative Multi-View Privileged Information Learning for Image Re-Ranking;Jun Li etc.;《 IEEE Transactions on Image Processing》;20200109;3490-3505页 * |
| 一种基于前景与背景划分的区域图像检索方法及实现;朱远清 等;《第十三届全国图象图形学学术会议论文集》;20061130;第570-574页 * |
| 基于卷积神经网络的实例图像检索;刘易斯;《中国优秀硕士学位论文全文数据库（信息科技辑）》;20200215;第I138-1503页 * |
| 基于特征性能增强和目标定位的图像检索研究;袁晖;《中国优秀硕士学位论文全文数据库（信息科技辑）》;20190815;I138-712页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112650869A (en) | 2021-04-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110162593B (en) | Search result processing and similarity model training method and device | |
| CN111538908B (en) | Search ranking method and device, computer equipment and storage medium | |
| CN106776673B (en) | Multimedia document summarization | |
| CN111581510A (en) | Shared content processing method and device, computer equipment and storage medium | |
| CN110929038B (en) | Knowledge graph-based entity linking method, device, equipment and storage medium | |
| JP2020123318A (en) | Method, apparatus, electronic device, computer-readable storage medium, and computer program for determining text relevance | |
| US10482146B2 (en) | Systems and methods for automatic customization of content filtering | |
| CN110674407A (en) | Hybrid recommendation method based on graph convolution neural network | |
| WO2022140900A1 (en) | Method and apparatus for constructing personal knowledge graph, and related device | |
| CN107291825A (en) | With the search method and system of money commodity in a kind of video | |
| CN111400615B (en) | Resource recommendation method, device, equipment and storage medium | |
| Yu et al. | A content-based goods image recommendation system | |
| CN114332680A (en) | Image processing method, video searching method, image processing device, video searching device, computer equipment and storage medium | |
| CN110059156A (en) | Coordinate retrieval method, apparatus, equipment and readable storage medium storing program for executing based on conjunctive word | |
| CN114186013A (en) | Entity recognition model hot updating method and device, equipment, medium and product thereof | |
| CN113722583A (en) | Recommendation method, recommendation model training method and related products | |
| Lee et al. | Efficient image retrieval using advanced SURF and DCD on mobile platform | |
| CN111507285A (en) | Face attribute recognition method and device, computer equipment and storage medium | |
| CN114358109A (en) | Feature extraction model training method, feature extraction model training device, sample retrieval method, sample retrieval device and computer equipment | |
| CN112256899A (en) | Image reordering method, related device and computer readable storage medium | |
| Su et al. | Hybrid recommender system based on deep learning model | |
| CN112650869B (en) | Image retrieval reordering method and device, electronic equipment and storage medium | |
| CN113449200B (en) | Article recommendation method and device and computer storage medium | |
| CN113076475B (en) | Information recommendation method, model training method and related equipment | |
| CN114996490A (en) | Movie recommendation method, system, storage medium and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |