CN110059198A - A kind of discrete Hash search method across modal data kept based on similitude - Google Patents
A kind of discrete Hash search method across modal data kept based on similitude Download PDFInfo
- Publication number
- CN110059198A CN110059198A CN201910277146.3A CN201910277146A CN110059198A CN 110059198 A CN110059198 A CN 110059198A CN 201910277146 A CN201910277146 A CN 201910277146A CN 110059198 A CN110059198 A CN 110059198A
- Authority
- CN
- China
- Prior art keywords
- sample
- matrix
- mode
- indicate
- hash codes
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of discrete Hash search methods across modal data kept based on similitude.The cross-module state retrieval data set being made of the sample comprising two mode is established, training set and test set are divided into；The objective function for establishing similitude in similitude and mode between keeping mode, solve to objective function and obtains Hash codes matrix by a kind of discrete optimizing method；Learn the hash function of each mode according to Hash codes matrix；The Hash codes of all samples in training set and test set are calculated using hash function；One mould measurement integrates as query set, another mode training set is retrieved set, calculates in query set that Hamming distance, sequence are used as search result between the Hash codes of sample and the Hash codes of sample in retrieved set.The present invention can effectively keep the similitude in similitude and mode between mode, and consider the discrete feature of Hash codes, be solved using a kind of method of discrete optimization to objective function, to improve the accuracy of cross-module state retrieval.
Description
Technical field
The present invention relates to a kind of a kind of cross-module state search methods in multimedia search technology field, more particularly, to one kind
The discrete Hash search method across modal data kept based on similitude.
Background technique
With the fast development of Internet information technique, the multimedia messages of various mode are in explosive growth on network.
Development trend is complied with, the retrieval of cross-module state becomes a most important problem, attracted the attention of many researchers.Across
Mode retrieval typical scene is exactly the query sample of a given mode, retrieves other similar mode.But due to
The presence of isomery wide gap can not directly measure the similitude between different modalities.Further, since the explosive increase of data, concern
The carrying cost and efficiency retrieved on a large scale are necessary.Hash method is that popular method, target exist in recent years
In data to be mapped as to compact binary code.By Hash, lower memory space can be used to save data, and pass through
Hamming distance measures the similitude between different modalities, and Hamming distance can quickly be counted by the xor operation of bit
It calculates.
In recent years, many cross-module state hash methods have been proposed in researcher.Most of cross-module state hash method it is main
Thinking is to learn hash function using training data, by the Feature Mapping in luv space to a public Hamming space,
And hash function should keep the semantic dependency in original feature space.Next it simply introduces some typical and relatively new
Across Modal Method.CVH is expanded by the spectrum Hash of single mode, by the distance minimization of Weight.IMH is by keeping mode
Between and mode in consistency learn linear hash function.CMFH uses Harmonious Matrix Factorization, to the difference of a sample
Mode learns unified Hash codes.SMFH is to be based on confederate matrix Factorization, while keeping local Geometrical consistency and mark
Consistency is signed to learn unified Hash codes.By keeping mode to learn hash function based on the mixing similitude of figure.
Similitude holding is one extremely important problem of cross-module state hash method.It is similar between most of method concern mode
Property, that is to say, that if an image pattern and a samples of text are semantically being mutually related, they should have
Similar Hash codes.In addition, similitude is also critically important in mode.Similitude is intended to keep the local geometric knot of each mode in mode
Structure.Certain methods keep similitude in mode using figure Laplce's regular terms, however only focus in belonging within k neighbour
Sample, the weight of the sample in relational matrix except k neighbour are arranged to 0, such as SMFH.In this way, in original feature space
Similar sample will obtain similar Hash codes, but the Hash codes of dissimilar sample are not necessarily dissimilar because they not by
To limitation.In addition, Hash codes are binary codes, study binary code is a discrete optimization problems of device, which is usually NP tired
Difficult problem.The strategy that most of existing cross-module state hash methods use is to loosen original discrete constraint for continuous constraint,
Then the successive value of acquisition is quantified as binary code by re-optimization objective function.However, this strategy that loosens will affect accessibility
Energy.
Summary of the invention
In view of the above-mentioned deficiencies in the prior art, it is an object of the present invention to provide it is a kind of based on similitude keep across modal data
Discrete Hash search method.
The technical solution adopted by the present invention the following steps are included:
1) the cross-module state retrieval data set being made of the sample comprising two mode, sample are established in the database of server
This two mode are respectively image modalities and text modality, and data set is divided into training set and test set；
2) objective function of similitude in similitude and mode between keeping mode is established, and passes through a kind of discrete optimizing method
Objective function is solved, Hash codes matrix is obtained；
3) the Hash codes matrix succeeded in school according to step 2), learns the hash function of each mode；
4) Hash codes of all samples in training set and test set are calculated using hash function；
5) using the test set of a mode as query set, using the training set of another mode as retrieved set, according to step
Rapid 4) mode obtains Hash codes, calculates the Hamming distance in query set in the Hash codes of sample and retrieved set between the Hash codes of sample
From being ranked up according to the sequence of Hamming distance from small to large to sample in retrieved set, the forward sample that sorts will be by as inspection
Hitch fruit.
The step 1) specifically:
Image and text are collected from webpage, and the identical piece image of corresponding meaning and a text are constituted into an image
Text pair, meaning identical finger describes same thing, such as the image and a text for describing people's surfing of width people's surfing
With regard to constituting an image text pair；To retrieve data set, an image text pair to building cross-module state by each image text
Characteristics of image and text feature constitute a sample；The training set of cross-module state retrieval data set has n sample, each sample packet
Feature containing two mode of image and text, X(1)Indicate the image modalities matrix that the feature of n image modalities is constituted,Each arrange represents the feature of the image modalities of a sample,Indicate the figure of p-th of sample
As the feature of mode, the as pth of image modalities matrix is arranged,Wherein d1Indicate the dimension of the feature of image modalities
Degree, R indicate set of real numbers；X(2)Indicate the text modality matrix that the feature of n text modality is constituted,Each arrange represents the feature of the text modality of a sample,Indicate p-th of sample
The feature of text modality, as the pth column of text modality matrix,Wherein d2Indicate the dimension of the feature of text modality
Degree；By the corresponding feature of two modeWithConstitute sample characteristics；
Y={ y1,y2,…,ynIndicate label matrix, Y ∈ { 0,1 }c×n, wherein c indicates classification sum, ypIndicate pth
The label vector of sample, i.e. the pth column of label matrix, yp={ y1p,y2p,…,yip,…,ynp, yipIndicate that pth sample exists
The label of i-th class classification；If p-th of sample belongs to the i-th class, the element y of the i-th row pth column in label matrix Yip=1,
Otherwise yip=0.
Data set is divided into training set and test set by the present invention, is extracted feature respectively to image and text, is wrapped in training set
Containing the characteristics of image and text feature for being used as training, include characteristics of image and text feature as test in test set.
The step 2) specifically includes:
2.1) for two kinds of different modalities of same sample, identical Hash codes are arrived in study, are able to maintain similar between mode
Property.Similarity matrix S is first constructed according to label matrix Y cosine similarity, the element of pth row q column is S in Spq=yp·
yq/(||yp||2||yq||2), wherein p and q is the ordinal number of sample, yp·yqIndicate the label vector y of p-th of samplepWith q
The label vector y of a sampleqBetween inner product, | | yp||2With | | yq||2Respectively indicate the label vector y of p-th of samplepWith q
The label vector y of a sampleqTwo norms；
Then, the loss function of similitude between keeping mode below is established:WhereinIt is F
Square of norm, B indicate the Hash codes matrix that the Hash codes of all samples are constituted, B ∈ { -1,1 }k×n, wherein k is Hash codes
Length；
2.2) it is directed to a mode, it is desirable to the local geometry that sample can be kept, i.e., in original feature space
In similar sample, it is desirable to after being mapped to Hamming space, their Hash codes are also similar.Mould is kept by using figure regular terms
Similitude in state establishes following holding mould for m-th of mode (m=1 indicates image modalities, and m=2 indicates text modality)
The loss function of similitude in state:
Wherein, bpAnd bqIt is the pth column and q column of Hash codes matrix B, W respectively(m)It is the weight matrix of m-th of mode,It is weight matrix W(m)Pth row q column element, LmIt is the Laplacian Matrix of m-th of mode, D(m)It is m-th of mould
The diagonal matrix of state,Indicate diagonal matrix D(m)Pth row q column element,Lm=D(m)-W(m)；tr
The mark of () representing matrix,Indicate square of 2 norms；
This method not only considers the sample close apart from a certain sample, it is also considered that the sample far apart from a certain sample, on
State weight matrix W(m)In elementSpecifically, more different Hash codes can be obtained in this way:
Wherein, e is natural constant,Indicate in m-th of mode with sample characteristicsApart from nearest k1A sample
The set that feature is constituted,Indicate in m-th of mode with sample characteristicsApart from farthest k2A sample characteristics are constituted
Set, μ be tradeoff parameter, the value of σ takes maximum
GatheringIn, with sample characteristicsDistance is closer, and weight is arranged bigger；GatheringIn,
With sample characteristicsDistance is remoter, and the absolute value of weight is arranged bigger.Weight is set according to above-mentioned formula, can both make phase
As Hash codes distance after sample mapping it is close, and the Hash codes distance after dissimilar sample mapping can be made remote.
2.3) loss function of similitude between keeping mode is combinedWith the loss function for keeping similitude in modeEstablish the overall goal function of following study Hash codes are as follows:
s.t.B∈{-1,1}k×n
Wherein, α indicate keep mode between similitude loss function tradeoff parameter, β1Indicate the mould of holding image modalities
The tradeoff parameter of the loss function of similitude, β in state2Indicate the power of the loss function of similitude in the mode of holding text modality
Weigh parameter, T representing matrix transposition；
2.4) due to the presence of Hash codes discrete constraint, solution procedure 2.1) objective function be a np problem, use
A kind of discrete optimizing method carries out solving overall goal function, specifically:
2.4.1) random initializtion Hash codes matrix B(0)∈{-1,1}k×n, B(0)Indicate initial Hash codes matrix B；Breathe out
The uncommon initial random generation of code matrix B, element therein are selected as -1 or 1.
2.4.2 solution) is iterated using following procedure:
First seek overall goal functionGradient:
Then iterative processing, the discrete Hash codes matrix B obtained using following formula according to iteration j(j)It handles
To the Hash codes matrix B of (j+1) secondary iteration(j+1):
Wherein, λ is learning rate；B(j)Indicate the Hash codes matrix that iteration j obtains, B(j+1)Indicate that (j+1) is secondary repeatedly
The Hash codes matrix B that generation obtains(j+1)；
Hash codes matrix is updated according to above-mentioned iterative formula and completes optimization process, obtains optimal Hash codes matrix B.
In the step 3), hash function uses simple Linear Mapping h1(x(1))=sign (P1 Tx(1)),Learning hash function is to learn two mapping matrix P1And P2, wherein P1Indicate image modalities
Mapping matrix, P2Indicate the mapping matrix of text modality, x(1)Indicate the feature of image modalities in sample, x(2)Indicate sample Chinese
The feature of this mode；
It solves following formula and obtains mapping matrix P1And P2:
Wherein,Indicate that the loss function of study mapping matrix, γ are tradeoff parameters；
Pass through orderIt solves to calculate and obtains P1: P1=(X(1)X(1)T+γI)-1X(1)BT, I expression unit matrix；
Pass through orderIt solves to calculate and obtains P2: P2=(X(2)X(2)T+γI)-1X(2)BT。
In the step 4), the Hash codes formula h of training set and test set image modalities1(x(1))=sign (P1 Tx(1))
It calculates, wherein x(1)Indicate the feature of the image modalities of sample, h1(x(1)) indicate by the sample image mode feature x(1)Meter
The Hash codes of calculating；The Hash codes formula of training set and test set text modalityIt calculates, wherein x(2)Indicate the feature of sample text mode, h2(x(2)) indicate by the sample text mode feature x(2)Calculated Hash codes.
Present invention reserved mapping matrix P after step 2) and step 3) training1,P2And it abandons step 2) and succeeds in school
Training set Hash codes.
The beneficial effects of the present invention are:
Similitude in similitude and mode between the mode that the present invention can effectively be kept simultaneously, and it is not concerned only with similar sample
This Hash codes, have also paid close attention to the Hash codes of dissimilar sample, the method for the present invention makes sample dissimilar in original feature space
This obtains dissimilar Hash codes after being mapped to Hamming space, and similar sample also obtains similar Hash codes, and Hash codes are more
Having any different property solves the problems, such as to learn the data retrieval of discrete Hash codes.
The present invention considers the discrete feature of Hash codes, is asked using a kind of method of discrete optimization objective function
Solution, to improve the accuracy of cross-module state data retrieval.
Detailed description of the invention
Fig. 1 is the implementation steps of the invention flow chart.
Fig. 2 is an example schematic of the image retrieval text on cross-module state data set Wiki.
Fig. 3 is an example schematic of the text retrieval image on cross-module state data set Wiki.
Specific embodiment
The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
As shown in Figure 1, specific embodiments of the present invention situation is as follows:
Specific implementation is to be described further with flow chart and cross-module state data set Wiki to technical solution of the present invention；Its
In, cross-module state data set Wiki derives from wikipedia, includes 2866 images collected from wikipedia article
Text pair.Image with 128 dimension SIFT feature indicate, text with 10 tie up LDA character representation.The image text of Wiki data set
This is to being divided into 10 semantic classes, each image text is to belonging to one kind therein.It randomly chooses 2173 samples and constitutes instruction
Practice collection, remaining 693 sample constitutes test set.
1) the cross-module state retrieval data set being made of the sample comprising two mode, sample are established in the database of server
This two mode are respectively image modalities and text modality, and data set is divided into training set and test set.
Image and text are collected from webpage, and the identical piece image of corresponding meaning and a text are constituted into an image
Text pair, meaning identical finger describes same thing, such as the image and a text for describing people's surfing of width people's surfing
With regard to constituting an image text pair；To retrieve data set, an image text pair to building cross-module state by each image text
Characteristics of image and text feature constitute a sample.
The training set of cross-module state retrieval data set has n sample, and each sample includes the spy of two mode of image and text
Sign, X(1)Indicate the image modalities matrix that the feature of n image modalities is constituted, Indicate pth
The feature of the image modalities of a sample,Wherein d1Indicate the dimension of the feature of image modalities, R indicates set of real numbers；
X(2)Indicate the text modality matrix that the feature of n text modality is constituted, It indicates p-th
The feature of the text modality of sample,Wherein d2Indicate the dimension of the feature of text modality；It is corresponding by two mode
FeatureWithConstitute sample characteristics；Y={ y1,y2,…,ynIndicate label matrix, Y ∈ { 0,1 }c×n, wherein c table
Show classification sum, ypIndicate the label vector of p-th of sample, yp={ y1p,y2p,…,yip,…,ynp, yipIndicate p-th of sample
In the label of the i-th class classification；If p-th of sample belongs to the i-th class, the element y of the i-th row pth column in label matrix Yip=
1, otherwise yip=0.
2) objective function of similitude in similitude and mode between keeping mode is established, and passes through a kind of discrete optimizing method
Objective function is solved, Hash codes matrix is obtained, learns Hash codes for training set.
2.1) similarity matrix S is first constructed according to label matrix Y cosine similarity, the element of pth row q column is in S
Spq=yp·yq/(||yp||2||yq||2), wherein p and q is the ordinal number of sample, yp·yqIndicate the label of p-th of sample to
Measure ypWith the label vector y of q-th of sampleqBetween inner product, | | yp||2With | | yq||2Respectively indicate the label of p-th of sample to
Measure ypWith the label vector y of q-th of sampleqTwo norms；
Then, the loss function of similitude between keeping mode below is established:WhereinIt is F
Square of norm, B indicate the Hash codes matrix that the Hash codes of all samples are constituted, B ∈ { -1,1 }k×n, wherein k is Hash codes
Length；
2.2) similitude in mode is kept by using figure regular terms, (m=1 indicates image mould for m-th of mode
State, m=2 indicate text modality), establish the following loss function for keeping similitude in mode:
Above-mentioned weight matrix W(m)In elementSpecifically:
2.3) loss function of similitude between keeping mode is combinedWith the loss function for keeping similitude in modeEstablish the overall goal function of following study Hash codes are as follows:
s.t.B∈{-1,1}k×n
2.4) it carries out solving overall goal function using a kind of discrete optimizing method, specifically:
2.4.1) random initializtion Hash codes matrix B(0)∈{-1,1}k×n, B(0)Indicate initial Hash codes matrix B；Breathe out
The uncommon initial random generation of code matrix B, element therein are selected as -1 or 1.
2.4.2 solution) is iterated using following procedure:
First seek overall goal functionGradient:
Then iterative processing, the discrete Hash codes matrix B obtained using following formula according to iteration j(j)It handles
To the Hash codes matrix B of (j+1) secondary iteration(j+1):
Wherein, λ is learning rate；B(j)Indicate the Hash codes matrix that iteration j obtains, B(j+1)Indicate that (j+1) is secondary repeatedly
The Hash codes matrix B that generation obtains(j+1)；
Hash codes matrix is updated according to above-mentioned iterative formula and completes optimization process, obtains optimal Hash codes matrix B.
3) the Hash codes matrix succeeded in school according to step 2), learns the hash function of each mode.
Hash function uses simple Linear Mapping h1(x(1))=sign (P1 Tx(1)),It learns
Practising hash function is to learn two mapping matrix P1And P2, wherein P1Indicate the mapping matrix of image modalities, P2Indicate text mould
The mapping matrix of state, x(1)Indicate the feature of image modalities in a certain sample, x(2)Indicate the feature of text modality in a certain sample；
It solves following formula and obtains mapping matrix P1And P2:
Pass through orderIt solves to calculate and obtains P1: P1=(X(1)X(1)T+γI)-1X(1)BT；
Pass through orderIt solves to calculate and obtains P2: P2=(X(2)X(2)T+γI)-1X(2)BT。
4) Hash codes of all samples in training set and test set are calculated using hash function.
The Hash codes formula h of training set and test set image modalities1(x(1))=sign (P1 Tx(1)) calculate, wherein x(1)
Indicate the feature of the image modalities of sample, h1(x(1)) indicate by the sample image mode feature x(1)Calculated Hash codes；
The Hash codes formula of training set and test set text modalityIt calculates, wherein x(2)Indicate sample text
The feature of this mode, h2(x(2)) indicate by the sample text mode feature x(2)Calculated Hash codes.
5) using the test set of a mode as query set, using the training set of another mode as retrieved set, according to step
Rapid 4) mode obtains Hash codes, in the Hamming in the Hash codes of sample in calculating query set and retrieved set between the Hash codes of sample
Distance is ranked up sample in retrieved set according to the sequence of Hamming distance from small to large, and the forward sample that sorts will be by conduct
Search result.
The present embodiment is used as evaluation criterion using mAP (mean Average Precision), and mAP value is bigger, the side of explanation
The cross-module state retrieval performance of method is better.With CMFH (referring to document Tang J, Wang K, Shao on cross-module state data set Wiki
L.Supervised matrix factorization hashing for cross-modal retrieval[J].IEEE
Transactions on Image Processing,2016,25(7):3157-3166)、 SMFH(Ding G,Guo Y,
Zhou J.Collective matrix factorization hashing for multimodal data[C]
.Proceedings of the IEEE conference on computer vision and pattern
recognition.2014:2075-2082.)、FSH(Liu H,Ji R,Wu Y,et al. Cross-modality binary
code learning via fusion similarity hashing[C].Proceedings of CVPR.2017:6345-
6353.) three kinds of cross-module state hash methods are compared, and preceding 100 samples are returned when Hash code length is 16 bit
MAP value is as shown in table 1.
MAP value on 1 Wiki data set of table
| Method | Image retrieval text | Text retrieval image |
| CMFH | 0.2295 | 0.3479 |
| SMFH | 0.2411 | 0.3658 |
| FSH | 0.2408 | 0.3871 |
| The present invention | 0.2455 | 0.4086 |
As it can be seen from table 1 the method for the present invention achieves highest mAP value, cross-module state compared with three kinds of control methods
Retrieval performance is best.
Fig. 2 gives an example of an image retrieval text on cross-module state data set Wiki, and return is sequence
In preceding 6 text, affiliated semantic classes is given above image and text.Query image belongs to geography class, solid line
Frame indicates the text retrieved and query image belongs to same semantic classes, and dotted line frame indicates the text retrieved and query image
It is not belonging to same semantic classes.As can be seen that method provided by the invention is better than to analogy from the search result of this example
Method.
Fig. 3 gives an example of a text retrieval image on cross-module state data set Wiki, and return is sequence
In preceding 6 image, affiliated semantic classes is given above image and text.Query text belongs to literature class, figure
The rimless image for indicating to retrieve and query text belong to same semantic classes outside piece, have dotted line frame expression to retrieve outside picture
Image and query text are not belonging to same semantic classes.It is from the search result of this example as can be seen that provided by the invention
Method is better than control methods.
In conclusion the method for the present invention can effectively keep the similitude in similitude and mode between mode, and not
The Hash codes of similar sample have been concerned only with, the Hash codes of dissimilar sample have also been paid close attention to, have been conducive to the more different Kazakhstan of study
Uncommon code, and solve the problems, such as to learn discrete Hash codes using a kind of discrete optimizing method, to improve the retrieval of cross-module state
Accuracy.
Claims (5)
1. a kind of discrete Hash search method across modal data kept based on similitude, it is characterised in that: method includes such as
Lower step:
1) the cross-module state retrieval data set being made of the sample comprising two mode is established in the database of server, sample
Two mode are respectively image modalities and text modality, and data set is divided into training set and test set；
2) objective function of similitude in similitude and mode between keeping mode is established, and by a kind of discrete optimizing method to mesh
Scalar functions are solved, and Hash codes matrix is obtained；
3) the Hash codes matrix succeeded in school according to step 2), learns the hash function of each mode；
4) Hash codes of all samples in training set and test set are calculated using hash function；
5) using the test set of a mode as query set, using the training set of another mode as retrieved set, according to step 4)
Mode obtains Hash codes, calculates the Hamming distance in query set in the Hash codes of sample and retrieved set between the Hash codes of sample,
Sample in retrieved set is ranked up according to the sequence of Hamming distance from small to large, the forward sample that sorts will be tied as retrieval
Fruit.
2. a kind of discrete Hash search method across modal data kept based on similitude according to claim 1,
It is characterized in that: the step 1) specifically:
Image and text are collected from webpage, and the identical piece image of corresponding meaning and a text are constituted into an image text
Right, to retrieve data set to building cross-module state by each image text, the characteristics of image and text of image text pair are special
Sign constitutes a sample；The training set of cross-module state retrieval data set has n sample, and each sample includes two moulds of image and text
The feature of state, X(1)Indicate the image modalities matrix that the feature of n image modalities is constituted, Table
Show the feature of the image modalities of p-th of sample,Wherein d1Indicate the dimension of the feature of image modalities, R indicates real
Manifold；X(2)Indicate the text modality matrix that the feature of n text modality is constituted, It indicates
The feature of the text modality of p-th of sample,Wherein d2Indicate the dimension of the feature of text modality；By two moulds
The corresponding feature of stateWithConstitute sample characteristics；Y={ y1,y2,…,ynIndicate label matrix, Y ∈ { 0,1 }c×n,
Middle c indicates classification sum, ypIndicate the label vector of p-th of sample, yp={ y1p,y2p,…,yip,…,ynp, yipIndicate pth
Label of a sample in the i-th class classification.
3. a kind of discrete Hash search method across modal data kept based on similitude according to claim 1,
Be characterized in that: the step 2) specifically includes:
2.1) similarity matrix S is first constructed according to label matrix Y cosine similarity, the element of pth row q column is S in Spq=
yp·yq/(||yp||2||yq||2), wherein p and q is the ordinal number of sample, yp·yqIndicate the label vector y of p-th of samplep
With the label vector y of q-th of sampleqBetween inner product, | | yp||2With | | yq||2Respectively indicate the label vector y of p-th of samplep
With the label vector y of q-th of sampleqTwo norms；
Then, the loss function of similitude between keeping mode below is established:WhereinIt is F norm
Square, B indicates the Hash codes matrix that the Hash codes of all samples are constituted, B ∈ { -1,1 }k×n, wherein k is the length of Hash codes；
2.2) for m-th of mode (m=1 indicates image modalities, and m=2 indicates text modality), foundation is following to keep phase in mode
Like the loss function of property:
Wherein, bpAnd bqIt is the pth column and q column of Hash codes matrix B, W respectively(m)It is the weight matrix of m-th of mode,It is
Weight matrix W(m)Pth row q column element, LmIt is the Laplacian Matrix of m-th of mode, D(m)It is pair of m-th of mode
Angle battle array,Indicate diagonal matrix D(m)Pth row q column element,Lm=D(m)-W(m)；Tr () table
Show the mark of matrix,Indicate square of 2 norms；
Above-mentioned weight matrix W(m)In elementSpecifically:
Wherein, e is natural constant,Indicate in m-th of mode with sample characteristicsApart from nearest k1A sample is special
The set constituted is levied,Indicate in m-th of mode with sample characteristicsApart from farthest k2What a sample characteristics were constituted
Set, μ are tradeoff parameters, and the value of σ takes maximum
2.3) loss function of similitude between keeping mode is combinedWith the loss function for keeping similitude in modeEstablish the overall goal function of following study Hash codes are as follows:
s.t.B∈{-1,1}k×n
Wherein, α indicate keep mode between similitude loss function tradeoff parameter, β1It indicates to keep in the mode of image modalities
The tradeoff parameter of the loss function of similitude, β2Indicate to keep the tradeoff of the loss function of similitude in the mode of text modality to join
Number, T representing matrix transposition；
2.4) it carries out solving overall goal function using a kind of discrete optimizing method, specifically:
2.4.1) random initializtion Hash codes matrix B(0)∈{-1,1}k×n, B(0)Indicate initial Hash codes matrix B；
2.4.2 solution) is iterated using following procedure:
First seek overall goal functionGradient:
Then iterative processing, the discrete Hash codes matrix B obtained using following formula according to iteration j(j)Processing obtains the
(j+1) the Hash codes matrix B of secondary iteration(j+1):
Wherein, λ is learning rate；B(j)Indicate the Hash codes matrix that iteration j obtains, B(j+1)Indicate that (j+1) secondary iteration obtains
Hash codes matrix B(j+1)；
Hash codes matrix is updated according to above-mentioned iterative formula and completes optimization process, obtains optimal Hash codes matrix B.
4. a kind of discrete Hash search method across modal data kept based on similitude according to claim 1,
Be characterized in that: in the step 3), hash function uses simple Linear Mapping h1(x(1))=sign (P1 Tx(1)), h2(x(2))
=sign (P2 Tx(2)), study hash function is to learn two mapping matrix P1And P2, wherein P1Indicate the mapping of image modalities
Matrix, P2Indicate the mapping matrix of text modality, x(1)Indicate the feature of image modalities in sample, x(2)Indicate text mould in sample
The feature of state；
It solves following formula and obtains mapping matrix P1And P2:
Wherein,Indicate that the loss function of study mapping matrix, γ are tradeoff parameters；
Pass through orderIt solves to calculate and obtains P1: P1=(X(1)X(1)T+γI)-1X(1)BT, I expression unit matrix；
Pass through orderIt solves to calculate and obtains P2: P2=(X(2)X(2)T+γI)-1X(2)BT。
5. a kind of discrete Hash search method across modal data kept based on similitude according to claim 1,
It is characterized in that: in the step 4), the Hash codes formula h of training set and test set image modalities1(x(1))=sign (P1 Tx(1)) calculate, wherein x(1)Indicate the feature of the image modalities of sample, h1(x(1)) indicate by the sample image mode feature x(1)
Calculated Hash codes；The Hash codes formula of training set and test set text modalityIt calculates,
Middle x(2)Indicate the feature of sample text mode, h2(x(2)) indicate by the sample text mode feature x(2)Calculated Hash
Code.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910277146.3A CN110059198B (en) | 2019-04-08 | 2019-04-08 | Discrete hash retrieval method of cross-modal data based on similarity maintenance |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910277146.3A CN110059198B (en) | 2019-04-08 | 2019-04-08 | Discrete hash retrieval method of cross-modal data based on similarity maintenance |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110059198A true CN110059198A (en) | 2019-07-26 |
| CN110059198B CN110059198B (en) | 2021-04-13 |
Family
ID=67318463
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910277146.3A Active CN110059198B (en) | 2019-04-08 | 2019-04-08 | Discrete hash retrieval method of cross-modal data based on similarity maintenance |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110059198B (en) |
Cited By (21)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110059154A (en) * | 2019-04-10 | 2019-07-26 | 山东师范大学 | It is a kind of that Hash search method is migrated based on the cross-module state for inheriting mapping |
| CN110569387A (en) * | 2019-08-20 | 2019-12-13 | 清华大学 | radar-image cross-modal retrieval method based on depth hash algorithm |
| CN110609914A (en) * | 2019-08-06 | 2019-12-24 | 厦门大学 | Online Hash learning image retrieval method based on rapid category updating |
| CN110674323A (en) * | 2019-09-02 | 2020-01-10 | 山东师范大学 | Unsupervised cross-modal Hash retrieval method and system based on virtual label regression |
| CN110704659A (en) * | 2019-09-30 | 2020-01-17 | 腾讯科技（深圳）有限公司 | Image list sorting method and device, storage medium and electronic device |
| CN111078952A (en) * | 2019-11-20 | 2020-04-28 | 重庆邮电大学 | Cross-modal variable-length Hash retrieval method based on hierarchical structure |
| CN111125457A (en) * | 2019-12-13 | 2020-05-08 | 山东浪潮人工智能研究院有限公司 | Deep cross-modal Hash retrieval method and device |
| CN111209415A (en) * | 2020-01-10 | 2020-05-29 | 重庆邮电大学 | Image-text cross-modal Hash retrieval method based on mass training |
| CN111522903A (en) * | 2020-04-01 | 2020-08-11 | 济南浪潮高新科技投资发展有限公司 | Deep hash retrieval method, equipment and medium |
| CN111639197A (en) * | 2020-05-28 | 2020-09-08 | 山东大学 | Cross-modal multimedia data retrieval method and system with label embedded online hash |
| CN112199531A (en) * | 2020-11-05 | 2021-01-08 | 广州杰赛科技股份有限公司 | Cross-modal retrieval method and device based on Hash algorithm and neighborhood map |
| CN112559810A (en) * | 2020-12-23 | 2021-03-26 | 上海大学 | Method and device for generating hash code by utilizing multi-layer feature fusion |
| CN112732976A (en) * | 2021-01-13 | 2021-04-30 | 天津大学 | Short video multi-label rapid classification method based on deep hash coding |
| CN112925962A (en) * | 2021-01-20 | 2021-06-08 | 同济大学 | Hash coding-based cross-modal data retrieval method, system, device and medium |
| CN113434671A (en) * | 2021-06-23 | 2021-09-24 | 平安国际智慧城市科技股份有限公司 | Data processing method and device, computer equipment and storage medium |
| CN115080801A (en) * | 2022-07-22 | 2022-09-20 | 山东大学 | Cross-modal retrieval method and system based on federal learning and data binary representation |
| CN115374165A (en) * | 2022-10-24 | 2022-11-22 | 山东建筑大学 | Data retrieval method, system and equipment based on triple matrix decomposition |
| CN115687571A (en) * | 2022-10-28 | 2023-02-03 | 重庆师范大学 | Depth unsupervised cross-modal retrieval method based on modal fusion reconstruction hash |
| CN116244484A (en) * | 2023-05-11 | 2023-06-09 | 山东大学 | Federal cross-modal retrieval method and system for unbalanced data |
| WO2024045866A1 (en) * | 2022-08-31 | 2024-03-07 | Huawei Technologies Co., Ltd. | System and method for cross-modal interaction based on pre-trained model |
| CN113434671B (en) * | 2021-06-23 | 2024-06-07 | 平安国际智慧城市科技股份有限公司 | Data processing method, device, computer equipment and storage medium |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104346440A (en) * | 2014-10-10 | 2015-02-11 | 浙江大学 | Neural-network-based cross-media Hash indexing method |
| CN106777318A (en) * | 2017-01-05 | 2017-05-31 | 西安电子科技大学 | Matrix decomposition cross-module state Hash search method based on coorinated training |
| CN107256271A (en) * | 2017-06-27 | 2017-10-17 | 鲁东大学 | Cross-module state Hash search method based on mapping dictionary learning |
| CN107273505A (en) * | 2017-06-20 | 2017-10-20 | 西安电子科技大学 | Supervision cross-module state Hash search method based on nonparametric Bayes model |
| CN107402993A (en) * | 2017-07-17 | 2017-11-28 | 山东师范大学 | The cross-module state search method for maximizing Hash is associated based on identification |
| CN108170755A (en) * | 2017-12-22 | 2018-06-15 | 西安电子科技大学 | Cross-module state Hash search method based on triple depth network |
| CN108334574A (en) * | 2018-01-23 | 2018-07-27 | 南京邮电大学 | A kind of cross-module state search method decomposed based on Harmonious Matrix |
| CN108510559A (en) * | 2017-07-19 | 2018-09-07 | 哈尔滨工业大学深圳研究生院 | It is a kind of based on have supervision various visual angles discretization multimedia binary-coding method |
| CN108595688A (en) * | 2018-05-08 | 2018-09-28 | 鲁东大学 | Across the media Hash search methods of potential applications based on on-line study |
| CN109299216A (en) * | 2018-10-29 | 2019-02-01 | 山东师范大学 | A kind of cross-module state Hash search method and system merging supervision message |
-
2019
- 2019-04-08 CN CN201910277146.3A patent/CN110059198B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104346440A (en) * | 2014-10-10 | 2015-02-11 | 浙江大学 | Neural-network-based cross-media Hash indexing method |
| CN106777318A (en) * | 2017-01-05 | 2017-05-31 | 西安电子科技大学 | Matrix decomposition cross-module state Hash search method based on coorinated training |
| CN107273505A (en) * | 2017-06-20 | 2017-10-20 | 西安电子科技大学 | Supervision cross-module state Hash search method based on nonparametric Bayes model |
| CN107256271A (en) * | 2017-06-27 | 2017-10-17 | 鲁东大学 | Cross-module state Hash search method based on mapping dictionary learning |
| CN107402993A (en) * | 2017-07-17 | 2017-11-28 | 山东师范大学 | The cross-module state search method for maximizing Hash is associated based on identification |
| CN108510559A (en) * | 2017-07-19 | 2018-09-07 | 哈尔滨工业大学深圳研究生院 | It is a kind of based on have supervision various visual angles discretization multimedia binary-coding method |
| CN108170755A (en) * | 2017-12-22 | 2018-06-15 | 西安电子科技大学 | Cross-module state Hash search method based on triple depth network |
| CN108334574A (en) * | 2018-01-23 | 2018-07-27 | 南京邮电大学 | A kind of cross-module state search method decomposed based on Harmonious Matrix |
| CN108595688A (en) * | 2018-05-08 | 2018-09-28 | 鲁东大学 | Across the media Hash search methods of potential applications based on on-line study |
| CN109299216A (en) * | 2018-10-29 | 2019-02-01 | 山东师范大学 | A kind of cross-module state Hash search method and system merging supervision message |
Non-Patent Citations (1)
| Title |
|---|
| LI MINGYANG: ""Discrete similarity preserving hashing for cross-modal retrival"", 《ICAIS2019：ARTIFICAL INTELLIGENCE AND SECURITY》 * |
Cited By (33)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110059154A (en) * | 2019-04-10 | 2019-07-26 | 山东师范大学 | It is a kind of that Hash search method is migrated based on the cross-module state for inheriting mapping |
| CN110059154B (en) * | 2019-04-10 | 2022-04-15 | 山东师范大学 | Cross-modal migration hash retrieval method based on inheritance mapping |
| CN110609914A (en) * | 2019-08-06 | 2019-12-24 | 厦门大学 | Online Hash learning image retrieval method based on rapid category updating |
| CN110609914B (en) * | 2019-08-06 | 2021-08-17 | 厦门大学 | Online Hash learning image retrieval method based on rapid category updating |
| CN110569387A (en) * | 2019-08-20 | 2019-12-13 | 清华大学 | radar-image cross-modal retrieval method based on depth hash algorithm |
| CN110569387B (en) * | 2019-08-20 | 2020-12-11 | 清华大学 | Radar-image cross-modal retrieval method based on depth hash algorithm |
| CN110674323B (en) * | 2019-09-02 | 2020-06-30 | 山东师范大学 | Unsupervised cross-modal Hash retrieval method and system based on virtual label regression |
| CN110674323A (en) * | 2019-09-02 | 2020-01-10 | 山东师范大学 | Unsupervised cross-modal Hash retrieval method and system based on virtual label regression |
| CN110704659A (en) * | 2019-09-30 | 2020-01-17 | 腾讯科技（深圳）有限公司 | Image list sorting method and device, storage medium and electronic device |
| CN110704659B (en) * | 2019-09-30 | 2023-09-26 | 腾讯科技（深圳）有限公司 | Image list ordering method and device, storage medium and electronic device |
| CN111078952B (en) * | 2019-11-20 | 2023-07-21 | 重庆邮电大学 | Cross-modal variable-length hash retrieval method based on hierarchical structure |
| CN111078952A (en) * | 2019-11-20 | 2020-04-28 | 重庆邮电大学 | Cross-modal variable-length Hash retrieval method based on hierarchical structure |
| CN111125457A (en) * | 2019-12-13 | 2020-05-08 | 山东浪潮人工智能研究院有限公司 | Deep cross-modal Hash retrieval method and device |
| CN111209415A (en) * | 2020-01-10 | 2020-05-29 | 重庆邮电大学 | Image-text cross-modal Hash retrieval method based on mass training |
| CN111209415B (en) * | 2020-01-10 | 2022-09-23 | 重庆邮电大学 | Image-text cross-modal Hash retrieval method based on mass training |
| CN111522903A (en) * | 2020-04-01 | 2020-08-11 | 济南浪潮高新科技投资发展有限公司 | Deep hash retrieval method, equipment and medium |
| CN111639197A (en) * | 2020-05-28 | 2020-09-08 | 山东大学 | Cross-modal multimedia data retrieval method and system with label embedded online hash |
| CN112199531A (en) * | 2020-11-05 | 2021-01-08 | 广州杰赛科技股份有限公司 | Cross-modal retrieval method and device based on Hash algorithm and neighborhood map |
| CN112199531B (en) * | 2020-11-05 | 2024-05-17 | 广州杰赛科技股份有限公司 | Cross-modal retrieval method and device based on hash algorithm and neighborhood graph |
| CN112559810A (en) * | 2020-12-23 | 2021-03-26 | 上海大学 | Method and device for generating hash code by utilizing multi-layer feature fusion |
| CN112732976A (en) * | 2021-01-13 | 2021-04-30 | 天津大学 | Short video multi-label rapid classification method based on deep hash coding |
| CN112925962A (en) * | 2021-01-20 | 2021-06-08 | 同济大学 | Hash coding-based cross-modal data retrieval method, system, device and medium |
| CN112925962B (en) * | 2021-01-20 | 2022-09-27 | 同济大学 | Hash coding-based cross-modal data retrieval method, system, device and medium |
| CN113434671A (en) * | 2021-06-23 | 2021-09-24 | 平安国际智慧城市科技股份有限公司 | Data processing method and device, computer equipment and storage medium |
| CN113434671B (en) * | 2021-06-23 | 2024-06-07 | 平安国际智慧城市科技股份有限公司 | Data processing method, device, computer equipment and storage medium |
| CN115080801A (en) * | 2022-07-22 | 2022-09-20 | 山东大学 | Cross-modal retrieval method and system based on federal learning and data binary representation |
| WO2024045866A1 (en) * | 2022-08-31 | 2024-03-07 | Huawei Technologies Co., Ltd. | System and method for cross-modal interaction based on pre-trained model |
| CN115374165B (en) * | 2022-10-24 | 2023-03-24 | 山东建筑大学 | Data retrieval method, system and equipment based on triple matrix decomposition |
| CN115374165A (en) * | 2022-10-24 | 2022-11-22 | 山东建筑大学 | Data retrieval method, system and equipment based on triple matrix decomposition |
| CN115687571B (en) * | 2022-10-28 | 2024-01-26 | 重庆师范大学 | Depth unsupervised cross-modal retrieval method based on modal fusion reconstruction hash |
| CN115687571A (en) * | 2022-10-28 | 2023-02-03 | 重庆师范大学 | Depth unsupervised cross-modal retrieval method based on modal fusion reconstruction hash |
| CN116244484A (en) * | 2023-05-11 | 2023-06-09 | 山东大学 | Federal cross-modal retrieval method and system for unbalanced data |
| CN116244484B (en) * | 2023-05-11 | 2023-08-08 | 山东大学 | Federal cross-modal retrieval method and system for unbalanced data |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110059198B (en) | 2021-04-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110059198A (en) | A kind of discrete Hash search method across modal data kept based on similitude | |
| Li et al. | Recent developments of content-based image retrieval (CBIR) | |
| Wang et al. | Richpedia: a large-scale, comprehensive multi-modal knowledge graph | |
| CN108132968B (en) | Weak supervision learning method for associated semantic elements in web texts and images | |
| CN106095893B (en) | A kind of cross-media retrieval method | |
| Wang et al. | Learning common and specific features for RGB-D semantic segmentation with deconvolutional networks | |
| Li et al. | Region-wise deep feature representation for remote sensing images | |
| Huang et al. | Identifying disaster related social media for rapid response: a visual-textual fused CNN architecture | |
| CN109299341A (en) | One kind confrontation cross-module state search method dictionary-based learning and system | |
| CN110674407A (en) | Hybrid recommendation method based on graph convolution neural network | |
| Xie et al. | Fast and accurate near-duplicate image search with affinity propagation on the ImageWeb | |
| Tang et al. | Large-scale remote sensing image retrieval based on semi-supervised adversarial hashing | |
| Zhang et al. | Adaptively Unified Semi-supervised Learning for Cross-Modal Retrieval. | |
| Li et al. | Online hashing for scalable remote sensing image retrieval | |
| Xu et al. | Instance-level coupled subspace learning for fine-grained sketch-based image retrieval | |
| Abdul-Rashid et al. | Shrec’18 track: 2d image-based 3d scene retrieval | |
| CN109284414A (en) | The cross-module state content search method and system kept based on semanteme | |
| Cheng et al. | Deep attentional fine-grained similarity network with adversarial learning for cross-modal retrieval | |
| Guo | Intelligent sports video classification based on deep neural network (DNN) algorithm and transfer learning | |
| Zhu et al. | Real-time data filling and automatic retrieval algorithm of road traffic based on deep-learning method | |
| Deng | Content-based image collection summarization and comparison using self-organizing maps | |
| Benhabiles et al. | Convolutional neural network for pottery retrieval | |
| Ma et al. | Dual modality collaborative learning for cross-source remote sensing retrieval | |
| Moumtzidou et al. | Verge in vbs 2017 | |
| Feng et al. | Scene semantic recognition based on probability topic model |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |