CN106649715B - A kind of cross-media retrieval method based on local sensitivity hash algorithm and neural network - Google Patents

A kind of cross-media retrieval method based on local sensitivity hash algorithm and neural network Download PDF

Info

Publication number
CN106649715B
CN106649715B CN201611190238.0A CN201611190238A CN106649715B CN 106649715 B CN106649715 B CN 106649715B CN 201611190238 A CN201611190238 A CN 201611190238A CN 106649715 B CN106649715 B CN 106649715B
Authority
CN
China
Prior art keywords
hash
function
neural network
text
local sensitivity
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201611190238.0A
Other languages
Chinese (zh)
Other versions
CN106649715A (en
Inventor
白亮
贾玉华
郭金林
谢毓湘
于天元
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
National University of Defense Technology
Original Assignee
National University of Defense Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by National University of Defense Technology filed Critical National University of Defense Technology
Priority to CN201611190238.0A priority Critical patent/CN106649715B/en
Publication of CN106649715A publication Critical patent/CN106649715A/en
Application granted granted Critical
Publication of CN106649715B publication Critical patent/CN106649715B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/40—Information retrieval; Database structures therefor; File system structures therefor of multimedia data, e.g. slideshows comprising image and additional audio data
    • G06F16/43—Querying
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Biophysics (AREA)
  • Evolutionary Computation (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Computational Linguistics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Multimedia (AREA)
  • Databases & Information Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The cross-media retrieval method based on local sensitivity hash algorithm and neural network that the invention discloses a kind of, it is related to cross-media retrieval technical field, this method includes that local sensitivity Hash and hash function learn two stages, in the local sensitivity Hash stage, m Hash table G=[g is mapped the image data to by local sensitivity hash algorithm1,g2,...,gm]∈Rk×mHash bucket in, wherein G be m Hash table set, gjIndicate that j-th of Hash table, k are the length that Hash bucket corresponds to Hash codes；Learn the stage in hash function, learns for text data to be respectively mapped to the hash function Ht=(Ht in m Hash table in its corresponding Hash bucket by neural network algorithm(1),Ht(2),...,Ht(m)), Ht(j), (1≤j≤m) indicates the hash function Ht corresponding to j-th of Hash table learnt.After having obtained the function in the two stages, coding further is carried out to all images and document and establishes index, to carry out more accurate retrieval.

Description

A kind of cross-media retrieval method based on local sensitivity hash algorithm and neural network
Technical field
The present invention relates to cross-media retrieval technical field, refer in particular to a kind of based on local sensitivity hash algorithm and neural network Cross-media retrieval method.
Background technique
In across media big data eras, do not brought all the time in the magnanimity multi-modal information of generation huge across media Search Requirement such as searches for image or video with text, and vice versa.For example, an entry on wikipedia generally comprises Text description and example image, the retrieval of these information need to construct cross media indexing and learning method.With traditional single matchmaker Physical examination rope is compared, and the key problem of cross-media retrieval how is excavated between the identical or related semantic object of different media representations Association.
At present worldwide, numerous solutions is proposed for the key problem of the cross-media retrieval.It is existing Cross-media retrieval method be broadly divided into two classes, one kind is the method based on theme: document [1] by theme proportion grading to not It is modeled with the correlation between the data of mode；Document [2] by CORR-LDA excavate between image and text marking The relationship of theme level；Markov random fields in conjunction with tradition LDA method, are proposed and are examined with brief text by document [3] The built-up pattern (MDRF) of the oriented and undirected probability graph of rope image；Document [4] proposes a kind of to utilize multiple medium types Micro-blog information come carry out obtain social event visualization summarize multimedia social event autoabstract frame.It is another kind of to be Method based on subspace: the core of this kind of methods is to seek to make the maximized subspace of different modalities data dependence [5].Sharma et al. proposes a kind of general multi-modal feature extraction framework technology, referred to as the multi-view analysis GMA of broad sense [6].Semantic viewpoint is introduced in the T-V CCA model that document [7] proposes, to improve multi-modal number different classes of in subspace According to classification accuracy.Document [8] proposes a kind of Bi-CMSRM method, constructs from the angle of optimization bi-directional list sequencing problem Computation model suitable for cross-media retrieval.
[1]Blei D M,Ng A Y,Jordan M I.Latent dirichlet allocation[J].the Journal of machine Learning research,2003,3:993-1022.
[2]Blei D M,Jordan M I.Modeling annotated data[C]//Proceedings of the 26th annual international ACM SIGIR conference on Research and developme nt in
information retrieval.ACM,2003:127-134.
[3]Jia Y,Salzmann M,Darrell T.Learning cross modality similarity for multinomial data[C]//Computer Vision(ICCV),2011IEEE International Conference on.
IEEE,2011:2407-2414.
[4]Bian J,Yang Y,Zhang H,et al.Multimedia Summarization for Social Events in Microblog Stream[J].IEEE Transactions on Multimedia,2015,17(2):216- 228.
[5]Hardoon D R,Szedmak S,ShaweTaylor J.Canonical correlation analysis:An overview with application to learning methods[J].Neural computation,2004,16(12):2639-2664.
[6]Abhishek Sharma,Abhishek Kumar,H Daume,and DavidWJacobs.2012.Generalized multi-view analysis:A discriminative latent space.In IEEE Conference on Computer Vision and Pattern Recognition.2160– 2167.
[7]Yunchao Gong,Qifa Ke,Michael Isard,and Svetlana Lazebnik.2013.A Multi-View Embedding Space for Modeling Internet Images,Tags,and Their Semantics.International Journal of Computer Vision(2013),1–24.
Wu F,Lu X,Zhang Z,et al.Cross-media semantic representation via bi- directional learning to rank[C]//Proceedings of the 21st ACM international conference on Multimedia.ACM,2013:877-886.
There is same technological deficiency in existing cross-media retrieval method, i.e., only only considered cross-media retrieval method Itself and have ignored some feasible optimization processings to document sets, due to exist in document sets largely with inquire incoherent text Shelves, therefore pre-process before accurately inquire to document sets improve in document sets relevant documentation proportion to mentioning It is of great significance for high recall precision.
Summary of the invention
For technical problem present in existing cross-media retrieval method, the present invention proposes a kind of to can be improved retrieval The cross-media retrieval method based on local sensitivity hash algorithm and neural network of accuracy.
The specific technical solution of the present invention is:
A kind of cross-media retrieval method based on local sensitivity hash algorithm and neural network, the cross-media retrieval method The following steps are included:
1) FCMR (Fast Cross-Media Retrieval, FCMR) model is established, the FCMR model was trained Journey includes local sensitivity Hash stage and hash function study stage；
2) all texts and image are mapped using the hash function that local sensitivity hash function and neural network learning arrive It establishes and indexes to Hamming space；
3) cross-media retrieval inquiry, including text query and image querying are carried out.
As the preferred technical solution of the present invention, in step 1) of the present invention, the local sensitivity Hash stage includes Hash bucket is mapped the image data to using local sensitivity hash algorithm, is specifically included image through local sensitivity hash algorithm Data are mapped to m Hash table G=[g1,g2,...,gm]∈Rk×mHash bucket in, wherein G be m Hash table set, gj Indicate that j-th of Hash table, k are the length that Hash bucket corresponds to Hash codes.
As the preferred technical solution of the present invention, in step 1) of the present invention, the hash function study stage includes Learn the hash function Ht that text data is mapped to Hash bucket using neural network algorithm, specifically includes and calculated by neural network Text data is respectively mapped to the hash function Ht=(Ht in m Hash table in its corresponding Hash bucket by calligraphy learning(1),Ht(2),...,Ht(m)), Ht(j), (1≤j≤m) indicates the hash function corresponding to j-th of Hash table learnt.
As the preferred technical solution of the present invention, in step 3) of the present invention,
The text query is to give a query text, passes through hash function Ht(j)The query text is mapped to m In Hash bucket in Hash table, then the image file stored in these Hash buckets just constitutes the arest neighbors of the query text, will The image pattern in identical Hash bucket is fallen in as candidate result collection with query text, and then in the arest neighbors of the query text It is accurately retrieved in range, calculate the distance between image that query text is concentrated with candidate result and is accurately examined Rope ranking；
The query image is mapped to give a query image by local sensitivity hash function by described image inquiry In Hash bucket in m Hash table, then the text file stored in these Hash buckets just constitutes the arest neighbors of the query image, And then precise search is carried out in the arest neighbors range of the query image.
As the preferred technical solution of the present invention, local sensitivity hash function of the present invention is defined as follows:
Wherein, hyperplane vectorMeet multi GaussianN (0,1) distribution；
Define a series of hash function h1,h2,...,hnK function component function g (x) therein is randomly selected, if choosing It is h1To hk, then g (x)=(h1(x),h2(x),...,hk(x)) m g (x) function: g, is chosen1(x),g2(x),...,gm(x), The then corresponding Hash table of each g (x) function；By m g (x) function by each of image space image pattern piPoint It is not mapped in m Hash table, image pattern p each in this wayiWill occur in some Hash bucket of m Hash table；So pi Corresponding Hash bucket can indicate in j-th of Hash table are as follows:
gj(pi)=< h1(pi),h2(pi)...,hk(pi) >, (0 < j≤m, 0 < i≤n) (2)
As the preferred technical solution of the present invention, m neural network NN being used in FCMR model of the present invention(j),(j∈ 1,2 ..., m) structure having the same；Each neural network NN(j)There are L layers, wherein input layer has dt neuron to correspond to The dimension of text feature, the position k that output layer has k neuron to correspond to Hash codes, the residue other than input layer and output layer L-2 layer for learning hash function；By each ti∈ T is as NN(j)Input, each layer of available neural network OutputL+1 layers withFor input, output
WhereinWithRespectively l layers and l+1 layers of feature representation；W(l+1)It is transition matrix；f(l+1)It is activation primitive；
The hash function Ht that neural network learning arrives(j)With tiTo input and exporting the Hash codes that length is k:
Wherein,It is a k dimension real-valued vectors, it will using sign functionIt is converted into Hash codes；
For training sampleHt(j)(ti) withShould be identical, that is, WithIt is as equal as possible.
Loss function is defined based on minimum variance are as follows:
Wherein,It is not put in marks Function Neural Network to tiPredicted value,Indicate piCorresponding to jth (0 < j ≤ m) Hash bucket in a Hash table Hash codes；
Training sample needed for training neural network is obtained from the local sensitivity Hash stage(i∈1,2,..., Nt, j ∈ 1,2 ..., m) passes through training neural network NN(j)Its study can be made to arrive tiIt is mapped toHash function.
As the preferred technical solution of the present invention, the training of neural network of the present invention is divided into pre-training and parameter adjustment, tool Body includes:
(1) stack self-encoding encoder (Stacked AutoEncoder, SAE) is applied to FCMR model sequentially to train Neural network NN(j)In each layer with initialization network parameter；
(2) it is based on the loss function formula (5), trains neural network to adjust network parameter by BP algorithm；
(3) variance and SSE based on all samples of text devise shown in whole loss function such as formula (6):
Compared with prior art, the beneficial effects of the present invention are:
The present invention is based on local sensitivity hash algorithm and neural networks, by eliminating document content largely unrelated with inquiry And the arest neighbors of a group polling is obtained, finally retrieval tasks are more efficiently carried out within the scope of the arest neighbors of inquiry document.
Detailed description of the invention
Fig. 1 is FCMR block schematic illustration of the invention.
Fig. 2 is that FCMR of the invention retrieves schematic diagram.
Specific embodiment
It elaborates now in conjunction with Figure of description to the present invention.
A kind of cross-media retrieval based on local sensitivity hash algorithm and neural network that the specific embodiment of the invention provides Method (Fast Cross-Media Retrieval, FCMR), the cross-media retrieval method mainly includes the following steps:
1) FCMR (Fast Cross-Media Retrieval, FCMR) model is established, the FCMR model was trained Journey includes local sensitivity Hash stage and hash function study stage；
2) all texts and image are mapped using the hash function that local sensitivity hash function and neural network learning arrive It establishes and indexes to Hamming space；
3) cross-media retrieval inquiry, including text query and image querying are carried out.
Wherein, more succinct in order to state symbol and algorithm, it describes to mention by taking two mode of text and image as an example below FCMR model out, model can easily expand to other mode, and the FCMR model includes local sensitivity Hash and Kazakhstan Uncommon two stages of function learning.
In the local sensitivity Hash stage, Hash bucket is mapped the image data to using local sensitivity hash algorithm, it is specific to wrap It includes and m Hash table G=[g is mapped the image data to by local sensitivity hash algorithm1,g2,...,gm]∈Rk×mHash bucket Interior, wherein R indicates real number field, and G is the set of m Hash table, gjIndicate that j-th of Hash table, k are that Hash bucket corresponds to Hash codes Length；
Learn the stage in hash function, learns the Hash letter that text data is mapped to Hash bucket using neural network algorithm Number Ht, specifically include by neural network algorithm study by text data be respectively mapped in m Hash table text data it is right Hash function Ht=(Ht in the Hash bucket answered(1),Ht(2),...,Ht(m)), Ht(j), (1≤j≤m) indicates the correspondence learnt In the hash function of j-th of Hash table.
The matrix description of text data are as follows: T=[t1,t2,...,tnt]∈Rdt×nt, wherein T is that the matrix of text data is retouched It states.Correspondingly, P=[p1,p2,...,pnp]∈Rdp×np, wherein P is the matrix description of image data.Wherein, tiWith piOne is a pair of It answers, the number of image text pair is n, i.e. nt=np=n replaces nt and np with n in following content.
If obtaining m Hash table with local sensitivity hash algorithm, need to design m minds corresponding with Hash table Through network text data to be mapped in m Hash table in Hash bucket corresponding to these text datas.Based on neural network The local sensitivity hash function that the hash function and local sensitivity Hash stage learnt uses, can establish multi-modal data Index, to carry out efficient cross-media retrieval task.
After establishing index, a query text is given, hash function Ht is passed through(j)The query text is mapped to m In Hash bucket in Hash table, then the image file stored in these Hash buckets just constitutes the arest neighbors of the query text, into And it is accurately retrieved within the scope of the arest neighbors of the query text；A query image is given, local sensitivity Hash is passed through The query image is mapped in the Hash bucket in m Hash table by function, then the text file stored in these Hash buckets is with regard to group Precise search is carried out at the arest neighbors of the query image, and then in the arest neighbors range of the query image.
The following detailed description of the local sensitivity hash algorithm in the specific embodiment of the invention, the local sensitivity hash algorithm It is mainly used to solve the problems, such as the approximate KNN search at higher dimensional space midpoint, local sensitivity hash function is defined as follows:
Wherein, hyperplane vectorMeet multi GaussianN (0,1) distribution.
Define a series of hash function h1,h2,...,hnK function component function g (x) therein is randomly selected, if choosing It is h1To hk, then g (x)=(h1(x),h2(x),...,hk(x)) m g (x) function: g, is chosen1(x),g2(x),...,gm(x), The then corresponding Hash table of each g (x) function.By m g (x) function by each of image space image pattern piPoint It is not mapped in m Hash table, image pattern p each in this wayiWill occur in some Hash bucket of m Hash table.
So piCorresponding Hash bucket can indicate in j-th of Hash table are as follows:
gj(pi)=< h1(pi),h2(pi)...,hk(pi) >, (0 < j≤m, 0 < i≤n) (2)
When inquiry, query text is given, Ht is utilized(j)Functional query text is mapped, and will be fallen in query text identical Hash bucket in image pattern as candidate result collection, calculate the distance between the image that query text and candidate result are concentrated And carry out accurate retrieval ranking.
Pass through local sensitivity hash algorithm, the sample p of image spacei, (0 < i≤n) is mapped in m Hash table, and Each pi, (0 < i≤n) can all appear in some Hash bucket of m Hash table together with sample similar with its.In this way, each A image pattern piSome Hash bucket all with jth (0 < j≤m) a Hash table establishes connection.It is mentioned above simultaneously, Due to p in modeliAnd tiThe description of same semantic different modalities, image pattern and samples of text be it is one-to-one, therefore, Each samples of text tiAlso some Hash bucket with jth (0 < j≤m) a Hash table establishes connection.So far, it is used By samples of text t in training neural network learningiIt is mapped to samples of text t in jth (0 < j≤m) a Hash tableiCorresponding Hash The training sample of the function of bucket:(i ∈ 1,2 ..., n, j ∈ 1,2 ..., m), whereinIndicate piCorresponding to The Hash codes of Hash bucket in j (0 < j≤m) a Hash table.
The following detailed description of the local sensitivity hash algorithm in the specific embodiment of the invention, as shown in Figure 1, Fig. 1 gives Hash function learns stage neural network structure, the m neural network NN used in Fig. 1 model(j),(j∈1,2,...,m) Structure having the same；Each neural network NN(j)There are L layers, wherein input layer has dt neuron to correspond to text feature Dimension, the position k that output layer has k neuron to correspond to Hash codes, remaining L-2 layers for learning hash function.By each ti∈ T is as NN(j)Input, the output of each layer of available neural networkL+1 layers withFor Input, output
WhereinWithRespectively l layers and l+1 layers of feature representation；W(l+1)It is transition matrix；f(l+1)It is activation primitive.
The hash function Ht that neural network learning arrives(j)With tiTo input and exporting the Hash codes that length is k:
Wherein,It is a k dimension real-valued vectors, it will using sign functionIt is converted into Hash codes.
Due to sign function non-differentiability, it is difficult to optimize, therefore is eliminated with the stage of neural network learning hash function Sign function, and added again in test phase.
For training sampleHt(j)(ti) withShould be identical, that is, WithIt is as equal as possible.
Loss function is defined based on minimum variance are as follows:
Wherein,It is not put in marks Function Neural Network to tiPredicted value.
Training sample needed for training neural network is obtained according to the local sensitivity Hash stage(i∈1, 2 ..., nt, j ∈ 1,2 ..., m) passes through training neural network NN(j)Its study can be made to arrive tiIt is mapped toHash Function.
The training of neural network is divided into pre-training and parameter adjustment, and pre-training can preferably initialization network parameter and prevent Only network falls into locally optimal solution, the training of neural network specifically includes the following steps:
(1) stack self-encoding encoder (Stacked AutoEncoder, SAE) is applied to FCMR model sequentially to train Neural network NN(j)In each layer with initialization network parameter.
(2) loss function formula (5) are based on, network parameter is adjusted by BP algorithm (back-propagation algorithm) Lai Xunlian network；
(3) variance and SSE based on all samples of text devise shown in whole loss function such as formula (6):
In order to enable neural network NN(j)The function Ht learnt(j)Samples of text data can be mapped to j well In the corresponding Hash bucket of its in Hash table, the embodiment of the present invention trains neural network NN using traditional back-propagation algorithm(j), final hash function Ht is obtained eventually by formula (4) in test phase(j)。
Wherein, the algorithmic procedure of the FCMR of the present embodiment is specific as follows:
Fig. 2 shows when only one Hash table, the schematic diagram that FCMR is retrieved, multiple Hash tables need to only use all minds Through e-learning to hash function map the text to Hamming space.
Obviously, the above embodiment of the present invention be only to clearly illustrate example of the present invention, and not be pair The restriction of embodiments of the present invention.For those of ordinary skill in the art, may be used also on the basis of the above description To make other variations or changes in different ways.There is no necessity and possibility to exhaust all the enbodiments.It is all this Made any modifications, equivalent replacements, and improvements etc., should be included in the claims in the present invention within the spirit and principle of invention Protection scope within.

Claims (3)

1. a kind of cross-media retrieval method based on local sensitivity hash algorithm and neural network, which is characterized in that described across matchmaker Body search method the following steps are included:
1) FCMR (Fast Cross-Media Retrieval, FCMR) model, the training process packet of the FCMR model are established Include local sensitivity Hash stage and hash function study stage；The local sensitivity Hash stage includes using local sensitivity Hash Algorithm maps the image data to Hash bucket, specifically includes and maps the image data to m Kazakhstan by local sensitivity hash algorithm Uncommon table G=[g1,g2,...,gm]∈Rk×mHash bucket in, wherein R indicate real number field, G be m Hash table set, gjIt indicates J-th of Hash table, k are the length that Hash bucket corresponds to Hash codes；The hash function study stage includes being calculated using neural network Text data is mapped to the hash function Ht of Hash bucket by calligraphy learning, is specifically included and is learnt by neural network algorithm by textual data According to the hash function Ht=(Ht being respectively mapped in m Hash table in its corresponding Hash bucket(1),Ht(2),...,Ht(m)), Ht(j), (1≤j≤m) indicates the hash function corresponding to j-th of Hash table learnt；
The local sensitivity hash function is defined as follows:
Wherein, hyperplane vectorMeet multi Gaussian N (0,1) distribution；
Define a series of hash function h1,h2,...,hn, k function component function g (x) therein is randomly selected, if that choosing is h1 To hk, then g (x)=(h1(x),h2(x),...,hk(x)) m g (x) function: g, is chosen1(x),g2(x),...,gm(x), then often The corresponding Hash table of a g (x) function；By m g (x) function by each of image space image pattern piIt reflects respectively It is mapped in m Hash table, image pattern p each in this wayiWill occur in some Hash bucket of m Hash table；So pi? Corresponding Hash bucket can indicate in j Hash table are as follows:
gj(pi)=< h1(pi),h2(pi)...,hk(pi) >, (0 < j≤m, 0 < i≤n) (2)
Wherein, the m neural network NN used in the FCMR model(j), (j ∈ 1,2 ..., m) structure having the same； Each neural network NN(j)There are L layers, the dimension that wherein input layer has dt neuron to correspond to text feature, output layer has k A neuron corresponds to the position k of Hash codes, is used to learn hash function for remaining L-2 layers in addition to input layer and output layer；It will be every One ti∈ T is as NN(j)Input, the output of each layer of available neural networkL+1 layers withFor input, output
WhereinWithRespectively l layers and l+1 layers of feature representation；W(l+1)It is transition matrix；f(l+1) It is activation primitive；
The hash function Ht that neural network learning arrives(j)With tiTo input and exporting the Hash codes that length is k:
Wherein,It is a k dimension real-valued vectors, it will using sign functionIt is converted into Hash codes；
Loss function is defined based on minimum variance are as follows:
Wherein,It is not put in marks Function Neural Network to tiPredicted value, Yi (j)Indicate piCorresponding to jth (0 < j≤m) The Hash codes of Hash bucket in a Hash table；
Training sample (t needed for training neural network is obtained from the local sensitivity Hash stagei,Yi (j)), (i ∈ 1,2 ..., nt, j ∈ 1,2 ..., m) passes through training neural network NN(j)Its study can be made to arrive tiIt is mapped to Yi (j)Hash function；
2) hash function arrived using local sensitivity hash function and neural network learning is by all text datas and image data It is mapped to Hamming space and establishes index；
3) cross-media retrieval inquiry, including text query and image querying are carried out.
2. a kind of cross-media retrieval method based on local sensitivity hash algorithm and neural network according to claim 1, It is characterized in that, in the step 3),
The text query is to give a query text, passes through hash function Ht(j)The query text is mapped to m Hash In Hash bucket in table, then the image file stored in these Hash buckets just constitutes the arest neighbors of the query text, will with look into It askes text and falls in the image pattern in identical Hash bucket as candidate result collection, and then in the arest neighbors range of the query text It is interior accurately to be retrieved, it calculates the distance between image that query text and candidate result are concentrated and carries out accurate retrieval and arrange Name；
The query image is mapped to m to give a query image, by local sensitivity hash function by described image inquiry In Hash bucket in Hash table, then the text file stored in these Hash buckets just constitutes the arest neighbors of the query image, into And precise search is carried out in the arest neighbors range of the query image.
3. a kind of cross-media retrieval method based on local sensitivity hash algorithm and neural network according to claim 2, It is characterized in that, the training of neural network is divided into pre-training and parameter adjustment, specifically include:
(1) stack self-encoding encoder (Stacked AutoEncoder, SAE) is applied to the training NN that FCMR model carrys out sequence(j) In each layer with initialization network parameter；
(2) loss function formula (5) are based on, network is trained to adjust network parameter by BP algorithm；
(3) variance and SSE based on all samples of text devise shown in whole loss function such as formula (6):
CN201611190238.0A 2016-12-21 2016-12-21 A kind of cross-media retrieval method based on local sensitivity hash algorithm and neural network Active CN106649715B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201611190238.0A CN106649715B (en) 2016-12-21 2016-12-21 A kind of cross-media retrieval method based on local sensitivity hash algorithm and neural network

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201611190238.0A CN106649715B (en) 2016-12-21 2016-12-21 A kind of cross-media retrieval method based on local sensitivity hash algorithm and neural network

Publications (2)

Publication Number Publication Date
CN106649715A CN106649715A (en) 2017-05-10
CN106649715B true CN106649715B (en) 2019-08-09

Family

ID=58834417

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201611190238.0A Active CN106649715B (en) 2016-12-21 2016-12-21 A kind of cross-media retrieval method based on local sensitivity hash algorithm and neural network

Country Status (1)

Country Link
CN (1) CN106649715B (en)

Families Citing this family (22)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107256271B (en) * 2017-06-27 2020-04-03 鲁东大学 Cross-modal Hash retrieval method based on mapping dictionary learning
CN107273529B (en) * 2017-06-28 2020-02-07 武汉图信科技有限公司 Efficient hierarchical index construction and retrieval method based on hash function
US11531859B2 (en) * 2017-08-08 2022-12-20 Samsung Electronics Co., Ltd. System and method for hashed compressed weighting matrix in neural networks
CN107729290B (en) * 2017-09-21 2021-05-11 北京大学深圳研究生院 Representation learning method of super-large scale graph by using locality sensitive hash optimization
CN107729557A (en) * 2017-11-08 2018-02-23 北京大学 A kind of classification of inventory information, search method and device
CN108280207A (en) * 2018-01-30 2018-07-13 深圳市茁壮网络股份有限公司 A method of the perfect Hash of construction
CN108319686B (en) * 2018-02-01 2021-07-30 北京大学深圳研究生院 Antagonism cross-media retrieval method based on limited text space
CN108229588B (en) * 2018-02-08 2020-04-07 重庆师范大学 Machine learning identification method based on deep learning
CN108229693B (en) * 2018-02-08 2020-04-07 徐传运 Machine learning identification device and method based on comparison learning
CN108345943B (en) * 2018-02-08 2020-04-07 重庆理工大学 Machine learning identification method based on embedded coding and contrast learning
CN108345942B (en) * 2018-02-08 2020-04-07 重庆理工大学 Machine learning identification method based on embedded code learning
CN108629049A (en) * 2018-05-14 2018-10-09 芜湖岭上信息科技有限公司 A kind of image real-time storage and lookup device and method based on hash algorithm
CN109947936B (en) * 2018-08-21 2021-03-02 北京大学 Method for dynamically detecting junk mails based on machine learning
CN110083762B (en) * 2019-03-15 2023-01-24 平安科技（深圳）有限公司 Room source searching method, device and equipment and computer readable storage medium
CN110674333B (en) * 2019-08-02 2022-04-01 杭州电子科技大学 Large-scale image high-speed retrieval method based on multi-view enhanced depth hashing
CN112699676B (en) * 2020-12-31 2024-04-12 中国农业银行股份有限公司 Address similarity relation generation method and device
CN112784838A (en) * 2021-01-28 2021-05-11 佛山市南海区广工大数控装备协同创新研究院 Hamming OCR recognition method based on locality sensitive hashing network
CN113515450A (en) * 2021-05-20 2021-10-19 广东工业大学 Environment anomaly detection method and system
CN113393107B (en) * 2021-06-07 2022-08-12 东方电气集团科学技术研究院有限公司 Incremental calculation method for state parameter reference value of power generation equipment
CN113449849B (en) * 2021-06-29 2022-05-27 桂林电子科技大学 Learning type text hash method based on self-encoder
CN113495886A (en) * 2021-09-07 2021-10-12 上海观安信息技术股份有限公司 Method and device for detecting pollution sample data for model training
CN114781642B (en) * 2022-06-17 2022-09-23 之江实验室 Cross-media corresponding knowledge generation method and device

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7860887B2 (en) * 2007-02-20 2010-12-28 The Invention Science Fund I, Llc Cross-media storage coordination
CN104199827B (en) * 2014-07-24 2017-08-04 北京大学 The high dimensional indexing method of large scale multimedia data based on local sensitivity Hash
CN104346440B (en) * 2014-10-10 2017-06-23 浙江大学 A kind of across media hash indexing methods based on neutral net
CN106202413B (en) * 2016-07-11 2018-11-20 北京大学深圳研究生院 A kind of cross-media retrieval method
CN106227851B (en) * 2016-07-29 2019-10-01 汤一平 The image search method of depth of seam division search based on depth convolutional neural networks

Also Published As

Publication number Publication date
CN106649715A (en) 2017-05-10

Similar Documents

Publication Publication Date Title
CN106649715B (en) A kind of cross-media retrieval method based on local sensitivity hash algorithm and neural network
Patil et al. Convolutional neural networks: an overview and its applications in pattern recognition
CN109299284B (en) Knowledge graph representation learning method based on structural information and text description
CN107832353B (en) False information identification method for social media platform
CN104899253B (en) Towards the society image across modality images-label degree of correlation learning method
CN109271539B (en) Image automatic labeling method and device based on deep learning
CN112905822B (en) Deep supervision cross-modal counterwork learning method based on attention mechanism
CN108959522B (en) Migration retrieval method based on semi-supervised countermeasure generation network
CN110647904A (en) Cross-modal retrieval method and system based on unmarked data migration
CN113761208A (en) Scientific and technological innovation information classification method and storage device based on knowledge graph
CN113065409A (en) Unsupervised pedestrian re-identification method based on camera distribution difference alignment constraint
CN112182275A (en) Trademark approximate retrieval system and method based on multi-dimensional feature fusion
CN111611395B (en) Entity relationship identification method and device
CN116578738B (en) Graph-text retrieval method and device based on graph attention and generating countermeasure network
Lv et al. Retrieval oriented deep feature learning with complementary supervision mining
Chen et al. Two-stream hierarchical similarity reasoning for image-text matching
CN117077071A (en) Data analysis method and system based on data classification
Wang et al. Learning based neural similarity metrics for multimedia data mining
CN116561272A (en) Open domain visual language question-answering method and device, electronic equipment and storage medium
Saito et al. Demian: Deep modality invariant adversarial network
CN116227486A (en) Emotion analysis method based on retrieval and contrast learning
Wang et al. Multilevel metric rank match for person re-identification
CN111723301B (en) Attention relation identification and labeling method based on hierarchical theme preference semantic matrix
Shafi et al. Urdu character recognition: A systematic literature review
CN114329064A (en) Video processing method, video processing device, computer equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant