CN108228757A - Image search method and device, electronic equipment, storage medium, program - Google Patents

Image search method and device, electronic equipment, storage medium, program Download PDF

Info

Publication number
CN108228757A
CN108228757A CN201711402436.3A CN201711402436A CN108228757A CN 108228757 A CN108228757 A CN 108228757A CN 201711402436 A CN201711402436 A CN 201711402436A CN 108228757 A CN108228757 A CN 108228757A
Authority
CN
China
Prior art keywords
image
word
pending
network
sample
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201711402436.3A
Other languages
Chinese (zh)
Inventor
张韵璇
李�诚
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Sensetime Technology Development Co Ltd
Original Assignee
Beijing Sensetime Technology Development Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Sensetime Technology Development Co Ltd filed Critical Beijing Sensetime Technology Development Co Ltd
Priority to CN201711402436.3A priority Critical patent/CN108228757A/en
Publication of CN108228757A publication Critical patent/CN108228757A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
    • G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • G06F16/5866—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computational Linguistics (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Biology (AREA)
  • Library & Information Science (AREA)
  • Databases & Information Systems (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Image Analysis (AREA)

Abstract

The embodiment of the invention discloses a kind of image search method and device, electronic equipment, storage medium, program, wherein method includes：It handles receiving pending data, obtains corresponding data characteristics；Using network is characterized, map operation is performed to the diagram data feature, obtains corresponding to the mappings characteristics of the data characteristics；Corresponding image is obtained as target image from the image set to be checked based on the mappings characteristics, and the image set to be checked includes at least one image.The above embodiment of the present invention is mapped by characterizing network, the characteristic information of image and word can be characterized simultaneously, and passing through the similarity between the distance between mappings characteristics characterize data, the similarity based on the feature after mapping is searched, you can is realized and is searched figure and with word to scheme to search figure.

Description

Image search method and device, electronic equipment, storage medium, program
Technical field
The present invention relates to computer vision technique, especially a kind of image search method and device, electronic equipment, storage are situated between Matter, program.
Background technology
The modeling of picture classification, picture searching be computer vision object identification, scene Recognition field major issue, There is important application in many fields.Wherein, picture classification can be used for knowledge mapping, the structure of scene system, realization " memory photograph album " Wait the function of hommizations.Picture searching is roughly divided into according to word search (i.e.：A word is inputted, searches for associated picture) With according to picture searching (i.e.：A picture is inputted, exports similar picture) etc..
However the modeling of picture classification, picture searching is sufficiently complex.
For picture classification model, first, classification present in reality is excessive, major class such as animal, plant, article, group Such as people, cat, dog, tree, flower, vehicle；In addition, same pictures often possess multiple classifications, for example, a dry goods runs quickly on grassland It runs, just contains multiple information such as horse, grassland, sky simultaneously.Above-mentioned reason can increase the training difficulty of picture classification model and And reduce the accuracy rate of picture classification.
For picture searching model, the problem of classification total amount present in similary reality is excessive；And for different visitors Family, the classification of demand are also not quite similar, and there are more picture may be rarer classification in certain client's photograph albums, this A little classifications can not possibly be completely covered substantially when search model is trained.Above-mentioned reason can increase the instruction of picture searching model Practice difficulty and reduce the accuracy rate of picture searching.
Invention content
An embodiment of the present invention provides a kind of picture search technologies.
Image search method provided in an embodiment of the present invention, which is characterized in that including：
It handles receiving pending data, obtains corresponding data characteristics, the pending data includes：It is pending Image and/or pending word；
Using network is characterized, map operation is performed to the data characteristics, the mapping for obtaining corresponding to the data characteristics is special Sign；Mappings characteristics described in word are used to characterize the characteristic information of image and word, and pass through the distance between mappings characteristics simultaneously Similarity between characterize data；
Corresponding image is obtained as target image, the image to be checked from image set to be checked based on the mappings characteristics Concentration includes at least one image.
In another embodiment based on the above method of the present invention, the pending data of described pair of reception is handled, Corresponding data characteristics is obtained, including：
Using convolutional neural networks, pending image is handled, the image for obtaining corresponding to the pending image is special Sign；
And/or using natural language processing network, pending word is handled, obtains corresponding to the pending text The character features of word.
It is described to utilize natural language processing network in another embodiment based on the above method of the present invention, treat place Reason word is handled, and obtains corresponding to the character features of the pending word, including：
Pending word is inputted into natural language processing network, it will by the input layer in the natural language processing network Sample word decomposes and is converted into one-hot encoding；
The one-hot encoding obtains corresponding to the predictive text feature vector of the sample word by hidden layer and output layer.
It is described to utilize natural language processing network in another embodiment based on the above method of the present invention, treat place Reason word is handled, and before obtaining the character features of the corresponding pending word, is further included：
The predictive text feature vector of sample word is obtained based on natural language processing network, based on the sample word Predictive text feature vector is trained the natural language processing network.
In another embodiment based on the above method of the present invention, the predictive text based on the sample word is special Sign vector is trained the natural language processing network, including：
Predictive text feature vector based on the sample word, the one-hot encoding for inputting hidden layer and hidden layer parameter calculate logarithm Posterior probability maximizes log posterior probability by adjusting hidden layer parameter；
It brings the hidden layer parameter for prolonging probability after the correspondence maximization logarithm into the natural language processing network, is instructed Natural language processing network after white silk.
It is described to be based on the mappings characteristics from image set to be checked in another embodiment based on the above method of the present invention It is middle obtain corresponding image as target image before, further include：
Using the characterization network, characteristics of image to be checked corresponding to image set to be checked performs map operation, is corresponded to The target signature collection of the characteristics of image to be checked.
In another embodiment based on the above method of the present invention, the image to be checked corresponding to image set to be checked is special Before sign performs map operation, further include：
At least one image to be checked is obtained from image library, based on image set to be checked described in the image construction to be checked；
Using neural network, feature extraction operation is carried out respectively to all images to be checked in image set to be checked, is obtained pair Answer the characteristics of image to be checked of the image to be checked.
It is described to be based on the mappings characteristics from image set to be checked in another embodiment based on the above method of the present invention It is middle to obtain corresponding image as target image, including：
It concentrates to search from target signature based on the mappings characteristics and obtains at least one target signature；The target signature pair Answer image set to be checked；
Corresponding image is obtained as target image from the image set to be checked based on the target signature.
It is described to be based on the mappings characteristics from target signature collection in another embodiment based on the above method of the present invention Middle lookup obtains at least one target signature, including：
It concentrates to search in the target signature based on the mappings characteristics and is less than predetermined threshold value with mappings characteristics distance Target signature, obtain at least one target signature.
It is described using network is characterized in another embodiment based on the above method of the present invention, to the data characteristics Before performing map operation, further include：
The characterization network is trained based on sample data；The sample data includes sample image and sample word, described Sample image and the sample word are labeled with mark class label respectively.
It is described that the characterization net is trained based on sample data in another embodiment based on the above method of the present invention Network, including：
Using sample image and sample word as pending image and pending word；
The corresponding sample character features of sample word are obtained based on natural language processing network, based on sample word spy Sign obtains corresponding word mappings characteristics through characterizing network, and each word mappings characteristics that distance is less than to preset value are aggregated to In one mapping set；
The image feature vector of sample image is obtained based on convolutional neural networks, based on described in described image feature vector warp It characterizes network and obtains corresponding image mappings characteristics；
The word mappings characteristics of the minimum preset quantity of distance are searched in the mapping set, based on obtained each word Mappings characteristics obtain corresponding to the prediction class label of the preset quantity of the sample image；
Prediction class label and mark class label based on the sample image are trained the characterization network.
It is described to be obtained based on obtained each word mappings characteristics in another embodiment based on the above method of the present invention The prediction classification of the preset quantity of the corresponding sample image, including：
Corresponding sample word is obtained based on obtained each word mappings characteristics, is obtained by the obtained sample word Semantic mark class label, the mark class label form the prediction classification of the corresponding sample image.
In another embodiment based on the above method of the present invention, the prediction classification based on the sample image and Mark class label is trained the characterization network, including：
The prediction classification of the sample image with mark class label is matched, matching degree is greater than or equal to default The corresponding character features vector sum described image feature vector of prediction classification of matching degree is saved in same in the characterization network In set；By matching degree be less than preset matching degree the corresponding character features vector of prediction classification not with described image feature vector It is saved in the characterization network in identity set；
Distance in the characterization network is made to be stored in a set less than each described eigenvector of preset value, the spy Sign vector includes image feature vector and character features vector.
It is described to obtain sample image based on convolutional neural networks in another embodiment based on the above method of the present invention Image feature vector before, further include：
Based on convolutional neural networks obtain sample image prognostic chart picture feature vector, obtain the prognostic chart picture feature to More than one corresponding prediction classification is measured, prediction classification and mark class label based on the sample image train the convolution Neural network.
The other side of the embodiment of the present invention provides a kind of image search apparatus, including：
Processing unit for handling receiving pending data, obtains corresponding data characteristics, the pending number According to including pending image and/or pending word；
Map unit for using network is characterized, map operation to be performed to the data characteristics, obtains corresponding to the data The mappings characteristics of feature；The mappings characteristics are used for while characterize the characteristic information of image and word, and pass through mappings characteristics The distance between similarity between characterize data；
Search unit obtains corresponding image as target figure for being based on the mappings characteristics from image set to be checked Picture, the image set to be checked include at least one image.
In another embodiment based on above device of the present invention, the processing unit, including：
Image processing module for utilizing convolutional neural networks, is handled pending image, corresponded to described in treat Handle the characteristics of image of image；
Language processing module for utilizing natural language processing network, is handled pending word, obtains corresponding institute State the character features of pending word.
In another embodiment based on above device of the present invention, the word processing module, specifically for that will wait to locate Word input natural language processing network is managed, is decomposed sample word simultaneously by the input layer in the natural language processing network It is converted into one-hot encoding；The one-hot encoding by hidden layer and output layer obtain correspond to the sample word predictive text feature to Amount.
In another embodiment based on above device of the present invention, the processing unit further includes：
Linguistic network training module, for based on natural language processing network obtain sample word predictive text feature to Amount, the predictive text feature vector based on the sample word are trained the natural language processing network.
In another embodiment based on above device of the present invention, the linguistic network training module, specifically for base The one-hot encoding and hidden layer parameter of predictive text feature vector, input hidden layer in the sample word calculate log posterior probability, Log posterior probability is maximized by adjusting hidden layer parameter；The hidden layer parameter band of probability will be prolonged after the correspondence maximization logarithm Enter the natural language processing network, the natural language processing network after being trained.
In another embodiment based on above device of the present invention, further include：
Compound mapping unit, for utilizing the characterization network, characteristics of image to be checked corresponding to image set to be checked performs Map operation obtains corresponding to the target signature collection of the characteristics of image to be checked.
In another embodiment based on above device of the present invention, further include：
Gather acquiring unit, for obtaining at least one image to be checked from image library, based on the image construction to be checked The image set to be checked；
Image set processing unit for utilizing neural network, carries out all images to be checked in image set to be checked respectively Feature extraction operation obtains corresponding to the characteristics of image to be checked of the image to be checked.
In another embodiment based on above device of the present invention, described search unit, including：
Target searching module concentrates lookup to obtain at least one target spy for being based on the mappings characteristics from target signature Sign；The target signature corresponds to image set to be checked；
Target Acquisition module obtains corresponding image conduct for being based on the target signature from the image set to be checked Target image.
In another embodiment based on above device of the present invention, the target searching module, specifically for being based on State mappings characteristics concentrates lookup, less than the target signature of predetermined threshold value, to be obtained with mappings characteristics distance in the target signature More than one target signature.
In another embodiment based on above device of the present invention, further include：
Training unit trains the characterization network for being based on sample data；The sample data include sample image and Sample word, the sample image and the sample word are labeled with mark class label respectively.
In another embodiment based on above device of the present invention, the training unit, including：Characteristic aggregation module, For using sample image and sample word as pending image and pending word；The sample image and sample word difference With mark class label；The corresponding sample character features of sample word are obtained based on natural language processing network, based on described Sample character features obtain corresponding word mappings characteristics through characterizing network, and each word that distance is less than to preset value maps In characteristic aggregation a to mapping set；
Sample mapping block, for obtaining the image feature vector of sample image based on convolutional neural networks, based on described Image feature vector obtains corresponding image mappings characteristics through the characterization network；
Tag Estimation module, the word for searching the minimum preset quantity of distance in the mapping set map special Sign obtains corresponding to the prediction class label of the preset quantity of the sample image based on obtained each word mappings characteristics；
Network training module is characterized, for the prediction class label based on the sample image and mark class label to institute Characterization network is stated to be trained.
In another embodiment based on above device of the present invention, the Tag Estimation module, specifically for being based on Each word mappings characteristics arrived obtain corresponding sample word, and semantic mark classification is obtained by the obtained sample word Label, the mark class label form the prediction classification of the corresponding sample image.
In another embodiment based on above device of the present invention, the characterization network training module, specifically for inciting somebody to action The prediction classification of the sample image is matched with mark class label, and matching degree is greater than or equal to the pre- of preset matching degree The corresponding character features vector sum described image feature vector of classification is surveyed to be saved in the characterization network in identity set；General It is not saved in the corresponding character features vector of prediction classification spent less than preset matching degree with described image feature vector described It characterizes in network in identity set；
Distance in the characterization network is made to be stored in a set less than each described eigenvector of preset value, the spy Sign vector includes image feature vector and character features vector.
In another embodiment based on above device of the present invention, further include：
Neural metwork training unit, for obtaining the prognostic chart picture feature vector of sample image based on convolutional neural networks, Obtain more than one corresponding prediction classification of the prognostic chart picture feature vector, prediction classification and mark based on the sample image It notes class label and trains the convolutional neural networks.
Other side according to embodiments of the present invention, a kind of electronic equipment provided, including processor, the processor Including image search apparatus as described above.
Other side according to embodiments of the present invention, a kind of electronic equipment provided, including：Memory, for storing Executable instruction；
And processor, it completes to scheme as described above to perform the executable instruction for communicating with the memory As the operation of searching method.
Other side according to embodiments of the present invention, a kind of computer storage media provided, for storing computer The instruction that can be read, described instruction are performed the operation for performing image search method as described above.
Other side according to embodiments of the present invention, a kind of computer program provided, including computer-readable code, It is characterized in that, when the computer-readable code in equipment when running, the processor execution in the equipment is used to implement The instruction of each step in image search method as described above.
A kind of image search method and device, electronic equipment, storage medium, journey provided based on the above embodiment of the present invention Sequence handles receiving pending data, obtains corresponding data characteristics；Using network is characterized, data characteristics execution is reflected Operation is penetrated, obtains the mappings characteristics of corresponding data feature；Corresponding image is obtained based on mappings characteristics from image set to be checked to make For target image；By characterizing the mapping of network, the characteristic information of image and word can be characterized simultaneously, and pass through mappings characteristics The distance between similarity between characterize data, the similarity based on the feature after mapping searched, you can realize with word Search figure and to scheme to search figure；And the picture search of unlimited word is realized, that is, is directed in advance without the word estimated, it can also Carry out picture search；Overcome the drawbacks of search of prior art image needs to limit a specific set of words.
Below by drawings and examples, technical scheme of the present invention is described in further detail.
Description of the drawings
The attached drawing of a part for constitution instruction describes the embodiment of the present invention, and is used to explain together with description The principle of the present invention.
With reference to attached drawing, according to following detailed description, the present invention can be more clearly understood, wherein：
Fig. 1 is the flow chart of image search method one embodiment of the present invention.
Fig. 2 is the structure diagram of one embodiment of natural language processing network.
Fig. 3 is the structure diagram of image search apparatus one embodiment of the present invention.
Fig. 4 is the structure diagram for realizing the terminal device of the embodiment of the present application or the electronic equipment of server.
Specific embodiment
Carry out the various exemplary embodiments of detailed description of the present invention now with reference to attached drawing.It should be noted that：Unless in addition have Body illustrates that the unlimited system of component and the positioned opposite of step, numerical expression and the numerical value otherwise illustrated in these embodiments is originally The range of invention.
Simultaneously, it should be appreciated that for ease of description, the size of the various pieces shown in attached drawing is not according to reality Proportionate relationship draw.
It is illustrative to the description only actually of at least one exemplary embodiment below, is never used as to the present invention And its application or any restrictions that use.
Technology, method and apparatus known to person of ordinary skill in the relevant may be not discussed in detail, but suitable In the case of, the technology, method and apparatus should be considered as part of specification.
It should be noted that：Similar label and letter represents similar terms in following attached drawing, therefore, once a certain Xiang Yi It is defined in a attached drawing, then in subsequent attached drawing does not need to that it is further discussed.
The embodiment of the present invention can be applied to the electronic equipments such as terminal device, computer system/server, can with it is numerous Other general or specialized computing system environments or configuration operate together.Suitable for electric with terminal device, computer system/server etc. The example of well-known terminal device, computing system, environment and/or configuration that sub- equipment is used together includes but not limited to： Personal computer system, server computer system, thin client, thick client computer, hand-held or laptop devices, based on microprocessor System, set-top box, programmable consumer electronics, NetPC Network PC, little types Ji calculate machine Xi Tong ﹑ large computer systems and Distributed cloud computing technology environment including any of the above described system, etc..
The electronic equipments such as terminal device, computer system/server can be in the department of computer science performed by computer system It is described under the general linguistic context of system executable instruction (such as program module).In general, program module can include routine, program, mesh Beacon course sequence, component, logic, data structure etc., they perform specific task or realize specific abstract data type.Meter Calculation machine systems/servers can be implemented in distributed cloud computing environment, and in distributed cloud computing environment, task is by by logical What the remote processing devices of communication network link performed.In distributed cloud computing environment, program module can be located at and include storage On the Local or Remote computing system storage medium of equipment.
At present, picture classification method mainly includes arest neighbors classification, Naive Bayes Classification, support vector machines and is based on Picture classification category of model of deep learning etc..Wherein, the picture classification model based on deep learning is for example including autocoding Device (Auto-encoder), limited Boltzmann machine (Restricted Boltzmann Machine, RBM), depth conviction net Network (Deep Belief Nets, DBN), convolutional neural networks (Convolutional Neural Networks, CNN), biology Heuristic models etc., these disaggregated models how many, picture class number can all be influenced by picture number when training.
Image searching method mainly includes perceiving hash algorithm (Perceptual hash algorithm) and word packet model (Bag-of-Words algorithms.Wherein, it is a kind of to scheme to search the model of figure to perceive hash algorithm, to every pictures generation one A fingerprint (fingerprint) character string, then the fingerprint of more different pictures, as a result closer, just illustrates that picture is more similar. Word packet model is a kind of model that figure is searched with word, first with the algorithm (Scale-invariant of detection local feature Feature transform, SIFT) extract SIFT feature respectively to multiple image, then by clustering algorithm to the whole of extraction A SIFT feature is clustered to obtain k cluster centre as vision word table, finally to each image, using word list as specification Distance of each word, nearest+1, can obtain during it is calculated each SIFT feature of the width image with word list Picture is converted for the combination of word, figure is searched so as to fulfill with word by the code book of the width image.But this method thinks word Between be independent, there is no consider semantic influence.In the implementation of the present invention, inventors discovered through research that, Above-mentioned perception hash algorithm and word packet model algorithm emphasize particularly on different fields, the former is according to picture removal search similar pictures, and the latter is root According to keyword search picture, unknown classification is not all solved the problems, such as well；It can not realize and be simultaneously scanned for word or image Image.
Fig. 1 is the flow chart of image search method one embodiment of the present invention.As shown in Figure 1, the embodiment method includes：
Step 101, the pending data of reception is handled, obtains corresponding data characteristics.
Wherein, pending data includes pending image and/or pending word.
Specifically, pending image can be individually received, is realized with picture search image；Or pending word is individually received, It realizes with text search image；Pending image and pending word can also be received simultaneously, while are realized with picture search figure Picture and with text search image；Before picture search, can pending figure be based on by neural network or other technologies means As obtaining characteristics of image, character features are obtained based on pending word.
Step 102, using network is characterized, map operation is performed to data characteristics, the mapping for obtaining corresponding data feature is special Sign.
Wherein, mappings characteristics for characterizing the characteristic information of image and word simultaneously, and pass through between mappings characteristics Similarity between characterize data；Image and word can be mapped to same characterization space by characterization network, in the table It is image or word that it is corresponding, which not differentiate between mappings characteristics, in sign space, according between mappings characteristics between all mappings characteristics Distance is (such as：Euclidean distance, Ming Shi distances, mahalanobis distance between feature vector etc.) classification, distance is less than the mapping of preset value There are there are similarity relations between similarity relation or image and word between the corresponding image of feature and image.
Step 103, corresponding image is obtained as target image from image set to be checked based on mappings characteristics.
Wherein, image set to be checked includes at least one image.
Based on a kind of image search method that the above embodiment of the present invention provides, handle receiving pending data, Obtain corresponding data characteristics；Using network is characterized, map operation is performed to data characteristics, obtains the mapping of corresponding data feature Feature；Corresponding image is obtained as target image from image set to be checked based on mappings characteristics；By characterizing the mapping of network, The characteristic information of image and word can be characterized simultaneously, and pass through similar between the distance between mappings characteristics characterize data Degree, the similarity based on the feature after mapping are searched, you can are realized and searched figure and with word to scheme to search figure；And it realizes unlimited The picture search of word is directed in advance without the word estimated, can also carry out picture search；Overcome prior art diagram The drawbacks of needing to limit a specific set of words as search.
The characteristics of based on characterization network, after obtaining matching characteristic vector corresponding with target text or target image, Can more than one search image be searched in the image set to be checked including more than one image by vector distance, so as to Realize that the picture classification for supporting a large amount of labels and the picture searching function of supporting unknown classification, wherein picture searching function are divided into Figure is searched and to scheme to search figure with word, a pictures can be matched with multiple search terms, realize the search of unknown classification.
Another embodiment of image search method of the present invention, on the basis of the various embodiments described above, operation 101 includes：
Using convolutional neural networks, pending image is handled, obtains corresponding to the characteristics of image of pending image；
And/or using natural language processing network, pending word is handled, obtains corresponding to pending word Character features.
In the present embodiment, can be by convolutional neural networks for the processing of image, it can also be real by other means Existing, the present embodiment only provides a kind of realization method, in order to those skilled in the art understand that and realizing；And for the place of word Reason belongs to technological means more advanced at present, but the processing of word can equally be led to using natural language processing network Other modes realization is crossed, the present embodiment only provides a kind of realization method, in order to those skilled in the art understand that and realizing；This reality Unified with nature Language Processing and a convolutional neural networks are applied, writings and image is mapped to same characterization network, characterizes network As the manifold of sample character set and sample graph image set, the picture search function by characterizing network is realized.
In a specific example of image search method the various embodiments described above of the present invention, natural language processing net is utilized Network handles pending word, obtains corresponding to the character features of pending word, including：
Pending word is inputted into natural language processing network, by the input layer in natural language processing network by sample Word decomposes and is converted into one-hot encoding；
One-hot encoding obtains the predictive text feature vector of corresponding sample word by hidden layer and output layer.
In the present embodiment, the net that structure includes input layer-hidden layer-output layer can be used in natural language processing network Network, the technology for being directed to a core are according to word frequency Huffman Huffman encodings so that the similar word of all word frequency is hidden The content for hiding layer activation is basically identical, and the higher word of the frequency of occurrences, the hiding number of layers that they activate is fewer, effective in this way Reduce the complexity of calculating.It is distributed with latent semantic analysis (Latent Semantic Index, LSI), potential Di Li Crays The classical processes of (Latent Dirichlet Allocation, LDA) are compared, and the context of word is utilized in Word2vec, semantic Information more is enriched and (Mikolov the article pointed out that the unit version of an optimization can train for one day in opinion with high efficiency Hundred billion words)；In addition, natural language processing technique possesses bilingual property, you can the term vector in different language is mapped to In one shared space, multilingual search is realized.
Fig. 2 is the structure diagram of one embodiment of natural language processing network.As shown in Fig. 2, input layer is by word It decomposes (" white ", " small ", " horse " etc.), is converted into and is encoded into one-hot (one-hot encoding, heat encode, and the thing in computer is all It is 01 expression, that is, binary system, one-hot is a kind of naturally to select) vector x of formIk(1 there are one only, Remaining is all 0) the row vector v of matrix W (VxN) (representing input layer to the weight matrix of hidden layer)wRepresent hidden layer variable and input layer Contact, W ' (NxV) is weight matrix of the hidden layer H to output layer, v 'wjIt is the jth row of W ', output layer is classification space, each A yj=wj|wIIt represents in input wIThe probability of lower output classification j, wjRepresent the classification of output, wIRepresent the information of input, | symbol Number represent conditional probability, wj|wIIt represents in given input wIUnder conditions of picture belong to the probability of classification j, i.e. yj.Output to Measure y=(y1,…,yj,…yV) represent the probability distribution that picture belongs to 1-V classes, the pass of each parameter and input and output in network System is as follows：
Wherein, T：Transposition；C：Constant term；h：The parameter of hidden layer；ujIt is the output (in real number field) of network, the meaning of exp It is yjIt normalizes between 0-1, so as to allow yjRepresent probability.
In a specific example of image search method the various embodiments described above of the present invention, natural language processing net is utilized Network handles pending word, before obtaining the character features of corresponding pending word, further includes：
The predictive text feature vector of sample word, the prediction based on sample word are obtained based on natural language processing network Character features vector is trained natural language processing network.
Using natural language processing technique, by the corresponding feature vector deposit characterization network of word.word embedding Word is embedded in, i.e., word words is mapped to the vector that dimension is d, then with cosine COS distances, the Euclidean distance between vector Or inner product describes the syntax and semantic similarity between word and word.Wherein Word2vec (word DUAL PROBLEMS OF VECTOR MAPPING) is to utilize depth The thought of study, can by training, the vector operation being reduced to the processing of content of text in V dimensional vector spaces, and to Similarity on quantity space can be used for representing the similarity on text semantic.The term vector of Word2vec outputs can be used to Many NLP (Neuro-Linguistic Programming, neural language rule) relevant work is done, for example clusters, look for together Adopted word, part of speech analysis etc..
In a specific example of image search method the various embodiments described above of the present invention, the prediction text based on sample word Word feature vector is trained natural language processing network, including：
Predictive text feature vector based on sample word, the one-hot encoding for inputting hidden layer and hidden layer parameter calculate log posterior Probability maximizes log posterior probability by adjusting hidden layer parameter；
Corresponding maximize is prolonged the hidden layer parameter of probability after logarithm and bring natural language processing network into, oneself after train Right Language Processing network.
In the present embodiment, network training target for maximize log posterior probability maxlogP (y | ωI) it is the network optimization Target, P (y | ωI) it is posterior probability, in addition log (logarithm) is that (this is a kind of very common processing for processing for convenience Method).Posterior probability is bigger, it is meant that under conditions of input, the type of picture is bigger according to the confidence level that y is distributed.) specific The formula for calculating posterior probability is as follows：
Wherein, ωO：The vector of output, actually y, o represent output.ωI：The information of input, I represent input.h： The parameter of hidden layer；ujIt is the output (in real number field) of network, exp is meant that a yjIt normalizes between 0-1, so as to allow yjGeneration Table probability.
In another embodiment of image search method of the present invention, on the basis of the various embodiments described above, it can also include：
Using characterizing network, characteristics of image to be checked corresponding to image set to be checked performs map operation, corresponded to described in The target signature collection of characteristics of image to be checked.
Specifically, images all in image set to be checked are performed into map operation by characterizing network, realized in characterization space The middle multiple mappings characteristics for obtaining corresponding image set to be checked, the process of the acquisition target signature collection can perform operation 103 Before, it can also be and operation performed to the characteristics of image to be checked in image set to be checked after image set to be checked is obtained, that is, exist Map operation is carried out to image set to be checked before operation 101 and operation 102.
Optionally, image set to be checked can be a database, and it is not absolutely required to perform spy in real time for this database Extraction operation is levied, this database can also prestore the target signature collection extracted by characterizing network.
It is corresponding to image set to be checked to treat in a specific example of image search method the various embodiments described above of the present invention It looks into before characteristics of image execution map operation, further includes：
At least one image to be checked is obtained from image library, based on image set to be checked described in image construction to be checked；
Using neural network, feature extraction operation is carried out respectively to all images to be checked in image set to be checked, is obtained pair Answer the characteristics of image to be checked of image to be checked.
The image set to be checked obtained in the present embodiment can derive from network or from local image library, such as：It will use Part or all in the mobile phone photo album (local image library) at family is as image set to be checked, in image set to be checked by characterization net Firstly the need of characteristics of image to be checked is obtained before network mapping, the present embodiment is obtained using neural network, based on characteristics of image to be checked Corresponding mappings characteristics can be just obtained, and then realize picture search.
In specific example, searching figure step with word can be divided into：A. word is inputted into natural language processing network, obtains characterization net Vector in network.B. the photo in photograph album is inputted into CNN, feature vector closely located (threshold value obtains in the training process) is made For search result.
Target Photo is inputted CNN convolutional neural networks by the step of to scheme to search figure as a., obtains the vector in characterization network. B. the photo in photograph album is inputted into CNN, feature vector closely located (threshold value obtains in the training process) is as search result.
In a specific example of image search method the various embodiments described above of the present invention, operation 103 includes：
It concentrates to search from target signature based on mappings characteristics and obtains at least one target signature；Target signature corresponds to figure to be checked Image set；
Corresponding image is obtained as target image from image set to be checked based on target signature.
Wherein, mappings characteristics and target signature are all to characterize the feature that network mapping obtains, and are found based on mappings characteristics Corresponding target signature, you can determine corresponding target image in image set to be checked.
It is special from target based on mappings characteristics in a specific example of image search method the various embodiments described above of the present invention It is searched in collection and obtains at least one target signature, including：
It is concentrated based on mappings characteristics in target signature and searches the target signature for being less than predetermined threshold value with mappings characteristics distance, obtained To at least one target signature.
In the present embodiment, target signature collection and pending image and/or word are mapped to table using characterizing network It levies in space, in characterization space, maps to obtain by characteristics of image or character features since mappings characteristics are not differentiated between, because This, in space is characterized, need to only be searched based on the distance of mappings characteristics, you can obtain corresponding pending image and/or The image to be found of word is realized and searches figure or with word to scheme to search figure.
In a still further embodiment of image search method of the present invention, on the basis of the various embodiments described above, operation 102 it Before, it further includes：
Characterization network is trained based on sample data.
Wherein, sample data includes sample image and sample word, and sample image and sample word are labeled with marking respectively Class label.
In the present embodiment, in order to realize that image and word can be mapped in same characterization space by characterization network, and be led to The distance between the mappings characteristics crossed in characterization space judge the similarity between image and image or image and word, need pair Characterization network is trained.
In a specific example of image search method the various embodiments described above of the present invention, trained and characterized based on sample data Network, including：
Using sample image and sample word as pending image and pending word；Sample image and sample word difference With mark class label；
The corresponding sample character features of sample word are obtained based on natural language processing network, are passed through based on sample character features It characterizes network and obtains corresponding word mappings characteristics, each word mappings characteristics that distance is less than to preset value are aggregated to a mapping In set；
The image feature vector of sample image is obtained based on convolutional neural networks, based on image feature vector through characterizing network Obtain corresponding image mappings characteristics；
The word mappings characteristics of the minimum preset quantity of distance are searched in mapping set, based on obtained each word mapping Feature obtains the prediction class label of the preset quantity of corresponding sample image；
Prediction class label and mark class label based on sample image are trained characterization network.
In the present embodiment, the prediction classification of corresponding sample image is obtained by characterizing network mapping, based on preset quantity Prediction classification with mark class label search the distance between corresponding feature vector, feature based vector in network is characterized To characterization network be trained, make prediction class label with mark class label in space is characterized corresponding feature vector one In a set；Characterization network is trained by being based on sample word and sample image, enables the characterization network of acquisition by language The similar writings and image of justice is mapped in a set, to realize by being searched in mapping set, you can obtain corresponding diagram The image or word that the needs of picture or word are searched for.
In a specific example of image search method the various embodiments described above of the present invention, based on obtained each word mapping Feature obtains the prediction classification of the preset quantity of corresponding sample image, including：
Corresponding sample word is obtained based on obtained each word mappings characteristics, is obtained by obtained sample word semantic Mark class label, mark class label forms the prediction classification of corresponding sample image.
Wherein, the characterization space for characterizing network mapping is RV(R represents dimension, R to vector spaceVIt is that the vector that dimension is V is empty Between), manifold of the characterization space as sample character set and sample graph image set, manifold is that part has Euclidean space property Space, manifold in mathematics for describing geometrical body, physically, the phase space of classical mechanics and construction general theory of relativity when The four-dimensional pseudo-Riemannian manifold of empty model is all the example of manifold；It therefore, can be by characterizing in space between two feature vectors Distance Judgment image and word the degree of correlation, realize word to image search.It is that writings and image is mapped to characterize network Same space has been arrived, for example the picture of " cat " this word and cat has been mapped to same vector, so either scheming figure Between, between word word or between figure word, the degree of correlation that can be transferred through vector inside characterization network judges whether unanimously.
In a specific example of image search method the various embodiments described above of the present invention, the prediction class based on sample image Characterization network is not trained with mark class label, including：
The prediction classification of sample image with mark class label is matched, matching degree is greater than or equal to preset matching The corresponding character features vector sum image feature vector of prediction classification of degree is saved in characterization network in identity set；It will matching Degree is not saved in characterization network less than the corresponding character features vector of prediction classification of preset matching degree with image feature vector In identity set；
Distance in characterization network is made to be stored in a set less than each feature vector of preset value, feature vector includes figure As feature vector and character features vector.
In the present embodiment, by training characterize network make in characterization space between each feature vector in each gathering away from From both less than preset value, therefore, more similar image can be found by the characterization network after training based on image or word Or word.
In a specific example of image search method the various embodiments described above of the present invention, obtained based on convolutional neural networks Before the image feature vector of sample image, further include：
The prognostic chart picture feature vector of sample image is obtained based on convolutional neural networks, obtains prognostic chart picture feature vector pair More than one the prediction classification answered, prediction classification and mark class label training convolutional neural networks based on sample image.
In the present embodiment, the convolutional neural networks of the image feature vector to obtaining sample image are trained, and are passed through The network that training obtains can more accurately obtain the prognostic chart picture feature vector of corresponding sample image, can be searched in network is characterized More accurately character features vector is obtained, ensure that the training effect to characterizing network；And specifically to convolutional neural networks Training, can be trained by the training method proposed in the prior art.
Such as：Mark class label and prediction class label based on sample image are calculated by loss function obtains convolution The error amount of neural network；
The parameter in convolutional neural networks is updated based on error amount；
Using the convolutional neural networks after undated parameter as convolutional neural networks, according to following methods iterative convolution nerve net Network：Feature based on extraction obtains prediction class label；Mark class label and prediction class label based on sample image lead to It crosses loss function and calculates the error amount for obtaining convolutional neural networks；Based on error amount by reversed gradient algorithm to convolutional Neural net Parameter in network is updated；Stop iteration until convolutional neural networks meet preset condition.
Wherein, the parameter in convolutional neural networks is updated based on error amount, can included：
The parameter in neural network is updated by reversed gradient algorithm based on error amount.
Preset condition includes following any one：
Loss function convergence, iterations reach preset times and error amount is less than preset value.
The image search method of the various embodiments described above proposed by the present invention has a variety of applications.It can be wrapped in specific example It includes：Using 1：Applied to picture classification, a large amount of classifications of convolutional neural networks model supports a, pictures can correspond to multiple marks Label.It can extend based on this and many functions related with picture classification, realize that different functions (such as will have in mobile phone The figure of Similar content merges).
Using 2：Applied to picture searching, in addition to the search of grader known class, for combination sort and unknown classification Search, can also (the characterization network of the diversified models of manifold, the distance in manifold can represent two by manifold The true similarity of object) in characterization range search go out the high picture of the degree of correlation.
The feature vector of closest predetermined number is found in network is characterized (such as：Preceding 5 vectors), these vectors Prediction classification of the corresponding classification as picture.For example, do not have in space " cat ", but have Tiger, then network is special Property be can allow " cat " map vector with Tiger from closely.It is similar with unknown classification " cat " " old so as to search out Tiger ".
User can be with fast search to certain pictures in mobile phone photo album.Since search model supports a large amount of classifications, unknown The search of classification, combination sort, it is ensured that search the very high picture of correlation.So the different need of different user can be met It asks, such as search " morningstar lily army horse-breeding farm ", grader did not contacted classification as morningstar lily army horse-breeding farm when training, but It is that " morningstar lily army horse-breeding farm " can be decomposed into thick classification known to several graders (using currently existing scheme, Penn by APP automatically TreeBank corpus, ICTCLAS corpus etc.) and auto-complete related category (being based on semantics recognition principle)：Horse, meadow, Sky (according to the classification that Category Relevance supplements, have in " horse " " stud-farm " in thick classification, in actual conditions, horse and meadow usual one Rise and occur, in the characterization network obtained by neural metwork training above, horse, the corresponding vector in meadow it is closer to the distance.So The auto-complete of classification can be realized by vector similar in selection), morningstar lily (Gansu place name).It is searched in network is characterized later Relevant picture.
It tags to picture, establishes " memory photograph album ", introduce face recognition technology, it can be by the photo of different people in photograph album It is clustered, generates someone exclusive photograph album, and face character label (expression, eye closing etc. of opening eyes), matching background are added for picture Label (if opening positioning function during picture shooting, can extract associated tag information) from location information, according to when Between arrange.It is formed " memory photograph album ".
Applied to logout, the information of a certain special scenes is taken together, abstract is formed, efficiency bar can be used as Tool.For example some meeting has been participated in recently, some pictures are had recorded in conference process, picture A PP can be automatically these photos Sort out (these pictures have identical feature, such as background information), it is convenient to inquire in the future.
Know figure function, do not know the picture of content for a user, can by model extraction characteristic information therein, And feature is converted into the textual annotation to picture, feed back to user.It may search for the picture of the same category simultaneously.
Natural language processing technique possesses bilingual property, by corpus (by identical information or the information of identical theme It is described with two or more language, and the set by manually or by computer building information between different language) with different Term vector in language is mapped in a shared space, realizes multilingual search.
One of ordinary skill in the art will appreciate that：Realizing all or part of step of above method embodiment can pass through The relevant hardware of program instruction is completed, and aforementioned program can be stored in a computer read/write memory medium, the program When being executed, step including the steps of the foregoing method embodiments is performed；And aforementioned storage medium includes：ROM, RAM, magnetic disc or light The various media that can store program code such as disk.
Fig. 3 is the structure diagram of image search apparatus one embodiment of the present invention.The device of the embodiment can be used for real The existing above-mentioned each method embodiment of the present invention.As shown in figure 3, the device of the embodiment includes：
Processing unit 31 is handled for the pending data to reception, obtains corresponding data characteristics.Wherein, it treats It handles data and includes pending image and/or pending word.
Map unit 32, for using network is characterized, performing map operation to data characteristics, obtaining corresponding data feature Mappings characteristics.
Wherein, mappings characteristics are for characterizing the characteristic information of image and word simultaneously, and pass through between mappings characteristics away from From the similarity between characterize data.
Search unit 33 obtains corresponding image as target image for being based on mappings characteristics from image set to be checked.
Wherein, image set to be checked includes at least one image.
Based on a kind of image search apparatus that the above embodiment of the present invention provides, handle receiving pending data, Obtain corresponding data characteristics；Using network is characterized, map operation is performed to data characteristics, obtains the mapping of corresponding data feature Feature；Corresponding image is obtained as target image from image set to be checked based on mappings characteristics；By characterizing the mapping of network, The characteristic information of image and word can be characterized simultaneously, and pass through similar between the distance between mappings characteristics characterize data Degree, the similarity based on the feature after mapping are searched, you can are realized and searched figure and with word to scheme to search figure；And it realizes unlimited The picture search of word is directed in advance without the word estimated, can also carry out picture search；Overcome prior art diagram The drawbacks of needing to limit a specific set of words as search.
Another embodiment of image search apparatus of the present invention, on the basis of the various embodiments described above, processing unit 31, packet It includes：
Image processing module for utilizing convolutional neural networks, is handled pending image, is obtained corresponding pending The characteristics of image of image；
Language processing module for utilizing natural language processing network, is handled pending word, is obtained correspondence and is treated Handle the character features of word.
In the present embodiment, can be by convolutional neural networks for the processing of image, it can also be real by other means Existing, the present embodiment only provides a kind of realization method, in order to those skilled in the art understand that and realizing；And for the place of word Reason belongs to technological means more advanced at present, but the processing of word can equally be led to using natural language processing network Other modes realization is crossed, the present embodiment only provides a kind of realization method, in order to those skilled in the art understand that and realizing；This reality Unified with nature Language Processing and a convolutional neural networks are applied, writings and image is mapped to same characterization network, characterizes network As the manifold of sample character set and sample graph image set, the picture search function by characterizing network is realized.
In a specific example of image search apparatus the various embodiments described above of the present invention, word processing module is specific to use In pending word is inputted natural language processing network, sample word is divided by the input layer in natural language processing network It solves and is converted into one-hot encoding；One-hot encoding obtains the predictive text feature vector of corresponding sample word by hidden layer and output layer.
In a specific example of image search apparatus the various embodiments described above of the present invention, processing unit 31 further includes：
Linguistic network training module, for based on natural language processing network obtain sample word predictive text feature to Amount, the predictive text feature vector based on sample word are trained natural language processing network.
In a specific example of image search apparatus the various embodiments described above of the present invention, linguistic network training module, tool Body is general for the predictive text feature vector based on sample word, the one-hot encoding of input hidden layer and hidden layer parameter calculating log posterior Rate maximizes log posterior probability by adjusting hidden layer parameter；The hidden layer parameter band of probability will be prolonged after corresponding maximization logarithm Enter natural language processing network, the natural language processing network after being trained.
In another embodiment of image search apparatus of the present invention, on the basis of the various embodiments described above, further include：
Compound mapping unit, for using network is characterized, characteristics of image to be checked corresponding to image set to be checked to perform mapping Operation obtains corresponding to the target signature collection of characteristics of image to be checked.
Specifically, images all in image set to be checked are performed into map operation by characterizing network, realized in characterization space The middle multiple mappings characteristics for obtaining corresponding image set to be checked, the process of the acquisition target signature collection can perform operation 103 Before, it can also be and operation performed to the characteristics of image to be checked in image set to be checked after image set to be checked is obtained, that is, exist Map operation is carried out to image set to be checked before operation 101 and operation 102.
In a specific example of image search apparatus the various embodiments described above of the present invention, further include：
Gather acquiring unit, it is to be checked based on image construction to be checked for obtaining at least one image to be checked from image library Image set；
Image set processing unit for utilizing neural network, carries out all images to be checked in image set to be checked respectively Feature extraction operation obtains corresponding to the characteristics of image to be checked of image to be checked.
In specific example, searching figure step with word can be divided into：A. word is inputted into natural language processing network, obtains characterization net Vector in network.B. the photo in photograph album is inputted into CNN, feature vector closely located (threshold value obtains in the training process) is made For search result.
Target Photo is inputted CNN convolutional neural networks by the step of to scheme to search figure as a., obtains the vector in characterization network. B. the photo in photograph album is inputted into CNN, feature vector closely located (threshold value obtains in the training process) is as search result.
In a specific example of image search apparatus the various embodiments described above of the present invention, search unit 33, including：
Target searching module concentrates lookup to obtain at least one target signature for being based on mappings characteristics from target signature； Target signature corresponds to image set to be checked；
Target Acquisition module obtains corresponding image as target figure for being based on target signature from image set to be checked Picture.
In a specific example of image search apparatus the various embodiments described above of the present invention, target searching module is specific to use The target signature for being less than predetermined threshold value with mappings characteristics distance is searched in being concentrated based on mappings characteristics in target signature, obtains one Above target signature.
In a still further embodiment of image search apparatus of the present invention, on the basis of the various embodiments described above, further include：
Training unit, for being based on sample data training characterization network.
Wherein, sample data includes sample image and sample word, and sample image and the sample word are labeled with respectively Mark class label.
In the present embodiment, in order to realize that image and word can be mapped in same characterization space by characterization network, and be led to The distance between the mappings characteristics crossed in characterization space judge the similarity between image and image or image and word, need pair Characterization network is trained.
In a specific example of image search apparatus the various embodiments described above of the present invention, training unit, including：
Characteristic aggregation module, for using sample image and sample word as pending image and pending word；Sample Image and sample word are respectively provided with mark class label；The corresponding sample of sample word is obtained based on natural language processing network Character features, corresponding word mappings characteristics are obtained based on sample character features through characterizing network, and distance is less than preset value Each word mappings characteristics are aggregated in a mapping set；
Sample mapping block, for obtaining the image feature vector of sample image based on convolutional neural networks, based on image Feature vector obtains corresponding image mappings characteristics through characterizing network；
Tag Estimation module, for searching the word mappings characteristics of the minimum preset quantity of distance, base in mapping set The prediction class label of the preset quantity of corresponding sample image is obtained in obtained each word mappings characteristics；
Network training module is characterized, for the prediction class label based on sample image and mark class label to characterizing net Network is trained.
In the present embodiment, the prediction classification of corresponding sample image is obtained by characterizing network mapping, based on preset quantity Prediction classification with mark class label search the distance between corresponding feature vector, feature based vector in network is characterized To characterization network be trained, make prediction class label with mark class label in space is characterized corresponding feature vector one In a set；Characterization network is trained by being based on sample word and sample image, enables the characterization network of acquisition by language The similar writings and image of justice is mapped in a set, to realize by being searched in mapping set, you can obtain corresponding diagram The image or word that the needs of picture or word are searched for.
In a specific example of image search apparatus the various embodiments described above of the present invention, Tag Estimation module is specific to use In obtaining corresponding sample word based on obtained each word mappings characteristics, semantic mark is obtained by obtained sample word Class label, mark class label form the prediction classification of corresponding sample image.
In a specific example of image search apparatus the various embodiments described above of the present invention, network training module, tool are characterized Body is used to match the prediction classification of sample image with mark class label, and matching degree is greater than or equal to preset matching degree The corresponding character features vector sum image feature vector of prediction classification be saved in characterization network in identity set；By matching degree Character features vector corresponding less than the prediction classification of preset matching degree is not saved in same in characterization network with image feature vector In one set；
Distance in characterization network is made to be stored in a set less than each feature vector of preset value, feature vector includes figure As feature vector and character features vector.
In a specific example of image search apparatus the various embodiments described above of the present invention, further include：
Neural metwork training unit, for obtaining the prognostic chart picture feature vector of sample image based on convolutional neural networks, Obtain more than one corresponding prediction classification of prognostic chart picture feature vector, prediction classification and mark classification mark based on sample image Sign training convolutional neural networks.
One side according to embodiments of the present invention, a kind of electronic equipment provided, including processor, processor includes this Invent the image search apparatus described in any of the above-described embodiment.
One side according to embodiments of the present invention, a kind of electronic equipment provided, including：Memory, can for storing Execute instruction；
And processor, for communicating to perform executable instruction image search method thereby completing the present invention with memory The operation of any of the above-described embodiment.
A kind of one side according to embodiments of the present invention, the computer storage media provided, can for storing computer The instruction of reading, described instruction are performed the operation for performing any of the above-described embodiment of image search method of the present invention.
A kind of one side according to embodiments of the present invention, the computer storage media provided, can for storing computer The instruction of reading, instruction are performed the operation for performing any of the above-described embodiment of image search method of the present invention.
Other side according to embodiments of the present invention, a kind of computer program provided, including computer-readable code, It is characterized in that, when computer-readable code in equipment when running, the processor execution in the equipment is used to implement the present invention The instruction of each step in any of the above-described embodiment of image search method.
The embodiment of the present invention additionally provides a kind of electronic equipment, such as can be mobile terminal, personal computer (PC), put down Plate computer, server etc..Below with reference to Fig. 4, it illustrates suitable for being used for realizing the terminal device of the embodiment of the present application or service The structure diagram of the electronic equipment 400 of device：As shown in figure 4, computer system 400 includes one or more processors, communication Portion etc., one or more of processors are for example：One or more central processing unit (CPU) 401 and/or one or more Image processor (GPU) 413 etc., processor can according to the executable instruction being stored in read-only memory (ROM) 402 or From the executable instruction that storage section 408 is loaded into random access storage device (RAM) 403 perform various appropriate actions and Processing.Communication unit 412 may include but be not limited to network interface card, and the network interface card may include but be not limited to IB (Infiniband) network interface card.
Processor can communicate with read-only memory 402 and/or random access storage device 430 to perform executable instruction, It is connected by bus 404 with communication unit 412 and is communicated through communication unit 412 with other target devices, is implemented so as to complete the application The corresponding operation of any one method that example provides for example, handling receiving pending image and/or pending word, obtains Obtain corresponding characteristics of image and/or character features；Using network is characterized, mapping behaviour is performed to characteristics of image and/or character features Make, obtain correspondence image feature and/or the mappings characteristics of character features；It is corresponding to image set to be checked to treat using characterizing network It looks into characteristics of image and performs map operation, obtain corresponding to the target signature collection of characteristics of image to be checked；It is special from target based on mappings characteristics It is searched in collection and obtains more than one target signature, corresponding target image is obtained based on target signature.
In addition, in RAM 403, it can also be stored with various programs and data needed for device operation.CPU401、ROM402 And RAM403 is connected with each other by bus 404.In the case where there is RAM403, ROM402 is optional module.RAM403 is stored Executable instruction is written in executable instruction into ROM402 at runtime, and it is above-mentioned logical that executable instruction performs processor 401 The corresponding operation of letter method.Input/output (I/O) interface 405 is also connected to bus 404.Communication unit 412 can be integrally disposed, It may be set to be with multiple submodule (such as multiple IB network interface cards), and in bus link.
I/O interfaces 405 are connected to lower component：Importation 406 including keyboard, mouse etc.；It is penetrated including such as cathode The output par, c 407 of spool (CRT), liquid crystal display (LCD) etc. and loud speaker etc.；Storage section 408 including hard disk etc.； And the communications portion 409 of the network interface card including LAN card, modem etc..Communications portion 409 via such as because The network of spy's net performs communication process.Driver 410 is also according to needing to be connected to I/O interfaces 405.Detachable media 411, such as Disk, CD, magneto-optic disk, semiconductor memory etc. are mounted on driver 410, as needed in order to be read from thereon Computer program be mounted into storage section 408 as needed.
Need what is illustrated, framework as shown in Figure 4 is only a kind of optional realization method, can root during concrete practice The component count amount and type of above-mentioned Fig. 4 are selected, are deleted, increased or replaced according to actual needs；It is set in different function component Put, can also be used it is separately positioned or integrally disposed and other implementations, such as GPU and CPU separate setting or can be by GPU collection Into on CPU, communication unit separates setting, can also be integrally disposed on CPU or GPU, etc..These interchangeable embodiments Each fall within protection domain disclosed by the invention.
Particularly, in accordance with an embodiment of the present disclosure, it may be implemented as computer above with reference to the process of flow chart description Software program.For example, embodiment of the disclosure includes a kind of computer program product, it is machine readable including being tangibly embodied in Computer program on medium, computer program are included for the program code of the method shown in execution flow chart, program code May include it is corresponding perform the corresponding instruction of method and step provided by the embodiments of the present application, for example, to receive pending image and/or Pending word is handled, and obtains corresponding characteristics of image and/or character features；Using characterizing network, to characteristics of image and/ Or character features perform map operation, obtain correspondence image feature and/or the mappings characteristics of character features；Using characterizing network, Characteristics of image to be checked corresponding to image set to be checked performs map operation, obtains corresponding to the target signature collection of characteristics of image to be checked； It concentrates to search from target signature based on mappings characteristics and obtains more than one target signature, corresponding target is obtained based on target signature Image.In such embodiments, the computer program can be downloaded and installed from network by communications portion 409 and/ Or it is mounted from detachable media 411.When the computer program is performed by central processing unit (CPU) 401, the application is performed Method in the above-mentioned function that limits.
Methods and apparatus of the present invention, equipment may be achieved in many ways.For example, software, hardware, firmware can be passed through Or any combinations of software, hardware, firmware realize methods and apparatus of the present invention, equipment.The step of for method Sequence is stated merely to illustrate, the step of method of the invention is not limited to sequence described in detail above, unless with other Mode illustrates.In addition, in some embodiments, the present invention can be also embodied as recording program in the recording medium, this A little programs include being used to implement machine readable instructions according to the method for the present invention.Thus, the present invention also covering stores to hold The recording medium of the program of row according to the method for the present invention.
Description of the invention provides for the sake of example and description, and is not exhaustively or will be of the invention It is limited to disclosed form.Many modifications and variations are obvious for the ordinary skill in the art.It selects and retouches It states embodiment and is to more preferably illustrate the principle of the present invention and practical application, and those of ordinary skill in the art is enable to manage The solution present invention is so as to design the various embodiments with various modifications suitable for special-purpose.

Claims (10)

1. a kind of image search method, which is characterized in that including：
The pending data of reception is handled, obtains corresponding data characteristics；The pending data includes：Pending figure Picture and/or pending word；
Using network is characterized, map operation is performed to the data characteristics, obtains corresponding to the mappings characteristics of the data characteristics；Institute State mappings characteristics for and meanwhile characterize the characteristic information of image and word, and pass through the distance between mappings characteristics characterize data it Between similarity；
Corresponding image is obtained from image set to be checked based on the mappings characteristics as target image, in the image set to be checked Including at least one image.
2. according to the method described in claim 1, it is characterized in that, described pair reception pending data handle, obtain Corresponding data characteristics, including：
Using convolutional neural networks, pending image is handled, obtains corresponding to the characteristics of image of the pending image；
And/or using natural language processing network, pending word is handled, obtains corresponding to the pending word Character features.
3. according to the method described in claim 2, it is characterized in that, described utilize natural language processing network, to pending text Word is handled, and obtains corresponding to the character features of the pending word, including：
Pending word is inputted into natural language processing network, by the input layer in the natural language processing network by sample Word decomposes and is converted into one-hot encoding；
The one-hot encoding obtains corresponding to the predictive text feature vector of the sample word by hidden layer and output layer.
4. according to the method described in claim 3, it is characterized in that, described utilize natural language processing network, to pending text Word is handled, and before obtaining the character features of the corresponding pending word, is further included：
The predictive text feature vector of sample word, the prediction based on the sample word are obtained based on natural language processing network Character features vector is trained the natural language processing network.
5. according to the method described in claim 4, it is characterized in that, the predictive text feature based on the sample word to Amount is trained the natural language processing network, including：
Predictive text feature vector based on the sample word, the one-hot encoding for inputting hidden layer and hidden layer parameter calculate log posterior Probability maximizes log posterior probability by adjusting hidden layer parameter；
The hidden layer parameter for prolonging probability after the correspondence maximization logarithm is brought into the natural language processing network, after being trained Natural language processing network.
6. a kind of image search apparatus, which is characterized in that including：
Processing unit is handled for the pending data to reception, obtains corresponding data characteristics；The pending data Including pending image and/or pending word；
Map unit for using network is characterized, map operation to be performed to the data characteristics, obtains corresponding to the data characteristics Mappings characteristics；The mappings characteristics pass through for characterizing the characteristic information of image and word simultaneously between mappings characteristics Similarity between characterize data；
Search unit obtains corresponding image as target image, institute for being based on the mappings characteristics from image set to be checked It states image set to be checked and includes at least one image.
7. a kind of electronic equipment, which is characterized in that including processor, the image that the processor includes described in claim 6 is searched Rope device.
8. a kind of electronic equipment, which is characterized in that including：Memory, for storing executable instruction；
And processor, for communicating to perform the executable instruction so as to complete claim 1 to 5 times with the memory The operation of one described image searching method of meaning.
9. a kind of computer storage media, for storing computer-readable instruction, which is characterized in that described instruction is performed When perform claim require 1 to 5 any one described image searching method operation.
10. a kind of computer program, including computer-readable code, which is characterized in that when the computer-readable code is being set During standby upper operation, the processor execution in the equipment is used to implement claim 1 to 5 any one described image searching method In each step instruction.
CN201711402436.3A 2017-12-21 2017-12-21 Image search method and device, electronic equipment, storage medium, program Pending CN108228757A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201711402436.3A CN108228757A (en) 2017-12-21 2017-12-21 Image search method and device, electronic equipment, storage medium, program

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201711402436.3A CN108228757A (en) 2017-12-21 2017-12-21 Image search method and device, electronic equipment, storage medium, program

Publications (1)

Publication Number Publication Date
CN108228757A true CN108228757A (en) 2018-06-29

Family

ID=62648427

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201711402436.3A Pending CN108228757A (en) 2017-12-21 2017-12-21 Image search method and device, electronic equipment, storage medium, program

Country Status (1)

Country Link
CN (1) CN108228757A (en)

Cited By (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109063068A (en) * 2018-07-23 2018-12-21 广州云测信息技术有限公司 A kind of picture retrieval method and device
CN109165563A (en) * 2018-07-27 2019-01-08 北京市商汤科技开发有限公司 Pedestrian recognition methods and device, electronic equipment, storage medium, program product again
CN109271401A (en) * 2018-09-26 2019-01-25 杭州大拿科技股份有限公司 Method, apparatus, electronic equipment and storage medium are corrected in a kind of search of topic
CN109522432A (en) * 2018-09-21 2019-03-26 重庆大学 A kind of image search method merging adaptive similarity and Bayesian frame
CN109543714A (en) * 2018-10-16 2019-03-29 北京达佳互联信息技术有限公司 Acquisition methods, device, electronic equipment and the storage medium of data characteristics
CN110070579A (en) * 2019-03-16 2019-07-30 平安科技（深圳）有限公司 Localization method, device, equipment and storage medium based on image detection
CN110084137A (en) * 2019-04-04 2019-08-02 百度在线网络技术（北京）有限公司 Data processing method, device and computer equipment based on Driving Scene
CN110083430A (en) * 2019-04-30 2019-08-02 成都市映潮科技股份有限公司 A kind of system theme color replacing options, device and medium
CN110737792A (en) * 2019-10-22 2020-01-31 广东小天才科技有限公司 Exercise searching method, exercise searching device, exercise searching equipment and storage medium
CN110807472A (en) * 2019-10-12 2020-02-18 北京达佳互联信息技术有限公司 Image recognition method and device, electronic equipment and storage medium
CN111651674A (en) * 2020-06-03 2020-09-11 北京妙医佳健康科技集团有限公司 Bidirectional searching method and device and electronic equipment
WO2020182112A1 (en) * 2019-03-13 2020-09-17 腾讯科技（深圳）有限公司 Image region positioning method, model training method, and related apparatus
CN112163636A (en) * 2020-10-15 2021-01-01 电子科技大学 Unknown pattern recognition method of electromagnetic signal radiation source based on twin neural network
CN112766284A (en) * 2021-01-26 2021-05-07 北京有竹居网络技术有限公司 Image recognition method and device, storage medium and electronic equipment
CN113495971A (en) * 2020-04-08 2021-10-12 百度在线网络技术（北京）有限公司 Data conversion model optimization method and device and electronic equipment
US20210374488A1 (en) * 2020-06-01 2021-12-02 Salesforce.Com, Inc. Systems and methods for a k-nearest neighbor based mechanism of natural language processing models
CN113780038A (en) * 2020-06-10 2021-12-10 深信服科技股份有限公司 Picture auditing method and device, computing equipment and storage medium

Citations (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050182755A1 (en) * 2004-02-14 2005-08-18 Bao Tran Systems and methods for analyzing documents over a network
CN101021849A (en) * 2006-09-14 2007-08-22 浙江大学 Transmedia searching method based on content correlation
CN101187927A (en) * 2007-12-17 2008-05-28 电子科技大学 Criminal case joint investigation intelligent analysis method
CN101281520A (en) * 2007-04-05 2008-10-08 中国科学院自动化研究所 Interactive physical training video search method based on non-supervision learning and semantic matching characteristic
CN101299243A (en) * 2008-06-27 2008-11-05 西安电子科技大学 Method of image segmentation based on immune spectrum clustering
CN102254043A (en) * 2011-08-17 2011-11-23 电子科技大学 Semantic mapping-based clothing image retrieving method
CN102436665A (en) * 2011-08-25 2012-05-02 清华大学 Two-dimensional plane representation method for images of alimentary tract
CN103020324A (en) * 2013-01-15 2013-04-03 周万荣 Searching method, searching equipment and searching system based on application
CN103559191A (en) * 2013-09-10 2014-02-05 浙江大学 Cross-media sorting method based on hidden space learning and two-way sorting learning
CN103729428A (en) * 2013-12-25 2014-04-16 中国科学院计算技术研究所 Big data classification method and system
CN103747331A (en) * 2013-12-23 2014-04-23 乐视致新电子科技（天津）有限公司 Interactive method of watching videos and device thereof
CN103927554A (en) * 2014-05-07 2014-07-16 中国标准化研究院 Image sparse representation facial expression feature extraction system and method based on topological structure
CN104200222A (en) * 2014-08-28 2014-12-10 中国人民解放军国防信息学院 Picture object identifying method based on factor graph model
CN104704499A (en) * 2012-06-21 2015-06-10 菲利普莫里斯生产公司 Systems and methods relating to network-based biomarker signatures
CN106227851A (en) * 2016-07-29 2016-12-14 汤平 Based on the image search method searched for by depth of seam division that degree of depth convolutional neural networks is end-to-end

Patent Citations (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050182755A1 (en) * 2004-02-14 2005-08-18 Bao Tran Systems and methods for analyzing documents over a network
CN101021849A (en) * 2006-09-14 2007-08-22 浙江大学 Transmedia searching method based on content correlation
CN101281520A (en) * 2007-04-05 2008-10-08 中国科学院自动化研究所 Interactive physical training video search method based on non-supervision learning and semantic matching characteristic
CN101187927A (en) * 2007-12-17 2008-05-28 电子科技大学 Criminal case joint investigation intelligent analysis method
CN101299243A (en) * 2008-06-27 2008-11-05 西安电子科技大学 Method of image segmentation based on immune spectrum clustering
CN102254043A (en) * 2011-08-17 2011-11-23 电子科技大学 Semantic mapping-based clothing image retrieving method
CN102436665A (en) * 2011-08-25 2012-05-02 清华大学 Two-dimensional plane representation method for images of alimentary tract
CN104704499A (en) * 2012-06-21 2015-06-10 菲利普莫里斯生产公司 Systems and methods relating to network-based biomarker signatures
CN103020324A (en) * 2013-01-15 2013-04-03 周万荣 Searching method, searching equipment and searching system based on application
CN103559191A (en) * 2013-09-10 2014-02-05 浙江大学 Cross-media sorting method based on hidden space learning and two-way sorting learning
CN103747331A (en) * 2013-12-23 2014-04-23 乐视致新电子科技（天津）有限公司 Interactive method of watching videos and device thereof
CN103729428A (en) * 2013-12-25 2014-04-16 中国科学院计算技术研究所 Big data classification method and system
CN103927554A (en) * 2014-05-07 2014-07-16 中国标准化研究院 Image sparse representation facial expression feature extraction system and method based on topological structure
CN104200222A (en) * 2014-08-28 2014-12-10 中国人民解放军国防信息学院 Picture object identifying method based on factor graph model
CN106227851A (en) * 2016-07-29 2016-12-14 汤平 Based on the image search method searched for by depth of seam division that degree of depth convolutional neural networks is end-to-end

Cited By (28)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109063068B (en) * 2018-07-23 2020-07-03 广州云测信息技术有限公司 Picture retrieval method and device
CN109063068A (en) * 2018-07-23 2018-12-21 广州云测信息技术有限公司 A kind of picture retrieval method and device
CN109165563A (en) * 2018-07-27 2019-01-08 北京市商汤科技开发有限公司 Pedestrian recognition methods and device, electronic equipment, storage medium, program product again
CN109165563B (en) * 2018-07-27 2021-03-23 北京市商汤科技开发有限公司 Pedestrian re-identification method and apparatus, electronic device, storage medium, and program product
CN109522432A (en) * 2018-09-21 2019-03-26 重庆大学 A kind of image search method merging adaptive similarity and Bayesian frame
CN109522432B (en) * 2018-09-21 2023-01-31 重庆大学 Image retrieval method integrating adaptive similarity and Bayes framework
CN109271401A (en) * 2018-09-26 2019-01-25 杭州大拿科技股份有限公司 Method, apparatus, electronic equipment and storage medium are corrected in a kind of search of topic
CN109543714A (en) * 2018-10-16 2019-03-29 北京达佳互联信息技术有限公司 Acquisition methods, device, electronic equipment and the storage medium of data characteristics
EP3940638A4 (en) * 2019-03-13 2022-05-11 Tencent Technology (Shenzhen) Company Limited Image region positioning method, model training method, and related apparatus
KR102646667B1 (en) 2019-03-13 2024-03-11 텐센트 테크놀로지(센젠) 컴퍼니 리미티드 Methods for finding image regions, model training methods, and related devices
WO2020182112A1 (en) * 2019-03-13 2020-09-17 腾讯科技（深圳）有限公司 Image region positioning method, model training method, and related apparatus
KR20210076110A (en) * 2019-03-13 2021-06-23 텐센트 테크놀로지(센젠) 컴퍼니 리미티드 Methods for finding image regions, model training methods and related devices
CN110070579A (en) * 2019-03-16 2019-07-30 平安科技（深圳）有限公司 Localization method, device, equipment and storage medium based on image detection
CN110084137A (en) * 2019-04-04 2019-08-02 百度在线网络技术（北京）有限公司 Data processing method, device and computer equipment based on Driving Scene
CN110083430B (en) * 2019-04-30 2022-03-29 成都映潮科技股份有限公司 System theme color changing method, device and medium
CN110083430A (en) * 2019-04-30 2019-08-02 成都市映潮科技股份有限公司 A kind of system theme color replacing options, device and medium
CN110807472B (en) * 2019-10-12 2022-08-12 北京达佳互联信息技术有限公司 Image recognition method and device, electronic equipment and storage medium
CN110807472A (en) * 2019-10-12 2020-02-18 北京达佳互联信息技术有限公司 Image recognition method and device, electronic equipment and storage medium
CN110737792A (en) * 2019-10-22 2020-01-31 广东小天才科技有限公司 Exercise searching method, exercise searching device, exercise searching equipment and storage medium
CN113495971A (en) * 2020-04-08 2021-10-12 百度在线网络技术（北京）有限公司 Data conversion model optimization method and device and electronic equipment
US20210374488A1 (en) * 2020-06-01 2021-12-02 Salesforce.Com, Inc. Systems and methods for a k-nearest neighbor based mechanism of natural language processing models
CN111651674B (en) * 2020-06-03 2023-08-25 北京妙医佳健康科技集团有限公司 Bidirectional searching method and device and electronic equipment
CN111651674A (en) * 2020-06-03 2020-09-11 北京妙医佳健康科技集团有限公司 Bidirectional searching method and device and electronic equipment
CN113780038A (en) * 2020-06-10 2021-12-10 深信服科技股份有限公司 Picture auditing method and device, computing equipment and storage medium
CN112163636A (en) * 2020-10-15 2021-01-01 电子科技大学 Unknown pattern recognition method of electromagnetic signal radiation source based on twin neural network
CN112163636B (en) * 2020-10-15 2023-09-26 电子科技大学 Unknown mode identification method of electromagnetic signal radiation source based on twin neural network
CN112766284A (en) * 2021-01-26 2021-05-07 北京有竹居网络技术有限公司 Image recognition method and device, storage medium and electronic equipment
CN112766284B (en) * 2021-01-26 2023-11-21 北京有竹居网络技术有限公司 Image recognition method and device, storage medium and electronic equipment

Similar Documents

Publication Publication Date Title
CN108228757A (en) Image search method and device, electronic equipment, storage medium, program
Yu et al. Hierarchical deep click feature prediction for fine-grained image recognition
US20190325259A1 (en) Feature extraction and machine learning for automated metadata analysis
Demir et al. A novel active learning method in relevance feedback for content-based remote sensing image retrieval
Meng et al. Oracle bone inscription detector based on ssd
CN107003977A (en) System, method and apparatus for organizing the photo of storage on a mobile computing device
EP3166020A1 (en) Method and apparatus for image classification based on dictionary learning
Grechikhin et al. User modeling on mobile device based on facial clustering and object detection in photos and videos
Niu et al. Knowledge-based topic model for unsupervised object discovery and localization
Gao et al. Fashion clothes matching scheme based on Siamese Network and AutoEncoder
Wang et al. Fabric identification using convolutional neural network
Zhang et al. Deep unsupervised self-evolutionary hashing for image retrieval
Chen et al. Visual-based deep learning for clothing from large database
Zhang et al. Semantically modeling of object and context for categorization
CN108268629A (en) Image Description Methods and device, equipment, medium, program based on keyword
Bouguila On multivariate binary data clustering and feature weighting
Ning et al. Integration of image feature and word relevance: Toward automatic image annotation in cyber-physical-social systems
Thompson et al. finFindR: Computer-assisted recognition and identification of bottlenose dolphin photos in r
CN113704534A (en) Image processing method and device and computer equipment
CN111898528A (en) Data processing method and device, computer readable medium and electronic equipment
John et al. A multi-modal cbir framework with image segregation using autoencoders and deep learning-based pseudo-labeling
CN112487154B (en) Intelligent search method based on natural language
Jain et al. Multi-label classification for images with labels for image annotation
Li et al. Automatic image annotation using adaptive weighted distance in improved K nearest neighbors framework
Li et al. Affective image classification via semi-supervised learning from web images

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication
RJ01 Rejection of invention patent application after publication

Application publication date: 20180629