CN102147815B - Method and system for searching images - Google Patents

Abstract

The present invention provides a picture search method and a picture search system, the method includes a client receiving query content, the query content includes the target picture to be queried, or the target picture to be queried and related information; the client obtains the visual word of the target picture , and select at least one target visual word dictionary corresponding to the query content from more than one visual word dictionary on the client according to preset rules, and obtain the target visual word of the visual word according to the target visual word dictionary; after encoding the target visual word Sent to the server to obtain the result picture matching the query content and/or the relevant information of the result picture. The method improves the image search speed by reducing the amount of data uploaded by the client, shortens the user's waiting time, and can improve the accuracy of system retrieval.

Description

Image search method and image search system

technical field

The invention relates to the technical field of picture identification and search, in particular to a picture search method and a picture search system.

Background technique

With the vigorous development of wireless networks and the continuous enhancement of the functions of mobile devices, users use mobile devices to search for picture information online more and more frequently. What first appeared was to use text to describe the content of pictures, and then perform subsequent retrieval/search based on the text content. However, because the text cannot accurately describe the content of the picture, and the retrieval result of the text retrieval image is often not the information that the user needs, the text retrieval method cannot satisfy the user.

Another content-based image search method is to use images as queries and search for similar images as a retrieval method, which can avoid the problem of inaccurate text descriptions caused by the above-mentioned text retrieval images. However, this content-based image search method is to directly transmit images to the server, which will generate a large amount of data transmission. In particular, in an unstable wireless network environment with limited bandwidth, image search often requires a long query response time.

As a result, people in the industry use visual descriptors to describe the picture, and convert the picture into a one-dimensional vector composed of multiple data, so that the transmission of pictures to the server is changed to the transmission of data vectors to the server. The way the visual descriptor describes the picture can improve the query response time of the picture, but limited by the quality of the current mobile network, the upload speed still cannot meet the actual needs of users. In view of this, how to provide an image retrieval method that can not only ensure the performance and efficiency of image retrieval, but also reduce the bandwidth requirement in image retrieval is a technical problem that needs to be solved at present.

Contents of the invention

Aiming at the defects in the prior art, the present invention provides a picture search method and a picture search system. The method and the system improve the picture retrieval speed by reducing the amount of data uploaded by the client without reducing the search performance, shortening the The user's waiting time is reduced, and the accuracy of the search system can be improved.

The image search method provided by the present invention includes:

The client receives the query content, the query content includes the target picture to be queried, or the target picture to be queried and related information;

The client obtains the visual word of the target picture, and selects at least one target visual word dictionary corresponding to the query content from more than one visual word dictionaries on the client according to preset rules, and obtains the target visual word of the visual word according to the target visual word dictionary word;

Encode the target visual word and send it to the server to obtain the result picture matching the query content and/or the relevant information of the result picture.

According to another aspect of the present invention, the present invention also provides an image search method, which includes:

The server receives the encoded target visual word and decodes the target visual word;

The server searches the index table corresponding to the visual word dictionary in the server based on the target visual word to obtain the result picture and/or the relevant information of the result picture, and sends it to the client;

The visual word dictionary is: a visual word dictionary established by clustering the visual features of all pictures in the server-side picture database.

According to another aspect of the present invention, the present invention also provides an image search system, which includes:

The receiving module, the client receives the query content including the target picture to be queried, or the target picture to be queried and related information;

The target visual word acquisition module, the client obtains the visual word of the target picture, and selects at least one target visual word dictionary corresponding to the query content from more than one visual word dictionary on the client according to preset rules, and, according to the target visual word dictionary Get the target visual word of the visual word;

The target visual word sending module encodes the target visual word and sends it to the server,

The receiving and searching module, the server receives the coded target visual word and decodes it, and searches the index table corresponding to the visual word dictionary of all pictures in the database based on the target visual word to obtain the result picture and/or the relevant information of the result picture;

A sending module, the server sends the result picture and/or related information of the result picture to the client.

The image search method and image search system of the present invention mainly compress target images into target visual words with visual content description capabilities on the client side, and then upload them to the server, thereby realizing low bit rate between the client and the server. The data transmission shortens the waiting time of the user when querying the target picture, and improves the response time of the server in the system at the same time, thereby improving the query efficiency in the picture search method.

Furthermore, the search method in the present invention can also improve the accuracy of search results. The present invention can be popularized and applied to the retrieval/search of various pictures, and can obtain the extended information of the result picture, so that the method has a wide application range, can be applied to various fields, and is convenient for users to search various types of information.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description These are some embodiments of the present invention. Those skilled in the art can also obtain other drawings based on these drawings without creative work.

Fig. 1 is a flow chart of steps of an embodiment of a picture search method in the present invention;

Fig. 2 is a flow chart of steps for screening effective visual dictionaries in the present invention;

Fig. 3 is a flow chart of the steps of an embodiment of the image search method in the present invention;

FIG. 4 is a schematic structural diagram of an embodiment of the image search system in the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are the Some, but not all, embodiments are invented. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present invention.

The present invention mainly provides a picture search method, the search method is mainly to use the visual word dictionary library preset in the client to obtain the target visual words for the target picture with a small amount of transmission data, and then encode these target visual words After that, it is sent to the server, and the result picture and/or related extended information has been obtained from the server. This method effectively reduces the number of visual words describing the target picture, reduces the amount of data transmitted to the server, and achieves low-bit data transmission between the client and the server. It can effectively solve the problem of long data transmission time under the current bandwidth limitation. At the same time, it can reduce the response time of the server, which can better save the waiting time of users.

Note the following descriptions:

Visual words: Discretely divide the visual feature space, each word is a division, and use image features to describe image content, which is the most basic data feature;

Visual word lexicon: A collection of visual words for all or a selected portion of images in an image database.

Referring to Fig. 1, Fig. 1 shows a flow chart of the steps of an embodiment of the image search method in the present invention, the steps of which include:

Step 101, the client receives query content, the query content includes the target picture to be queried/query picture, or the target picture to be queried and related information; the relevant information here is information other than the target picture. For example, the relevant information may be text information describing the target image, geographic location information, publishing house information, publishing house barcode, publishing house logo or electronic label, and the like.

Step 102, the client obtains the visual word of the target picture, and selects at least one target visual word dictionary corresponding to the query content from more than one visual word dictionaries on the client according to preset rules, and obtains the visual word according to the target visual word dictionary the target visual word of

The visual words of the target picture can be generated by acquiring more than one visual feature of the target picture, and converting the features into visual words in the original visual word dictionary according to the mapping rules between visual features and visual words. Preferably, the client's original visual word dictionary is the same as the server's original visual word dictionary, and the client's original visual word dictionary can be preset on the client and can be updated from the server in real time.

The original visual word dictionary can be generated in such a way as to acquire more than one visual feature of the server-side database picture, and generate multiple classes by clustering the visual features of the database picture. The specific methods of this and subsequent clustering methods can be K-means clustering, hierarchical clustering, spectral clustering, etc., where spectral clustering can refer to the literature "Ng A., Jordan M., and Weiss Y. On Spectral Clustering : Analysis and an algorithm. NIPS, 849-856, 2001". The class center of each class represents the class, which is called a visual word, that is, each class is a visual word, and the collection of visual words in the entire database constitutes the original visual word dictionary.

Specifically, in this embodiment, visual features such as the color histogram, texture map, scale-invariant descriptor, gradient position orientation histogram or direction gradient histogram of the target image can be extracted;

Then, according to the mapping rules of the visual features and the visual words, the color histogram, texture map, scale-invariant descriptor (SIFT), gradient position orientation histogram (GLOH) or direction gradient histogram ( HOG) into all visual words corresponding to the server-side original visual word dictionary.

Sub-step 1021, according to the type of the query content, from one or more visual word dictionaries preset on the client side, search for a visual word dictionary and a prediction loss function matching the type of the query content. That is to say, the mapping rule between the query content type and the visual word dictionary is preset. For example, the query content is a picture and text information describing the picture, and the visual word dictionary is a visual word dictionary corresponding to the text information.

In particular, the one or more visual word dictionaries preset by the client are obtained by the client from the server in advance, and the client regularly updates the one or more visual word dictionaries. Or, when a new picture is added on the server side, the client side may be prompted to update its internal visual word dictionary database. The process of how the server acquires the visual word dictionary database is described in detail below through subsequent steps P1 to P3.

Sub-step 1022, using the prediction loss function to calculate the prediction loss value of each visual word dictionary in the visual word dictionary library for the visual word of the target picture, and obtain one or more visual word dictionaries within the threshold range.

Wherein, the predictive loss function is used to calculate the predictive loss value of each visual word dictionary in the visual word dictionary library to the visual word of the target picture, and the specific calculation method of the predictive loss value can be selected from the following first calculation method to the third calculation method any kind.

The first calculation method: the cosine distance between the visual word of the target picture and the class center of the picture class where the target visual word dictionary is located; or

The second calculation method: the cosine distance between the visual word of the target picture and the class center of the picture class where the target visual word dictionary is located, and the weighted sum of the Euclidean distances between the related information and the similar information of the picture class where the visual word dictionary is located;

The third calculation method: the product of the visual similarity distance between the target picture and the picture class of the visual word dictionary of the target visual word dictionary, and the Euclidean distance between the related information and the same kind of information of the picture class of the visual word dictionary.

For example, the prediction loss function _fpredict (q _i , C _j ) is formulated as:

f _predict (q _i , C _j )=α·Vd _ij +β·Rd _ij

f _prediction (q _i , C _j ) represents the prediction loss value of the target picture q _i and the picture class C _j where the visual word dictionary is located, and Vd _ij is the cosine distance between the visual word of the target picture and the class center of the picture class where the target visual word dictionary is located , Rd _ij is the Euclidean distance between the relevant information and the same kind of information in the image category of the visual word dictionary. α and β are real numbers and can be set according to experience or requirements.

The cosine distance Vd _ij calculation formula of the visual word of the target picture and the class center of the picture class where the target visual word dictionary is located is,

${\mathrm{<span\; class="notranslate">\; Vd</span>}}_{\mathrm{<span\; class="notranslate">\; ij</span>}}<span\; class="notranslate">\; =</span>{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>\stackrel{<span\; class="notranslate">\; \&Right\; Arrow;</span>}{{\mathrm{<span\; class="notranslate">\; BOW</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}<span\; class="notranslate">\; ,</span>\stackrel{<span\; class="notranslate">\; \&Right\; Arrow;</span>}{{\mathrm{<span\; class="notranslate">\; BOW</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}_{\mathrm{<span\; class="notranslate">\; co</span>}\mathrm{<span\; class="notranslate">\; sin</span>}\mathrm{<span\; class="notranslate">\; e</span>}}<span\; class="notranslate">\; =</span>\frac{\stackrel{<span\; class="notranslate">\; \&Right\; Arrow;</span>}{{\mathrm{<span\; class="notranslate">\; BOW</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}<span\; class="notranslate">\; \&CenterDot;</span>\stackrel{<span\; class="notranslate">\; \&Right\; Arrow;</span>}{{\mathrm{<span\; class="notranslate">\; BOW</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}}}{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>\stackrel{<span\; class="notranslate">\; \&Right\; Arrow;</span>}{{\mathrm{<span\; class="notranslate">\; BOW</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; \&CenterDot;</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>\stackrel{<span\; class="notranslate">\; \&Right\; Arrow;</span>}{{\mathrm{<span\; class="notranslate">\; BOW</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}<span\; class="notranslate">\; ;</span>$

图片i为目标图片的视觉单词，

为目标视觉单词词典所在图片类C_j的类中心。

Picture i is the visual word of the target picture,

is the class center of the picture class C _j where the target visual word dictionary is located.

The formula for calculating the Euclidean distance Rd _ij of related information and the same kind of information in the image category of the visual word dictionary is

${\mathrm{<span\; class="notranslate">\; Rd</span>}}_{\mathrm{<span\; class="notranslate">\; ij</span>}}<span\; class="notranslate">\; =</span>{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}_{\mathrm{<span\; class="notranslate">\; co</span>}\mathrm{<span\; class="notranslate">\; sin</span>}\mathrm{<span\; class="notranslate">\; e</span>}}<span\; class="notranslate">\; =</span>\sqrt{{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}$

R _i is the relevant information in the query content for picture i, and R _j is the similar information value of the picture category C _j where the target visual word dictionary is located.

In addition, the types of query content in the above sub-step 1021 may include: target pictures, target pictures and texts, target pictures and signals detected by sensors, and the target pictures and object recognition software recognizes object tags in the pictures. Among them, the signal detected by the sensor may include the geographical location information detected by the Global Positioning System (GPS), the barcode information of the book or commodity barcode scanned by the barcode scanner, and the electronic tag information (RFID) read by the electronic tag reader. wait. The object tags recognized by the object recognition software may include face recognition by face recognition software, text recognition by character recognition system software (ORC), and the like.

For example, when the type of the query content is the target picture category, the visual word dictionary database is a visual word dictionary database based on visual similarity established based on similarity of pictures.

When the type of query content is the target image and the signal detected by the sensor, if the query content is a landmark image, and the signal detected by the sensor can be the building in the landmark image, the geographic location information corresponding to the landmark image, and the geographic location corresponding to the building The geographical location information corresponding to the natural landscape in the information or landmark picture. At this time, the visual word dictionary is a visual word dictionary corresponding to the geographic location information.

When the type of query content is a target picture and the object recognition software recognizes the object label in the picture, if the query content is a book picture, the object recognition software recognizes that the object label in the picture can be the book publisher's logo in the book picture or name. At this time, the visual word dictionary base is a visual word dictionary base corresponding to the logo or name of the publishing house.

The query content is a photo of the product, and the object recognition software recognizes that the object label in the picture can be the trademark of the product, or the barcode scanner scans the barcode corresponding to the product (physical object) in the photo, and the visual word dictionary library is the corresponding trademark or barcode A dictionary of visual words.

The query content is the guide map of the museum exhibition room. The object recognition software recognizes that the object label in the picture is the barcode or electronic label in the guide picture, and the visual word dictionary is the visual word dictionary corresponding to the barcode or electronic tag. In this step, the picture set is divided into multiple categories, so that the coupled visual words of the divided picture set are the largest, thereby achieving the purpose of reducing the dimension of the visual word dictionary.

Step 103, encode the target visual word and send it to the server, so as to obtain and display the result picture matching the query content and/or the relevant information of the result picture.

In the above-mentioned sub-step 1021, when the client pre-presets one or more visual word dictionaries for the client and obtains from the server in advance, the step of the server pre-establishing one or more visual word dictionaries includes:

The first step P1: Divide the pictures in the server database into various types of picture sets by using the picture set division method.

Wherein, the sub-step of the first step P1 is to divide all the pictures into multiple picture sets by utilizing the visual similarity between the pictures. Alternatively, the sub-step of the first step P1 is to divide all the pictures into multiple picture sets by using the information related to the pictures, such as the date when the pictures were taken, text tags, electronic tags, etc. Of course, the sub-steps of the first step P1 can also divide all the pictures into multiple sets by utilizing the visual similarity between the pictures and information related to the pictures such as date of photographing, text label, electronic label, etc.

The second step P2: establish a visual word dictionary corresponding to each picture set, and analyze the visual word dictionary corresponding to each picture. In particular. The visual word dictionary here can be the original visual word dictionary of the picture by clustering the visual features of the picture collection; or, the visual word dictionary here is: clustering the visual features of the picture set to build the visual word of the picture Dictionary, based on the screening rules of the effective visual word dictionary, determine the effective visual word dictionary representing the original visual word dictionary, use the effective visual word dictionary as the visual word dictionary, and then take the dimension of the visual word dictionary (dimensions in the N-axis coordinate system) can be relatively reduced.

The third step P3: (the first way to obtain the visual word dictionary database) If the visual word dictionary meets the conditions for establishing the visual word dictionary database, then the visual word dictionary sets corresponding to various types of picture collections form a visual word dictionary database.

Wherein: the condition for establishing the visual word dictionary library can be: the number of visual words in the visual word dictionary of each picture collection after division is less than or equal to the total number of visual words in the visual word dictionary of the server database; and count its visual words for each picture collection after division and calculate the entropy of the probability distribution of visual words whose information entropy is less than the set threshold.

Finally, the server sends the established visual word dictionary to the client and stores it for subsequent use. When the server has a new picture, it can update its own visual word dictionary and update the client's visual word dictionary at the same time.

Compared with the prior art, the screening rules of the effective visual word dictionary in the present embodiment can be (that is, the screening rules of the effective visual word dictionary used in the second step P2 can be):

Step P41: selecting a certain number of pictures from a certain type of pictures as sample pictures, and converting the features of the sample pictures into visual words in the original visual word dictionary;

Step P42: Query in the visual word index table of the original visual word dictionary according to the visual word of the sample picture, and obtain the original query result;

Step P43: Combine any visual words belonging to the original visual word dictionary to form a screened visual word dictionary, and based on the screened visual word dictionary, convert the features of the sample picture into corresponding first visual words in the screened visual word dictionary. Word, and adopt the first visual word to inquire in the visual word index table of described original visual word dictionary, obtain the first query result corresponding to screening visual word dictionary;

Step P44: Analyze the original query results and the first query results of all sample pictures, if the first query results match the original query results, then use the current filtered visual word dictionary as the visual word dictionary; otherwise, use the original visual word dictionary Select a visual word in the dictionary and add it to the current filtered visual word dictionary, and return to the step of obtaining the first query result.

It should be noted that: the visual word dictionaries corresponding to the above-mentioned various types of picture collections are generated in a manner of clustering the visual features of the picture collections to establish the visual word dictionaries of the pictures.

Compared with the prior art, the search method in this embodiment only needs to transmit tens of bits of encoded data to the server, so as to achieve the purpose of faster query by the client, and at the same time improve the process of querying the target image by the client. In the transmission efficiency, and shorten the response query time of the server.

In particular, the image search method of this embodiment is mainly applied to image query in mobile terminals. These mobile terminals adaptively select a suitable visual word dictionary for query information and obtain target visual words with visual description capabilities. In order to effectively reduce the data volume of the target picture to be queried, and then realize the low-bit data transmission between the client and the server, shorten the waiting time of the user when querying the target picture, and improve the response time of the server at the same time, thereby improving the Query efficiency of image search methods.

Furthermore, the search method in the present invention can also improve the accuracy of retrieval results. The present invention can be popularized and applied to the retrieval/search of various pictures, and can obtain the extended information of the result pictures, so that the method has a wide application range, can be used in various fields, and is convenient for users to search various types of information.

Referring to Fig. 2, Fig. 2 shows a flow chart of specific steps for screening effective visual dictionaries in the present invention; that is, specific calculations for screening effective visual dictionaries in the above-mentioned embodiment of the index construction method for distributed image search Steps include:

为查询结果中排在第j位的图片，

的视觉单词向量为

The first step 201: select N _sample pictures from the entire picture database, use these sample pictures as query pictures in the visual word index table, and retrieve the results of the first R query pictures. For the i-th picture, its query result

is the image ranked jth in the query result,

The visual word vector of

从原始视觉单词词典的子集中筛选出有效视觉单词词典。The second step 202: calculate the term frequency-inverse document frequency (TF-IDF) of each result picture, The TF-IDF is

A valid visual word dictionary is filtered from a subset of the original visual word dictionary.

w_i为图片i的权重为0，测试子集train_V为空；The third step 203: set the number of iterations as d=1, the effective visual word dictionary min_V _j is empty, the candidate visual word set cadi_V _j =V (V is the original visual word dictionary), and the number of elements is N _cv , N _sample A set of weights for an image

w _i is the weight of picture i is 0, and the test subset train_V is empty;

Fourth step 204: if the number of iterations d>α or lost _Rank <β, then end.

train_V_t＝min_V∪{wd_t}。Fifth step 205: otherwise, add N _cv visual words in the candidate visual word set to the test subset tran_V, thereby generating N _cv test subsets train_V ₁ , . . .

train_V _t =min_V∪{wd _t }.

The sixth step 206: each test subset is used as a visual word dictionary, and the local feature vector S _i of the query picture i is converted into a visual word vector according to the visual word dictionary, and the visual word vector of the picture i corresponding to the test subset train_V _k is

对于测试子集train_V_k和图片I_i，总错误率Lost(I_i)^k计算方法为如下的M1至M4所示：Step 7 207: Calculate the total error rate caused by using each test subset to describe each query image

For the test subset train_V _k and picture I _i , the calculation method of the total error rate Lost(I _i ) ^k is as shown in M1 to M4 as follows:

映射为成原始视觉单词词典视觉向量

为映射向量；M1, will

Mapping to the original visual word dictionary visual vector

is the mapping vector;

和查询图片i的内容相似性

计算方法为：M2, calculate when the query picture is described by the test subset train_V _k , the result picture

Content similarity to query image i

The calculation method is:

${<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>\stackrel{<span\; class="notranslate">\; \&Right\; Arrow;</span>}{\mathrm{<span\; class="notranslate">\; gBO</span>}{\mathrm{<span\; class="notranslate">\; W</span>}}_{{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; \&Center\; Dot;</span>\stackrel{<span\; class="notranslate">\; \&Right\; Arrow;</span>}{{\mathrm{<span\; class="notranslate">\; BOW</span>}}_{{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}_{\mathrm{<span\; class="notranslate">\; co</span>}\mathrm{<span\; class="notranslate">\; sin</span>}\mathrm{<span\; class="notranslate">\; e</span>}}<span\; class="notranslate">\; =</span>\frac{\stackrel{<span\; class="notranslate">\; \&Right\; Arrow;</span>}{{\mathrm{<span\; class="notranslate">\; BOW</span>}}_{{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}}}<span\; class="notranslate">\; \&Center\; Dot;</span>\stackrel{<span\; class="notranslate">\; \&Right\; Arrow;</span>}{{\mathrm{<span\; class="notranslate">\; wxya</span>}}_{{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span>}}{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>\stackrel{<span\; class="notranslate">\; \&Right\; Arrow;</span>}{\mathrm{<span\; class="notranslate">\; BO</span>}{\mathrm{<span\; class="notranslate">\; W</span>}}_{{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; \&Center\; Dot;</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>\stackrel{<span\; class="notranslate">\; \&Right\; Arrow;</span>}{\mathrm{<span\; class="notranslate">\; gBO</span>}{\mathrm{<span\; class="notranslate">\; W</span>}}_{{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}<span\; class="notranslate">\; ;</span>$

M3, calculate the error rate Lost(I _i ) ^k caused by the test subset train_V _k describing the query image i

$\mathrm{<span\; class="notranslate">\; Lost</span>}{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; \&times;</span>\underset{\mathrm{<span\; class="notranslate">\; r</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; R</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\mathrm{<span\; class="notranslate">\; R</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Center\; Dot;</span>{\mathrm{<span\; class="notranslate">\; T.I.</span>}}_{{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}}}<span\; class="notranslate">\; \&Center\; Dot;</span>{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>\stackrel{<span\; class="notranslate">\; \&Right\; Arrow;</span>}{\mathrm{<span\; class="notranslate">\; gBO</span>}{\mathrm{<span\; class="notranslate">\; W</span>}}_{{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; \&CenterDot;</span>\stackrel{<span\; class="notranslate">\; \&Right\; Arrow;</span>}{{\mathrm{<span\; class="notranslate">\; BOW</span>}}_{{\mathrm{<span\; class="notranslate">\; A</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}_{\mathrm{<span\; class="notranslate">\; co</span>}\mathrm{<span\; class="notranslate">\; sin</span>}\mathrm{<span\; class="notranslate">\; e</span>}}<span\; class="notranslate">\; ;</span>$

为与结果图片排序位置递增的函数，可以设置

for the picture with the result A function that increases the sorting position, you can set

M4, calculate the total error rate of the query image described by the test subset train_V _k

${\mathrm{<span\; class="notranslate">\; lost</span>}}_{\mathrm{<span\; class="notranslate">\; Rank</span>}}^{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{{\mathrm{<span\; class="notranslate">\; N</span>}}_{\mathrm{<span\; class="notranslate">\; sample</span>}}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}}\mathrm{<span\; class="notranslate">\; Lost</span>}{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; I</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; .</span>$

The eighth step 208: select the test subset that minimizes the total error rate lost _Rank , and update the effective visual word dictionary and the candidate visual word set. The specific method is: if the test subset is train_V _MIN , then the effective visual word dictionary is min_V = train_V _MIN , cadi_V = cadi_V - {wd _MIN }.

Ninth step 209: update the weight of each query image, the calculation method for updating the weight of query image i is:

The tenth step 210: update the number of iterations d=d+1, and return to the fourth step 204.

On the basis of the above-mentioned embodiments, the following takes the query information only including pictures as an example to describe in detail, and the query steps are as follows:

In the first step, the client obtains the target image to be searched.

In the second step, the client obtains more than one feature of the target picture, and converts these features into visual words.

Specifically, in this embodiment, visual features such as color histogram, texture map, scale-invariant descriptor, gradient position orientation histogram or direction gradient histogram of the target picture may be extracted.

Then, according to the mapping rules of visual features and visual words, the color histogram, texture map, scale-invariant descriptor (SIFT), gradient position orientation histogram (GLOH) or orientation gradient histogram (HOG) of the target picture Converts to sight words in the client's sight word dictionary.

The third step is to search for a target visual word dictionary matching the target picture from one or more visual word dictionary databases of the client. The visual word dictionaries of these clients are pre-downloaded and obtained by the client from the server. That is to say, the client is preset with a visual word dictionary library corresponding to the server.

In particular, when the query content is only the target picture, the client selects the visual word dictionary database of visual similarity established according to the similarity of the picture, and calculates any visual word dictionary database of the visual similarity between the target picture and the visual word dictionary database. The visual similarity distance of the picture class where the word dictionary is located, the visual word dictionary with the smallest similarity distance is selected as the visual word dictionary matching the target picture, that is, the target visual word dictionary. The visual similarity distance is the class center cosine distance between the visual word of the target picture and the picture class where the visual word dictionary is located.

The fourth step is to analyze the visual word and the target visual word dictionary to obtain the target visual word corresponding to the target picture; specifically, according to the visual word dictionary, the visual word of the target picture is screened, and the visual word belonging to the visual word dictionary is selected. Sight words as target sight words;

The fifth step is to compress the target visual word into a data packet according to the Huffman encoding method; the specific operation is to scan the probability of occurrence of each target visual word, and establish a Huffman tree, using '0' and '1 'Encode the target word, the greater the probability, the fewer the number of encoding digits, save the visual word and the corresponding encoding in the Huffman encoding table and send it to the client.

In the sixth step, the server decodes the data packet into the target visual word according to the Huffman coding table, searches the visual word index table of the original visual word dictionary inside it according to the target visual word, and obtains more than one visual word corresponding to the target visual word The result picture, and/or obtain the extended information of the result picture, and send the result picture and/or the extended information to the client for display.

According to another aspect of the present invention, the present invention also provides a picture search method, as shown in Figure 3, the steps include:

Step 301: The server receives the encoded target visual word and decodes the target visual word.

Step 302: The server searches the index table corresponding to the visual word dictionary in the server based on the target visual word to obtain the result picture and/or related information of the result picture.

The visual word dictionary is: a visual word dictionary established by clustering the visual features of all or part of the pictures in the server-side picture database.

Step 303: Send the result picture and/or related information of the result picture to the client for display.

In the above embodiment, fewer target visual word query result pictures are used, which can improve the efficiency of the target picture query on the basis of the original retrieval performance, shorten the waiting time of the user, and then realize the low-bandwidth situation. To achieve the purpose of image query.

According to another aspect of the present invention, the present invention also provides a picture search system, as shown in Figure 4, which includes:

Receiving module 401, the client receives the query content including the target picture to be queried, or the target picture to be queried and related information;

The target visual word acquisition module 402, the client acquires the visual word of the target picture, and selects at least one target visual word dictionary corresponding to the query content from more than one visual word dictionaries on the client according to preset rules, and, according to the target visual word The dictionary gets the target visual word of the visual word;

The target visual word sending module 403 sends the target visual word to the server after encoding,

Receiving and searching module 404, the server receives and decodes the coded target visual word, and searches the index table corresponding to the visual word dictionary of all pictures in the database based on the target visual word to obtain the result picture and/or the relevant information of the result picture;

Sending module 405, the server sends the result picture and/or related information of the result picture to the client.

The above-mentioned picture query system or picture search system automatically selects a visual word dictionary suitable for the type of query information according to the combination type of the query information, converts the picture into a visual word according to the visual word dictionary, and further compresses the visual word into a file with a small amount of data. The data package of the target visual word, and then quickly and accurately obtain the result picture of the target picture to be retrieved and its related extended information according to the data package.

Among them, in the process of obtaining the target picture, the picture search system effectively divides the database picture set according to the picture division criterion, so that the types of visual words of the divided pictures are much smaller than the visual types of the original database picture set, thus Effectively reduce the number of visual words describing pictures, realize the conversion of target pictures into data packets of tens of bits of target visual words, reduce the amount of data transmitted to the server, and achieve low-bit transmission between the client and the server. It can effectively solve the problem of long data transmission time under the current bandwidth limitation, and can better save the waiting time of users. The above search method is suitable for different types of queries and has strong scalability.

The client mentioned in this embodiment may be a mobile terminal, such as a mobile phone, an IPAD, a tablet computer, and the like.

In particular, the client in this embodiment may include:

The receiving module receives the query content including the target picture to be queried, or the target picture to be queried and related information;

The target visual word acquisition module obtains the visual word of the internal target picture, and selects at least one target visual word dictionary corresponding to the query content from more than one visual word dictionary on the client according to preset rules, and, according to the target visual word dictionary Get the target visual word of the visual word;

The target visual word sending module encodes the target visual word and sends it to the server,

The result picture receiving module is used for receiving and displaying the result picture searched and sent by the server and/or the relevant information of the result picture.

The modules displayed in the image search system above are only schematically showing their internal structural relationships. The same module may be used for transmission or reception multiple times in a certain system, client or other structures, or one of the above-mentioned modules may be used at intervals. A module, the above embodiment is only a schematic illustration, and it is not limited to the structural arrangement and connection relationship in FIG. 4 . In addition, it is also possible to add some other modules in the image search system and client that can implement some steps in the image search method of the present invention, all of which belong to the content of the present invention.

Finally, it should be noted that: the order of the steps in the above image search method can be performed in parallel or interchanged, and the above embodiment is only for illustrative illustration, and does not limit the execution order of the steps. In addition, the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it can still be applied to the foregoing embodiments Modifications are made to the recorded technical solutions, or equivalent replacements are made to some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims (9)

1. A method for image search, characterized in that, comprising: The client receives the query content, the query content includes the target picture to be queried, or the target picture to be queried and related information; The client obtains the visual words of the target image, and according to the type of the query content, from one or more visual word dictionaries preset by the client, finds the visual word dictionary and the prediction loss function matching the type of the query content, using The prediction loss function calculates the prediction loss value of each visual word dictionary in the visual word dictionary to the visual word of the target picture, obtains one or more visual word dictionaries within the threshold range, and obtains the target of the visual word according to the target visual word dictionary sight words; Encode the target visual word and send it to the server to obtain the result picture matching the query content and/or the relevant information of the result picture. 2. The image search method according to claim 1, wherein the type of the query content comprises: Target pictures, target pictures and texts, target pictures and signals detected by sensors, target pictures and object recognition software to identify the object labels in the pictures; Among them, the signal detected by the sensor includes the geographical position information detected by the global positioning system device, the barcode barcode of the book or commodity is scanned by the barcode scanner, and the electronic label is read by the electronic label reader; Object tags recognized by object recognition software include face recognition software to recognize faces, and text recognition system software to recognize text. 3. The image search method according to claim 1, characterized in that: One or more visual word dictionaries preset by the client are obtained by the client from the server in advance, and the client regularly updates the one or more visual word dictionaries; The step that described service end sets up one or more visual word dictionaries includes: Divide the pictures in the server database into various types of picture sets by using the picture set division method, and establish the visual word dictionary corresponding to each picture set, analyze the visual word dictionary corresponding to each picture, if the visual word dictionary satisfies the visual word dictionary library Establish conditions, then the collection of visual word dictionaries corresponding to various types of picture collections form a visual word dictionary library; Among them: the conditions for establishing the visual word dictionary database are: After division, the number of visual words in the visual word dictionary of each picture collection is less than or equal to the total number of visual words in the visual word dictionary of the server database; And the probability distribution of the visual words in the picture set is counted, and the entropy of the probability distribution of the visual words is calculated, and the information entropy of the probability distribution is less than the set threshold. 4. The image search method according to claim 3, characterized in that: The visual word dictionary is: the visual features of the picture collection are clustered to establish the original visual word dictionary of the picture; or, The visual word dictionary of the picture is established by clustering the visual features of the picture collection, based on the screening rules of the effective visual word dictionary, the effective visual word dictionary representing the original visual word dictionary is determined, and the effective visual word dictionary is used as the visual word dictionary. 5. The image search method according to claim 3, characterized in that: The steps of dividing the pictures in the server database into various types of picture sets by using the picture set division method include: Divide all pictures into multiple picture sets by utilizing the visual similarity between pictures; or, Divide all pictures into multiple picture collections using information about pictures; or All the pictures are divided into multiple sets by using the visual similarity between the pictures and the information related to the pictures. 6 . The image search method according to claim 5 , wherein the information related to the image includes the date when the image was taken, a text label, and an electronic label. 7. The image search method according to claim 1, characterized in that: In the step of using the prediction loss function to calculate the prediction loss value of each visual word dictionary in the visual word dictionary library to the visual word of the target picture, the calculation method of the prediction loss value is as follows: the cosine distance between the visual word of the target image and the class center of the image class in which the target visual word dictionary is located; or The cosine distance between the visual word of the target picture and the class center of the picture class where the target visual word dictionary is located, and the weighted sum of the Euclidean distances between the related information and the same kind of information of the picture class where the visual word dictionary is located; or The product of the visual similarity distance between the target picture and the picture class of the visual word dictionary of the target visual word dictionary, and the Euclidean distance between the related information and the same kind of information of the picture class of the visual word dictionary. 8. The image search method according to claim 4, wherein: Based on the screening rules of the effective visual word dictionary, the steps of determining a visual word dictionary representative of the original visual word dictionary include: Selecting a certain number of pictures from a certain type of pictures as sample pictures, and converting the features of the sample pictures into visual words in the original visual word dictionary; Query in the visual word index table of the original visual word dictionary according to the visual word of the sample picture, and obtain the original query result; Combining any visual words belonging to the original visual word dictionary to form a screening visual word dictionary, based on the screening visual word dictionary, converting the features of the sample picture into corresponding first visual words in the screening visual word dictionary, and Use the first visual word to query in the visual word index table of the original visual word dictionary, and obtain the first query result corresponding to the filtered visual word dictionary; Analyze the original query result and the first query result of all sample pictures, if the first query result matches the original query result, then adopt the current screening visual word dictionary as the visual word dictionary; otherwise select from the original visual word dictionary A visual word is added to the current filtered visual word dictionary, and the process returns to the step of obtaining the first query result. 9. The image search method according to claim 1, further comprising: The server receives the encoded target visual word and decodes the target visual word; The server searches the index table corresponding to the visual word dictionary in the server based on the target visual word to obtain the result picture and/or the relevant information of the result picture, and sends it to the client; The visual word dictionary is: a visual word dictionary established by clustering the visual features of all or part of the pictures in the server-side picture database.

Images