WO2017000109A1

WO2017000109A1 - Search method, search apparatus, user equipment, and computer program product

Info

Publication number: WO2017000109A1
Application number: PCT/CN2015/082628
Authority: WO
Inventors: 姚聪; 周舒畅; 周昕宇; 吴育昕
Original assignee: 北京旷视科技有限公司; 北京小孔科技有限公司
Priority date: 2015-06-29
Filing date: 2015-06-29
Publication date: 2017-01-05
Also published as: CN105518678A; CN105518678B

Abstract

A search method, a search apparatus, a user equipment, and a computer program product. The search method used for a server comprises: receiving a search request, the search request comprising a target image of a target object to be searched for (S210); extracting character information and image characteristic associated with the target object from the target image (S220); searching for related object information associated with the target object according to the character information and the image characteristic (S230); and sending the related object information (S240). Related object information of a target object can be accurately and conveniently searched for, thereby improving use experience of users.

Description

Search method, search device, user equipment, and computer program product

Technical field

The present disclosure relates to the field of information technology, and more particularly, to a search method, a search device, a user equipment, and a computer program product.

Background technique

With the development of the Internet and the popularity of user equipment, e-commerce based on user equipment has been booming in recent years. Searching and purchasing goods on the Internet through user devices has become a common activity in people's daily lives. The user equipment is, for example, a smart phone, a tablet computer, a notebook computer, or the like. The Internet provides a new trading platform and entertainment platform. For example, you can buy goods on the Internet, download music, watch videos online, and more.

Typically, the search can be performed on the Internet based on the keywords of the object to be searched. The keyword-based object search system relies on the character representation entered by the user. However, when the keyword input by the user is inaccurate or there is an error, it is difficult to obtain a satisfactory search result. As the scale of e-services such as e-commerce continues to expand and the number and variety of goods or services grows rapidly, consumers may need to spend more time browsing to find objects or products of interest.

Therefore, it is desirable to provide a search technology to help users accurately search for goods or services of interest, and provide richer information and more detailed services, thereby improving the user experience.

Summary of the invention

Embodiments of the present disclosure provide a search method, a search device, a user device, and a computer program product, which enable accurate and convenient searching of related object information of a target object, thereby improving a user's use experience.

In a first aspect, a search method is provided, which is applied to a server, the search method may include: receiving a search request, the search request including a target image of a target object to be searched; and extracting the target from the target image Character information and image features associated with the object; searching for related object information associated with the target object based on the character information and the image feature; transmitting the related object information.

In conjunction with the first aspect, in an implementation of the first aspect, the extracting the character information and the image feature associated with the target object from the target image may include: utilizing optical character recognition The OCR identifies a character and a symbol from the target image; and selects an identification character for identifying the target object from the recognized characters and symbols as character information associated with the target object.

In conjunction with the first aspect and the foregoing implementation manner, in another implementation manner of the first aspect, the searching for related object information associated with the target object based on the character information and the image feature may include: The character information and the image feature search for the related object information from a pre-established object database, wherein the object database includes image features, character information, and associated information of each candidate object.

In conjunction with the first aspect and the foregoing implementation manner, in another implementation manner of the first aspect, the searching for the related object information from the pre-established object database based on the character information and the image feature may include: Calculating image feature similarity between the target object and each candidate object by image features of the target image and image features of the respective candidate objects; calculating the target object based on character information of the target image and character information of each candidate object Character information similarity with each candidate object; searching for related object information associated with the target object from the plurality of candidate objects based on the image feature similarity and the character information similarity.

In combination with the first aspect and the foregoing implementation manner, in another implementation manner of the first aspect, the searching for the location and the location information based on the image feature similarity and the character information similarity The related object information associated with the target object may include: performing weighted averaging on the image feature similarity and the character information similarity to obtain an average similarity between the target object and each candidate object; according to the average similarity The descending order of degrees selects a predetermined number of candidate objects from the plurality of candidate objects; information corresponding to the selected candidate objects is used as related object information associated with the target object.

In conjunction with the first aspect and the foregoing implementation manner, in another implementation manner of the first aspect, the image feature based on the target image and the image feature of each candidate object are used to calculate between the target object and each candidate object. The image feature similarity may include calculating a cosine similarity between the image feature of the target object and an image feature between the respective candidate objects as the image feature similarity.

In conjunction with the first aspect and the foregoing implementation manner, in another implementation manner of the first aspect, the character information based on the target image and the character information of each candidate object are calculated between the target object and each candidate object. The character information similarity may include: calculating an edit distance between the character information of the target object and the character information of each candidate object; based on the edit distance, the length of the character information of the target object, the character of the candidate object The length of the information is used to calculate the similarity of the character information.

In conjunction with the first aspect and the foregoing implementation manner, in another implementation of the first aspect, the extracting the character information and the image feature associated with the target object from the target image may include at least one of the following operations One: calculating a color histogram feature of the target image as the image feature; and calculating a word bag model feature of the target image as the image feature.

In conjunction with the first aspect and the above implementation thereof, in another implementation of the first aspect, the target image may satisfy a predetermined condition.

In a second aspect, a search method is provided for application to a user equipment. The search method may include: collecting a target image of the target object to be searched; determining whether the target image satisfies a predetermined condition; and when the target image satisfies a predetermined condition, issuing a search request for the target object, the search request including The target image; receiving related object information associated with the target object, wherein the related object information is obtained based on character information and image feature search associated with the target object extracted from the target image.

With reference to the second aspect, in an implementation manner of the second aspect, the determining whether the target image meets the predetermined condition may include: determining an illumination parameter in the process of acquiring the target image; and when the illumination parameter is greater than or equal to When the illuminance is preset, it is determined that the target image satisfies a predetermined condition.

With reference to the second aspect and the foregoing implementation manner, in another implementation manner of the second aspect, the determining whether the target image meets a predetermined condition may include: determining an average gradient of pixel points of an edge of the collected target image; When the average gradient of the pixel points of the edge of the target image is less than the preset gradient threshold, it is determined that the target image satisfies a predetermined condition.

In a third aspect, a search device is provided for use in a server. The search device can include a transceiver that receives a search request, the search request including a target image of a target object to be searched, a processor, a memory, and computer program instructions stored in the memory. Performing the steps of: extracting, from the target image, character information and image features associated with the target object when the computer program instructions are executed by the processor; searching and said based on the character information and image features Relevant object information associated with the target object; the searched related object information is provided to the transceiver for transmission.

In conjunction with the third aspect, in an implementation of the third aspect, the extracting the character information and the image feature associated with the target object from the target image may include: using the optical character recognition OCR from the target Identifying characters and symbols in the image; selecting an identification character for identifying the target object from the recognized characters and symbols as character information associated with the target object.

With reference to the third aspect and the foregoing implementation manner, in another implementation manner of the third aspect, Searching for related object information associated with the target object based on the character information and the image feature may include: searching for the related object information from a pre-established object database based on the character information and the image feature, wherein The object database includes image features, character information, and associated information of each candidate object.

With reference to the third aspect and the foregoing implementation manner, in another implementation manner of the third aspect, the searching for the related object information from the pre-established object database based on the character information and the image feature may include: Calculating image feature similarity between the target object and each candidate object by image features of the target image and image features of the respective candidate objects; calculating the target object based on character information of the target image and character information of each candidate object Character information similarity with each candidate object; searching for related object information associated with the target object from the plurality of candidate objects based on the image feature similarity and the character information similarity.

In conjunction with the third aspect and the foregoing implementation manner, in another implementation manner of the third aspect, the searching, searching, and searching from the multiple candidate objects based on the image feature similarity and the character information similarity The related object information associated with the target object may include: performing weighted averaging on the image feature similarity and the character information similarity to obtain an average similarity between the target object and each candidate object; according to the average similarity The descending order of degrees selects a predetermined number of candidate objects from the plurality of candidate objects; information corresponding to the selected candidate objects is used as related object information associated with the target object.

In conjunction with the third aspect and the foregoing implementation manner, in another implementation manner of the third aspect, the image feature based on the target image and the image feature of each candidate object are calculated between the target object and each candidate object. The image feature similarity may include calculating a cosine similarity between the image feature of the target object and an image feature between the respective candidate objects as the image feature similarity.

In conjunction with the third aspect and the foregoing implementation manner, in another implementation manner of the third aspect, the character information based on the target image and the character information of each candidate object are calculated between the target object and each candidate object. The character information similarity may include: calculating an edit distance between the character information of the target object and the character information of each candidate object; based on the edit distance, the length of the character information of the target object, and the candidate objects The length of the character information is used to calculate the similarity of the character information.

In conjunction with the third aspect and the above implementation thereof, in another implementation of the third aspect, extracting the character information and the image feature associated with the target object from the target image may include at least one of the following operations: Calculating a color histogram feature of the target image as the image feature; And calculating a word bag model feature of the target image as the image feature.

In conjunction with the third aspect and the above implementation thereof, in another implementation of the third aspect, the target image may satisfy a predetermined condition.

In a fourth aspect, a user equipment is provided. The user equipment may include: an image collector for acquiring a target image of the target object to be searched; a processor for determining whether the target image satisfies a predetermined condition; and a transceiver, when the target image satisfies a predetermined condition, The issuing a search request for the target object, the search request including the target image, and receiving related object information associated with the target object, wherein the related object information is based on extraction from the target image Character information and image feature search associated with the target object are obtained.

In conjunction with the fourth aspect, in an implementation manner of the fourth aspect, the user equipment may further include an illuminometer for measuring an illumination parameter of the target object, the processor may instruct the illuminometer to be in image collection The illumination parameter of the target object is measured during the process of acquiring the target image, and determining that the target image satisfies a predetermined condition when the illumination parameter is greater than or equal to the preset illumination.

In conjunction with the fourth aspect and the above implementation manner, in another implementation of the fourth aspect, the processor analyzes the target image to determine an average gradient of pixel points of the edge thereof, and a pixel point at an edge of the target image When the average gradient is less than the preset gradient threshold, it is determined that the target image satisfies a predetermined condition.

In a fifth aspect, a computer program product for searching for an object is provided, which can include a computer readable storage medium. Storing computer program instructions on the computer readable storage medium, the computer program instructions being executed by a processor to cause the processor to: receive a search request, the search request including a target image of a target object to be searched; Extracting character information and image features associated with the target object in the target image; searching for related object information associated with the target object based on the character information and the image feature; and transmitting the related object information.

In a sixth aspect, a computer program product for searching for an object is provided, which can include a computer readable storage medium. Computer program instructions are stored on the computer readable storage medium. The computer program instructions may be executed by a processor to cause the processor to: acquire an object image of a target object to be searched using an image collector; determine whether the target image satisfies a predetermined condition; and when the target image satisfies a predetermined condition Transmitting, by the transceiver, a search request for the target object, the search request including the target image; and receiving, by the transceiver, related object information associated with the target object, wherein the related object information is based on the The character information and image feature associated with the target object extracted in the target image are searched.

In a technical solution for a search method, a search device, and a computer program product for a server according to an embodiment of the present disclosure, character information and image features associated with the target object are extracted from a target image of a target object to be searched for The search is performed based on the character information and the image feature, and the related object information of the target object can be searched accurately and conveniently, thereby improving the user experience.

In a technical solution for a search method for a user equipment, the user equipment, and a computer program product according to an embodiment of the present disclosure, when a target image of the collected target object satisfies a predetermined condition, a search request is issued based on the target image, so that It can accurately and conveniently search related object information of the target object, thereby improving the user experience.

DRAWINGS

In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings used in the embodiments or the description of the prior art will be briefly described below. Obviously, the drawings in the following description are only some of the disclosure. For the embodiments, other drawings can also be obtained from those skilled in the art based on these drawings.

FIG. 1(a) schematically illustrates an application scenario according to an embodiment of the present disclosure;

Figure 1 (b) schematically illustrates a schematic diagram of a target image taken by a user device;

2 is a flow chart that schematically illustrates a search method for a server in accordance with an embodiment of the present disclosure;

3 is a flow chart schematically illustrating related object information of a target object based on image features and character information in the search method of FIG. 2;

4 is a flow chart that schematically illustrates a search method for a user equipment in accordance with an embodiment of the present disclosure;

FIG. 5 is a block diagram schematically illustrating a first search device according to an embodiment of the present disclosure; FIG.

FIG. 6 is a block diagram schematically illustrating a second search device for a server according to an embodiment of the present disclosure; FIG.

FIG. 7 is a block diagram schematically illustrating a user equipment in accordance with an embodiment of the present disclosure.

detailed description

The technical solutions in the embodiments of the present disclosure are clearly and completely described in the following with reference to the accompanying drawings in the embodiments of the present disclosure. It is obvious that the described embodiments are a part of the embodiments of the present disclosure, and not all of the embodiments. All of those obtained by those of ordinary skill in the art based on the embodiments of the present disclosure The embodiments thereof are all within the scope of the protection of the present disclosure.

FIG. 1(a) schematically illustrates an application scenario in accordance with an embodiment of the present disclosure. As shown in FIG. 1(a), the user equipment 10 is communicatively coupled to the search server 20 via a network. The user device 10 is, for example, a smart phone, a tablet computer, a notebook computer, or the like. The search server 20 is a cloud server, a web server, or the like. Communication between user device 10 and search server 20 may be implemented using a variety of techniques including, but not limited to, the Internet, local area networks, third generation mobile communication technologies, and the like. For example, a user of a user device browses a Taobao web page to expect to purchase a particular item, ie, a target object. The user equipment is connected to the search server of Taobao via the Internet.

Generally, the user inputs a keyword of the product to be purchased in the Taobao webpage of the user equipment, and the user equipment transmits the keyword to the search server of Taobao via the Internet, and the latter performs a search based on the keyword and via the Internet. Sending the search result to the user equipment. When the keyword input by the user is inaccurate or there is an error, it is difficult to obtain a satisfactory search result. Moreover, due to the large number and variety of goods or services, a plurality of items associated with keywords may be included in the search results, which may make it impossible for the user to find the target object to be purchased from the search results.

In the embodiment of the present disclosure, the user equipment 10 performs image acquisition on a target object to be purchased using a camera or the like, and transmits the collected target image to the search server 20. The search server 20 extracts character information and/or image information from the target image, and performs a search based on the extracted information, and transmits the search result to the user device via the Internet. In the target image, it usually carries rich information about the target object, such as the appearance, name, trademark, manufacturer, date of manufacture, and the like of the target object. Based on the rich information in the target image, the search server can more accurately search for the target object of the user, thereby improving the accuracy of the search. In addition, the search server can automatically extract information in the target image without requiring the user to manually input keywords or the like, which makes the user's search operation more convenient.

FIG. 1(b) schematically illustrates a schematic diagram of a target image taken by the user device 10. As shown in Fig. 1(b), the captured target images (1), (2), and (3) are respectively Evian mineral water, calbee potato chips, and blue moon laundry liquid. The target image (1) includes information on the appearance of the shape of the bottled water, the name of the evian, the shape of the mountain, the capacity of 550 ml, etc., based on the information, the search server 20 can accurately search for the target object of the user. However, if the user enters the keyword "Evian mineral water", it will search for Evian mineral water in different packaging, different series and different capacities. Similarly, the target image (2) of Fig. 1(b) also includes a wealth of information: for example, the brand name "Carle B", the product content "Potato Chips", the taste series "Barbecue", bags Appearance, capacity "90g", etc.; the target image (3) of Figure 1(b) also contains a wealth of information: for example, the brand name "blue moon", the product content "laundry liquid", the barreled product Appearance, capacity "2kg", product series "clear clove" and so on. Based on the rich information contained in the target image, the search server 20 can accurately search for each target object.

2 is a flow chart that schematically illustrates a search method 200 for a server in accordance with an embodiment of the present disclosure. The search method 200 is applicable to a search server as shown in FIG. 1(a). As shown in FIG. 2, the search method 200 may include receiving a search request including a target image of a target object to be searched (S210); extracting characters associated with the target object from the target image Information and image features (S220); searching for related object information associated with the target object based on the character information and the image feature (S230); transmitting the related object information (S240).

In S210, the server receives a search request from the user device, the search request including a target image of the target object to be searched. The target image is any one of the target images as shown in FIG. 1(b). The target image contains various information of the target object to be searched, including but not limited to brand name, object content, series, appearance, capacity, production date, and the like. The target image may be collected by the user equipment by using the image collection device, or may be received by the user equipment from other electronic devices. The manner in which the user equipment acquires the target image does not constitute a limitation on the embodiments of the present disclosure.

As described in connection with FIG. 1(a), the server extracts information from the target image to search for a target object. Accordingly, the image quality of the target image will directly affect the search results. For example, in the target image (1) of FIG. 1(b), if the target image is blurred and information such as the brand name evian, capacity, and the like cannot be extracted, it is difficult to accurately search for the target object. Therefore, a request can be made for the target image, for example, the target image satisfies a predetermined condition. The predetermined condition may be a condition regarding the brightness of the target image or a condition regarding the sharpness of the target image.

For example, when the brightness of the target image is greater than or equal to the preset brightness threshold, it is determined that the target image satisfies a predetermined condition; when the brightness of the target image is less than the preset brightness threshold, it is determined that the target image does not satisfy the predetermined condition. Alternatively, when the sharpness of the target image is greater than or equal to the preset sharpness threshold, it is determined that the target image satisfies a predetermined condition; when the brightness of the target image is less than the preset sharpness threshold, it is determined that the target image does not satisfy the predetermined condition. The preset brightness threshold or the preset definition threshold may be adjusted according to the processing capability of the server. For example, when the processing capability of the server is strong, the preset brightness threshold or the preset definition threshold may be set to a lower value; when the processing capability of the server is weak, the preset brightness threshold or the preset definition threshold may be set. Is a higher value.

In S220, character information associated with the target object is extracted from the target image and Image features. The character information included in the target image is, for example, a product name, a capacity, a brand name, a date of manufacture, and the like, and the character information is a character or a symbol. The image special diagnosis included in the target image is a color component of the image, a composition ratio of each color component, and the like. Typically, different techniques are employed to extract character information and image features in the target image.

For character information, the character information in the target image can be extracted by using Optical Character Recognition (OCR) technology. In OCR technology, the server determines its shape by detecting the dark and bright patterns of the target image, and then uses the character recognition method to translate the shape into computer text. Alternatively, other techniques may be employed to perform character recognition on the target image to obtain character information therein.

Character information associated with the target object may be extracted from the target image by recognizing words and symbols from the target image using optical character recognition OCR; selecting for identification from the identified characters and symbols The identification character of the target object is the character information associated with the target object. As mentioned before, rich information is included in the target image, and some of the information may be closely related to the search of the target object, such as product name, brand, capacity, and the like. However, the target image may also include information that is not related to the search of the target object, such as components, security reminders, etc., which may be information related to all similar products, which cannot be used to identify the target object. Therefore, after performing character recognition on the target image, it is necessary to filter out information required for searching the target object, that is, an identification character for identifying the target object.

Image features are index-valued image feature representations, such as using vectors to represent image characteristics. The image features of the target image may be represented in various ways that are present or appearing in the future. Here, a color histogram and a Bag of Words feature are taken as an example of an image feature. It is to be noted that, in the application, any one of the color histogram and the word bag model feature may be used to represent the image feature of the target image, and both the color histogram and the bag model feature may be used to represent the image feature of the target image. That is, the extracting the image object associated with the target object from the target image includes at least one of: calculating a color histogram feature of the target image as the image feature; and calculating a location The word bag model feature of the target image is used as the image feature.

A color histogram is a statistical representation of the color characteristics of an image that is used to represent the proportion of different colors in the entire target image, without concern for the spatial location of each color. Color histograms are closely related to how color space is represented. Common color histograms include RGB spatial color histograms, HSV spatial color histograms, and Lab space color histograms. In different color spaces The color histogram of the target image has different values.

The word bag model feature is a statistical representation of the texture features of an image that can effectively describe the overall and local characteristics of the image. For example, the word bag model feature of the target image can be obtained by extracting feature descriptors from the target image, such as Scale Invariant Feature Transform (SIFT), Directional Histogram (HOG, Histogram of Oriented Gradient). ); for each descriptor, search for the most similar cluster center in the pre-accurate codebook, and count the frequency of occurrence of different cluster centers in the target image to form a histogram; The processing is performed to obtain the word bag model feature of the target image. The pre-accurate codebook can be obtained by randomly extracting a large number of image descriptors (for example, SIFT, HOG, etc.) from the set of training images, and clustering the image descriptors by using a clustering algorithm to obtain multiple Category, all the categories obtained by clustering constitute the codebook.

In S230, related object information associated with the target object is searched based on the character information and the image feature obtained in S220. Specifically, the related object information is searched from a pre-established object database based on the character information and the image feature. The object database includes image features, character information, and associated information of respective candidate objects.

Assuming that the object database P contains N objects, each object p _j can be represented by a triplet {f _I (p _j ), f _T (p _j ), a(p _j )}, where j=1, 2 ,...,N. f _I (p _j ) represents an image feature of the object p _j , which may be a color histogram feature, or a bag model feature, or a vector composed of color histogram features and word bag model features. f _T (p _j ) is character information of the object p _j , which is typically a character string such as a name, a brand, a content, and the like. a(p _j ) represents other associated information associated with the object p _j , such as price, sales volume, user rating, promotional video, and hyperlinks. Alternatively, each object p _j can also be represented by a binary group {f _I (p _j ), f _T (p _j )}. It is assumed that the image feature and the character information of the target image of the target object q to be searched are f _I (q) and f _T (q), respectively, and accordingly, the character information f _T (q) and the image obtained in S220 can be obtained. The feature f _I (q) is compared with the character information f _T (p _j ) of each candidate object in the object database P and the image feature f _I (p _j ) to perform a search.

FIG. 3 is a flowchart schematically illustrating related object information (S230) of searching for a target object based on image features and character information in the search method of FIG. 2. As shown in FIG. 3, image feature similarity between the target object and each candidate object is calculated based on image features of the target image and image features of the respective candidate objects (S231); character information based on the target image and each device Selecting character information of the object to calculate a character information similarity between the target object and each candidate object (S232); performing weighted averaging on the image feature similarity and the character information similarity to obtain the target object and each device Selecting an average similarity between the objects (S233); selecting a predetermined number of candidate objects from the plurality of candidate objects in descending order of the average similarity (S234); corresponding to the selected candidate objects The information is related object information associated with the target object (S235). The following describes the target object q and the object database P including N objects p _j as an example.

In S231, a cosine similarity s _I (q, p _j ) between the image feature f _I (q) of the target object q and the image feature f _I (p _j ) between the respective candidate objects p _j can be calculated. ) as the image feature similarity. The cosine similarity s _I (q, p _j ) can be calculated by the following formula (1):

Formula 1),

Where ||f _I (q)|| is the modulus of the image feature f _I (q), and ||f _I (p _j )|| is the modulus of the image feature f _I (p _j ). The cosine similarity shown in equation (1) is only a representation of the similarity of image features. In practice, other functions may also be adopted to represent the image feature similarity. For example, a Pearson correlation coefficient between the image feature of the target object and the image feature between each candidate object may be taken as the image feature similarity. .

In S232, the character information similarity between the target object q and each candidate object p _j may be calculated as follows: calculating character information f _T (q) of the target object q and each candidate object p _j An edit distance d(f _T (q), f _T (p _j )) between the character information f _T (p _j ); based on the edit distance, the length of the character information f _T (q) of the target object, The character information similarity is calculated by the length of the character information f _T (p _j ) of the candidate object. Edit distance is the minimum number of edit operations required to convert from one string to another between two strings. The allowed editing operations include replacing one character with another, inserting a character, and deleting One character. Therefore, the edit distance d(f _T (q), f _T (p _j )) is the minimum number of editing operations required to convert the character information f _T (q) into the character information f _T (p _j ). Character information length f _T (q), for example, the number of characters and symbols included in the character information f _T (q) in. Alternatively, the length of the object character information f _T (p _j), for example, the number of characters and symbols included in the character information f _T (p _j) in. For example, the character information similarity s _T (q, p _j ) can be calculated by the following formula (2):

Formula (2)

Where d(f _T (q), f _T (p _j )) is the edit distance between the character information f _T (q) and the character information f _T (p _j ), and L(f _T (q)) is a character The length of the information f _T (q), L(f _T (p _j )), is the length of the character information f _T (p _j ).

In S233, the image feature similarity s _I (q, p _j ) and the character information similarity s _T (q, p _j ) are weighted and averaged to obtain an average between the target object and each candidate object. Similarity. For example, the average similarity s(q, p _j ) can be calculated by the following formula (3):

s(q,p _j )=ω·s _I (q,p _j )+(1-ω)s _T (q,p _j ) Equation (3),

Where ω is the weight coefficient. The weight coefficient ω is an adjustable parameter, and its value range is [0, 1], and the typical value is ω=0.6. When the weight coefficient ω increases, the image feature similarity s _I (q, p _j ) increases in the average similarity, and the character information similarity s _T (q, p _j ) decreases in the average similarity. When the weight coefficient ω decreases, the image feature similarity s _I (q, p _j ) decreases in the average similarity, and the character information similarity s _T (q, p _j ) increases in the average similarity.

In S234, a predetermined number of candidate objects are selected from the plurality of candidate objects in descending order of the average similarity s(q, p _j ). In S233, the average similarity s(q, p _j ) between the target object and each candidate object is calculated, j=1, 2, . . . , N, that is, N average similarities are obtained, for which N The average similarities may be arranged in descending order, and for example, a predetermined number of R candidate objects with an average degree of similarity are selected, and the R candidate objects are search results. The average similarity between the R candidate objects and the target object is high, indicating that the R candidate objects are closer to the target object, so that there is a larger target object that the user desires. R is a configurable parameter, and its typical value can be set to 10, 20, 100, and so on.

In S235, information corresponding to the selected R candidate objects is taken as related object information associated with the target object. The picture, the character description, the related information, and the like of the R objects are used as related object information. The related information is, for example, price, sales volume, user rating, promotional video, hyperlink, and the like.

In the above S233 to S235, related object information associated with the target object is searched from the plurality of candidate objects based on the image feature similarity and the character information similarity. Replacing the S233 to S235, for example, the related object information may be searched in such a manner that R1 candidate objects are selected from the plurality of candidate objects in descending order of image feature similarity; in descending order of similarity of character information R2 candidate objects are selected from the plurality of candidate objects; information corresponding to the selected R1 candidate objects and R2 candidate objects is used as related object information associated with the target object. R1 is a natural number smaller than N. R2 is also a natural number smaller than N.

In S240, the server sends the searched related object information as a search result to the user equipment. The server can transmit the correlation by using various networks or communication technologies such as the Internet and a local area network. Object information. The related object information is, for example, a picture, a text description, and associated information of the R candidate objects, or a picture, a text description, and associated information of the R1 plus R2 candidate object described above. After receiving the related object information, the user equipment may display the related object information on the screen of the user equipment for the user to view.

In a technical solution for a search method of a server according to an embodiment of the present disclosure, character information and image features associated with the target object are extracted from a target image of a target object to be searched based on the character information and The image feature performs a search, and can accurately and conveniently search related object information of the target object, thereby improving the user experience. In addition, the step of manually inputting a keyword by the user is eliminated by automatically recognizing the character information contained in the target image.

4 is a flow chart that schematically illustrates a search method 400 for a user device in accordance with an embodiment of the disclosure. The search method 400 is applied to the user equipment shown in FIG. 1(a). As shown in FIG. 4, the search method 400 may include: collecting a target image of a target object to be searched (S410); determining whether the target image satisfies a predetermined condition (S420); and when the target image satisfies a predetermined condition, issuing a search request for the target object, the search request including the target image (S430); receiving related object information associated with the target object (S440), wherein the related object information is based on from the target image The extracted character information and image feature search associated with the target object are obtained.

In S410, the image capturing device in the user device may be utilized to collect the target image of the target object to be searched. For example, if the blue moon laundry liquid in the user's FIG. 1(b) is exhausted and it is desired to purchase the blue moon laundry liquid, the user utilizes an image capturing device built in the user device 10 or an image acquisition connected to the user device. The device performs image acquisition on the existing Blue Moon laundry detergent. The positional relationship between the image capture device and the user equipment does not constitute a limitation on the embodiments of the present disclosure.

In S420, it is determined whether the target image satisfies a predetermined condition. Since the server is to extract information from the target image to search for the target object, the image quality of the target image will directly affect the search result. Taking the target image (1) of FIG. 1(b) as an example, if the target image is blurred and information such as the brand name evian, capacity, etc. cannot be extracted, it is difficult to accurately search for the target object. A requirement may be made for the target image at S420, for example, the target image satisfies a predetermined condition. The predetermined condition may be a condition regarding the brightness of the target image or a condition regarding the sharpness of the target image.

As an example of judging whether or not the predetermined condition is satisfied based on the brightness of the target image, the target image acquired in S410 may be converted into image data of the HSL color space in which the luminance information is included in the image data of the HSL color space. Then, the average value of the illumination components (ie, L components) of all pixels in the image data of the HSL color space is counted

Average value of the illumination component used for the target image

When the predetermined brightness threshold T _L is greater than or equal to, it may be determined that the target image satisfies a predetermined condition. Average value of the illumination component used for the target image

When it is less than the predetermined brightness threshold T _L , it may be judged that the target image does not satisfy the predetermined condition. The predetermined brightness threshold T _{L is} typically 64. Alternatively, the quality of the target image can be indirectly determined by measuring the lighting conditions in the image acquisition environment. For example, the illumination parameter in the process of acquiring the target image may be determined; when the illumination parameter is greater than or equal to the preset illumination, determining that the target image satisfies a predetermined condition; when the illumination parameter is less than the preset illumination, determining the location The target image does not satisfy the predetermined condition.

As an example of judging whether or not a predetermined condition is satisfied based on the sharpness of the target image, an edge of the target image acquired in S410 may be extracted using a predetermined algorithm (for example, Canny algorithm) in S420, and each of the edges located in the target image is calculated. The gradient G of the pixel, and then further calculate the average of the gradients of all the pixel points at the edge in the target image

The average of the gradients of all pixel points at the edge of the target image

When it is greater than or equal to the preset gradient threshold T _G , it may be determined that the target image satisfies a predetermined condition. The average of the gradients of all pixel points at the edge of the target image

When it is less than the preset gradient threshold T _G , it may be determined that the target image does not satisfy the predetermined condition. The preset gradient threshold T _{G is} typically 100.

The predetermined brightness threshold T _L or the preset gradient threshold T _G described above may be adjusted according to the processing capability of the server performing the search. For example, when the processing capability of the server is strong, the predetermined brightness threshold T _L or the preset gradient threshold T _{G may be} set to a lower value; when the processing capability of the server is weak, the predetermined brightness threshold T _L or a preset gradient may be used. The threshold T _{G is} set to a higher value.

When it is judged in S420 that the target image satisfies the predetermined condition, a search request for the target object is issued in S430, the search request including the target image. Then, the search server 20 as shown in FIG. 1(a) extracts character information and image features associated with the target object from the target image, and performs a search based on the character information and the image feature, ie, The various steps of the search method described in connection with FIG. 2 are performed. Since the brightness or sharpness of the target image is good, the character information and the image feature can be accurately extracted in the server, thereby ensuring the accuracy of the search.

When it is judged in S420 that the target image does not satisfy the predetermined condition, it means that the target image acquired in S410 does not meet the requirement, which may make it difficult to accurately extract the character information and the image feature therein. At this time, a retake prompt message may be output in the user equipment to prompt re-execution S410 to collect the target image of the target object to be searched. In the retake prompt message, it is also possible to specifically list the reason why the target image does not satisfy the predetermined condition. For example, the average of the illumination components of the target image

When less than the predetermined brightness threshold T _L , the brightness may be indicated in the replay prompt message; the average of the gradients of all the pixel points located at the edge of the target image

When the preset gradient threshold T _G is smaller than the preset gradient threshold T _G , it is indicated that the sharpness is insufficient in the replay prompt message. In this way, the shooting of the target image can be adjusted according to the replay prompt message until the target image that satisfies the predetermined condition is acquired. Alternatively, when it is judged in S420 that the target image does not satisfy the predetermined condition, the setting parameters of the image pickup device may be automatically adjusted in accordance with the determination result of S420 until the target image satisfying the predetermined condition is acquired.

After the user device issues a search request to the server in S430, the server performs the search method described in connection with FIGS. 2 and 3 and obtains related object information associated with the target object. That is, the related object information is obtained based on character information and image feature search associated with the target object extracted from the target image. Correspondingly, the user equipment receives relevant object information associated with the target object in S440. The user equipment can receive the related object information from the server by using various networks or communication technologies such as the Internet and a local area network. The related object information is, for example, a picture of a plurality of candidate objects, a text description, and associated information. The associated information is, for example, price, sales volume, user rating, promotional video, hyperlinks, etc., which assists the user in performing selection operations among a plurality of candidate objects. After receiving the related object information, the user equipment may display the related object information on the screen of the user equipment for the user to view.

Therefore, in the process of the user device taking an image of the target object, the user device can automatically calculate the illumination condition and the degree of clarity of the image. If the lighting conditions and clarity of the image meet the requirements, the user device is allowed to issue a search request based on the acquired target image. If the lighting conditions and clarity of the image do not meet the requirements, the user equipment is prompted or automatically instructed to re-shoot until the desired target image is obtained.

In a technical solution for a search method of a user device according to an embodiment of the present disclosure, when a target image of the acquired target object satisfies a predetermined condition, a search request is issued based on the target image, so that the target object can be searched accurately and conveniently Relevant object information, thereby improving the user experience.

FIG. 5 is a block diagram schematically illustrating a first search device 500 in accordance with an embodiment of the present disclosure. The first search device 500 can be applied to a user equipment or server. As shown in FIG. 5, the first data processing apparatus 500 may include one or more processors 510, a storage unit 520, an input unit 530, an output unit 540, a communication unit 550, and an image acquisition unit 560. These components are interconnected by a bus system 570 and/or other form of connection mechanism (not shown). It should be noted that the first search shown in Figure 5 The components and structures of the device 500 are merely exemplary and not limiting. The first search device 500 may also have other components and structures as needed, and may, for example, not include the input unit 530, the output unit 540, and the image acquisition unit 560. Wait.

Processor 510 can be a central processing unit (CPU) or other form of processing unit with data processing capabilities and/or instruction execution capabilities, and can control other components in first search device 500 to perform desired functions.

Storage unit 520 can include one or more computer program products, which can include various forms of computer readable storage media, such as volatile memory and/or nonvolatile memory. The volatile memory may include, for example, a random access memory (RAM) and/or a cache or the like. The nonvolatile memory may include, for example, a read only memory (ROM), a hard disk, a flash memory, or the like. One or more computer program instructions may be stored on the computer readable storage medium, and the processor 510 may execute the program instructions to implement various of the search methods described above in connection with FIGS. 2 and 3 of embodiments of the present disclosure. Step, at this time, the first search device 500 can be included in the server. Alternatively, the processor 510 can execute the program instructions to implement the various steps of the search method described above in connection with FIG. 4 of the embodiments of the present disclosure, at which time the first search device 500 can be included in the user equipment. Various applications and various data such as an operating state of the display screen, an operational state of the application, and the like can also be stored in the computer readable storage medium.

The input unit 530 may be a unit used by a user to input an instruction, and may include one or more of a keyboard, a mouse, a microphone, a touch screen, and the like. The output unit 540 may output various information (such as an image or a sound) to the outside (for example, a user), and may include one or more of a display, a speaker, and the like. Communication unit 550 can communicate with other units (e.g., personal computers, servers, mobile stations, base stations, etc.) via a network or other technology, which can be the Internet, a wireless local area network, a mobile communication network, and the like.

In the technical solution of the first search device 500 of the embodiment of the present disclosure, the character information and the image feature associated with the target object are extracted from the target image of the target object to be searched, based on the character information and the image feature. The search can accurately and conveniently search for related object information of the target object, thereby improving the user experience.

FIG. 6 is a block diagram schematically illustrating a second search device 600 for a server in accordance with an embodiment of the present disclosure. The second search device 600 is applicable to a search server as shown in FIG. 1(a). As shown in FIG. 6, the second search device 600 may include a first receiving unit 610, an extracting unit 620, and a search. Unit 630 and first transmitting unit 640.

The first receiving unit 610 receives a search request including a target image of a target object to be searched for. The target image is any one of the target images as shown in FIG. 1(b). The target image contains various information of the target object to be searched, including but not limited to brand name, object content, series, appearance, capacity, production date, and the like. The target image may be collected by the user equipment by using the image collection device, or may be received by the user equipment from other electronic devices, and the manner in which the target image is acquired does not constitute a limitation on the embodiments of the present disclosure. The first receiving unit 610 corresponds to the communication unit 550 in FIG. 5 and can be implemented by using a radio frequency circuit and a signal receiving circuit.

The image quality of the target image will directly affect the search results. For example, in the target image (1) of FIG. 1(b), if the target image is blurred and information such as the brand name evian, capacity, and the like cannot be extracted, it is difficult to accurately search for the target object. Therefore, the target image preferably satisfies a predetermined condition. The predetermined condition may be a condition regarding the brightness of the target image or a condition regarding the sharpness of the target image. When the brightness of the target image is greater than or equal to the preset brightness threshold, it is determined that the target image satisfies a predetermined condition; when the brightness of the target image is less than the preset brightness threshold, it is determined that the target image does not satisfy the predetermined condition. Alternatively, when the sharpness of the target image is greater than or equal to the preset sharpness threshold, it is determined that the target image satisfies a predetermined condition; when the brightness of the target image is less than the preset sharpness threshold, it is determined that the target image does not satisfy the predetermined condition. The preset brightness threshold or the preset definition threshold may be adjusted according to the processing capability of the server. For example, when the processing capability of the server is strong, the preset brightness threshold or the preset definition threshold may be set to a lower value; when the processing capability of the server is weak, the preset brightness threshold or the preset definition threshold may be set. Is a higher value.

The extracting unit 620 extracts character information and image features associated with the target object from the target image. The character information included in the target image is, for example, a product name, a capacity, a brand name, a date of manufacture, and the like, and the character information is a character or a symbol. The image special diagnosis included in the target image is a color component of the image, a composition ratio of each color component, and the like. Typically, different techniques are employed to extract character information and image features in the target image. Extraction unit 620 can be implemented using the memory and processor of FIG.

For the character information, the extracting unit 620 may extract the character information in the target image using OCR technology or other techniques. In OCR technology, the server determines its shape by detecting the dark and bright patterns of the target image, and then uses the character recognition method to translate the shape into computer text. The extracting unit 620 may include an OCR module, and may extract the target pair from the target image by the following operation Like associated character information: identifying characters and symbols from the target image by using optical character recognition OCR; selecting identification characters for identifying the target object from the recognized characters and symbols as related to the target object Linked character information. Rich information is included in the target image, and some of the information may be closely related to the search of the target object, such as product name, brand, capacity, and the like. However, the target image may also include information that is not related to the search of the target object, such as components, security reminders, etc., which may be information related to all similar products, which cannot be used to identify the target object. Therefore, after performing the character recognition on the target image, the extracting unit 620 needs to filter out information required for searching the target object, that is, the identification character for identifying the target object.

Image features are index-valued image feature representations, such as using vectors to represent image characteristics. The image features of the target image may be represented in various ways that are present or appearing in the future. The extracting unit 620 may include an image feature extraction module, and the image feature extraction module may perform at least one of: extracting an image feature: calculating a color histogram feature of the target image as the image feature; and calculating the image The word bag model feature of the target image is used as the image feature. That is, the extracting unit 620 may represent the image features of the target image using at least one of a color histogram and a bag model feature.

A color histogram is a statistical representation of the color characteristics of an image that is used to represent the proportion of different colors in the entire target image, without concern for the spatial location of each color. Common color histograms include RGB spatial color histograms, HSV spatial color histograms, and Lab space color histograms. In different color spaces, the color histogram of the target image has different values. The word bag model feature is a statistical representation of the texture features of an image that can effectively describe the overall and local characteristics of the image. For example, the extracting unit 620 can obtain the word bag model feature of the target image by extracting feature descriptors such as SIFT, HOG, etc. from the target image; for each descriptor, searching for the most similar in the pre-accurate codebook The clustering center counts the frequency of occurrence of different clustering centers in the target image to form a histogram; normalizes the histogram to obtain the word bag model feature of the target image. The pre-accurate codebook can be obtained by randomly extracting a large number of image descriptors from a set of training images, and clustering the image descriptors by using a clustering algorithm to obtain a plurality of categories, and all the clusters are obtained. The category is the codebook.

The search unit 630 searches for related object information associated with the target object based on the character information and the image feature. For example, the search unit 630 searches the related object information from a pre-established object database based on the character information and the image feature. The object database includes image features, character information, and associated information of respective candidate objects. As described above, it is assumed that the object database P contains N objects, and each object p _j can be represented by a triplet {f _I (p _j ), f _T (p _j ), a(p _j )}, where j The meaning of each component in the =1, 2, ..., N, triplet is as described above. Alternatively, each object p _j can also be represented by a binary group {f _I (p _j ), f _T (p _j )}. It is assumed that the image features and character information of the target image of the target object q to be searched are f _I (q) and f _T (q), respectively, and accordingly, the search unit 630 can pass the character information f _T extracted by the extracting unit 620 ( q) The image feature f _I (q) is compared with the character information f _T (p _j ) of each candidate object in the object database P and the image feature f _I (p _j ) to perform a search. Search unit 630 can be implemented using the memory and processor of FIG.

The searching unit 630 is operable to search for related object information associated with the target object: calculating image features similar to the target object and each candidate object based on the image features of the target image and the image features of the respective candidate objects Calculating a similarity of character information between the target object and each candidate object based on the character information of the target image and the character information of each candidate object; and based on the image feature similarity and the character information similarity Searching for related object information associated with the target object from the plurality of candidate objects.

As an example, the search unit 630 may calculate a cosine similarity s _I (q, between the image feature f _I (q) of the target object q and the image feature f _I (p _j ) between the respective candidate objects p _j , p _j ) as the image feature similarity. Typically, the search unit 630 can calculate the cosine similarity s _I (q, p _j ) according to the above formula (1), and can be specifically referred to the description above in connection with the formula (1). Further, the search unit 630 may also take the Pearson correlation coefficient between the image feature of the target object and the image feature between the respective candidate objects as the image feature similarity.

The search unit 630 may be calculated as the similarity of the character information: calculating the edit distance d between the target object character information q f _T (q) of each candidate character information of the object p _j f _T (p _j) (f _T (q), f _T (p _j )); based on the edit distance, the length of the character information f _T (q) of the target object, the length of the character information f _T (p _j ) of the candidate object To calculate the similarity of the character information. The edit distance d(f _T (q), f _T (p _j )) is the minimum number of editing operations required to convert the character information f _T (q) into the character information f _T (p _j ). Character information length f _T (q), for example, the number of characters and symbols included in the character information f _T (q) in. Character information length f _T (p _j), for example, the number of characters and symbols included in the character information f _T (p _j) in. The search unit 630 can calculate the character information similarity s _T (q, p _j ), for example, by the above formula (2). Alternatively, the search unit 630 may also use the edit distance d(f _T (q), f _T (p _j )) as the character information similarity.

The searching unit 630 may search for related object information by selecting R1 candidate objects from the plurality of candidate objects in descending order of image feature similarity; from the plurality of devices in descending order of character information similarity R2 candidate objects are selected among the selected objects; information corresponding to the selected R1 candidate objects and R2 candidate objects is used as related object information associated with the target object. R1 is a natural number smaller than N. R2 is also a natural number smaller than N.

Alternatively, the searching unit 630 may further search for related object information based on the image feature similarity and the character information similarity in such a manner that the image feature similarity and the character information similarity are weighted and averaged to obtain the target. An average similarity between the object and each of the candidate objects; selecting a predetermined number of candidate objects from the plurality of candidate objects in descending order of the average similarity; using information corresponding to the selected candidate object as Relevant object information associated with the target object.

The search unit 630 can calculate the average similarity using the above formula (3), and specifically refer to the description made above in connection with the formula (3). After calculating the average similarity s(q, p _j ) between the target object and each candidate object, j=1, 2, . . . , N, the search unit 630 may decrement the N average similarities. Arranging in order, and selecting, for example, a predetermined number of R candidate objects with an average degree of similarity, and information corresponding to the selected R candidate objects as related object information associated with the target object, ie, search result . R is a configurable parameter, and its typical value can be set to 10, 20, 100, and so on.

The first sending unit 640 sends the related object information, that is, the searched related object information is sent to the user equipment as a search result. The first transmitting unit 640 can transmit the related object information by using various networks or communication technologies such as the Internet and a local area network. The related object information is, for example, a picture, a text description, and associated information of the R candidate objects, or a picture, a text description, and associated information of the R1 plus R2 candidate object described above. After receiving the related object information, the user equipment may display the related object information on the screen of the user equipment for the user to view. The first transmitting unit 640 may correspond to the communication unit 550 in FIG. 5 and may be implemented by using a radio frequency circuit and a signal transmitting circuit.

In the technical solution of the second search device 600 for a server according to an embodiment of the present disclosure, the character information and the image feature associated with the target object are extracted from the target image of the target object to be searched, based on the character The information and image feature performs a search, and can accurately and conveniently search for related object information of the target object, thereby improving the user experience. In addition, the step of manually inputting a keyword by the user is eliminated by automatically recognizing the character information contained in the target image.

FIG. 7 is a block diagram that schematically illustrates a user device 700 in accordance with an embodiment of the present disclosure. The user equipment 700 corresponds to the user equipment shown in FIG. 1(a). As shown in FIG. 7, the user equipment 700 may include an image acquisition unit 710, a determination unit 720, a second transmission unit 730, and a second reception unit 740.

The image acquisition unit 710 collects a target image of the target object to be searched for. Image acquisition unit 710 is typically disposed in the user device. For example, if the user's blue moon laundry liquid is exhausted and it is desired to purchase the blue moon laundry liquid, the user uses the image acquisition unit 710 to perform image acquisition on the existing blue moon laundry liquid. The image acquisition unit 710 is illustrated as being included in the user equipment in FIG. 7, but the image acquisition unit 710 may also be external to the user equipment, coupled to the user equipment, and capable of receiving instructions of the user equipment, And transmitting the acquired target image to the user equipment. The positional relationship between the image capture device and the user equipment does not constitute a limitation on the embodiments of the present disclosure. The image acquisition unit 710 can be a camera, a camera, or the like. The image acquisition unit 710 corresponds to the image acquisition unit 560 of FIG.

The judging unit 720 judges whether or not the target image satisfies a predetermined condition. Since the server is to extract information from the target image to search for the target object, the image quality of the target image will directly affect the search result. The determining unit 720 can make a request for the target image using a predetermined condition. The predetermined condition may be a condition regarding the brightness of the target image or a condition regarding the sharpness of the target image. The determining unit 720 can be implemented using the memory and processor in FIG.

As an example in which the determination unit 720 determines whether the predetermined condition is satisfied based on the brightness of the target image, the determination unit 720 may convert the acquired target image into image data of an HSL color space in which the luminance information is included in the image data of the HSL color space. Then, the judging unit 720 counts the average value of the illumination components (ie, the L component) of all the pixels in the image data of the HSL color space.

And comparing it to a predetermined brightness threshold T _L . Average value of the illumination component used for the target image When the predetermined brightness threshold T _L is greater than or equal to, the determination unit 720 may determine that the target image satisfies a predetermined condition. Average value of the illumination component used for the target image

When it is less than the predetermined brightness threshold T _L , the determination unit 720 may determine that the target image does not satisfy the predetermined condition. The predetermined brightness threshold T _{L is} typically 64. Alternatively, the determining unit 720 can also indirectly determine the quality of the target image by measuring the lighting conditions in the image capturing environment by means of the illuminometer. For example, the user equipment 700 may further include an illuminometer 750 for measuring an illumination parameter of the target object, the determination unit 720 communicating with the illuminometer to determine an illumination parameter in the process of acquiring the target image; When the parameter is greater than or equal to the preset illuminance, it is determined that the target image satisfies a predetermined condition; when the illumination parameter is less than the preset illuminance, it is determined that the target image does not satisfy the predetermined condition.

As an example in which the determination unit 720 determines whether the predetermined condition is satisfied based on the sharpness of the target image, the determination unit 720 may extract an edge of the acquired target image using a predetermined algorithm (for example, the Canny algorithm), and calculate the edge located in the target image. The gradient G of each pixel, and then further calculate the average of the gradients of all the pixel points at the edge in the target image

When it is greater than or equal to the preset gradient threshold T _G , the determining unit 720 may determine that the target image satisfies a predetermined condition. The average of the gradients of all pixel points at the edge of the target image

When it is less than the preset gradient threshold T _G , the determination unit 720 may determine that the target image does not satisfy the predetermined condition. The preset gradient threshold T _{G is} typically 100.

The second transmitting unit 730 issues a search request for the target object when the target image satisfies a predetermined condition, the search request including the target image. A search device as shown in FIG. 5 or FIG. 6 extracts character information and image features associated with the target object from the target image, and performs a search based on the character information and the image features. For example, after receiving the search request, the first receiving unit 610 in FIG. 6 extracts an image feature associated with the target object from the target image by using an image feature extraction module therein, and utilizes the image feature The COR module extracts character information associated with the target object from the target image; the search unit 630 searches for the related object information associated with the target object from the object database based on the character information and the image feature; The two transmitting unit 640 transmits the searched related object information to the user equipment. Since the brightness or sharpness of the target image is good, the character information and the image feature can be accurately extracted in the server, thereby ensuring the accuracy of the search. The second transmitting unit 730 corresponds to the transceiver unit 550 in FIG. 5, and can be implemented by using a radio frequency circuit and a signal transmitting circuit.

When the judging unit 720 judges that the target image does not satisfy the predetermined condition, it means that the acquired target image does not meet the requirement, which may result in difficulty in accurately extracting the character information and the image feature therein. At this time, the user equipment 700 may further include an output unit for outputting a retake prompt message to prompt the user to operate the image collection device to collect the target image of the target object to be searched. In the retake prompt message, it is also possible to specifically list the reason why the target image does not satisfy the predetermined condition. For example, the average of the illumination components of the target image

When less than the predetermined brightness threshold T _L , it may be indicated that the brightness is insufficient in the re-scuing message; the average of the gradients of all the pixel points located at the edge of the target image

When the preset gradient threshold T _G is smaller than the preset gradient threshold T _G , it is indicated that the sharpness is insufficient in the replay prompt message. In this way, the shooting of the target image can be adjusted according to the replay prompt message until the target image that satisfies the predetermined condition is acquired. Alternatively, when the determination unit 720 determines that the target image does not satisfy the predetermined condition, the setting parameters of the image acquisition unit 710 may be automatically adjusted until the target image satisfying the predetermined condition is acquired.

The second receiving unit 740 receives related object information associated with the target object. After the second transmitting unit 730 issues a search request to the server, the server performs the search method described in connection with FIGS. 2 and 3, and obtains related object information associated with the target object. Correspondingly, the second receiving unit 740 receives related object information associated with the target object. The related object information is obtained based on character information and image feature search associated with the target object extracted from the target image. The second receiving unit 740 can receive the related object information from the server through various networks or communication technologies such as the Internet, a local area network, and the like. The related object information is, for example, a picture of a plurality of candidate objects, a text description, and associated information. The associated information is, for example, price, sales volume, user rating, promotional video, hyperlinks, etc., which assists the user in performing selection operations among a plurality of candidate objects. After receiving the related object information, the second receiving unit 740 may display the related object information on the screen of the user equipment for the user to view. The second receiving unit 740 corresponds to the transceiver unit 550 in FIG. 5 and can be implemented by using a radio frequency circuit and a signal receiving circuit.

In the technical solution of the user equipment according to the embodiment of the present disclosure, when the target image of the acquired target object satisfies a predetermined condition, the search request is issued based on the target image, so that the related object information of the target object can be accurately and conveniently searched, Thereby improving the user experience.

After the first search device and the second search device according to an embodiment of the present disclosure are described above, an electronic device or server including any of the first search device and the second search device is also within the scope of the present disclosure.

Those of ordinary skill in the art will appreciate that the elements and algorithm steps of the various examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, or in computer software and electronic hardware. Come together to achieve. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the solution. A person skilled in the art can use different methods for implementing the described functions for each particular application, but such implementation should not be considered to be beyond the scope of the present invention.

In the several embodiments provided by the present application, it should be understood that the disclosed apparatus and method may be implemented in other manners. For example, the device embodiments described above are merely illustrative. For example, the division of the unit is only a logical function division. In actual implementation, there may be another division manner, for example, multiple units or components may be combined or Can be integrated into another device, or some features can be ignored or not executed.

The above is only a specific embodiment of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily think of changes or substitutions within the technical scope of the present invention. It should be covered by the scope of the present invention. Therefore, the scope of the invention should be determined by the scope of the appended claims.

Claims

A search method is applied to a server, the search method includes:

Receiving a search request including a target image of a target object to be searched;

Extracting character information and image features associated with the target object from the target image;

Searching for related object information associated with the target object based on the character information and the image feature;

Send the related object information.
The search method according to claim 1, wherein extracting character information and image features associated with the target object from the target image comprises:

Identifying text and symbols from the target image using optical character recognition OCR;

An identification character for identifying the target object is selected from the recognized characters and symbols as character information associated with the target object.
The search method according to claim 1, wherein said searching for related object information associated with said target object based on said character information and said image feature comprises:

Searching for the related object information from a pre-established object database based on the character information and the image feature,

The object database includes image features, character information, and associated information of each candidate object.
The search method according to claim 3, wherein said searching for said related object information from a pre-established object database based on said character information and said image feature comprises:

Calculating image feature similarity between the target object and each candidate object based on image features of the target image and image features of the respective candidate objects;

Calculating a similarity of character information between the target object and each candidate object based on the character information of the target image and the character information of each candidate object;

Searching for related object information associated with the target object from the plurality of candidate objects based on the image feature similarity and the character information similarity.
The search method according to claim 4, wherein said searching for related object information associated with said target object from said plurality of candidate objects based on said image feature similarity and said character information similarity comprises:

Performing a weighted average of the image feature similarity and the character information similarity to obtain the target The average similarity between the object and each candidate;

Selecting a predetermined number of candidate objects from the plurality of candidate objects in descending order of the average similarity;

Information corresponding to the selected candidate object is used as related object information associated with the target object.
The search method according to claim 4, wherein the calculating the image feature similarity between the target object and each candidate object based on the image feature of the target image and the image feature of each candidate object comprises: calculating the target object The cosine similarity between the image features and the image features between the respective candidate objects is taken as the image feature similarity.
The search method according to claim 4, wherein the calculating the similarity of the character information between the target object and each of the candidate objects based on the character information of the target image and the character information of each candidate object comprises:

Calculating an edit distance between the character information of the target object and the character information of each candidate object;

The character information similarity is calculated based on the edit distance, the length of the character information of the target object, and the length of the character information of the candidate object.
The search method according to any one of claims 1 to 7, wherein said extracting character information and image features associated with said target object from said target image comprises at least one of the following operations:

Calculating a color histogram feature of the target image as the image feature; and

A word bag model feature of the target image is calculated as the image feature.
The search method according to claim 1, wherein said target image satisfies a predetermined condition.
A search method is applied to a user equipment, and the search method includes:

Collecting a target image of the target object to be searched;

Determining whether the target image satisfies a predetermined condition;

Sending a search request for the target object when the target image satisfies a predetermined condition, the search request including the target image;

A related object information associated with the target object is received, wherein the related object information is obtained based on character information and image feature search associated with the target object extracted from the target image.
The search method according to claim 10, wherein said judging whether said target image is satisfied The booking conditions include:

Determining an illumination parameter during the process of acquiring the target image;

When the illumination parameter is greater than or equal to the preset illumination, it is determined that the target image satisfies a predetermined condition.
The search method according to claim 10, wherein said determining whether said target image satisfies predetermined conditions comprises:

Determining an average gradient of pixels of an edge of the acquired target image;

When the average gradient of the pixel points of the edge of the target image is less than the preset gradient threshold, it is determined that the target image satisfies a predetermined condition.
A search device is applied to a server, the search device comprising:

a transceiver receiving a search request, the search request including a target image of a target object to be searched;

processor;

Memory; and

Computer program instructions stored in the memory perform the following steps when the computer program instructions are executed by the processor:

Extracting character information and image features associated with the target object from the target image;

Searching for related object information associated with the target object based on the character information and image features;

The searched related object information is provided to the transceiver for transmission.
The search apparatus according to claim 13, wherein said extracting character information and image features associated with said target object from said target image comprises:

Identifying text and symbols from the target image using optical character recognition OCR;

An identification character for identifying the target object is selected from the recognized characters and symbols as character information associated with the target object.
The search apparatus according to claim 13, wherein said searching for related object information associated with said target object based on said character information and said image feature comprises:

Searching for the related object information from a pre-established object database based on the character information and the image feature,

The object database includes image features, character information, and associated information of each candidate object.
The search device according to claim 15, wherein said said based on said character information and said Searching for the related object information from the pre-established object database includes:

Calculating image feature similarity between the target object and each candidate object based on image features of the target image and image features of the respective candidate objects;

Calculating a similarity of character information between the target object and each candidate object based on the character information of the target image and the character information of each candidate object;

Searching for related object information associated with the target object from the plurality of candidate objects based on the image feature similarity and the character information similarity.
The search apparatus according to claim 16, wherein said searching for related object information associated with said target object from said plurality of candidate objects based on said image feature similarity and said character information similarity comprises:

Performing a weighted average of the image feature similarity and the character information similarity to obtain an average similarity between the target object and each candidate object;

Selecting a predetermined number of candidate objects from the plurality of candidate objects in descending order of the average similarity;

Information corresponding to the selected candidate object is used as related object information associated with the target object.
The search device according to claim 16, wherein the calculating the image feature similarity between the target object and each candidate object based on the image feature of the target image and the image feature of each candidate object comprises: calculating the target object The cosine similarity between the image features and the image features between the respective candidate objects is taken as the image feature similarity.
The search apparatus according to claim 16, wherein the character information similarity between the target object and each candidate object is calculated based on the character information of the target image and the character information of each candidate object, including:

Calculating an edit distance between the character information of the target object and the character information of each candidate object;

The character information similarity is calculated based on the edit distance, the length of the character information of the target object, and the length of the character information of each candidate object.
The search device according to any one of claims 13 to 19, wherein extracting character information and image features associated with the target object from the target image comprises at least one of the following operations:

Calculating a color histogram feature of the target image as the image feature; and

A word bag model feature of the target image is calculated as the image feature.
The search device according to claim 13, wherein said target image satisfies a predetermined condition.
A user equipment comprising:

An image collector for collecting a target image of a target object to be searched;

a processor, configured to determine whether the target image meets a predetermined condition;

a transceiver, when the target image satisfies a predetermined condition, the issuing a search request for the target object, the search request including the target image, and receiving related object information associated with the target object, where The related object information is obtained based on character information and image feature search associated with the target object extracted from the target image.
A user equipment according to claim 22, wherein

The user equipment further includes an illuminometer for measuring an illumination parameter of the target object,

The processor instructs the illuminometer to measure an illumination parameter of the target object in a process of acquiring an image of the target by the image collector, and determining that the target image satisfies a predetermined condition when the illumination parameter is greater than or equal to a preset illuminance.
The user equipment according to claim 22, wherein said processor analyzes the target image to determine an average gradient of pixel points of the edge thereof, and determines when the average gradient of the pixel points of the edge of the target image is less than a preset gradient threshold The target image satisfies a predetermined condition.
A computer program product for searching for an object, comprising a computer readable storage medium having stored thereon computer program instructions, the computer program instructions being executed by a processor to cause the processor to:

Receiving a search request including a target image of a target object to be searched;

Extracting character information and image features associated with the target object from the target image;

Searching for related object information associated with the target object based on the character information and the image feature;

Send the related object information.
A computer program product for searching for an object, comprising a computer readable storage medium having stored thereon computer program instructions, the computer program instructions being executed by a processor to cause the processor to:

Acquiring a target image of the target object to be searched by using an image collector;

Determining whether the target image satisfies a predetermined condition;

Transmitting, by the transceiver, the target object when the target image satisfies a predetermined condition Searching for a request, the search request including the target image;

The related object information associated with the target object is received by the transceiver, wherein the related object information is obtained based on character information and image feature search associated with the target object extracted from the target image.