DE212011100024U1 - Search with common image audio queries - Google Patents

Abstract

A system, comprising: a data processing device and a computer storage medium encoded with a computer program, the program comprising instructions that, when executed by the computing device, cause the computing device to perform operations, comprising: receiving a common image audio query from a client device, the common image audio query containing query image data defining a query image and query audio data defining a query audio; Determining image query feature data based on the query image data, the query image feature data describing image features of the query image; Determining polling audio feature data based on the audio data, the polling audio feature data describing audio characteristics of the polling audio; Providing the query image feature data and the query audio feature data to a common image audio relevance model, wherein the common image audio relevance model is adapted to generate relevance score values for a plurality of resources, each resource including resource image data defining a resource image for the resource and textual data; containing a resource text for the resource, and where each relevance score is a measure of the relevance of the corresponding image audio query resource;

Description

GENERAL PRIOR ART

The present specification relates to the processing of queries, in particular queries containing both an image and an associated audio.

The Internet provides access to a wide variety of resources, such as video files, image files, audio files, or content-targeted web pages, book articles, or consumer products. A search system may choose one or more resources in response to receiving a search query. A search query is data that a user submits to a search engine to satisfy the user's information needs. The search system selects and evaluates resources based on their relevance to the search query. Search results are typically ordered by scores and posted on a search results page.

For searching image resources, a search system may determine the relevance of an image for a text query based on the textual content of the resource in which the image resides, and also based on relevance feedback associated with the image. Some search systems seek image resources as input using query images. A query image is an image, such as a jpeg file, that is used by a search engine as input to a search processing operation. Related images can be found by processing other images and identifying images that are similar in visual appearance to the query image. The use of query images becomes more prevalent with the advent of smartphones incorporating cameras. For example, a user with a smartphone can now take a picture of an object of interest and submit the picture to a search engine. The search engine then searches for image resources using the image as a query image.

However, viewers interpret images in a much more subjective way than text. While the images that are identified may be similar in appearance to the query image, many of the images may not be of interest to the viewer. For example, a user may perform a search over an image of a car. The user may be interested in other cars of this brand, but an image search could possibly only respond to images of cars from different brands based on visual similarity.

BRIEF SUMMARY OF THE INVENTION

In general, an innovative aspect of the subject matter described in this specification may be embodied in systems for receiving a common image audio query from a client device, the common image audio query containing query image data defining a query image and query audio data representing a query audio define; Determining image query feature data based on the query image data, the query image feature data describing image features of the query image; Determining polling audio feature data based on the audio data, the polling audio feature data describing audio characteristics of the polling audio; Providing the query image feature data and the query audio feature data to a common image audio relevance model, wherein the common image audio relevance model is adapted to generate relevance score values for a plurality of resources, each resource including resource image data defining a resource image for the resource and textual data; which defines a resource text for the resource, and wherein each relevance score is a measure of the relevance of the corresponding image audio interrogation resource; Ordering the resources according to the corresponding relevance score; and providing data defining search results indicating the order of the resources to the client device. Other embodiments of this aspect include corresponding systems, devices, and computer programs that are configured to perform acts encoded on computer storage devices.

Another aspect of the subject matter described in this patent may be implemented in systems for accessing image annotation data describing a plurality of annotation pairs, each annotation pair including image data defining an image and textual data associated with the image; Accessing resources, each resource defining a resource image for the resource and text data defining resource text for the resource; and training a common image-audio relevance model on the image annotation data and the resources to generate relevance score values for a plurality of resources, and wherein each relevance score is a measure of the relevance of a corresponding image-audio resource, the query image data defining a query image. and query audio data defining query audio contains. Other embodiments of this aspect include corresponding systems, devices, and computer programs that are configured to perform acts encoded on computer storage devices.

Certain embodiments of the subject matter described in this specification may be implemented to realize one or more of the following advantages. Adding audio to an image query can improve the relevance of search results for the query. The relevance can be improved by providing both information that can assist the system in extracting the object of interest in an image and also by providing information that complements the user's search beyond what can be found in the image , This information can also be added in a variety of other ways. In some embodiments, a portion of the image of the user who circles a circle on the image using a touchscreen may be selected to include the object of interest. The user may also outline the object of interest more closely than with a circle or other shape, and may also draw the outline using other input methods. In some embodiments, the user may add additional information regarding the image using a drop-down menu box. The menu box may have various categories of items, such as shopping categories including shoes, shirts, pants, and other similar categories.

The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects and advantages of the subject matter will be apparent from the description, drawings and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

1 Figure 10 is a block diagram of an example environment in which a common image audio search system provides search services.

2A and 2 B are exemplary images for a common image audio query.

3A FIG. 10 is a block diagram of an exemplary process for querying a common image audio search system.

3B FIG. 10 is a block diagram of an exemplary process for training a common image audio relevance model.

4 FIG. 10 is a flowchart of an example process for training a common image audio relevance model.

5 FIG. 10 is a flowchart of an exemplary process for ranking resources for a common image audio query.

DETAILED DESCRIPTION

 § 1.0 Overview

An application running on a mobile phone allows a user to take a picture of an object and speak into the phone to record the user's speech. The audio recording is paired with the image to form a common image audio query. The mobile device then submits the common image audio query to a search system.

The search system receives the common image audio query and determines text data based on the language and generates image feature data based on the image. The search system uses the text data and the image feature data as inputs to a common image audio relevance model that compares resources with the input data. The resources can be any of the resources found on the Internet, including web pages, documents, images, and video. As an example, each resource may be a document for a product containing an image of the product and associated text data of the product. The common image audio relevancy model compares the query image feature data with the image feature data of each resource and the query text data with the corresponding resource text data, and calculates a relevance score value for each resource. The system organizes the resources according to the relevance score values and presents search results to the user. The search results contain links to the ordered resources and may also contain additional information about each resource, such as thumbnails of the resource image or subsets of the resource text.

In order for the common image audio relevancy model to determine the relevance of a common image audio query for the resources, the model is first trained. Training the model involves using image annotation data, which are annotation pairs. Each annotation pair is an image paired with text data associated with the image. These annotation pairs are used as training inputs for the common image audio relevance model along with training and test resources for the annotation pairs. The common image audio relevance model is trained until the test resources are ranked in a way that is deemed acceptable, as defined by one or more criteria.

1 is a block diagram of an example environment 100 in which a common image audio search system 106 Provides search services. The exemplary environment 100 contains a network 104 such as the Internet, which is a user device 102 with a search system 106 combines. The user device 102 transmits a common image audio query 120 that is a pairing of image data 122 and audio data 124 contains, over the network 104 to the search system 106 , Exemplary audio 124 is a voice recording. The system 106 processes the image data 122 and audio data 124 and compare them to a collection of resources 116 , where is a relevance score for each resource 116 is calculated. The system 106 classifies these resources 116 according to their relevance scores and sends a list of search results, each of which is a resource link 130 to a corresponding resource to the user device 102 ,

The user device 102 is an electronic device that is under the control of a user and over the network 104 resources 116 request and receive. To exemplary user facilities 102 include personal computers, mobile communication devices, and other devices that can send and receive data over the network. A user device 102 usually contains a user application, eg. For example, use a web browser to send and receive data over the network 104 to enable. The user device 102 can also have a camera and a microphone to capture an image 122 and audio 124 contain. The user equipment also includes an application that stores the audio 124 with the picture 122 paired to form a common image audio query. The query audio 124 usually contains voice data, which gives more information about the picture 122 or via the search parameters of the user.

For example, assume that a user searches for a water bottle and the query image 122 one from the user device 102 taken picture of a water bottle is. See for example 2A , In 2A The picture may contain more than one water bottle. After taking the image, the user specifies that the water bottle in the image is the object of interest by displaying the query image 122 with the query audio 124 "Water bottle" is extended. Alternatively, the user may provide more specific information by, for example, "red water bottle" than the query audio 124 receives. The query audio 124 may also include position information, for example, in the query image 122 If there is more than one object, the user can do so by submitting the query studio 124 Specify "red water bottle on the right side".

Even with only one object in the image, audio information can enhance the results of a search. For example, contains 2 B in the picture only the water bottle. However, if a search were performed to find similar images based only on the visual features, the results may only include bottles that are similar in shape and color, and may not contain other types of water bottles. By expanding the image with the audio, e.g. For example, "water bottle" or "bicycle water bottle", the system provides additional information to the search system, and the search system uses this additional information to provide search results that are likely to satisfy the user's information needs.

Furthermore, the user can also provide parameters through the use of audio to limit the search results. For example, the user may be looking in a product database to find a water bottle for sale. The user can give the image to the search system 122 the water bottle and the query audio 124 "Water bottle under $ 10, brand X" or as another example "this water bottle in blue" deliver.

Referring again to 1 receives the search system 106 the common image audio query that the image data 122 and the audio data 124 contains, through the network 104 from the user device 102 , In some implementations, the search system includes 106 an image processing device 110 for generating image feature data from the image data 122 , Alternatively, the search system sends the image data to other implementations 122 to a separate image processing device 110 and receives the image feature data from the separate image processing device 110 , Analogously, the search system 106 also a voice processing device 112 for extracting text data from the audio data 124 included, or it may be the audio data 124 to a separate language processing device 112 deliver and receive the text data.

The search system 106 uses the image feature data and the text data derived from the common image audio query as input to a common image audio relevance model 108 , The common image-audio relevance model 108 receives these two inputs and also receives resources 116 , The common image-audio relevance model 108 judges every resource 116 indicating a relevance measure of the resource 116 for the common image audio query.

REL_i

ein Relevanzpunktwert für eine Ressource R_i;

S

die Audiodaten 124;

I

die Bilddaten 122; und

R_i

eine gegebene Ressource in einer Ressourcendatenbank oder einem Ressourcenzwischenspeicher sind. Die Funktion f(S, I, R) wird unter Bezugnahme auf 3B unten ausführlicher beschrieben.

In some implementations, the search system computes using the common image audio relevance model 108 a score for each resource according to the following rating function: REL _i = f (S, I, R _i ) in which

REL _i

a relevance score for a resource R _i ;

S

the audio data 124 ;

I

the image data 122 ; and

R _i

are a given resource in a resource database or a resource cache. The function f (S, I, R) will be described with reference to 3B described in more detail below.

For a resource 116 is any data over a network 104 can be provided and associated with a resource address or indexed in a database. In some implementations, a resource database includes 114 a collection of resources 116 where each resource 116 contains a resource image and a resource text. An example of a resource database 114 is a product database that contains product documents that include an image of a product and descriptive data such as brand name, price, and a text description.

For every ith resource, the search system determines 106 Resource image feature data from the resource image in a manner similar to how it determines query image feature data from the query image. The search system 106 also determines resource text data from the resource 116 , The common image-audio relevance model 108 then compares the query image feature data with the resource image feature data and the query text data with the resource text data of a resource 116 and computes a relevance point value REL _i for the resource 116 , The model 108 Returns the relevance score values to the search system 106 , The search system 106 then allocates the resources according to the relevance score and returns search results 130 , classified according to the relevance score values of the resources, to the user equipment 102 ,

§ 2.0 Processing a common image-audio query

3A is a block diagram of an example process 300 to query a common image audio search system. The search system 106 receives the common image audio query, the image data 302 and audio data 304 includes. This data is received by the network, and in some implementations, the image data is 302 a picture taken by a user from a query object. The audio data 304 Contain language recorded by the user, which contains information about the query object or about the desired query results. These are paired together as the common image audio query.

The audio file 304 contains the language related audio. The voice data 304 are converted into textual data using a speech recognition algorithm 308 transformed. The text 308 is further analyzed using natural language processing techniques to determine the content of the textual data 308 to parse. For example, the picture 302 in the common image audio query contain a water bottle, as in 2A , The audio accompanying this image 304 can simply be "water bottle". The search system 106 converts this language 304 in text data 308 and uses the text 308 as a search parameter when comparing with resource text data.

Using natural language processing, the search system can 106 determine spatial areas of the image to include or exclude them. For example, the audio 304 the language "water bottle on the right side of the picture" included. The search system 106 converts this language 304 in text data 308 and parse the statement. The system 106 determines from the phrase "on the right side of the image" that the right side of the image is an area of interest and thus ignores on the left side of the image 302 recognized features and objects and focuses only on those who find it on the right.

Using natural language processing, the search system can 106 Detect sensations for particular features or characteristics. For example, the picture 302 in the common image audio query contain a red water bottle, as in 2 B , The audio 304 however, may contain the language "blue water bottles only, not red". The search system 106 converts this language 304 in text data 308 and parse the statement to interpret that the user only wants blue water bottles in the search results, as opposed to the red water bottle in the picture 302 ,

From the image data 302 The image audio query generates the search system 106 Image feature value data 306 , Image feature value data 306 are value scores, the visual characteristics of a section of an image 302 represent. The section of the picture can be the wholeness of the picture 302 or Contain a section of the image. In some implementations, the image features 306 Include color, texture, edges, saturation and other characteristics. Exemplary processes for extracting values of image features 306 from which a feature point value can be calculated include processes for generating color histograms, texture detection processes (eg, based on spatial variation in pixel intensities), scale invariant feature transformation, edge detection, corner detection, and geometric blur.

The common image-audio relevance model 108 receives the image feature data 306 and text data 308 , The model 108 also uses resources 314 in a collection of resources too. After every resource 314 has been accessed, generates the model 108 Resource image feature data from the resource image in a manner similar to the query image 302 , The model 108 also determines text data based on the resource 314 such as text on a web page containing the image, or text associated with the image according to a database schema (eg, a database of commercial products). The model 108 compares the query image feature data with the resource image feature data and the query text data with the resource text data, and calculates a relevance score for that resource 314 , The model 108 calculates relevance score values for each resource in the collection of resources, classifies the resources according to scores, and responds with a ranked list of resources 312 , The search system 106 then generates search results that reference the images and resources, and delivers the search results to the user.

In some implementations, this process may be iterated one or more times. For example, the system can 106 after generating a list of resources by relevance 312 for the picture audio query 302 . 304 use one or more of the highest rated resource images to perform another query. This can produce an improved list of relevance resources. Alternatively, or in combination, the system may include resource text data from one or more highest ranked resources in addition to or instead of the original query text data 308 use.

§ 3.0 Training the common image-audio relevance model

Thus the common picture audio relevance model 310 Calculate relevance point values correctly, it will be trained first. 3B is a block diagram of an example process 350 to train a common image-audio relevance model 108 , The model is trained with pairs of notes. Analogously to a common image-audio query, a comment pair has image data 352 and associated audio data 354 on. The set of annotation pairs can be divided into a training amount and a test amount.

By taking note pairs from the training set, image feature data becomes 358 using similar image processing algorithms as those used in the query image from the annotation image data 352 generated. text data 360 are extracted from the annotated audio data using similar speech recognition techniques and natural language processing techniques as those used in the query audio 354 certainly. A training model 362 receives as input the image feature data 358 and the text data 360 , The training model 362 also receives as input a resource 356 with a predetermined relevance to the annotation pair 352 . 354 , This predetermined relevance may be binary (eg, relevant / not relevant) or located on a relative scale (eg, highly relevant, somewhat relevant, not relevant) or on a scale of more refined values. The model 362 generates resource image feature data from the resource image and determines resource text data from the resource text. By comparing the annotated image feature data 352 with the resource image feature data and the annotation text data 354 with the resource text data calculates the training model 362 a relevance score. Weights, which correspond to the image features and text features, are adjusted to produce a score in the correct range of the predetermined relevance. This process is repeated for different resources and with different training comment pairs, all with predetermined relevances.

With the test set of annotation data, the trained model can then be verified. The trained model may receive as input annotation pairs from the test set along with resources having a predetermined relevance for each of the test pairs. The test pairs and resources would be processed to generate feature data, as happens with the training pairs. The model would then generate relevance score values for each of these sets of inputs. If the recovation point values. within a threshold range of acceptability, then the model is adequately trained. However, if the model generates relevance point values that are not within the threshold range of acceptability, then the model is not adequately trained and the training process can be repeated with the training set of annotation data and the assigned weights can be re-evaluated and adjusted.

This threshold range can be set in many different ways. For example, each of the qualitative scale values in the predetermined relevance scale may be assigned relevance score ranges. For example, if the relevance point values generated by the model range from 1 to 100, the threshold may be set to greater than or equal to 50 for a binary predetermined relevance scale and less than 50 for non-relevant. Alternatively, the threshold can be made stricter by, for example, assigning over 75 to be relevant and less than 25 to irrelevant. This can provide a more effective image-audio-relevance model, but may also require more iterations of training to manufacture. Alternatively, the threshold of acceptability may become more qualitative. For example, for a given annotation pair, there may be a set of resources with a predetermined ranking of more relevant to less relevant. The acceptability of training the model can be evaluated by examining how closely the trained model comes to provide the correct rating of the resources for the annotation pair.

§ 3.1 Selection of Note Pairing Data

The annotation data can be obtained in a variety of ways. In one implementation, the annotation data is derived from a product database, the product database having a collection of product documents. Each product document has an image of a product and associated text with information about the product, such as a description, prices, sellers of the product, and reviews and ratings of both the product and the seller of the product. The note pair 352 . 354 Contains the image of a product document and a subset of the text from the same document. This would also have a predetermined relevance between the product document and the pair of comments made from this product 352 . 354 allow. Because the note pair 352 . 354 created from this product document, the comment pair for the product document must be highly relevant.

In another implementation, the annotation data is derived from selection data from image search result data. The query input text input by users to an image search system may be as the annotation text data 354 of a pair of comments. The note image data 352 for the pair can be selected from images that are the most popular image search results according to the query input. The popularity of results can be determined by statistical measures such as CTR. Alternatively, the annotation data may be from product search result data selection data. The query input may be again as the annotation text data 354 be used for a comment pair. The note picture 352 can be obtained from the product image of the most popular product documents selected by users for this query input. This would also provide product documents for use as resources of high predetermined relevance.

In another implementation, the annotation data is derived from selection data from general web search result data. Query input text entered by users in a Web search system may be considered the annotation text data 354 of a pair of comments. The web search system can respond to general web resources, including web sites, images, and product documents. If the user selects a product document as a result of the web search, the product image may be designated as the annotated image data 352 used for the comment pair. The product document is then used as the resource of known high relevance.

In another implementation, human commentators can be used to provide training data. The commentators can use a photograph to add the annotation picture 352 and voice or text data for the annotation text data 354 supply resources they want to search for. The commentators can then search a product document or other resource database and find resources that are both related and unrelated to the photography and speech data they provided. Any resource found by the commenters can then flag it as a good quality match or a poor quality match. In another implementation, the commenters can be used to rank the quality of matches determined by an automated procedure. For example, any of the previously discussed procedures may be used to obtain annotation data from a product database, product search selection data, image search selection data, or web search selection data, and human commentators may classify the relevance of each annotation pair for the resource selected by the automated process.

§ 3.2 Exemplary evaluation models

A variety of models can be used to add the relevance function f (S, I, R) too and examples are described below. An example model implements a relevance function that is a linear combination of individual models trained on image feature data and audio and text data, ie f (S, I, R) = cf _S (S, R) + (I-c) f _I (I, R) where f _{S is} a weighting function trained on the voice and text data, and f _{I is} a weighting function trained on the image feature data. The mixing parameter c is a value that is adjusted between 0 and 1.

Another exemplary model implements a relevance function f (S, I, R) that limits the set of resource elements considered to only those text descriptions containing the words in S. With this limited set, the model then evaluates the relevance of the image feature data. Thus, the relevance function would be (S, I, R) = f _I (I, R) f (S, R) where f (S, R) = 1 if text S is in resource R and otherwise f (S, R) = 0.

Another exemplary model implements a relevance function f (S, I, R), where an image feature relevance function is learned for each possible choice of S, ie f (S, I, R) = W _S · Φ (I, R) where Φ (I, R) is the feature representation of the image and the resource and W _{S is} a learned feature weight matrix of images and resources representing features. W _S is a 1 × | Φ (I, R) | matrix or a vector of dimension | Φ (I, R) |, that is, the number of features used to represent the image and the resource.

Yet another exemplary model implements a relevance function using a linear ranking function, ie f (S, I, R) = W · Φ (S, I, R).

Another exemplary model implements a relevance function, which is an extension of the approach described in the incorporated by reference herein "Large Scale Image Annotation: Learning to Rank with Joint Word-Image Embeddings" by Jason Weston, Samy Bengio and Nicolas Usunier ( "Weston Tower") is located. The approach in the Weston essay involves training on an "embedding-space" representation of arbitrary dimension, where the distance between two elements in the space denotes their similarity. This model includes the function f (S, I, R) = (W _SI · Φ _SI (S, I)) · (W _R · Φ _R (R)) where W _SI and W _{R are} matrices, and the method learns both matrices and an embedment space of dimension R, which is typically of a low dimension. W _SI is an R × | Φ _SI (S, I) | matrix, where R is the size of the embedment space and | Φ _SI (S, I) | is the number of features used to represent text and image together. W _R is an R × | Φ _R (R) | matrix, where | Φ _R (R) | is the number of features used to represent the resource. The embedding room in the Weston Tower used only pictures and markings. The approach is extended here by concatenating the speech and image features into a single feature space Φ _SI (S, I).

Another exemplary model implements a relevance function that further expands the approach of the Weston essay. The relevance function is defined by f (S, I, R) = Σ (W _S * Φ _S (S)) * (W _I * Φ _I (I)) * (W _{R R} (R)) where the * operation is the component multiplication of vectors. This feature allows for more complex nonlinear interactions between the features of the image, the language, and the resource.

§ 4.0 Exemplary processes

4 is a flowchart of an example process 400 to train a common image-audio relevance model 105 , The process 400 can in the search system 106 be implemented and is to train a common image-audio relevance model 108 used.

The process 400 uses image annotation data ( 402 ). The search system 106 For example, accesses image annotation data from a product database. The search system 106 can also access image annotation data from product search selection data. In another example, the search system takes effect 106 on image annotation data of image search selection data. In another implementation, the search system takes effect 106 on image annotation data of web search selection data. The search system 106 can also access image annotation data from data annotated by human commentators. Human commentators can generate their own image and voice data for commenting, or access data to be commented from a product database or other automated process.

The process 400 accesses resources ( 404 ). For example, the search system takes effect 106 resources that include product documents from a product database.

The process 400 trains a common image-audio relevance model on the image annotation data and resources ( 406 ). For example, the search system trains 106 a common image audio relevancy model using the image annotation data from the product database and the resources from the product database. For example, the common image audio relevance model may be determined according to one of the techniques described in section 3.2 training algorithms described above, or other training algorithms may be used.

5 shows a flowchart of an example process 500 to classify resources for a common image audio query. The process 500 can in the search system 106 and is used to classify resources for a common image audio query.

The process 500 receives a common image audio query ( 502 ). The search system 106 For example, it receives a common image audio request from a user device through the network.

The process 500 determines query image feature data ( 504 ). For example, the search engine generates 106 Image feature value data from the query image received from the user device.

The process 500 determines query audio feature data ( 506 ). For example, the search system processes 106 the audio data to generate text data from audio data including voice data.

The process 500 provides query image feature data and query audio feature data to the common image audio relevance model ( 508 ). For example, the search system provides 106 Query image feature data and text data to the common image audio relevance model. The common image-audio relevance model is trained to generate relevance score values for a collection of resources.

The process 500 allocates resources according to their relevance scores ( 510 ). For example, the search system orders 106 the resources from the most relevant to the least relevant for the image-audio query.

The process 500 returns search results indicating the order of resources ( 512 ). For example, the search system provides 106 Search results comprising a list of resource addresses classified from the most relevant to the least relevant to the user equipment.

§ 5.0 Additional Implementation Details

Implementations of the subject matter and operations described in this specification may be implemented in digital electronic circuitry or in computer software, firmware, or hardware, including the structures disclosed in this patent and their structural equivalents, or combinations of one or more of them. Implementations of the subject matter described in this specification may be implemented as one or more computer programs, i. H. one or more modules of computer program instructions encoded on a computer storage medium for execution by or for controlling the operation of the computing device. Alternatively or additionally, the program instructions may be encoded on an artificially generated broadcast signal, such as a machine-generated electrical, optical or electromagnetic signal generated to encode information for transmission to a suitable receiver device for execution by a data processing device. A computer storage medium may be or may be included in a computer readable storage device, a computer readable storage substrate, a random or serial access storage array, or a random or serial access device, or a combination of one or more of them. In addition, although a computer storage medium is not a broadcast signal, a computer storage medium may be a source or destination of computer program instructions encoded in an artificially generated broadcast signal. The computer storage medium may also be one or more separate physical components or media (eg, multiple CDs, floppy disks, or other storage devices) or included therein.

The operations described in this patent may be implemented as operations performed by a data processing device on data stored on or received from other sources by one or more computer readable storage devices.

The term "data processing device" includes all types of devices, devices, and machines for processing data, including, for example, one programmable processor, a computer, a system on a chip, or several individual or combinations of the above. The device may include special logic circuitry, such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). The apparatus may include, in addition to hardware, code that generates an execution environment for the computer program in question, such as code, the processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of forms them. The device and execution environment may implement various computing model infrastructures, such as web services, distributed computing and network computing infrastructures.

A computer program (also known as a program, software, software application, script or code) may be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and may be used in any form, including as a independent program or as a module, component, subroutine, object or any other entity suitable for use in a computing environment. A computer program may or may not be equivalent to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (for example, one or more scripts stored in a markup language document), a single file dedicated to the program in question, or multiple coordinated files (eg, files that store one or more modules, partial programs or sections of code). A computer program may be used to run on a computer or on multiple computers that are in one place or distributed over multiple locations and connected through a communication network.

The processes and logic flows described in this patent may be performed by one or more programmable processors that execute one or more computer programs to perform actions by working on input data and generating an output. The processes and logic flows may also be performed by special logic circuitry, such as a Field Programmable Gate Array (FPGA) or Application Specific Integrated Circuit (ASIC), and the device may also be implemented as such.

For example, processors suitable for executing a computer program include both general purpose and specialty microprocessors and one or more arbitrary processors of any type of digital computer. Generally, a processor will receive instructions and data from a read only memory or random access memory or both. The essential elements of a computer are a processor for performing actions in accordance with instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices for storing data, such as magnetic, magneto-optical disks, or optical disks, or will be operably coupled to receive data from or transfer data to them, or both. However, a computer need not have such facilities. In addition, a computer may be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device (such as a USB Universal serial bus, just to name a few, means suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and memory devices including, for example, semiconductor memory devices such as EPROM, EEPROM, and Flash Magnetic disks, such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks The processor and memory may be supplemented or integrated with special logic circuitry.

To provide for interaction with a user, implementations of the subject matter described in this specification may be implemented on a computer having a display device, such as a CRT or LCD (Liquid Crystal Display) monitor to display information to the user, and a keyboard and a pointing device, such as a mouse or a trackball, through which the user can provide input to the computer. Other types of devices may be used to also provide interaction with a user; for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, voice or tactile input. Additionally, a computer may interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending web pages to a web browser on a client device of the user in response to requests received from the web browser.

While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any disclosures or claims that may be claimed, but rather as descriptions of features that are inherent in particular implementations of particular disclosures. Certain features described in this patent in the context of separate implementations may also be implemented in combination in a single implementation. Conversely, various features described in the context of a single implementation may also be implemented in multiple implementations separately or in any suitable subcombination. Moreover, although features above may be described as acting and even initially claimed as such in certain combinations, in some instances, one or more features of a claimed combination may be removed from the combination in some instances, and the claimed combination may be a partial combination or combination relate to a variation of a sub-combination.

Similarly, while operations in the drawings are shown in a particular order, this should not be construed analogously as requiring that such operations be performed in the particular order shown, or in a sequential order, or that all operations illustrated are performed as desired To achieve results. In some circumstances, multitasking and parallel processing may be beneficial. In addition, the separation of various system components in the above-described implementations should not be construed as requiring separation in all implementations, and it should be understood that the described program components and systems may generally be integrated together into a single software product several software products.

Thus, certain implementations of the subject have been described. Other implementations are within the scope of the following claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve desirable results. In addition, the processes shown in the accompanying figures do not necessarily require the particular order or sequential order shown to achieve desired results. In certain implementations, multitasking and parallel processing may be beneficial.

Claims (13)

A system comprising: a data processing device and a computer storage medium encoded with a computer program, the program comprising instructions that, when executed by the computing device, cause the computing device to perform operations comprising: Receiving a common image audio query from a client device, the shared image audio query including query image data defining a query image and query audio data defining a query audio; Determining image query feature data based on the query image data, the query image feature data describing image features of the query image; Determining polling audio feature data based on the audio data, the polling audio feature data describing audio characteristics of the polling audio; Providing the query image feature data and the query audio feature data to a common image audio relevance model, wherein the common image audio relevance model is adapted to generate relevance score values for a plurality of resources, each resource including resource image data defining a resource image for the resource and textual data; which defines a resource text for the resource, and wherein each relevance score is a measure of the relevance of the corresponding image audio interrogation resource; Ordering the resources according to the corresponding relevance score and Deliver data defining search results indicating the order of resources to the client device. The system of claim 1, wherein the query audio feature data includes data pertaining to a query object in the query image data by further describing the query object, wherein the query object is a subset of the query image that contains image data representing an object of interest. The system of claim 1 or 2, wherein the data further describing the query object includes information indicating a position of the query object. The system of any of claims 1 to 3, wherein the query audio feature data includes data defining one or more restrictions on the search results. A system according to any one of claims 1 to 4, wherein one or more resource images are used to refine the search results. The system of any one of claims 1 to 5, wherein one or more resource text data is used to refine the search results. A system comprising: a data processing device and a computer storage medium encoded with a computer program, the program comprising instructions that, when executed by the computing device, cause the computing device to perform operations comprising: Accessing image annotation data describing a plurality of annotation pairs, each annotation pair including image data defining an image and text data associated with the image; Accessing resources, each resource defining a resource image for the resource and text data defining resource text for the resource; and Training a common image-audio relevance model on the image annotation data and the resources to generate relevance score values for a plurality of resources, and wherein each relevance score is a measure of the relevance of a corresponding image-audio resource, the query image data defining a query image, and Contains query data defining query audio. The system of claim 7, wherein the annotation pair text data is converted from a speech-defining audio file. The system of claim 7 or 8, wherein the image retention data includes annotation pairs obtained from selection data for search results in response to a user search query, wherein the selection data identifies a selection of a reference to the annotated image from search results in response to the user query and the annotation text data includes the user query text , The system of any one of claims 7 to 9, wherein the image annotation data includes annotation pairs obtained from a product database, the product database including product resources defining a product resource image and product product data for multiple products, the annotation image comprising a product resource image and the annotation text data comprising a subset the corresponding product text data. The system of any one of claims 7 to 10, wherein a portion of the image annotation data and resources of known relevance are divided into a training amount and a test amount, wherein the training amount is used to train the common image audio relevance model and the test set is used to test the common image audio relevance model. A computer storage medium encoded with a computer program, the program comprising instructions that, when executed by a client device, cause the client device to perform operations comprising: Receiving a common image audio query from a client device, the shared image audio query including query image data defining a query image and query audio data defining a query audio; Determining image query feature data based on the query image data, the query image feature data describing image features of the query image; Determining polling audio feature data based on the audio data, the polling audio feature data describing audio characteristics of the polling audio; Providing the query image feature data and the query audio feature data to a common image audio relevance model, wherein the common image audio relevance model is adapted to generate relevance score values for a plurality of resources, each resource including resource image data defining a resource image for the resource and textual data; which defines a resource text for the resource, and wherein each relevance score is a measure of the relevance of the corresponding image audio interrogation resource; Ordering the resources according to the corresponding relevance score and Deliver data defining search results indicating the order of resources to the client device. A computer storage medium encoded with a computer program, the program comprising instructions that, when executed by a client device, cause the client device to perform operations, comprising: accessing image annotation data describing a plurality of annotation pairs, each annotation pair including image data defining an image and text data that are associated with the image contains; Accessing resources, each resource defining a resource image for the resource and text data defining resource text for the resource; and training a common image-audio relevance model on the image annotation data and the resources to generate relevance score values for a plurality of resources, and wherein each relevance score is a measure of the relevance of a corresponding image-audio resource, the query image data defining a query image. and query audio data defining query audio contains.

Images