DE102017125474A1 - CONTEXTUAL COMMENTING OF INQUIRIES - Google Patents

Abstract

Methods, systems and apparatuses, including computer programs encoded on a computer storage medium, for contextualizing requests are disclosed. In one aspect, a method includes receiving an image presented on a display of a computing device and a transcription of an utterance spoken by a user of the computing device, identifying a particular sub-image contained in the image, and based on performing image recognition on the particular sub-image, determining one or more first labels that indicate a context of the particular sub-image. The method also includes, based on performing text recognition on a portion of the image other than the particular sub-image, determining one or more second labels that indicate the context of the particular sub-image based on the transcription, the first Labels and the second labels, generating a search query and delivering, for output, the search query.

Description

TERRITORY

The present description relates to search engines.

BACKGROUND

Generally, a search request includes one or more expressions that are presented upon a request to perform a search towards a search engine. For example, a user may enter query expressions of a search query by typing on a keyboard or, in the case of a voice request, by speaking the query expressions into a microphone of a computing device. Voice queries can be processed using speech recognition technology.

SUMMARY

In some implementations, an image corresponding to a portion of a display of a computing device may be analyzed to help a query processing system respond to a natural language query. For example, a user may ask a question about a photograph that the user is looking at on the computing device, such as "What is this?". The computing device may capture the user's utterance and capture a respective image of the computing device that the user is viewing. The computing device processes the utterance to produce a transcription of the utterance spoken by the user of the computing device. The computing device transmits the transcription and the image to a server.

The server receives the transcription and the image from the computing device. The server can identify visual and textual content in the image. The server generates labels for the image that correspond to a content of the image, such as locations, entities, names, species of animals, etc. The server may identify a particular sub-image in the image. The particular subpicture may be a photograph or a drawing. In some aspects, the server identifies a portion of the particular sub-image that is likely to be of primary interest to the user, such as a historical landmark in the image. The server may perform an image recognition on the particular sub-image to generate labels for the particular sub-image. The server may also generate labels for textual content in the image, such as comments corresponding to the particular sub-image, by performing text recognition on a portion of the image other than the particular sub-image. The server can generate a search query based on the received transcription and the generated labels. Furthermore, the server may be configured to provide the search query for output to a search engine.

An innovative aspect of the subject matter described in this specification is embodied in methods including the actions of receiving an image presented on or corresponding to at least a portion of a display of a computing device and receiving a transcription from, or, what corresponds to it, an utterance spoken by a user of the computing device, typically at the time the image is presented, identifying a particular sub-image contained in the image, and, based on performing an image recognition on the particular sub-image Determining one or more first labels that indicate a context of the particular sub-image. The method also includes, based on performing text recognition on a portion of the image other than the particular sub-image, determining one or more second labels that indicate the context of the particular sub-image based on the transcription, the first Labels and the second labels, generating a search query and delivering, for output, the search query.

Such method steps or other combinations of steps as described herein may be performed automatically and without further user intervention, such as in response to an automatic determination by the computing device that the method should be performed at a particular time, or after a particular one Keystroke, a spoken command, or other indication from a user of the computing device that such a procedure should be performed. The methods described herein may therefore provide a more efficient user interface to the user device by reducing the input required by a user to achieve a desirable or desirable search query generation.

Other implementations thereof and other aspects include corresponding systems, devices, and computer programs that are configured to perform the actions of the methods encoded on computer memory devices.

Implementations may each optionally include one or more of the following features. For example, the methods may include a weighting of the first label that different from that of the second labels. The methods may also include generating the search query by replacing one or more of the first labels or the second labels for terms of transcription. In some aspects, the methods include generating for each of the first labels and the second labels a label confidence value indicating a likelihood that the label corresponds to a portion of the particular sub-image that is of primary interest to the user and selecting one or more of the first labels and the second labels based on the respective label confidence values, the query being generated based on the one or more selected first labels and second labels. Furthermore, the methods may include accessing historical query data containing previous search queries provided by other users, generating, based on the transcription, the first labels and the second labels, one or more candidate search queries, comparing the historical query data with the one or more candidate search queries, and, based on comparing the historical query data with the one or more candidate search queries, selecting the search query among the one or more candidate search queries.

The methods may include generating, based on the transcription, the first labels and the second labels, one or more candidate search queries, determining, for each of the one or more candidate search queries, a query confidence score, a probability indicates that the candidate query is an exact override of the transcription and selecting, based on the query confidence values, a particular candidate query as the search query. In addition, the methods may include identifying one or more images included in the image, generating for each of the one or more images included in the image, an image confidence score indicating a probability that Image is an image of primary interest to the user, and, based on the image confidence values for the one or more images, selecting the particular sub-image. The methods may include receiving data indicative of a selection of a control event at the computing device, the control event identifying the particular sub-image. In some aspects, the computing device is configured to capture the image and to capture audio data that corresponds to the utterance in response to detecting a predefined keyword ("hotword").

Further, the methods may include receiving an additional image of the computing device and an additional transcription of an additional utterance spoken by a user of the computing device, identifying an additional particular sub-image contained in the additional image based on performing an image recognition on the additional determined one Subimage, determining one or more additional first labels indicating a context of the additional particular sub-image based on performing text recognition on a portion of the additional image other than the additional particular sub-image, determining one or more additional second labels indicating the context of the additional particular sub-image based on the additional transcription, the additional first labels, and the additional second labels, generating a B error and a completion of the command. In this case, performing the command may include performing one or more of storing the additional image in memory, storing the particular sub-image in memory, uploading the additional image to a server, uploading the particular sub-image to the server, importing of the additional image to an application on the computing device and importing the particular sub-image to the application of the computing device. In certain aspects, the methods may include identifying metadata associated with the particular sub-image, wherein determining the one or more first labels that indicate the context of the particular sub-image is further based on the metadata associated with the particular one Sub-picture are associated.

Advantageous implementations may include one or more of the following features. The methods may determine a context of an image corresponding to a portion of a display of a computing device to assist in processing natural language queries. The context of the image can be determined by image and / or text recognition. Specifically, the context of the image may be used to rewrite or transcribe a transcription of a user's utterance. The methods may generate labels related to the context of the image and replace the labels for portions of the transcription. For example, when looking at a photograph, a user may be on a computing device and ask, "Where was this taken?". The methods may determine that the user refers to the photograph on the screen of the computing device. The methods may extract information about the photograph to determine a context of the photograph, and a context of other parts of the image that do not contain the photo. In this case, the context information may be used to determine a location at which the photo was taken. As such, the methods may use images corresponding to displays of computing devices to aid in the generation of search queries.

In some aspects, the methods may identify a particular sub-picture in the image that is a primary focus of the user. The methods may generate labels corresponding to the particular sub-image and weight labels corresponding to the particular sub-image different from those of other labels so that the context of the image may be determined more effectively. The methods may weight labels based on a particular subpicture's fame in the image, a frequency at which the particular subpicture labels appear in historical queries, a frequency at which the particular subpicture labels appear in recent searches, and so on Method identify primary points of user interest in the image to determine a context of the image as a whole.

The details of one or more embodiments of the invention are set forth in the accompanying drawings and the description below. Other features and advantages of the invention will become apparent from the description, the drawings and the claims.

list of figures

• 1 FIG. 12 is a graphical representation of an example environment for contextualizing a query.
• 2 FIG. 4 is a graphical representation of an exemplary system for contextualizing a query.
• 3 Fig. 10 is a flowchart illustrating an example process for contextualizing a request.
• 4 FIG. 10 is a flowchart illustrating an example process for selecting a particular sub-image using confidence values. FIG.
• 5 FIG. 10 is a flow chart illustrating an example process for generating a search query using selected labels. FIG.
• 6 FIG. 10 is a diagram of an example computing device and example mobile computing device. FIG.

Like reference numerals and designations in the various drawings indicate like elements.

DETAILED DESCRIPTION

1 is a graphical representation of an example environment 100 for contextualizing a request. The environment 100 contains a user 102 and a computer device 104 , In the neighborhood 100 provides the user 102 a statement 103 , such as a request, to the computing device 104. The user 102 may ask a question about one or more objects displayed on a graphical display of computing device 104. For example, the utterance 103 include a request, such as "What is this?". In this case, the user can 102 refer to objects, such as an image, text, video or any combination thereof, displayed on the graphical display of the computing device 104. The computer device 104 may include one or more computing devices, such as a laptop, a desktop, a smartphone, a tablet, or any other computing device that is known.

The comment 103 of the user 102 can be contextually ambiguous. In this case, the utterance 103 Do not refer directly to the contents of a computer by a name 102 is shown. However, a context of the displayed objects may be determined and the context may be combined with a transcription corresponding to the utterance 103 used to make the request unique.

The computer device 104 can be configured to take a picture 106 to capture that on a display of the computing device 104 is presented when the utterance 103 of the user 102 Will be received. For example, the computing device 104 capture a portion of the ad that contains a photograph 108 and comments 116 that match the photograph contains, but not a logo icon 120 contains, such as an application title, the computing device 104 to run. In some examples, the picture is the same 106 a screenshot of the computer device 104 , Alternatively or additionally, the computing device 104 persistently or permanently capture the displayed content and send or transmit certain sub-images upon detection of the utterance 103. Furthermore, the picture can 106 on capturing a predefined hotword in the utterance 103 be detected. The computer device 104 can the utterance 103 transcribe. In some implementations, the computing device may 104 Audio data according to the statement 103 to send a speech recognition engine and a transcription of the utterance 103 received by the speech recognition engine.

Transcription according to the statement 103 and the picture 106 can be transmitted to a server over a network for processing (eg, to make the utterance clear). The server can be configured to display a context of the image 106 by analyzing the image 106 to determine. The server can change the context of the image 106 by identifying and analyzing images of photographs in the image. For example, a photograph 108 may be analyzed to identify the photograph 108 one or more entities in the photograph 108 contains. With reference to the exemplary environment 100 of the 1 can the photography 108 are identified by the server and then analyzed to determine that the photograph contains 108 entities, such as the Eiffel Tower 110 and a dog 112 in front of the Eiffel Tower 110 ,

In some examples, the server performs image recognition on the particular sub-image 108 by. The image recognition is performed to determine one or more first labels that indicate a context of the particular sub-image. For example, the server may have an image recognition on the photograph 108 carry out and determine the first labels, that of photography 108 such as Eiffel Tower, France, Paris and dog. Image recognition can be a determination of entities in a focus in photography 108 , entities in the foreground and background of photography 108 , relative sizes of entities in photography 108 and the like included. In some examples, the server may identify metadata associated with the particular subpicture or photograph 108 in 1 , The server can use the metadata to determine the first labels that match the particular sub-image.

In addition, the server can do a text recognition on the image 106 carry out. The server can do a text recognition on a partial area of the image 106 perform another than photography 108 is. The portion of the image 106 may be a title 114 the photograph 108 and / or comments 116 that focus on photography 108 refer, included. For example, the picture contains 106 of the 1 a title 114 indicating a location at which the photograph 108 was taken, such as Paris, France. The picture 106 also contains comments or comments 116 referring to photography 108, such as "Dave ~ So cool, France is my favorite.""Sarah ~ I did not know you had a Golden, I have one!" and "Abby ~ I was just in Paris, when were you there? ".

The title 114 and the comments 116 of the picture 106 can be processed by the server via text recognition. By performing text recognition, the server may designate one or more second labels that further indicate the context of the particular sub-picture. For example, the server may have a text recognition on the title 114 to verify that the location of the particular subpicture is Paris, France. Furthermore, the server can do a text recognition on the comments 116 to verify that the location of the particular subpicture is Paris, France (eg, by performing a text recognition on the phrase "I was just in Paris."). In addition, the server can do a text recognition on the comments 116 perform to determine that the dog 112 in photography 108 a Golden Retriever is (eg by performing a text recognition on the phrase "I did not know that you had a golden ..."). As such, the server may generate one or more second labels, such as Paris, France, and Golden Retriever.

The server may be configured to generate a search query based on the received transcription, the first labels, and the second labels. The server can automatically generate the search query without further user intervention. For example, in response to automatic determination by the computing device 104 in that the method should be executed at a certain time, after a certain keystroke preceding the utterance, after a spoken command / word contained in the utterance or any other indication from the user 102 the computing device 104 that such a method should be executed before the transcription and before the image is received by the server.

The search query can be generated by overwriting the transcription. In some aspects, transcription may be overridden by replacing one or more of the first and / or second labels with the transcription. For example, the transcription may include "What's this?". In this case, the phrase "The Eiffel Tower" for the term "Das" in the transcription can be replaced. Therefore, the search query can be overridden to include the following: "What is the Eiffel Tower?".

In some aspects, the server is configured to generate a label trust value for each of the first and second labels. In this case, the label trust values can be one indicate relative likelihood that each label corresponds to a subset of the particular subpicture that is of primary interest to the user 102 is. For example, a first label may contain "Eiffel Tower" with a confidence score of 0.8, and may include a second label "Golden Retriever" with a confidence score of 0.5. In this case, the trust values may indicate that the first label corresponds to an entity that is more likely to be of primary interest to the user 102 based on the larger respective label trust value.

Labels can be selected to generate the search query based on the trusted values. For example, a certain number of labels with the highest confidence value can be selected to generate a query in combination with the transcription. In another example, all labels that satisfy a particular label confidence threshold may be used in combination with the transcription to generate the search query. In another example, the server may generate label confidence values based on a frequency at which the labels appear in recent searches, a frequency at which labels appear in all historical queries, and so on.

The server can be configured to access historical search request data. The historical query data may include a number of previous search queries made by the user 102 and / or other users are supplied. The server may generate one or more candidate search queries based on the transcription, the first labels, and the second labels, and compare the historical query data with the candidate search queries. Based on comparing the historical query data with the one or more candidate search queries, the server may select a particular candidate query as the search query. For example, the server may search the particular candidate search query based on a comparison between a frequency of candidate search queries appearing in recent search queries, such as queries entered by the user, and / or a frequency of candidate search queries entered in historical queries appear, such as queries entered by all users into a search engine.

The server may be configured to deliver the generated search request for output. For example, the server may be configured to deliver the generated search query to a search engine. In another example, the server may generate the search query and send the search request to the computing device 102. In this case, the search request to the user 102 may be audible or visual by the computing device 104 to verify that the server has accurately or accurately overwritten the request.

The server may be further configured to generate the generated query for output and / or a search result to the computing device 104 to deliver. In this case, the computer device 104 be configured to receive the search request and to output a search result that matches the search request 122 such as "You look at a photograph of the Eiffel Tower".

2 Figure 3 is a graphical representation of an exemplary system 200 for contextualizing a request. The system 200 contains the user 102 , the computer device 104 , a server 206 , an image recognition engine 208 and a text recognition engine 210 , The computing device 104 is in communication with the server 206 over one or more networks. The computer device 104 may be a microphone or other detection mechanisms for detecting user utterances 102 contain.

In one example, the user may 102 provide an utterance to the computing device 104. The utterance may be through the computing device 102 recorded and transcribed. As such, the computing device 104 a transcription 204 generate the utterance of the user 102 equivalent. The computer device 104 can also be configured to take a picture 202 a graphical display of the computer device 104 capture. The computing device 104 may capture the image 202 upon detecting the utterance of the user 102 or transcribing the utterance. Additionally or alternatively, the computing device 104 be configured to display the displayed contents of the computing device 104 permanently record. In this case, a particular subpicture can be transcribed 204 to the server 206 sent to a detection of the utterance.

In another example, the computing device may 104 be configured to the utterance of the user 102 to the server 206 to send. For example, the computing device 104 be configured to capture a predefined hotword in the utterance and, upon detection of the hotword, the utterance to the server 206 send. In this case, the server 206 configured to generate a transcription according to the utterance.

In a case (A), the server receives 206 the transcription 204 and the picture 202 of the computer device 104 , The computer device 104 can transcription 204 and the picture 202 automatically to the server 206 send. The computing device 104 may also transcribe 204 and the picture 202 on a user input. For example, the user may provide the utterance as well as a touch input to the graphical display of the computing device 104, indicating that the user is requesting a transcription corresponding to the utterance and the image to the server 206 to be sent.

In a case (B), the server identifies 206 a particular sub-picture 207 of the picture 202 and sends the particular sub-picture 207 to an image recognition engine 208. In some aspects, the server is 206 in communication with the image recognition engine 208 over the network. Other aspects include the server 206 and the image recognition engine 208 integrated into a single system.

In some examples, the picture may 202 contain several pictures. The server 206 may analyze the multiple images to determine the particular sub-image 207 that is likely to be of interest to the user 102. In addition, the server can 206 receive a user input indicating that the particular sub-picture 207 the pictures in the picture 202 of primary interest to the user 102 is. The server 206 can provide a picture confidence value for each of the multiple pictures in the picture 202 produce. The image confidence value may indicate a relative likelihood that an image will be an image of primary interest to the user 102 is. The server 206 can the particular sub picture 207 or the image of primary interest to the user 102 determine based on the confidence values generated. For example, the server 206 identify that the display of the computing device 104 a first subarea and a second subarea. The first portion may include a photograph, and the second portion may include a logo image corresponding to a title of the application using the computing device. The server may be configured to produce a confidence level of 0.9 for the first subarea and a confidence level of 0.3 for the second subarea. In this case, the server determines 206 that the first subset is likely of primary interest to the user 102 is based on the confidence values generated.

The server may be configured to the particular sub-picture 207 based on receiving data indicative of a selection of a control event. The control event may correspond to the user 102 inputting to the computing device 104 supplies. Specifically, the control event may correspond to that of the user 102 with the display of the computer device 104 interacts. For example, the user 102 interact with a portion of the ad that corresponds to the particular subpicture 207 equivalent. The server 206 may receive data indicating that the user 102 was interacting with a portion of the display corresponding to the particular sub-picture 207, and may therefore determine that the portion of the display is the particular sub-picture 207 equivalent.

At an event (C), the image recognition engine runs 208 an image recognition on the particular sub-image 207 by. The image recognition engine 208 performs image recognition to labels 209 for the particular sub-picture 207 indicating a context of the particular sub-picture. The labels 209 can be entities in the given subpicture 207 such as trees or a dog. The labels 209 may also correspond to entities containing specific locations or landmarks, such as the Eiffel Tower. The labels 209 can be used individually or in combination to create a context of the particular subpicture 207 to determine.

The image recognition engine 208 may be configured to a subarea of the particular subpicture 207 to determine that of a primary focus of the user 102 is. For example, the image recognition engine 208 the particular sub-picture 207 analyze to determine that the particular subpicture 207 Contains entities such as the Eiffel Tower and a dog. The image recognition engine 208 can analyze the entities in the particular sub-image 207 and determine that the Eiffel Tower is larger than the dog in size. Based on the determination that the Eiffel Tower is proportionally larger in size to the dog, the image recognition engine 208 may determine that the Eiffel Tower 110 probably of primary interest to the user 102 is. Additionally or alternatively, the image recognition engine 208 may be configured to include other aspects of the particular sub-image 207 such as foreground versus background, entities in a focus of the particular sub-image 207 and similar. For example, the image recognition engine 208 Determine that the Eiffel Tower is in focus in the particular subpicture 207 is and that the dog is out of focus. As such, the image recognition engine 208 determine that the Eiffel Tower is likely of primary interest to the user 102 is.

At an event (D), the server identifies 206 one or more subareas 211 of the picture 202 that the particular subpicture does not contain. The one or more subareas 211 become the text recognition engine 210 Posted. In some aspects, the server is 206 in communication with the text recognition engine 210 over the network. In other aspects, the server 206 and the text recognition engine 210 integrated into a single system. Furthermore, the server can 206 , the image recognition engine 208 and the text recognition engine 210 may be integrated into a single system. In some examples, the one or more subregions may 211 a title contained in the picture 202 contained, comments or comments included in image 202, or any content in the image 202 that is not the particular subpicture 207 contains.

For a result (E), the text recognition engine performs 210 a text recognition at the one or more subregions 211 of the picture 202 through, the or the particular sub-picture 207 does not contain or contain. The text recognition engine 210 performs text recognition to labels 212 for the one or more subareas 211 to create a context of the particular subpicture 207 Show. For example, the subareas 211 Comments include, for example, "Dave ~ So cool, France is my favorite.""Sarah ~ I did not know you had a Golden, I also have one!" And "Abby ~ I was in Paris just when you were there ? ". The labels 212 may directly text in the one or more sections 211 correspond. In this case, the labels 212 Include expressions, such as "France" or "Paris". The labels 212 can be from the text in the one or more sub-sections 211 be derived. In this case, the labels 212 derived from the phrase "Golden Retriever". The labels 212 can be used individually or in combination to create a context of the particular subpicture 207 to determine.

By performing text recognition, the text recognition engine 210 may include one or more labels 212 determine which continues the context of the particular subpicture 207 Show. For example, the text recognition engine 210 may do a text recognition on the annotations 116 to verify that the location of the particular subpicture is Paris, France (eg, by performing text recognition on the phrase "I was just in Paris."). In addition, the text recognition engine 210 do a text recognition on the annotations to determine that the dog is in the particular subpicture 207 a Golden Retriever is (eg by performing a text recognition on the phrase "I did not know that you had a golden ..."). As such, the text recognition engine 210 create one or more labels 212, such as Paris, France and Golden Retriever.

At an event (F) the server generates 206 a search query 213 using the transcription 204 , the labels 209 from the image recognition engine 208 and the labels 212 from the text recognition engine 210 , The server 206 may do the search 213 automatically without further user intervention. For example, in response to automatic determination by the computing device 104 in that the method should be executed at a certain time, after a certain keystroke preceding the utterance, after a spoken command / hotword contained in the utterance, or any other indication from the user 102 the computing device 104 that such a procedure prior to transcription 204 and before the image 202 through the server 206 is received, is to execute.

The server 206 can transcription 204 as the search query 213 overwrite. The server 206 may be a subset of the labels of the image recognition engine 209 and the text recognition engine 212 in the transcription 204 insert to the search query 213 to create. For example, the server 206 may label the "Eiffel Tower" in the transcription 204 insert so that the query generated 213 "What is the Eiffel Tower?" Contains.

The server continues to deliver 206 at an event (F), the generated search request 213 for output. For example, the server 206 deliver the search query 213 to a search engine. The server 206 can receive search results from the search engine and deliver the search results to the computing device 104 via the network. In some aspects, the computing device 104 may receive the search results and provide the search results as an audible or visual output. For example, the server 206 the search query 213 "What is the Eiffel Tower?" Generate and the generated query 213 to the computing device 104 deliver. In this case, the computing device 104 may be configured to generate the generated query 213 to the user 102 to audition for verification before the search query 213 is entered to a search engine.

In some examples, the server generates 206 the search query 213 according to generated weights of the labels 209 and 212 , In this case, server 206 may provide a first weighting for the image labels 209 generate a second weighting for the text labels 212 different. For example, the server 206 determine that the picture labels 209 more relevant for transcription 204 than the text labels 212 are. As such, the server can 206 a greater emphasis on the picture labels 209 through more weights of the picture labels 209 as the text labels 212 place.

The server 206 may be configured to provide an additional image of the computing device 104 and an additional transcription of a by a user of the computing device 104 to receive spoken additional utterance. The server 206 may identify an additional particular sub-image contained in the additional image and the additional particular sub-image to the image recognition engine 208 send to perform image recognition on the additional designated sub-picture. The image recognition engine 208 may be configured to generate one or more additional first labels for the additional particular sub-image indicating a context of the additional particular sub-image. Likewise, the server may be configured to provide a portion of the additional image that does not contain the additional particular sub-image to the text recognition engine 210 to generate one or more additional second labels based on performing text recognition on the portion of the additional image that is other than the additional particular sub-image.

The server 206 may use the additional transcription, additional first labels, and additional second labels to generate a command or action. The command can be through the server 206 be done automatically to the computing device 104 be delivered and the like. In some examples, the command may include one or more actions, such as storing the additional image in memory, storing the additional particular sub-image in memory, uploading the additional image to the server 206 , uploading the additional specific subpicture to the server 206 , importing the additional image to an application of the computing device 104 and importing the particular sub-image for use by the computing device 104 , For example, the user 102 may be viewing visual and textual content in a news application on the display of the computing device 104 look at. Using the received transcription and the generated labels, the server 206 may be configured to capture a portion of an image in the application annotation application and to upload the portion of the image to the cloud for storage.

For certain aspects, the server provides 206 the search query 213 to the computer device 104 , In this case, the computer device 104 the search query 213 for verification by the user 102 deliver before the search query 213 as input to a search engine supplies. As such, the search query 213 by the user 102 accepted, modified or rejected. For example, the user may respond in response to receiving the search request 213 at the computer device 104 provide a user input indicating that the search query 213 to deliver to a search engine. In another example, the user may 102 provide a user input indicating that the search query 213 is to be modified before it is delivered to the search engine. As such, the user can search query 213 modify directly or after another search from the server 206 ask. In another example, the user may 102 provide a user input indicating that the search query 213 is rejected. As such, the user can 102 after another search from the server 206 Provide questions or another statement to be used in generating another query.

3 is a flowchart illustrating an exemplary process 300 to contextualize a request. The process 300 can be performed by one or more servers or other computing devices. For example, operations of the process 300 through the server 206 of the 2 be performed. Operations of the process 300 may also be implemented as instructions stored on a non-transitory computer-readable medium, and when the instructions are executed by one or more servers (or other computing devices), the instructions cause the one or more servers to perform operations of the process 300 carry out.

At one step 310 the server receives an image and a transcription of an utterance. The image may correspond to a graphical display of a computing device in communication with the server. For example, the computing device may capture the image upon receiving the utterance. In some aspects, the image may correspond to a graphical display of the computing device when the computing device is in a camera mode. As such, the image may correspond to a photograph that captures the computing device or that is being viewed by a camera in communication with the computing device. Furthermore, the image may correspond to a video captured by the camera of the computing device or a video displayed in a display of the computing device. Additionally or alternatively, the computing device may send background noise that is detected while the utterance is being received. In this case, the server can use the background noise to to generate additional labels and / or to evaluate the generated labels.

The transcription may correspond to an utterance received by the computing device. In some aspects, the transcription is generated by the computing device based on the received utterance. In other aspects, the transcription corresponds to a user input received by the computing device. For example, a user may enter a question via a keyboard or user interface of the computing device. The computing device may generate the transcription based on the input and provide the transcription to the server.

At one step 320 the server identifies a particular subpicture that is included in the image. The server is configured to identify a particular sub-picture from one or more pictures in the picture. The particular subpicture may be an image that is likely to be in the primary focus or interest of a user. For example, the image may contain a photograph as well as several other graphical symbols. The server may be configured to analyze the image to determine that the photograph is of primary interest to the user, while the other graphical symbols in the display are not of primary interest to the user.

At one step 330 The server determines one or more first labels based on an image identifier of the particular sub-image. The server may perform image recognition on the particular sub-image in the image to identify one or more entities in the particular sub-image and to generate respective labels for the one or more entities. Each of the one or more entities may correspond to one or more respective first labels. The first labels may be partially determined using metadata associated with the particular sub-image. The first labels can display a context of the particular sub-picture. In certain aspects, the server is configured to perform image recognition over the entire image. In this case, the first labels can be generated for all entities identified by image recognition in the image.

At one step 340 The server determines one or more second labels based on text recognition at a portion of the image other than the particular sub-image. The server may perform a text recognition on the portion of the image other than the particular sub-image to identify a textual content of the image for use in generating labels indicative of a context of the content. The textual content may be identified using one or more second labels that indicate a context of the particular content.

The second labels may be determined in part using metadata associated with the partition. For example, the server may be configured to access and capture code related to displaying content on the display of the computing device. In this case, the server can access a tag code and capture the tag code to analyze for metadata that can be used in the generation of the second labels. In some aspects, the server is configured to perform text recognition over the entire image. In this case, the second labels may be generated for the entire textual content identified by a text recognition in the image.

At one step 350 The server generates a search query based on the transcription, the first labels, and the second labels. Specifically, the server is configured to generate the search query based on the transcription and the labels. In some examples, the server is configured to generate multiple candidate search queries based on the transcription and the labels. Candidate queries can be sequenced based on historical query data. As such, a candidate query with a top rank may be selected as the search query.

At one step 360 the server returns the search query for output. The selected search query can be delivered directly to a search engine. In this case, the server may also be configured to receive one or more search results from the search engine and to provide the search results for output. For example, the server may provide the search query to the search engine, select a particular search result, and provide the search result to the computing device for audible or visual output.

In other aspects, the query may be delivered to the computing device. The computing device may provide the search query for audible or visual output. In this case, the search request may be verified by a user before being provided as input to a search engine.

4 is a flowchart illustrating an exemplary process 400 to select a particular subpicture using Represents trustworthy. The process 400 can be performed by one or more servers or other computing devices. For example, operations of the process 400 through the server 206 of the 2 be performed. Operations of the process 400 may also be implemented as instructions stored on a non-transitory computer-readable medium, and when the instructions are executed by one or more servers (or other computing devices), the instructions cause the one or more servers to perform operations of the process 400 carry out.

At one step 410 the server identifies images contained in an image. In certain aspects, the server receives an image from a computing device and identifies a plurality of images in the image. The server may be configured to perform an image recognition on the image to identify the plurality of images. The pictures may contain photographs, symbols, drawings, pictures and the like. The images may vary in size, shape and type. In some aspects, the images correspond to a still image of a video. For example, the image may be from a web page containing multiple images and a video playing in the background. The image may correspond to a single captured frame of the video playing on the web page.

At one step 420 The server generates a trust value for each of the identified images. The confidence values may each indicate a probability that an image is an image of primary interest to a user viewing the image. The confidence values can be determined based on various features of the image. For example, the server may generate larger confidence values for large images over small images in the image. In another example, the server may generate greater confidence values for images having a large number of identifiable entities in the image, such as landmarks, humans, or animals, and vice versa.

At one step 430 The server selects a specific subpicture based on the trusted values. The server may be configured to select the particular subpicture based on the highest confidence value. As such, the confidence values of the images may be compared to determine which image is associated with the greatest confidence value. In some examples, the server selects multiple images. In this case, the server may be configured to select images when each of the selected images meets a predetermined image confidence threshold. This may be the case if multiple images in the image contain similar entities or objects. For example, two images in one image may contain the Eiffel Tower and a third image may not contain the Eiffel Tower. As such, the two respective images containing the Eiffel Tower may be selected as the particular sub-images due to the similar content in each of the two images.

5 is a flowchart illustrating an exemplary process 500 to generate a search query using selected labels. The process 500 can be performed by one or more servers or other computing devices. For example, operations of the process 500 may be performed by the server 206 of the 2 be performed. Operations of the process 500 may also be implemented as instructions stored on a non-transitory computer-readable medium, and when the instructions are executed by one or more servers (or other computing devices), the instructions cause the one or more servers to perform operations of the process 500 carry out.

At one step 510 The server generates a trust value for each of the first labels and the second labels. The first labels may correspond to a particular sub-image identified in an image, and the second labels may correspond to a portion of the image other than the particular sub-image. For example, the particular sub-image may be a photograph of the Eiffel Tower in the image, and the portion of the image other than the particular sub-image may include commentary on the photograph. The confidence values for the first and second labels each indicate a likelihood that the respective label corresponds to a portion of the particular sub-image that is of primary interest to the user.

At one step 520 The server selects one or more of the first labels and the second labels based on the trusted values. For example, the server can select a single label with the largest trust value. In another example, the server is configured to select labels with trust values that satisfy a predetermined confidence threshold. In another example, the server is configured to select a predetermined number of labels with the highest confidence values.

At one step 530 The server generates a search query using a received transcription, the selected first labels and the selected second labels. The server may be configured to deliver the generated search request for output. For example, the server may be configured to match the query generated to deliver to a search engine. In another example, the server may generate the search query and send the search request to a computing device. In this case, the search request may be delivered to a user acoustically or visually by the computing device.

6 FIG. 10 is a diagram of an example computing device 600 and an example mobile computing device. FIG 650 which can be used with the techniques described herein. The computing device 600 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The mobile computing device 650 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart phones, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are meant to be exemplary only, and are not intended to limit implementations of the inventions described and / or claimed in this document.

The computer device 600 contains a processor 602 a memory 604, a memory device 606 , a high-speed interface 608 connected to the memory 604 connects, and high-speed expansion ports 610, and a low-speed interface 612 connected to a low-speed bus 614 and the memory device 606 combines. Each of the components 602 . 604 . 606 . 608 . 610 and 612 is interconnected using various buses and may be mounted on a common motherboard or in other ways as appropriate. The processor 602 may process instructions for execution within the computing device 600, including instructions stored in memory 604 or on the storage device 606 are stored to display graphical information for a GUI on an external input / output device, such as a display 616 that with the high-speed interface 608 is coupled. In other implementations, multiple processors and / or multiple buses may be used, as appropriate, along with multiple memories and types of memory. Likewise, multiple computing devices 600 may be connected, with each device providing portions of the necessary operations (eg, as a server bank, a group of blade servers, or a multi-processor system).

The memory 604 stores information within the computing device 600. In one implementation, the memory is 604 a volatile memory unit or units. In another implementation, the memory is 604 a non-volatile storage unit or units. The memory 604 may also be another form of computer-readable medium, such as a magnetic or optical disk.

The storage device 606 may provide a mass storage for the computing device 600. In one implementation, the storage device may 606 a computer readable medium, such as a floppy disk device, hard disk device, optical disk device or tape device, flash memory, or other similar solid state storage device or set of devices, including devices in a storage area network or other configurations. A computer program product may be tangibly embodied in an information carrier. The computer program product may also include instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory 604 , the storage device 606 or a memory on the processor 602 ,

The high speed control 608 manages bandwidth intensive operations for the computing device 600 while the low-speed controller 612 manages operations that are less bandwidth intensive. Such an allocation of functions is only exemplary. In one implementation, the high-speed control is 608 with the memory 604 , the display 616 (eg, coupled via a graphics processor or accelerator), and high speed expansion ports 610 that can accommodate various expansion cards (not shown). In implementation, the low-speed controller 612 is the memory device 606 and the low speed expansion port 614. The low-speed expansion port, which may include various communication ports (eg, USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or more input / output devices, such as a keyboard, pointing device, scanner, or network device, such as for example, a circuit switch or a router, eg via a network adapter.

The computer device 600 can be implemented in a number of different forms, as shown in the figure. For example, it can act as a standard server 620 be implemented, or several times in a group of such Servers. It may also be implemented as part of a rack server system 624. In addition, it may be implemented in a personal computer, such as a laptop computer 622 , Alternatively, components of the computing device 600 combined with other components in a mobile device (not shown), such as the mobile computing device 650 , Each of such devices may include one or more of computer devices 600 . 650 included, and an entire system may consist of multiple computer devices 600 . 650 be formed, which communicate with each other.

The mobile computing device 650 contains a processor 652 , a store 664 , an input / output device, such as a display 654, a communication interface 666 and a transceiver 668 , among other components. The mobile computing device 650 may also be provided with a storage device, such as a microdrive or other device, to provide additional storage. Each of the components 650 . 652 . 664 . 654 . 666 and 668 is interconnected using different buses, and several of the components may be mounted on a common motherboard or in other ways as appropriate.

The processor 652 can execute instructions within the mobile computing device 650, including instructions stored in memory 664 are stored. The processor may be implemented as a chip group of chips containing separate and multiple analog and digital processors. For example, the processor may coordinate the other components of the mobile computing device 650 such as control of user interfaces, applications through the device 650 and wireless communication through the device 650.

The processor 652 can with a user through a control interface 658 and a display interface 656 that with an ad 654 are coupled, communicate. The ad 654 For example, a TFT-LCD (Thin Film Transistor Liquid Crystal Display) or OLED (Organic Light Emitting Diode) display or other suitable display technology may be used. The display interface 656 may be a suitable circuit for driving the display 654 to present graphical and other information to a user. The control interface 658 can receive commands from a user and submit them to the processor 652 convert. In addition, an external interface 662 in communication with the processor 652 to enable near field communication of the mobile computing device 650 with other devices. For example, the external interface 662 may provide for wired communication in some implementations, or for wireless communication in other implementations, and multiple interfaces may also be used.

The memory 664 stores information within the mobile computing device 650. The memory 664 may be implemented as one or more of a computer-readable medium or media, a volatile storage device or units, or a nonvolatile storage device or devices. An expansion memory 654 may also be provided and with the device 650 via the expansion interface 652 connected, for example, a SIMM (Einzelreihiger memory module) card interface. Such an expansion memory 654 may provide additional storage for the device 650, or may also provide applications or other information for the device 650 to save. Specifically, the extended memory 654 may include instructions to perform or supplement the processes described above, and may also include secure information. Thus, the extended memory 654 For example, it may be available as a security module for device 650 and may be programmed with instructions that may safely use the device 650 allow. Additionally, secure applications may be provided via the SIMM cards, along with additional information such as placing identification information on the SIMM card in a non-hackable manner.

The memory may include, for example, a flash memory and / or an NVRAM memory, as discussed below. In one implementation, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory 664 , the expansion memory 654 , the memory at the processor 652 or a propagated signal, for example via a transceiver 668 or an external interface 662 can be received.

The device 650 can communicate wirelessly via a communication interface 666, which may include a digital signal processing circuit where necessary. The communication interface 666 can under for communications various modes, such as GSM voice calls, SMS, EMS or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000 or GPRS, among others. Such communication may be, for example, via a radio frequency transceiver 668 respectively. Additionally, near field communication may occur, such as using Bluetooth, Wi-Fi, or another such transceiver (not shown). Additionally, a GPS (Global Positioning System) receiver module 650 may provide additional navigation and location related data to the device 650 which can be used as suitable by applications on the device 650 to run.

The device 650 can also communicate acoustically using audio codec 660, which can receive spoken information from a user and convert it into usable digital information. Audio codec 660 may equally produce acoustic sound to a user, such as through a speaker, eg, in a handset of device 650. Such sound may include a sound of voice phone calls, may include and may include recorded sound (eg, voice messages, music files, etc.) Also contain sound that is generated by applications on the device 650 are in operation.

The computer device 650 can be implemented in a number of different forms, as shown in the figure. For example, it can be used as a cellular telephone 680 be implemented. It can also be part of a smartphone 682 , a personal assistant or other similar mobile device.

A number of implementations have been described. Nevertheless, it will be understood that various modifications can be made without departing from the spirit and scope of the disclosure. For example, various forms of the operations shown above may be used with steps rearranged, added, or removed.

Embodiments of the invention and all of the functional operations described in this specification may be implemented in a digital electronic circuit, or in computer software, firmware, or hardware, including the structures disclosed in this specification, and theirs structural equivalents, or in combination of one or more of them. Embodiments of the invention may be implemented as one or more computer program products, such as e.g. one or more modules of computer program instructions encoded on a computer readable medium for execution by or for operation by a data processing device. The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a storage device, a composition of matter that effects a machine-readable propagated signal, or a combination of one or more of them. The term "data processing device" includes all devices, devices and machines for processing data, including, by way of example, a programmable processor, a computer or multiple processors or computers. The apparatus may include, in addition to hardware, code that generates an execution environment for the computer program in question, e.g. a code that forms a processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A propagated signal is an artificially generated signal, e.g. a machine-generated electrical, optical or electromagnetic signal generated to encode information for transmission to a suitable receiver device.

A computer program (also known as a program, software, software application, script or code) may be written in any form of programming language, including compiled or interpreted languages, and may be used in any form, including as a stand-alone program or as a module , a component, subroutine, or other device suitable for use in a computer environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (eg, one or more scripts stored in a recording language document), in a single file destined for the program in question, or in several coordinated ones Files (eg files that store one or more modules, subroutines or sections of a code). A computer program may be used to run on a computer or on multiple computers distributed in one or more locations and interconnected by a communication network.

The processes and logical operations described in this specification may be performed by one or more programmable processors that execute one or more computer programs to perform functions by working on input data and generating a computer program Issue. The processes and logical operations may also be performed by, and a device may also be implemented as, a special logic circuit such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Intergrated Circuit).

Processors suitable for executing a computer program include, by way of example, both general and specific microprocessors, and any one or more processors of any type of digital computer. Generally, a processor will receive instructions and data from a read only memory or random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include or be operably coupled to one or more mass storage devices for storing data for receiving data from or transferring data thereto, or both, e.g. magnetic, magneto-optical disks or optical disks. However, a computer does not have to have such devices. In addition, a computer may be embedded in another device, e.g. a tablet computer, a mobile phone, a personal digital assistant (PDA), a mobile audio player, a global positioning system (GPS) receiver, just to name a few. Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media and memory devices including, by way of example, semiconductor memory devices, e.g. EPROM, EEPROM, and flash memory devices; magnetic plates, e.g. internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM discs. The processor and memory may be supplemented or included by a special logic circuit.

To provide for interaction with a user, embodiments of the invention may be implemented on a computer having a display device, such as a display device. a CRT (CRT) or LCD (liquid crystal display) monitor for displaying information to the user, and a keyboard and a pointing device, such as a keyboard. a mouse or trackball, allowing the user to provide input to the computer. Other types of devices may equally well be used to provide interaction with a user; For example, feedback provided to the user may be some form of sensory feedback, e.g. a visual feedback, an acoustic feedback or a tactile feedback; and an input from the user may be received in any form, including audible, verbal or tactile input.

Embodiments of the invention may be implemented in a computer system that includes a backend component, such as a backend component. as a data server or that contains a middleware component, e.g. an application server or that contains a frontend component, such as a client computer having a graphical user interface or web browser that allows a user to interact with an implementation of the invention, or any combination of one or more of such backend, middleware, or frontend components. The components of the system may be interconnected by some form or medium of digital data communication, e.g. a communication network. Examples of communication networks include a local area network ("LAN") and a wide area network ("WAN"), e.g. the Internet.

The computer system may include clients and servers. A client and a server are generally remote from each other and typically interact through a communication network. The relationship between client and server is created by means of computer programs that run on the respective computers and that have a client-server relationship with one another.

While this description contains many specifics, these should not be considered as limitations on the scope of the invention or on what may be claimed, but rather as descriptions of features specific to particular embodiments of the invention. Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately or in any suitable subcombination in several embodiments. Moreover, although features above may be described as acting in certain combinations, and even initially claimed as such, in some instances one or more features of a claimed combination may be excluded from the combination and the claimed combination may include be a sub-combination or a variation of a sub-combination directed.

Likewise, while operations in the drawings are shown in a particular order, it should not be understood that it is required that such operations be performed in the particular order shown or in a sequential order, or that all illustrated operations be performed to achieve desired results. In some circumstances, multitasking and parallel processing may be beneficial. Moreover, the separation of various system components in the embodiments described above should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems are generally integrated with one another in a single software product or into multiple Software products can be packed.

In any case, where an HTML file is specified, other file types or formats may be substituted. For example, an HTML file can be replaced by XML, JSON, plain text, or other types of files. In addition, where a table or hash table is specified, other data structures (such as spreadsheets, relational databases, or structured files) may be used.

Certain embodiments of the invention have been described. Other embodiments are within the scope of the following claims. For example, the steps recited in the claims may be performed in a different order and still achieve desirable results.

Claims (20)

Computer-implemented method comprising: Receiving an image presented on a display of a computing device and a transcription of an utterance spoken by a user of the computing device; Identifying a particular sub-picture contained in the picture; based on performing an image recognition on the particular sub-image, determining one or more first labels that indicate a context of the particular sub-image; based on performing text recognition on an area of the image other than the particular sub-image, determining one or more second labels that indicate the context of the particular sub-image; based on the transcription, the first labels and the second labels generating a search query; and Deliver, to the output, the query. Method according to Claim 1 wherein generating the search query comprises replacing one or more of the first labels or the second labels for terms of transcription. Method according to one of the preceding claims, comprising: Generating, for each of the first labels and the second labels, a label confidence value indicating a likelihood that the label corresponds to a portion of the particular sub-image that is of primary interest to the user; and Selecting one or more of the first labels and the second labels based on the respective label confidence values, wherein the search request is generated based on the one or more selected first labels and second labels. Method according to one of the preceding claims, wherein generating a search query comprises: Access historical query data containing previous search queries provided by other users; Generating, based on the transcription, the first labels and the second labels, one or more candidate search queries; Comparing the historical query data with the one or more candidate search queries; and based on comparing the historical query data with the one or more candidate search queries, selecting the search query among the one or more candidate search queries. Method according to one of Claims 1 to 3 comprising: based on the transcription, the first labels and the second labels, generating one or more candidate search queries; Determining, for each of the one or more candidate search queries, a request trust value indicating a probability that the candidate search query is an accurate override of the transcription; and selecting, based on the query confidence values, a particular candidate query as the search query. The method of any one of the preceding claims, wherein identifying the particular sub-image contained in the image comprises: identifying one or more sub-images contained in the image; Generating, for each of the one or more sub-images contained in the image, an image confidence value indicating a likelihood that a sub-image is of primary interest to the user; and selecting the particular sub-image based on the image confidence values for the one or more sub-images. Method according to one of Claims 1 to 5 wherein identifying the particular sub-image contained in the image includes receiving data indicative of a selection of a control event at the computing device, the control event identifying the particular sub-image. The method of any one of the preceding claims, wherein the computing device is configured to capture the image and to acquire audio data corresponding to the utterance in response to detecting a predefined hotword. Method according to one of the preceding claims, comprising: Receiving an additional image corresponding to at least one further portion of the display of the computing device and an additional transcription of an additional utterance spoken by a user of the computing device; Identifying an additionally determined subpicture included in the additional image; determining, based on performing an image recognition on the additionally determined sub-image, one or more additional first labels indicating a context of the additional particular sub-image; and based on performing text recognition on a portion of the additional image that is other than the additional particular sub-image, determining one or more additional second labels that indicate the context of the additional particular sub-image; based on the additional transcription, the additional first labels and the additional second labels, generating a command; and Perform the command. Method according to Claim 9 wherein performing the command comprises performing one or more of storing the additional image in a memory, storing the particular sub-image in the memory, uploading the additional image to a server, uploading the particular sub-image to the server, importing the additional one Image to an application of the computing device and importing the particular sub-image to use the computing device. Method according to one of the preceding claims, comprising: Identifying metadata associated with the particular subimage wherein determining from the one or more first labels that indicate the context of the particular sub-image is still based on the metadata associated with the particular sub-image. System comprising: one or more computers and one or more memory devices that store instructions that are operable when executed by the one or more computers to cause the one or more computers to perform operations that include: Receiving an image presented on a display of a computing device and a transcription of an utterance spoken by a user of the computing device; and Identifying a particular sub-picture contained in the picture; based on performing an image recognition on the particular sub-image, determining one or more first labels that indicate a context of the particular sub-image; based on performing text recognition on a portion of the image other than the particular sub-image, determining one or more second labels that indicate the context of the particular sub-image; based on the transcription, the first labels and the second labels generating a search query; and Deliver, to the output, the query. System after Claim 12 wherein the operation for generating a search query comprises weighting the first labels differently from the second labels. System after Claim 12 or 13 wherein the operation for generating the search request comprises replacing one or more of the first labels or the second labels for terms of the transcription. System according to one of Claims 12 to 14 wherein the operations include: generating, for each of the first labels and the second labels, a label confidence value indicating a likelihood that the label corresponds to a portion of the particular sub-image that is of primary interest to the user; Selecting one or more of the first labels and the second labels based on the respective label confidence values, wherein the search request is generated based on the one or more selected first labels and second labels. System according to one of Claims 12 to 15 wherein the operation to generate a search query comprises: accessing historical query data containing previous search queries provided by other users; Generating, based on the transcription, the first labels and the second labels of one or more candidate search queries; Comparing the historical query data with the one or more candidate search queries; and based on comparing the historical query data with the one or more candidate search queries, selecting the search query among the one or more candidate search queries. A non-transitory computer-readable medium storing software that includes instructions executable by one or more computers that, in such an embodiment, cause the one or more computers to perform operations that include: Receiving an image presented on a display of a computing device and a transcription of an utterance spoken by a user of the computing device; Identifying a particular sub-picture contained in the picture; based on performing an image recognition on the particular sub-image, determining one or more first labels that indicate a context of the particular sub-image; based on performing text recognition on a portion of the image other than the particular sub-image, determining one or more second labels that indicate the context of the particular sub-image; based on the transcription, the first labels and the second labels, generating a search query; and Deliver, to the output, the query. Non-volatile computer readable medium after Claim 17 wherein the operation for generating a search query comprises weighting the first labels differently from the second labels. Non-volatile computer readable medium after Claim 17 or 18 wherein the operation for generating the search request comprises replacing one or more of the first labels or the second labels for terms of the transcription. Non-volatile computer readable medium after one of the Claims 17 to 19 wherein the operations include: generating, for each of the first labels and the second labels, a label confidence value indicating a likelihood that the label corresponds to a portion of the particular sub-image that is of primary interest to the user; Selecting one or more of the first labels and the second labels based on the respective label confidence values, the query being generated based on the one or more selected first labels and second labels.

Images