KR102140177B1 - Answering questions using environmental context - Google Patents

Abstract

Receive speech and environmental data, obtain transcriptions for speech, identify entities using environmental data, submit a query to the natural language query processing engine that includes data identifying at least a portion of the transcription and entities, And an apparatus comprising methods, systems, and a computer program encoded on a computer storage medium for obtaining one or more results for a query.

Description

Question answer using environment context {ANSWERING QUESTIONS USING ENVIRONMENTAL CONTEXT}

This application is filed on September 10, 2012, US provisional application number 61/698,934, filed on September 10, 2012, 61/698,949, filed on September 25, 2012, U.S. Patent Application No. 13/626,439, Priority is claimed on U.S. Patent Application No. 13/626,351 filed on September 25, 2012, and U.S. Patent Application No. 13/768,232 filed on February 15, 2013, all contents of which are hereby incorporated by reference. Included in the specification.

This specification relates to identifying results of a query based on natural language query and environmental information, and is intended to answer questions using environmental information (eg, context).

Generally, a search query includes one or more terms that the user submits to the search engine when the user requests the search engine to perform the search. Among other approaches, a user may enter query terms of a search query by typing on the keyboard or by speaking query terms into the microphone of the mobile device in the context of a voice query. Speech queries can be processed using speech recognition technology.

According to some innovative aspects of the subject matter described herein, environmental information (eg, ambient noise) can help the query processing system answer natural language queries. For example, a user may ask a question about the television program they are watching (eg, "Who is the main character in this movie?"). The user's mobile device detects the user's utterance and environmental information that may include the soundtrack audio of the television program. The mobile computing device encodes speech and environmental information as waveform data, and provides the waveform data to a server-based computing environment.

The computing environment separates the utterance from the environment data of the waveform data, and then acquires a transcription for the utterance. The computing environment further identifies the environment data and entity data related to the utterance, for example, by identifying the name of the movie. Then, from the transcription and the entity data, the computing environment can identify one or more results (eg, results in response to the user's question). In particular, the one or more results may include an answer to the user's question of "What actor is in this movie?" The computing environment can provide these results to the user of the mobile computing device.

The innovative aspects of the subject matter described herein can be implemented in a method, the method comprising receiving audio data encoding speech and environmental data, obtaining transcription of the speech, and entity using the environmental data And submitting a query comprising at least a portion of the transcription and data identifying the entity to a natural language query processing engine, and obtaining one or more results for the query.

Other embodiments of these aspects include computer programs encoded on corresponding systems, apparatuses, and computer storage devices, configured to perform the operations of the present invention.

These and other embodiments may optionally include one or more of the features described below, respectively. For example, the representation of at least one result is output. The entity can be identified further using the utterance. Create the above query. Generating the query includes associating the warrior with data identifying the entity. The associating operation includes tagging the transcript with data identifying the entity. The associating action further includes replacing a portion of the transcript with data identifying the entity. The replacing operation further includes replacing one or more words of the transcription with data identifying the entity. The operation of receiving the environment data further includes receiving environment audio data, environment image data, or both. The operation of receiving the environmental audio data further includes receiving additional audio data including background noise.

According to some innovative aspects of the subject matter described herein, an item of media content is identified based on environmental audio data and spoken natural language queries. For example, a user may ask a question about the television program they are viewing, such as "What are we watching now?". The question may include keywords (eg, “watching”) suggesting that the question is about a television show and not about other forms of media content. The user's mobile device detects the user's speech and environmental data that may include background audio of the television program. The mobile computing device encodes the speech and the environmental data into waveform data, and provides the waveform data to a server-based computing environment.

The computing environment separates the utterance from the environment data of the waveform data, and then processes the utterance to obtain transcription for the utterance. From the transcription, the computing environment detects any content type-specific keywords (eg, keyword "view"). The computing environment can then identify items of media content based on the environment data, and select a specific item of multi-content that matches a specific content type associated with keywords from the identified items. The computing environment provides a representation of a particular item of multimedia content to a user of the mobile computing device.

The innovative aspects of the subject matter described herein can be implemented in methods, which include (i) receiving audio data encoding a spoken natural language query and (ii) receiving environmental audio data, the spoken nature. Obtaining a transcription for a language query, determining a specific content type associated with one or more keywords in the transcription, providing at least a portion of the environment audio data to a content recognition engine, and by the content recognition engine And outputting a content item matching the specific content type.

Other embodiments of these aspects include computer programs encoded in corresponding systems, apparatuses, and computing storage devices and configured to perform the operations of the methods.

These and other embodiments may optionally include one or more of the features described below, respectively. For example, specific content types include movie content type, music content type, television show content type, audio podcast content type, book content type, artwork content type, trailer content type, video podcast content type, internet Video content type, or video game content type. The operation of receiving the environmental audio data further includes receiving additional audio data including background noise. The background noise is associated with a specific content type. Additional environmental data including video data or image data is received. The video data or the image data is associated with the specific content type. The operation of providing at least a portion of the environmental audio data to the content recognition engine further includes providing the portion of the environmental audio data to an audio fingerprinting engine. The determining of the specific content type may include, for each of the plurality of content types, identifying one or more keywords mapped to at least one of the plurality of content types using one or more databases. It includes more. The plurality of content types include the specific content type, and the mapped operation further includes an operation in which at least one of the keywords is mapped to the specific content type. Data identifying the content item is output.

The features include, for example, providing data identifying the specific content type to the content recognition engine, and identifying the content item is receiving data identifying the content item from the content recognition engine It further includes. The operation of receiving two or more content recognition candidates from the content recognition system and the operation of identifying the content item further include selecting a specific content recognition candidate based on the specific content type. Each of the two or more content recognition candidates is associated with a ranking score, and the method further includes adjusting the ranking scores of the two or more content recognition candidates based on the specific content type. The two or more content recognition candidates are ranked based on the adjusted ranking scores.

One or more details of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other potential features, aspects, and advantages of the subject matter will become apparent from the detailed description, drawings, and claims above.

1 depicts an exemplary system for identifying content item data based on environmental audio data and spoken natural language queries.
2 depicts a flow chart for an exemplary process of identifying content item data based on environmental audio data and extracted natural language queries.
3A and 3B depict portions of an exemplary system for identifying content items.
4 depicts an exemplary system for identifying media content items based on environmental image data and spoken natural language queries.
5 depicts a system that identifies one or more results based on environmental audio data and speech.
6 depicts a flow chart for an example process for identifying one or more results based on environmental data and speech.
7 depicts a computer system and a mobile computer device that can be used to implement the techniques described herein.
Like reference symbols in the various drawings indicate like elements.

A computing environment that responds to spoken natural language queries using environment information as context can process queries using multiple processes. In the examples of some of the processes shown in FIGS. 1 to 4, the computing environment can identify media content based on environmental information such as ambient noise. In the examples of the other processes shown in FIGS. 5 and 6, the computing environment is a spoken nature in context derived from environment information, such as data identifying media content, to provide a more satisfactory answer to spoken natural language queries. You can increase the language query.

More specifically, FIG. 1 shows a system 100 for identifying content item data based on environmental audio data and spoken natural language queries. Briefly, the system 100 can identify content item data that matches based on environmental audio data and associates a particular content type with a spoken natural language query. System 100 includes a mobile computing device 102, a disambiguation engine 104, a speech recognition engine 106, a keyword mapping engine 108, and a content recognition engine 110. Mobile computing device 102 communicates with clarification engine 104 via one or more networks. Mobile device 110 may include a microphone, camera, or other detection mechanisms for detecting utterances from user 112 and/or environmental data associated with user 112.

In some examples, user 112 is watching a TV program. In the example shown, user 112 wants to know who supervised the currently displayed TV program. In some examples, the user 112 may not know the name of the TV program currently being displayed, and thus may ask, "Who supervised this show?" The mobile computing device 102 detects this utterance along with environmental audio data associated with the environment of the user 112.

In some examples, the environment audio data associated with the environment of the user 112 may include background noise of the environment of the user 112. For example, the environmental audio data may include sounds of a TV program. In some examples, the environmental audio data associated with the currently displayed TV program is the audio of the currently displayed TV program (eg, the conversation of the currently displayed TV program, the soundtrack audio associated with the currently displayed TV program, etc.) It may include.

In some examples, mobile computing device 102 may detect environmental audio data after detecting an utterance, detect environmental audio data concurrently with detection of an utterance, or both. During operation (A), the mobile computing device 102 processes the detected speech and environmental audio data to generate waveform data 114 representing the detected speech and environmental audio data, and the waveform data 114 (for example, (For example, over a network). In some examples, environmental audio data is streamed from mobile computing device 110.

Clarification engine 104 receives waveform data 114 from mobile computing device 102. During operation (B), the articulation engine 104 processes the waveform data 114, including the separation (or extraction) of the utterance from other parts of the waveform data 114, and the utterance (eg, over a network). ) To the speech recognition engine 106. For example, the clarification engine 104 separates the utterance ("Who supervised this show") from the background noise of the environment of the user 112 (eg, the audio of the TV program currently being displayed).

In some examples, the disambiguation engine 104 may be configured to enable the separation of speech from background noise by identifying a portion of the waveform data 114 that includes speech activity or speech activity associated with a user of the computing device 102. Use a detector. In some examples, the utterance is related to a query (eg, a query related to the TV currently being displayed). In some examples, waveform data 114 includes detected speech. Correspondingly, the disambiguation engine 104 may request environmental audio data related to the utterance from the mobile computing device 102.

The speech recognition engine 106 receives a portion of the waveform data 114 corresponding to the utterance from the clarification engine 104. During operation (C), the speech recognition engine 106 obtains a transcription of the utterance and provides the transcription to the keyword mapping engine 108. Specifically, the speech recognition engine 106 processes the utterance received from the speech recognition engine 106. In some examples, processing of speech by speech recognition engine 106 includes generating transcription of speech. Generating the transcription of an utterance includes transferring the utterance into data related to text or text. In other words, the speech recognition system 106 may provide a language expression in the literal form of speech.

For example, the speech recognition system 106 transcribes the utterance to create a warrior, "Who directed this show?" In other embodiments, speech recognition system 106 provides two or more utterances of speech. For example, the speech recognition system 106 transcribes the utterance, "Who supervised this show?" And "Who supervised this shoe?"

The keyword mapping engine 108 receives transcriptions from the speech recognition engine 106. During operation (D), the keyword mapping engine 108 identifies one or more keywords in the transcription associated with a particular content type, and provides the specific content type to the clarification engine 104. In some embodiments, one or more content types include'movie','music','TV show','audio podcast','image','artwork' ,'Book','magazine','trailer','video podcast','internet video', or'video game'.

For example, the keyword mapping engine 108 identifies the keyword "directed" from the warrior "Who directed this show?". The keyword "directed" relates to the type of'TV Show' content. In some embodiments, the keyword of the transcription identified by keyword mapping engine 108 is associated with two or more content types. For example, the keyword "directed" is related to'TV Show' and'Movie' content types.

In some embodiments, keyword mapping engine 108 identifies two or more keywords in a transcription associated with a particular content type. For example, keyword mapping engine 108 identifies keywords “directed” and “show” associated with a particular content type. In some embodiments, two or more keywords identified are associated with the same content type. For example, the identified keywords "directed" and "show" are all related to the'TV Show' content type. In some embodiments, two or more of the identified keywords are associated with different content types. For example, the identified keyword "directed" is associated with the'movie' content type, and the identified keyword "show" is associated with the'TV show' content type. Keyword mapping engine 108 sends the specific content type to clarification engine 108 (eg, over a network).

In some embodiments, keyword mapping engine 108 uses one or more databases to map at least one of the keywords to at least one of the plurality of content types for each of the plurality of content types. Identifies one or more keywords in the transcription associated with. Specifically, the keyword mapping engine 108 includes (or communicates with) one database (or multiple databases). The database includes or is associated with mappings between keywords and content types. Specifically, the database establishes a connection (eg, mapping) between keywords and content types, such as allowing the keyword mapping engine 108 to identify one or more keywords in the transcription associated with particular content types. to provide.

In some embodiments, one or more mappings between keywords and content types may include unidirectional (eg, one-way mapping (ie, mapping from keywords to content types)). In some embodiments, one or more mappings between keywords and content types are bidirectional (eg, bi-directional mapping (ie, mapping from keywords to content types and content types). To keywords) In some embodiments, one or more databases map one or more keywords to two or more content types.

For example, the keyword mapping engine 108 uses one or more databases that map the keyword "directed" to'movie' and'TV show' content types. In some embodiments, the mapping between keywords and content types includes a mapping between multiple types (ie word families) and content types for the root keyword. Other forms of keywords may include other grammatical categories, such as tense (eg, past, present, future) and parts of speech (eg, nouns, verbs). For example, the database may be a root word "supervised" word family such as "directors", "direction", "directed" to one or more content types. May include mapping.

Clarification engine 104 receives data from keyword mapping engine 108 that identifies a particular content type associated with transcription of the utterance. Moreover, as described above, the articulation engine 104 receives waveform data 114 that includes environmental audio data associated with the utterance from the mobile computing device 102. During operation (E), the articulation engine 104 then provides environmental audio data and specific content types to the content recognition engine 110.

For example, the disambiguation engine 104 may display the currently displayed TV including the audio of the currently displayed TV program (eg, the conversation of the currently displayed TV program, the soundtrack audio of the currently displayed TV program, etc.). The environmental audio data related to the program and the specific content type of the transcription of the utterance (eg, the'TV show' content type) are transmitted to the content recognition engine 110.

In some embodiments, the articulation engine 104 provides a portion of the environmental audio data to the content recognition engine 110. In some examples, a portion of the environmental audio data may include background noise detected by mobile computing device 102 after detecting an utterance. In some examples, the portion of the environmental audio data may include background noise detected by the mobile computing device 102 at the same time as detection of the utterance.

In some embodiments, the background noise (of waveform data 114) is associated with a particular content type associated with the keyword of the transcription. For example, the warrior “Who supervised this show” warrior’s keyword “directed” is related to the “TV Show” content type, and the background noise (eg, environmental audio data related to the TV program currently being displayed). ) Also related to the'TV Show' content type.

The content recognition engine 110 receives environmental audio data and specific content types from the articulation engine 104. During operation (F), the content recognition engine 110 identifies content item data, which is based on the environmental audio data and matches a particular content type, and provides the content item data to the clarification engine 104. Specifically, the content recognition engine 110 properly processes the environmental audio data to identify content item data associated with the environmental audio data (eg, the name of a TV show, the name of a song, etc.). Moreover, the content recognition engine 110 matches the identified content item data with a specific content type (eg, the content type of the transcription of the utterance). The content recognition engine 110 transmits the identified content item data to the clarification engine 104 (eg, over a network).

For example, the content recognition engine 110 is based on environmental audio data related to the currently displayed TV program, and further identifies content item data matching the'TV show' content type. To this end, the content recognition engine 110 is based on a portion of the environment audio data received by the content recognition engine 110, a conversation of a currently displayed TV program, or a content item based on soundtrack audio associated with the currently displayed TV program. Identifies the data.

In some embodiments, the content recognition engine 110 is an audio fingerprinting engine that uses content fingerprints that use wavelets to identify content item data. Specifically, the content recognition engine 110 converts the waveform data 114 into a spectrogram. The content recognition engine 110 extracts spectral images from the spectrogram. Spectral images may be represented as wavelets. For each of the spectral images extracted from the spectrogram, the content recognition engine 110 extracts “top” wavelets based on the size of each of the wavelets. For each spectral image, the content recognition engine 9110 calculates the wavelet signature of the image. In some examples, wavelet signature is a truncated, quantized form of wavelet decomposition of an image.

For example, in order to describe an m×n image with a wavelet, m×n wavelets return without compression. Moreover, the content recognition engine 110 uses a subset of wavelets that characterize most of the song. Specifically, when t << m × n, t “top” wavelets (by size) are selected. Moreover, the content recognition engine 110 is concise of the sparse wavelet-vector described above using MinHash, for example, to compute sub-fingerprints for sparse bit vectors. Create an expression.

In some examples, when the environmental audio data includes at least the soundtrack audio associated with the currently displayed TV program, the content recognition engine 110 is based on the soundtrack audio associated with the currently displayed TV program and is also referred to as a'TV show. 'Identifies content item data matching the content type. Thus, in some examples, the content recognition engine 110 identifies content item data related to the name of the currently displayed TV program. For example, the content recognition engine 110 may associate a specific content item (eg, a specific TV show) with a theme song (eg, soundtrack audio), and the specific content item (eg, the specific It may be determined that the TV show) matches the specific content type (eg, the'TV show' content type). Accordingly, the content recognition engine 110 is related to the specific content item (eg, currently displayed TV program) based on environmental audio data (eg, soundtrack audio) and additionally includes the specific content type (eg For example, data matching the'TV show' content type) (eg, the name of the specific TV show) may be identified.

The disambiguation engine 104 receives the identified content item data from the content recognition engine 110. In operation (G), the articulation engine 104 then provides the identified content item data to the mobile computing device 102. For example, the disambiguation engine 104 sends the identified content item data related to the currently displayed TV program (eg, the name of the currently displayed TV program) to the mobile computing device 102.

In some examples, one or more of the mobile computing device 102, the disambiguation engine 104, the speech recognition engine 106, the keyword mapping engine 108, and the content recognition engine 110 are mobile computing device 102. ), clarification engine 104, speech recognition engine 106, keyword mapping engine 108, and a subset (or each) of content recognition engine 110. In some embodiments, one or more of the articulation engine 104, speech recognition engine 106, keyword mapping engine 108, and content recognition engine 110 are one or more computing servers, distributed computing system. Or, it may be implemented using one or more computing devices, such as one server farm or cluster.

In some embodiments, as described above, environmental audio data may be streamed from mobile computing device 110 to clarification engine 104. When the environmental audio data is streamed, the above-described processing (e.g., operations (A)-(H)) is performed when the environmental audio data is received by the articulation engine 104 (i.e., is performed gradually). ). In other words, when each portion of the environmental audio data is received (eg, streamed) by the disambiguation engine 104, operations (A)-(H) are performed repeatedly until the content item data is identified. do.

2 shows a flow diagram of an example process 200 for identifying content item data based on environmental audio data and spoken natural language queries. The example process 200 can be executed by one or more computing devices. For example, mobile computing device 102, clarification engine 104, speech recognition engine 106, keyword mapping engine 108, and/or content recognition engine 110 execute exemplary process 200. Can be used for

Audio data encoding the spoken natural language query and environmental audio data are received (202). For example, masterpiece engine 104 receives waveform data 114 from mobile computing device 102. Waveform data 114 includes the user's uttered natural query (eg, “Who supervised this show?”) and environmental audio data (eg, the audio of the TV program currently being displayed). The clarification engine 104 separates the spoken natural language query ("Who supervised the show?") from the background noise of the environment of the user 112 (eg, the audio of the currently displayed TV program).

Transcription of the natural language query is obtained (204). For example, the speech recognition system 106 transcribes the natural language query to create a transcription of the natural language query (eg, "Who supervised this show?").

The particular content type associated with one or more keywords in the transcription is determined (206). For example, one or more keywords within a warrior (eg, "Who supervised this show?") associated with a particular content type (eg,'TV Show' content type) by keyword mapping engine 108. (E.g., "has been supervised"). In some embodiments, the keyword mapping engine 108 uses one or more databases to map at least one of the keywords to at least one of the plurality of content types for each of the plurality of content types, one in the transcription. Or more keywords. The database provides a link (eg, mapping) between keywords (eg, “has been supervised”) and content types (eg,'TV Show' content type).

At least a portion of the environmental audio data is provided to the content recognition engine 208. For example, the articulation engine 104 provides the content recognition engine 110 with at least a portion of the environment audio data (eg, audio of a currently displayed TV program) encoded by the waveform data 114. In some examples, the disambiguation engine 104 is also content-aware of a particular content type (eg, a'TV Show' content type) associated with one or more keywords (eg, "has been supervised") within the enterprise. Provided to the engine 110.

Content items that are output by the content recognition engine and match specific content types are identified (210 ). For example, the content recognition engine 110 is based on environmental audio data (eg, audio of a currently displayed TV program) and content that matches a specific content type (eg,'TV show' content type). Identifies item or content item data.

3A and 3B show portions of systems 300a and 300b, respectively, for identifying content item data. In particular, FIGS. 3A and 3B are clarification engines 304a and 304b, respectively; And content recognition engines 310a and 310b, respectively. The clarification engines 304a, 304b are similar to the clarification engine 104 of the system 100 shown in FIG. 1; The content recognition engines 310a and 310b are similar to the content recognition engine 110 of the system 100 shown in FIG. 1.

3A shows a portion 300a that includes a content recognition engine 310a. The content recognition engine 310a may identify content items matching a specific content type based on the environment data. In other words, the content recognition engine 310a appropriately processes the environment data based on the environment data to identify the content item, and further selects one or more identified content item data so that the selected content item data matches a specific content type. have.

Specifically, during operation (A), the clarification engine 304a provides environment data and specific content types to the content recognition engine 310a. In some embodiments, the articulation engine 304a provides a portion of the environment data to the content recognition engine 310a.

The content recognition engine 310a receives environmental data and specific content types from the clarification engine 304a. Then, during operation (B), the content recognition engine 310a identifies content item data matching a specific content type based on the environment data and provides the identified content item data to the clarification engine 304a. Specifically, the content recognition engine 310a identifies content item data (eg, a TV show name, song title, etc.) based on the environment data. The content recognition engine 310a then selects one or more of the identified content item data that matches the particular content type. In other words, the content recognition engine 310a filters the identified content item data based on the specific content type. The content recognition engine 310a transmits the identified content item data (eg, over a network) to the clarification engine 304a.

In some examples, as described above with respect to FIG. 1, when the environment data includes at least a soundtrack audio associated with the currently displayed TV program, the content recognition engine 310a is configured to display the currently displayed TV program. Content item data is identified based on the associated soundtrack audio. The content recognition engine 310a then filters the identified content item data based on the “TV Show” content type. For example, the content recognition engine 310a identifies “theme song name” and “TV show name” associated with soundtrack audio. The content recognition engine 310a then filters the identified content item data such that the identified content item data also matches the “TV Show” content type. For example, the content recognition engine 310a selects "TV show name" identification data and transmits "TV show name" identification data to the clarification engine 304a.

In some examples, the content recognition engine 310a selects a corpus (or index) based on the content type (eg, “TV Show content type”). Specifically, the content recognition engine 310a may have access to the first index related to the “TV Show” content type and the second index related to the “Movie” content type. The content recognition engine 310a appropriately selects the first index based on the “TV Show” content type. Thus, by selecting the first index (and not selecting the second index), the content recognition engine 310a can more effectively identify content item data (eg, the name of a TV show).

The disambiguation engine 304a receives content item data from the content recognition engine 310a. For example, the clarification engine 304a receives “TV show name” identification data from the content recognition engine 310a. The clarification engine 304a then provides identification data to a third party (eg, the mobile computing device 102 of FIG. 1) during operation (C). For example, the disambiguation engine 304a provides the data “TV Show Name” identification data to a third party.

3B is a diagram illustrating a part 300b including the content recognition engine 310b. The content recognition engine 310b may identify content item data based on the environment data. In other words, the content recognition engine 310b appropriately processes the environment data to identify the content item data based on the environment data, and provides the content item data to the clarification engine 304b. The content recognition engine 310b selects at least one of the identified content item data such that the selected content item data matches a specific content type.

Specifically, during operation (A), the articulation engine 304b provides environmental data to the content recognition engine 310b. In some embodiments, the articulation engine 304b provides a portion of the environment data to the content recognition engine 310b.

The content recognition engine 310b receives environmental data from the clarification engine 304b. Thereafter, during operation (B), the content recognition engine 310b identifies the content item data based on the environment data and provides the identified content item data to the clarification engine 304b. Specifically, the content recognition engine 310b identifies content item data associated with two or more content items (eg, TV show name, song title, etc.) based on the environment data. The content recognition engine 310b sends two or more candidates representing the identified content item data (eg, over a network) to the disambiguation engine 304b.

In some examples, as described above with respect to FIG. 1, when the environment data includes at least soundtrack audio associated with the currently displayed TV program, the content recognition engine 310b is a soundtrack associated with the currently displayed TV program. Identifies content item data associated with two or more content items based on audio. For example, the content recognition engine 310b identifies the “theme song name” and “TV show name” associated with the soundtrack audio, and the “theme song name” and the “TV show name” identification data clarification engine 304b To be transferred.

The articulation engine 304b receives two or more candidates from the content recognition engine 310b. For example, the clarification engine 304b receives “theme song name” and “TV show name” candidates from the content recognition engine 310b. Thereafter, during operation (C), the disambiguation engine 304b selects one of the two or more candidates based on the particular content type, and selects the selected candidate by a third party (eg, the mobile computing device 102 of FIG. 1 ). To give. Specifically, as described above with respect to FIG. 1, the disambiguation engine 304b may receive a particular content type (eg, associated with the utterance) in advance. The clarification engine 304b selects a specific candidate from two or more candidates based on a specific content type. Specifically, the clarification engine 304b selects a specific candidate that matches a specific content type from two or more candidates. For example, the clarification engine 304b selects the'TV Show Name' candidate because the'TV Show Name' candidate matches the'TV Show' content type.

In some embodiments, two or more candidates from content recognition engine 310b are associated with a ranking score. The ranking score can be associated with any score metric determined by the clarification engine 304b. The clarification engine 304b may also adjust the ranking scores of two or more candidates based on a particular content type. Specifically, the clarification engine 304b may increase the ranking score of one or more candidates when each candidate matches a specific content type. For example, the ranking score of the candidate'TV Show Name' may increase as it matches the'TV Show' content type. In addition, the clarification engine 304b can reduce the ranking score of one or more candidates when each candidate does not match a particular content type. For example, the ranking score of the candidate “theme song name” may decrease as it does not match the “TV show” content type.

In some embodiments, two or more candidates may be ranked based on ranking scores each adjusted by clarification engine 304b. For example, as the clarification engine 304b has a higher adjusted ranking score, as the “TV Show Name” candidate has a higher adjusted ranking score when compared to the adjusted ranking score of the “Theme Song Name” candidate. You can give the rank of'theme song name' higher than the candidate. In some examples, the clarification engine 304b selects the highest ranked candidate (eg, having the highest adjusted ranking score).

4 illustrates a system 400 for identifying content item data based on environmental image data and spoken natural language queries. In short, the system 400 can identify content item data that matches a particular content type associated with the spoken natural language query based on the environment data. The system 400 includes a mobile computing device 102 of the system 100 shown in FIG. 1, a clarification engine 104, a speech recognition engine 106, a keyword mapping engine 108 and a content recognition engine 110. Each includes a mobile computing device 402, a speech recognition engine 406, a keyword mapping engine 408, and a content recognition engine 410, respectively.

In some examples, user 112 is viewing a CD album cover of a movie's soundtrack. In the illustrated example, the user 112 wants to know what songs are recorded in the soundtrack. In some examples, the user 112 may not know the name of the movie soundtrack, and thus may ask, ＂What song is there?＂ or ＂What song was played in the movie?＂ . The mobile computing device 402 detects these utterances as well as environmental image data associated with the environment of the user 112.

In some examples, environment image data associated with the environment of user 112 may include image data of user 112 environment. For example, the environmental image data includes an image of a CD album cover that depicts images associated with a movie (eg, an image of a movie poster of an associated movie). In some examples, mobile computing device 402 utilizes the camera of mobile computing device 402 to capture an image (or video) of a CD album cover to sense environmental image data.

During operation (A), the mobile computing device 402 processes the sensed utterance to generate waveform data 414 representing the sensed utterance and transmits the waveform data 414 and environmental image data (eg, over a network). ) To the clarification engine 404.

During operation (B), the articulation engine 404 receives waveform data 414 and environmental image data from the mobile computing device 402. The articulation engine 404 processes the waveform data 414 (eg, over a network) and sends a utterance to the speech recognition engine 406. In some examples, the utterance is related to a query (eg, a query related to a movie soundtrack).

The speech recognition engine 406 receives the utterance from the articulation engine 404. During operation (C), the speech recognition engine 406 obtains the transcription of the utterance and provides the transcription to the keyword mapping engine 408. Specifically, the speech recognition engine 406 processes the utterance received from the speech recognition engine 406 by generating transcription of speech.

For example, the speech recognition engine 406 transcribes the utterance to create a transcript, "What song is there?" In some embodiments, speech recognition engine 406 provides two or more transcriptions of speech. For example, the speech recognition engine 406 transcribes the utterance so that WhatWhat songs are on this?＂ and 놀이What plays are on this? )＂ warriors.

The keyword mapping engine 408 receives transcriptions from the speech recognition engine 406. During operation (D), the keyword mapping engine 408 identifies one or more keywords in the transcription associated with the particular content type and provides the specific content type to the clarification engine 404.

For example, the keyword mapping engine 408 identifies the keyword “song” from the transcription of “What song is there?”. The keyword “song” is related to the “music” content type. In some embodiments, the keyword of the transcription identified by keyword mapping engine 408 is associated with two or more content types. For example, the keyword “song” is associated with “music” and “singer” content types. Keyword mapping engine 408 sends a specific content type (eg, over a network) to clarification engine 408.

In some embodiments, similar to the above, the keyword mapping engine 408 maps, for each of the multiple content types, at least one keyword and at least one of the multiple content types, one or more. Using databases, one or more keywords associated with a particular content type within the enterprise are identified. For example, the keyword mapping engine 408 uses one or more databases that map “song” to “music” and “singer” content types.

Clarification engine 404 receives a specific content type associated with transcription of speech from keyword mapping engine 408. In addition, as described above, the clarification engine 404 receives environmental image data associated with the utterance. During operation (E), the clarification engine 404 provides environment image data and a specific content type to the content recognition engine 410.

For example, the articulation engine 404 may provide environmental image data (eg, images of a movie poster CD album cover) and specific content types (eg, “music” content types) associated with the movie soundtrack. Content recognition engine 410.

The content recognition engine 410 receives environmental image data and specific content types from the disambiguation engine 404. Then, during operation F, the content recognition engine 410 identifies content item data that is based on the environment image data and matches a particular content type and provides the identified content item data to the clarification engine 404. Specifically, the content recognition engine 410 appropriately processes the environment image data to identify content item data (eg, the name of the content item). In addition, the content recognition engine 410 matches the identified content item with a specific content type (eg, the content type of the speech transcription). The content recognition engine 408 transmits the identified content item data (eg, over a network) to the disambiguation engine 408.

For example, the content recognition engine 410 is based on environmental image data related to the image of the movie poster CD album cover, and further identifies data matching the “music” content type.

In some examples, if the environmental image data includes at least a movie poster associated with the CD album cover, the content recognition engine 410 is based on the movie poster associated with the CD album cover and also matches the “music” content type. Identifies item data. Thus, in some examples, the content recognition engine 410 identifies content item data associated with the title of the movie soundtrack. For example, the content recognition engine 410 allows a specific content item (eg, a specific movie soundtrack) to be associated with a movie poster, and a specific content item (eg, a specific movie soundtrack) to a specific content type (eg For example, it can be judged that it is consistent with “Music” content type). Thus, the content recognition engine 410 is based on the environmental image data (eg, the image of the CD album cover) and further matches a specific content type (eg, “music” content type), a specific content item (eg For example, data related to a specific movie soundtrack (eg, the title of a specific movie soundtrack) may be identified.

The disambiguation engine 404 receives the identified content item data from the content recognition engine 410. Then in operation (G), the disambiguation engine 404 provides the identified content item data to the mobile computing device 402. For example, the disambiguation engine 404 sends the identified content item data (eg, the title of the movie soundtrack) associated with the movie soundtrack to the mobile computing device 402.

As mentioned above, FIGS. 1-4 illustrate various exemplary processes within a computing environment capable of identifying media content (or other content) based on environmental information such as ambient noise. Other processes for identifying content can also be used. In general, FIGS. 5 and 6 provide spoken natural language queries for contexts in which the computing environment originates from environmental information, such as data identifying media content, to provide a more satisfactory answer to spoken natural language queries. It represents an exemplary process that can be increased.

More specifically, FIG. 5 shows a system 500 for identifying one or more results based on environmental audio and speech. In some examples, the one or more results may represent one or more answers to natural language queries. The system 500 includes a mobile computing device 502, a coordination engine 504, a speech recognition engine 506, a content identification engine 508, and a natural language query processing engine 510. Mobile computing device 502 communicates with coordination engine 504 via one or more networks. Mobile device 510 may include a microphone, camera, or other sensing mechanism for sensing utterances from user 512 and/or environmental data associated with user 512.

Similar to system 100 of FIG. 1, user 512 is watching a television program. In the illustrated example, the user 512 wants to know who supervised the television program (eg, entity) that is currently playing. In some examples, user 512 may not know the name of the television program currently being played, and thus may ask the question "Who supervised this show?" The mobile computing device 502 detects this utterance, as well as environmental data associated with the user 512 environment.

In some examples, environment data associated with the environment of the user 512 may include background noise of the environment of the user 512. For example, environmental data includes television program (eg, entity) sounds. In some examples, environmental data associated with the currently displayed television program may include audio of the currently displayed television program (eg, conversation of the currently displayed television program, soundtrack audio associated with the current display television program, etc.). Can. In some examples, the environmental data may include environmental audio data, environmental image data, or both. In some examples, the mobile computing device 502 detects environmental audio data after detecting an utterance; Simultaneously detecting environmental audio data while detecting ignition; Or both. During operation (A), the mobile computing device 502 generates sensed speech and environmental data to generate waveform data 514 representing the detected speech and the detected environmental audio data (eg, sound of a television program). Process and send the waveform data 514 to the coordination engine 504 (eg, over the network).

Coordination engine 504 receives waveform data 514 from mobile computing device 502. During operation (B), the coordination engine 504 processes the waveform data 514, including separating (or extracting) utterances from other portions of the waveform data 514 and waveform data corresponding to the waveform ( 514) to the speech recognition engine 506 (eg, over a network). For example, the coordination engine 504 separates the utterance ("Who supervised this show") from the background noise of the user 512 environment (eg, the audio of the currently displayed television program). In some examples, coordination engine 504 uses a voice detector to identify a portion of waveform data 514 that includes voice activity to facilitate separation of speech from background noise. Can. In some examples, the utterance is associated with a query (eg, a query related to the currently displayed television program).

The speech recognition engine 506 receives a portion of the waveform data 514 corresponding to the utterance from the coordination engine 504. During operation (C), the coordination engine 506 acquires a transcription of ignition and provides the transcription to the coordination engine 504. In particular, the speech recognition engine 506 appropriately processes a portion of the waveform data 514 corresponding to the utterance received from the coordination engine 504. In some examples, processing the portion of the waveform data 514 corresponding to the speech by the speech recognition engine 506 includes generating a transcription of the speech. Generating the transcription of the utterance may include transferring the utterance into text or text-related data. In other words, the speech recognition engine 506 can provide a language representation in the written form of the utterance.

For example, the speech recognition engine 506 transcribes the utterance to create a transcript of "Who supervised this show?" In some embodiments, speech recognition engine 506 provides two or more transcriptions of the speech. For example, the speech recognition engine 506 may generate warriors for "Who directed this show?" and "Who directed this shoe?" The ignition can be transferred.

Coordination engine 504 receives transcription of speech from speech recognition engine 506. Further, as mentioned above, the coordination engine 504 receives waveform data 514 including environmental audio data associated with the utterance from the mobile computing device 502. Coordination engine 504 now uses the environmental data to identify the entity. In particular, the coordination engine 504 obtains data identifying the entity from the content identification engine 508. To do so, during operation (D), the coordination engine 504 provides a portion of the waveform data 514 corresponding to the environmental data and speech to the content identification engine 508 (eg, over a network).

For example, the coordination engine 504 currently displays television that includes audio of the currently displayed television program (eg, a conversation of the currently displayed television program, soundtrack audio associated with the currently displayed television program, etc.). A portion of the waveform 514 corresponding to the environmental data and utterances associated with the program (eg, an entity) (“Who supervised this show?”) is sent to the content identification engine 508.

In some embodiments, coordination engine 504 provides a portion of environmental data to content identification engine 508. In some examples, a portion of the environmental data may include background noise sensed by the mobile computing device 502 after ignition detection. In some examples, some of the environmental data may include background noise sensed by the mobile computing device 502 at the same time as ignition detection.

The content identification engine 508 receives a portion of the waveform 514 corresponding to the environmental data and speech from the coordination engine 504. During operation (E), the content identification engine 508 identifies data identifying an entity (eg, content item data) based on environmental data and speech, and the data identifying the entity to the coordination engine 504. (For example, through a network). In particular, the content identification engine 508 is configured to identify environmental data (e.g., the name of a television show, the name of a song, etc.) and data associated with the entity (e.g., content item data) to identify the data. A portion of the waveform 514 corresponding to ignition is appropriately processed.

For example, the content identification engine 508 processes environmental audio data to identify content item data associated with the currently displayed television program. In some embodiments, the content identification engine 508 is the system 100 of FIG. 1.

The coordination engine 504 receives data identifying an entity (eg, content item data) from the content identification engine 508. Furthermore, as described above, the coordination engine 504 receives transcription from the speech recognition engine 506. During operation (F), the coordination engine 504 now provides the query including the transcription and data identifying the entity to the natural language query processing engine 510 (eg, over a network). For example, the coordination engine 504 provides the query and content item data ('television show name') including the transcription of the utterance ("Who supervised this show?") to the natural language query processing engine 510. do.

In some examples, coordination engine 504 generates a query. In some examples, coordination engine 504 obtains a query (eg, from a third server). For example, the coordination engine 504 submits the transcript of the utterance, and data identifying the entity to a third server, and returns a query based on the data identifying the transcript and entity.

In some embodiments, generating a query by the coordination engine 504 may include associating the transcription of the utterance with data identifying an entity (eg, content item data). In some examples, associating the transcription of the utterance with content item data may include tagging data identifying the entity to the transcription. For example, the coordination engine 504 may tag the warrior “Who directed this show?” with “TV Show Name” or other identifying information associated with content item data (eg, identification (ID) number). have. In some examples, associating the transcription of the utterance with data identifying the entity includes replacing a portion of the transcription with data identifying the entity. For example, the coordination engine 504 may replace a portion of the warrior "Who supervised this show" with data identifying the'television show name' or'television show name'. In some examples, replacing a portion of the transcript with data identifying the entity may include replacing one or more words of the transcript of the utterance with data identifying the entity. For example, the coordination engine 504 may replace the data identifying'TV Show Name' or'TV Show Name' in the warrior "Who supervised this show?" For example, the replacement would include a warrior who included "Who directed'television show name'" or "Who directed'ID number'?" Can cause.

The natural language query processing engine 510 receives data from the coordination engine 504 that identifies the query and entities (eg, content item data) that contain transcriptions. During operation (G), the natural language query processing engine 510 processes the query appropriately and, based on the processing, provides one or more results to the coordination engine 504 (eg, over a network). In other words, the coordination engine 510 obtains one or more results for the query (eg, from the natural language query processing engine 510).

Specifically, the natural language query processing engine 510 obtains information resources (transcription of speech and content item data) related to the query (from a collection of information resources). In some examples, the natural language query processing engine 510 matches the query against database information (eg, text documents, images, audio, video, etc.) and how well each object in the database matches the query. The score for is calculated. The natural language query processing engine 510 identifies one or more results based on the matched entities (eg, those having a score higher than a threshold score).

For example, the natural language query processing engine 510 receives a query containing the transcription of the utterance "Who supervised this show?" and the'television show name' (or other identifying information). The natural language query processing engine 510 matches the query against database information and provides one or more results matching the query. The natural language query processing engine 510 calculates the score of each matched entity.

Coordination engine 504 receives one or more results from natural language query processing engine 510. In operation (H), the coordination engine 504 now provides the one or more results to the mobile computing device 502 (eg, over a network). For example, the coordination engine 504 sends one or more results (eg, the name of the director of the television show) to the mobile computing device 502.

In some examples, one or more mobile computing devices 502, coordination engine 504, speech recognition engine 506, content identification engine 508, and natural language query processing engine 510 are mobile computing devices 502. , Coordination engine 504, speech recognition engine 506, content identification engine 508, and a subset (or each) of natural language query processing engine 510. In some embodiments, one or more coordination engines 504, speech recognition engines 506, content identification engines 508, and natural language query processing engines 510 may include one or more computing servers, distributed computing systems, or server farms. Or it may be implemented by one or more computing devices, such as clusters.

6 shows a flow diagram of an example process 600 for identifying one or more results based on environmental data and speech. The example process 600 can be executed using one or more computing devices. For example, mobile computing device 502, coordination engine 504, speech recognition engine 506, content identification engine 508, and/or natural language query processing engine 510 execute example process 600. Can be used to

Audio data encoding speech and environmental data is received (602). For example, coordination engine 504 receives waveform data 514 from mobile computing device 502. Waveform data 514 includes the user's speech (eg, “Who supervised this show?”) and environmental data (eg, the audio of the television program currently displayed). In some examples, receiving the environmental data may include receiving environmental audio data, environmental image data, or both. In some examples, receiving the environmental data includes receiving additional audio data including background noise.

A warrior of utterance is obtained (604). For example, the coordination engine 504 uses the speech recognition engine 506 to acquire transcription of speech. The speech recognition engine 506 transcribes the utterance to create a transcript of the utterance (eg, "Who supervised this show?").

The entity is identified using environmental data (606). For example, the coordination engine 504 uses the content identification engine 508 to obtain data that identifies the entity. The content identification engine 508 identifies environmental data (eg, content item data) to identify data associated with the entity (eg, content item data) associated with the environmental data (eg, television show name, song title, etc.). For example, environmental audio data (associated with the displayed television program) can be properly processed. In some examples, the content identification engine 508 can further process the waveform 514 corresponding to the utterance (simultaneously or following processing of the environmental data) to identify the entity.

In some examples, coordination engine 504 generates a query. In some examples, the step in which the query is generated by the coordination engine 504 may include associating the transcription of the utterance with data identifying the entity. In some examples, associating the transcription of the utterance with content item data may include replacing a portion of the transcription with data identifying an entity. In some examples, replacing a portion of the transcript with data identifying the entity may include replacing one or more words of the transcript of the utterance with data identifying the entity.

The query is submitted to the natural language processing engine (608). For example, coordination engine 504 submits the query to natural language processing engine 510. The query may include data identifying at least a portion of the transcript and entities (eg, content item data). For example, the coordination engine 504 provides query and content item data ('television show name') including the transcription of the utterance ("Who supervised this show?") to the natural language query processing engine 510. do.

One or more results for the query are obtained (610). For example, the coordination engine obtains one or more results for the query (eg, the name of the director of the television show) from the natural language query processing engine 510. In some examples, coordination engine 504 now provides the one or more results to mobile computing device 502.

7 shows an example of a typical computing device 700 and a typical mobile computing device 750 in which the techniques described herein can be utilized. Computing device 700 is used to represent various types of digital computers, such as laptops, desktops, workstations, PDAs, servers, blade servers, mainframes, and other suitable computers. Mobile computing device 750 is used to represent various types of mobile devices, such as PDAs, cell phones, smart phones, and other similar computing devices. The components, their connections and relationships, and their functions shown herein are meant for illustration only, and are not intended to limit the embodiment of the technology described or claimed herein.

The computing device 700 includes a processor 702, a memory 704, a storage device 706, a fast interface 708 and a fast expansion port 710 connecting to the memory 704, and a slow bus 714 and storage. And a low speed interface 712 that connects to the device 706. Each of the components 702, 704, 706, 708, 710, and 512 are connected to each other using various buses, and may be mounted in a general motherboard or other manner as appropriate. The processor 702 can process instructions for execution within the computing device 700, which includes graphical information for the GUI on an external input/output device, such as a display 716 connected to the high-speed interface 708. For display, instructions stored in memory 704 or storage device 706 are included. In other embodiments, multiple processors and/or multiple buses may be used with multiple memories and memory types, as appropriate. In addition, multiple computing devices 700 may be connected in a form (eg, a server bank, a group of blade servers, or a multi-processor system) in which each device provides a portion of the required operation.

Memory 704 stores information in computing device 700. In one embodiment, memory 704 is a volatile memory unit or units. In another embodiment, the memory 704 is a non-volatile memory unit or units. Further, the memory 704 may be other types of computer readable media, such as magnetic or optical disks.

The storage device 706 can provide mass storage for the computing device 700. In one embodiment, storage device 706 includes a floppy disk device, hard disk device, optical disk device, or tape device, flash memory or other similar solid state memory device, or a device present in a storage area network or other configuration. Device array. Computer program products may be embodied tangibly in an information medium. In addition, the computer program product may include instructions that, when executed, perform one or more methods as described above. The information carrier is a computer or machine readable medium, such as memory on memory 704, storage device 706, processor 702.

The low-speed control 712 manages low-band-intensive operations, while the high-speed control 708 manages band-intensive operations on the computing device 700. The arrangement of these functions is merely exemplary. In one embodiment, high speed control 708 is coupled to memory 704, display 716 (eg, via a graphics processor or accelerator), and is capable of accommodating various expansion cards (not shown). It is connected to the expansion port 710. In some embodiments, slow control 712 is coupled to storage device 706 and slow expansion port 714. The low speed expansion port, which may include various communication ports (eg, USB, Bluetooth, Ethernet, wireless Ethernet), is connected to one or more input/output devices such as a keyboard, pointing device, scanner, or a network adapter, for example. Through it, it can be connected to a networking device such as a switch or router.

The computing device 700 can be implemented in a number of different forms, as shown in the figure. For example, computing device 700 may be implemented as standard server 720 or multiple times in a group of such servers. In addition, computing device 700 may be implemented as part of rack server system 724. In addition, computing device 700 may be implemented within a personal computer, such as laptop computer 722. Optionally, components from computing device 700 can be combined with other components in a mobile device (not shown), such as device 750. Each of these devices includes one or more computing devices 700, 750, and the entire system can be comprised of multiple computing devices 700, 750 communicating with each other.

Computing device 750 includes a processor 752, memory 764, input/output devices such as display 754, communication interface 766, and transceiver 768, among other components. Further, the device 750 may be provided with a storage device, such as a micro drive or other device, to provide additional storage. Each of the components 750, 752, 764, 754, 766, and 568 are connected to each other using various buses, and some of the components may be mounted on a conventional motherboard or other suitable method.

Processor 752 executes instructions within computing device 750, which include instructions stored in memory 764. The processor can be implemented as a chip set of chips, including individual and multiple analog and digital processors. The processor can provide coordination between other components of the device 750, such as, for example, control of the user interface, applications executed by the device 750, and wireless communication by the computing device 750. have.

The processor 752 can communicate with the user through a control interface 758 and a display interface 756 connected to the display 754. The display 754 may be, for example, a TFT-Thin-Film-Tansistor Liquid Crystal Display (LCD) or Organic Light Emitting Diode (OLED) display, or other suitable display technology. Display interface 756 can include suitable circuitry to drive display 754 to present graphics and other information to the user. Control interface 758 receives instructions from the user and translates them to submit to processor 752. Moreover, an extension interface 762 may be provided for communication with the processor 752 to enable short-range communication between the device 750 and other devices. The extended interface 762, for example, provides wired communication in some embodiments and wireless communication in other embodiments, and multiple interfaces may also be used.

Memory 764 stores information in computing device 750. The memory 764 can be implemented as one or more of a computer-readable medium or media, a volatile memory unit or units, or a non-volatile memory unit or units. In addition, an extended memory 774 may be provided and connected to the device 750 through an extended interface 774 that includes, for example, a SIMM (Single In Line Memory Module) card interface. The extended memory 774 may provide extra storage space for the device 750, and may also store other information for the application or device 750. In particular, the extended memory 774 includes instructions for executing or assisting the above-described process, and may also include security information. Thus, for example, the extended memory 774 can be provided as a security module for the device 750 and can be programmed with instructions that enable secure use of the device 750. Moreover, the security application can be provided through the SIMM card along with additional information such as placing identification information on the SIMM card in a non-hackable manner.

The memory can include, for example, flash memory and/or NVRAM memory, as discussed below. In one embodiment, the computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods as described above. The information carrier is a computer-or machine, such as a memory 764, an extended memory 774, a memory on the processor 752, or a transmitted signal that can be received, for example, via a transceiver 768 or an extended interface 762. -It is a readable medium.

The device 750 can communicate wirelessly through a communication interface 766 that includes digital signal processing circuitry as needed. The communication interface 766 can provide communication under various modes or protocols, such as GSM voice calls, SMS, EMS, or MMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS. Such communication may be performed, for example, via radio-frequency transceiver 768. In addition, short range communication may be performed using, for example, Bluetooth, WiFi, or other such transceivers (not shown). In addition, the Global Position System (GPS) receiver module 770 can provide additional navigation- and location-related wireless data to the device 750. This wireless data can be appropriately used by applications running on the device 750.

Further, the device 750 may communicate audibly using an audio codec 760 that receives utterance information from a user and converts the utterance information into usable digital information. In addition, the audio codec 760 generates a voice that the user can hear, such as through a speaker in the handset of the device 750. Such voices may include voices from voice telephone calls, recorded voices (eg, voice messages, music files, etc.), and voices generated by applications running on device 750 as well. It may include.

The computing device 750 may be implemented in a plurality of various forms, as shown in the figure. For example, computing device 750 may be implemented as cellular telephone 780. Further, the computing device 750 can be implemented as part of a smartphone 782, PDA, or other similar mobile device.

Various embodiments of the various systems and methods described herein can be realized with digital electronic circuits, integrated circuits, application specific integrated circuits (ASICs) designed for specific purposes, computer hardware, firmware, software, and/or combinations thereof. have. Embodiments include embodiments of one or more computer programs, which are executable and/or interpretable on a programmable system, the programmable system being connected to a storage system to transmit and receive data and instructions, dedicated or general purpose And at least one programmable processor, at least one input device, and at least one output device.

Such computer programs (also called programs, software, software applications, or code) include machine instructions for programmable processors, and can be implemented in a high-level procedural and/or object-oriented programming language, and/or assembly/machine language. have. As used herein, the terms "machine-readable medium", "computer-readable medium" are devices and/or devices (eg, used to provide machine instructions and/or data to a programmable processor) Refers to magnetic disks, optical disks, memory, programmable logic devices (PLDs), and machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide instructions and/or data to a programmable processor.

In order to provide user interaction, the systems and methods described herein include a display device (eg, a cathode ray tube (CRT) or liquid crystal display (LCD) monitor) and a user that displays information to a user. It may be implemented on a computer having a keyboard and a pointing device (eg, mouse or trackball) used for input to a computer. Other categories of devices can also be used to provide user interaction. For example, the feedback provided to the user can be any form of perceptual feedback (visual, auditory or tactile feedback), and input from the user can be received in any form, including acoustic, voice or tactile input.

The various systems and methods described herein may include a back-end component (eg, data server), or middleware component (eg, application server) or front-end component (eg, as described herein). To a computing system that includes a graphical user interface (GUI) or a client computer with a web browser that a user can use to interact with embodiments of the systems and methods, or any combination of these backend, middleware, or frontend components. Can be implemented. The components of the system may be interconnected by any form or medium of digital data communication (eg, a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), and the Internet.

The computing system can include a client and a server. The client and server are generally separate from each other and interact through a communication network. The relationship between the client and the server is caused by computer programs running on each computer and having a client-server relationship to each other.

Although this specification includes some features, these should not be construed as limiting the scope of the disclosed content or the content that can be claimed, but should be construed as a description of features of exemplary embodiments of the disclosed content. Certain features that are described in the context of separate embodiments herein may be provided in combination in one embodiment. Conversely, various features described in the context of one embodiment may be provided in multiple embodiments in separate or any suitable subcombination. Furthermore, even if the features operate under certain conditions above or even have been raised as such, one or more features from the combinations raised may in some cases be deleted from the combination, and the combinations raised may be sub-combinations or variations of sub-combinations Can be directed to.

Similarly, although the operations are shown in a specific order in the figure, it is understood that such operations must be performed in a specific order or time series order shown, or that all shown actions are performed to achieve a desired result. It should not be. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the division of various system components in the embodiments described above should not be construed as requiring that division in all embodiments, and the described program components and systems are generally integrated into one software product or packaged as multiple software products. It should be understood as possible.

That is, specific embodiments of the present invention have been described. Other embodiments are also within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve the desired result. A number of embodiments have been described. Nevertheless, it will be understood that various modifications may be made without departing from the scope and spirit of the invention. For example, the various types of flows shown above can be used with reordering, adding, and removing steps. Accordingly, other embodiments are within the scope of the following claims.

104: Clarification Engine
106: speech recognition engine
108: keyword mapping engine
110: content recognition engine

Claims (20)

As a computer-implemented method,
Generating, by the mobile device, questions about unidentified media content items other devices are playing near the mobile device and (ii) generating audio recordings of the environment audio, the questions being natural language queries uttered by the user ;
In response to forwarding the audio recording to the front end server of the natural language processing system, different parts of the audio recording are processed by a speech recognition engine server associated with the natural language processing system and a content identification engine server associated with the natural language processing system. Receiving an answer to the question based on processing; And
And in response to the question, providing, by a mobile device, an answer to a question regarding the unidentified media content item. According to claim 1,
Identifying, by the mobile device, one or more keywords corresponding to the question,
Associating, by the mobile device, one or more keywords with one or more types of media content, and
And providing, by the mobile device, an answer based on the question and one or more types of media content. According to claim 2,
The one or more types of media content,
A computer-implemented method comprising at least one of: movies, music, television shows, audio podcasts, images, artwork, books, magazines, trailers, videos, podcasts, internet videos, and video games. According to claim 2,
The step of providing an answer based on one or more types of media content is:
Identifying two or more candidate answers to the question,
Generating ranked scores for each of two or more candidate answers, the ranking scores being based on one or more types of media content, and
And providing an answer based on the question and ranking score. According to claim 1,
A computer implemented method further comprising the step of streaming environmental audio. According to claim 1,
A computer-implemented method, wherein the speech recognition engine server associated with the natural language processing system and the content identification server associated with the natural language processing system are the same server. According to claim 1,
Detecting, by the mobile device, environmental image data associated with the item of media content, and
And providing, by the mobile device, an answer based on the question and environment image data. The method of claim 7,
Identifying, by the mobile device, one or more types of media content based on the environmental image data, and
And providing, by the mobile device, an answer based on the question, environmental image data, and one or more types of media content. As a system,
It includes one or more computers and one or more storage devices that store instructions that, when executed by the one or more computers, cause one or more computers to perform the operations, the operations being:
Generating, by the mobile device, (i) a question about an unidentified media content item that another device is playing near the mobile device, and (ii) generating an audio recording of the environment audio, the question being a natural language query uttered by the user ;
In response to forwarding the audio recording to the front end server of the natural language processing system, different parts of the audio recording are processed by a speech recognition engine server associated with the natural language processing system and a content identification engine server associated with the natural language processing system. Receiving an answer to the question based on processing; And
In response to the question, providing, by the mobile device, an answer to the question regarding the unidentified media content item. The method of claim 9,
The above operations,
Identifying one or more keywords corresponding to the question,
Associating one or more keywords with one or more types of media content, and
And providing an answer based on a question and one or more types of media content. The method of claim 10,
The one or more types of media content,
A system that includes at least one of movies, music, television shows, audio podcasts, images, artwork, books, magazines, trailers, videos, podcasts, internet video and video games. The method of claim 10,
The step of providing an answer based on one or more types of media content is:
Identifying two or more candidate answers to the question,
Generating a ranking score for each of two or more candidate answers, the ranking score being based on one or more types of media content, and
And further comprising providing an answer based on the question and ranking score. The method of claim 9,
The above operations,
And streaming environmental audio. The method of claim 9,
A system characterized in that the speech recognition engine server associated with the natural language processing system and the content identification server associated with the natural language processing system are the same server. The method of claim 9,
The above operations,
Detecting environmental image data associated with the media content item, and
A system comprising providing answers based on question and environmental image data. The method of claim 15,
The above operations,
Identifying one or more types of media content based on the environmental image data, and
A system comprising providing an answer based on a question, environmental image data, and one or more types of media content. A non-transitory computer readable medium storing software executable by one or more computers and including instructions that, when executed, cause one or more computers to perform the actions, the actions being:
(i) generating a question about an unidentified media content item that another device is playing near the mobile device and (ii) generating an audio recording of the environment audio, the question being a natural language query uttered by the user;
In response to forwarding the audio recording to the front end server of the natural language processing system, different parts of the audio recording are processed by a speech recognition engine server associated with the natural language processing system and a content identification engine server associated with the natural language processing system. Receiving an answer to the question based on processing; And
And in response to the question, providing an answer to a question about the unidentified media content item. The method of claim 17, wherein the operations,
Identifying one or more keywords corresponding to the question,
Associating one or more keywords with one or more types of media content, and
A non-transitory computer readable medium comprising providing an answer based on a question and one or more types of media content. The method of claim 17,
The above operations,
And streaming environmental audio. The method of claim 17,
The above operations,
Detecting environmental image data associated with the media content item, and
A non-transitory computer readable medium comprising providing answers based on question and environmental image data.

Images