US20220148590A1 - Architecture for multi-domain natural language processing - Google Patents
Architecture for multi-domain natural language processing Download PDFInfo
- Publication number
- US20220148590A1 US20220148590A1 US17/454,716 US202117454716A US2022148590A1 US 20220148590 A1 US20220148590 A1 US 20220148590A1 US 202117454716 A US202117454716 A US 202117454716A US 2022148590 A1 US2022148590 A1 US 2022148590A1
- Authority
- US
- United States
- Prior art keywords
- data
- intent
- nlu
- domain
- natural language
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003058 natural language processing Methods 0.000 title 1
- 238000012545 processing Methods 0.000 claims abstract description 121
- 238000000034 method Methods 0.000 claims abstract description 65
- 230000004044 response Effects 0.000 claims description 49
- 238000004891 communication Methods 0.000 claims description 10
- 230000008569 process Effects 0.000 abstract description 52
- 238000013518 transcription Methods 0.000 abstract description 31
- 230000035897 transcription Effects 0.000 abstract description 31
- 230000003993 interaction Effects 0.000 abstract description 20
- 230000009471 action Effects 0.000 abstract description 8
- 239000000945 filler Substances 0.000 description 10
- 238000010586 diagram Methods 0.000 description 5
- ZYXYTGQFPZEUFX-UHFFFAOYSA-N benzpyrimoxan Chemical compound O1C(OCCC1)C=1C(=NC=NC=1)OCC1=CC=C(C=C1)C(F)(F)F ZYXYTGQFPZEUFX-UHFFFAOYSA-N 0.000 description 4
- 230000000977 initiatory effect Effects 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 230000001413 cellular effect Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000009118 appropriate response Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000004883 computer application Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 238000013077 scoring method Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
- 238000012549 training Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/22—Procedures used during a speech recognition process, e.g. man-machine dialogue
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/279—Recognition of textual entities
- G06F40/284—Lexical analysis, e.g. tokenisation or collocates
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/279—Recognition of textual entities
- G06F40/289—Phrasal analysis, e.g. finite state techniques or chunking
- G06F40/295—Named entity recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/30—Semantic analysis
- G06F40/35—Discourse or dialogue representation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/40—Processing or translation of natural language
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/40—Processing or translation of natural language
- G06F40/55—Rule-based translation
- G06F40/56—Natural language generation
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L13/00—Speech synthesis; Text to speech systems
- G10L13/08—Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/26—Speech to text systems
Definitions
- Spoken language processing systems include various modules and components for receiving speech input from a user, determining what the user said, and determining what the user meant.
- a spoken language processing system includes an automatic speech recognition (“ASR”) module that receives audio input of a user utterance and generates one or more likely transcriptions of the utterance.
- Spoken language processing systems may also include a natural language understanding (“NLU”) module that receives textual input, such as a transcription of a user utterance, and determines the meaning of the text in a way that can be acted upon, such as by a computer application. For example, a user of a mobile phone may speak a spoken command to initiate a phone call. Audio of the spoken command can be transcribed by the ASR module, and the NLU module can determine the user's intent (e.g., that the user wants to initiate the phone call feature) from the transcription and generate a command to initiate the phone call.
- ASR automatic speech recognition
- NLU natural language understanding
- TTS Text-to-speech
- a TTS system may receive text input and provide an audio presentation of the text input to a user.
- a TTS system may be configured to “read” text to a user, such as the text of an email or a list of reminders.
- GPS global positioning systems
- Some systems combine both speech recognition and TTS.
- GIS global positioning systems
- users may then continue to interact with such systems while receiving directions.
- the GPS system provides the next direction or series of directions, the user may use one of any number of predetermined commands (e.g., “cancel route,” “next turn”).
- other non-spoken user interactions may be used to interact with content that is presented aurally.
- turn-by-turn directions can be displayed via a touch screen display that allows users to select, via a touch screen or keyboard, a particular route to bypass.
- FIG. 1 is a block diagram of an illustrative networked environment in which a spoken language processing system may be implemented, showing illustrative interactions between a spoken language processing system, a client device, and a user.
- FIG. 2 is a block diagram of an illustrative spoken language processing system showing data flows between various modules.
- FIG. 3 is a block diagram of an illustrative multi-domain natural language understanding module.
- FIG. 4 is a flow diagram of an illustrative process for saving history, generating hints, and using the history and hints to process user interactions.
- FIG. 5 is a flow diagram of an illustrative process for processing an utterance that may apply to one of multiple domains.
- an automatic speech recognition (“ASR”) module of a spoken language processing system may utilize various models (e.g., language models, acoustic models) when determining the content of a spoken user interaction, also known as an utterance.
- the ASR module may utilize specialized models for a particular subject matter, also known as a domain, when the ASR module knows the domain to which the utterance relates. Such specialized models can improve the efficiency and accuracy of the ASR module.
- NLU natural language understanding
- Other modules of a spoken language processing system such as a natural language understanding (“NLU”) module, may interpret the user's words (e.g., as received from an ASR module) to determine what action the user would like to initiate, also known as a user intent.
- An NLU module may be configured to interpret the user's words into an action within a particular domain. If the user's words do not support an interpretation within the particular domain for which the NLU module is configured, the NLU processing may be inefficient and error-prone, or the NLU module may not be able to generate any reasonable interpretation.
- Some spoken language processing systems are configured to process utterances within one of any number of different domains. Such systems choose one domain in which to process an utterance. For example, a system may expect a user utterance to relate to a currently active domain and therefore only process the utterance with respect to that particular domain.
- a user may initially wish to get directions, and may initiate a series of interactions (e.g., utterances and responses) with the spoken language processing system in order to get directions. At some point while the user is getting directions, the user may decide to perform a different task instead, such as initiating playback of music. The user may issue a spoken command to initiate playback of music.
- the spoken language processing system expecting an utterance in the “directions” domain, may not understand or may misinterpret the spoken command. In some cases, the spoken language processing system may not understand the spoken command even if the spoken command is formed properly such that it would be understood by the spoken language processing system if it was processed in the music domain.
- a user utterance may be processed by an ASR module.
- a transcription or N-best list of transcriptions from the ASR module may be provided to a multi-domain NLU engine.
- the multi-domain NLU engine may use any number of individual domain-specific NLU modules to process the transcription in multiple domains at the same time or substantially the same time (e.g., in parallel or asynchronously).
- a spoken language processing system may be more responsive to user utterances and allow users to arbitrarily switch domains without first informing the spoken language processing system of the domain to which the user is switching for a particular utterance.
- Additional aspects of the disclosure relate to analyzing previous user interactions and generating processing hints, also referred to herein simply as hints, to aid in processing of user utterances.
- Each user interaction with a user device or a spoken language processing system may create history, or context, that can be used to determine the user's intent when processing a subsequent utterance.
- specific hints may be generated regarding what the user is likely to do next. For example, a user may engage in a series of directions-related and music-related activities (e.g., getting driving directions to various locations and playing music from various artists). History regarding these activities may be saved and then considered by the various modules of the spoken language processing system. Subsequent interactions may be more likely to relate to the same or similar domains as the previous interactions.
- recent interactions may be used to generate hints regarding what the user would like to do next. For example, if a spoken language processing system prompts a user for the artist or title of a requested song, there is a substantial likelihood that the next user utterance will relate to an artist or a song title. However, as mentioned above, the utterance may be processed in multiple domains so that if the user does decide to do something different or otherwise does not provide an artist or song title, the spoken language processing system may still process the utterance accurately and respond appropriately.
- a client device may include a speech recognition engine and provide the features described herein for processing user utterances in multiple domains to determine the most appropriate domain for any given utterance.
- a user may issue spoken commands or otherwise make spoken utterances to a client device, such as a mobile phone or tablet computer.
- the utterances may be transmitted to a network-accessible speech recognition server, which processes the utterances and returns a response.
- FIG. 1 illustrates such a spoken language processing system 100 in communication with a client device 102 .
- the spoken language processing system 100 may use an ASR module to process the utterance and transcribe what the user said.
- history data may be saved regarding the utterance.
- the history data may include a copy of the transcription or N-best transcriptions, or summary data regarding the transcription.
- the history data may also include a timestamp so that the spoken language processing system 100 may determine which history data is the most recent, etc.
- the spoken language processing system 100 may use a multi-domain NLU engine to determine what the user would like to do, also known as the user intent, based on the transcription from the ASR module.
- the multi-domain NLU engine may also consider hints or history based on previous user interactions or other data when determining the user intent.
- the multi-domain NLU engine can include any number of domain-specific NLU modules configured to operate on text related to a particular subject matter or in a particular domain (e.g., getting directions, shopping, initiating communication with a contact, performing a search, or playing music). Domain-specific NLU modules may process text, such as a transcription, with respect to a particular domain and produce a result indicating the user's intent. Such domain-specific NLU modules are known in the art and will not be described in detail here.
- each of the domain-specific NLU modules can process the transcription in parallel, with each module producing a separate result indicating the user's intent (e.g., the user wants directions, or the user wants to listen to music).
- the multi-domain NLU engine or some other module or component of the spoken language processing system 100 can then select a particular domain-specific result on which the spoken language processing system 100 will base its response. The selection may be based on a likelihood that each individual result is reflective of the user's actual intent.
- the spoken language processing system 100 may receive and respond to several utterances related to the “directions” domain at (A). For each of the utterances, however, the domain-specific NLU modules for other domains also process the utterance in parallel with the directions domain-specific NLU module. The results returned from the other domain-specific NLU modules may be less likely to reflect the user's actual intent, however, due to the language used by the user and the language that is typically used when a user intends to interact with the individual domains. As a result, the spoken language processing system 100 may decide to produce a response to the utterance based on the result returned by the directions domain-specific NLU module.
- the response generated by the spoken language processing system 100 may be a clarifying question (e.g., if the user utterance was ambiguous), requested information (e.g., if the user utterance requested driving directions), a command for execution by the client device (e.g., if the user utterance was a spoken command to initiate voice dialing of a phone), etc.
- the spoken language processing system 100 may use a text-to-speech (“TTS”) module to generate synthesized speech that the user may consume or with which the user may interact. If the user has submitted spoken commands requesting driving directions, the spoken language processing system 100 may use the TTS module to generate audio of the driving directions.
- the spoken language processing system 100 may transmit the TTS audio to the client device 102 at (B).
- the client device 102 may receive the TTS audio and play it to the user at (C).
- the TTS audio may include directions that the user requested.
- the user may issue a spoken command that is related to a different domain at (D).
- the user utterance may be “Play Beethoven's 5th Symphony.” Such an utterance is a command to initiate playback of a particular musical recording.
- the client device 102 may transmit data regarding the user's utterance to the spoken language processing system 100 at (E).
- the spoken language processing system 100 can process the utterance in a multi-domain NLU engine at (F).
- the NLU module for the music domain may produce several potential interpretations, including one indicating that the user intends to hear a recording of Beethoven's 5th Symphony, while the NLU module for the directions domain may return several potential interpretations, including one indicating that the user intends to get directions to a restaurant named Beethoven's that is located on 5th Street.
- the domain-specific NLU modules may, in some embodiments, return scores regarding the likelihood that each result corresponds to the user's actual intent.
- the result from the music domain may have a high score because Beethoven's 5th Symphony is an actual music recording.
- the score may be further increased.
- the user's recent history includes other requests to hear music recordings of Beethoven's music or those of another classical composer, the score may be increased further.
- the result from the directions domain may also have a relatively high score, due to the recent history of directions-related interactions. The score may be relatively high particularly if there is a restaurant named Beethoven's and the restaurant is on 5th Street.
- the score may be low enough that the music domain interpretation's score is higher and therefore is the one on which the spoken language processing system 100 bases its response.
- the spoken language processing system 100 can prepare a response at (H).
- the response may be an executable command that the client device 102 executes.
- the executable command may indicate the task to perform (e.g., play music), and parameters or other data relevant to the task (e.g., a song ID or the artist and song title).
- the spoken language processing system 100 can transmit the response to the client device 102 at (I).
- the client device 102 can execute the response at (J), playing the requested musical performance to the user.
- the spoken language processing system 100 can prepare other responses, such as streaming the selected song, causing another system to stream the selected song, etc.
- the response executed by or transmitted from the spoken language processing system 100 can vary depending on the domain and the capabilities of the client device 102 , among other factors.
- FIG. 2 illustrates a spoken language processing system 100 and a client device 102 .
- the spoken language processing system 100 can be a network-accessible system in communication with the client device 102 via a communication network 150 , such as a cellular telephone network or the Internet.
- a user may use the client device 102 to submit utterances, receive information, and initiate various processes, either on the client device 102 or at the spoken language processing system 100 .
- the user can issue spoken commands to the client device 102 in order to get directions or listen to music, as described above.
- the client device 102 can correspond to a wide variety of electronic devices.
- the client device 102 may be a mobile device that includes one or more processors and a memory which may contain software applications executed by the processors.
- the client device 102 may include a speaker or other audio output component for presenting or facilitating presentation of audio content.
- the client device 102 may contain a microphone or other audio input component for accepting speech input on which to perform speech recognition.
- the client device 102 may be a mobile phone, personal digital assistant (“PDA”), mobile gaming device, media player, electronic book reader, tablet computer, laptop computer, and the like.
- the software of the client device 102 may include components for establishing communications over wireless communication networks or directly with other computing devices.
- the spoken language processing system 100 can be any computing system that is configured to communicate via a communication network.
- the spoken language processing system 100 may include any number of server computing devices, desktop computing devices, mainframe computers, and the like.
- the spoken language processing system 100 can include several devices physically or logically grouped together, such as an application server computing device configured to perform speech recognition on an utterance and a database server computing device configured to store records and speech recognition models.
- the spoken language processing system 100 can include an orchestrator module 110 , an ASR module 112 , a multi-domain NLU engine 114 , a dialog manager 116 , a natural language generation (“NLG”) module 118 , and a TTS module 120 .
- the spoken language processing system 100 can include various modules and components combined on a single device, multiple instances of a single module or component, etc.
- the spoken language processing system 100 may include a separate orchestrator 110 server that may be configured with a dialog manager module 116 ; a server or group of servers configured with ASR and NLU modules 112 , 114 ; and a server or group of servers configured with NLG and TTS modules 118 , 120 .
- the various devices of the spoken language processing system 100 may communicate via an internal communication network, such as a corporate or university network configured as a local area network (“LAN”) or a wide area network (“WAN”). In some cases, the devices of the spoken language processing system 100 may communicate over an external network, such as the Internet, or a combination of internal and external networks.
- an internal communication network such as a corporate or university network configured as a local area network (“LAN”) or a wide area network (“WAN”).
- LAN local area network
- WAN wide area network
- the devices of the spoken language processing system 100 may communicate over an external network, such as the Internet, or a combination of internal and external networks.
- the features and services provided by the spoken language processing system 100 may be implemented as web services consumable via a communication network 150 .
- the spoken language processing system 100 is provided by one more virtual machines implemented in a hosted computing environment.
- the hosted computing environment may include one or more rapidly provisioned and released computing resources, which computing resources may include computing, networking and/or storage devices.
- a hosted computing environment may also be referred to as a cloud computing environment.
- the network 150 may be a publicly accessible network of linked networks, possibly operated by various distinct parties, such as the Internet.
- the network 150 may include a private network, personal area network (“PAN”), LAN, WAN, cable network, satellite network, etc. or some combination thereof, each with access to and/or from the Internet.
- PAN personal area network
- the devices of the spoken language processing system 100 may be located within a single data center, and may communicate via a private network as described above.
- the client device 102 may communicate with spoken language processing system 100 via the Internet.
- the client device 102 may have access to the Internet via a wired or WiFi connection, or via a cellular telephone network (e.g., a Long Term Evolution or LTE network).
- a cellular telephone network e.g., a Long Term Evolution or LTE network
- the client device 102 may transmit audio and other data to the spoken language processing system 100 , and receive audio, executable commands, and other data from the spoken language processing system 100 in response.
- the orchestrator 110 may oversee receipt of input from the client device 102 and routing of the data within the spoken language processing system 100 in order to process the input and generate a response.
- the input from the client device 102 may be a user utterance transmitted to the spoken language processing system 100 via the network 150 .
- the orchestrator 110 may route the utterance to the ASR module 112 to be processed into a transcription or a lattice or N-best list of transcriptions.
- Hints can be provided to the ASR module 112 to aid in processing.
- a hint may be used to inform the ASR module 112 that the utterance is likely to include a title to a song by a particular artist because the user was just prompted for a song title after requesting to hear music by the artists.
- the ASR module 112 may use the hint to narrow its search space regarding which words may be included in the utterance, potentially providing an improvement in efficiency and accuracy.
- the transcription or N-best list of transcriptions generated by the ASR module 112 can be provided to the multi-domain NLU engine 114 to determine the specifics of the request and to formulate a command to satisfy the request.
- the dialog manager 116 or some other component of the spoken language processing system 100 may generate a hint for the multi-domain NLU engine 114 , such as data indicating the likelihood that the transcription relates to a song title of the particular artist due to the previous TTS prompt to the user for a song title to play. By hinting this likelihood to the multi-domain NLU engine 114 , NLU processing may determine the user's intent more efficiently and accurately, and scores, if used, may be weighted accordingly.
- the orchestrator 110 may provide data to the dialog manager 116 regarding the most likely user intents.
- the dialog manager 116 can then formulate a response to the user that prompts the user for the appropriate information.
- the response may be generated in the form of semantic frames (or some other semantic representation of text) which indicate the information that is required or desirable.
- Another component such as the NLG module 118 , can generate text for the question to be asked of the user.
- information about the desired output may be provided to an NLG module 118 in the form of semantic frames to generate natural language text for the responses and prompts.
- hints may also be provided to aid in NLG processing.
- the NLG module 118 can analyze the semantic frame (e.g., prompt the user for more information about getting directions to a specific location).
- the NLG module 118 can then generate textual content (including TTS markup regarding the words on which to put spoken emphasis, and other such features of spoken language) that sounds natural when read or synthesized into speech by the TTS module 120 .
- the orchestrator 110 may employ the TTS module 120 to generate an audio presentation of text content and TTS markup, whether the text is a user prompt, information requested by the user, or some other text. The orchestrator 110 may then transmit the audio generated by the TTS module 120 to the client device 102 . In addition, the orchestrator 110 or some other component of the spoken language processing system 100 may save history regarding the fully processed interaction, generate hints for subsequent interactions, and the like.
- FIG. 3 illustrates a detailed example of a multi-domain NLU engine 114 .
- the multi-domain NLU engine 114 can receive transcriptions of user utterances, textual input from users (e.g., input that users have typed on a client device 102 and submitted) or other textual input.
- the multi-domain NLU engine 114 can output interpretations of the textual input that may be acted upon by other components of the spoken language processing system 100 .
- the interpretations may include a user intent (e.g., “play a song”) and one or more named entities that add information to be used in acting upon the user intent (e.g., a “song” entity with a value “Gimme Shelter” and an “artist” entity with a value “Rolling Stones”).
- a multi-domain NLU engine 114 may include any number of single-domain NLU modules 202 .
- the multi-domain NLU engine 114 may include a music domain 202 a , a directions domain 202 b , and a phone dialing domain 202 c .
- a cross-domain ranker 204 may receive the results provided by various single-domain NLU modules 202 (e.g., the N-best list from each single domain NLU module 202 ) and compile them into a combined result (e.g., one N-best list including the results from all single-domain NLU modules 202 ).
- the slot filler 206 can correct erroneous data in the results (e.g., correct song titles by checking against a catalog of known song titles).
- the slot filler 206 can filter data that is included in the results so that only information useful to other components of the spoken language processing system 100 is included.
- the context ranker 208 can determine which result is the most appropriate.
- the context ranker 208 may analyze scores to choose a single domain or result with which to proceed, or the context ranker 208 may generate an N-best list of likely user intents across all domains.
- Each single-domain NLU module 202 may include several modules that implement various portions of NLU domain processing. As seen in FIG. 3 , a single-domain NLU module 202 can include a tokenizer 220 , a named entity recognizer 222 , and an intent classifier 224 . Some embodiments may include fewer or additional modules rather than only those shown in FIG. 3 .
- the multi-domain NLU engine 114 can receive a transcription or N-best list of transcriptions, such as those generated by an ASR module 112 .
- the multi-domain NLU engine 114 can also receive hints, such as those generated by the dialog manager 116 .
- the multi-domain NLU engine 114 may then initiate processing of the transcription in each available single-domain NLU module 202 .
- the processing in the individual domains may proceed in parallel, such that processing of a transcription in multiple domains may take substantially the same amount of time as processing in a single domain.
- the tokenizer 220 can transform a transcription from a collection of words into a series of tokens that may be processed.
- the tokenizer 220 may use various lexica to normalize words. For example, the tokenizer 220 can remove punctuations, convert digits to words (e.g., “911” to “nine one one”), etc.
- the named entity recognizer 222 can label or tag individual words or tokens for further processing. For example, in the utterance “I want to play Gimme Shelter by the Rolling Stones,” the words “I,” “want,” and “to” may be labeled as “other” because they do not, by themselves, indicate any specific user intent or provide any meaningful information.
- the next word, “play,” indicates the command that is to be performed. Such a word or token may be labeled as a user intent, because the user intends for the system to play a song.
- the phrase “Gimme Shelter” may be labeled as a song title, while the phrase “Rolling Stones” may be labeled as an artist.
- the words “by the” may be labeled “other,” because they are not actionable and provide no additional information. However, even words labeled “other” may be used to label other tokens.
- the phrase “by the” may be an important indicator that what follows is the name of a musical artist, particularly when the words “by the” are preceded by a song title.
- the named entity recognizer 222 can use hints to aid processing.
- hint information indicating that the input likely relates to the artist “Rolling Stones” can be used if the input from the ASR does not match exactly the artist name “Rolling Stones,” or if “Rolling Stones” can apply to multiple named entities in the music domain 202 a (e.g., a song, album, and artist). In such cases, if there are several different possible options for “Rolling Stones,” including artist, the named entity recognizer can select or bias the results towards “Rolling Stones” being the artist.
- the intent classifier 224 can receive input from the named entity recognizer 222 and determine which intent, known to the single-domain NLU module 202 , describes the most likely user intent. For example, the named entity recognizer 222 may label the token “play” as a user intent. The intent classifier 224 can determine which command or other response most likely captures the user intent. Illustratively, the music domain may determine that a programmatic “playSong( . . . )” command corresponds best to the user intent. The command may include arguments or parameters, such as one parameter for a song ID, or multiple parameters for the artist and song title. In some embodiments, the intent classifier 224 can use hints to aid processing. If an utterance can apply to several different intents (e.g., play a song, buy a song, etc.), the hints can help the intent classifier 224 to select or bias the results towards one that is consistent with the hint.
- the intent classifier 224 can use hints to aid processing. If an utterance can apply to several different
- the cross domain ranker 204 can receive output from the several domain-specific NLU modules 202 a , 202 b and 202 c and combine the results into a single result or N-best list of results.
- domain-specific NLU modules 202 output an N-best list of results and scores associated with those results.
- the scores may indicate a confidence in each result or a likelihood that each result is the correct result.
- the cross domain ranker 204 can then combine each of the N-best lists into a single N-best list based on the scores.
- the music domain 202 a may produce an N-best list of interpretations and corresponding scores, and the scores may be between 0.70 and 0.99 (where 0 is the minimum score and 1 is the maximum score).
- the phone domain 202 c may also produce an N-best list of interpretations and corresponding scores. In the case of the phone domain 202 c , however, the scores may be between 0.01 and 0.10. Each of the other domain-specific NLU modules 202 can produce its own N-best list of results and corresponding scores.
- the cross domain ranker 204 can combine the scores into single sorted N-best list of results and corresponding scores, such that the most likely interpretation is the first result in the N-best list or has the highest corresponding score. This scoring method is illustrative only, and not intended to be limiting. Other methods of scoring or ranking N-best lists, known to those of skill in the art, may also or alternatively be used.
- domain-specific NLU modules 202 are configured to recognize when an utterance is likely outside if its own domain.
- the domain-specific NLU modules 202 may be exposed to text from other domains during the training processes.
- Such domain-specific NLU modules 202 can assign lower scores to their results—including the top result of the domain-specific N-best list—due to the reduced likelihood that the results are correct.
- the cross domain ranker 204 can then simply combine the results into a single N-best list or choose the best result, because the domain-specific NLU modules 202 have already normalized the data based on their ability to recognize when utterances are likely outside the specific domain that they are configured to process.
- the domain-specific NLU modules 202 may not be configured to recognize when utterances are likely outside of their specific domains.
- the cross domain ranker 204 can normalize the scores from the domain-specific NLU modules 202 when generating the combined N-best list.
- the music domain 202 a may return an N-best list of results with scores between 0.70 and 0.99.
- the phone domain 202 c may also return an N-best list of results with scores between 0.70 and 0.99.
- the cross domain ranker 204 can use hints to assign weights to the scores from the various domain-specific NLU modules 202 or to otherwise normalize the results from the various domain-specific NLU modules 202 .
- the cross domain ranker 204 can produce a combined N-best list that is sorted according to normalized scores or which otherwise indicates which results are the most likely across results from all domain-specific NLU modules 202 .
- the slot filler 206 can ensure that all data required to fully implement the user intent is present, and can also remove any unnecessary data.
- the domain-specific NLU modules 202 may use more data in producing the N-best lists than is otherwise necessary for other components of the spoken language processing system 100 to take action in response to the user utterance.
- the slot filler 206 can therefore remove unnecessary information from the NLU results or otherwise modify the NLU results accordingly.
- the slot filler 206 can ensure that all information necessary implement the user intent is present. In the present example, the user may have said “Give Me Shelter” when issuing the voice command to play the song by the Rolling Stones.
- the slot filler may determine that there is no song by the Rolling Stones called “Give Me Shelter.” By searching a catalog or other data store for songs by the Rolling Stones with a similar name, the slot filler 206 may determine that correct interpretation is to play the song “Gimme Shelter” by the Rolling Stones. In some cases, the slot filler 206 may require more information from the user, and the dialog manager 116 may be employed to prompt the user for the missing information. As another example, the user utterance may be “Remind me to pay the bills on the day after tomorrow.”
- the user intent as determined by the intent classifier 224 in conjunction with the named entity recognizer 222 , may correspond to execution of a command such as “createReminder( . . .
- the command may require a parameter for the day on which the reminder is to be created. In addition, the command may require the day to be specified unambiguously.
- the slot filler 206 can translate the phrase “day after tomorrow” into a specific date for use with the “createReminder( . . . )” command. For example, if the current data is November 9 , then the slot filler 206 can calculate that the day to be used in the “createReminder( . . . )” command is November 11.
- the context ranker 208 can use hints to select the most likely result from the N-best list or to re-order the results in the N-best list. For example, the current hint may indicate that the user is likely issuing a command to play a particular song. If the top-scored or most likely result in N-best list is that the user wants to buy the song “Gimme Shelter” by the Rolling Stones, rather than play the song, the context ranker 208 can re-rank the results in the N-best list such that the most likely result is that the user wants to play the song. As described above, due to the multi-domain nature of the NLU module 114 , users can issue voice commands related to any domain rather than only the previously used domain or a currently active domain.
- the context ranker 208 may not necessarily select or elevate the result that most closely corresponds to the current hint.
- the most likely preliminary result in the N-best list may be substantially more likely than the result that most closely corresponds to the current hint (e.g., the difference between scores associated with the two items exceeds a threshold, or the result that most closely corresponds to the current hint is ranked below a threshold).
- the context ranker 208 may not increase the score or rank of the result that most closely corresponds to the hint.
- a spoken language processing system 100 may use the process 400 to save history data regarding user utterances and other interactions, and to generate hints based on prior user utterances and interactions.
- the history and hints may be used in processing subsequent user utterances and interactions.
- the process 400 begins at block 402 .
- the process 400 may begin automatically upon initiation of a speech recognition session.
- the process 400 may be embodied in a set of executable program instructions stored on a computer-readable medium, such as one or more disk drives, of a computing system of the spoken language processing system 100 , such as an orchestrator 110 server.
- the executable program instructions can be loaded into memory, such as RAM, and executed by one or more processors of the computing system.
- the spoken language processing system 100 can receive an utterance from a client device 102 .
- the user utterance may be a spoken command to play a recorded music file.
- the spoken language processing system 100 can access history and/or hints that may be useful in processing the utterance.
- the history can be a record of user intents and/or responses associated with the user within a particular time frame or during a particular voice command session. Hints may be generated during or after processing of user utterances and saved for use in processing subsequent utterances.
- data other than user utterances may be received from the client device 102 . Such data may be saved in the history and/or used to generate hints to aid in further processing, and those hints may be accessed at block 406 to process the current utterance.
- a user of a client device 102 may raise the phone up to the user's head in a vertical position.
- the mobile phone may include proximity sensors to determine that the phone is currently positioned near the user's head.
- Data regarding this particular user action may be transmitted to the spoken language processing system 100 and stored as history.
- Such a user action may indicate that he user is about to initiate voice dialing, and therefore the data may be useful in generating hints, as described below.
- the spoken language processing system 100 can process the utterance as described in detail below with respect to FIG. 5 .
- history and hints based on the utterance, the processing of the utterance, and/or the response generated may be saved for use in processing further utterances.
- the orchestrator 110 can determine at decision block 412 whether a response is to be sent to the client device 102 . For example, if an utterance is received that includes a request for playback of a particular musical recording, such as Beethoven's 5th Symphony, the dialog manager 116 of the spoken language processing system 100 may prompt the user to identify the particular recording artist (e.g., the Chicago Symphony Orchestra or the Berlin Symphony Orchestra). As described in detail herein, a TTS audio response prompting the user for the name of the recording artist may be generated and ready for transmission to the client device 102 . In addition, history may be saved regarding the prompt, and the history can be used during a subsequent execution of the process 400 . If a prompt, executable command, or some other response is ready for transmission, the orchestrator 110 can transmit the response to the client device 102 at block 414 . Otherwise, the process 400 can terminate.
- a prompt, executable command, or some other response is ready for transmission.
- Data received subsequent to the execution of the process 400 may cause the process 400 to be executed again.
- the user may provide a response to a prompt for a particular recording artist by submitting the utterance “the Chicago Symphony Orchestra.”
- data regarding the history and/or hints saved during previous executions of the process 400 may be accessed, and at block 408 the utterance may be processed.
- the hint used during processing of the subsequently received utterance may indicate that the utterance likely relates to an artist for the request to play a musical recording of Beethoven's 5th symphony.
- FIG. 5 illustrates a sample process 500 for processing a user utterance, received from a client device 102 , with a multi-domain NLU engine.
- the process 500 may be used to process user utterances in multiple domains simultaneously or substantially simultaneously, and to select the best result or result most likely to be accurate from among the multiple results that are calculated.
- the process 500 begins at block 502 .
- the process 500 may begin automatically upon receipt of an utterance from a client device 102 .
- the process 500 may be embodied in a set of executable program instructions stored on a computer-readable medium, such as one or more disk drives, of a computing device associated with the spoken language processing system 100 .
- the executable program instructions can be loaded into memory, such as RAM, and executed by one or more processors of the computing system.
- an utterance may be received by the spoken language processing server 100 .
- the user may have submitted a spoken command to initiate playback of a particular music recording. (e.g., “play Beethoven's 5th symphony by the Chicago Symphony Orchestra”).
- the utterance may be transcribed into a likely transcription or N-best list of likely transcriptions by an ASR module 112 of the spoken language processing system 100 . If any hints are available, they may be provided to the ASR module 112 in order to allow ASR module 112 to operate more efficiently and provide potentially more accurate results.
- the transcription or N-best list of likely transcriptions may be processed by any number of single-domain NLU modules in a multi-domain NLU engine.
- Each domain may process the transcription with respect to a particular subject matter, as described above.
- the domains may process the transcription in parallel or asynchronously so as to reduce the user-perceived time required to perform multi-domain NLU processing in comparison with single-domain NLU processing.
- a score may be assigned to each of the results.
- the most likely interpretation of user intent may be selected from the N-best list returned from the various single domain NLU modules.
- response is selected based on the hint that was generated during or after processing of a previously received utterance or after receiving other information from the client device 102 .
- the spoken language processing system 100 may determine a response to the utterance.
- the response can vary depending upon which domain provided the most likely analysis of the user's intent.
- an executable command to play the requested music may be the most appropriate response.
- a prompt may be generated in order to get the additional information.
- the spoken language processing system can determine whether the response includes content to be presented aurally to the user, such a text that is to be read to the user. For example, if the spoken language processing system 100 determined that the user is to be prompted for additional information regarding the user's utterance, then the process may proceed to block 516 . Otherwise, if no TTS processing is required, the process 500 may proceed to block 520 , where a command corresponding to the likely user intent is generated or some other action is performed.
- an NLG module 118 may be employed to generate text for a spoken response such that the language of the response will sound natural to the user. For example, if the user has requested information about a particular product, information about the product (e.g., price, features, availability) may be obtained and provided to the NLG module 118 .
- the NLG module 118 may output a string of text that can be synthesized as speech by a TTS module 112 .
- the TTS module 112 can generate synthesized speech from a text input, such as text generated by an NLG module 118 .
- a software module can reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of a non-transitory computer-readable storage medium.
- An exemplary storage medium can be coupled to the processor such that the processor can read information from, and write information to, the storage medium.
- the storage medium can be integral to the processor.
- the processor and the storage medium can reside in an ASIC.
- the ASIC can reside in a user terminal.
- the processor and the storage medium can reside as discrete components in a user terminal.
Abstract
Features are disclosed for processing a user utterance with respect to multiple subject matters or domains, and for selecting a likely result from a particular domain with which to respond to the utterance or otherwise take action. A user utterance may be transcribed by an automatic speech recognition (“ASR”) module, and the results may be provided to a multi-domain natural language understanding (“NLU”) engine. The multi-domain NLU engine may process the transcription(s) in multiple individual domains rather than in a single domain. In some cases, the transcription(s) may be processed in multiple individual domains in parallel or substantially simultaneously. In addition, hints may be generated based on previous user interactions and other data. The ASR module, multi-domain NLU engine, and other components of a spoken language processing system may use the hints to more efficiently process input or more accurately generate output.
Description
- This application is a continuation of U.S. patent application Ser. No. 16/400,905, filed May 1, 2019, which is a continuation of U.S. patent application Ser. No. 15/966,400, filed Apr. 30, 2018, now U.S. Pat. No. 10,283,119, which is a continuation of U.S. patent application Ser. No. 15/694,996, filed Sep. 4, 2017, now U.S. Pat. No. 9,959,869, which is a continuation of U.S. patent application Ser. No. 15/256,176, filed Sep. 2, 2016, now U.S. Pat. No. 9,754,589, which is a continuation of U.S. patent application Ser. No. 14/754,598, filed Jun. 29, 2015, now U.S. Pat. No. 9,436,678, which is a continuation of U.S. application Ser. No. 13/720,909, filed Dec. 19, 2012, now U.S. Pat. No. 9,070,366, all of which are incorporated by reference herein.
- Spoken language processing systems include various modules and components for receiving speech input from a user, determining what the user said, and determining what the user meant. In some implementations, a spoken language processing system includes an automatic speech recognition (“ASR”) module that receives audio input of a user utterance and generates one or more likely transcriptions of the utterance. Spoken language processing systems may also include a natural language understanding (“NLU”) module that receives textual input, such as a transcription of a user utterance, and determines the meaning of the text in a way that can be acted upon, such as by a computer application. For example, a user of a mobile phone may speak a spoken command to initiate a phone call. Audio of the spoken command can be transcribed by the ASR module, and the NLU module can determine the user's intent (e.g., that the user wants to initiate the phone call feature) from the transcription and generate a command to initiate the phone call.
- Text-to-speech (“TTS”) systems convert text into sound using a process sometimes known as speech synthesis. In a common implementation, a TTS system may receive text input and provide an audio presentation of the text input to a user. For example, a TTS system may be configured to “read” text to a user, such as the text of an email or a list of reminders.
- Some systems combine both speech recognition and TTS. For example, global positioning systems (“GPS”) can receive a user's spoken input regarding a particular address, generate directions for travelling to the address, and aurally present the directions to the user. In many cases, users may then continue to interact with such systems while receiving directions. After the GPS system provides the next direction or series of directions, the user may use one of any number of predetermined commands (e.g., “cancel route,” “next turn”). In addition, other non-spoken user interactions may be used to interact with content that is presented aurally. For example, turn-by-turn directions can be displayed via a touch screen display that allows users to select, via a touch screen or keyboard, a particular route to bypass.
- Embodiments of various inventive features will now be described with reference to the following drawings. Throughout the drawings, reference numbers may be re-used to indicate correspondence between referenced elements. The drawings are provided to illustrate example embodiments described herein and are not intended to limit the scope of the disclosure.
-
FIG. 1 is a block diagram of an illustrative networked environment in which a spoken language processing system may be implemented, showing illustrative interactions between a spoken language processing system, a client device, and a user. -
FIG. 2 is a block diagram of an illustrative spoken language processing system showing data flows between various modules. -
FIG. 3 is a block diagram of an illustrative multi-domain natural language understanding module. -
FIG. 4 is a flow diagram of an illustrative process for saving history, generating hints, and using the history and hints to process user interactions. -
FIG. 5 is a flow diagram of an illustrative process for processing an utterance that may apply to one of multiple domains. - Spoken language processing systems can perform more efficiently and accurately when they know the subject matter to which the spoken interactions that they process relate. For example, an automatic speech recognition (“ASR”) module of a spoken language processing system may utilize various models (e.g., language models, acoustic models) when determining the content of a spoken user interaction, also known as an utterance. The ASR module may utilize specialized models for a particular subject matter, also known as a domain, when the ASR module knows the domain to which the utterance relates. Such specialized models can improve the efficiency and accuracy of the ASR module. Other modules of a spoken language processing system, such as a natural language understanding (“NLU”) module, may interpret the user's words (e.g., as received from an ASR module) to determine what action the user would like to initiate, also known as a user intent. An NLU module may be configured to interpret the user's words into an action within a particular domain. If the user's words do not support an interpretation within the particular domain for which the NLU module is configured, the NLU processing may be inefficient and error-prone, or the NLU module may not be able to generate any reasonable interpretation.
- Some spoken language processing systems are configured to process utterances within one of any number of different domains. Such systems choose one domain in which to process an utterance. For example, a system may expect a user utterance to relate to a currently active domain and therefore only process the utterance with respect to that particular domain. A user may initially wish to get directions, and may initiate a series of interactions (e.g., utterances and responses) with the spoken language processing system in order to get directions. At some point while the user is getting directions, the user may decide to perform a different task instead, such as initiating playback of music. The user may issue a spoken command to initiate playback of music. However, the spoken language processing system, expecting an utterance in the “directions” domain, may not understand or may misinterpret the spoken command. In some cases, the spoken language processing system may not understand the spoken command even if the spoken command is formed properly such that it would be understood by the spoken language processing system if it was processed in the music domain.
- Aspects of this disclosure relate to processing user utterances in multiple domains and, based on the results, selecting the most likely or appropriate domain in which to generate a response or otherwise take further action. A user utterance may be processed by an ASR module. A transcription or N-best list of transcriptions from the ASR module may be provided to a multi-domain NLU engine. The multi-domain NLU engine may use any number of individual domain-specific NLU modules to process the transcription in multiple domains at the same time or substantially the same time (e.g., in parallel or asynchronously). After the domain-specific NLU modules have produced results (e.g., determined one or more likely interpretations) and, optionally, scores for those results (such as a confidence score or a likelihood score), the most likely or appropriate interpretation may be selected. In this way, a spoken language processing system may be more responsive to user utterances and allow users to arbitrarily switch domains without first informing the spoken language processing system of the domain to which the user is switching for a particular utterance.
- Additional aspects of the disclosure relate to analyzing previous user interactions and generating processing hints, also referred to herein simply as hints, to aid in processing of user utterances. Each user interaction with a user device or a spoken language processing system may create history, or context, that can be used to determine the user's intent when processing a subsequent utterance. In addition, specific hints may be generated regarding what the user is likely to do next. For example, a user may engage in a series of directions-related and music-related activities (e.g., getting driving directions to various locations and playing music from various artists). History regarding these activities may be saved and then considered by the various modules of the spoken language processing system. Subsequent interactions may be more likely to relate to the same or similar domains as the previous interactions. In addition, recent interactions may be used to generate hints regarding what the user would like to do next. For example, if a spoken language processing system prompts a user for the artist or title of a requested song, there is a substantial likelihood that the next user utterance will relate to an artist or a song title. However, as mentioned above, the utterance may be processed in multiple domains so that if the user does decide to do something different or otherwise does not provide an artist or song title, the spoken language processing system may still process the utterance accurately and respond appropriately.
- Although aspects of the embodiments described in the disclosure will focus, for the purpose of illustration, on a spoken language processing system exchanging data with a separate client device via a network, one skilled in the art will appreciate that the techniques disclosed herein may be applied to any number of software processes or applications. For example, a client device may include a speech recognition engine and provide the features described herein for processing user utterances in multiple domains to determine the most appropriate domain for any given utterance. Various aspects of the disclosure will now be described with regard to certain examples and embodiments, which are intended to illustrate but not limit the disclosure.
- With reference to an illustrative example, a user may issue spoken commands or otherwise make spoken utterances to a client device, such as a mobile phone or tablet computer. The utterances may be transmitted to a network-accessible speech recognition server, which processes the utterances and returns a response.
FIG. 1 illustrates such a spokenlanguage processing system 100 in communication with aclient device 102. - The spoken
language processing system 100 may use an ASR module to process the utterance and transcribe what the user said. In addition, history data may be saved regarding the utterance. The history data may include a copy of the transcription or N-best transcriptions, or summary data regarding the transcription. The history data may also include a timestamp so that the spokenlanguage processing system 100 may determine which history data is the most recent, etc. - The spoken
language processing system 100 may use a multi-domain NLU engine to determine what the user would like to do, also known as the user intent, based on the transcription from the ASR module. The multi-domain NLU engine may also consider hints or history based on previous user interactions or other data when determining the user intent. The multi-domain NLU engine can include any number of domain-specific NLU modules configured to operate on text related to a particular subject matter or in a particular domain (e.g., getting directions, shopping, initiating communication with a contact, performing a search, or playing music). Domain-specific NLU modules may process text, such as a transcription, with respect to a particular domain and produce a result indicating the user's intent. Such domain-specific NLU modules are known in the art and will not be described in detail here. - Advantageously, in a multi-domain NLU engine, each of the domain-specific NLU modules can process the transcription in parallel, with each module producing a separate result indicating the user's intent (e.g., the user wants directions, or the user wants to listen to music). The multi-domain NLU engine or some other module or component of the spoken
language processing system 100 can then select a particular domain-specific result on which the spokenlanguage processing system 100 will base its response. The selection may be based on a likelihood that each individual result is reflective of the user's actual intent. - As illustrated in
FIG. 1 , the spokenlanguage processing system 100 may receive and respond to several utterances related to the “directions” domain at (A). For each of the utterances, however, the domain-specific NLU modules for other domains also process the utterance in parallel with the directions domain-specific NLU module. The results returned from the other domain-specific NLU modules may be less likely to reflect the user's actual intent, however, due to the language used by the user and the language that is typically used when a user intends to interact with the individual domains. As a result, the spokenlanguage processing system 100 may decide to produce a response to the utterance based on the result returned by the directions domain-specific NLU module. - In some cases, the response generated by the spoken
language processing system 100 may be a clarifying question (e.g., if the user utterance was ambiguous), requested information (e.g., if the user utterance requested driving directions), a command for execution by the client device (e.g., if the user utterance was a spoken command to initiate voice dialing of a phone), etc. For example, the spokenlanguage processing system 100 may use a text-to-speech (“TTS”) module to generate synthesized speech that the user may consume or with which the user may interact. If the user has submitted spoken commands requesting driving directions, the spokenlanguage processing system 100 may use the TTS module to generate audio of the driving directions. The spokenlanguage processing system 100 may transmit the TTS audio to theclient device 102 at (B). Theclient device 102 may receive the TTS audio and play it to the user at (C). As seen inFIG. 1 , the TTS audio may include directions that the user requested. - After any number of utterances related to the directions domain, the user may issue a spoken command that is related to a different domain at (D). Illustratively, the user utterance may be “Play Beethoven's 5th Symphony.” Such an utterance is a command to initiate playback of a particular musical recording. The
client device 102 may transmit data regarding the user's utterance to the spokenlanguage processing system 100 at (E). - After generating a transcript for the utterance through the use of the ASR module, the spoken
language processing system 100 can process the utterance in a multi-domain NLU engine at (F). In the present example, the NLU module for the music domain may produce several potential interpretations, including one indicating that the user intends to hear a recording of Beethoven's 5th Symphony, while the NLU module for the directions domain may return several potential interpretations, including one indicating that the user intends to get directions to a restaurant named Beethoven's that is located on 5th Street. There may be any number of other domain-specific NLU modules that return results, and the results may be more or less plausible depending on the particular domain. - In addition to the results, the domain-specific NLU modules may, in some embodiments, return scores regarding the likelihood that each result corresponds to the user's actual intent. In the current example, the result from the music domain may have a high score because Beethoven's 5th Symphony is an actual music recording. In addition, if the user has access to a musical recording of Beethoven's 5th Symphony, the score may be further increased. Moreover, if the user's recent history includes other requests to hear music recordings of Beethoven's music or those of another classical composer, the score may be increased further. The result from the directions domain may also have a relatively high score, due to the recent history of directions-related interactions. The score may be relatively high particularly if there is a restaurant named Beethoven's and the restaurant is on 5th Street. However, due to the lack of a completely applicable result (e.g., the terms “play” and “symphony” do not relate to directions), the score may be low enough that the music domain interpretation's score is higher and therefore is the one on which the spoken
language processing system 100 bases its response. - After automatically selecting the interpretation from the music domain at (G), the spoken
language processing system 100 can prepare a response at (H). In the current example, the response may be an executable command that theclient device 102 executes. The executable command may indicate the task to perform (e.g., play music), and parameters or other data relevant to the task (e.g., a song ID or the artist and song title). The spokenlanguage processing system 100 can transmit the response to theclient device 102 at (I). Theclient device 102 can execute the response at (J), playing the requested musical performance to the user. In some embodiments, the spokenlanguage processing system 100 can prepare other responses, such as streaming the selected song, causing another system to stream the selected song, etc. In practice, the response executed by or transmitted from the spokenlanguage processing system 100 can vary depending on the domain and the capabilities of theclient device 102, among other factors. - Prior to describing embodiments of processes for analyzing user utterances in a multi-domain NLU engine in detail, several illustrative interactions and an example environment in which the processes may be implemented will be described.
FIG. 2 illustrates a spokenlanguage processing system 100 and aclient device 102. The spokenlanguage processing system 100 can be a network-accessible system in communication with theclient device 102 via acommunication network 150, such as a cellular telephone network or the Internet. A user may use theclient device 102 to submit utterances, receive information, and initiate various processes, either on theclient device 102 or at the spokenlanguage processing system 100. For example, the user can issue spoken commands to theclient device 102 in order to get directions or listen to music, as described above. - The
client device 102 can correspond to a wide variety of electronic devices. In some embodiments, theclient device 102 may be a mobile device that includes one or more processors and a memory which may contain software applications executed by the processors. Theclient device 102 may include a speaker or other audio output component for presenting or facilitating presentation of audio content. In addition, theclient device 102 may contain a microphone or other audio input component for accepting speech input on which to perform speech recognition. Illustratively, theclient device 102 may be a mobile phone, personal digital assistant (“PDA”), mobile gaming device, media player, electronic book reader, tablet computer, laptop computer, and the like. The software of theclient device 102 may include components for establishing communications over wireless communication networks or directly with other computing devices. - The spoken
language processing system 100 can be any computing system that is configured to communicate via a communication network. For example, the spokenlanguage processing system 100 may include any number of server computing devices, desktop computing devices, mainframe computers, and the like. In some embodiments, the spokenlanguage processing system 100 can include several devices physically or logically grouped together, such as an application server computing device configured to perform speech recognition on an utterance and a database server computing device configured to store records and speech recognition models. - The spoken
language processing system 100 can include anorchestrator module 110, anASR module 112, amulti-domain NLU engine 114, adialog manager 116, a natural language generation (“NLG”)module 118, and aTTS module 120. In some embodiments, the spokenlanguage processing system 100 can include various modules and components combined on a single device, multiple instances of a single module or component, etc. For example, the spokenlanguage processing system 100 may include aseparate orchestrator 110 server that may be configured with adialog manager module 116; a server or group of servers configured with ASR andNLU modules TTS modules language processing system 100 may communicate via an internal communication network, such as a corporate or university network configured as a local area network (“LAN”) or a wide area network (“WAN”). In some cases, the devices of the spokenlanguage processing system 100 may communicate over an external network, such as the Internet, or a combination of internal and external networks. - In some embodiments, the features and services provided by the spoken
language processing system 100 may be implemented as web services consumable via acommunication network 150. In further embodiments, the spokenlanguage processing system 100 is provided by one more virtual machines implemented in a hosted computing environment. The hosted computing environment may include one or more rapidly provisioned and released computing resources, which computing resources may include computing, networking and/or storage devices. A hosted computing environment may also be referred to as a cloud computing environment. - The
network 150 may be a publicly accessible network of linked networks, possibly operated by various distinct parties, such as the Internet. In other embodiments, thenetwork 150 may include a private network, personal area network (“PAN”), LAN, WAN, cable network, satellite network, etc. or some combination thereof, each with access to and/or from the Internet. For example, the devices of the spokenlanguage processing system 100 may be located within a single data center, and may communicate via a private network as described above. Theclient device 102 may communicate with spokenlanguage processing system 100 via the Internet. Theclient device 102 may have access to the Internet via a wired or WiFi connection, or via a cellular telephone network (e.g., a Long Term Evolution or LTE network). - In operation, the
client device 102 may transmit audio and other data to the spokenlanguage processing system 100, and receive audio, executable commands, and other data from the spokenlanguage processing system 100 in response. Theorchestrator 110 may oversee receipt of input from theclient device 102 and routing of the data within the spokenlanguage processing system 100 in order to process the input and generate a response. - For example, the input from the
client device 102 may be a user utterance transmitted to the spokenlanguage processing system 100 via thenetwork 150. Theorchestrator 110 may route the utterance to theASR module 112 to be processed into a transcription or a lattice or N-best list of transcriptions. Hints can be provided to theASR module 112 to aid in processing. Illustratively, a hint may be used to inform theASR module 112 that the utterance is likely to include a title to a song by a particular artist because the user was just prompted for a song title after requesting to hear music by the artists. TheASR module 112 may use the hint to narrow its search space regarding which words may be included in the utterance, potentially providing an improvement in efficiency and accuracy. - The transcription or N-best list of transcriptions generated by the
ASR module 112 can be provided to themulti-domain NLU engine 114 to determine the specifics of the request and to formulate a command to satisfy the request. In addition, thedialog manager 116 or some other component of the spokenlanguage processing system 100 may generate a hint for themulti-domain NLU engine 114, such as data indicating the likelihood that the transcription relates to a song title of the particular artist due to the previous TTS prompt to the user for a song title to play. By hinting this likelihood to themulti-domain NLU engine 114, NLU processing may determine the user's intent more efficiently and accurately, and scores, if used, may be weighted accordingly. If additional information is required (e.g., the utterance was ambiguous or did not include all necessary information), theorchestrator 110 may provide data to thedialog manager 116 regarding the most likely user intents. Thedialog manager 116 can then formulate a response to the user that prompts the user for the appropriate information. The response may be generated in the form of semantic frames (or some other semantic representation of text) which indicate the information that is required or desirable. Another component, such as theNLG module 118, can generate text for the question to be asked of the user. - When text or synthesized speech is to be transmitted to the
client device 102, information about the desired output may be provided to anNLG module 118 in the form of semantic frames to generate natural language text for the responses and prompts. In some embodiments, hints may also be provided to aid in NLG processing. TheNLG module 118 can analyze the semantic frame (e.g., prompt the user for more information about getting directions to a specific location). TheNLG module 118 can then generate textual content (including TTS markup regarding the words on which to put spoken emphasis, and other such features of spoken language) that sounds natural when read or synthesized into speech by theTTS module 120. - The
orchestrator 110 may employ theTTS module 120 to generate an audio presentation of text content and TTS markup, whether the text is a user prompt, information requested by the user, or some other text. Theorchestrator 110 may then transmit the audio generated by theTTS module 120 to theclient device 102. In addition, theorchestrator 110 or some other component of the spokenlanguage processing system 100 may save history regarding the fully processed interaction, generate hints for subsequent interactions, and the like. -
FIG. 3 illustrates a detailed example of amulti-domain NLU engine 114. Themulti-domain NLU engine 114 can receive transcriptions of user utterances, textual input from users (e.g., input that users have typed on aclient device 102 and submitted) or other textual input. Themulti-domain NLU engine 114 can output interpretations of the textual input that may be acted upon by other components of the spokenlanguage processing system 100. The interpretations may include a user intent (e.g., “play a song”) and one or more named entities that add information to be used in acting upon the user intent (e.g., a “song” entity with a value “Gimme Shelter” and an “artist” entity with a value “Rolling Stones”). - A
multi-domain NLU engine 114 may include any number of single-domain NLU modules 202. For example, themulti-domain NLU engine 114 may include amusic domain 202 a, adirections domain 202 b, and aphone dialing domain 202 c. Across-domain ranker 204 may receive the results provided by various single-domain NLU modules 202 (e.g., the N-best list from each single domain NLU module 202) and compile them into a combined result (e.g., one N-best list including the results from all single-domain NLU modules 202). Theslot filler 206 can correct erroneous data in the results (e.g., correct song titles by checking against a catalog of known song titles). In addition, theslot filler 206 can filter data that is included in the results so that only information useful to other components of the spokenlanguage processing system 100 is included. Thecontext ranker 208 can determine which result is the most appropriate. Thecontext ranker 208 may analyze scores to choose a single domain or result with which to proceed, or thecontext ranker 208 may generate an N-best list of likely user intents across all domains. - Each single-domain NLU module 202 may include several modules that implement various portions of NLU domain processing. As seen in
FIG. 3 , a single-domain NLU module 202 can include atokenizer 220, a namedentity recognizer 222, and anintent classifier 224. Some embodiments may include fewer or additional modules rather than only those shown inFIG. 3 . - In operation, the
multi-domain NLU engine 114 can receive a transcription or N-best list of transcriptions, such as those generated by anASR module 112. Themulti-domain NLU engine 114 can also receive hints, such as those generated by thedialog manager 116. Themulti-domain NLU engine 114 may then initiate processing of the transcription in each available single-domain NLU module 202. The processing in the individual domains may proceed in parallel, such that processing of a transcription in multiple domains may take substantially the same amount of time as processing in a single domain. - The
tokenizer 220 can transform a transcription from a collection of words into a series of tokens that may be processed. Thetokenizer 220 may use various lexica to normalize words. For example, thetokenizer 220 can remove punctuations, convert digits to words (e.g., “911” to “nine one one”), etc. - The named
entity recognizer 222 can label or tag individual words or tokens for further processing. For example, in the utterance “I want to play Gimme Shelter by the Rolling Stones,” the words “I,” “want,” and “to” may be labeled as “other” because they do not, by themselves, indicate any specific user intent or provide any meaningful information. The next word, “play,” indicates the command that is to be performed. Such a word or token may be labeled as a user intent, because the user intends for the system to play a song. The phrase “Gimme Shelter” may be labeled as a song title, while the phrase “Rolling Stones” may be labeled as an artist. Finally, the words “by the” may be labeled “other,” because they are not actionable and provide no additional information. However, even words labeled “other” may be used to label other tokens. For example, the phrase “by the” may be an important indicator that what follows is the name of a musical artist, particularly when the words “by the” are preceded by a song title. In some embodiments, the namedentity recognizer 222 can use hints to aid processing. For example, hint information indicating that the input likely relates to the artist “Rolling Stones” can be used if the input from the ASR does not match exactly the artist name “Rolling Stones,” or if “Rolling Stones” can apply to multiple named entities in themusic domain 202 a (e.g., a song, album, and artist). In such cases, if there are several different possible options for “Rolling Stones,” including artist, the named entity recognizer can select or bias the results towards “Rolling Stones” being the artist. - The
intent classifier 224 can receive input from the namedentity recognizer 222 and determine which intent, known to the single-domain NLU module 202, describes the most likely user intent. For example, the namedentity recognizer 222 may label the token “play” as a user intent. Theintent classifier 224 can determine which command or other response most likely captures the user intent. Illustratively, the music domain may determine that a programmatic “playSong( . . . )” command corresponds best to the user intent. The command may include arguments or parameters, such as one parameter for a song ID, or multiple parameters for the artist and song title. In some embodiments, theintent classifier 224 can use hints to aid processing. If an utterance can apply to several different intents (e.g., play a song, buy a song, etc.), the hints can help theintent classifier 224 to select or bias the results towards one that is consistent with the hint. - The
cross domain ranker 204 can receive output from the several domain-specific NLU modules cross domain ranker 204 can then combine each of the N-best lists into a single N-best list based on the scores. Returning to the example above, themusic domain 202 a may produce an N-best list of interpretations and corresponding scores, and the scores may be between 0.70 and 0.99 (where 0 is the minimum score and 1 is the maximum score). Thephone domain 202 c may also produce an N-best list of interpretations and corresponding scores. In the case of thephone domain 202 c, however, the scores may be between 0.01 and 0.10. Each of the other domain-specific NLU modules 202 can produce its own N-best list of results and corresponding scores. Thecross domain ranker 204 can combine the scores into single sorted N-best list of results and corresponding scores, such that the most likely interpretation is the first result in the N-best list or has the highest corresponding score. This scoring method is illustrative only, and not intended to be limiting. Other methods of scoring or ranking N-best lists, known to those of skill in the art, may also or alternatively be used. - The technique described above can be used when domain-specific NLU modules 202 are configured to recognize when an utterance is likely outside if its own domain. For example, the domain-specific NLU modules 202 may be exposed to text from other domains during the training processes. Such domain-specific NLU modules 202 can assign lower scores to their results—including the top result of the domain-specific N-best list—due to the reduced likelihood that the results are correct. The
cross domain ranker 204 can then simply combine the results into a single N-best list or choose the best result, because the domain-specific NLU modules 202 have already normalized the data based on their ability to recognize when utterances are likely outside the specific domain that they are configured to process. - In some embodiments, the domain-specific NLU modules 202 may not be configured to recognize when utterances are likely outside of their specific domains. In such cases, the
cross domain ranker 204 can normalize the scores from the domain-specific NLU modules 202 when generating the combined N-best list. Returning the previous example, themusic domain 202 a may return an N-best list of results with scores between 0.70 and 0.99. Thephone domain 202 c may also return an N-best list of results with scores between 0.70 and 0.99. Thecross domain ranker 204 can use hints to assign weights to the scores from the various domain-specific NLU modules 202 or to otherwise normalize the results from the various domain-specific NLU modules 202. As a result, thecross domain ranker 204 can produce a combined N-best list that is sorted according to normalized scores or which otherwise indicates which results are the most likely across results from all domain-specific NLU modules 202. - The
slot filler 206 can ensure that all data required to fully implement the user intent is present, and can also remove any unnecessary data. For example, the domain-specific NLU modules 202 may use more data in producing the N-best lists than is otherwise necessary for other components of the spokenlanguage processing system 100 to take action in response to the user utterance. Theslot filler 206 can therefore remove unnecessary information from the NLU results or otherwise modify the NLU results accordingly. In addition, theslot filler 206 can ensure that all information necessary implement the user intent is present. In the present example, the user may have said “Give Me Shelter” when issuing the voice command to play the song by the Rolling Stones. The slot filler may determine that there is no song by the Rolling Stones called “Give Me Shelter.” By searching a catalog or other data store for songs by the Rolling Stones with a similar name, theslot filler 206 may determine that correct interpretation is to play the song “Gimme Shelter” by the Rolling Stones. In some cases, theslot filler 206 may require more information from the user, and thedialog manager 116 may be employed to prompt the user for the missing information. As another example, the user utterance may be “Remind me to pay the bills on the day after tomorrow.” The user intent, as determined by theintent classifier 224 in conjunction with the namedentity recognizer 222, may correspond to execution of a command such as “createReminder( . . . ).” The command may require a parameter for the day on which the reminder is to be created. In addition, the command may require the day to be specified unambiguously. Theslot filler 206 can translate the phrase “day after tomorrow” into a specific date for use with the “createReminder( . . . )” command. For example, if the current data is November 9, then theslot filler 206 can calculate that the day to be used in the “createReminder( . . . )” command is November 11. - The
context ranker 208 can use hints to select the most likely result from the N-best list or to re-order the results in the N-best list. For example, the current hint may indicate that the user is likely issuing a command to play a particular song. If the top-scored or most likely result in N-best list is that the user wants to buy the song “Gimme Shelter” by the Rolling Stones, rather than play the song, thecontext ranker 208 can re-rank the results in the N-best list such that the most likely result is that the user wants to play the song. As described above, due to the multi-domain nature of theNLU module 114, users can issue voice commands related to any domain rather than only the previously used domain or a currently active domain. In some cases, therefore, thecontext ranker 208 may not necessarily select or elevate the result that most closely corresponds to the current hint. For example, the most likely preliminary result in the N-best list may be substantially more likely than the result that most closely corresponds to the current hint (e.g., the difference between scores associated with the two items exceeds a threshold, or the result that most closely corresponds to the current hint is ranked below a threshold). In such cases, thecontext ranker 208 may not increase the score or rank of the result that most closely corresponds to the hint. - With reference now to
FIG. 4 , asample process 400 for managing the generation and usage of hints in a spoken language processing system will be described. Advantageously, a spokenlanguage processing system 100 may use theprocess 400 to save history data regarding user utterances and other interactions, and to generate hints based on prior user utterances and interactions. The history and hints may be used in processing subsequent user utterances and interactions. - The
process 400 begins atblock 402. Theprocess 400 may begin automatically upon initiation of a speech recognition session. Theprocess 400 may be embodied in a set of executable program instructions stored on a computer-readable medium, such as one or more disk drives, of a computing system of the spokenlanguage processing system 100, such as anorchestrator 110 server. When theprocess 400 is initiated, the executable program instructions can be loaded into memory, such as RAM, and executed by one or more processors of the computing system. - At
block 404, the spokenlanguage processing system 100 can receive an utterance from aclient device 102. As described above, the user utterance may be a spoken command to play a recorded music file. - At
block 406, the spokenlanguage processing system 100 can access history and/or hints that may be useful in processing the utterance. As described above, the history can be a record of user intents and/or responses associated with the user within a particular time frame or during a particular voice command session. Hints may be generated during or after processing of user utterances and saved for use in processing subsequent utterances. In some cases, data other than user utterances may be received from theclient device 102. Such data may be saved in the history and/or used to generate hints to aid in further processing, and those hints may be accessed atblock 406 to process the current utterance. For example, a user of aclient device 102, such as a mobile phone, may raise the phone up to the user's head in a vertical position. The mobile phone may include proximity sensors to determine that the phone is currently positioned near the user's head. Data regarding this particular user action may be transmitted to the spokenlanguage processing system 100 and stored as history. Such a user action may indicate that he user is about to initiate voice dialing, and therefore the data may be useful in generating hints, as described below. - At
block 408, the spokenlanguage processing system 100 can process the utterance as described in detail below with respect toFIG. 5 . Atblock 410, history and hints based on the utterance, the processing of the utterance, and/or the response generated may be saved for use in processing further utterances. - Subsequent to, or in parallel with, saving history and/or hints, the
orchestrator 110 can determine atdecision block 412 whether a response is to be sent to theclient device 102. For example, if an utterance is received that includes a request for playback of a particular musical recording, such as Beethoven's 5th Symphony, thedialog manager 116 of the spokenlanguage processing system 100 may prompt the user to identify the particular recording artist (e.g., the Chicago Symphony Orchestra or the Berlin Symphony Orchestra). As described in detail herein, a TTS audio response prompting the user for the name of the recording artist may be generated and ready for transmission to theclient device 102. In addition, history may be saved regarding the prompt, and the history can be used during a subsequent execution of theprocess 400. If a prompt, executable command, or some other response is ready for transmission, theorchestrator 110 can transmit the response to theclient device 102 atblock 414. Otherwise, theprocess 400 can terminate. - Data received subsequent to the execution of the
process 400 may cause theprocess 400 to be executed again. Returning to an example described above, the user may provide a response to a prompt for a particular recording artist by submitting the utterance “the Chicago Symphony Orchestra.” Atblock 406, data regarding the history and/or hints saved during previous executions of theprocess 400 may be accessed, and atblock 408 the utterance may be processed. Based on the history (e.g., a request for playback of a Beethoven's 5th Symphony followed by a prompt for an artist's name), the hint used during processing of the subsequently received utterance may indicate that the utterance likely relates to an artist for the request to play a musical recording of Beethoven's 5th symphony. -
FIG. 5 illustrates asample process 500 for processing a user utterance, received from aclient device 102, with a multi-domain NLU engine. Advantageously, theprocess 500 may be used to process user utterances in multiple domains simultaneously or substantially simultaneously, and to select the best result or result most likely to be accurate from among the multiple results that are calculated. - The
process 500 begins atblock 502. Theprocess 500 may begin automatically upon receipt of an utterance from aclient device 102. Theprocess 500 may be embodied in a set of executable program instructions stored on a computer-readable medium, such as one or more disk drives, of a computing device associated with the spokenlanguage processing system 100. When theprocess 500 is initiated, the executable program instructions can be loaded into memory, such as RAM, and executed by one or more processors of the computing system. - At
block 504, an utterance may be received by the spokenlanguage processing server 100. For example, the user may have submitted a spoken command to initiate playback of a particular music recording. (e.g., “play Beethoven's 5th symphony by the Chicago Symphony Orchestra”). Atblock 506, the utterance may be transcribed into a likely transcription or N-best list of likely transcriptions by anASR module 112 of the spokenlanguage processing system 100. If any hints are available, they may be provided to theASR module 112 in order to allowASR module 112 to operate more efficiently and provide potentially more accurate results. - At blocks 508-1 to 508-n, the transcription or N-best list of likely transcriptions may be processed by any number of single-domain NLU modules in a multi-domain NLU engine. Each domain may process the transcription with respect to a particular subject matter, as described above. The domains may process the transcription in parallel or asynchronously so as to reduce the user-perceived time required to perform multi-domain NLU processing in comparison with single-domain NLU processing. In addition, a score may be assigned to each of the results.
- At
block 510, the most likely interpretation of user intent may be selected from the N-best list returned from the various single domain NLU modules. In some embodiments, response is selected based on the hint that was generated during or after processing of a previously received utterance or after receiving other information from theclient device 102. - At
block 512, the spokenlanguage processing system 100 may determine a response to the utterance. The response can vary depending upon which domain provided the most likely analysis of the user's intent. Returning to the previous example, if the music domain produced an analysis of the utterance with the highest score, then an executable command to play the requested music may be the most appropriate response. However, if not all of the required information was included in the utterance, a prompt may be generated in order to get the additional information. - At
decision block 514, the spoken language processing system can determine whether the response includes content to be presented aurally to the user, such a text that is to be read to the user. For example, if the spokenlanguage processing system 100 determined that the user is to be prompted for additional information regarding the user's utterance, then the process may proceed to block 516. Otherwise, if no TTS processing is required, theprocess 500 may proceed to block 520, where a command corresponding to the likely user intent is generated or some other action is performed. - At
block 516, anNLG module 118 may be employed to generate text for a spoken response such that the language of the response will sound natural to the user. For example, if the user has requested information about a particular product, information about the product (e.g., price, features, availability) may be obtained and provided to theNLG module 118. TheNLG module 118 may output a string of text that can be synthesized as speech by aTTS module 112. Atblock 518, theTTS module 112 can generate synthesized speech from a text input, such as text generated by anNLG module 118. - Depending on the embodiment, certain acts, events, or functions of any of the processes or algorithms described herein can be performed in a different sequence, can be added, merged, or left out altogether (e.g., not all described operations or events are necessary for the practice of the algorithm). Moreover, in certain embodiments, operations or events can be performed concurrently, e.g., through multi-threaded processing, interrupt processing, or multiple processors or processor cores or on other parallel architectures, rather than sequentially.
- The various illustrative logical blocks, modules, routines and algorithm steps described in connection with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. The described functionality can be implemented in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the disclosure.
- The steps of a method, process, routine, or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of a non-transitory computer-readable storage medium. An exemplary storage medium can be coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium can be integral to the processor. The processor and the storage medium can reside in an ASIC. The ASIC can reside in a user terminal. In the alternative, the processor and the storage medium can reside as discrete components in a user terminal.
- Conditional language used herein, such as, among others, “can,” “could,” “might,” “may,” “e.g.,” and the like, unless specifically stated otherwise, or otherwise understood within the context as used, is generally intended to convey that certain embodiments include, while other embodiments do not include, certain features, elements and/or steps. Thus, such conditional language is not generally intended to imply that features, elements and/or steps are in any way required for one or more embodiments or that one or more embodiments necessarily include logic for deciding, with or without author input or prompting, whether these features, elements and/or steps are included or are to be performed in any particular embodiment. The terms “comprising,” “including,” “having,” and the like are synonymous and are used inclusively, in an open-ended fashion, and do not exclude additional elements, features, acts, operations and so forth. Also, the term “or” is used in its inclusive sense (and not in its exclusive sense) so that when used, for example, to connect a list of elements, the term “or” means one, some, or all of the elements in the list.
- Conjunctive language such as the phrase “at least one of X, Y and Z,” unless specifically stated otherwise, is to be understood with the context as used in general to convey that an item, term, etc. may be either X, Y or Z, or a combination thereof. Thus, such conjunctive language is not generally intended to imply that certain embodiments require at least one of X, at least one of Y and at least one of Z to each be present.
- While the above detailed description has shown, described and pointed out novel features as applied to various embodiments, it can be understood that various omissions, substitutions and changes in the form and details of the devices or algorithms illustrated can be made without departing from the spirit of the disclosure. As can be recognized, certain embodiments of the inventions described herein can be embodied within a form that does not provide all of the features and benefits set forth herein, as some features can be used or practiced separately from others. The scope of certain inventions disclosed herein is indicated by the appended claims rather than by the foregoing description. All changes which come within the meaning and range of equivalency of the claims are to be embraced within their scope.
Claims (21)
1. (canceled)
2. A system comprising:
computer-readable memory storing executable instructions; and
one or more processors in communication with the computer-readable memory, wherein the one or more processors are programmed by the executable instructions to:
receive movement data representing a movement of a user device;
generate, using the movement data, hint data regarding a future user utterance;
receive, subsequent to generating the hint data, natural language data representing a user utterance;
determine, based at least partly on the hint data, to perform natural language understanding (“NLU”) processing on the natural language data;
perform the NLU processing on the natural language data to generate intent data; and
generate response data based at least partly on the intent data.
3. The system of claim 2 , wherein the one or more processors are further programmed by the executable instructions to determine, based at least partly on the hint data, that the natural language data is associated with an intent of a plurality of intents, and wherein the intent data represents the intent.
4. The system of claim 2 , wherein the one or more processors are further programmed by the executable instructions to determine, based at least partly on the hint data, that the natural language data is associated with an NLU domain of a plurality of NLU domains, and wherein the intent data represents an intent associated with the NLU domain.
5. The system of claim 4 , wherein the NLU domain is associated with intents related to at least one of: phone dialing, shopping, getting directions, playing music, or performing a search.
6. The system of claim 2 , wherein the movement of the user device comprises a vertical rise of the user device.
7. The system of claim 2 , wherein the one or more processors are further programmed by the executable instructions to:
generate second intent data based at least partly on the natural language data, wherein the intent data represents a first intent, and wherein the second intent data represents a second intent different from the first intent; and
rank the intent data and the second intent data based at least partly on the hint data.
8. The system of claim 2 , wherein to perform the NLU processing, the one or more processors are further programmed by the executable instructions to generate NLU result data using the natural language data and an NLU subsystem of a plurality of NLU subsystems,
wherein the NLU subsystem is associated with an NLU domain of a plurality of NLU domains, and
wherein the NLU result data represents a named entity of a plurality of named entities associated with an intent represented by the intent data.
9. The system of claim 2 , wherein the one or more processors are further programmed by the executable instructions to:
receive audio data representing the user utterance; and
generate the natural language data using the audio data and an automatic speech recognition (“ASR”) subsystem.
10. The system of claim 2 , wherein the executable instructions to generate the response data comprise executable instructions to:
generate response text data using a natural language generation subsystem; and
generate response audio data using the response text data and a text-to-speech subsystem.
11. The system of claim 2 , wherein one or more processors are further programmed by the executable instructions to send the response data to the user device, and wherein the user device is configured to present a response using the response data.
12. A computer-implemented method comprising:
under control of one or more computing devices configured with specific computer-executable instructions,
receiving movement data representing a movement of a user device;
generating, using the movement data, hint data regarding a future user utterance;
receiving, subsequent to generating the hint data, natural language data representing a user utterance;
determining, based at least partly on the hint data, to perform natural language understanding (“NLU”) processing on the natural language data;
performing the NLU processing on the natural language data to generate intent data; and
generating response data based at least partly on the intent data.
13. The computer-implemented method of claim 12 , further comprising determining, based at least partly on the hint data, that the natural language data is associated with an intent of a plurality of intents, wherein the intent data represents the intent.
14. The computer-implemented method of claim 12 , further comprising determining, based at least partly on the hint data, that the natural language data is associated with an NLU domain of a plurality of NLU domains, wherein the intent data represents and intent associated with the NLU domain.
15. The computer-implemented method of claim 12 , further comprising:
generating second intent data based at least partly on the natural language data, wherein the intent data represents a first intent, and wherein the second intent data represents a second intent different from the first intent; and
ranking the intent data and the second intent data based at least partly on the hint data.
16. The computer-implemented method of claim 16 , wherein ranking the intent data and the second intent data based at least partly on the hint data comprises adjusting a rank of at least one of the intent data or the second intent data.
17. The computer-implemented method of claim 12 , further comprising determining that the movement data represents a vertical rise of the user device.
18. The computer-implemented method of claim 12 , wherein performing the NLU processing comprises generating NLU result data using the natural language data and an NLU subsystem of a plurality of NLU subsystems,
wherein the NLU subsystem is associated with an NLU domain of a plurality of NLU domains, and
wherein the NLU result data represents a named entity of a plurality of named entities associated with an intent represented by the intent data.
19. The computer-implemented method of claim 12 , further comprising:
receiving audio data representing the user utterance; and
generating the natural language data using the audio data and an automatic speech recognition (“ASR”) subsystem.
20. The computer-implemented method of claim 12 , wherein generating the response data comprises:
generating response text data using a natural language generation subsystem; and
generating response audio data using the response text data and a text-to-speech subsystem.
21. The computer-implemented method of claim 12 , further comprising sending the response data to the user device, and wherein the user device is configured to present a response using the response data.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/454,716 US20220148590A1 (en) | 2012-12-19 | 2021-11-12 | Architecture for multi-domain natural language processing |
Applications Claiming Priority (7)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US13/720,909 US9070366B1 (en) | 2012-12-19 | 2012-12-19 | Architecture for multi-domain utterance processing |
US14/754,598 US9436678B2 (en) | 2012-12-19 | 2015-06-29 | Architecture for multi-domain natural language processing |
US15/256,176 US9754589B2 (en) | 2012-12-19 | 2016-09-02 | Architecture for multi-domain natural language processing |
US15/694,996 US9959869B2 (en) | 2012-12-19 | 2017-09-04 | Architecture for multi-domain natural language processing |
US15/966,400 US10283119B2 (en) | 2012-12-19 | 2018-04-30 | Architecture for multi-domain natural language processing |
US16/400,905 US11176936B2 (en) | 2012-12-19 | 2019-05-01 | Architecture for multi-domain natural language processing |
US17/454,716 US20220148590A1 (en) | 2012-12-19 | 2021-11-12 | Architecture for multi-domain natural language processing |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/400,905 Continuation US11176936B2 (en) | 2012-12-19 | 2019-05-01 | Architecture for multi-domain natural language processing |
Publications (1)
Publication Number | Publication Date |
---|---|
US20220148590A1 true US20220148590A1 (en) | 2022-05-12 |
Family
ID=53441819
Family Applications (7)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/720,909 Active 2033-08-26 US9070366B1 (en) | 2012-12-19 | 2012-12-19 | Architecture for multi-domain utterance processing |
US14/754,598 Active US9436678B2 (en) | 2012-12-19 | 2015-06-29 | Architecture for multi-domain natural language processing |
US15/256,176 Active US9754589B2 (en) | 2012-12-19 | 2016-09-02 | Architecture for multi-domain natural language processing |
US15/694,996 Active US9959869B2 (en) | 2012-12-19 | 2017-09-04 | Architecture for multi-domain natural language processing |
US15/966,400 Active US10283119B2 (en) | 2012-12-19 | 2018-04-30 | Architecture for multi-domain natural language processing |
US16/400,905 Active US11176936B2 (en) | 2012-12-19 | 2019-05-01 | Architecture for multi-domain natural language processing |
US17/454,716 Pending US20220148590A1 (en) | 2012-12-19 | 2021-11-12 | Architecture for multi-domain natural language processing |
Family Applications Before (6)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US13/720,909 Active 2033-08-26 US9070366B1 (en) | 2012-12-19 | 2012-12-19 | Architecture for multi-domain utterance processing |
US14/754,598 Active US9436678B2 (en) | 2012-12-19 | 2015-06-29 | Architecture for multi-domain natural language processing |
US15/256,176 Active US9754589B2 (en) | 2012-12-19 | 2016-09-02 | Architecture for multi-domain natural language processing |
US15/694,996 Active US9959869B2 (en) | 2012-12-19 | 2017-09-04 | Architecture for multi-domain natural language processing |
US15/966,400 Active US10283119B2 (en) | 2012-12-19 | 2018-04-30 | Architecture for multi-domain natural language processing |
US16/400,905 Active US11176936B2 (en) | 2012-12-19 | 2019-05-01 | Architecture for multi-domain natural language processing |
Country Status (1)
Country | Link |
---|---|
US (7) | US9070366B1 (en) |
Families Citing this family (243)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8645137B2 (en) | 2000-03-16 | 2014-02-04 | Apple Inc. | Fast, language-independent method for user authentication by voice |
US8677377B2 (en) | 2005-09-08 | 2014-03-18 | Apple Inc. | Method and apparatus for building an intelligent automated assistant |
US9318108B2 (en) | 2010-01-18 | 2016-04-19 | Apple Inc. | Intelligent automated assistant |
US8073681B2 (en) | 2006-10-16 | 2011-12-06 | Voicebox Technologies, Inc. | System and method for a cooperative conversational voice user interface |
US7818176B2 (en) | 2007-02-06 | 2010-10-19 | Voicebox Technologies, Inc. | System and method for selecting and presenting advertisements based on natural language processing of voice-based input |
US8977255B2 (en) | 2007-04-03 | 2015-03-10 | Apple Inc. | Method and system for operating a multi-function portable electronic device using voice-activation |
US8140335B2 (en) | 2007-12-11 | 2012-03-20 | Voicebox Technologies, Inc. | System and method for providing a natural language voice user interface in an integrated voice navigation services environment |
US10002189B2 (en) | 2007-12-20 | 2018-06-19 | Apple Inc. | Method and apparatus for searching using an active ontology |
US9330720B2 (en) | 2008-01-03 | 2016-05-03 | Apple Inc. | Methods and apparatus for altering audio output signals |
US8996376B2 (en) | 2008-04-05 | 2015-03-31 | Apple Inc. | Intelligent text-to-speech conversion |
US9305548B2 (en) | 2008-05-27 | 2016-04-05 | Voicebox Technologies Corporation | System and method for an integrated, multi-modal, multi-device natural language voice services environment |
US20100030549A1 (en) | 2008-07-31 | 2010-02-04 | Lee Michael M | Mobile device having human language translation capability with positional feedback |
US8676904B2 (en) | 2008-10-02 | 2014-03-18 | Apple Inc. | Electronic devices with voice command and contextual data processing capabilities |
US8326637B2 (en) | 2009-02-20 | 2012-12-04 | Voicebox Technologies, Inc. | System and method for processing multi-modal device interactions in a natural language voice services environment |
US10255566B2 (en) | 2011-06-03 | 2019-04-09 | Apple Inc. | Generating and processing task items that represent tasks to perform |
US10241752B2 (en) | 2011-09-30 | 2019-03-26 | Apple Inc. | Interface for a virtual digital assistant |
US10241644B2 (en) | 2011-06-03 | 2019-03-26 | Apple Inc. | Actionable reminder entries |
US9431006B2 (en) | 2009-07-02 | 2016-08-30 | Apple Inc. | Methods and apparatuses for automatic speech recognition |
US8943094B2 (en) | 2009-09-22 | 2015-01-27 | Next It Corporation | Apparatus, system, and method for natural language processing |
US10276170B2 (en) | 2010-01-18 | 2019-04-30 | Apple Inc. | Intelligent automated assistant |
US8682667B2 (en) | 2010-02-25 | 2014-03-25 | Apple Inc. | User profiling for selecting user specific voice input processing information |
US9262612B2 (en) | 2011-03-21 | 2016-02-16 | Apple Inc. | Device access using voice authentication |
US10057736B2 (en) | 2011-06-03 | 2018-08-21 | Apple Inc. | Active transport based notifications |
US8994660B2 (en) | 2011-08-29 | 2015-03-31 | Apple Inc. | Text correction processing |
US10134385B2 (en) | 2012-03-02 | 2018-11-20 | Apple Inc. | Systems and methods for name pronunciation |
US9280610B2 (en) | 2012-05-14 | 2016-03-08 | Apple Inc. | Crowd sourcing information to fulfill user requests |
US10417037B2 (en) | 2012-05-15 | 2019-09-17 | Apple Inc. | Systems and methods for integrating third party services with a digital assistant |
US9721563B2 (en) | 2012-06-08 | 2017-08-01 | Apple Inc. | Name recognition system |
US10026394B1 (en) * | 2012-08-31 | 2018-07-17 | Amazon Technologies, Inc. | Managing dialogs on a speech recognition platform |
US9547647B2 (en) | 2012-09-19 | 2017-01-17 | Apple Inc. | Voice-based media searching |
US9070366B1 (en) | 2012-12-19 | 2015-06-30 | Amazon Technologies, Inc. | Architecture for multi-domain utterance processing |
US9038142B2 (en) * | 2013-02-05 | 2015-05-19 | Google Inc. | Authorization flow initiation using short-term wireless communication |
BR112015018905B1 (en) | 2013-02-07 | 2022-02-22 | Apple Inc | Voice activation feature operation method, computer readable storage media and electronic device |
US10652394B2 (en) | 2013-03-14 | 2020-05-12 | Apple Inc. | System and method for processing voicemail |
US10748529B1 (en) | 2013-03-15 | 2020-08-18 | Apple Inc. | Voice activated device for use with a voice-based digital assistant |
US9953630B1 (en) * | 2013-05-31 | 2018-04-24 | Amazon Technologies, Inc. | Language recognition for device settings |
US9582608B2 (en) | 2013-06-07 | 2017-02-28 | Apple Inc. | Unified ranking with entropy-weighted information for phrase-based semantic auto-completion |
WO2014197336A1 (en) | 2013-06-07 | 2014-12-11 | Apple Inc. | System and method for detecting errors in interactions with a voice-based digital assistant |
WO2014197334A2 (en) | 2013-06-07 | 2014-12-11 | Apple Inc. | System and method for user-specified pronunciation of words for speech synthesis and recognition |
WO2014197335A1 (en) | 2013-06-08 | 2014-12-11 | Apple Inc. | Interpreting and acting upon commands that involve sharing information with remote devices |
US10176167B2 (en) * | 2013-06-09 | 2019-01-08 | Apple Inc. | System and method for inferring user intent from speech inputs |
EP3008641A1 (en) | 2013-06-09 | 2016-04-20 | Apple Inc. | Device, method, and graphical user interface for enabling conversation persistence across two or more instances of a digital assistant |
US10296160B2 (en) | 2013-12-06 | 2019-05-21 | Apple Inc. | Method for extracting salient dialog usage from live data |
US9741343B1 (en) * | 2013-12-19 | 2017-08-22 | Amazon Technologies, Inc. | Voice interaction application selection |
US9430463B2 (en) | 2014-05-30 | 2016-08-30 | Apple Inc. | Exemplar-based natural language processing |
US9715875B2 (en) | 2014-05-30 | 2017-07-25 | Apple Inc. | Reducing the need for manual start/end-pointing and trigger phrases |
US10170123B2 (en) | 2014-05-30 | 2019-01-01 | Apple Inc. | Intelligent assistant for home automation |
CN106471570B (en) | 2014-05-30 | 2019-10-01 | 苹果公司 | Order single language input method more |
US9633004B2 (en) | 2014-05-30 | 2017-04-25 | Apple Inc. | Better resolution when referencing to concepts |
US9338493B2 (en) | 2014-06-30 | 2016-05-10 | Apple Inc. | Intelligent automated assistant for TV user interactions |
US9818400B2 (en) | 2014-09-11 | 2017-11-14 | Apple Inc. | Method and apparatus for discovering trending terms in speech requests |
US9898459B2 (en) | 2014-09-16 | 2018-02-20 | Voicebox Technologies Corporation | Integration of domain information into state transitions of a finite state transducer for natural language processing |
EP3195145A4 (en) | 2014-09-16 | 2018-01-24 | VoiceBox Technologies Corporation | Voice commerce |
US9668121B2 (en) | 2014-09-30 | 2017-05-30 | Apple Inc. | Social reminders |
US10127911B2 (en) | 2014-09-30 | 2018-11-13 | Apple Inc. | Speaker identification and unsupervised speaker adaptation techniques |
US10074360B2 (en) | 2014-09-30 | 2018-09-11 | Apple Inc. | Providing an indication of the suitability of speech recognition |
US10817672B2 (en) * | 2014-10-01 | 2020-10-27 | Nuance Communications, Inc. | Natural language understanding (NLU) processing based on user-specified interests |
US9747896B2 (en) | 2014-10-15 | 2017-08-29 | Voicebox Technologies Corporation | System and method for providing follow-up responses to prior natural language inputs of a user |
US10431214B2 (en) * | 2014-11-26 | 2019-10-01 | Voicebox Technologies Corporation | System and method of determining a domain and/or an action related to a natural language input |
US10614799B2 (en) | 2014-11-26 | 2020-04-07 | Voicebox Technologies Corporation | System and method of providing intent predictions for an utterance prior to a system detection of an end of the utterance |
US9690776B2 (en) * | 2014-12-01 | 2017-06-27 | Microsoft Technology Licensing, Llc | Contextual language understanding for multi-turn language tasks |
US11094320B1 (en) * | 2014-12-22 | 2021-08-17 | Amazon Technologies, Inc. | Dialog visualization |
US10460720B2 (en) * | 2015-01-03 | 2019-10-29 | Microsoft Technology Licensing, Llc. | Generation of language understanding systems and methods |
US10152299B2 (en) | 2015-03-06 | 2018-12-11 | Apple Inc. | Reducing response latency of intelligent automated assistants |
US9721566B2 (en) | 2015-03-08 | 2017-08-01 | Apple Inc. | Competing devices responding to voice triggers |
US9886953B2 (en) | 2015-03-08 | 2018-02-06 | Apple Inc. | Virtual assistant activation |
US10567477B2 (en) | 2015-03-08 | 2020-02-18 | Apple Inc. | Virtual assistant continuity |
US10460227B2 (en) | 2015-05-15 | 2019-10-29 | Apple Inc. | Virtual assistant in a communication session |
US10083688B2 (en) | 2015-05-27 | 2018-09-25 | Apple Inc. | Device voice control for selecting a displayed affordance |
US9979748B2 (en) | 2015-05-27 | 2018-05-22 | Cisco Technology, Inc. | Domain classification and routing using lexical and semantic processing |
US10200824B2 (en) | 2015-05-27 | 2019-02-05 | Apple Inc. | Systems and methods for proactively identifying and surfacing relevant content on a touch-sensitive device |
US9578173B2 (en) | 2015-06-05 | 2017-02-21 | Apple Inc. | Virtual assistant aided communication with 3rd party service in a communication session |
US11025565B2 (en) | 2015-06-07 | 2021-06-01 | Apple Inc. | Personalized prediction of responses for instant messaging |
US20160378747A1 (en) | 2015-06-29 | 2016-12-29 | Apple Inc. | Virtual assistant for media playback |
US10631057B2 (en) | 2015-07-24 | 2020-04-21 | Nuance Communications, Inc. | System and method for natural language driven search and discovery in large data sources |
US10847175B2 (en) * | 2015-07-24 | 2020-11-24 | Nuance Communications, Inc. | System and method for natural language driven search and discovery in large data sources |
US10255921B2 (en) * | 2015-07-31 | 2019-04-09 | Google Llc | Managing dialog data providers |
US10747498B2 (en) | 2015-09-08 | 2020-08-18 | Apple Inc. | Zero latency digital assistant |
US10671428B2 (en) | 2015-09-08 | 2020-06-02 | Apple Inc. | Distributed personal assistant |
US10740384B2 (en) | 2015-09-08 | 2020-08-11 | Apple Inc. | Intelligent automated assistant for media search and playback |
US10331312B2 (en) | 2015-09-08 | 2019-06-25 | Apple Inc. | Intelligent automated assistant in a media environment |
US9697820B2 (en) | 2015-09-24 | 2017-07-04 | Apple Inc. | Unit-selection text-to-speech synthesis using concatenation-sensitive neural networks |
US10366158B2 (en) | 2015-09-29 | 2019-07-30 | Apple Inc. | Efficient word encoding for recurrent neural network language models |
US10236017B1 (en) * | 2015-09-29 | 2019-03-19 | Amazon Technologies, Inc. | Goal segmentation in speech dialogs |
US11010550B2 (en) | 2015-09-29 | 2021-05-18 | Apple Inc. | Unified language modeling framework for word prediction, auto-completion and auto-correction |
US11587559B2 (en) | 2015-09-30 | 2023-02-21 | Apple Inc. | Intelligent device identification |
US10691473B2 (en) | 2015-11-06 | 2020-06-23 | Apple Inc. | Intelligent automated assistant in a messaging environment |
US10956666B2 (en) | 2015-11-09 | 2021-03-23 | Apple Inc. | Unconventional virtual assistant interactions |
US10049668B2 (en) | 2015-12-02 | 2018-08-14 | Apple Inc. | Applying neural network language models to weighted finite state transducers for automatic speech recognition |
US10223066B2 (en) | 2015-12-23 | 2019-03-05 | Apple Inc. | Proactive assistance based on dialog communication between devices |
US10417346B2 (en) | 2016-01-23 | 2019-09-17 | Microsoft Technology Licensing, Llc | Tool for facilitating the development of new language understanding scenarios |
WO2017138777A1 (en) * | 2016-02-12 | 2017-08-17 | Samsung Electronics Co., Ltd. | Method and electronic device for performing voice based actions |
US10282417B2 (en) | 2016-02-19 | 2019-05-07 | International Business Machines Corporation | Conversational list management |
US10446143B2 (en) | 2016-03-14 | 2019-10-15 | Apple Inc. | Identification of voice inputs providing credentials |
US10304444B2 (en) * | 2016-03-23 | 2019-05-28 | Amazon Technologies, Inc. | Fine-grained natural language understanding |
US9934775B2 (en) | 2016-05-26 | 2018-04-03 | Apple Inc. | Unit-selection text-to-speech synthesis based on predicted concatenation parameters |
US9972304B2 (en) | 2016-06-03 | 2018-05-15 | Apple Inc. | Privacy preserving distributed evaluation framework for embedded personalized systems |
US11227589B2 (en) | 2016-06-06 | 2022-01-18 | Apple Inc. | Intelligent list reading |
US10249300B2 (en) | 2016-06-06 | 2019-04-02 | Apple Inc. | Intelligent list reading |
US10049663B2 (en) | 2016-06-08 | 2018-08-14 | Apple, Inc. | Intelligent automated assistant for media exploration |
DK179309B1 (en) | 2016-06-09 | 2018-04-23 | Apple Inc | Intelligent automated assistant in a home environment |
US10586535B2 (en) | 2016-06-10 | 2020-03-10 | Apple Inc. | Intelligent digital assistant in a multi-tasking environment |
US10192552B2 (en) | 2016-06-10 | 2019-01-29 | Apple Inc. | Digital assistant providing whispered speech |
US10509862B2 (en) | 2016-06-10 | 2019-12-17 | Apple Inc. | Dynamic phrase expansion of language input |
US10067938B2 (en) | 2016-06-10 | 2018-09-04 | Apple Inc. | Multilingual word prediction |
US10490187B2 (en) | 2016-06-10 | 2019-11-26 | Apple Inc. | Digital assistant providing automated status report |
DK201670540A1 (en) | 2016-06-11 | 2018-01-08 | Apple Inc | Application integration with a digital assistant |
DK179415B1 (en) | 2016-06-11 | 2018-06-14 | Apple Inc | Intelligent device arbitration and control |
DK179049B1 (en) | 2016-06-11 | 2017-09-18 | Apple Inc | Data driven natural language event detection and classification |
DK179343B1 (en) | 2016-06-11 | 2018-05-14 | Apple Inc | Intelligent task discovery |
US10319365B1 (en) * | 2016-06-27 | 2019-06-11 | Amazon Technologies, Inc. | Text-to-speech processing with emphasized output audio |
WO2018023106A1 (en) | 2016-07-29 | 2018-02-01 | Erik SWART | System and method of disambiguating natural language processing requests |
KR20180022021A (en) | 2016-08-23 | 2018-03-06 | 삼성전자주식회사 | Method and electronic device for recognizing voice |
US10360300B2 (en) | 2016-08-24 | 2019-07-23 | Microsoft Technology Licensing, Llc | Multi-turn cross-domain natural language understanding systems, building platforms, and methods |
US10474753B2 (en) | 2016-09-07 | 2019-11-12 | Apple Inc. | Language identification using recurrent neural networks |
US10043516B2 (en) * | 2016-09-23 | 2018-08-07 | Apple Inc. | Intelligent automated assistant |
US11281993B2 (en) | 2016-12-05 | 2022-03-22 | Apple Inc. | Model and ensemble compression for metric learning |
US10600418B2 (en) | 2016-12-07 | 2020-03-24 | Google Llc | Voice to text conversion based on third-party agent content |
US10593346B2 (en) | 2016-12-22 | 2020-03-17 | Apple Inc. | Rank-reduced token representation for automatic speech recognition |
US10068567B1 (en) * | 2016-12-29 | 2018-09-04 | Amdocs Development Limited | System, method, and computer program for automatic management of intent classification |
US10229680B1 (en) * | 2016-12-29 | 2019-03-12 | Amazon Technologies, Inc. | Contextual entity resolution |
US11204787B2 (en) | 2017-01-09 | 2021-12-21 | Apple Inc. | Application integration with a digital assistant |
US11010601B2 (en) | 2017-02-14 | 2021-05-18 | Microsoft Technology Licensing, Llc | Intelligent assistant device communicating non-verbal cues |
US11100384B2 (en) | 2017-02-14 | 2021-08-24 | Microsoft Technology Licensing, Llc | Intelligent device user interactions |
US10467510B2 (en) | 2017-02-14 | 2019-11-05 | Microsoft Technology Licensing, Llc | Intelligent assistant |
KR102388539B1 (en) * | 2017-04-30 | 2022-04-20 | 삼성전자주식회사 | Electronic apparatus for processing user utterance |
DK201770383A1 (en) | 2017-05-09 | 2018-12-14 | Apple Inc. | User interface for correcting recognition errors |
US10417266B2 (en) | 2017-05-09 | 2019-09-17 | Apple Inc. | Context-aware ranking of intelligent response suggestions |
US10726832B2 (en) | 2017-05-11 | 2020-07-28 | Apple Inc. | Maintaining privacy of personal information |
DK201770439A1 (en) | 2017-05-11 | 2018-12-13 | Apple Inc. | Offline personal assistant |
US10395654B2 (en) | 2017-05-11 | 2019-08-27 | Apple Inc. | Text normalization based on a data-driven learning network |
DK180048B1 (en) | 2017-05-11 | 2020-02-04 | Apple Inc. | MAINTAINING THE DATA PROTECTION OF PERSONAL INFORMATION |
DK179496B1 (en) | 2017-05-12 | 2019-01-15 | Apple Inc. | USER-SPECIFIC Acoustic Models |
US11301477B2 (en) | 2017-05-12 | 2022-04-12 | Apple Inc. | Feedback analysis of a digital assistant |
DK201770427A1 (en) | 2017-05-12 | 2018-12-20 | Apple Inc. | Low-latency intelligent automated assistant |
DK179745B1 (en) | 2017-05-12 | 2019-05-01 | Apple Inc. | SYNCHRONIZATION AND TASK DELEGATION OF A DIGITAL ASSISTANT |
DK201770431A1 (en) | 2017-05-15 | 2018-12-20 | Apple Inc. | Optimizing dialogue policy decisions for digital assistants using implicit feedback |
DK201770432A1 (en) | 2017-05-15 | 2018-12-21 | Apple Inc. | Hierarchical belief states for digital assistants |
US20180336275A1 (en) | 2017-05-16 | 2018-11-22 | Apple Inc. | Intelligent automated assistant for media exploration |
US10311144B2 (en) | 2017-05-16 | 2019-06-04 | Apple Inc. | Emoji word sense disambiguation |
US20180336892A1 (en) | 2017-05-16 | 2018-11-22 | Apple Inc. | Detecting a trigger of a digital assistant |
DK179560B1 (en) | 2017-05-16 | 2019-02-18 | Apple Inc. | Far-field extension for digital assistant services |
US10403278B2 (en) | 2017-05-16 | 2019-09-03 | Apple Inc. | Methods and systems for phonetic matching in digital assistant services |
US10657328B2 (en) | 2017-06-02 | 2020-05-19 | Apple Inc. | Multi-task recurrent neural network architecture for efficient morphology handling in neural language modeling |
CN109102802B (en) * | 2017-06-21 | 2023-10-17 | 三星电子株式会社 | System for processing user utterances |
KR102445382B1 (en) * | 2017-07-10 | 2022-09-20 | 삼성전자주식회사 | Voice processing method and system supporting the same |
US11494395B2 (en) | 2017-07-31 | 2022-11-08 | Splunk Inc. | Creating dashboards for viewing data in a data storage system based on natural language requests |
US20190034555A1 (en) * | 2017-07-31 | 2019-01-31 | Splunk Inc. | Translating a natural language request to a domain specific language request based on multiple interpretation algorithms |
US10901811B2 (en) | 2017-07-31 | 2021-01-26 | Splunk Inc. | Creating alerts associated with a data storage system based on natural language requests |
US10580406B2 (en) * | 2017-08-18 | 2020-03-03 | 2236008 Ontario Inc. | Unified N-best ASR results |
US10515625B1 (en) | 2017-08-31 | 2019-12-24 | Amazon Technologies, Inc. | Multi-modal natural language processing |
US10445429B2 (en) | 2017-09-21 | 2019-10-15 | Apple Inc. | Natural language understanding using vocabularies with compressed serialized tries |
US10755051B2 (en) | 2017-09-29 | 2020-08-25 | Apple Inc. | Rule-based natural language processing |
US10769186B2 (en) * | 2017-10-16 | 2020-09-08 | Nuance Communications, Inc. | System and method for contextual reasoning |
US11372862B2 (en) | 2017-10-16 | 2022-06-28 | Nuance Communications, Inc. | System and method for intelligent knowledge access |
US10636424B2 (en) | 2017-11-30 | 2020-04-28 | Apple Inc. | Multi-turn canned dialog |
JP2019106054A (en) * | 2017-12-13 | 2019-06-27 | 株式会社東芝 | Dialog system |
US10733982B2 (en) | 2018-01-08 | 2020-08-04 | Apple Inc. | Multi-directional dialog |
US10770094B2 (en) * | 2018-01-09 | 2020-09-08 | Intel IP Corporation | Routing audio streams based on semantically generated result sets |
US10733375B2 (en) | 2018-01-31 | 2020-08-04 | Apple Inc. | Knowledge-based framework for improving natural language understanding |
KR20190101630A (en) * | 2018-02-23 | 2019-09-02 | 삼성전자주식회사 | System for processing user utterance and controlling method thereof |
US10789959B2 (en) | 2018-03-02 | 2020-09-29 | Apple Inc. | Training speaker recognition models for digital assistants |
US11354521B2 (en) | 2018-03-07 | 2022-06-07 | Google Llc | Facilitating communications with automated assistants in multiple languages |
EP3723084A1 (en) | 2018-03-07 | 2020-10-14 | Google LLC | Facilitating end-to-end communications with automated assistants in multiple languages |
US10460734B2 (en) | 2018-03-08 | 2019-10-29 | Frontive, Inc. | Methods and systems for speech signal processing |
US10592604B2 (en) | 2018-03-12 | 2020-03-17 | Apple Inc. | Inverse text normalization for automatic speech recognition |
US20190295541A1 (en) * | 2018-03-23 | 2019-09-26 | Polycom, Inc. | Modifying spoken commands |
US11461779B1 (en) * | 2018-03-23 | 2022-10-04 | Amazon Technologies, Inc. | Multi-speechlet response |
US10818288B2 (en) | 2018-03-26 | 2020-10-27 | Apple Inc. | Natural assistant interaction |
US10909331B2 (en) | 2018-03-30 | 2021-02-02 | Apple Inc. | Implicit identification of translation payload with neural machine translation |
US11113473B2 (en) * | 2018-04-02 | 2021-09-07 | SoundHound Inc. | Interpreting expressions having potentially ambiguous meanings in different domains |
US11145294B2 (en) | 2018-05-07 | 2021-10-12 | Apple Inc. | Intelligent automated assistant for delivering content from user experiences |
US10928918B2 (en) | 2018-05-07 | 2021-02-23 | Apple Inc. | Raise to speak |
US11520989B1 (en) * | 2018-05-17 | 2022-12-06 | Workday, Inc. | Natural language processing with keywords |
US10984780B2 (en) | 2018-05-21 | 2021-04-20 | Apple Inc. | Global semantic word embeddings using bi-directional recurrent neural networks |
US11127395B1 (en) * | 2018-05-23 | 2021-09-21 | Amazon Technologies, Inc. | Device-specific skill processing |
KR20190134107A (en) * | 2018-05-24 | 2019-12-04 | 삼성전자주식회사 | Electronic device which is processing user's voice and method for providing voice recognition control thereof |
DK180639B1 (en) | 2018-06-01 | 2021-11-04 | Apple Inc | DISABILITY OF ATTENTION-ATTENTIVE VIRTUAL ASSISTANT |
US10892996B2 (en) | 2018-06-01 | 2021-01-12 | Apple Inc. | Variable latency device coordination |
DK179822B1 (en) | 2018-06-01 | 2019-07-12 | Apple Inc. | Voice interaction at a primary device to access call functionality of a companion device |
US11386266B2 (en) | 2018-06-01 | 2022-07-12 | Apple Inc. | Text correction |
DK201870355A1 (en) | 2018-06-01 | 2019-12-16 | Apple Inc. | Virtual assistant operation in multi-device environments |
US10504518B1 (en) | 2018-06-03 | 2019-12-10 | Apple Inc. | Accelerated task performance |
US10896675B1 (en) | 2018-06-29 | 2021-01-19 | X Development Llc | Multi-tiered command processing |
US11163952B2 (en) * | 2018-07-11 | 2021-11-02 | International Business Machines Corporation | Linked data seeded multi-lingual lexicon extraction |
US11217244B2 (en) * | 2018-08-07 | 2022-01-04 | Samsung Electronics Co., Ltd. | System for processing user voice utterance and method for operating same |
WO2020040753A1 (en) * | 2018-08-21 | 2020-02-27 | Google Llc | Automated assistant invocation of second interactive module using supplemental data provided by first interactive module |
US10224035B1 (en) * | 2018-09-03 | 2019-03-05 | Primo Llc | Voice search assistant |
WO2020060151A1 (en) * | 2018-09-19 | 2020-03-26 | Samsung Electronics Co., Ltd. | System and method for providing voice assistant service |
US11868728B1 (en) * | 2018-09-19 | 2024-01-09 | Amazon Technologies, Inc. | Multi-domain skills |
US11010561B2 (en) | 2018-09-27 | 2021-05-18 | Apple Inc. | Sentiment prediction from textual data |
US10839159B2 (en) | 2018-09-28 | 2020-11-17 | Apple Inc. | Named entity normalization in a spoken dialog system |
US11170166B2 (en) | 2018-09-28 | 2021-11-09 | Apple Inc. | Neural typographical error modeling via generative adversarial networks |
US11462215B2 (en) | 2018-09-28 | 2022-10-04 | Apple Inc. | Multi-modal inputs for voice commands |
US11232264B2 (en) * | 2018-10-19 | 2022-01-25 | Verint Americas Inc. | Natural language processing with non-ontological hierarchy models |
US11037048B2 (en) * | 2018-10-22 | 2021-06-15 | Moveworks, Inc. | Virtual conversation method or system |
US11475898B2 (en) | 2018-10-26 | 2022-10-18 | Apple Inc. | Low-latency multi-speaker speech recognition |
US11404058B2 (en) * | 2018-10-31 | 2022-08-02 | Walmart Apollo, Llc | System and method for handling multi-turn conversations and context management for voice enabled ecommerce transactions |
US11238850B2 (en) | 2018-10-31 | 2022-02-01 | Walmart Apollo, Llc | Systems and methods for e-commerce API orchestration using natural language interfaces |
KR20200054360A (en) * | 2018-11-05 | 2020-05-20 | 삼성전자주식회사 | Electronic apparatus and control method thereof |
KR20200052612A (en) * | 2018-11-07 | 2020-05-15 | 삼성전자주식회사 | Electronic apparatus for processing user utterance and controlling method thereof |
US11830485B2 (en) * | 2018-12-11 | 2023-11-28 | Amazon Technologies, Inc. | Multiple speech processing system with synthesized speech styles |
US11087749B2 (en) * | 2018-12-20 | 2021-08-10 | Spotify Ab | Systems and methods for improving fulfillment of media content related requests via utterance-based human-machine interfaces |
CN109739965B (en) * | 2018-12-29 | 2022-07-15 | 深圳前海微众银行股份有限公司 | Method, device and equipment for migrating cross-domain conversation strategy and readable storage medium |
US11638059B2 (en) | 2019-01-04 | 2023-04-25 | Apple Inc. | Content playback on multiple devices |
CN109949817B (en) * | 2019-02-19 | 2020-10-23 | 一汽-大众汽车有限公司 | Voice arbitration method and device based on dual-operating-system dual-voice recognition engine |
US11348573B2 (en) | 2019-03-18 | 2022-05-31 | Apple Inc. | Multimodality in digital assistant systems |
WO2020208972A1 (en) * | 2019-04-08 | 2020-10-15 | ソニー株式会社 | Response generation device and response generation method |
WO2020214988A1 (en) * | 2019-04-17 | 2020-10-22 | Tempus Labs | Collaborative artificial intelligence method and system |
US11681923B2 (en) | 2019-04-19 | 2023-06-20 | Samsung Electronics Co., Ltd. | Multi-model structures for classification and intent determination |
US10997968B2 (en) * | 2019-04-30 | 2021-05-04 | Microsofttechnology Licensing, Llc | Using dialog context to improve language understanding |
US11475884B2 (en) | 2019-05-06 | 2022-10-18 | Apple Inc. | Reducing digital assistant latency when a language is incorrectly determined |
US11307752B2 (en) | 2019-05-06 | 2022-04-19 | Apple Inc. | User configurable task triggers |
DK201970509A1 (en) | 2019-05-06 | 2021-01-15 | Apple Inc | Spoken notifications |
US11423908B2 (en) | 2019-05-06 | 2022-08-23 | Apple Inc. | Interpreting spoken requests |
US11140099B2 (en) | 2019-05-21 | 2021-10-05 | Apple Inc. | Providing message response suggestions |
DK201970511A1 (en) | 2019-05-31 | 2021-02-15 | Apple Inc | Voice identification in digital assistant systems |
US11496600B2 (en) | 2019-05-31 | 2022-11-08 | Apple Inc. | Remote execution of machine-learned models |
US11289073B2 (en) | 2019-05-31 | 2022-03-29 | Apple Inc. | Device text to speech |
DK180129B1 (en) | 2019-05-31 | 2020-06-02 | Apple Inc. | User activity shortcut suggestions |
US11360641B2 (en) | 2019-06-01 | 2022-06-14 | Apple Inc. | Increasing the relevance of new available information |
US11227599B2 (en) | 2019-06-01 | 2022-01-18 | Apple Inc. | Methods and user interfaces for voice-based control of electronic devices |
US11501753B2 (en) | 2019-06-26 | 2022-11-15 | Samsung Electronics Co., Ltd. | System and method for automating natural language understanding (NLU) in skill development |
WO2021033889A1 (en) * | 2019-08-20 | 2021-02-25 | Samsung Electronics Co., Ltd. | Electronic device and method for controlling the electronic device |
US11669684B2 (en) | 2019-09-05 | 2023-06-06 | Paro AI, LLC | Method and system of natural language processing in an enterprise environment |
US11488406B2 (en) | 2019-09-25 | 2022-11-01 | Apple Inc. | Text detection using global geometry estimators |
US11636438B1 (en) | 2019-10-18 | 2023-04-25 | Meta Platforms Technologies, Llc | Generating smart reminders by assistant systems |
US11567788B1 (en) | 2019-10-18 | 2023-01-31 | Meta Platforms, Inc. | Generating proactive reminders for assistant systems |
US11710476B2 (en) | 2020-04-27 | 2023-07-25 | Robert Bosch Gmbh | System and method for automatic testing of conversational assistance |
US11823082B2 (en) | 2020-05-06 | 2023-11-21 | Kore.Ai, Inc. | Methods for orchestrating an automated conversation in one or more networks and devices thereof |
US11061543B1 (en) | 2020-05-11 | 2021-07-13 | Apple Inc. | Providing relevant data items based on context |
US11038934B1 (en) | 2020-05-11 | 2021-06-15 | Apple Inc. | Digital assistant hardware abstraction |
US11490204B2 (en) | 2020-07-20 | 2022-11-01 | Apple Inc. | Multi-device audio adjustment coordination |
US11438683B2 (en) | 2020-07-21 | 2022-09-06 | Apple Inc. | User identification using headphones |
CN111933146B (en) * | 2020-10-13 | 2021-02-02 | 苏州思必驰信息科技有限公司 | Speech recognition system and method |
US11620990B2 (en) * | 2020-12-11 | 2023-04-04 | Google Llc | Adapting automated speech recognition parameters based on hotword properties |
US20220335223A1 (en) * | 2021-04-16 | 2022-10-20 | Accenture Global Solutions Limited | Automated generation of chatbot |
EP4089570A1 (en) * | 2021-05-14 | 2022-11-16 | Deutsche Telekom AG | Techniques to provide a customized response for users communicating with a virtual speech assistant |
US11908452B1 (en) * | 2021-05-20 | 2024-02-20 | Amazon Technologies, Inc. | Alternative input representations for speech inputs |
US11922938B1 (en) | 2021-11-22 | 2024-03-05 | Amazon Technologies, Inc. | Access to multiple virtual assistants |
US20230230584A1 (en) * | 2022-01-18 | 2023-07-20 | Samsung Electronics Co., Ltd. | System and method for simultaneously identifying intent and slots in voice assistant commands |
US11868344B1 (en) * | 2022-09-09 | 2024-01-09 | Tencent America LLC | System, method, and computer program for cross-lingual text-to-SQL semantic parsing with representation mixup |
US11822894B1 (en) * | 2022-12-30 | 2023-11-21 | Fmr Llc | Integrating common and context-specific natural language understanding processing in a virtual assistant application |
Citations (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7430510B1 (en) * | 2004-03-01 | 2008-09-30 | At&T Corp. | System and method of using modular spoken-dialog components |
US20100121636A1 (en) * | 2008-11-10 | 2010-05-13 | Google Inc. | Multisensory Speech Detection |
US20110022292A1 (en) * | 2009-07-27 | 2011-01-27 | Robert Bosch Gmbh | Method and system for improving speech recognition accuracy by use of geographic information |
US20110112921A1 (en) * | 2009-11-10 | 2011-05-12 | Voicebox Technologies, Inc. | System and method for providing a natural language content dedication service |
US20110238409A1 (en) * | 2010-03-26 | 2011-09-29 | Jean-Marie Henri Daniel Larcheveque | Semantic Clustering and Conversational Agents |
US20120254227A1 (en) * | 2011-03-31 | 2012-10-04 | Microsoft Corporation | Augmented Conversational Understanding Architecture |
US20130138424A1 (en) * | 2011-11-28 | 2013-05-30 | Microsoft Corporation | Context-Aware Interaction System Using a Semantic Model |
US20140095147A1 (en) * | 2012-10-01 | 2014-04-03 | Nuance Communications, Inc. | Situation Aware NLU/NLP |
US9959869B2 (en) * | 2012-12-19 | 2018-05-01 | Amazon Technologies, Inc. | Architecture for multi-domain natural language processing |
Family Cites Families (15)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP4438028B2 (en) * | 1998-07-27 | 2010-03-24 | キヤノン株式会社 | Information processing apparatus and method, and storage medium storing the program |
US8374875B2 (en) | 2000-01-31 | 2013-02-12 | Intel Corporation | Providing programming information in response to spoken requests |
US6829603B1 (en) * | 2000-02-02 | 2004-12-07 | International Business Machines Corp. | System, method and program product for interactive natural dialog |
US7092928B1 (en) * | 2000-07-31 | 2006-08-15 | Quantum Leap Research, Inc. | Intelligent portal engine |
US6754626B2 (en) * | 2001-03-01 | 2004-06-22 | International Business Machines Corporation | Creating a hierarchical tree of language models for a dialog system based on prompt and dialog context |
US8566102B1 (en) * | 2002-03-28 | 2013-10-22 | At&T Intellectual Property Ii, L.P. | System and method of automating a spoken dialogue service |
US20040162724A1 (en) * | 2003-02-11 | 2004-08-19 | Jeffrey Hill | Management of conversations |
US7421393B1 (en) * | 2004-03-01 | 2008-09-02 | At&T Corp. | System for developing a dialog manager using modular spoken-dialog components |
KR100612839B1 (en) * | 2004-02-18 | 2006-08-18 | 삼성전자주식회사 | Method and apparatus for domain-based dialog speech recognition |
US7640160B2 (en) * | 2005-08-05 | 2009-12-29 | Voicebox Technologies, Inc. | Systems and methods for responding to natural language speech utterance |
US7949529B2 (en) * | 2005-08-29 | 2011-05-24 | Voicebox Technologies, Inc. | Mobile systems and methods of supporting natural language human-machine interactions |
US7925507B2 (en) * | 2006-07-07 | 2011-04-12 | Robert Bosch Corporation | Method and apparatus for recognizing large list of proper names in spoken dialog systems |
US9298287B2 (en) | 2011-03-31 | 2016-03-29 | Microsoft Technology Licensing, Llc | Combined activation for natural user interface systems |
US9105268B2 (en) | 2012-09-19 | 2015-08-11 | 24/7 Customer, Inc. | Method and apparatus for predicting intent in IVR using natural language queries |
US10282419B2 (en) * | 2012-12-12 | 2019-05-07 | Nuance Communications, Inc. | Multi-domain natural language processing architecture |
-
2012
- 2012-12-19 US US13/720,909 patent/US9070366B1/en active Active
-
2015
- 2015-06-29 US US14/754,598 patent/US9436678B2/en active Active
-
2016
- 2016-09-02 US US15/256,176 patent/US9754589B2/en active Active
-
2017
- 2017-09-04 US US15/694,996 patent/US9959869B2/en active Active
-
2018
- 2018-04-30 US US15/966,400 patent/US10283119B2/en active Active
-
2019
- 2019-05-01 US US16/400,905 patent/US11176936B2/en active Active
-
2021
- 2021-11-12 US US17/454,716 patent/US20220148590A1/en active Pending
Patent Citations (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7430510B1 (en) * | 2004-03-01 | 2008-09-30 | At&T Corp. | System and method of using modular spoken-dialog components |
US20100121636A1 (en) * | 2008-11-10 | 2010-05-13 | Google Inc. | Multisensory Speech Detection |
US20110022292A1 (en) * | 2009-07-27 | 2011-01-27 | Robert Bosch Gmbh | Method and system for improving speech recognition accuracy by use of geographic information |
US20110112921A1 (en) * | 2009-11-10 | 2011-05-12 | Voicebox Technologies, Inc. | System and method for providing a natural language content dedication service |
US20110238409A1 (en) * | 2010-03-26 | 2011-09-29 | Jean-Marie Henri Daniel Larcheveque | Semantic Clustering and Conversational Agents |
US20120254227A1 (en) * | 2011-03-31 | 2012-10-04 | Microsoft Corporation | Augmented Conversational Understanding Architecture |
US20130138424A1 (en) * | 2011-11-28 | 2013-05-30 | Microsoft Corporation | Context-Aware Interaction System Using a Semantic Model |
US20140095147A1 (en) * | 2012-10-01 | 2014-04-03 | Nuance Communications, Inc. | Situation Aware NLU/NLP |
US9959869B2 (en) * | 2012-12-19 | 2018-05-01 | Amazon Technologies, Inc. | Architecture for multi-domain natural language processing |
US10283119B2 (en) * | 2012-12-19 | 2019-05-07 | Amazon Technologies, Inc. | Architecture for multi-domain natural language processing |
Also Published As
Publication number | Publication date |
---|---|
US20170116985A1 (en) | 2017-04-27 |
US11176936B2 (en) | 2021-11-16 |
US9070366B1 (en) | 2015-06-30 |
US20190325873A1 (en) | 2019-10-24 |
US9754589B2 (en) | 2017-09-05 |
US10283119B2 (en) | 2019-05-07 |
US20180315425A1 (en) | 2018-11-01 |
US20180012597A1 (en) | 2018-01-11 |
US20150302002A1 (en) | 2015-10-22 |
US9959869B2 (en) | 2018-05-01 |
US9436678B2 (en) | 2016-09-06 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11176936B2 (en) | Architecture for multi-domain natural language processing | |
US11398236B2 (en) | Intent-specific automatic speech recognition result generation | |
US9454957B1 (en) | Named entity resolution in spoken language processing | |
US9240187B2 (en) | Identification of utterance subjects | |
US9502032B2 (en) | Dynamically biasing language models | |
US10964312B2 (en) | Generation of predictive natural language processing models | |
US20180190288A1 (en) | System and method of performing automatic speech recognition using local private data | |
JP6588637B2 (en) | Learning personalized entity pronunciation | |
JP7200405B2 (en) | Context Bias for Speech Recognition | |
US8949266B2 (en) | Multiple web-based content category searching in mobile search application | |
US9922650B1 (en) | Intent-specific automatic speech recognition result generation | |
US20110060587A1 (en) | Command and control utilizing ancillary information in a mobile voice-to-speech application | |
US20110054894A1 (en) | Speech recognition through the collection of contact information in mobile dictation application | |
US20110054898A1 (en) | Multiple web-based content search user interface in mobile search application | |
US11501764B2 (en) | Apparatus for media entity pronunciation using deep learning | |
US10102845B1 (en) | Interpreting nonstandard terms in language processing using text-based communications | |
US10140981B1 (en) | Dynamic arc weights in speech recognition models | |
US20220020365A1 (en) | Automated assistant with audio presentation interaction |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE AFTER FINAL ACTION FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: ADVISORY ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |