KR102312218B1 - contextual hotwords - Google Patents

Abstract

Methods, systems and apparatus comprising computer programs encoded on a computer storage medium for contextual hotwords are disclosed. In an aspect, during a boot process of a computing device, a method includes determining, by the computing device, a context associated with the computing device. The operations further include determining the hotword based on the context associated with the computing device. The operations also include, after determining the hotword, receiving audio data corresponding to the utterance. The operations further include determining that the audio data includes a hotword. The operations further include, in response to determining that the audio data includes the hotword, performing an operation associated with the hotword.

Description

contextual hotwords

FIELD OF THE INVENTION The present invention relates generally to automated speech processing.

The reality of a speech-enabled home or other environment - that is, when a user utters a question or command aloud or a computer-based system composes and responds to the question and/or causes the command to be performed Things - this is coming to us. A voice recognition capable environment (eg, home, work, school, etc.) may be implemented using a network of connected microphone devices distributed across several rooms or areas of the environment. This network of microphones gives users the right to verbally query the system from essentially anywhere in the environment without having to have a computer or other devices in front of or near them. For example, while cooking in the kitchen, the user may ask the system "how many milliliters are in three cups?" and in response, from the system in the form of synthesized speech output, for example. A response may be received. Alternatively, the user may ask, "When can I reach the nearest gas station?" Or, as you prepare to leave the house, you can ask a system question like “Should I wear a coat today?”

The user may also make queries and/or commands to the system related to the user's personal information. For example, a user could ask the system "When are you meeting John?" or you could instruct the system to "remind me to call John when I get home."

In the case of speech recognition capable systems, the way the user interacts with the system is primarily designed to be through voice input, although not exclusively. Consequently, a system that potentially picks up all utterances made in the surrounding environment, including those not directed to the system, is capable of identifying when any given utterance is directed in the opposite direction, e.g., towards an individual present in the surrounding environment. There must be several ways. One way to achieve this is to use reserved hotwords with predetermined words spoken to get the system's attention according to agreement among users in the environment. In an exemplary environment, the hot word used to draw the attention of the system is the word "OK computer." As a result, each time the word "ok computer" is pronounced, it is picked up by the microphone and delivered to the system, performing voice modeling techniques to determine if a hotword has been spoken, and if so, awaits the next command or query. Thus, utterances directed to the system have the general form [hotword] [query], in this example, "hotword" is "ok computer" and "query" is system, either alone or in combination with a server over a network. may be any query, command, declaration, or other request that may be voice recognized, parsed and performed by

When a user provides several hotword-based commands to a voice recognition capable system, such as a cell phone, the user's interaction with the phone can become awkward. The user can say "Okay computer, play my homework list." The phone may start playing the first song in the playlist. The user may want to skip to the next song and say, "Okay computer, next." To proceed to another song, the user can say "Okay computer, next" again. To reduce the need to repeat the hotword over and over, the phone may be configured to recognize "next" as both a hotword and a query, or in this case a command. With this feature, the user's interaction with the phone and the music application becomes more natural because the user can only say "next" to advance to the next song instead of "ok computer, next".

To achieve this, the speech recognition capable system determines the current context of the system and identifies the associated hotwords. Contexts may be based on applications running on the system, system location, system movement, or other similar circumstances. The system may use the system's context to identify additional hotwords. For example, when music is playing, the system may identify the hotwords "next", "stop" and "back" to control the music. The system may use the system's context to identify additional hotwords. For example, when music is playing, the system may identify the hotwords "next", "stop" and "back" to control the music. The system may require a hotword model for each identified hotword. The system may use the hotword models to recognize new hotwords by processing an audio characteristic of audio data corresponding to the user's voice and applying the hotword models to the audio characteristic. The system recognizes the voice hotword and performs the corresponding task. Since the system is playing music, the user says "stop", and when "stop" becomes an active hotword, then the system can stop playing the music.

According to an innovative aspect of the subject matter disclosed in this application, a method for anti-rollback security comprises:

determining, by a computing device, a context associated with the computing device; determining a hotword based on the context associated with the computing device; after determining the hot word, receiving audio data corresponding to an utterance; determining that the audio data includes the hotword; and in response to determining that the audio data includes the hotword, performing an operation associated with the hotword.

These and other implementations may each optionally include one or more of the following features. The determining that the audio data includes the hotword includes determining that the audio data includes the hotword without performing speech recognition on the audio data. Determining that the audio data includes the hotword may include: extracting audio features of the audio data corresponding to the utterance; generating a hotword confidence score by processing the audio features; determining whether the hotword confidence score satisfies a hotword confidence threshold; and based on determining whether the hotword confidence score satisfies a hotword confidence threshold, determining that the audio data corresponding to the utterance includes the hotword. The method further comprises, after determining the hotword, receiving a hotword model corresponding to the hotword, wherein determining that the audio data includes the hotword comprises: using the hotword model , determining that the audio data includes the hotword.

The method includes identifying, by the computing device, an application running on the computing device. The context is based on the application running on the computing device. The method includes determining, by the computing device, that the context is no longer associated with the computing device; and determining that subsequently received audio data comprising the hotword does not trigger an action. The method further includes providing, for output, data identifying the hotword. The method includes identifying, by the computing device, movement of the computing device. The context is based on movement of the computing device. The method further includes identifying, by the computing device, a location of the computing device. The context is based on the location of the computing device. Performing the operation associated with the hotword includes: performing speech recognition on a portion of the audio data that does not include the hotword. The operation is based on the transcription of the portion of the audio that does not contain the hotword. The audio data includes only the hotword. The initial portion of the audio data includes the hotword.

Other embodiments of this aspect include computer programs recorded on corresponding systems, devices and computer storage device, each configured to perform the operations of the methods.

The subject matter described in this application may have one or more of the following advantages. The computing device is capable of recognizing and operating in response to a command that is shorter than conventional systems that typically require the user to say a hotword that follows the command. Accordingly, less processing of recognized commands is required and, therefore, less resource consumption (including memory for storage of input terms and power consumption) is required for the computing device to respond, and thus the computing device is faster and respond effectively. The computing device may recognize and operate in response to different commands without performing voice recognition for the commands. It can reduce the computing resources and battery power required to recognize the query and command, because the computing device can recognize the query and command as one term, processing two or more different terms in separate operational steps. because you don't have to.

The details of one or more embodiments of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features, aspects and advantages of the subject matter will become apparent from the detailed description, drawings and claims.

1 depicts an exemplary system for identifying and processing contextual hotwords.
2 depicts an exemplary process for identifying and processing contextual hotwords.
3 shows an example of a computing device and a mobile computing device.
In the drawings, like reference numbers indicate corresponding parts throughout.

1 depicts an exemplary system 100 for identifying and processing context hotwords. Briefly, as described in greater detail below, computing device 104 begins playing music in response to utterance 106 , “Ok computer, play music.” The computing device 104 begins playing the music, and the computing device 104 may recognize the additional hotword “next” as a command to advance to the next song.

More specifically, starting at step A of FIG. 1 , the user 102 makes an utterance 106 in the vicinity of the computing device 104 , “Okay computer, play music.” A microphone of the computing device 104 receives the utterance 106 and processes audio data corresponding to the utterance. Initial processing of the audio data may include filtering the audio data and converting the audio data from an analog signal to a digital signal.

When the computing device 104 processes the audio data, the computing device may store the audio data in a buffer for further processing. With the audio data in the buffer, computing device 104 can identify whether the audio data includes any spoken words. One way to identify words spoken by the computing device 104 is to use the hotword detector 108 . The hotword detector 108 is configured to identify a hotword included in the audio data.

In some implementations, the hotword detector 108 may be configured to identify hotwords in the initial portion of the utterance 106 . In this example, if hotword detector 108 detects acoustic characteristics of audio data that are characteristic of active hotword 112 , voice 106 "Okay computer, play music" hotword 110 "Okay" The hotword detector 108 can detect that it contains "ok computer." The acoustic properties may be Mel Frequency Septral Coefficients (MFCCs) representing the short term power spectra of the utterance or may be Mel scale filter bank energies for the utterance 106 . For example, based on generating MFCCs from audio data, and classifying whether MFCCs contain MFCCs similar to MFCCs that are characteristic of the hot word "ok computer" as stored in hotword models 114, The hotword detector 108 may detect that the utterance 106 “Okay computer, play music” includes the hotword 110 “Okay computer”. As another example, generating mel-scale filter bank energies from the audio data, and a mel-scale filter bank that is characteristic of a hotword “ok computer” in which mel-scale filter bank energies are stored in hotword models 114 . Based on the inclusion of mel-scale filter bank energies similar to the energies, the hotword detector 108 determines that the utterance 106 “Ok computer, play music” includes the hotword 110 “OK computer”. can be detected.

The hotword detector 108 determines that the audio data corresponding to the utterance 106 includes the hotword 110 , and the computing device 104 performs speech recognition or speech recognition for the audio data corresponding to the utterance 106 . semantic interpretation can be performed. The speech recognizer 116 may perform speech recognition on the portion of the audio data that follows the hot word 110 . In this example, speech recognizer 116 may identify words 118 “play music”.

In some implementations, the voice recognizer 116 may not be located on the computing device 104 . Instead, the computing device 104 provides audio data corresponding to the portion of the utterance 106 after the hotword 110 to a server, eg, server 120 . Server 120 performs speech recognition and returns a record of audio data to computing device 104 . Computing device 104 identifies words in utterance 106 , and performs computing device semantic interpretation and identifies any spoken commands. Computing device 104 identifies the instruction and executes the instruction. In this example and step B, computing device 104 plays music 122 upon identifying “play music” command 118 .

In music playback 122 , computing device 104 is running a music application in the foreground or background. The computing device 104 may include a context identifier 124 and an active hotword selector 126 . The context identifier 124 may be configured to identify a current context of the computing device 104 . Active hotword selector 126 may use the current context of computing device 104 to select active hot words. In this example, the context of the device may relate to playing music 122 and running a music application. Developers of music applications want what users can say to interact with the music application and the respective actions for each hotword, so the active hotword selector 126 selects the music application to identify any hotwords. You can check the code of A music application may identify hotwords such as “play,” “next,” “stop,” and “back.” Based on the context of the actively playing music, the active hotword selector 126 may select the hotwords of “next”, “stop” and “back” and store them in active hotwords 112 .

In some implementations, the context identifier 124 can use the location of the computing device 104 to determine the context. For example, the context identifier 124 may determine that the location corresponds to the home of the user 102 of the computing device 104 . The active hotword selector 126 indicates that the device is in the user's 102 home to identify hotwords such as "warmer" and "cooler" so that the user can control the temperature of the home. situation can be used. Similar to the example above, active hotword selector 126 may store the hotwords “warmer” and “cooler” in active hotwords 112 while user 102 is at home.

In some implementations, the context identifier 124 can use the motion of the computing device 104 to determine the context. For example, the context identifier 124 may determine that the movement of the computing device 104 corresponds to the speed and motion of a typical vehicle. The context identifier 124 may also compare the speed and movement of the computing device 104 to determine whether the computing device 104 is moving along a road to increase confidence that the computing device 104 is in a vehicle. have. In this example, active hotword selector 126 may use the context of the computing device to identify hotword “directions” so that the user can request directions to a particular location. The active hotword selector 126 may store the hotword “direction” in the active hotwords 112 while the computing device 104 is in the vehicle.

In some implementations, the context identifier 124 may use the day of the week or the time or both to determine the context. For example, the context identifier 124 may determine that the context corresponds to an evening time, such as between 9:00 PM and midnight. In this example, the active hotword selector 126 may use the evening time context to identify “a hotword set alarm that causes the user 102 to set an alarm.” Active hotword selector 126 ) may store the hot word “alarm set” in active hotwords 112 for a time period from 9:00 pm to midnight.

In some implementations, the context identifier 124 can use the user's 102 past actions and patterns to identify hotwords. The context identifier 124 may identify actions the user performs, perhaps at different times of the day, and may determine a context based on those actions. Active hotword selector 126 may identify relevant hotwords and store these hotwords in active hotwords 112 for a period of time. For example, the context identifier 124 may determine that the user 102 is reading news on the computing device 104 between 8:00 AM and 10:00 AM. The active hotword selector 126 may select “news” as the active hotword 112 . Using “news” as a hot word, user 102 could say “news” to open the news application from 8 am to 10 am. A news application may have its own corresponding hotwords. The active hotword selector 126 selects hotwords such as “sports,” “local,” and “national” as active hotwords when a news application is open on the computing device 104 . It can be identified as (112).

In some implementations, the context identifier 124 can identify past actions that are not necessarily time dependent. For example, user 102 may habitually check the weather on computing device 102 . The context identifier 124 may determine that the context of the computing device almost always corresponds to the time the user 102 checked the weather. In this case, the active hotword selector 126 may identify the hotword “weather” and store the hotword in the active hotwords 112 . When using “weather” as the active hotword, user 102 just opens the weather application and says “weather” to check the weather.

In some implementations, the context identifier 124 can determine the context based on the state of the computing device 104 . For example, the state of computing device 104 may be “locked”. In this case, the active hotword selector 126 may identify the hotword “unlock” and store the hotword in the active hotword 112 when the device is locked. Using "unlock" as an active hotword, the user can unlock the phone by saying "unlock". To enhance security, computing device 104 may verify that the speaker is the user 102 using speaker identification techniques. In this case, the corresponding hotword model will be trained using the user's 102 voice. For example, computing device 104 may “unlock” user 102 multiple times so that computing device 104 or server 120 can build a hotword model specific to user 102 with voice samples. may be urged to do so.

Active hotwords 112 include the new hotword, and computing device 104 uses hotword models 114 to determine whether computing device 104 has a hotword for the newly added hotword. ) can be checked. For example, when active hotword selector 126 stores “next” in active hotwords 112 , computing device 104 determines whether hotword models 114 include a hotword model for “next”. ) is determined. If a hotword model for “next” is in the hotword models 114 , then the hotword detector 108 can begin to detect the hotword “next” and the computing device 104 performs stages C, D and E can be skipped. If the hotword model for “next” is not in the hotword models 114 , in step C, the computing device 104 sends a request 128 for the hotword model for “next” to the server 120 . do.

In some implementations, computing device 104 can require a hotword model for a hotword word that already has a corresponding hotword model in hotword model 114 . The computing device 104 may request the hotword model periodically, eg, once a week, to ensure that the locally stored hotword model is as accurate as possible. Computing device 104 may also request the hotword model in response to feedback from user 102 . In some examples, user 102 may say a hotword such as “next,” and the computer cannot advance the song. The hot word detector 108 may process the audio data corresponding to "next", but cannot identify the hot word because the hot word confidence score does not satisfy the threshold.

User 102 may advance a song by selecting a next song button on computing device 104 . When computing device 104 detects such a series of actions, computing device 104 may request an updated hotword model for “next”. Alternatively, the computing device 104 may update the hotword model for “next” using audio data that meets the threshold but does not generate a hotword confidence score that exceeds a lower hotword confidence threshold. have. In some implementations, computing device 104 can calculate a noise level for audio data that did not generate a hotword confidence score that satisfies a threshold. If the noise level is greater than the noise threshold, the computing device 104 may have too much background noise and may not update the hotword model with the corresponding audio data.

Server 120 receives a request 128 for a hotword model and identifies a corresponding hotword model 130 in stage D. Server 120 may correspond to one or more servers accessible via network 132 , such as the Internet. Data that can be accessed by one server can be accessed by other servers. In addition to identifying and providing hotword models, server 102 is configured to receive audio data and generate hotword models 130 based on the received audio data.

To generate hotword models 130 , server 120 receives and collects voice data 134 . The server 120 may receive voice data 134 from servers that perform voice recognition. Servers may perform speech recognition and provide audio data to server 120 to generate hotword models. With the collected speech data 134 , the speech recognizer 136 identifies words in the collected speech data.

The voice recognizer 136 provides recorded data and voice data to the voice data tokenizer 138 . The voice data tokenizer 138 divides the audio data into parts corresponding to different words. For example, if the collected voice data 134 corresponds to the word “play next that song”, then the voice recognizer generates a translation of “play next that song” and the voice data tokenizer 138 uses the audio data tokenize into 4 sections. There is one section for “play”, another section for “that”, another section for “next”, and another section for “song”.

Speech recognizer 136 and speech data tokenizer 138 may tokenize many speech samples and provide tokenized speech samples to hotword model generator 140 . The hotword model generator 140 processes multiple samples of the same word to generate a hotword model for the word. For example, the hotword model generator 140 may receive a number of speech samples corresponding to the word “next”. The hotword model generator 140 extracts audio characteristics of a voice sample to generate a hotword model for "next". The hotword model generator 140 stores the hotword model for "next" in the hotword models 130 .

Hotword models 130 are ready to be provided by the server to computer devices to recognize these specific hotwords. In some implementations, server 120 may not have the hotword required in hotword models 130 . In this case, the server 120 may analyze the collected voice data 134 using the voice recognizer 136 and the voice data tokenizer 138 to identify audio samples corresponding to the requested word. Alternatively, server 120 may request computing device 104 to collect samples. Computing device 104 may request the user to repeat a word multiple times. Computing device 104 may provide audio data to server 120 for processing to generate a hotword model. In some implementations, the requested hotword may be more than one word. In this example, hotword model generator 140 may combine hotword models 130 to generate a hotword model for a multi-word hotword.

In some implementations, hotword model generator 140 can generate hotword models that include noise specific to a specific context. The hotword model generator 140 may not select all voice tokens corresponding to the target hotword. Instead, the hotword model generator 140 selects speech tokens containing background noise that are likely to be present in the corresponding context. For example, the hotword model generator 140 may generate a hotword model for "next" with speech tokens that include "next" and have background music. If server 120 receives a request for a hotword model "next" and the request indicates that the context is music playback, then server 120 may provide a hotword model of "next" configured for background music. can If server 120 receives a request for a hotword model "next" and the request indicates that the context is photo appreciation, then server 120 may provide a hotword model "next" configured to be free from background noise. can

In step E, the server 120 provides a response 142 containing the requested hotword model. In the example shown in FIG. 1 , server 120 provides a hotword model for “next” to computing device 104 . Computing device 104 stores the hotword model in hotword word models 114 and 114 and updates the indicator in active hotwords 112 to have a corresponding hotword model stored in computing device 104 . can be displayed.

In some implementations, active hotword selector 126 can remove hot words from active hotwords 112 . When the context identifier 124 indicates that the context has changed, the active hotword selector 126 may update the active hotwords 112 based on the new context. Continuing with some of the above examples, active hotword selector 126 may remove the hotword “news” from active hotwords 112 after 10 am. Similarly, the active hotword selector 126 may remove the hotword “set alarm” after midnight and add the hotword “set alarm” back to the active hotwords 112 after 9 pm. . In some implementations, when the active hotword selector 126 removes the hotword active hotwords 112 , the corresponding hotword model remains in the hotword models 114 .

In some implementations, active hotword selector 126 can remove hotwords from active hotwords 112 even when the same application is running on computing device 104 . When the computing device 104 executes the music application, the active hotword selector 126 selects the hotwords “play,” “next,” “stop,” and “back.” Identify and load hotword models corresponding to hotword models 114 . In this example, the hotwords do not need to be added to the active hotwords 112 for the computing device 104 to request the hotword model. While music is playing, active hotword selectors 126 may include the hotwords "next", "pause" and "back" as active hotwords. When the music is stopped and the music application remains open, the active hotword selector 126 may update the active hotwords 112 to "Next", "Play" and "Back".

In some implementations, the user interface generator 144 creates a user interface for display on the computing device 104 . The user interface may present active hotwords 112 . For example, the user interface may instruct the user 102 to say “Next,” “Stop,” or “Back” to control the music, when the song begins to play. When the music is paused, the user interface may indicate that the user 102 can say “next”, “play” and “back” to control the music. User interface generator 144 may also generate a notification when a hotword is active.

For example, the user interface may indicate that the hotword “set alarm” is activated when the current time reaches 9 PM. Similarly, the user interface may indicate that the hotword “set alarm” is no longer active when the current time reaches midnight. The user interface may exhibit the effect of speaking each hotword. For example, when the music application is active and music is playing, the user interface shows that “Next” advances to the next song in the playlist, “Stop” stops the current song from playing, and “Back” indicates that the playlist may indicate a return to the previous song of

In some implementations, the user interface generator 144 can also generate the user interface to control when different hotwords are activated. The control interface may include a context when the hotword is activated and allows the user to update the context when the hotword is activated. Additionally or alternatively, the control interface allows user 102 to indicate active hotwords for each context. For example, the control interface may indicate that the hot word “news” has been activated from 8 am to 10 am. User 102 can adjust the context so that the hot word “news” is active from 8 am to noon.

The control interface may also indicate that when music is playing, the hotwords "next", "stop" or "back" have been activated. User 102 may update the hotwords for the music playback context only to “next” and “pause”. In some implementations, the control interface may also provide the user 102 the ability to add custom hotwords to existing or custom contexts. For example, the user 102 may enter the control interface “call mom” as a hotword, always activate the hotword, and detect a hotword that calls the contact “mom”. have. In addition, user 102 can add "up" and "down" to the music playback context and specify hotwords to control the volume. The user may also add a new context corresponding to the time period 11:00 AM to 1:00 PM. The user may add the hotword “order lunch” to activate during that time period and indicate that the hotword is to open a food ordering application.

In step F, user 102 speaks utterance 146 that includes hotword 148 . Computing device 104 receives utterance 146 via a microphone and processes corresponding audio data. The hotword detector 108 compares the hotword models 114 of the active hotwords 112 to identify whether the utterance 146 contains any active hotwords. When the hot word detector 108 identifies the hot word, the computing device performs the corresponding instruction. In the example shown in FIG. 1 , user 102 says “next”. Active hotwords 112 may be “pause,” “next,” and “back.” The hotword detector 108 compares the audio data corresponding to the utterance 146 with hotword models 114 corresponding to “pause,” “next,” and “back” and the utterance 146 is a hotword “ decide to include " Based on detecting the hotword “next” corresponding to the instruction to proceed to the next song, the computing device proceeds to the next song 150 in step G.

In some implementations, hotword detector 108 can detect hotwords that are not among active hotwords 112 but whose models are still stored in hotword models. In this case, hotword detector 108 may provide instructions to user interface generator 144 to create a user interface indicating that the hotword is not currently active. For example, user 102 may say “play” when music is playing. The hot word detector 108 may identify the hot word “play”. Because the hotword is not active, the computing device 104 does not perform any action. However, the user interface generator 144 may generate an interface indicating that the hotword "play" is not active and the active hotwords are "stop," "next," and "back."

In some implementations, computing device 104 may still be configured to identify the default hotword “ok computer”. In this example, computing device 104 may process audio data following the “ok computer” using speech recognition and execute appropriate actions based on subsequent recording of the audio data. For example, if the user 102 says "OK computer, next" while music is playing, then the computing device 104 identifies the "OK computer" hotword and selects the audio data including the instruction "next". After recording the sequel, proceed to the next song.

Similarly, a contextual hotword may follow an instruction. Continuing the example of "order lunch", user 102 may say "order lunch at a sandwich deli" while the hotword is active. In this example, the hotword detector 108 identifies the “Order lunch” hotword. Speech recognizer 116 generates the notation "Sandwich Deli." Computing device 104 may open a food ordering application and open a sandwich deli menu.

In some implementations, the hotword detector 108 generates a hotword confidence score for each initial portion of the processed audio data. If the hotword confidence score satisfies the threshold, the hotword detector 108 determines that the audio data contains a hotword. For example, if the hotword confidence score is 0.9, and the hotword confidence threshold is 0.8, the hotword detector 108 determines that the audio data includes a hotword.

In some implementations, if the hotword confidence score has a range below a threshold, the user interface generator 144 can generate an interface confirming that the user 102 has spoken the hotword. For example, the hotword confidence score may be 0.7. If the range is between 0.6 and 0.8, the user interface generator 144 may generate a user interface that requests the user 102 to confirm or repeat the hot word. In some implementations, once the user confirms that the user 102 has spoken the hotword, the computing device 104 may use the audio data to update the hotword model to improve future performance. The computing device 104 cannot use the audio data if there is too much noise in the audio data.

2 depicts an exemplary process 200 for identifying and processing contextual hotwords. In general, process 200 identifies hotwords based on the context of the device and assigns actions to the hotwords so that when a user speaks the hotword, the process performs the corresponding action. Process 200 will be described as being performed by a computer system including one or more computers, eg, system 100 shown in FIG. 1 .

The system determines ( 210 ) a context associated with the computing device. In some implementations, the system identifies an application running on the system. The system determines the context based on the application running on the system. For example, the application may be a music application. In this case, the context may be playing music. The system can also distinguish between applications running in the background and foreground. For example, a music application that runs and plays music in the background or foreground may have the same context as music playback. Applications such as browsers running in the background may not affect the context. In some implementations, the context can also relate to the state of the device, such as when the device is locked. Context may also relate to being displayed on a screen, eg, a “home screen”.

In some implementations, the context may be based on the movement of the system. For example, if the system is moving at a speed similar to that of a car, the system may determine that the context is "in a car." In some implementations, the context can be based on the location of the computing device. For example, the system may be located in the user's home. In this case, the context of the device may be “at home”. In some implementations, a context may be a combination of contexts. For example, if the system is locked and at the user's home, the context may be "locked at home".

The system determines the hotword 220 based on a context associated with the system. In some implementations, the system determines the hotword based on the identified hotwords in software. For example, a music application may identify hotwords for when the context is "music is playing" and when the music is paused and when the music application is open.

In some implementations, the system can determine hotwords based on previous usage. For example, if the user generally reads news during a certain time range, then the system may determine the hotword "news" when the context is in that time range. In some implementations, the system can display the hotword and indicate on the display the action performed by the hotword. The system may provide notifications when a hotword is activated and when the system deactivates a hotword.

After the system determines the hotword, it receives audio data corresponding to the utterance 230 . In some implementations, the system requests a hotword model for the hotword from the server, and the system determines the hotword. For example, if the system determines the hotword "next", then the system may request the hotword model for "next" from the server.

The system determines that the audio data contains a hotword 240 . In some implementations, the system determines that the audio data includes a hotword without performing speech recognition on the audio data. In some implementations, the system determines that the audio data includes a hotword by extracting audio features of the audio data corresponding to the utterance. The system generates a hotword confidence score by processing the audio features and comparing the possible audio features to the features of the hotword model.

If the hotword confidence score satisfies the hotword confidence threshold, then the system determines that the audio data includes the hotword. If the hotword confidence score does not satisfy the hotword confidence threshold, then the system determines that the audio data does not contain the hotword. For example, if the hotword confidence threshold is 0.8, then, audio data having a hotword confidence score of 0.8 or higher is labeled as containing a hotword, and audio data having a hotword confidence score of less than 0.8 does not contain the hotword. labeled as

In some implementations, the audio data includes only hotwords. For example, the user can only say the hotword "next". In some implementations, the initial portion of the audio data includes a hotword. For example, the hotword may be "order lunch" and the user may say "order lunch at the sandwich deli". In this case, the system identifies the hot word "lunch order" and uses speech recognition to process the audio data portion following the hot word.

The system performs an action associated with the hot word 250 in response to determining that the audio data includes the hot word. In some implementations, the system uses a hotword to identify an action. For example, when music is played and the user says "next", the system advances the song.

In some implementations, the system removes a hotword from the active hotword list when the context is no longer valid. For example, if the user stops playing music, then the system removes the hotword "next" from the list of active hotwords. In this case, when the user says "next", the system does nothing in response to "next".

In some implementations, the operation may be based on the hotword and any audio data following the hotword, or based only on audio data following the hotword. For example, the active hotword may be “directions,” and the user may say “directions to home.” In this case, the system may open the mapping application and begin providing directions to the user's home to the user.

In some implementations, the operation may depend on audio data following the hotword. There may be cases where an active hotword has different tasks depending on the audio data that follows the hotword. For example, the user may be at home and the system may be playing music. A user at home can cause the system to activate the hotword "increase" to increase the thermostat's temperature. A system playing music may also increase the volume of the music by the system activating the hotword "increase". In this case, the system may mitigate this conflict in one or more of the following ways.

The system can update the hotwords to "Raise the temperature" and "Turn up the volume". Just as the user says "turn up the volume" to increase the music volume or "raise the temperature" to increase the temperature of the thermostat, when the user says "raise" the system may display it on the screen. Alternatively, the system may keep "Raise" as a hotword and ask for a suggestion after "Raise" or ask for clarification from the user if the system performs speech recognition on some audio data after "Raise" . For example, the user may say "raise up". The system may display a request to clarify what to increment or play a synthesized voice requested by the user.

In some implementations, the system can detect an active hotword only in the presence of a suggestion or suffix for the spoken hotword, sometimes only in the presence of a suggestion or suffix for the spoken hotword. For example, the active hotword may be “set alarm”. The system can recognize the hotword "set alarm" only when a suggestion like "6 am" or "tomorrow morning" comes. In this case, the system may perform speech recognition on the portion of audio data following the hot word, and may not recognize the hot word unless additional speech terms follow the hot word. In some implementations, the additional phonetic terms are suggestions that the hotword accepts. The hotword "set an alarm" may allow suggestions like "6am", but not "tuna sandwich".

In some implementations, the system can determine that the audio data includes one or more hotwords. This can happen because the currently active hotwords sound similar. For example, the two active hotwords may be "next" and "text". In some implementations, the system can determine that the audio data includes a hotword, not necessarily a hotword. If the system determines that the two or more hot word models match the audio data, then the system may perform speech recognition on the portion of the audio data including the hot word to determine which hot word the user said.

3 shows an example of a computing device 300 and a mobile computing device 350 that may be used to implement the techniques described herein. Computing device 300 represents various forms of digital computers such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes and other suitable computers. intended to bet Mobile computing device 350 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, and other similar computing devices. The components presented herein, their connections and relationships, and their functions, are intended to be illustrative only and are not intended to limit the embodiments of the invention described and/or claimed herein.

The computing device 300 includes a processor 302 , a memory 304 , a storage device 306 , a high-speed interface 308 coupled to the memory 304 and multiple high-speed expansion ports 310 , and a low-speed expansion port 314 and and a low-speed interface 312 coupled to the storage device 306 . Each of the processor 302 , memory 304 , storage device 306 , high-speed interface 308 , high-speed expansion port 310 , and low-speed interface 312 are interconnected using various buses and are on a common motherboard. or may be mounted in any other suitable manner. Processor 302 processes instructions executed within computing device 300 , including instructions stored in memory 304 or storage device 306 , to display 316 coupled to high-speed interface 308 . Graphical information for the GUI can be displayed on the same external input/output device. In other implementations, multiple processors and/or multiple buses may be used as appropriate with multiple memories and memory types. Also, multiple computing devices may be connected, each providing some of the necessary operations (eg, a server bank, a group of blade servers, or a multi-processor system). Memory 304 stores information within computing device 300 . In some implementations, memory 304 is a volatile memory unit or units. In some implementations, memory 304 is a non-volatile memory unit or units. Memory 304 may also be another form of computer readable medium, such as a magnetic disk or optical disk.

The storage device 306 may provide a mass storage device for the computing device 300 . In some implementations, the storage device 306 is a device including a floppy disk device, a hard disk device, an optical disk device or tape device, a flash memory or other similar solid state memory device, or a device of a storage area network or other configuration. It may be or include a computer-readable medium such as an array. The instructions may be stored on an information medium. When executed by one or more processing devices (eg, processor 302 ), the instructions perform one or more methods as described above.

The instructions may also be stored on one or more storage devices, such as a computer readable medium or a machine readable medium (eg, memory 304 , storage 306 , or memory on processor 302 ).

High speed interface 308 manages bandwidth intensive operations for computing device 300 , while low speed interface 312 manages low bandwidth intensive operations. This function assignment is merely exemplary. In some implementations, high-speed interface 308 connects to memory 304 , display 316 (eg, via a graphics processor or accelerator), and high-speed expansion ports 310 that can accommodate various expansion cards. are combined In the implementation, the low-speed expansion port 314 is coupled to the storage device 306 and the low-speed expansion port 314 . A low-speed expansion port, which may include a variety of communication ports (eg, USB, Bluetooth, Ethernet, wireless Ethernet), is provided via one or more input/output devices such as a keyboard, pointing device, scanner, or via, for example, a network adapter. It may be coupled to a networking device such as a router or switch.

Computing device 300 may be implemented in a number of different forms as shown in the figures. For example, the computing device may be implemented as a standard server 320 , or multiple times as a group of such servers. The computing device may also be implemented in a personal computer, such as a laptop computer 322 . It may also be implemented as part of a rack server system 324 . Alternatively, components from computing device 300 may be combined with other components within a mobile device, such as mobile computing device 350 . Each of these devices may include one or more of computing device 300 and mobile computing device 350 , and the overall system may be comprised of multiple computing devices communicating with each other.

Mobile computing device 350 includes processor 352 , memory 364 , input/output devices such as display 354 , communication interface 366 and transceiver 368 , among other components. Mobile computing device 350 may also include a storage device, such as a micro drive or other device, to provide additional storage. Each of the processor 352 , memory 364 , display 354 , communication interface 366 , and transceiver 368 are interconnected using various buses, some components being on a common motherboard or in other ways as appropriate. can be installed with

Processor 352 may execute instructions within mobile computing device 350 , including instructions stored in memory 364 . The processor 352 may be implemented as a chipset of a chip that includes separate and multiple analog and digital processors. The processor 352 may be, for example, controlling user interfaces, applications executed by the mobile computing device 350 , and other components of the mobile computing device 350 , such as wireless communication by the mobile computing device 350 . It can provide coordination of elements.

The processor 352 may communicate with a user through a control interface 358 and a display interface 356 coupled to a display 354 . Display 354 may be, for example, a Thin-Film-Transistor Liquid Crystal Display (TFT LCD) or Organic Light Emitting Diode (OLED) display or other suitable display technology. Display interface 356 may include suitable circuitry for driving display 354 to provide graphics and other information to a user. Control interface 358 may receive commands from a user and translate these commands to communicate the commands to processor 352 . In addition, the external interface 362 may be provided to communicate with the processor 352 and to enable short-range communication of the mobile communication device 350 with other devices. External interface 362 may, for example, provide wired communication in some implementations, or wireless communication in other implementations, and multiple interfaces may also be used.

Memory 364 stores information within computing device 350 . Memory 364 may be implemented in one or more of a computer-readable medium or media, a volatile memory unit or units, or a non-volatile memory unit or units. Expansion memory 374 may also be provided and coupled to mobile computing device 350 via expansion interface 372 , which may include, for example, a Single In Line Memory Module (SIMM) card interface. This expansion memory 374 may provide extra storage space for the mobile computing device 350 , or may further store applications or other information for the mobile computing device 350 . Specifically, the expansion memory 374 may include instructions for performing or supplementing the above-described processing, and may also include security information. Thus, for example, expansion memory 374 may be provided as a secure module for mobile computing device 350 and may be programmed with instructions to enable secure use of mobile computing device 350 . Also, for example, a secure application that places identification information on the SIMM card in an unhackable manner may be provided via the SIMM card along with the additional information.

The memory may include, for example, flash memory and/or NVRAM memory (non-active random access memory), as described below. In some implementations, the instructions are stored on an information carrier and, when executed by one or more processing devices (eg, the processor 352 ), the instructions perform one or more methods, such as the method described above. The instructions may also be stored on one or more storage devices, such as one or more computer readable media or machine readable media (eg, memory 364 , extended memory 374 , or memory on processor 352 ). . In some implementations, the instructions may be received in a propagated signal, for example, via transceiver 368 or external interface 362 .

Mobile computing device 350 may communicate wirelessly via communication interface 366 , which may include digital signal processing circuitry, if desired. Communication interface 366 is inter alia Global System for Mobile communications (GSM), Short Message Service (SMS), Enhanced Messaging Service (EMS), or Multimedia Messaging Service (MMS), code division multiple access (CDMA), TDMA It can provide communication under various modes or protocols, such as time division multiple access (PDC), Personal Digital Cellular (PDC), Wideband Code Division Multiple Access (WCDMA), CDMA2000 or General Packet Radio Service (GPRS). Such communication may occur, for example, via transceiver 368 using radio frequencies. Also, short-range communications using, for example, Bluetooth, Wi-Fi or other such transceivers may occur. In addition, the Global Positioning System (GPS) receiver module 370 provides the mobile computing device 350 with additional navigation-related wireless data and location-related wireless data that can be suitably used by applications running on the mobile computing device 350 . can do.

Mobile computing device 350 may also communicate audibly using audio codec 360 that may receive spoken information from a user and convert the spoken information into usable digital information. Audio codec 360 may also generate audible sound for a user, such as through a speaker in a handset of mobile computing device 350 , for example. Such sound may include sound from a voice phone call, may include recorded sound (eg, voice messages, music files, etc.), and may also include applications running on mobile computing device 350 . It may include sound generated by

Mobile computing device 350 may be implemented in a number of different forms as shown in the figures. For example, this mobile computing device may be implemented as a cellular phone 380 . This computing device may also be implemented as part of a smart phone 382 , personal digital assistant, or other similar mobile device.

Various implementations of the systems and descriptions described herein may be realized in digital electronic circuitry, integrated circuits, specially designed application specific integrated circuits (ASICs), computer hardware, firmware, software, and/or combinations thereof. Such various implementations may include implementation in one or more computer programs executable and/or interpretable on a programmable system comprising at least one programmable processor, the at least one programmable processor comprising: a special purpose or general purpose processor coupled to receive data and instructions from the system, at least one input device and at least one output device, and transmit data and instructions to a storage system, at least one input device and at least one output device; can

Such computer programs (also referred to as programs, software, software applications or code) contain machine instructions for a programmable processor and are written in high-level procedural and/or object-oriented programming language, and/or assembly/machine language. can be implemented. As used herein, the terms "machine readable medium" and "computer readable medium" refer to machine instructions and/or to a programmable processor, including a machine readable medium that receives machine instructions as a machine readable signal. refers to any computer program product, apparatus, and/or device (eg, magnetic disks, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide data. The term “machine-readable signal” refers to any signal used to provide machine instructions and/or data to a programmable processor.

To provide interaction with a user, the systems and techniques described herein may include a display device (eg, a cathode ray tube (CRT) or liquid crystal display (LCD) monitor) for displaying information to a user; and a keyboard and a pointing device (eg, a mouse or trackball) through which the user can provide input to the computer. Other types of devices may be used to provide interaction with the user. For example, the feedback provided to the user may be any form of sensory feedback (eg, visual feedback, auditory feedback, or tactile feedback) and the input from the user may be any form including acoustic, voice, or tactile input. can be received as

The systems and technologies described herein may include a backend component (eg, a data server) or a middleware component (eg, an application server) or may include a frontend component (eg, a user a client computer having a web browser or graphical user interface capable of interacting with its implementation) or any combination of such back-end, middleware or front-end components. The components of the system may be interconnected by digital data communication (eg, a communication network) in any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), and the Internet.

A computing system may include clients and servers. Clients and servers are typically remote from each other and typically interact through a communications network. The relationship of client and server arises by virtue of computer programs running on each computer and having a client-server relationship to each other.

Although some implementations have been described above, other modifications are possible. For example, a client application is described as accessing a delegate(s), but in other implementations, the delegate(s) has access to other applications implemented by one or more processors, such as an application running on one or more servers. can be employed by Furthermore, the logic flows shown in the figures do not require the specific order shown or sequential order to achieve desirable results. In addition, other actions may be provided or removed from the described flow, and other components may be added or removed from the described systems. Accordingly, other implementations are within the scope of the following claims.

Claims (21)

A computer-implemented method comprising:
determining, by a computing device, a context associated with the computing device;
determining a hotword based on the context associated with the computing device, and after determining the hotword, receiving a hotword model corresponding to the hotword, wherein the hotword model is in the context associated with the computing device. including noise specific to - and;
after determining the hot word, receiving audio data corresponding to an utterance;
determining, using the hotword model, that the audio data includes the hotword;
in response to determining that the audio data includes the hotword, performing an operation associated with the hotword;
determining, by the computing device, that the context is no longer associated with the computing device; and
determining that subsequently received audio data comprising the hotword does not trigger an action
A computer-implemented method. According to claim 1,
Determining that the audio data includes the hotword comprises:
extracting audio features of the audio data corresponding to the utterance;
generating a hotword confidence score by processing the audio features;
determining whether the hotword confidence score satisfies a hotword confidence threshold; and
based on determining whether the hotword confidence score satisfies a hotword confidence threshold, determining that the audio data corresponding to the utterance includes the hotword.
A computer-implemented method. 3. The method of claim 1 or 2,
providing, for output, data identifying the hotword
A computer-implemented method. According to claim 1,
identifying, by the computing device, movement of the computing device;
wherein the context is based on movement of the computing device.
A computer-implemented method. According to claim 1,
identifying, by the computing device, an application running on the computing device;
wherein the context is based on the application running on the computing device.
A computer-implemented method. According to claim 1,
identifying, by the computing device, a location of the computing device;
wherein the context is based on a location of the computing device.
A computer-implemented method. delete According to claim 1,
Performing the operation associated with the hot word includes:
performing speech recognition on a portion of the audio data that does not include the hotword;
wherein the operation is based on a transcription of the portion of the audio that does not include the hotword.
A computer-implemented method. According to claim 1,
wherein the audio data includes only the hotword
A computer-implemented method. According to claim 1,
wherein the initial portion of the audio data includes the hotword.
A computer-implemented method. As a system,
one or more computers and one or more storage devices;
the one or more storage devices store instructions that, when executed by the one or more computers, are operable to cause the one or more computers to perform operations;
The actions are:
determining, by a computing device, a context associated with the computing device;
determining a hotword based on the context associated with the computing device, and after determining the hotword, receiving a hotword model corresponding to the hotword, wherein the hotword model is in the context associated with the computing device. including noise specific to - and;
receiving audio data corresponding to the utterance after determining the hot word;
determining, using the hotword model, that the audio data includes the hotword;
in response to determining that the audio data includes the hotword, performing an operation associated with the hotword;
determining, by the computing device, that the context is no longer associated with the computing device; and
determining that subsequently received audio data comprising the hotword does not trigger an action.
system. 12. The method of claim 11,
Determining that the audio data includes the hotword comprises:
extracting audio features of the audio data corresponding to the utterance;
generating a hotword confidence score by processing the audio features;
determining whether the hotword confidence score satisfies a hotword confidence threshold; and
based on determining whether the hotword confidence score satisfies a hotword confidence threshold, determining that the audio data corresponding to the utterance includes the hotword.
system. 13. The method of any one of claims 11 or 12,
The actions are:
further comprising, by the computing device, identifying an application running on the computing device;
wherein the context is based on the application running on the computing device.
system. delete 12. The method of claim 11,
Performing the operation associated with the hot word includes:
performing speech recognition on a portion of the audio data that does not include the hotword;
wherein the operation is based on a representation of a portion of the audio that does not include the hotword.
system. A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers, comprising:
when executed, cause the one or more computers to perform operations;
The actions are:
determining, by a computing device, a context associated with the computing device;
determining a hotword based on the context associated with the computing device, and after determining the hotword, receiving a hotword model corresponding to the hotword, wherein the hotword model is in the context associated with the computing device. including noise specific to - and;
receiving audio data corresponding to the utterance after determining the hot word;
determining, using the hotword model, that the audio data includes the hotword;
in response to determining that the audio data includes the hotword, performing an operation associated with the hotword;
determining, by the computing device, that the context is no longer associated with the computing device; and
determining that subsequently received audio data comprising the hotword does not trigger an action.
Non-transitory computer-readable media. A non-transitory computer readable medium storing a computer program configured to perform the method of claim 1 when executed by a processor. delete delete delete delete

Images