KR101752119B1 - Hotword detection on multiple devices - Google Patents

Abstract

Methods, systems, and apparatus are disclosed that include computer programs encoded on a computer storage medium for hot word detection in multiple devices. In one aspect, a method includes actions of a step of receiving, by a first computing device, audio data corresponding to a utterance. These actions further include determining a first value corresponding to the likelihood that the utterance includes a hot word. The actions further comprising receiving a second value corresponding to the likelihood that the utterance includes a hot word, the second value being determined by the second computing device. These actions further include comparing the first value to the second value. These actions further include initiating a speech recognition process on the audio data based on comparing the first value and the second value.

Description

[0001] HOT WORD DETECTION ON MULTIPLE DEVICES [0002]

This specification generally relates to recognizing words that people are speaking, or systems and techniques for what is called speech recognition.

A speech-enabled home or other environment - that is, a user simply has to say loudly a query or command, and the computer-based system processes and responds to the query and / The reality of what is being done - before us. A voice-enabled environment (e.g., home, workplace, school, etc.) may be implemented using a network of connected microphone devices distributed throughout various rooms or areas of the environment. Through the network of such microphones, the user has the ability to query the system verbally from virtually anywhere in the environment without having to have a computer or other device in front of or even near it. For example, during cooking in the kitchen, the user may ask the system "how many milliliters in three cups ?," and in response, In the form of a response, you can receive. Alternatively, the user may ask the system to "close when the nearest gas station closes" or "should I wear a coat today?" You can ask the same questions.

In addition, the user can ask / query or issue a query to the system, which is related to the user's personal information. For example, the user might ask the system to "When is my meeting with John?" Or remind the system "Remind me to call John when I get home. call John when I get back home) ".

For voice-enabled systems, the manner in which the user interacts with the system is primarily designed to use voice input, although not entirely. Thus, a system that potentially captures all of the utterances made in the environment, including those that are not pointing to the system, determines that any given utterance is directed to the system rather than to an individual present in the environment It must have some way. One way to achieve this is to use a hotword, which is reserved as a predetermined word that is said to consume the system's attention, by agreement among users in the environment. In an exemplary environment, the hot words used to draw attention to the system are the words "OK computer ". Thus, whenever the words "OK computer" are said, it is picked up by the microphone and delivered to the system, and the system performs speech recognition techniques to determine if the hot word has been spoken, . Thus, the voices directed to the system have the general form of a [hot word] [query], in which the "hot word" is the "OK computer" and the "query" is the system, either alone or with the server , Speech, recognized, parsed, and / or acted upon by the user.

According to one innovative aspect of the subject matter described in this specification, the user device receives the utterance spoken by the user. The user device determines whether the utterance includes a hot word and calculates a hot word confidence score indicating the likelihood that the utterance includes the hot word. The user device sends the score to other nearby user devices. The other user devices may have received the same utterance. The other user devices calculate hot word reliability scores and transmit their scores to the user device. The user device compares the hotword reliability scores. If the user device has the highest hot word reliability score, the user device remains active and prepares for processing additional audio. If the user device does not have the highest hot word reliability score, the user device does not process the additional audio.

In general, another innovative aspect of the subject matter described herein is a method comprising: receiving, by a first computing device, audio data corresponding to speech; Determining a first value corresponding to the likelihood that the utterance includes a hot word; Receiving a second value corresponding to a likelihood that the utterance includes a hot word, the second value being determined by a second computing device; Comparing the first value with the second value; And initiating speech recognition processing on the audio data based on comparing the first value and the second value.

These and other embodiments may each optionally include one or more of the following features. The actions further comprise determining that the first value satisfies a hot word score threshold. The actions further comprising transmitting the first value to the second computing device. The actions further comprise determining an activation state of the first computing device based on comparing the first value with the second value. The act of determining an activation state of the first computing device based on comparing the first value with the second value further comprises determining that the activation state is active. The actions comprising: receiving, by the first computing device, additional audio data corresponding to additional speech; Determining a third value corresponding to the likelihood that the additive utterance includes the hot word; Receiving a fourth value corresponding to the likelihood that the utterance includes the hot word, the fourth value being determined by a third computing device; Comparing the first value with the second value; And determining that the activated state of the first computing device is inactive, based on comparing the first value to the second value.

The act of transmitting the first value to the second computing device further comprises transmitting the first value to a server, via a local network, or via a short range radio. Receiving a second value corresponding to the likelihood that the utterance includes the hot word, wherein the second value is determined by the second computing device, is received from the server, through the local network, And receiving, via radio, a second value determined by the second computing device. The actions identifying the second computing device; And determining that the second computing device is configured to respond to voices comprising the hot word. Wherein the act of sending the first value to the second computing device further comprises transmitting a first identifier for the first computing device. Receiving a second value corresponding to the likelihood that the utterance includes the hot word, wherein the second value is determined by a second computing device, receives a second identifier for the second computing device . The action of determining that the activation state is active further comprises determining that a certain amount of time has elapsed since the step of receiving the audio data corresponding to the speech. The actions further comprise continuing to transmit the first value for a specified amount of time based on determining that the activation state is active.

Other embodiments of this aspect include corresponding systems, devices, and computer programs recorded on computer storage devices, each configured to perform the operations of the methods.

Certain embodiments of the subject matter described in this specification may be implemented to realize one or more of the following advantages. Multiple devices can detect a hot word and only one device will respond to the hot word.

The details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims.

Figure 1 is a drawing of an exemplary system for hot word detection.
Figure 2 is a drawing of an exemplary process for hot word detection.
Figure 3 shows an example of a computing device and a mobile computing device.
In the various figures, like reference numerals and names denote like elements.

In the not too distant future, it is possible that many devices can still be listening to hot words. When a single user has a plurality of devices (e.g., telephone, tablet, TV, etc.) trained to respond to their voice, it inhibits responding to hot words in devices that are not likely to be what the user intends to address May be desirable. For example, when a user speaks a hot word towards a device, if any of their other devices are nearby, they are also likely to trigger a voice search. In many cases, this is not the intent of the user. Thus, it can be advantageous if only a single device, specifically a device that is the subject of what the user is talking about triggers. The present specification deals with the problem of selecting the correct device for responding to a hot word and suppressing the reaction to the hot word at other devices.

1 is a diagram of an exemplary system 100 for hot word detection. Generally, the system 100 shows that the user 102 speaks voices 104 that are detected by the microphones of the computing devices 106, 108, and 110. Computing devices 106, 108, and 110 process the utterance 104 to determine the likelihood that the utterance 104 includes a hot word. Computing devices 106, 108, and 110 each send data to each other that indicates the likelihood that the utterance 104 includes a hot word. Computing devices 106, 108, and 110 each compare the data and a computing device that computes the highest likelihood that the utterance 104 will include a hot word will initiate speech recognition for the utterance 104 do. Computing devices that have not computed the highest likelihood that the utterance 104 will include a hot word will not initiate speech recognition for the voice after the utterance 104. [

Computing devices located near each other identify each other before sending data to the other computing device indicating the likelihood that the utterance 104 will correspond to a hot word. In some implementations, the computing devices identify each other by searching the local network for other devices configured to respond to the hot words. For example, the computing device 106 may look for other devices configured to respond to hot words and search the local area network and identify the computing device 108 and the computing device 110.

In some implementations, the computing devices identify other nearby computing devices that are configured to respond to the hot words by identifying the user logged in to each device. For example, user 102 is logged into computing devices 106, 108, and 110. The user 102 has a computing device 106 at the user's hand. The computing device 108 is located on a table, and the computing device 110 is located on a nearby wall. Computing device 106 detects computing devices 108 and 110, and each computing device shares information related to a user logged into the computing device, such as a user identifier. In some implementations, the computing devices identify other nearby computing devices configured to respond to the hot word by identifying computing devices configured to respond when the hot word is spoken by the same user through speaker identification . For example, the user 102 has configured the computing devices 106, 108, and 110, respectively, to respond to the voice of the user 102 when the user 102 speaks a hot word. Computing devices share speaker identification information by providing a user identifier for user 102 to each other computing device. In some implementations, the computing devices may identify other computing devices configured to respond to hot words over short-range wireless. For example, the computing device 106 may transmit a signal over short-range radio to retrieve other computing devices configured to respond to a hot word. Computing devices may use one or a combination of these techniques to identify other computing devices configured to respond to hot words

Once computing devices 106, 108, and 110 have identified other computing devices configured to respond to hot words, computing devices 106, 108, and 110 may share and store device identifiers for identified computing devices do. The identifiers may be based on the type of device, the IP address of the device, the MAC address, the name given to the device by the user, or any similar unique identifier. For example, the device identifier 112 for the computing device 106 may be a "phone. &Quot; The device identifier 114 for the computing device 108 may be a "tablet ". The device identifier 116 for the computing device 110 may be a "thermostat ". Computing devices 106, 108, and 110 store device identifiers for other computing devices configured to respond to hot words. Each computing device has a device group in which the computing device stores device identifiers. For example, the computing device 106 may be a device group that enumerates "tablets" and "thermostats" as two devices that are calculated by the computing device 106 and that will receive the possibility that the audio data contains hot words (118). The computing device 108 may be configured to include a group of devices 120 that enumerate the "phone" and "thermostat" as two devices, which are calculated by the computing device 108 and that will receive the possibility that the audio data contains a hot word I have. Computing device 110 has a device group 122 that enumerates "phone" and "tablet" as two devices, calculated by computing device 110, to receive the possibility that the audio data contains a hot word.

When the user 102 speaks a vocalization 104 of "OK computer ", each computing device with a microphone near the user 102 detects and processes the vocalization 104. [ Each computing device detects vocalization 104 through an audio input device, such as a microphone. Each microphone provides audio data to its audio subsystem. Each audio subsystem buffers, filters, and digitizes audio data. In some implementations, each computing device may also perform endpointing and speaker identification for audio data. The audio subsystem provides processed audio data to a hotworder. HotWorder compares the processed audio data to known hotword data and calculates a confidence score indicating the likelihood that utterance 104 will correspond to the hot word. Hotwaders can extract audio features, such as filterbank energy or mel-frequency cepstral coefficient, from the processed audio data. Hotwaders can process these audio features by using classifying windows, for example, using a support vector machine or neural network. Based on the processing of the audio features, HotWonder 124 calculates a reliability score of 0.85, HotWonder 126 calculates a reliability score of 0.6, and HotWonder 128 calculates a reliability score of 0.45. In some implementations, the confidence score can be normalized to a scale of 0 to 1, and a higher number indicates greater confidence that vocalization 104 will include the hot word.

Each computing device sends its own reliability score data packet to the other computing devices in the device group. Each reliability score data packet includes a respective reliability score and a respective device identifier for the computing device. For example, the computing device 106 may send a reliability score data packet 130 containing a reliability score of 0.85 and a device identifier "telephone " to the computing devices 108 and 110, which are computing devices in the device group 118 . The computing device 108 sends a confidence score data packet 132 containing a confidence score of 0.6 and a device identifier "tablet " to the computing devices 106 and 110, which are computing devices in the device group 120. The computing device 110 sends a reliability point data packet 134 containing a reliability score of 0.45 and a device identifier "temperature controller" to the computing devices 106 and 108, which are computing devices in the device group 118 .

In some implementations, the computing device may transmit a confidence point data packet if the confidence score meets a hot word score threshold. For example, if the hotword score threshold is 0.5, computing device 110 will not send confidence score data packet 134 to other computing devices in device group 122. Computing devices 106 and 108 will still send confidence point data packets 130 and 132 to computing devices in device groups 118 and 120, respectively.

In some implementations, the computing device transmitting the reliability point data packet may send the reliability point data packet directly to the other computing devices. For example, the computing device 106 may transmit the confidence score data packet 130 to the computing devices 108 and 110 over short-range wireless. The communication protocol used between the two computing devices may be a universal plug and play. In some implementations, the computing device transmitting the reliability point data packet may broadcast the reliability point data packet. In this case, the confidence score data packet may be received by the computing devices in the device group and by other computing devices. In some implementations, the computing device transmitting the reliability point data packet may send a reliability point data packet to the server, and then the server sends the reliability point data packet to the computing devices in the device group. The server may be located within the local area network of computing devices or may be accessible via the Internet. For example, the computing device 108 sends a reliability score data packet 132 and a list of computing devices in the device group 120 to the server. The server sends the confidence score data packet 132 to the computing devices 106 and 110. In case the computing device sends a reliability point data packet to another computing device, the receiving computing device may return confirmation that the receiving computing device has received the reliability point data packet.

Each computing device uses the score comparator to compare the hotword reliability scores received by the computing device. For example, the computing device 106 computed a hot word reliability score of 0.85 and received hot word reliability scores of 0.6 and 0.45. In this case, the score comparator 136 compares the three scores and identifies the highest score of 0.85. For computing devices 108 and 110, score comparators 138 and 140 arrive at similar conclusions that identify the highest score of 0.85 corresponding to computing device 106.

A computing device that determines that its own hot word reliability score is highest will initiate voice recognition for voice data after hot word utterance. For example, the user may speak of an "OK computer ", and the computing device 106 may determine that it has the highest hot word reliability score. The computing device 106 will initiate speech recognition for the audio data received after the hot word. If the user speaks an "Alice call, " the computing device 106 will process the utterance and execute the appropriate command. In some implementations, receiving a hot word may cause the computing devices receiving the hot word to activate from a sleep state. In this case, the computing device with the highest hot word reliability score remains in the awake state, while other computing devices that do not have the highest hot word reliability score do not process the voice data after hot word utterance, ≪ / RTI >

As illustrated in FIG. 1, the score comparator 136 has identified the hotword reliability score corresponding to the computing device 106 as the highest. Therefore, the device state 142 is "awake ". Score comparators 138 and 140 also identified the hotword reliability score corresponding to computing device 106 as the highest. Therefore, device states 144 and 146 are "asleep ". In some implementations, the activation state of the computing device may not be affected. For example, the user 102 may be viewing a movie at the computing device 108 and have the computing device 106 in the hands of the user. When the user 102 speaks an "OK computer ", the computing device 106 initiates speech recognition for audio data after the hot word since it has the highest hot word reliability score. The computing device 108 does not initiate speech recognition for the audio data after the hot word and continues to play back the movie.

In some implementations, the computing device that determines that it has the highest hot word reliability score waits for a certain amount of time before beginning to perform speech recognition on the voice after the hot word. This allows a computing device that computes the highest hot word reliability score to begin performing voice recognition for voice after hot words without waiting for a higher hot word reliability score. The point comparator 136 of the computing device 106 receives from the computing devices 108 and 110 the hot word reliability scores of 0.6 and 0.45 respectively as well as the hot word reliability scores of 0.85 from the hot worder 124 . Computing device 106 waits 500 milliseconds prior to performing speech recognition for speech after the hot word, from when hotware 124 computes the hot word reliability score of the "Ok computer" audio data. If the score comparator receives a higher score, the computing device may not wait for a certain amount of time before setting the device state to "sleep ". For example, the hotwonder 126 of the computing device 108 computes a hot word reliability score of 0.6 and receives hot word reliability scores of 0.85 and 0.45. When the computing device 108 receives a hot word reliability score of 0.85, the computing device 108 may set the device state 144 to "sleep ". This assumes that the computing device 108 receives a hotword reliability score of 0.85 within a certain amount of time after the hotword 126 computes a hotword reliability score of 0.6.

In some implementations, if the computing device has the highest hot word reliability score, the computing device continues to broadcast a reliability point data packet for a certain amount of time to ensure that other computing devices receive the reliability point data packet . This strategy would be most applicable when the computing device replies acknowledgment when it receives a reliability point data packet from another computing device. Thus, if computing device 106 sends a confidence score data packet 130 to computing devices in device group 118 and receives confirmation before a certain amount of time, such as 500 milliseconds, computing device 106 is hot It is possible to start performing speech recognition on the speech after the word. In the event that the computing devices broadcast their reliability score data packets and do not expect to be acknowledged, the computing devices may determine their hot word reliability scores for a certain amount of time, such as 500 milliseconds, It can continue to broadcast until it receives a higher hot word reliability score. For example, the computing device 110 computes a hotword reliability score of 0.45 and begins to broadcast a reliability score data packet 134. After 300 milliseconds, the computing device 110 receives the reliability score data packet 130 and stops broadcasting the reliability score data packet 134 because the reliability score data packet 130 is 0.85 hot Because the word reliability score is higher than the hot word reliability score of 45. As another broadcast example, the computing device 106 calculates a hotword reliability score of 0.45 and begins to broadcast a reliability score data packet 130. [ After 500 milliseconds, the computing device 106 stops broadcasting the reliability score data packet 130 and begins performing voice recognition for the voice after the hot word. The computing device 106 may receive the reliability point data packets 132 and 134 before 500 milliseconds have elapsed but since the hot word reliability scores in the reliability point data packets 132 and 134 are lower than 0.85, Wait until device 500 milliseconds have elapsed.

In some implementations, the computing device may begin performing voice recognition for voice after hot words until the computing device receives a higher hot word reliability score. The hotword computes a hotword confidence score, and if the hotword confidence score satisfies the threshold, the computing device performs speech recognition for the voice after the hotword. The computing device may perform speech recognition without any indication to the user of speech recognition. This allows the computing device to display results based on speech recognition to the user more quickly than if the computing device waited until doing so verified that the computing device had calculated the highest hotword score, To the user. As an example, the computing device 106 calculates a hotword reliability score of 0.85 and begins performing speech recognition for the voice after the hot word. The computing device 106 receives the reliability score data packets 132 and 134 and determines that the hot word reliability score of 0.85 is the highest. The computing device 106 continues to perform speech recognition for the voice after the hot word and present the results to the user. For the computing device 108, the hotwirder 126 computes a hotword reliability score of 0.6, and the computing device 108 begins performing speech recognition for the voice after the hot word without displaying data to the user do. When the computing device 108 receives the reliability point data packet 130 that includes the hot word reliability of 0.85, the computing device stops performing voice recognition. No data is displayed to the user, and the user is likely to be impressed that the computing device 108 has remained in the "sleep" state.

In some implementations, scores may be reported from the HotWorder prior to the end of the hot word, e.g., a partial hot word, to avoid any wait time after the hot word is spoken. For example, when the user is speaking "Ok computer ", the computing device can calculate a partial hot word reliability score if the user completes saying" OK comp ". The computing device may then share a partial hotword reliability score with other computing devices. A computing device with the highest partial hot-word reliability score can continue to process the user's voice.

In some implementations, when a computing device determines that a hot word reliability score meets a threshold, the computing device may generate audible or noisy speech, e.g., at a particular frequency or frequency pattern. The tone will signal other computing devices that the computing device will continue to process the audio data after the hot word. Other computing devices will receive this tone and stop processing the audio data. For example, the user refers to "Ok computer ". One of the computing devices calculates a hotword reliability score that is above a threshold. If the computing device determines that the hotword reliability score is above the threshold, the computing device produces a tone of 18 kilohertz. Other computing devices in the vicinity of the user may be calculating the hotword reliability score and calculating the hotword reliability score when the other computing devices receive the tone. When the other computing devices receive the tone, the other computing devices stop processing the user's voice. In some implementations, the computing device may encode a hot word reliability score in an audible or audible tone. For example, if the hotword reliability score is 0.5, the computing device may generate an audible or an audible tone that includes a frequency pattern that encodes a score of 0.5.

In some implementations, the computing devices may use different audio metrics to select a computing device to continue processing the user's voice. For example, computing devices can use loudness to determine which computing device will continue to process the user's voice. A computing device that detects the largest voice can continue to process the user's voice. As another example, a computing device that is currently in use or has an active display may notify other computing devices that it will continue to process the user's voice if it detects a hot word.

In some implementations, each computing device in the vicinity of the user while the user is speaking receives the audio data and sends the audio data to the server to improve speech recognition. Each computing device can receive audio data corresponding to the user ' s voice. Although only one computing device appears to be processing the user's voice to the user, each computing device may transmit audio data to the server. The server can then use the audio data received from each computing device to improve speech recognition because the server can compare different audio samples corresponding to the same utterance. For example, the user says, "Ok computer, remind me to buy milk." When the user finishes saying "Ok computer", it is likely that nearby computing devices have determined which computing device has the highest hot word reliability score, and that computing device reminds the user to "buy milk Will treat and respond to the words when speaking of them. Other computing devices will also receive a "remind me to buy milk". Other computing devices will not respond to the "remind me to buy milk" voices, but other computing devices can send audio data corresponding to "remind me to buy milk" to the server. Computing devices that respond to "remind to buy milk" can also send its audio data to the server. The server can process the audio data to improve speech recognition because the server has different audio samples from different computing devices corresponding to the same "remember to buy milk" utterance.

2 is a diagram of an exemplary process 200 for hot word detection. Process 200 may be performed by a computing device, such as computing device 108 from FIG. The process 200 determines whether to perform speech recognition for the portion of the utterance after the hot word by calculating a value corresponding to the likelihood that the utterance will include a hot word and comparing the value with other values computed by other computing devices .

The computing device receives the audio data corresponding to the utterance (210). The user speaks voices and the microphone of the computing device receives the voiced audio data. The computing device processes the audio data by buttering, filtering, endpointing, and digitizing the audio data. By way of example, the user may speak "Ok, computer" and the microphone of the computing device will receive audio data corresponding to "Ok, computer". The audio subsystem of the computing device will sample, buffer, filter, and end point the audio data for further processing by the computing device.

The computing device determines 220 a first value corresponding to the likelihood that the utterance includes a hot word. The computing device determines a first value, which may be referred to as a hot word reliability score, by comparing the audio data of the utterance to a group of audio samples containing the hot word or by analyzing the audio characteristics of the audio data of utterance do. The first value may be normalized to a scale of 0 to 1, where 1 represents the highest probability that the utterance will include the hot word. In some implementations, the computing device identifies the second computing device and determines that the second computing device is configured to respond to voices, including hot words, and configured by the user to respond to hot words. The user may be logged in to both the computing device and the second computing device. Both the computing device and the second computing device may be configured to respond to the user's voice. The computing device and the second computing device may be connected to the same local area network. Both the computing device and the second computing device may be located within a certain distance of each other, such as 10 meters, as determined by GPS or signal strength. For example, these computing devices may communicate by short-range wireless. The computing device may detect the strength of the signal transmitted by the second computing device as 5 dBm and translate it to a corresponding distance, such as 5 meters.

The computing device receives (230) a second value corresponding to the likelihood that the utterance includes a hot word, and a second value determined by the second computing device. The second computing device receives the vocalization through the microphone of the second computing device. The second computing device processes the received audio data corresponding to the utterance and determines a second value or a second hot word reliability score. The second hot word reliability score reflects the likelihood that the utterance, calculated by the second computing device, contains a hot word. In some implementations, a computing device transmits a first value to a second computing device using one or more of the following techniques. The computing device may transmit a first value to a second computing device via a server accessible through the Internet, through a server located in the local area network, or directly via a local area network or short-range wireless. The computing device may only transmit a first value to the second computing device, or the computing device may broadcast a first value such that other computing devices may also receive the first value. The computing device may receive the second value from the second computing device using the same or a different technique as the computing device transmits the first value.

In some implementations, the computing device may calculate a speech intensity score for voicing or a signal-to-noise ratio for voicing. The computing device may combine the sound intensity score, the signal-to-noise ratio, and the hot word reliability score to determine a new value for comparison with similar values from other computing devices. For example, a computing device may calculate a hotword reliability score and a signal-to-noise ratio. The computing device can then combine the two scores and compare them with similarly computed scores from other computing devices. In some implementations, the computing device may calculate different scores and send each score to other computing devices for comparison. For example, the computing device may calculate a sound intensity score and a hot word reliability score for utterance. The computing device may then send the scores to other computing devices for comparison.

In some implementations, the computing device may transmit the first identifier along with the first value. The identifier may be based on one or more of the address of the computing device, the name of the computing device given by the user, or the location of the computing device. For example, the identifier may be "69.123.132.43" or "phone. &Quot; Similarly, the second computing device may transmit the second identifier along with the second value. In some implementations, the computing device may send the first identifier to the particular computing devices previously identified as being configured to respond to the hot word by the computing device. For example, a computing device may have previously identified a second computing device as being configured to respond to a hot word, because in addition to being able to respond to a hot word, the same user as the computing device has been logged into the second computing device have.

The computing device compares the first value to the second value (240). The computing device then initiates a speech recognition process on the audio data based on the result of the comparison (250). In some implementations, for example, the computing device initiates speech recognition when the first value is greater than or equal to a second value. If the user speaks "ok computer, call Carol", the computing device will begin processing "Carol call" by performing speech recognition for "Carol call" because the first value is above the second value. In some implementations, the computing device sets the activation state. If the first value is greater than or equal to the second value, the computing device sets the activation state as active or "awake. &Quot; In the "awake" state, the computing device displays results from speech recognition.

In some implementations, the computing device compares the first value with the second value and determines that the first value is less than the second value. The computing device sets the activation state to inactive or "sleep" based on the determination that the first value is less than the second value. In the "sleep" state, the computing device is not visible to the user, either active or processing audio data.

In some implementations, when the computing device determines that the first value is greater than or equal to the second value, the computing device may wait for a certain amount of time before setting the activation state to active. The computing device may wait a certain amount of time to increase the likelihood that the computing device will not receive a higher value from another computing device. The amount of time of a particular amount may be fixed or may vary depending on the technique by which computing devices transmit and receive values. In some implementations, when the computing device determines that the first value is greater than or equal to the second value, the computing device may continue to transmit the first value for a certain amount of time. By continually transmitting the first value for a particular amount of time, the computing device increases the likelihood that the first value is received by the other computing devices. In case the computing device determines that the first value is less than the second value, the computing device may cease transmission of the first value.

In some implementations, the computing device may consider the side information when deciding whether to execute an instruction after the hot word. One example of additional information may be part of the vocalization after the hot word. Typically, audio data after a hot word corresponds to a command to a computing device such as "Sally Call," " play Halloween Movie, "or" set heat to 70 degrees & . The computing device may identify a typical device that can handle the type of request or handle the request. A request to call a person will typically be handled by the phone based on typical preprogrammed uses or based on usage patterns of the user of the device. If a user typically watches a movie on a tablet, the tablet can handle a request to play the movie. If the temperature controller can control the temperature, the temperature controller can handle the temperature controls.

In order for the computing device to consider the portion of speech after the hot word, the computing device will have to initiate speech recognition for the audio data if it is likely to identify the hot word. The computing device may classify the command portion of the utterance and calculate the frequency of commands in that classification. The computing device may send its frequency to other computing devices along with the hot word reliability score. Each computing device may use its frequencies and hot word reliability scores to determine whether to execute a command after the hot word.

For example, if a user utters "OK computer, play Michael Jackson" and the computing device is the phone the user uses to listen to music at 20% of the time, Can be transmitted together with the word reliability score. A computing device, such as a tablet, that a user uses to listen to music at 5% of the time may send that information to other computing devices along with a hotword reliability score. The computing devices may use a combination of the hotword reliability score and the percentage of music playback time to determine whether to execute the command.

FIG. 3 shows an example of a computing device 300 and a mobile computing device 350 that may be utilized to implement the techniques described herein. The computing device 300 is intended to represent various types of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The mobile computing device 350 is intended to represent various types of mobile devices, such as personal digital assistants, cell phones, smart phones, and other similar computing devices. The components, their connections and relationships, and their functions illustrated herein are by way of example only and are not intended to be limiting.

The computing device 300 includes a processor 302, a memory 304, a storage device 306, a memory 304 and a high speed interface 308 coupled to the plurality of high speed expansion ports 310, 314 and a low speed interface 312 connected to the storage device 306. Each of the processor 302, the memory 304, the storage device 306, the high speed interface 308, the high speed expansion ports 310 and the low speed interface 312 are interconnected using various busses, May be mounted on the board or, suitably in other manners. The processor 302 may include instructions stored in the memory 304 or stored in the storage device 306 to display graphical information for the GUI at an external input and output device, such as the display 316 coupled to the high- And processing instructions within computer device 300 for execution. In other implementations, multiple processors and / or busses may be used with multiple memories and memory types, as appropriate. In addition, multiple computing devices may be connected, and each device provides portions of the required operations (e.g., as a server bank, group of blade servers, or a multi-processor system).

The memory 304 stores information within the computing device 300. In some implementations, memory 304 is a volatile memory unit or unit. In some implementations, memory 304 is a non-volatile memory unit or unit. The memory 304 may also be another type of computer-readable medium, such as a magnetic or optical disk.

The storage device 306 may provide mass storage for the computing device 300. In some implementations, the storage device 306 may include a floppy disk device, a hard disk device, an optical disk device, or a device in a tape device, flash memory or other similar solid state memory device, or in a storage area network or other configurations Or a computer-readable medium, such as an array of devices. The instructions may be stored in an information carrier. The instructions, when executed by one or more processing devices (e.g., processor 302), perform one or more methods, such as those described above. The instructions may also be stored by one or more storage devices, such as computer- or machine-readable media (e.g., memory 304, storage device 306, or memory of processor 302).

The high speed interface 308 manages bandwidth intensive operations for the computing device 300, while the low speed interface 312 manages low bandwidth intensive operations. The assignment of these functions is only an example. In some implementations, the high speed interface 308 may include a memory 304, a display 316 (e.g., via a graphics processor or accelerator), and a high speed expansion port (not shown) 310 < / RTI > In an implementation, the low speed interface 312 is coupled to the storage device 306 and the slow expansion port 314. The slow expansion port 314, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be connected to one or more input / output devices such as a keyboard, pointing device, scanner, May be coupled to the device (e.g., via a network adapter).

The computing device 300 may be implemented in a number of different forms, as shown in the figures. For example, it may be implemented as standard server 320, or multiple times in a group of such servers. In addition, it may be implemented as a personal computer, such as laptop computer 322. [ It may also be implemented as part of the rack server system 324. [ Alternatively, components from the computing device 300 may be combined with other components (not shown) within the mobile device, such as the mobile computing device 350. Each of such devices may include one or more of computing device 300 and mobile computing device 350, and the entire system may be comprised of a plurality of computing devices in communication with one another.

The mobile computing device 350 includes among other components a processor 352, a memory 364, input and output devices such as a display 354, a communication interface 366, and a transceiver 368. The mobile computing device 350 may also include a storage device, such as a micro-drive or other device, to provide additional storage. The processor 352, the memory 364, the display 354, the communication interface 366 and the transceiver 368 are each interconnected using a variety of buses, and several of the components may be interconnected on a common motherboard, It can be mounted in other ways.

The processor 352 may execute instructions in the mobile computing device 350, including instructions stored in the memory 364. [ The processor 352 may be implemented as a chipset of chips comprising a plurality of separate analog and digital processors. The processor 352 may be any suitable computing device that can be used by the mobile computing device 350 to communicate with other mobile computing devices 350, such as, for example, controls of user interfaces, applications executed by the mobile computing device 350, And may provide coordination of components.

The processor 352 may communicate with the user via the control interface 358 and the display interface 356 coupled to the display 354. The display 354 may be, for example, a thin film transistor (TFT) display or an organic light emitting diode (OLED) display, or other suitable display technology. Display interface 356 may include suitable circuitry for driving display 354 to present graphics and other information to a user. The control interface 358 may receive instructions from the user and convert them to submit them to the processor 352. In addition, external interface 362 may provide for communication with processor 352 to enable close communication between mobile computing device 350 and other devices. The external interface 362 may, for example, provide wired communication in some implementations, or wireless communications in other implementations, and multiple interfaces may also be used.

The memory 364 stores information within the mobile computing device 350. The memory 364 may be embodied in one or more of computer-readable media or media, volatile memory units or units, or non-volatile memory units or units. An expansion memory 374 is also provided and may be coupled to the mobile computing device 350 via an expansion interface 372, which may include, for example, a Single In Line Memory Module (SIMM) card interface. Extension memory 374 may provide additional storage space for mobile computing device 350 or may also store applications or other information for mobile computing device 350. [ In particular, the expansion memory 374 may include instructions to perform or supplement the processes described above, and may also include security information. Thus, for example, the extended memory 374 may be provided as a security module for the mobile computing device 350 and may be programmed with instructions that allow secure use of the mobile computing device 350. In addition, secure applications can be provided over SIMM cards, along with additional information, such as putting the identification information in a non-hackable manner on the SIMM card.

The memory may include, for example, flash memory and / or NVRAM memory (non-volatile random access memory), as discussed below. In some implementations, the instructions may be stored in an information carrier. The instructions, when executed by one or more processing devices (e.g., processor 352), perform one or more methods, such as those described above. The instructions may also be stored by one or more storage devices, such as one or more computer- or machine-readable media (e.g., memory 364, expansion memory 374, or memory of processor 352) . In some implementations, the instructions may be received in the propagated signal, e.g., via transceiver 368 or external interface 362.

The mobile computing device 350 may wirelessly communicate via the communication interface 366, which may include digital signal processing circuitry as needed. The communication interface 366 may be implemented in a variety of communication systems such as Global System for Mobile communications, SMS (Short Message Service), EMS (Enhanced Messaging Service), or MMS messaging (Multimedia Messaging Service), code division multiple access (CDMA) , Communications in various modes or protocols, such as time division multiple access (TDMA), personal digital cellular (PDC), wideband code division multiple access (WCDMA), CDMA2000, or general packet radio service . Such communication may occur through the transceiver 368 using, for example, radio frequency. In addition, short-range communications may occur using, for example, Bluetooth, WiFi, or other such transceivers (not shown). In addition, a Global Positioning System (GPS) receiver module 370 may provide additional navigation-and location-related wireless data to the mobile computing device 350, which data may be stored in applications running on the mobile computing device 350 . ≪ / RTI >

The mobile computing device 350 may also audibly communicate using an audio codec 360, which may receive verbal information from the user and convert it into usable digital information. The audio codec 360 may likewise generate an audible sound for the user, e.g., via a speaker, e.g., in a handset of the mobile computing device 350. Such a tone may include notes from voice telephone calls and may include recorded notes (e.g., voice messages, music files, etc.) and may also be executed by applications running on the mobile computing device 350 ≪ / RTI >

The mobile computing device 350 may be implemented in a number of different forms, as shown in the figures. For example, it may be implemented as a cellular phone 380. It may also be implemented as part of a smart phone 382, personal digital assistant, or other similar mobile device.

Various implementations of the systems and techniques described herein may be realized with digital electronic circuits, integrated circuits, specially designed application specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof . These various implementations include a storage system, at least one input device, and at least one programmable, which may be special or general purpose, coupled to receive data and instructions from the at least one output device and to transmit data and instructions thereto. And may include implementations in one or more computer programs executable and / or interpretable in a programmable system including a processor.

These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in an advanced procedure and / or in an object oriented programming language and / . As used herein, the terms machine-readable medium and computer-readable medium include machine-readable storage media for receiving machine instructions as machine-readable signals to cause machine instructions and / Refers to any storage medium including devices and / or devices (e.g., magnetic disks, optical disks, memory, programmable logic devices (PLDs)) used to provide data. The term machine-readable signal refers to any signal that is used to provide machine instructions and / or data to the programmable processor.

To provide for interaction with a user, the systems and techniques described herein include a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) for displaying information to a user, May be implemented in a computer having a keyboard and pointing device (e.g., a mouse or trackball) available to provide input to a computer. Other types of devices may also be used to provide interaction with the user; For example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); The input from the user may be received in any form, including acoustic, voice, or tactile input.

The systems and techniques described herein may be implemented in a system that includes a back end component (e.g., as a data server), or a middleware component (e.g., an application server), or a front end component A client computer with a graphical user interface or a web browser that can be used to interact with the implementation of the disclosed systems and techniques), or any combination of such back-end, middleware, or front- . The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

The computing system may include clients and servers. Clients and servers are typically far apart and typically interact through a communications network. The relationship between a client and a server is generated by computer programs running on their respective computers and having a client-server relationship with each other.

Although a few implementations have been described in detail above, other modifications are possible. For example, while a client application is described as accessing a delegate (s), in other implementations, the delegate (s) may include other applications implemented by one or more processors, such as an application executing on one or more servers Lt; / RTI > In addition, the logic flows depicted in the Figures do not require the specific order shown, or sequential order, to achieve the desired results. In addition, other actions may be provided, and from the described flows, actions may be removed and other components added to, or removed from, the described systems. Accordingly, other implementations are within the scope of the following claims.

Claims (22)

A computer-implemented method for determining which of a plurality of computing devices to perform automatic speech recognition,
Receiving, by the first computing device, audio data corresponding to utterance;
The method of claim 1, further comprising: prior to starting automated speech recognition processing on the audio data, using a classifier to classify the audio data as comprising a specific hot word or not including the specific hot word Processing the audio data;
Based on processing the audio data using the classifier to classify the audio data as comprising a specific hot word or not including the specific hot word, Determining a first value that reflects the first value;
Receiving a second value determined by the second computing device, the second value reflecting the second possibility that the utterance includes the particular hot word;
Comparing the first value that reflects the first possibility that the utterance includes the specific hot word and the second value that reflects the second possibility that the utterance includes the particular hot word; And
Based on the first value reflecting the first possibility that the utterance includes the specific hot word and the second value reflecting the second possibility that the utterance will include the particular hot word, Determining whether to start performing automatic speech recognition processing on the audio data
≪ / RTI > The method according to claim 1,
Determining that the first value satisfies a hotword score; And
Sending the first value to the second computing device based on determining that the first value satisfies the hot word score, the first value reflecting the first possibility that the utterance includes the particular hot word
≪ / RTI > The method according to claim 1,
Based on the first value reflecting the first possibility that the utterance includes the specific hot word and the second value reflecting the second possibility that the utterance will include the particular hot word, Determining an activation state of the first computing device. 4. The method of claim 3, wherein the utterance includes a first value that reflects the first possibility to include the particular hot word and a second value that reflects the second possibility that the utterance includes the particular hot word. Based on the comparison, determining the activation state of the first computing device comprises:
Determining that the activation state of the first computing device is active. The method according to claim 1,
Receiving, by the first computing device, additional audio data corresponding to additional speech;
Processing the additional audio data using a classifier that classifies audio data as containing a specific hot word or not including the specific hot word before starting automatic speech recognition processing on the additional audio data;
Based on processing the additional audio data by using the classifier that classifies the audio data as including a specific hot word or not including the specific hot word, 3 determining a third value that reflects the likelihood;
Receiving a fourth value determined by the third computing device, the fourth value reflecting the fourth possibility that the additive utterance will include the particular hot word;
Comparing the third value that reflects the third possibility that the additive utterance includes the specific hot word and the fourth value that reflects the fourth possibility that the utterance includes the specific hot word; And
Wherein the additive utterance is based on comparing the third value that reflects the third possibility to include the particular hot word and the fourth value that reflects the fourth possibility that the additive utterance includes the particular hot word And determining whether to start performing automatic speech recognition processing on the additional audio data
≪ / RTI > The method according to claim 1,
Receiving a second value, determined by the second computing device, the second value reflecting the second possibility that the utterance includes the particular hot word comprises:
Receiving a second value from the server, through the local network, or via a short-range wireless communication channel, the vocalization reflecting the second possibility to include the particular hot word. The method according to claim 1,
Determining that the second computing device is configured to respond to voices comprising the particular hot word,
Wherein the step of comparing the first value, which reflects the first possibility that the utterance includes the specific hot word, and the second value, which reflects the second possibility that the utterance includes the specific hot word, 2 < / RTI > computing device is configured to respond to voices comprising the particular hot word. The method according to claim 1,
Receiving a second value, determined by the second computing device, the second value reflecting the second possibility that the utterance includes the particular hot word comprises:
And receiving a second identifier of the second computing device. 5. The method of claim 4, wherein determining whether to start performing automatic speech recognition processing on the audio data further comprises: determining that a specific amount of time has elapsed since receiving the audio data corresponding to the utterance / RTI > 5. The method of claim 4,
The method comprising transmitting the first value reflecting the first possibility that the utterance will include the particular hot word for a specified amount of time based on determining that the activation state is active. The method according to claim 1,
Based on the first value reflecting the first possibility that the utterance includes the specific hot word and the second value reflecting the second possibility that the utterance will include the particular hot word, The step of determining that the first value reflecting the first possibility that the utterance includes the specific hot word is greater than the second value reflecting the second possibility that the utterance includes the particular hot word Including,
Wherein the step of determining whether to perform automatic speech recognition processing on the audio data comprises:
Wherein the first value reflecting the first possibility that the utterance includes the particular hot word is determined based on determining that the utterance is greater than the second value reflecting the second possibility to include the particular hot word And determining to start performing automatic speech recognition processing on the audio data. The method according to claim 1,
Based on the first value reflecting the first possibility that the utterance includes the specific hot word and the second value reflecting the second possibility that the utterance will include the particular hot word, The step of determining that the first value reflecting the first possibility that the utterance includes the specific hot word is less than the second value reflecting the second possibility that the utterance includes the particular hot word Including,
Wherein the step of determining whether to perform automatic speech recognition processing on the audio data comprises:
Wherein the first value reflecting the first possibility that the utterance includes the particular hot word is determined based on determining that the utterance is less than the second value reflecting the second possibility to include the particular hot word And deciding not to start performing automatic speech recognition processing on the audio data. 2. The method of claim 1, wherein processing the audio data using a classifier that classifies audio data as comprising a particular hot word or not includes the specific hot word comprises:
Extracting filterbank energies or mel-frequency cepstral coefficients from the audio data. 2. The method of claim 1, wherein processing the audio data using a classifier that classifies audio data as comprising a particular hot word or not includes the specific hot word comprises:
Processing the audio data using a support vector machine or a neural network. As a computing device,
And at least one storage device operable, when executed by the computing device, to store instructions operable to cause the computing device to perform operations for determining which of a plurality of computing devices to perform automatic speech recognition, The operations include:
Receiving, by the first computing device, audio data corresponding to the utterance;
Processing the audio data using a classifier that classifies the audio data as comprising a specific hot word or not including the specific hot word before starting automatic speech recognition processing on the audio data;
Based on processing the audio data using the classifier to classify the audio data as comprising a specific hot word or not including the specific hot word, Determining a first value that reflects the first value;
Receiving a second value determined by the second computing device, the second value reflecting the second possibility that the utterance includes the particular hot word;
Comparing the first value that reflects the first possibility that the utterance includes the particular hot word and the second value that reflects the second possibility that the utterance includes the particular hot word; And
Based on the first value reflecting the first possibility that the utterance includes the specific hot word and the second value reflecting the second possibility that the utterance will include the particular hot word, An operation for determining whether to start performing automatic speech recognition processing on the audio data
Gt; computing device. ≪ / RTI > 16. The method of claim 15, wherein the operations comprise:
Determining that the first value satisfies a hot word score; And
Sending the first value to the second computing device, the first value reflecting the first possibility that the utterance includes the particular hot word, based on determining that the first value satisfies the hot word score
Lt; / RTI > 16. The method of claim 15, wherein the operations comprise:
Based on the first value reflecting the first possibility that the utterance includes the specific hot word and the second value reflecting the second possibility that the utterance will include the particular hot word, The method further comprising: determining an activation state of the first computing device. 18. The method of claim 17, wherein the first value reflecting the first possibility that the utterance includes the particular hot word and the second value reflecting the second possibility that the utterance includes the particular hot word Based on the comparison, the act of determining the activation state of the first computing device comprises:
And determining that the activation state of the first computing device is active. 16. The method of claim 15, wherein the operations comprise:
Receiving, by the first computing device, additional audio data corresponding to additional speech;
Processing the additional audio data using a classifier that classifies the audio data as containing a specific hot word or not including the specific hot word before starting automatic speech recognition processing on the additional audio data;
Based on processing the additional audio data by using the classifier that classifies the audio data as including a specific hot word or not including the specific hot word, 3 < / RTI >
Receiving a fourth value determined by the third computing device, the fourth value reflecting the fourth possibility that the additive utterance includes the particular hot word;
The third value reflecting the third possibility that the additive utterance includes the specific hot word and the fourth value reflecting the fourth possibility that the additive utterance will include the specific hot word; And
Wherein the additive utterance is based on comparing the third value that reflects the third possibility to include the particular hot word and the fourth value that reflects the fourth possibility that the additive utterance includes the particular hot word An operation of determining whether to start performing automatic speech recognition processing on the additional audio data
Lt; / RTI > 16. The method of claim 15,
The operation of receiving a second value determined by the second computing device, the second value reflecting the second possibility that the utterance includes the particular hot word comprises:
Receiving from the server, via the local network, or via a short-range wireless communication channel, a second value that reflects the second possibility that the utterance will include the particular hot word. 16. The method of claim 15, wherein the operations comprise:
Further comprising determining that the second computing device is configured to respond to voices comprising the particular hot word,
The first value reflecting the first possibility that the utterance includes the particular hot word and the second value reflecting the second possibility that the utterance will include the particular hot word may be compared to the second value, Wherein the computing device is responsive to determining that it is configured to respond to voices comprising the particular hot word. 25. A computer readable storage medium storing software comprising instructions executable by one or more computers,
Wherein the instructions, when executed, cause the one or more computers to perform operations to determine which of a plurality of computing devices to perform automatic speech recognition, the operations comprising:
Receiving, by the first computing device, audio data corresponding to the utterance;
Processing the audio data using a classifier that classifies the audio data as comprising a specific hot word or not including the specific hot word before starting automatic speech recognition processing on the audio data;
Based on processing the audio data using the classifier to classify the audio data as comprising a specific hot word or not including the specific hot word, Determining a first value that reflects the first value;
Receiving a second value determined by the second computing device, the second value reflecting the second possibility that the utterance includes the particular hot word;
Comparing the first value that reflects the first possibility that the utterance includes the particular hot word and the second value that reflects the second possibility that the utterance includes the particular hot word; And
Based on the first value reflecting the first possibility that the utterance includes the specific hot word and the second value reflecting the second possibility that the utterance will include the particular hot word, An operation for determining whether to start performing automatic speech recognition processing on the audio data
Gt; computer-readable < / RTI >

Images