JP2024508033A - Instant learning of text-speech during dialogue - Google Patents

Abstract

A method for instantaneous text-to-speech (TTS) learning during interaction includes receiving user pronunciations (202) of particular words present in a question spoken by a user. The method also includes receiving a TTS pronunciation (204) of the same particular word present in the TTS input, the TTS pronunciation of the particular word being different from a user pronunciation of the particular word. The method also includes obtaining user pronunciation-related features (210) and TTS pronunciation-related features (230) that are associated with a particular word. The method also includes generating a pronunciation decision (250) that selects one of the user pronunciation or the TTS pronunciation of the particular word associated with the highest confidence. The method also includes providing TTS audio that includes a synthesized speech representation of a response to the question using the user pronunciation or TTS pronunciation for the particular word.

Description

The present disclosure relates to instantaneous text-to-speech learning during interaction.

Users often interact with voice-enabled devices, such as smartphones, smart watches, and smart speakers, through digital assistants. These digital assistants provide interaction with users, allowing them to complete tasks and get answers to their questions through all-natural conversational interactions. Ideally, during an interaction between a user and a digital assistant, the user can communicate via verbal questions directed to his or her voice-enabled device running the digital assistant as if the user were speaking to another person. You should be able to interact as if you were there. The digital assistant would provide these verbal questions to an Automatic Speech Recognizer (ASR) system to process and recognize the verbal requests so that the action can be taken. Additionally, the digital assistant will also utilize text-to-speech (TTS) systems to convert textual representations of responses to questions into synthetic speech for audible output from the user's voice-enabled device. Often, there is lexical overlap between a verbal question and a corresponding TTS response during a digital assistant interaction, such that the user's pronunciation of the words in the verbal question, when output audibly as synthesized speech, is different from the question. different from the TTS pronunciation of the same word present in the digital assistant response.

"Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions", J. Shen et al., available at e.g. https://arxiv.org/abs/1712.05884

One aspect of the present disclosure, when executed on data processing hardware, causes the data processing hardware to determine whether a user pronunciation of a particular word or a text-to-speech pronunciation of a particular word is better used for text-to-speech audio. Provided is a computer-implemented method for performing an operation for selecting a more reliable object. The operations include receiving user pronunciations of particular words present in the question spoken by the user. The operations also include receiving a TTS pronunciation of the same particular word present in the text-to-speech (TTS) input. The TTS input includes textual representations of responses to questions, and the TTS pronunciation of a particular word is different from the user pronunciation of the particular word. The operations also include obtaining user pronunciation-related features associated with the user pronunciation of the particular word and obtaining TTS pronunciation-related features associated with the TTS pronunciation of the particular word. The operation determines, as output from a pronunciation decision model configured to receive user pronunciation-related features and TTS pronunciation-related features as input, the user pronunciation of a particular word or It also includes generating a pronunciation decision that selects one of the TTS pronunciations of a particular word. The operation uses either the user pronunciation for the particular word or the TTS pronunciation for the particular word selected by the pronunciation decision output from the pronunciation decision model for audible output from the user device associated with the user. and providing TTS audio that includes a synthesized speech representation of the response.

Implementations of this disclosure may include one or more of the following optional features. In some implementations, the operations include receiving audio data corresponding to a question spoken by a user and using an automatic speech recognizer (ASR) to process the audio data to generate a representation of the question. The method further includes the step of: In these implementations, receiving the user pronunciation of the particular word may include extracting the user pronunciation of the particular word from an intermittent state of the ASR while processing audio data using the ASR, or at least one of: extracting a user acoustic representation of a particular word from audio data that conveys a user pronunciation of the particular word; or processing the audio data to generate a user phonemic representation that conveys a user pronunciation of the particular word. include. The user pronunciation-related features associated with the user pronunciation of a particular word may also include one or more confidence features associated with the ASR recognizing the particular word in the audio data.

In some examples, the user pronunciation-related features that are associated with a user's pronunciation of a particular word are the user's geographic region when the question was spoken by the user, the linguistic demographic information associated with the user, the user's and/or the frequency with which the user pronunciation is used when pronouncing a particular word in a previous question that has been spoken by other users. Receiving a TTS pronunciation of a particular word comprises, as input to the TTS system, receiving TTS input containing a textual representation of the response to the question and, as output from the TTS system, an initial synthesis of the response to the question. generating an initial sample of TTS audio that includes a phonetic representation; and extracting a TTS acoustic representation of a particular word from the initial sample of TTS audio, the TTS acoustic representation conveying a TTS pronunciation of the particular word. may include.

Optionally, receiving the TTS pronunciation of the particular word may include processing the textual representation of the response to the question to generate a TTS phoneme representation conveying the TTS pronunciation of the particular word. In some instances, the TTS pronunciation-related features associated with the TTS pronunciation of a particular word are estimated using verified preferred pronunciations for the particular word or pronunciation mining from one or more auxiliary sources. pronunciation complexity, which indicates the likelihood of user mispronunciation of a particular word; Contains at least one of the following characteristics:

In some implementations, after generating a pronunciation decision that selects either the user pronunciation of a particular word or the TTS pronunciation of the particular word, the operation receiving explicit feedback from the user indicating whether the user prefers the user pronunciation of a particular word or the TTS pronunciation of the particular word; and creating a pronunciation decision model based on the explicit feedback from the user. and updating. Here, when explicit feedback from the user indicates that the user prefers the user pronunciation of a particular word, the behavior is to use the user pronunciation of the particular word when generating TTS audio containing the particular word. The method further includes the step of updating the TTS system to ensure that the TTS system is updated. In some examples, after providing TTS audio for audible output from the user device, the operation receives audio data corresponding to the next question spoken by the user or another user that includes the particular word. step and whether the user or another user pronounced the particular word in the next question the same way as either the user pronunciation for the particular word selected by the pronunciation determination or the TTS pronunciation for the particular word. The method further includes determining implicit user feedback and updating the pronunciation decision model based on the implicit user feedback.

Another aspect of the present disclosure is to have data processing hardware in communication with the data processing hardware and, when executed on the data processing hardware, cause the data processing hardware to specify a user pronunciation of a particular word or a particular and memory hardware storing instructions for performing operations for selecting which of the text-to-speech pronunciations of a word is more reliable for use in text-to-speech audio. The operations include receiving user pronunciations of particular words present in the question spoken by the user. The operations also include receiving a TTS pronunciation of the same particular word present in the text-to-speech (TTS) input. The TTS input includes textual representations of responses to questions, and the TTS pronunciation of a particular word is different from the user pronunciation of the particular word. The operations also include obtaining user pronunciation-related features associated with the user pronunciation of the particular word and obtaining TTS pronunciation-related features associated with the TTS pronunciation of the particular word. The operation determines, as output from a pronunciation decision model configured to receive user pronunciation-related features and TTS pronunciation-related features as input, the user pronunciation of a particular word or It also includes generating a pronunciation decision that selects one of the TTS pronunciations of a particular word. The operation uses either the user pronunciation for the particular word or the TTS pronunciation for the particular word selected by the pronunciation decision output from the pronunciation decision model for audible output from the user device associated with the user. and providing TTS audio containing a synthesized speech representation of the response.

This aspect may include one or more of the following optional features. In some implementations, the operations include receiving audio data corresponding to a question spoken by a user and processing the audio data to generate a representation of the question using an automatic speech recognizer (ASR). It further includes: In these implementations, receiving the user pronunciation of a particular word may involve extracting the user pronunciation of the particular word from an intermittent state of the ASR while processing audio data using ASR, or receiving the user pronunciation of the particular word extracting a user acoustic representation of a particular word from audio data that conveys a user pronunciation of the particular word; or processing the audio data to generate a user phonemic representation that conveys a user pronunciation of the particular word. include. The user pronunciation-related features associated with the user pronunciation of the particular word may include one or more confidence features associated with the ASR recognizing the particular word in the audio data.

In some examples, the user pronunciation-related features that are associated with a user's pronunciation of a particular word are the user's geographic region when the question was spoken by the user, the linguistic demographic information associated with the user, the user's and/or the frequency with which the user pronunciation is used when pronouncing a particular word in a previous question that has been spoken by other users. Receiving a TTS pronunciation of a particular word consists of receiving, as input to the TTS system, TTS input containing a textual representation of the response to the question, and as output from the TTS system, the initial synthesis of the response to the question. generating an initial sample of TTS audio that includes a phonetic representation; extracting a TTS acoustic representation of a specific word from the initial sample of TTS audio, the TTS acoustic representation conveying the TTS pronunciation of the specific word; and may include.

Optionally, receiving the TTS pronunciation of the particular word may include processing the textual representation of the response to the question to generate a TTS phoneme representation that conveys the TTS pronunciation of the particular word. In some instances, the TTS pronunciation-related features associated with the TTS pronunciation of a particular word are estimated using verified preferred pronunciations for the particular word or pronunciation mining from one or more auxiliary sources. pronunciation complexity, which indicates the likelihood of user mispronunciation of a particular word; Contains at least one of the following characteristics:

In some implementations, after generating a pronunciation decision that selects either the user pronunciation of a particular word or the TTS pronunciation of the particular word, the operation receives explicit feedback from users indicating whether they prefer the user pronunciation of a particular word or the TTS pronunciation of a particular word, and develops a pronunciation decision model based on explicit feedback from the user. and updating. Here, when explicit feedback from the user indicates that the user prefers the user pronunciation of a particular word, the behavior is to use the user pronunciation of the particular word when generating TTS audio containing the particular word. further including updating the TTS system to In some examples, after providing TTS audio for audible output from the user device, the operation receives audio data corresponding to the next question spoken by the user or another user that includes the particular word. and whether the user or other users pronounced the particular word in the next question the same way as either the user pronunciation for the particular word or the TTS pronunciation for the particular word selected by the pronunciation determination. The method further includes determining implicit user feedback and updating the pronunciation decision model based on the implicit user feedback.

The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims.

1 is a schematic diagram of an example audio environment; FIG. FIG. 2 is a schematic diagram of an example pronunciation decision model for determining whether a user pronunciation of a particular word or a text-to-speech pronunciation of a particular word is more reliable for use in text-to-speech audio. 1 is a schematic diagram of an example text-to-speech model for generating TTS audio; FIG. FIG. 2 is a schematic diagram of an example of a dialogue in which a user's pronunciation for a specific word is different and more reliable than the TTS pronunciation for the same word; 2 is a flowchart of an example arrangement of operations for a method of instantaneous text-to-speech learning during interaction between a user and a digital assistant. 1 is a schematic diagram of an example computing device that may be used to implement the systems and methods described herein.

Like reference symbols in the various drawings indicate similar elements.

Interactions between a user and a digital assistant, such as when a user issues a verbal question to the digital assistant to perform an action, and the digital assistant audibly outputs a text-to-speech (TTS) response related to the question. There is often lexical overlap between verbal questions and TTS responses. For example, a user can speak the voice search question "Where is Bexar?" and the corresponding audio data is automatically translated into the corresponding notation for interpretation by a natural language understanding (NLU) module. It is passed to a speech recognizer (ASR) and then provided as a search question to a search engine to retrieve the results. Using the retrieved results, the digital assistant may generate a TTS input that includes a textual representation of the response to the question. Here, the textual representation of the response would be "Bear is located in Texas," and this would be passed to the TTS system to generate corresponding TTS audio, including synthesized speech, for audible output from the device associated with the user. given to the TTS input.

In some cases, users pronounce words in a verbal question differently than the way a TTS system would pronounce the same words present in the textual representation of the response to the question provided as TTS input. do. Differences between competing pronunciations include the lack of normalization between ASR and TTS systems, which are generally trained separately and often become out of sync in terms of updates, and the lack of normalization between ASR and TTS systems, which are typically trained separately and often become out of sync in terms of updates; Unusual words/terms (e.g. contact names, neologisms, etc.) that the user has unknowingly mispronounced, the ASR system is robust and correctly recognizes and the TTS system pronounces correctly, and the user has intentionally mispronounced the words/terms This can be due to various reasons, such as words being mispronounced to induce the ASR system.

Example implementations herein identify when the user pronunciation of a term spoken by the user during a question differs from the TTS pronunciation of the same term present in response to the question, and use a pronunciation decision model to: It is directed to determining whether a user pronunciation or a TTS pronunciation should be used to pronounce a word in TTS audio that conveys a response to the user as a synthesized speech. As described in further detail below, the pronunciation decision model is trained and continuously updated to choose between competing pronunciations of the same word based on TTS pronunciation-related features and/or user pronunciation-related features. good. That is, based on the user pronunciation-related features associated with the user pronunciation of a particular word and the TTS pronunciation-related features associated with the TTS pronunciation of the particular word, the pronunciation decision model determines the confidence level of the user pronunciation of the particular word and the same identification. The reliability of the TTS pronunciation of the word may be estimated to determine whether the user pronunciation or the TTS pronunciation is more reliable.

Referring to FIG. 1, in some implementations, a voice environment 100 includes a user 10 speaking a question 12 to a voice-enabled device 110 (also referred to as device 110 or user device 110). Specifically, the user 10 is interacting with a digital assistant 115 (also referred to as a "digital assistant interface") running on the user device 110, and a question 12 spoken by the user 10 causes the digital assistant to Request 115 to perform an action. User 10 (ie, the speaker of question 12) may speak question 12 and request a response from digital assistant 115 or cause digital assistant 115 to perform a task specified by question 12. Device 110 is configured to capture sound from one or more users 10 within audio environment 100. A voice-enabled system (e.g., digital assistant 115) of or associated with device 110 responds to question 12, performs one or more actions specified by question 12, and outputs an audible output from device 110. The response to the question may be provided as a synthesized voice 154 for the purpose.

Here, device 110 captures audio data 14 corresponding to verbal questions 12 by user 10. Device 110 may represent any computing device associated with user 10 and capable of receiving audio data 14. Some examples of user devices 110 include mobile devices (e.g., mobile phones, tablets, laptops, e-book readers, etc.), computers, wearable devices (e.g., smart watches), music players, casting devices, smart home appliances (e.g., Examples include, but are not limited to, smart TVs) and Internet of Things (IoT) devices, remote controls, smart speakers, etc. The device 110 is in communication with the data processing hardware 112 and the data processing hardware 112 and, when executed by the data processing hardware 112, causes the data processing hardware 112 to receive information related to voice and/or text processing. It includes memory hardware 114 that stores instructions for performing one or more operations. In some examples, device 110 includes one or more applications (i.e., software applications), each application associated with one or more audio processing systems/models 140, 150, 200. can be used to perform various functions within the application. For example, device 110 may run digital assistant 115 that is configured to communicate synthetically played audio 154 (also referred to as synthetic speech 154) to user 10 to converse with user 10 and perform various tasks. Assist.

Device 110 communicates an audio capture device (e.g., a microphone) 116 and an audible audio signal (e.g., a synthesized playback signal 154 from device 110) for capturing and converting audio data 14 within audio environment 100 into an electrical signal. It further includes an audio subsystem with an audio output device (eg, speaker) 118 for playing. While in the illustrated example device 110 implements a single audio capture device 116, device 110 may implement an array of audio capture devices 116 without departing from the scope of this disclosure, thereby providing an array of One or more audio capture devices 116 within may not physically reside on device 110, but may be in communication with an audio subsystem (eg, a peripheral of device 110). For example, device 110 may correspond to a vehicle infotainment system that utilizes an array of microphones located throughout the vehicle. Similarly, audio output device 118 may include one or more speakers residing on or in communication with device 110, or in combination with one or more speakers residing on device 110 and one One or more other speakers may be physically removed from device 110 but in communication with device 110.

Further, device 110 may be configured to communicate with remote system 130 via network 120. Remote system 130 may include remote resources 132, such as remote data processing hardware 134 (eg, a remote server or CPU) and/or remote memory hardware 136 (eg, remote database or other storage hardware). Device 110 may utilize remote resources 132 to perform various functionality related to audio processing and/or synthetic playback transmission. For example, device 110 is configured to perform speech recognition using automatic speech recognition (ASR) system 140 and/or text-to-speech conversion using text-to-speech (TTS) system 150. . Additionally, pronunciation decision model 200 determines between user pronunciations 202 or TTS pronunciations 204 for particular words that are present in question 12 spoken by user 10 as well as in TTS input 152 corresponding to the textual representation of the response to question 12. may generate a pronunciation determination 250 to be selected. TTS system 150 creates TTS audio 154 that includes a synthesized speech representation of the response to question 12, whereby TTS audio 154 is a user pronunciation 202 or TTS Pronounce certain words using one of the pronunciations 204.

These systems/models 140, 150, 200 may reside on the device 110 (referred to as on-device systems) or may reside remotely (e.g., reside on a remote system 130) in communication with the device 110. good. In some examples, some of these systems 140, 150, 200 reside locally or on-device while others reside remotely. In other words, any of these systems 140, 150, 200 may be local, remote, or both in any combination. For example, systems 140, 150, 200 may reside on remote system 130 if the size or processing requirements of systems 140, 150, 200 are significant. Additionally, if device 110 may support the size or processing requirements of one or more systems 140, 150, 200, one or more systems 140, 150, 200 may include data processing hardware 112 and/or memory hardware. 114 may be used to reside on the device 110. Optionally, one or more of systems 140, 150, 200 may reside locally/on-device or remotely. For example, when a connection to network 120 between device 110 and remote system 130 is available, one or more of systems 140, 150, 200 may be running on remote system 130 by default; or network 120 is unavailable, systems 140, 150, 200 instead run locally on device 110.

In the illustrated example, ASR system 140 receives as input audio data 14 corresponding to question 12 and processes audio data 14 to produce a representation 142 of question 12 as output. ASR system 140 may include natural language understanding (NLU) functionality to perform query interpretation (eg, semantic analysis) on notation 142. Notation 142 is represented by a sequence of texts that digital assistant 115 may use to generate a response to question 12, and more particularly to generate a TTS input 152 that corresponds to a textual representation of the response to question 12. contains grapheme representations. For example, continuing the example from before, user 10 can speak the question "Where is Bexar?" which is captured by ASR system 140 which generates the corresponding notation 142. It is converted into corresponding audio data 14 for processing. The standard pronunciation for the word Bexar in the Texas context is "

”, whereas user 10 unknowingly spelled the word Bexar based on its English spelling by pronouncing the “X” clearly.

'' was mispronounced. In particular, the ASR system 140 is robust and accurately recognizes user mispronunciations of the word Bexar as a city in Texas.

After generating notation 142 (and performing any NLU functionality), digital assistant 115 may then use notation 142 to determine a response to question 12. For example, to determine a location relative to a city name (e.g., Bexar), the digital assistant 115 may request the search engine to display the notation 142 (“Where is Bexar?”) or the identifying portions of the notation (e.g., “where” and “ You can pass a search string containing "Bexar"). The search engine may return one or more search results that the digital assistant 115 then interprets to generate a TTS input 152 that includes a textual representation of the response to the question 12. In particular, the word "Bexar" is present in both the representation 142 of question 12 and the TTS input 152, such that both the representation 142 and the textual representation of the TTS input 152 share the same grapheme representation of the word "Bexar". A textual representation may include a sequence of graphemes/characters of a particular natural language. The grapheme/character series may include letters, numbers, punctuation marks and/or other special characters.

TTS system 150 may convert TTS input 152 to corresponding TTS audio 154 (eg, synthesized speech) that device 110 audibly outputs to communicate the response to question 12 to user 10 . Prior to audible output of TTS audio 154, TTS system 150 and/or pronunciation decision model 200 determines what the user pronunciation 202 of a particular word present in question 12 is the TTS pronunciation 204 of the same particular word present in TTS input 152. You can distinguish between different things. In the illustrated example, the user pronunciation 202 of the word Bexar is “

” while the TTS pronunciation 204 of the word Bexar is “

”. In some implementations, pronunciation decision model 200 includes user pronunciation-related features (UPF) 210 associated with user pronunciation 202 of a particular word and TTS pronunciation-related features (TTSPF) 230 associated with TTS pronunciation 204 of a particular word. get. Although a number of different UPFs 210 and TTSPFs 230 may be obtained and form the basis for generating pronunciation decisions 250, as described in more detail below with reference to FIG. 2, as depicted in the present example in FIG. In addition to UPF210, which includes user statistics showing that users frequently mispronounce Bexar, Pronunciation Decision Model 200 also shows that Bexar is widely used in the Texas context.

I get TTSPF230 indicating that it is pronounced as ``. Therefore, the pronunciation decision model 200 estimates against the TTS pronunciation 204 of Bexar due to the fact that users commonly mispronounce Bexar and the TTS pronunciation 204 is widely adopted when spoken in a Texas context. The reliability of Bexar's user pronunciation 202 may be estimated to be lower than the reliability determined by Bexar. In particular, " for the word Bexar present in the textual representation of the TTS response.

The TTS pronunciation 204 of `` is not verified, but instead is learned by the TTS system 150 through pronunciation mining from one or more auxiliary sources. For example, auxiliary information sources may include audio and/or video web sources that may be mined for pronunciations of various terms in various contexts for use by TTS system 150.

When the TTS system 150 generates TTS audio 154 from the TTS input 152, the TTS output 154 includes a textual representation of the response to the question 12. Contains synthesized speech that approximates how it would be played. Continuing with the example, pronunciation decision 250 selects a TTS pronunciation 204 for a particular word Bexar, maintains the use of TTS pronunciation 204 in TTS audio 154, and maintains the use of TTS pronunciation 204 in TTS audio 154 for audible output from user device 110. 154 will be provided. Therefore, TTS audio 154 is audibly output as a synthesized speech representation of the response to question 12, and for the textual representation of the word Bexar present in TTS input 152, "

” using TTS pronunciation 204.

In some examples, after providing TTS audio 154 for audible output using a selected one of user pronunciation 202 or TTS pronunciation 204 for a particular word, pronunciation decision model 200 and/or TTS system 150 to provide explicit feedback to user 10 indicating whether user 10 agrees with pronunciation decision 250 or instead prefers the other of user pronunciation 202 or TTS pronunciation 204 that was not selected in pronunciation decision 250. prompt. More specifically, explicit feedback indicating a preferred pronunciation for a particular word is provided by the pronunciation decision model 200 as a TTSPF 230 when making the next pronunciation decision 250 against competing user and TTS pronunciations of a particular word. May be used. User 10 determines when model 200 estimates confidence for users and TTS pronunciations 202, 204 that are close (e.g., within a threshold confidence range) and/or that confidence satisfies a confidence threshold. be encouraged to provide explicit feedback when they are unable to do so. Explicit feedback may be used to update pronunciation decision model 200 and/or TTS system 150. Prior to updating pronunciation decision model 200 and/or TTS system 150, verifying that the preferred pronunciation for a particular word is a reasonable variant and/or that explicit feedback is not an adversarial user correction. Additional verification steps may be performed to confirm.

In an alternative implementation of the above example, TTSPF230 would write "

” indicates that user pronunciation 202 of System 150 will modify TTS audio 154 to adopt user pronunciation 202. In this example implementation, if the TTS system 150 is a user-specific local/custom TTS system 150, then the TTSPF 230 has the preferred pronunciation for the word Bexar.

As a result of the user pronunciation 202 of "

The TTS system 150 may be retrained to generate the TTS audio 154 by pronouncing it as ``.''.

TTS system 150 may include any type of TTS system capable of converting input text (eg, TTS input) to a corresponding synthesized speech representation as TTS audio 154. For example, the TTS system 150 may utilize a parametric TTS model or a TTS model 300 (see FIG. TTS model) may be used. In some implementations, the TTS model processes embeddings that are encoded representations of audio features (e.g., features of the TTS input 152) to represent the audio waveform (e.g., the audio signal over time) that represents the TTS audio 154. generate a time-domain audio waveform (time-domain audio waveform) that determines the amplitude of Once generated, TTS system 150 communicates TTS audio 154 to device 110 so that device 110 can output TTS audio 154 as a synthesized speech representation of the response to question 12. For example, device 110 is selected from one or more speakers 118 by pronunciation determination 250 output from model 200.

outputs audible TTS audio 154 of "Bexar is in Texas" using TTS pronunciation 204 of "Bexar is in Texas". Here, TTS model 300 of TTS system 150 is configured to control speech-related attributes of synthesized speech 154. In other words, the TTS model 300 is configured to simulate the voice of a human speaker in terms of naturalness, while also being able to generate diverse synthetic speech by modeling detailed latent features. . Although FIG. 1 depicts an example of a TTS system 150 in the context of an application to a digital assistant 115, the TTS system 150 (e.g., using a TTS model 300) may also be used for other text-to-text applications, such as voice search, navigation, or document reading. It can be applied to voice scenarios.

In some implementations, if the TTS model 300 associated with the TTS system 150 resides on the user device 110 but is trained as a global model that is shared across users within a geographic region, the model 300 may be updated/replayed. Federated learning techniques are used to train. For example, the user device 110 retrains/updates the parameters of an on-device version of the TTS model 300 that it is running locally, and then transfers the updated parameters or training losses to the server for updating the global TTS model 300. May be shared. In doing so, the Global TTS Model 300 can receive data from multiple users without requiring users across the user population to share their audio data, question content, or other information that users do not wish to share. May be retrained/updated using parameter updates/training losses.

Referring to FIG. 3, in some examples, a TTS model 300 includes an encoder-decoder network architecture having an encoder 302 and a decoder 304. In some implementations, the encoder-decoder 302, 304 structure corresponds to a series-to-sequence recurrent neural network (RNN) in Tacotron 2 (e.g., Shen, Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions, https:// available at arxiv.org/pdf/1712.05884.pdf, incorporated herein by reference). In some configurations, encoder 302 receives as input a TTS input 152 or an embedding (e.g., a character embedding) corresponding to TTS input 152, and as output each Mel frequency that decoder 304 is to subsequently produce. It is configured to generate a context vector Vc for the spectrogram. The context vector Vc is of fixed length and may define features that appear at particular positions that generally correspond to sequences of characters forming the textual representation of the TTS input 152. In some configurations, TTS input 152 includes a grapheme sequence that is first converted (e.g., via a normalization engine, such as a grapheme-phoneme model) into a corresponding phoneme sequence prior to being input to encoder 302. .

Encoder 302 may include one or more convolutional layers followed by a bidirectional long short-term memory (LTSM) layer. Neurons in each convolutional layer may receive input from a small subset of neurons in previous layers. In this regard, neuronal connectivity allows the convolutional layer to learn filters that activate when certain hidden features appear at positions in the sequence of characters corresponding to the textual representation of the TTS input 152. In some implementations, the filter in each convolutional layer may span a series of characters (eg, 4, 5, or 6 characters). Each convolutional layer may be followed by a batch normalization and rectification linear unit (RELU). If encoder 302 includes one or more convolutional layers, bidirectional LSTM layers may follow these convolutional layers. Here, the bidirectional LSTM is configured to process the hidden features generated by the final convolutional layer to generate a sequential feature representation of the sequence of characters corresponding to the TTS input 152. The sequential feature representation may include a sequence of feature vectors.

In some implementations, encoder 302 is configured to receive sequential feature representations from encoder 302 and to process the sequential feature representations to generate a context vector Vc for each decoder output step. Including networks. That is, the attention network can generate a fixed length context vector Vc for each frame of the Mel frequency spectrogram that decoder 304 will subsequently generate. A frame refers to a unit of mel frequency spectrogram based on a small portion (eg, 10 millisecond samples) of the input signal. The architecture of the attention network may vary depending on the particular TTS system 150. Some examples of attention networks include additive attention networks, location-sensitive attention networks, Gaussian mixture model (GMM) attention networks (e.g., to improve generalization to long utterances), forward attention networks, and stepwise monotone attention networks. or involving a dynamic convolutional attention network. The attention network allows the TTS model 300 to generate an output sequence (e.g., an output log mel spectrogram frame of sequence).

The decoder 304 receives an output audio signal AS (e.g., configured as a neural network (eg, an autoregressive recurrent neural network) to generate an output sequence (mel frequency spectrogram). For example, based on the context vector V _C , decoder 304 predicts a representation of the audio signal (eg, a mel frame or a spectrogram frame) from the encoded representation produced by encoder 302. That is, the decoder 304 is configured to receive as input one or more context vectors _VC , and for each context vector _VC , the mel frequency spectrogram is a frequency domain representation of the sound. may generate corresponding frames. In some examples, decoder 304 includes an architecture similar to Tacotron 2. In other words, decoder 304 may include an architecture with prinets, long short-term memory (LSTM) subnetworks, linear projections, and convolutional postnets.

In some configurations, TTS model 300 also includes a speech synthesizer 306 (also referred to as synthesizer 306). Synthesizer 306 can be any network configured to receive the Mel frequency spectrogram and to generate output samples of TTS audio 154 based on the Mel frequency spectrogram as synthesized speech. In some other implementations, synthesizer 306 includes a vocoder. For example, speech synthesizer 306 may include a WaveRNN vocoder (e.g., "Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions", J. Shen et al., available at, e.g., https://arxiv.org/abs/1712.05884). possible, as described by). Here, the WaveRNN vocoder may generate a 16-bit signal sampled at 24 kHz subject to the spectrogram predicted by the TTS model 300. In some other implementations, synthesizer 306 is a trainable spectrogram waveform inverter. After synthesizer 306 generates the waveform, an audio subsystem uses the waveform to generate TTS audio 154 and provides TTS audio 154 for audible playback (e.g., on device 110) as synthesized speech. Or the generated waveform can be provided to another system so that the other system can generate and play TTS audio 154. Generally speaking, synthesizer 306 has little or no influence on the resulting pronunciation, prosody, and/or style of synthesized speech 154; It only affects the audio fidelity of TTS Audio 154 because it converts the mel frames or spectrogram frames output by TTS Audio 154 into waveforms.

FIG. 2 shows an example pronunciation decision model that generates a pronunciation decision 250 that selects whether a user pronunciation 202 of a particular word or a different TTS pronunciation 204 of the same particular word is more reliable for use in TTS audio 154. 200 schematic diagrams are shown. Pronunciation decision model 200 may be executed on user device 110, remote system 130, or a combination thereof. In some implementations, the pronunciation decision model 200 is configured to make pronunciation decisions 250 specific to the user 10 based on learning from interactions/interactions between the user 10 and the digital assistant 115. It is user-specific in that it is customized. In other implementations, the pronunciation decision model 200 is shared by multiple users and instantly learns to base the pronunciation decisions 250 on interactions that all users 10 have with their digital assistants 115.

Pronunciation decision model 200 may receive user pronunciation 202 for a particular word using various techniques. For example, the user pronunciation 202 of a particular word may be extracted from an intermittent state of the ASR system 140 while the ASR system 140 is processing audio data 14 corresponding to the question 12 spoken by the user. In another implementation, the model 200 receives a user pronunciation 202 of a particular word by extracting the user acoustic representation of the particular word from audio data. Here, the user acoustic representation conveys the user pronunciation 202 for a particular word. In yet another implementation, audio data 14 corresponding to question 12 is processed to generate a user phoneme representation that conveys the user's pronunciation of the particular word. The presented disclosure is not limited to any specific technique for obtaining user pronunciations of words.

Pronunciation decision model 200 may receive TTS pronunciations 204 for particular words using various techniques. In one example, pronunciation decision model 200 receives a phoneme representation that conveys a TTS pronunciation 204 of a particular word. For example, TTS system 150 may include a grapheme-phoneme model (eg, the normalization engine of FIG. 3) that converts grapheme representations of TTS input 152 into corresponding phoneme representations. In another example, a TTS acoustic representation (eg, a waveform) of a particular word is extracted from an initial sample of TTS audio produced (but not audibly output) by the TTS system 150. Here, the acoustic representation conveys the TTS pronunciation 204 of a particular word. A TTS acoustic representation may also be extracted from the audio signal As (eg, an output sequence Mel frequency spectrogram) generated as an output from the decoder 304 of the TTS model 300 of FIG. In yet another example, TTS pronunciation 204 is extracted from an intermittent state of TTS system 150 while processing a textual representation of TTS input 152 for conversion to speech. For example, the extracted intermittent state of the TTS system 150 may include the sequence context vector Vc output from the encoder 302 of the TTS model 300 of FIG.

For a particular word that is present both in question 12 spoken by user 10 and in TTS input 152 that includes a textual representation of the response to question 12, TTS system 150 or pronunciation decision model 200 determines the user's pronunciation of the particular word. 202 and TTS pronunciation 204 differ from each other and therefore which one will ultimately be used to pronounce a particular word within the TTS audio 154 provided for audible output as synthesized speech. May identify when there is conflict regarding. Pronunciation decision model 200 generates pronunciation decisions 250 based on one or more user pronunciation-related features (UPF) 210 and/or one or more TTS pronunciation-related features (TTSPF) 230 associated with a particular word. be trained as such. Here, pronunciation decision 250 selects one of user pronunciation 202 or TTS pronunciation 204 associated with the highest degree of confidence to use to pronounce the particular word in TTS audio 154 that conveys the response to question 12. In some examples, pronunciation decision model 200 includes user pronunciation confidence estimator 220 and TTS pronunciation estimator 240. In these examples, the user pronunciation confidence estimator 220 estimates the user pronunciation confidence 222 for the user pronunciation 202 of a particular word indicating the likelihood that the user pronunciation 202 is preferred. On the other hand, the TTS pronunciation estimator 240 estimates a TTS pronunciation confidence level 242 for the TTS pronunciation 204 of a particular word indicating the possibility that the TTS pronunciation 204 is preferred. In some implementations, pronunciation decision model 200 generates pronunciation decision 250 as a log-likelihood ratio between user pronunciation confidence 222 and TTS pronunciation confidence 242.

The UPF 210 associated with the user pronunciation 202 of a particular word includes a pronunciation decision model 200 in ascertaining the reliability (or unreliability) associated with the pronunciation of the particular word when spoken by the user 10 during question 12. may include many different features that convey information useful to the user. For example, the representation 142 of question 12 may convey the context of a particular word in light of the word type of the particular word and other words recognized within the representation. Additionally, linguistic demographic information associated with user 10 may convey the user's proficiency in a particular language. For example, a user 10 who speaks American English as a native language may be more likely to mispronounce the name of an Italian restaurant, whereas a bilingual English and Italian speaker is less likely to mispronounce Italian names/words. . The UPF may provide any type of demographic information associated with the user 10, such as age and gender. The geographic region in which user 10 is located when question 12 is spoken may serve as a UPF 210 that is associated with user pronunciation 202 of a particular word. For example, in question 12 for navigation instructions to street addresses with unusual names, the TTS system 150 may have a tendency to mispronounce unusual names. Therefore, the UPF 210, which identifies that the user's geographic region is close to the requested street address with the unusual name, serves as a strong indicator that the user pronunciation 202 of the unusual name may be more reliable than the TTS pronunciation 204 of the unusual name. can do. A user may explicitly grant access to a user's geographic region, and may deny access to a geographic region at any time. In this scenario, combined with the user 10's geographic region provided by UPF 210 and TTSPF 230, the location/dialect in which the TTS system 150 is trained may be outside the geographic region associated with the requested street address. , thereby further increasing the user pronunciation reliability 222 and/or lowering the TTS pronunciation reliability 242.

Additionally, historical information providing the frequency of using the user pronunciation 202 when pronouncing particular words in previous questions spoken by the user and/or other users is provided as input to the pronunciation decision model 200. Can serve as UPF210. For example, multiple users across a geographic region often use the same user pronunciation 202 for a particular word, resulting in a nascent pronunciation and/or unknown dialect that the TTS system 150 and/or ASR system 140 has not been trained on. can indicate change. In addition to improving the confidence pronunciation decisions 250 produced by the model 200, the TTS system 150 is updated to learn to create TTS audio 154 that reflects the user pronunciation 202 of a particular word. Knowledge of emerging pronunciations and/or unknown dialect variations in pronouncing particular words may be used.

In some examples, by using the user pronunciation 202 of a particular word in a previous question 12 to establish a threshold frequency for user 10, the implied user indicates the user pronunciation 202 as the preferred pronunciation of the particular word. Serves as feedback. For example, in one or more previous interaction sessions in which decision model 200 selected TTS pronunciation 204 over a different user pronunciation 202 to pronounce a particular word in TTS audio 154, user 10 and/or other users Subsequent questions 12 that continue to use user pronunciation 202 to pronounce a particular word may indicate that user 10 and/or other users prefer user pronunciation 202 over TTS pronunciation 204. This implicit feedback may be further used to train/update the TTS system 150.

In some additional implementations, the UPF 210 associated with a user pronunciation 202 of a particular word includes one or more ASR confidences indicating the likelihood that recognition of the particular word by the ASR system 140 in the audio data 14 is correct. Contains degree features. Pronunciation decision model 200 may obtain ASR confidence features directly from ASR system 140 used to generate representation 142 of question 12. ASR confidence features include, without limitation, a likelihood/posterior score associated with recognizing a particular word, a confidence score indicating the probability that a particular word was correctly recognized (ASR system 140 or (obtained by a confidence model), and any recognition confusability associated with recognizing the word. Here, recognition confusability may be ascertained from the entropy of the posterior word lattice of candidate recognition results for the notation 142 and/or the posterior distribution of possible recognition results of the neural network model of the ASR system 140.

In one example, following a previous misrecognition of a particular word by ASR system 140 when pronounced using TTS pronunciation 204, a subsequent recognition of a particular word instead pronouncing the word using user pronunciation 202 A success may indicate that the user pronunciation 202 is unreliable because the user 10 may have chosen the user pronunciation 202 simply to fool the ASR system 140 and avoid false recognition errors. Such knowledge may be used to update the ASR system 140 to be robust in recognizing TTS pronunciations 204 of particular words. Additionally or alternatively, a previous misrecognition of a particular word by the ASR 140 when pronounced using the user pronunciation 202 may result in a subsequent recognition of the particular word pronounced using the TTS pronunciation 204 instead. The knowledge that the ASR system 140 has been followed may be used to update the ASR system 140 to be robust in recognizing user pronunciations 202.

Similar to UPF 210, TTSPF 230 associated with TTS pronunciation 204 of a particular word includes the reliability for using TTS pronunciation 204 to pronounce a particular word present in TTS input 152 received by TTS system 150. may include a number of different features that convey information useful to pronunciation decision model 200 in ascertaining (or unreliability). For example, a TTS input 152 containing a textual representation of a response to question 12 may convey the context of a particular word in light of the word type of the particular word and other words present in the TTS input 152. Additionally, the geographic region/dialect in which TTS system 150 is trained may be provided to pronunciation decision model 200 as TTSPF 230.

In some examples, TTSPF 230 indicates whether TTS pronunciation 204 includes a verified preferred pronunciation for a particular word. The TTS pronunciation 204 that is confirmed as a preferred pronunciation increases the TTS pronunciation reliability 242 estimated by the TTS pronunciation reliability estimator 240 with respect to the TTS pronunciation 204 and increases the TTS pronunciation reliability 242 estimated by the TTS pronunciation reliability estimator 220 with respect to the user pronunciation 202. It can serve as a strong indicator for reducing the user pronunciation reliability 222 estimated by . Preferred pronunciations for particular words are manually verified during training of TTS system 150 by training TTS system 150 on training sample pairs that map audio of preferred pronunciations of particular words with corresponding grapheme representations. good. For example, if user 10, who does not speak Swedish as a native language, is traveling abroad in Sweden and speaks question 12 for directions to a city in Sweden, but mispronounces the city name, TTS system 150 responds to By indicating that the TTS pronunciation 204 of the city name has been trained and verified, the TTSPF 230 increases the confidence that the TTS pronunciation 204 is more reliable than the user pronunciation 202.

The preferred pronunciation comprises the user 10 providing a preferred pronunciation for a word specified by the user and configuring the TTS system 150 to use the preferred pronunciation to pronounce the word in the TTS audio 154. The preferred pronunciation of confirmation may also be included. This scenario is common for pronouncing custom or contact names that may have a unique pronunciation that the TTS system 150 would otherwise pronounce differently based solely on the graphemic representation of the name. Similarly, ASR system 140 may have difficulty accurately recognizing these contact names due to their unique pronunciation. To provide a preferred pronunciation for a particular word (e.g., a name or other proper noun), a user may speak the word using the preferred pronunciation, and the TTS system 150 translates the preferred pronunciation into the corresponding grapheme. may be mapped to a representation such that the TTS system 150 pronounces the word using the preferred pronunciation when the corresponding grapheme representation is present in the TTS input 152.

In other examples, user 10 provides a preferred pronunciation for a particular word via explicit feedback during a previous interaction session with digital assistant 115. In these examples, the audible output of TTS audio 154 conveying a response to a question, in which certain words are pronounced differently than the way the words were pronounced during the question spoken by user 10, is followed by digital assistant 115. may prompt the user 10 to provide explicit feedback to indicate which pronunciation the user prefers to pronounce the word in the next TTS audio 154. In another scenario, without being prompted, the user 10 may say that if he is not satisfied with the pronunciation of a particular word in the TTS audio 154, he can say "I didn't say it correctly" or "[use the preferred pronunciation]?" Explicit feedback may be provided by issuing a response such as ``. The fact that TTSPF 230 indicates that the TTS pronunciation 204 of a particular word includes the user-confirmed preferred pronunciation may explain why the user consciously chooses to use a different user pronunciation 202 of the particular word when speaking Question 12. Seemingly inconsistent with the view, user 10 intentionally mispronounces certain words, or at least pronounces them inconsistently with the pronunciation preferred by user 10, in an attempt to deceive ASR system 140 and avoid false recognition errors. It is possible.

In some implementations, TTSPF 230 indicates whether TTS pronunciation 204 includes an unconfirmed pronunciation for a particular word. Unconfirmed pronunciations may increase confidence in the reliability of TTS pronunciations, but to a lesser extent than confirmed pronunciations. The TTS pronunciation 204 for a particular word may be unconfirmed if the pronunciation is learned/estimated by the TTS system 150 via pronunciation mining from one or more auxiliary sources. For example, in the example above, the TTS system 150 uses the TTS pronunciation 204 for the word/term Bexar as the standard pronunciation in the Texas context.

' may be learned as unconfirmed pronunciations that are correctly estimated using pronunciation mining from audio and/or video sources.

Additionally or alternatively, other TTSPFs 230 that are input to the pronunciation decision model 200 for making pronunciation decisions 250 include, without limitation, pronunciation variants and pronunciation complexity features. A phonetic variant feature may indicate whether there are any other variants for pronouncing a particular word. These variants of pronunciation of a particular word may be learned by pronunciation decision model 200 over time. Different pronunciations may be mapped to different contexts. For example, a majority of speakers may use a particular pronunciation of a particular word, while speakers in a particular geographic area may pronounce the same word exclusively differently. On the other hand, the pronunciation complexity feature may indicate the likelihood of user mispronunciation of a particular word. The TTS system 150 may provide pronunciation complexity features that indicate a strong likelihood of user mispronunciation by identifying specific phoneme sequences/expressions that may be found difficult for the user to pronounce. TTSPF 230 may further show mispronunciation statistics indicating how often the user population mispronounces particular words in the question.

FIG. 4 shows a user pronunciation 202 for a particular word present in question 12 that is different from the TTS pronunciation 204 of the same word present in the TTS input 152 conveying the response to question 12 performed on user 10 and user device 110. provides a schematic diagram 400 of an example interaction between a digital assistant 115 and a digital assistant 115. In the illustrated example, user 10 speaks question 12 directed to digital assistant 115, "Play songs by A$AP Rocky." In this example, the user pronunciation 202 of the word "A$AP" is A-S-A-P, and the ASR system 140 is robust and correctly recognizes the user pronunciation 202 as "A$AP." Here, ASR system 140 generates representation 142 of question 12 and applies NLU functionality to cause digital assistant 115 to cause music player application to audibly output a musical song by an artist named A$AP Rocky. Interpret question 12 as a request. Digital assistant 115 also generates a response to question 12 that confirms that digital assistant 115 understands question 12 and is in the process of fulfilling question 12. Here, the digital assistant 115 generates a text representation of "Playing latest songs by A$AP Rocky" in response to question 12 and for conversion to TTS audio 154. is provided as a TTS input 152 to the TTS system 150.

However, in this scenario, the TTS system 150 does not know how to pronounce the word "A$AP" that is present in the TTS input 152, and therefore there is no TTS pronunciation 204 for the word "A$AP." The pronunciation decision model 200 is based on the UPF 210 that includes one or more ASR confidence features indicating that it has a strong confidence in recognizing the word A$AP pronounced using the user pronunciation 202. A pronunciation decision 250 may be generated that concludes that the pronunciation confidence level 222 is high and therefore selects the user pronunciation 202 of A-S-A-P to pronounce the word A$AP in the TTS audio 154. In the illustrated example, TTS audio 154 is audibly output from user device 110 and includes a synthesized speech representation of a response to question 12 that uses user pronunciation 202 to pronounce the word "A$AP."

If the pronunciation decision 250 is associated with a confidence level that does not meet a certain threshold, the model 200 and/or the TTS system 150 determines that there may be some doubt about using the user pronunciation 202 and therefore the user User 10 may be prompted to indicate whether they agree with pronunciation decision 250 or prefer a different pronunciation. The user's response to the prompt may include explicit feedback as described above. The prompt may be an audible prompt in which the TTS system 150 outputs a synthesized voice asking the user 10 whether he or she agrees with the pronunciation decision or prefers a different pronunciation. The prompt may additionally or alternatively include a visual prompt that displays a notification on a screen in communication with user device 110. Here, the user may provide user input instructions to select a graphical element that indicates a preferred pronunciation or, at a minimum, satisfaction or dissatisfaction with the pronunciation decision 250.

If pronunciation determination 250 selects user pronunciation 202 as more reliable than TTS pronunciation 204 (or TTS pronunciation 204 is not available), then TTS system 150 is unable to create TTS audio 154, or TTS audio 154 is There may be scenarios where the user pronunciation 202 is used to pronounce a word that would otherwise contain synthetic quality that does not meet the quality criteria. This scenario may occur when the user pronunciation 202 of a particular word is spoken in a different accent/dialect than the accent/dialect in which the TTS system 150 (and associated TTS model 300) was trained. TTS system 150 may utilize a variety of different techniques to create TTS audio 154 using user pronunciations 202 on the fly. In some examples, TTS system 150 uses acoustic representations of particular words extracted from audio data 14 that correspond to questions 12 spoken by the user. In these examples, acoustic representations of particular words extracted from audio data 14 may be inserted into TTS audio 154. On the other hand, TTS system 150 may further extract phonemic representations from the acoustic representation of the particular word and use the phonemic representation to create TTS audio 154 with the user pronunciation 202 of the particular word.

In an additional example, TTS system 150 obtains a latent representation derived from a portion of audio data 14 corresponding to verbal question 12 that includes user pronunciation 202 of a particular word. In these examples, the latent representation directs the TTS system 150 (and associated TTS model) to create TTS audio 154 that pronounces the particular word using the user pronunciation 202. In yet another example, speech conversion technology is applied to a portion of the audio data 14 corresponding to the oral question 12 that includes the user pronunciation 202 to create a TTS that uses the user pronunciation 202 to pronounce a particular word in a synthesized voice. Create audio 154.

A software application (ie, software resource) may refer to computer software that causes a computing device to perform a task. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

Non-transitory memory may be a physical device used to temporarily or permanently store programs (eg, sequences of instructions) or data (eg, program state information) for use by a computing device. Non-transitory memory may be volatile and/or non-volatile addressable semiconductor memory. Examples of non-volatile memory include flash memory and read-only memory (ROM)/programmable read-only memory (PROM)/erasable programmable read-only memory (EPROM)/electronically erasable programmable read-only memory (EEPROM) (e.g. (typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), as well as disk or tape. Not done.

FIG. 5 shows an arrangement of operations for a method 500 of selecting either a user pronunciation 202 or a different TTS pronunciation 204 of a word that is present in both the verbal question 12 and the TTS input 152 containing a textual representation of the response to question 12. 1 is an example flowchart. At act 502, method 500 includes receiving a user pronunciation 202 of a particular word present in question 12 spoken by user 10. At act 504, method 500 includes receiving a TTS pronunciation 204 of the same particular word present in TTS input 152. TTS input 152 includes a textual representation of the response to question 12, and the TTS pronunciation 204 of the particular word differs from the user pronunciation 202 of the particular word. At act 506, method 500 includes obtaining user pronunciation-related features 210 that are associated with user pronunciation 202 of the particular word. At act 508, method 500 includes obtaining TTS pronunciation-related features 230 that are associated with TTS pronunciation 204 of the particular word. At operation 510, method 500 determines the highest confidence level for use in TTS audio 154 as an output from pronunciation decision model 200 configured to receive user pronunciation-related features 210 and TTS pronunciation-related features 230 as input. The method includes generating a pronunciation decision 250 that selects either an associated user pronunciation 202 of the particular word or a TTS pronunciation 204 of the particular word. At act 512, the method 500 determines the user pronunciation 202 for the particular word or the particular word selected by the pronunciation determination 250 output from the pronunciation determination model 200 for audible output from the user device 110 associated with the user 10. and providing TTS audio 154 containing a synthesized speech representation of the response to question 12 using one of the TTS pronunciations 204 for the question.

FIG. 6 is a schematic diagram of an example computing device 600 (eg, system 600) that may be used to implement the systems and methods described in this document. Computing device 600 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components illustrated herein, their connections and relationships, and their functions are meant to be illustrative only and are not meant to limit the example implementations of the invention described and/or claimed herein. .

Computing device 600 includes a processor 610 (e.g., data processing hardware 610), memory 620 (e.g., memory hardware 620), a storage device 630, a high-speed interface/controller 640 that connects to memory 620 and high-speed expansion port 650; Includes a low speed interface/controller 660 that connects to a low speed bus 670 and storage device 630. Each of components 610, 620, 630, 640, 650, and 660 may be interconnected using various buses and mounted on a common motherboard or in other manners as appropriate. Processor 610 (i.e., data processing hardware 112 of user device 110 or data processing hardware 134 of remote system 130) executes within computing device 600, including instructions stored in memory 620 or in storage device 630. and display graphic information for a graphical user interface (GUI) on an external input/output device, such as a display 680 coupled to high-speed interface 640. Memory 620 and storage device 630 may include memory hardware 114 of user device 110 or memory hardware 136 of remote system 130. In other implementations, multiple processors and/or multiple buses may be used with multiple memories and multiple types of memory, where appropriate. Similarly, multiple computing devices 600 may be connected, each device providing a portion of the required operation (eg, as a server bank, a group of blade servers, or a multiprocessor system).

Memory 620 stores information within computing device 600 on a non-transitory basis. Memory 620 may be a computer readable medium, a volatile memory unit, or a non-volatile memory unit. Non-transitory memory 620 is a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by computing device 600. good. Examples of non-volatile memory include flash memory and read-only memory (ROM)/programmable read-only memory (PROM)/erasable programmable read-only memory (EPROM)/electronically erasable programmable read-only memory (EEPROM) (e.g. (typically used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), as well as disk or tape. Not done.

Storage device 630 can provide mass storage for computing device 600. In some implementations, storage device 630 is a computer readable medium. In various different implementations, storage device 630 includes a floppy disk device, a hard disk device, an optical disk device or tape device, a flash memory or other similar solid-state memory device, or a device in a storage area network or other configuration. Can be an array of devices. In additional implementations, a computer program product is tangibly embodied on an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine readable medium, such as memory 620, storage device 630 or memory on processor 610.

High speed controller 640 manages bandwidth intensive operations for computing device 600, while low speed controller 660 manages low bandwidth intensive operations. Such assignment of responsibilities is merely illustrative. In some implementations, high-speed controller 640 is coupled to memory 620, display 680 (e.g., through a graphics processor or accelerator), and high-speed expansion port 650 that can receive various expansion cards (not shown). Ru. In some implementations, low speed controller 660 is coupled to storage device 630 and low speed expansion port 690. The low-speed expansion port 690 may include a variety of communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), such as a keyboard, pointing device, scanner, or network device, such as a switch or router, through a network adapter. etc., may be coupled to one or more input/output devices.

Computing device 600 may be implemented in several different forms, as illustrated in the figures. For example, it may be implemented as a standard server 600a or a stack of such servers 600a, as a laptop computer 600b, or as part of a rack server system 600c.

Various implementations of the systems and techniques described herein include digital electronic and/or optical circuitry, integrated circuitry, specially designed ASICs (Application Specific Integrated Circuits), computer hardware, firmware, software, and/or a combination thereof. These various implementations include a storage system, dedicated or general purpose, coupled to receive data and instructions from, and transmit data and instructions to, at least one input device and at least one output device. may include one or more computer program implementations that are executable and/or interpretable on a programmable system that may include at least one programmable processor.

These computer programs (also known as programs, software, software applications or code) contain machine instructions for a programmable processor and are written in high-level procedural and/or object-oriented programming languages and/or in assembly/machine language. It can be implemented with As used herein, the terms "machine-readable medium" and "computer-readable medium" include a machine-readable medium that receives machine instructions as a machine-readable signal and provides machine instructions and/or data to a programmable processor. Refers to any computer program product, non-transitory computer readable medium, apparatus and/or device (eg, magnetic disk, optical disk, memory, programmable logic device (PLD)) used for. The term "machine readable signal" refers to any signal used to provide machine instructions and/or data to a programmable processor.

The processes and logic flows described herein refer to one or more data processing hardware components that execute one or more computer programs to perform functions by operating on input data and producing output. This can be done by a programmable processor, also called hardware. The process and logic flow may also be performed by dedicated logic circuitry, such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from read-only memory and/or random access memory. The essential elements of a computer are a processor for executing instructions, and one or more memory devices for storing instructions and data. Generally, the computer further includes, or is operatively coupled to, one or more mass storage devices for storing data, such as magnetic, magneto-optical or optical disks, for receiving data from or receiving data therefrom. or both. However, a computer does not need to have such a device. Computer readable media suitable for storing computer program instructions and data include, by way of example, semiconductor memory devices such as EPROM, EEPROM and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD- Includes all forms of non-volatile memory, media and memory devices, including ROM and DVD-ROM discs. The processor and memory can be supplemented by or incorporated into dedicated logic circuitry.

To provide for interaction with a user, one or more aspects of the present disclosure may include a display device, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor or touch screen, as well as an optional It can be implemented on a computer with a keyboard and pointing device, such as a mouse or trackball, that optionally allows a user to provide input to the computer. Other types of devices can also be used to provide interaction with the user, for example the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback or tactile feedback; Input from the user can then be received in any form, including acoustic, audio or tactile input. In addition, the computer may cause a web page to appear on a web browser in response to a request received from a web browser on a user's client device, for example, by sending the document to the device used by the user and receiving the document from the device. You can interact with users by sending them.

Some implementation examples were described. Nevertheless, it will be understood that various changes may be made without departing from the spirit and scope of the disclosure. Accordingly, other implementations are within the scope of the following claims.

10 users
12 questions
14 Audio data
100 Audio environment
110 Voice-enabled devices
112 Data processing hardware
114 Memory hardware
115 Digital Assistant
116 Audio Capture Device
118 Audio output device
120 network
130 remote system
132 remote resources
134 Remote Data Processing Hardware
136 Remote Memory Hardware
140 Automatic Speech Recognition (ASR) System
142 Notation
150 Text-to-Speech (TTS) System
152 TTS input
154 TTS Audio
200 Pronunciation determination model
202 User pronunciation
204 TTS pronunciation
210 User pronunciation related features (UPF)
220 User pronunciation reliability estimator
222 User pronunciation reliability
230 TTS pronunciation related features (TTSPF)
240 TTS pronunciation reliability estimator
242 TTS pronunciation reliability
250 Pronunciation determined
300 TTS model
302 encoder
304 decoder
306 Synthesizer
600 computing devices
600a standard server
600b laptop computer
600c rack server system
610 processor
620 memory
630 storage device
640 High Speed Interface/Controller
650 high speed expansion port
660 low speed interface/controller
670 low speed bus
680 display
690 Low Speed Expansion Port

Claims (24)

A computer-implemented method (500), when executed on data processing hardware (610), causes the data processing hardware (610) to perform an operation, the operation comprising:
receiving a user pronunciation (202) of a particular word present in the question (12) spoken by the user;
receiving a TTS pronunciation (204) of the same particular word present in a text-to-speech (TTS) input (152), said TTS input (152) comprising a textual representation of a response to said question (12); , the TTS pronunciation (204) of the particular word is different from the user pronunciation (202) of the particular word;
obtaining user pronunciation-related features (210) associated with the user pronunciation (202) of the particular word;
obtaining TTS pronunciation-related features (230) associated with the TTS pronunciation (204) of the particular word;
as an output from a pronunciation decision model (200) configured to receive as input said user pronunciation related features (210) and said TTS pronunciation related features (230) for use in TTS audio (154). generating a pronunciation decision (250) that selects one of the user pronunciation (202) of the particular word associated with a degree or the TTS pronunciation (204) of the particular word;
the user pronunciation (202) for the particular word selected by the pronunciation decision (250) output from the pronunciation decision model (200) for audible output from a user device (110) associated with the user; or providing the TTS audio (154) comprising a synthesized speech representation of the response to the question (12) using the one of the TTS pronunciations (204) for the particular word. Method(500). The said operation is
receiving audio data (14) corresponding to the question (12) spoken by the user;
and processing the audio data (14) to generate a representation (142) of the question (12) using an automatic speech recognizer (ASR (140)). Method(500). receiving the user pronunciation (202) of the specific word;
extracting the user pronunciation (202) of the particular word from the intermittent state of the ASR (140) while processing the audio data (14) using the ASR (140);
extracting a user acoustic representation of the particular word from the audio data (14), wherein the user acoustic representation conveys the user pronunciation (202) of the particular word;
3. The method (500) of claim 2, comprising at least one of: processing the audio data (14) to generate a user phoneme representation conveying the user pronunciation (202) of the particular word. one of the user pronunciation-related features (210) associated with the user pronunciation (202) of the particular word that is associated with the ASR (140) recognizing the particular word in the audio data (14); The method (500) of claim 2 or 3, comprising a plurality of reliability features. The user pronunciation related feature (210) associated with the user pronunciation (202) of the specific word,
the geographic region of the user when the question (12) was spoken by the user;
linguistic demographic information associated with said user;
1. Frequency of using the user pronunciation (202) when pronouncing the particular word in previous questions (12) spoken by the user and/or other users. The method according to any one of (500) to (4) above. receiving the TTS pronunciation (204) of the specific word;
receiving as input to a TTS system (150) said TTS input (152) comprising a textual representation of said response to said question (12);
generating as output from the TTS system (150) an initial sample of TTS audio (154) comprising an initial synthesized speech representation of the response to the question (12);
extracting a TTS acoustic representation of the particular word from the initial sample of the TTS audio (154), the TTS acoustic representation conveying the TTS pronunciation (204) of the particular word; A method (500) according to any one of claims 1 to 5. receiving the TTS pronunciation (204) of the particular word, processing the textual representation of the response to the question (12) to convey the TTS pronunciation (204) of the particular word; 7. A method (500) according to any one of claims 1 to 6, comprising the step of generating a representation. The TTS pronunciation related feature (230) associated with the TTS pronunciation (204) of the specific word,
Is it a confirmed preferred pronunciation for said particular word?
an unconfirmed pronunciation for said particular word that is inferred using pronunciation mining from one or more auxiliary sources;
a pronunciation variant feature indicating whether there are any other variants for pronouncing said particular word;
8. The method (500) of any one of claims 1 to 7, including at least one of a pronunciation complexity feature indicating a likelihood of user mispronunciation of the particular word. after said operation generates said pronunciation decision (250) selecting said one of said user pronunciation (202) of said particular word or said TTS pronunciation (204) of said particular word;
Whether the user prefers the user pronunciation of the specific word (202) or the TTS pronunciation of the specific word (204) to pronounce the specific word in subsequent TTS output (204) receiving explicit feedback from said user indicating that;
9. The method (500) of any preceding claim, further comprising: updating the pronunciation decision model (200) based on the explicit feedback from the user. when the operation generates TTS audio (154) including the particular word when the explicit feedback from the user indicates that the user prefers the user pronunciation (202) of the particular word; 10. The method (500) of claim 9, further comprising updating a TTS system (150) to use the user pronunciation (202) of the particular word. after the operation provides the TTS audio (154) for audible output from the user device (110);
receiving audio data (14) corresponding to a next question (12) spoken by the user or another user that includes the particular word;
In the same way as the one of the user pronunciation (202) for the specific word selected by the pronunciation determination (250) or the TTS pronunciation (204) for the specific word, the user or the other user determining implicit user feedback indicating whether the particular word in question (12) has been pronounced;
11. The method (500) of any preceding claim, further comprising: updating the pronunciation decision model (200) based on the implicit user feedback. said operation is such that said implicit feedback is similar to said one of said user pronunciation (202) for said particular word selected by said pronunciation determination (250) or said TTS pronunciation (204) for said particular word; 12. The method of claim 11, further comprising updating the TTS system (150) based on the implicit user feedback when the user indicates that he has pronounced the particular word in the next question (12). Method described (500). data processing hardware (610);
a system (600) comprising: memory hardware (620) in communication with the data processing hardware (610), the memory hardware (620) executing on the data processing hardware (610); It stores an instruction that causes the data processing hardware (610) to perform an operation when the operation is performed.
receiving a user pronunciation (202) of a particular word present in the question (12) spoken by the user;
receiving a TTS pronunciation of the same specific word present in a text-to-speech (TTS) input (152), said TTS input (152) comprising a textual representation of a response to said question (12); the TTS pronunciation (204) of the word is different from the user pronunciation (202) of the specific word;
obtaining a user pronunciation related feature (210) associated with the user pronunciation (202) of the specific word;
obtaining a TTS pronunciation related feature (230) associated with the TTS pronunciation (204) of the specific word;
as an output from a pronunciation decision model (200) configured to receive as input said user pronunciation related features (210) and said TTS pronunciation related features (230) for use in TTS audio (154). generating a pronunciation decision (250) that selects one of the user pronunciation (202) of the particular word associated with a degree or the TTS pronunciation (204) of the particular word;
the user pronunciation (202) for the particular word selected by the pronunciation decision (250) output from the pronunciation decision model (200) for audible output from a user device (110) associated with the user; or providing said TTS audio (154) comprising a synthesized speech representation of said response to said question (12) using said one of said TTS pronunciations (204) for said particular word ( 600). The said operation is
receiving audio data (14) corresponding to the question (12) spoken by the user;
14. Processing the audio data (14) to generate a representation (142) of the question (12) using an automatic speech recognizer (ASR) (140). System(600). receiving the user pronunciation (202) of the specific word;
extracting the user pronunciation (202) of the particular word from the intermittent state of the ASR (140) while processing the audio data (14) using the ASR (140);
extracting a user acoustic representation of the particular word from the audio data (14), wherein the user acoustic representation conveys the user pronunciation (202) of the particular word;
15. The system (600) of claim 14, comprising at least one of: processing the audio data (14) to generate a user phoneme representation that conveys the user pronunciation (202) of the particular word. one of the user pronunciation-related features (210) associated with the user pronunciation (202) of the particular word that is associated with the ASR (140) recognizing the particular word in the audio data (14); or a plurality of reliability features. The user pronunciation related feature (210) associated with the user pronunciation (202) of the specific word,
the geographic region of the user when the question (12) was spoken by the user;
linguistic demographic information associated with said user;
13. Frequency of using the user pronunciation (202) when pronouncing the particular word in previous questions (12) spoken by the user and/or other users. 16. The system (600) according to any one of paragraphs 1 to 16. receiving the TTS pronunciation (204) of the specific word;
receiving as input to a TTS system (150) said TTS input (152) comprising a textual representation of said response to said question (12);
producing as an output from the TTS system (150) an initial sample of TTS audio (154) comprising an initial synthesized speech representation of the response to the question (12);
extracting a TTS acoustic representation of the particular word from the initial sample of the TTS audio (154), the TTS acoustic representation conveying the TTS pronunciation (204) of the particular word; A system (600) according to any one of claims 13 to 17. Receiving the TTS pronunciation (204) of the particular word processes the textual representation of the response to the question (12) to convey the TTS pronunciation (204) of the particular word. 19. The system (600) of any one of claims 13-18, comprising generating a representation. The TTS pronunciation related feature (230) associated with the TTS pronunciation (204) of the specific word,
Is it a confirmed preferred pronunciation for said particular word?
an unconfirmed pronunciation for said particular word that is inferred using pronunciation mining from one or more auxiliary sources;
a pronunciation variant feature indicating whether there are any other variants for pronouncing said particular word;
20. The system (600) of any one of claims 13-19, comprising at least one of a pronunciation complexity feature indicating a likelihood of user mispronunciation of the particular word. after said operation generates said pronunciation decision (250) selecting said one of said user pronunciation (202) of said particular word or said TTS pronunciation (204) of said particular word;
Whether the user prefers the user pronunciation of the specific word (202) or the TTS pronunciation of the specific word (204) to pronounce the specific word in subsequent TTS output (204) receiving explicit feedback from said user indicating that;
21. The system (600) of any one of claims 13-20, further comprising: updating the pronunciation decision model (200) based on the explicit feedback from the user. when the operation generates TTS audio (154) including the particular word when the explicit feedback from the user indicates that the user prefers the user pronunciation (202) of the particular word; 22. The system (600) of claim 21, further comprising updating a TTS system (150) to use the user pronunciation (202) of the particular word. after the operation provides the TTS audio (154) for audible output from the user device (110);
receiving audio data (14) corresponding to a next question (12) spoken by the user or another user that includes the particular word;
In the same way as the one of the user pronunciation (202) for the specific word selected by the pronunciation determination (250) or the TTS pronunciation (204) for the specific word, the user or the other user determining implicit user feedback indicating whether the particular word in question (12) has been pronounced;
23. The system (600) of any one of claims 13-22, further comprising: updating the pronunciation decision model (200) based on the implicit user feedback. said operation is such that said implicit feedback is similar to said one of said user pronunciation (202) for said particular word selected by said pronunciation determination (250) or said TTS pronunciation (204) for said particular word; 24. The method of claim 23, further comprising updating the TTS system (150) based on the implicit user feedback when the user indicates that he has pronounced the particular word in the next question (12). System described (600).

Images