US20220148566A1 - Text-to-Speech Adapted by Machine Learning - Google Patents
Text-to-Speech Adapted by Machine Learning Download PDFInfo
- Publication number
- US20220148566A1 US20220148566A1 US17/580,289 US202217580289A US2022148566A1 US 20220148566 A1 US20220148566 A1 US 20220148566A1 US 202217580289 A US202217580289 A US 202217580289A US 2022148566 A1 US2022148566 A1 US 2022148566A1
- Authority
- US
- United States
- Prior art keywords
- tts
- voice
- text
- speech
- attribute
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000010801 machine learning Methods 0.000 title description 18
- 230000015572 biosynthetic process Effects 0.000 claims abstract description 47
- 238000003786 synthesis reaction Methods 0.000 claims abstract description 47
- 238000000034 method Methods 0.000 claims description 34
- 238000012545 processing Methods 0.000 claims description 10
- 230000008859 change Effects 0.000 claims description 8
- 230000002194 synthesizing effect Effects 0.000 claims description 6
- 230000000694 effects Effects 0.000 claims description 5
- 230000006399 behavior Effects 0.000 abstract description 28
- 239000013598 vector Substances 0.000 abstract description 11
- 230000002996 emotional effect Effects 0.000 abstract description 6
- 239000003607 modifier Substances 0.000 abstract description 3
- 230000036651 mood Effects 0.000 abstract description 2
- 230000006870 function Effects 0.000 description 34
- 238000012549 training Methods 0.000 description 15
- 230000006978 adaptation Effects 0.000 description 9
- 230000008569 process Effects 0.000 description 9
- 230000004044 response Effects 0.000 description 9
- 238000005259 measurement Methods 0.000 description 4
- 230000001131 transforming effect Effects 0.000 description 4
- 241000699670 Mus sp. Species 0.000 description 3
- 238000006243 chemical reaction Methods 0.000 description 3
- 238000002474 experimental method Methods 0.000 description 3
- 230000014509 gene expression Effects 0.000 description 3
- 230000015654 memory Effects 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- 230000003044 adaptive effect Effects 0.000 description 2
- 238000013459 approach Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 230000003278 mimic effect Effects 0.000 description 2
- 206010048909 Boredom Diseases 0.000 description 1
- 241000238558 Eucarida Species 0.000 description 1
- 230000037007 arousal Effects 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000003542 behavioural effect Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 239000003795 chemical substances by application Substances 0.000 description 1
- 239000002537 cosmetic Substances 0.000 description 1
- 238000003066 decision tree Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 230000010006 flight Effects 0.000 description 1
- 229910052734 helium Inorganic materials 0.000 description 1
- 239000001307 helium Substances 0.000 description 1
- SWQJXJOGLNCZEY-UHFFFAOYSA-N helium atom Chemical compound [He] SWQJXJOGLNCZEY-UHFFFAOYSA-N 0.000 description 1
- 230000006698 induction Effects 0.000 description 1
- 238000012417 linear regression Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000003058 natural language processing Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 230000035484 reaction time Effects 0.000 description 1
- 230000002040 relaxant effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000001228 spectrum Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000012706 support-vector machine Methods 0.000 description 1
- 230000009897 systematic effect Effects 0.000 description 1
- 230000001225 therapeutic effect Effects 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
- 230000001755 vocal effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/30—Semantic analysis
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L13/00—Speech synthesis; Text to speech systems
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L13/00—Speech synthesis; Text to speech systems
- G10L13/02—Methods for producing synthetic speech; Speech synthesisers
- G10L13/033—Voice editing, e.g. manipulating the voice of the synthesiser
- G10L13/0335—Pitch control
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L13/00—Speech synthesis; Text to speech systems
- G10L13/02—Methods for producing synthetic speech; Speech synthesisers
- G10L13/04—Details of speech synthesis systems, e.g. synthesiser structure or memory management
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L13/00—Speech synthesis; Text to speech systems
- G10L13/08—Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
- G10L13/10—Prosody rules derived from text; Stress or intonation
Definitions
- the present invention is in the field of computer-based speech synthesis.
- This application provides an improved approach for adapting a voice to a particular customer for improving effectiveness of advertising.
- the present disclosure is directed to systems, methods, and computer software code for voice adaptation.
- Various embodiments provide for configuring a synthetic voice for best pleasantness, for improved selling, and in other ways adaptive and reactive to particular user profiles, particular situations, and particular speech semantics.
- Various embodiments provide for testing and analysis to find improved configurations.
- Some embodiments use concatenative speech synthesis. Some embodiments use fully parametric synthesis. Some embodiments configure TTS (text to speech) voice parameters based on particular attributes. Some embodiments include speaker voice analysis. Some embodiments store TTS voice parameters in user profiles. Some embodiments use attributes stored in user profiles. Some embodiments use situational attributes. Some embodiments use semantic attributes. Some embodiments determine TTS parameters by applying functions to attributes. Some functions use formulas. Some functions use algorithms. Some functions use models trained by machine learning techniques. Some embodiments assign a worth value to each behavior of listeners in order to train or update models. Various parameters and various attributes are appropriate for various embodiments.
- FIG. 1 illustrates conventional parametric speech synthesis.
- FIG. 2 illustrates parametric speech synthesis according to TTS voice attributes.
- FIG. 3 illustrates parametric speech synthesis for mimicking a speaker's voice.
- FIG. 4 illustrates parametric speech synthesis responsive to attributes stored in a user profile.
- FIG. 5 illustrates modifying parameters for parametric speech synthesis responsive to listener behavior.
- FIG. 6 illustrates parametric speech synthesis responsive to situational attributes.
- FIG. 7 illustrates parametric speech synthesis responsive to situational attributes and analysis of a speaker's voice.
- FIG. 8 illustrates modifying parametric speech synthesis responsive to listener behaviors and to semantics of natural language expressions.
- FIG. 9 illustrates example rules for assigning parameters for parametric speech synthesis according to relevant attributes.
- FIG. 10 illustrates example physical embodiments of the invention.
- FIG. 11 illustrates a computer processing system according to an embodiment of the invention.
- FIG. 12 illustrates a process for assigning TTS voice attributes to mimic a particular speaker's voice.
- FIG. 13 illustrates a process for synthesizing speech according to a model of user profile attributes.
- FIG. 14 illustrates a process for training a model of user profile attributes in response to user behavior, as affected by variations in TTS voice parameters.
- FIG. 15 illustrates a process for synthesizing speech according to a function of a model applied to a particular speaker's voice and situational attributes.
- attribute refers to inherently present aspects of things, such as a person's gender or age. Any particular thing has a value for each attribute, such as 25 years for an age and female for a gender.
- Parameters are the configurable aspects of things that are configurable, such as the baseline pitch and the range of a synthesized voice. Parameters have attributes, such as 240 hertz for a baseline pitch and 2 octaves for a range.
- Modules, as described in this disclosure, are pieces of hardware or software. The type of implementation of one module need not be the same as that of any other module.
- Speech synthesis can be concatenative, which requires many recorded data samples and therefore much data storage. Speech synthesis can be fully parametric, which requires little data and is adaptable in more dimensions and over greater ranges. Hybrid methods between concatenative synthesis and parametric synthesis are also appropriate. Articulatory methods, in which parameters apply to models of the human vocal tract, are also appropriate.
- Speech synthesis consumes text, and produces a sequence of audio samples that represent the corresponding speech audio.
- Various embodiments respond to text in many formats, such as text readable by humans, text marked up with meta-data, and symbolic representations of phonemes.
- FIG. 1 shows a speech synthesis system 100 and its operation.
- a parametric speech synthesis module 101 consumes input text and produces speech audio according to a set of internal parameters.
- a listener 102 hears the speech audio.
- FIG. 2 shows such an embodiment of a speech system 200 and its operation.
- a parametric speech synthesis module 201 consumes input text and produces speech audio according to a set of TTS voice parameters.
- a listener 102 hears the speech audio.
- the system uses internal default values for each parameter, unless an external voice description overrides the parameter value with an external TTS voice parameter, in which case the system uses the value of the external TTS voice parameter.
- the TTS voice parameters come directly from a set of voice attributes 202 .
- voice attributes such as values, stored in text files on a non-transitory computer readable medium, and transferred through a transitory computer readable medium to a computer.
- TTS systems may have different application programming interfaces (APIs) and accept different sets of parameters.
- Some systems take as input a sentence type (declarative, interrogative, imperative, exclamatory, optative) and use it to configure sentence prosody.
- Some TTS embodiments notably concatenative and hybrid TTS systems, offer prepackaged voices with human-sounding names such as Siri from Apple, Alexa from Amazon, and Samantha from Nuance; these have predefined internal parameter values.
- Some other TTS systems, including parametric synthesis TTS and sometimes hybrid TTS are more flexible and can accept as input high-level parameters such as the gender of the voice, an age, a dialect, and a speech rate, from which they derive lower level parameters.
- Some TTS systems accept mid-level parameters such as pitch range or volume range. Some TTS systems take as input very low-level parameters such as flutter, roughness, breath, formant frequency, formant bandwidth, formant amplitude, nasal DSP parameters, which are generated from higher-level parameters.
- Some TTS systems allow markup in the input text to provide fine control over prosody, including phrase level accents, intonation, word emphasis, and pitch contours. Some systems accept affect high-level emotional tags (modifiers) that represent positions along the emotional continuum (a two-dimensional space based on Positive-Negative and Active-Passive axes) such as joy, sadness, anger, fear, and boredom. Submissive and authoritative tones can also be created. Some systems give control of the level of articulation. Some TTS systems accept a combination of parameter levels, where lower-level parameter values may combine with, or override, values derived from higher-level parameters.
- FIG. 3 shows such an embodiment of a speech system 300 and its operation.
- a parametric speech synthesis module 201 consumes input text and operates according to a set of TTS voice parameters to produce speech audio for a listener 102 .
- the speech synthesis is according to a set of TTS voice parameters from a set of voice attributes 202 .
- the system assigns values of voice attributes to the set of voice attributes 202 by performing a voice analysis in a voice analysis module 303 .
- the voice analysis produces detected voice attributes and consumes speech.
- the speech comes from a speaker 304 .
- FIG. 12 shows a process of using the embodiment of FIG. 3 . First, the system performs voice analysis of a speaker to determine detected values of voice attributes 1201 . Next, the system assigns values of voice attributes to a set of voice attributes 1202 . Finally, the system provides voice attributes as TTS voice parameters to parametric speech synthesis 1203 .
- the system of FIG. 3 is useful for creating particularly interesting voices. For example, by analyzing the voice of James Earl Jones the system can produce synthesized speech in his voice. Likewise, systems can mimic the voices of Oprah Winfrey, Bart Simpson, or Darth Vader. Some embodiments allow customers to provide recordings of their own voice or the voice of friends or relatives and produce custom TTS voices for personal computers, mobile devices, and robots. This allows for entertaining, sentimental and therapeutic uses of computer generated voices that sound like specific people.
- Some embodiments use speech from listener 102 for the source input to the voice analysis 303 .
- the voice attributes 202 enable the speech synthesis 201 to produce speech audio that resembles the sound of a user's voice.
- the voice analysis 303 is not done on an utterance basis; it requires a substantial amount of data from speaker 304 .
- Some systems further provide an ability to customize parameters to change the sound of a user's voice for novelty or educational effects.
- One such customization causes an extreme increase in the voice pitch so that it sounds like the system is speaking in the voice of the user as if the user has inhaled helium.
- One such customization deepens the lower voice formants slightly, and sharpens phoneme transitions in order to provide a user the sound of their own voice with a more authoritative tone and more clear enunciation. This can help users train to give powerful lectures or speeches.
- a user profile is a set of attributes applicable to a person or a class of people. Typical user profiles include attributes such as gender, age range, ethnicity, and musical interests. Whereas such information is applicable to classes of people, some user profiles include information that is specific to a particular person, such as a birth date and home location. Whereas such information is essentially static, some user profiles include dynamic information, such as a current location and musical interests.
- FIG. 4 shows such an embodiment of a speech system 400 and its operation.
- a parametric speech synthesis module 201 consumes input text and, according to a set of TTS voice parameters, produces speech audio for a listener 102 .
- a function module 406 produces the TTS voice parameters. The function does so by transforming user profile attributes from a user profile 405 according to a model 407 .
- FIG. 13 shows a process 1300 of using the embodiments of FIG. 4 . First, the system reads user profile attribute values 1301 . Next, the system reads a model 1302 . Next, it executes a function on the user profile and model to produce TTS voice parameters 1203 . Finally, the system performs parametric speech synthesis on text according to the TTS voice parameters 1304 .
- system designers or system users are able to configure the TTS voice parameters in the user profile.
- a default dialect exists for voice synthesis in each language or country.
- a system designer tunes the TTS voice parameters for particular dialects in those countries.
- a system supports different types of voice-enabled devices, and each device has a device profile, which acts as a user profile). Some devices have female-gendered TTS voice parameters and other devices have male-gendered TTS voice parameters. As a result, the system guides users of the different devices to have the system designer's desired expectations of the devices' behaviors.
- TTS voice parameters are specifically parameters that can affect a listener's perceived nature of a voice such that it enables listeners to distinguish between different voices.
- Some system parameters, volume (gain control) in particular, are useful for adaptation in various applications, but they affect the speech audio output of speech synthesis module 201 . They are not TTS voice parameters per se, although in a noisy environment, the choice of a signal-to-noise ratio (SNR) is such a parameter, and some embodiments pass that parameter to a gain control unit, which sets the volume relative to the average noise volume. Though a higher volume is preferable for older users, it is does not affect the perceived nature of the voice. In contrast, some embodiments support TTS parameters that define the relative amplitudes of different formants or different ranges of frequency spectra. Such parameters are TTS voice parameters.
- Some embodiments attempt to optimize TTS voice parameters for the maximum satisfaction of listeners. For example, many listeners prefer to hear a voice that uses a familiar dialect. Many male listeners prefer to hear a relatively high pitch female voice from speech-enabled devices, whereas many female listeners prefer a male voice or a relatively low-pitched female voice from speech-enabled devices. In those cases, the improved set of TTS voice parameters is different for different user profiles.
- Some embodiments attempt to improve TTS voice parameters for purposes other than maximizing satisfaction of listeners, such as maximizing the likelihood that the user will make a purchase in response to an advertisement (ad conversion).
- the TTS voice parameter values that increase user satisfaction also increase ad conversion.
- the TTS voice parameter values that maximize ad conversion are not the same as those that maximize user satisfaction. For example, some listeners are more likely to follow the purchase advice of an advertisement if delivered in a deeper voice than the listener would find most comfortable.
- functions such as function 406 in FIG. 4 , compute TTS voice parameter output from one or more inputs.
- functions compute their TTS voice parameter output by using a model having specified rule-based algorithms coded by a developer.
- functions compute their TTS parameter output by using a model that they train by machine learning (ML).
- ML machine learning
- Some ML embodiments train a model from a training set.
- Some ML embodiments update a model dynamically, and thereby adapt continuously during the operation of the system.
- Some embodiments improve TTS voice parameters for a particular user profiles by experimentation. Some embodiments do so by AB testing. Some embodiments use algorithms of multivariable optimization that are more complex, such as open source or proprietary linear equation solvers, where each TTS voice parameter is a variable.
- Some embodiments perform multiple experiments across various speech utterances for a single listener. Some embodiments perform multiple experiments across speech utterances directed at a large number of listeners. Thereby, such embodiments use crowdsourcing for the generation of experimental results. In either case, some embodiments make only fine-grained experimental variations, and collect data across a large number of utterances. Making coarse-grained variations to a machine voice, such that any particular user perceives it as having more than one voice, cause perceptions of awkwardness that would mask useful experimental results.
- FIG. 5 shows such an embodiment of a speech system 500 and its operation.
- a parametric speech synthesis module 201 consumes input text and, according to a set of TTS voice parameters, produces speech audio for a listener 102 .
- a function module 506 produces the TTS voice parameters. The function does so by transforming user profile attributes from a user profile 505 according to a model 507 .
- model 507 is a trained model.
- FIG. 14 shows a process 1400 of using the embodiments of FIG. 5 . First, the system reads user profile attribute values 1401 . Next, the system reads the model 1402 .
- the system executes a function on the user profile attribute values and model to produce TTS voice parameters 1403 .
- the system performs parametric speech synthesis on text according to the TTS voice parameters 1404 .
- the system detects a user behavior in response to the synthesized speech audio 1405 .
- the system executes a training function on the user profile attribute values, TTS voice parameters, and user behavior to produce an update to the model.
- the system observes a resulting user behavior.
- a worth value such as a number between 0 and 1.
- a worth value indicates a degree of desirability of the behavior.
- the system detects a desirable behavior, such as clicking on an ad; the worth value is 1.
- the system detects that a desirable behavior did not occur, within a certain period of time, and treats the absence of a desirable behavior as an undesirable behavior; the worth value is 0.
- the training module 508 receives as input a behavior, a worth value, user profile attributes and TTS voice parameters. This collected input is a training sample for the trained model 507 , and model 507 is updated accordingly. Thereby, a feedback loop improves the TTS voice parameters that the function module 506 produces.
- Some embodiments maintain a stored model. For each utterance, they read the stored parameters, make certain changes to one or more parameters, and then, if they detect the desired behavior, store the changed set of parameters. If the desired behavior did not occur, the system discards the changed parameters. Some systems change parameters randomly in very small ways, and allow the parameter set to evolve gradually by natural selection. Some embodiments use systematic walks over certain ranges of each parameter in sequence. Some embodiments change groups of parameters together, and train the model 507 over large numbers of utterances in order to avoid local maxima or minima within the parameter state space.
- Various embodiments incorporate machine learning within the measurement module 508 and between the measurements module 508 and the stored model 507 .
- Different embodiments with different purposes or different target industries use different means of capturing user behavior after the system synthesizes TTS text and outputs the audio to the user. For example: systems that provide clickable ads detect whether a listener clicks an ad before a specific period of time; mobile phone software-based systems detect which app a user opens next; piloted airplanes measure the reaction time before a pilot follows an instruction in the utterance; shopping assistants measure the number of items and total amount of money spent by a shopper; child attendant systems measure the level and type of activity of children.
- systems that provide clickable ads detect whether a listener clicks an ad before a specific period of time
- mobile phone software-based systems detect which app a user opens next
- piloted airplanes measure the reaction time before a pilot follows an instruction in the utterance
- shopping assistants measure the number of items and total amount of money spent by a shopper
- child attendant systems measure the level and type of activity of children.
- TTS voice parameters are used in different situations, even for the same user profile. This is especially true for speech synthesis in mobile devices, such as phones, vehicles, and robots. It is also true for stationary devices for which situations change over time, such as dynamic billboards, information kiosks, and music players. Generally, speech-enabled systems use sensors to detect situational attributes.
- FIG. 6 shows such an embodiment of a speech system 600 and its operation.
- a parametric speech synthesis module 201 consumes input text and, according to a set of TTS voice parameters, produces speech audio for a listener 102 .
- the TTS voice parameters are situational attributes.
- Some phones, vehicles, and music players use microphones to detect ambient noise levels and assign TTS voice parameter values to deepen the voice by lowering the low frequencies.
- Some vehicles and robots use image and vision processing to detect the number of people present and their identities and assign TTS voice parameter values that the present users prefer and to which they respond more. For example, children respond more to high-pitched voices, and ones that are more melodic.
- Some dynamic billboards use internal clocks and time sync signals to detect the time of day and assign TTS voice parameters for faster speech rate during morning and work hours, but a slower speech rate during relaxing evening hours.
- Some information kiosks measure usage frequency to detect busyness and assign TTS voice parameters for a high level of arousal when usage is frequent.
- Some music players identify the genre of music playing in order to detect the type of mood and listeners likely to be present and assign TTS voice parameters for with a resonance to match the style and particular piece of music.
- Some embodiments provide behavior measurement and model training, as described above in the feedback and experimentation section, in order to determine which voice attributes are most useful for each situation. Experimentation will likely find, for example, that a level of enunciation that would sound awkward in a quiet environment improves understandability by listeners in a situation with a high noise level, affecting the measure of worth. Many other model experiments involving situational attributes are possible.
- Speaker voices change from one situation to another. Speaker voice analysis can provide useful information for determination of suitable TTS voice parameters in combination with situational attributes. Some embodiments perform voice analysis on ambient speakers. Some embodiments perform voice analysis on a single person who is both a speaker and a listener interacting with the system.
- FIG. 7 shows such an embodiment of a speech system 700 and its operation.
- a parametric speech synthesis module 201 consumes input text and, according to a set of TTS voice parameters, produces speech audio for a listener 102 .
- a function module 706 produces the TTS voice parameters. The function does so by transforming user situational attributes according to a model 707 .
- model 707 is a specified model, a fixed trained model, or an adaptable model.
- the function further takes in a set of voice attributes 202 .
- the system assigns values of voice attributes to the set of voice attributes 202 by performing a voice analysis in a voice analysis module 303 .
- the voice analysis produces detected voice attributes and consumes speech.
- the speech comes from a speaker 304 .
- FIG. 15 shows a process 1500 of using the embodiments of FIG. 7 .
- the system analyzes speech from a speaker to produce a set of voice attributes 1501 .
- the system reads the voice attributes, situational attributes, and model 1502 .
- the system executes a function on the voice attributes, situational attributes, and model to produce TTS voice parameters 1503 .
- the system performs parametric speech synthesis on text according to the TTS voice parameters 1504 .
- TTS voice parameters are different depending on the semantic attributes of the text. Some embodiments perform natural language processing on the text to extract semantic attributes. Some embodiments receive semantic attributes with the text. Some systems choose TTS voice parameters that are friendly and submissive when the text contains a question to the listener, but become slightly more authoritative when responding to a question, and more commanding when giving instructions.
- Some embodiments from which listeners expect to hear different voices, change the voice gender based on the domain of conversation. If the text regards the topic of bricklaying, such embodiments choose a male voice, whereas, if the text regards the topic of cosmetics, such embodiments choose a female voice.
- Some embodiments adapt TTS parameters based on the proportion of proper nouns and uncommon words in the text. In particular, they increase articulation for such words. Some embodiments adapt TTS parameters based on the number of dependent clauses or the number of modifiers (adjectives and adverbs) in the text. In particular, they slow the rate of speech for such sentences. Some embodiments adapt TTS parameters based on the emotional charge in the text to be spoken. Some examples of emotional charge attributes include happiness, sadness, novelty, excitement, inquisitiveness, and certainty. For example, a happy tone is appropriate to announce the availability of a low-cost flight for certain desired dates, whereas a low tone is appropriate for responding with a failure.
- Some embodiments that adapt TTS voice parameters based on semantic attributes are natural language (NL) based systems. Some embodiments use training to improve the effectiveness of synthesized speech on particular users or classes of users for particular expression semantics.
- NL natural language
- FIG. 8 shows such an embodiment of a speech system 800 and its operation.
- a parametric speech synthesis module 201 consumes input text and, according to a set of TTS voice parameters, produces speech audio for a listener 102 .
- a function module 806 produces the TTS voice parameters. The function does so by transforming both semantic attributes and attributes of a user profile 505 according to a model 807 .
- model 807 is adaptable. After the parametric speech synthesis module 201 produces speech audio, the system observes a responsive user behavior, or determines that a desirable behavior did not occur within a particular time period. The system 800 assigns a worth value to the behavior. The worth value, together with all the other dimensions of the training vector (user profile attributes, semantic attributes and TTS voice parameters) are training sample inputs to the training module 808 . The training module 808 , in response to the training samples, produces an update to the model 807 . Thereby, a feedback loop improves the TTS voice parameters that the function module 806 produces for particular listeners hearing expressions with particular semantics.
- TTS voice parameters For example, some function modules apply a quadratic equation of certain factors to an age user profile attribute in order to produce a speech rate TTS voice parameter. Experimentation eventually settles on a set of effective factors, which produce TTS parameters from the function that are slower for very young and very old listeners, but fastest for listeners in the 16 to 25 year old age range.
- a more advanced function for a dialect TTS voice parameter uses a home region user profile attributes, but scales that inversely by the speech rate parameter such that the dialect mimics that typical of the user profile home region most strongly for younger and older listeners, but allows greater dialect difference for middle-aged listeners.
- FIG. 9 shows several examples of basic rules of specified models.
- FIG. 9( a ) assigns a pitch parameter to value LOW if a user profile attribute, listener gender, is FEMALE, otherwise sets the pitch parameter to HIGH.
- FIG. 9( b ) assigns a first formant baseline frequency parameter, F1_freq_min, to 240-40*noise_level hertz. This deepens the voice, while maintaining the high frequencies that distinguish phonemes, when the situational noise attribute, noise_level, is high.
- FIG. 9( c ) assigns a voice_gender TTS voice parameter to be MALE for a semantic attribute associated with the topic BRICKLAYING, otherwise assigns the voice_gender TTS voice parameter to FEMALE.
- Some embodiments use machine learning (ML) techniques to estimate, for each utterance, improved TTS voice parameters.
- ML techniques are appropriate. They generally train by attempting to reduce a cost function to approximate optimal parameters.
- Some such embodiments set values of the vector B equal to a model associated with one or more of user profile attributes, situational attributes, voice attributes, and semantic attributes to train a model.
- the system uses the model as a predictor.
- Each pair of a measured behavior vector B resulting from a corresponding choice of a parameter vector P is a training pair.
- Such systems process the training pairs using a ML algorithm to find a prediction model F that reduces a cost function over the training set.
- the error is measured as the distance between the TTS voice parameters P and the predicted TTS voice parameters F(B).
- the error measure is the Euclidian distance (sum of squares).
- the error is the norm ⁇ P ⁇ F(B) ⁇ of the error vector P ⁇ F(B).
- the best model minimizes ⁇ P ⁇ F(B) ⁇ . Accordingly, the ML algorithm seeks to approach this best function F(B) and will at least reduce the error measure in most cases.
- ML embodiments use multivariate linear regression, support vector machines, neural networks, and decision tree induction. Some ML embodiments are supervised, and apply past learning to new data. Some ML embodiments are unsupervised, and draw inferences from large datasets.
- Some embodiments use processors with instruction sets and compilers optimized for machine learning algorithms.
- Some large-scale systems use distributed storage and processing architectures such as the open source Hadoop or other proprietary architectures.
- Some large-scale systems use field programmable gate array, application specific integrated circuit, or custom system-on-chip chips. Some such chips, or components of such chips, use hardwired logic that are not software programmable.
- Most large-scale systems receive their configuration and control from remote computers over networks. Such systems do not have direct input and output interface such as keyboards, mice, displays, or touch screens. However, some such systems have processors, such as digital signal processors and multi-core processors. Some systems have non-transitory computer readable media such as solid-state storage device and Flash RAM. Some systems have transitory computer readable media such as Ethernet and PCI-Express connections.
- Some embodiments are small-scale devices that support single users. Some such embodiments are specialized personal computers, mobile phones, tablets, virtual assistant devices, and household automatons. Some such devices have keyboards, mice, liquid crystal displays, touch screens, accelerometers, microphones, speakers, and gestural limbs.
- FIG. 10 shows some physical embodiments.
- FIG. 10( a ) is a server farm.
- FIG. 10( b ) is a mobile phone.
- FIG. 10( c ) is a disk non-transitory computer readable medium.
- FIG. 10( d ) is a Flash RAM non-transitory computer readable medium.
- FIG. 10( e ) is the bottom side of a system-on-chip.
- FIG. 10( f ) is the top side of a system-on-chip.
- FIG. 11 shows a software-implemented embodiment that runs on computer processors.
- Computer system 1100 comprises parallel processors 1101 and 1102 , which connect to interconnect 1103 , through which the processors can execute software from instructions and operate on data stored in random access memory (RAM) 1104 and non-transitory, non-volatile memory 1105 .
- Software running on computer system 1100 accesses the Internet through network interface 1108 , provides a GUI through display controller 1106 , and accepts user input through I/O controller 1107 , all of which are also connected through interconnect 1103 .
- the processors are ARM instruction set processors. In some embodiments they are x86 processors.
- memories, controllers, and interfaces are all on the same system-on-chip. In some embodiments, some elements are in different chips.
- the non-volatile memory is a hard disk drive. In some embodiments, it is a solid-state drive.
- the display controller connects to a local device display panel through a mobile industry processor interface (MIPI) display serial interface (DSI). In some embodiments, the display controller connects to a HDMI connector. In various embodiments, the I/O controller interfaces to touch screens, keyboards, mice, microphones, speakers, and USB connectors.
- the network interface is an Ethernet cable interface, Wi-Fi interface, Bluetooth interface, and 5G LTE interface.
- Various embodiments use suitable mapping functions on different combinations of user profile attributes, voice attributes, situational attributes, and semantic attributes to determine TTS voice parameters.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Acoustics & Sound (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Artificial Intelligence (AREA)
- General Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Machine Translation (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
Description
- This application is a continuation of U.S. Non-Provisional application Ser. No. 16/742,006, entitled “Text-to-Speech Adapted by Machine Learning”, filed Jan. 14, 2020, which is a continuation of U.S. Non-Provisional application Ser. No. 15/406,213, entitled “Parametric Adaptation Of Voice Synthesis”, filed Jan. 13, 2017, which claims the benefit of U.S. Provisional Application Ser. No. 62/438,873, entitled “Parametric Adaptation Of Voice Synthesis”, filed Dec. 23, 2016. All of the above-referenced applications are hereby incorporated herein by reference in their entirety for all purposes.
- The present invention is in the field of computer-based speech synthesis.
- Research shows that certain voices are more pleasant to listeners and that the preference varies by listener. Advertisers know that certain voices are more successful for selling, and furthermore that the best voice for selling is different from one product to another. Previous work on voice adaptation attempted to make a speech output sound like a particular person.
- This application provides an improved approach for adapting a voice to a particular customer for improving effectiveness of advertising.
- Current systems for voice adaptation do not configure a voice for best pleasantness, for improved selling, or in other ways adaptive and reactive to particular user profiles, particular situations, or particular speech semantics. Furthermore, conventional systems do not aid in finding improved configurations.
- The present disclosure is directed to systems, methods, and computer software code for voice adaptation. Various embodiments provide for configuring a synthetic voice for best pleasantness, for improved selling, and in other ways adaptive and reactive to particular user profiles, particular situations, and particular speech semantics. Various embodiments provide for testing and analysis to find improved configurations.
- Some embodiments use concatenative speech synthesis. Some embodiments use fully parametric synthesis. Some embodiments configure TTS (text to speech) voice parameters based on particular attributes. Some embodiments include speaker voice analysis. Some embodiments store TTS voice parameters in user profiles. Some embodiments use attributes stored in user profiles. Some embodiments use situational attributes. Some embodiments use semantic attributes. Some embodiments determine TTS parameters by applying functions to attributes. Some functions use formulas. Some functions use algorithms. Some functions use models trained by machine learning techniques. Some embodiments assign a worth value to each behavior of listeners in order to train or update models. Various parameters and various attributes are appropriate for various embodiments.
-
FIG. 1 illustrates conventional parametric speech synthesis. -
FIG. 2 illustrates parametric speech synthesis according to TTS voice attributes. -
FIG. 3 illustrates parametric speech synthesis for mimicking a speaker's voice. -
FIG. 4 illustrates parametric speech synthesis responsive to attributes stored in a user profile. -
FIG. 5 illustrates modifying parameters for parametric speech synthesis responsive to listener behavior. -
FIG. 6 illustrates parametric speech synthesis responsive to situational attributes. -
FIG. 7 illustrates parametric speech synthesis responsive to situational attributes and analysis of a speaker's voice. -
FIG. 8 illustrates modifying parametric speech synthesis responsive to listener behaviors and to semantics of natural language expressions. -
FIG. 9 illustrates example rules for assigning parameters for parametric speech synthesis according to relevant attributes. -
FIG. 10 illustrates example physical embodiments of the invention. -
FIG. 11 illustrates a computer processing system according to an embodiment of the invention. -
FIG. 12 illustrates a process for assigning TTS voice attributes to mimic a particular speaker's voice. -
FIG. 13 illustrates a process for synthesizing speech according to a model of user profile attributes. -
FIG. 14 illustrates a process for training a model of user profile attributes in response to user behavior, as affected by variations in TTS voice parameters. -
FIG. 15 illustrates a process for synthesizing speech according to a function of a model applied to a particular speaker's voice and situational attributes. - Throughout this disclosure, the term attribute refers to inherently present aspects of things, such as a person's gender or age. Any particular thing has a value for each attribute, such as 25 years for an age and female for a gender. Parameters are the configurable aspects of things that are configurable, such as the baseline pitch and the range of a synthesized voice. Parameters have attributes, such as 240 hertz for a baseline pitch and 2 octaves for a range. Modules, as described in this disclosure, are pieces of hardware or software. The type of implementation of one module need not be the same as that of any other module.
- Many methods of speech synthesis, also known as text-to-speech (TTS) are well known. Speech synthesis can be concatenative, which requires many recorded data samples and therefore much data storage. Speech synthesis can be fully parametric, which requires little data and is adaptable in more dimensions and over greater ranges. Hybrid methods between concatenative synthesis and parametric synthesis are also appropriate. Articulatory methods, in which parameters apply to models of the human vocal tract, are also appropriate.
- Speech synthesis consumes text, and produces a sequence of audio samples that represent the corresponding speech audio. Various embodiments respond to text in many formats, such as text readable by humans, text marked up with meta-data, and symbolic representations of phonemes.
-
FIG. 1 shows aspeech synthesis system 100 and its operation. A parametricspeech synthesis module 101 consumes input text and produces speech audio according to a set of internal parameters. Alistener 102 hears the speech audio. - Some systems allow voice designers to configure the voice.
FIG. 2 shows such an embodiment of aspeech system 200 and its operation. A parametricspeech synthesis module 201 consumes input text and produces speech audio according to a set of TTS voice parameters. Alistener 102 hears the speech audio. - The system uses internal default values for each parameter, unless an external voice description overrides the parameter value with an external TTS voice parameter, in which case the system uses the value of the external TTS voice parameter. In the embodiment of
FIG. 2 , the TTS voice parameters come directly from a set of voice attributes 202. Various formats and methods are appropriate to represent, store, and transmit voice attributes, such as values, stored in text files on a non-transitory computer readable medium, and transferred through a transitory computer readable medium to a computer. - In various embodiments, TTS systems may have different application programming interfaces (APIs) and accept different sets of parameters. Some systems take as input a sentence type (declarative, interrogative, imperative, exclamatory, optative) and use it to configure sentence prosody. Some TTS embodiments, notably concatenative and hybrid TTS systems, offer prepackaged voices with human-sounding names such as Siri from Apple, Alexa from Amazon, and Samantha from Nuance; these have predefined internal parameter values. Some other TTS systems, including parametric synthesis TTS and sometimes hybrid TTS, are more flexible and can accept as input high-level parameters such as the gender of the voice, an age, a dialect, and a speech rate, from which they derive lower level parameters. Some TTS systems accept mid-level parameters such as pitch range or volume range. Some TTS systems take as input very low-level parameters such as flutter, roughness, breath, formant frequency, formant bandwidth, formant amplitude, nasal DSP parameters, which are generated from higher-level parameters.
- Some TTS systems allow markup in the input text to provide fine control over prosody, including phrase level accents, intonation, word emphasis, and pitch contours. Some systems accept affect high-level emotional tags (modifiers) that represent positions along the emotional continuum (a two-dimensional space based on Positive-Negative and Active-Passive axes) such as joy, sadness, anger, fear, and boredom. Submissive and authoritative tones can also be created. Some systems give control of the level of articulation. Some TTS systems accept a combination of parameter levels, where lower-level parameter values may combine with, or override, values derived from higher-level parameters.
- Some embodiments attempt to make a voice sound like a particular speaker.
FIG. 3 shows such an embodiment of aspeech system 300 and its operation. A parametricspeech synthesis module 201 consumes input text and operates according to a set of TTS voice parameters to produce speech audio for alistener 102. The speech synthesis is according to a set of TTS voice parameters from a set of voice attributes 202. The system assigns values of voice attributes to the set of voice attributes 202 by performing a voice analysis in avoice analysis module 303. The voice analysis produces detected voice attributes and consumes speech. The speech comes from aspeaker 304.FIG. 12 shows a process of using the embodiment ofFIG. 3 . First, the system performs voice analysis of a speaker to determine detected values of voice attributes 1201. Next, the system assigns values of voice attributes to a set of voice attributes 1202. Finally, the system provides voice attributes as TTS voice parameters toparametric speech synthesis 1203. - The system of
FIG. 3 is useful for creating particularly interesting voices. For example, by analyzing the voice of James Earl Jones the system can produce synthesized speech in his voice. Likewise, systems can mimic the voices of Oprah Winfrey, Bart Simpson, or Darth Vader. Some embodiments allow customers to provide recordings of their own voice or the voice of friends or relatives and produce custom TTS voices for personal computers, mobile devices, and robots. This allows for entertaining, sentimental and therapeutic uses of computer generated voices that sound like specific people. - Some embodiments use speech from
listener 102 for the source input to thevoice analysis 303. By performing analysis on a significantly large amount of speech, the voice attributes 202 enable thespeech synthesis 201 to produce speech audio that resembles the sound of a user's voice. In the systems of interest, thevoice analysis 303 is not done on an utterance basis; it requires a substantial amount of data fromspeaker 304. - Some systems further provide an ability to customize parameters to change the sound of a user's voice for novelty or educational effects. One such customization causes an extreme increase in the voice pitch so that it sounds like the system is speaking in the voice of the user as if the user has inhaled helium. One such customization deepens the lower voice formants slightly, and sharpens phoneme transitions in order to provide a user the sound of their own voice with a more authoritative tone and more clear enunciation. This can help users train to give powerful lectures or speeches.
- A user profile is a set of attributes applicable to a person or a class of people. Typical user profiles include attributes such as gender, age range, ethnicity, and musical interests. Whereas such information is applicable to classes of people, some user profiles include information that is specific to a particular person, such as a birth date and home location. Whereas such information is essentially static, some user profiles include dynamic information, such as a current location and musical interests.
- Some embodiments store TTS voice parameters within user profiles, such as TTS voice parameters described above with respect to
FIG. 3 .FIG. 4 shows such an embodiment of aspeech system 400 and its operation. A parametricspeech synthesis module 201 consumes input text and, according to a set of TTS voice parameters, produces speech audio for alistener 102. Afunction module 406 produces the TTS voice parameters. The function does so by transforming user profile attributes from auser profile 405 according to amodel 407.FIG. 13 shows aprocess 1300 of using the embodiments ofFIG. 4 . First, the system reads user profile attribute values 1301. Next, the system reads amodel 1302. Next, it executes a function on the user profile and model to produceTTS voice parameters 1203. Finally, the system performs parametric speech synthesis on text according to theTTS voice parameters 1304. - In various embodiments, system designers or system users are able to configure the TTS voice parameters in the user profile. In some embodiments, a default dialect exists for voice synthesis in each language or country. A system designer tunes the TTS voice parameters for particular dialects in those countries. In some embodiments, a system supports different types of voice-enabled devices, and each device has a device profile, which acts as a user profile). Some devices have female-gendered TTS voice parameters and other devices have male-gendered TTS voice parameters. As a result, the system guides users of the different devices to have the system designer's desired expectations of the devices' behaviors.
- Note that TTS voice parameters are specifically parameters that can affect a listener's perceived nature of a voice such that it enables listeners to distinguish between different voices. Some system parameters, volume (gain control) in particular, are useful for adaptation in various applications, but they affect the speech audio output of
speech synthesis module 201. They are not TTS voice parameters per se, although in a noisy environment, the choice of a signal-to-noise ratio (SNR) is such a parameter, and some embodiments pass that parameter to a gain control unit, which sets the volume relative to the average noise volume. Though a higher volume is preferable for older users, it is does not affect the perceived nature of the voice. In contrast, some embodiments support TTS parameters that define the relative amplitudes of different formants or different ranges of frequency spectra. Such parameters are TTS voice parameters. - Some embodiments attempt to optimize TTS voice parameters for the maximum satisfaction of listeners. For example, many listeners prefer to hear a voice that uses a familiar dialect. Many male listeners prefer to hear a relatively high pitch female voice from speech-enabled devices, whereas many female listeners prefer a male voice or a relatively low-pitched female voice from speech-enabled devices. In those cases, the improved set of TTS voice parameters is different for different user profiles.
- Some embodiments attempt to improve TTS voice parameters for purposes other than maximizing satisfaction of listeners, such as maximizing the likelihood that the user will make a purchase in response to an advertisement (ad conversion). In some situations, at coarse granularity, the TTS voice parameter values that increase user satisfaction also increase ad conversion. However, in some situations, at coarse granularity, and always at fine granularity, the TTS voice parameter values that maximize ad conversion are not the same as those that maximize user satisfaction. For example, some listeners are more likely to follow the purchase advice of an advertisement if delivered in a deeper voice than the listener would find most comfortable.
- In some embodiments, functions, such as
function 406 inFIG. 4 , compute TTS voice parameter output from one or more inputs. In some embodiments, functions compute their TTS voice parameter output by using a model having specified rule-based algorithms coded by a developer. In some embodiments, functions compute their TTS parameter output by using a model that they train by machine learning (ML). Some ML embodiments train a model from a training set. Some ML embodiments update a model dynamically, and thereby adapt continuously during the operation of the system. - Some embodiments improve TTS voice parameters for a particular user profiles by experimentation. Some embodiments do so by AB testing. Some embodiments use algorithms of multivariable optimization that are more complex, such as open source or proprietary linear equation solvers, where each TTS voice parameter is a variable.
- Some embodiments perform multiple experiments across various speech utterances for a single listener. Some embodiments perform multiple experiments across speech utterances directed at a large number of listeners. Thereby, such embodiments use crowdsourcing for the generation of experimental results. In either case, some embodiments make only fine-grained experimental variations, and collect data across a large number of utterances. Making coarse-grained variations to a machine voice, such that any particular user perceives it as having more than one voice, cause perceptions of awkwardness that would mask useful experimental results.
- Measuring user behavior improves performance of TTS parameters.
FIG. 5 shows such an embodiment of aspeech system 500 and its operation. A parametricspeech synthesis module 201 consumes input text and, according to a set of TTS voice parameters, produces speech audio for alistener 102. Afunction module 506 produces the TTS voice parameters. The function does so by transforming user profile attributes from auser profile 505 according to amodel 507. In the embodiment ofFIG. 5 ,model 507 is a trained model.FIG. 14 shows aprocess 1400 of using the embodiments ofFIG. 5 . First, the system reads user profile attribute values 1401. Next, the system reads themodel 1402. Next, the system executes a function on the user profile attribute values and model to produceTTS voice parameters 1403. Next, the system performs parametric speech synthesis on text according to theTTS voice parameters 1404. Next, the system detects a user behavior in response to the synthesizedspeech audio 1405. Finally, the system executes a training function on the user profile attribute values, TTS voice parameters, and user behavior to produce an update to the model. - After the parametric
speech synthesis module 201 produces a speech audio response, the system observes a resulting user behavior. Some embodiments assign to the resulting behavior a worth value, such as a number between 0 and 1. A worth value indicates a degree of desirability of the behavior. In some embodiments, the system detects a desirable behavior, such as clicking on an ad; the worth value is 1. In some embodiments, the system detects that a desirable behavior did not occur, within a certain period of time, and treats the absence of a desirable behavior as an undesirable behavior; the worth value is 0. The training module 508 receives as input a behavior, a worth value, user profile attributes and TTS voice parameters. This collected input is a training sample for the trainedmodel 507, andmodel 507 is updated accordingly. Thereby, a feedback loop improves the TTS voice parameters that thefunction module 506 produces. - Some embodiments maintain a stored model. For each utterance, they read the stored parameters, make certain changes to one or more parameters, and then, if they detect the desired behavior, store the changed set of parameters. If the desired behavior did not occur, the system discards the changed parameters. Some systems change parameters randomly in very small ways, and allow the parameter set to evolve gradually by natural selection. Some embodiments use systematic walks over certain ranges of each parameter in sequence. Some embodiments change groups of parameters together, and train the
model 507 over large numbers of utterances in order to avoid local maxima or minima within the parameter state space. - Various embodiments incorporate machine learning within the measurement module 508 and between the measurements module 508 and the stored
model 507. - Different embodiments with different purposes or different target industries use different means of capturing user behavior after the system synthesizes TTS text and outputs the audio to the user. For example: systems that provide clickable ads detect whether a listener clicks an ad before a specific period of time; mobile phone software-based systems detect which app a user opens next; piloted airplanes measure the reaction time before a pilot follows an instruction in the utterance; shopping assistants measure the number of items and total amount of money spent by a shopper; child attendant systems measure the level and type of activity of children. By testing thousands of small TTS voice parameter variations across millions of utterances it is possible to fine tune the improved TTS voice parameters to cause listeners that fit a particular user profile to click any particular ad.
- A system that reads bedtime stories to children detects how long it takes children to fall asleep. A system designed to report information to airplane pilots detects how often pilots fall asleep during long flights. A system that teaches massively open online courses measures student success rates on quizzes. A virtual assistant that responds to user queries measures the effectiveness of its responses by detecting when users do not follow with similar queries.
- Sometimes it is useful to have different TTS voice parameters in different situations, even for the same user profile. This is especially true for speech synthesis in mobile devices, such as phones, vehicles, and robots. It is also true for stationary devices for which situations change over time, such as dynamic billboards, information kiosks, and music players. Generally, speech-enabled systems use sensors to detect situational attributes.
- Such systems detect situational attributes and map them to TTS voice parameters. Some embodiments accept such situation attributes to adapt speech synthesis.
FIG. 6 shows such an embodiment of aspeech system 600 and its operation. A parametricspeech synthesis module 201 consumes input text and, according to a set of TTS voice parameters, produces speech audio for alistener 102. The TTS voice parameters are situational attributes. - Some phones, vehicles, and music players use microphones to detect ambient noise levels and assign TTS voice parameter values to deepen the voice by lowering the low frequencies. Some vehicles and robots use image and vision processing to detect the number of people present and their identities and assign TTS voice parameter values that the present users prefer and to which they respond more. For example, children respond more to high-pitched voices, and ones that are more melodic. Some dynamic billboards use internal clocks and time sync signals to detect the time of day and assign TTS voice parameters for faster speech rate during morning and work hours, but a slower speech rate during relaxing evening hours. Some information kiosks measure usage frequency to detect busyness and assign TTS voice parameters for a high level of arousal when usage is frequent. Some music players identify the genre of music playing in order to detect the type of mood and listeners likely to be present and assign TTS voice parameters for with a resonance to match the style and particular piece of music.
- Some embodiments provide behavior measurement and model training, as described above in the feedback and experimentation section, in order to determine which voice attributes are most useful for each situation. Experimentation will likely find, for example, that a level of enunciation that would sound awkward in a quiet environment improves understandability by listeners in a situation with a high noise level, affecting the measure of worth. Many other model experiments involving situational attributes are possible.
- Speaker voices change from one situation to another. Speaker voice analysis can provide useful information for determination of suitable TTS voice parameters in combination with situational attributes. Some embodiments perform voice analysis on ambient speakers. Some embodiments perform voice analysis on a single person who is both a speaker and a listener interacting with the system.
FIG. 7 shows such an embodiment of aspeech system 700 and its operation. A parametricspeech synthesis module 201 consumes input text and, according to a set of TTS voice parameters, produces speech audio for alistener 102. Afunction module 706 produces the TTS voice parameters. The function does so by transforming user situational attributes according to amodel 707. In various embodiments,model 707 is a specified model, a fixed trained model, or an adaptable model. The function further takes in a set of voice attributes 202. The system assigns values of voice attributes to the set of voice attributes 202 by performing a voice analysis in avoice analysis module 303. The voice analysis produces detected voice attributes and consumes speech. The speech comes from aspeaker 304. -
FIG. 15 shows aprocess 1500 of using the embodiments ofFIG. 7 . First, the system analyzes speech from a speaker to produce a set of voice attributes 1501. Next, the system reads the voice attributes, situational attributes, andmodel 1502. Next, the system executes a function on the voice attributes, situational attributes, and model to produceTTS voice parameters 1503. Finally, the system performs parametric speech synthesis on text according to theTTS voice parameters 1504. - Sometimes it is useful to have different TTS voice parameters depending on the semantic attributes of the text. Some embodiments perform natural language processing on the text to extract semantic attributes. Some embodiments receive semantic attributes with the text. Some systems choose TTS voice parameters that are friendly and submissive when the text contains a question to the listener, but become slightly more authoritative when responding to a question, and more commanding when giving instructions.
- Some embodiments, from which listeners expect to hear different voices, change the voice gender based on the domain of conversation. If the text regards the topic of bricklaying, such embodiments choose a male voice, whereas, if the text regards the topic of cosmetics, such embodiments choose a female voice.
- Some embodiments adapt TTS parameters based on the proportion of proper nouns and uncommon words in the text. In particular, they increase articulation for such words. Some embodiments adapt TTS parameters based on the number of dependent clauses or the number of modifiers (adjectives and adverbs) in the text. In particular, they slow the rate of speech for such sentences. Some embodiments adapt TTS parameters based on the emotional charge in the text to be spoken. Some examples of emotional charge attributes include happiness, sadness, novelty, excitement, inquisitiveness, and certainty. For example, a happy tone is appropriate to announce the availability of a low-cost flight for certain desired dates, whereas a low tone is appropriate for responding with a failure. In the response “I am sorry, I found no flight under $500 for Sunday, but I have found one for Tuesday. Would you like that?” the first part uses a slightly sad tone, but the second part (starting at “but”) uses a happy tone. An intelligent agent that constructs answers to a user and has a notion of failure or success in meeting her expectations converts such semantic distinctions into emotional attributes, which the TTS reflects. Some embodiments support sentences with contrastive emphasis, which adapt TTS for particular words indicate contrasting meanings. This affects prosody parameters. Consider the responses, “I did not find a flight for Sunday, but I have one for Tuesday!”, with the words “not” and “Tuesday” emphasized and the response, “I didn't find a flight for Sunday, but I have one for Tuesday!”, with the words “Sunday” and “Tuesday” emphasized.
- Some embodiments that adapt TTS voice parameters based on semantic attributes are natural language (NL) based systems. Some embodiments use training to improve the effectiveness of synthesized speech on particular users or classes of users for particular expression semantics.
-
FIG. 8 shows such an embodiment of aspeech system 800 and its operation. A parametricspeech synthesis module 201 consumes input text and, according to a set of TTS voice parameters, produces speech audio for alistener 102. Afunction module 806 produces the TTS voice parameters. The function does so by transforming both semantic attributes and attributes of auser profile 505 according to amodel 807. - In the embodiment of
FIG. 8 ,model 807 is adaptable. After the parametricspeech synthesis module 201 produces speech audio, the system observes a responsive user behavior, or determines that a desirable behavior did not occur within a particular time period. Thesystem 800 assigns a worth value to the behavior. The worth value, together with all the other dimensions of the training vector (user profile attributes, semantic attributes and TTS voice parameters) are training sample inputs to thetraining module 808. Thetraining module 808, in response to the training samples, produces an update to themodel 807. Thereby, a feedback loop improves the TTS voice parameters that thefunction module 806 produces for particular listeners hearing expressions with particular semantics. - Different embodiments with different purposes or different target industries use different functions to produce TTS voice parameters. For example, some function modules apply a quadratic equation of certain factors to an age user profile attribute in order to produce a speech rate TTS voice parameter. Experimentation eventually settles on a set of effective factors, which produce TTS parameters from the function that are slower for very young and very old listeners, but fastest for listeners in the 16 to 25 year old age range. A more advanced function for a dialect TTS voice parameter uses a home region user profile attributes, but scales that inversely by the speech rate parameter such that the dialect mimics that typical of the user profile home region most strongly for younger and older listeners, but allows greater dialect difference for middle-aged listeners.
-
FIG. 9 shows several examples of basic rules of specified models.FIG. 9(a) assigns a pitch parameter to value LOW if a user profile attribute, listener gender, is FEMALE, otherwise sets the pitch parameter to HIGH.FIG. 9(b) assigns a first formant baseline frequency parameter, F1_freq_min, to 240-40*noise_level hertz. This deepens the voice, while maintaining the high frequencies that distinguish phonemes, when the situational noise attribute, noise_level, is high.FIG. 9(c) assigns a voice_gender TTS voice parameter to be MALE for a semantic attribute associated with the topic BRICKLAYING, otherwise assigns the voice_gender TTS voice parameter to FEMALE. - Experienced system designers, with good intuition and thorough study, are able to create effective rules. However, particularly for numerical functions, machine algorithms applied to large experimental data sets can produce better rules and formulas across much larger numbers of simultaneous variables.
- Some embodiments use machine learning (ML) techniques to estimate, for each utterance, improved TTS voice parameters. Many ML techniques are appropriate. They generally train by attempting to reduce a cost function to approximate optimal parameters. The trained model is a function F that takes as input observable behaviors, such as examples enumerated above, as a vector B=[b1 b2 . . . bN] and yields as output a TTS voice parameter vector P=[p1 p2 . . . pM], that is, P=F(B) that is tuned according to the vector B by the trained model. Some such embodiments set values of the vector B equal to a model associated with one or more of user profile attributes, situational attributes, voice attributes, and semantic attributes to train a model. The system uses the model as a predictor. Each pair of a measured behavior vector B resulting from a corresponding choice of a parameter vector P is a training pair. Such systems process the training pairs using a ML algorithm to find a prediction model F that reduces a cost function over the training set. The error is measured as the distance between the TTS voice parameters P and the predicted TTS voice parameters F(B).
- In some embodiments, the error measure is the Euclidian distance (sum of squares). The error is the norm ∥P−F(B)∥ of the error vector P−F(B). The best model minimizes ∥P−F(B)∥. Accordingly, the ML algorithm seeks to approach this best function F(B) and will at least reduce the error measure in most cases.
- Various ML embodiments use multivariate linear regression, support vector machines, neural networks, and decision tree induction. Some ML embodiments are supervised, and apply past learning to new data. Some ML embodiments are unsupervised, and draw inferences from large datasets.
- Various embodiments implement different combinations of components and method steps using software. Current general-purpose computer processors offer sufficient performance for all processing aspects in systems for a single or small number of simultaneous users. Server farms can be appropriate for some implementations. However, for large numbers of users, and in particular for enough users to gather statistically significant behavioral data to fine-grained tuning of a model, the data storage and processing requirements exceed what farms of general-purpose processor servers can practically accommodate.
- To meet the demanding requirements of large-scale systems, some embodiments use processors with instruction sets and compilers optimized for machine learning algorithms. Some large-scale systems use distributed storage and processing architectures such as the open source Hadoop or other proprietary architectures. Some large-scale systems use field programmable gate array, application specific integrated circuit, or custom system-on-chip chips. Some such chips, or components of such chips, use hardwired logic that are not software programmable.
- Most large-scale systems receive their configuration and control from remote computers over networks. Such systems do not have direct input and output interface such as keyboards, mice, displays, or touch screens. However, some such systems have processors, such as digital signal processors and multi-core processors. Some systems have non-transitory computer readable media such as solid-state storage device and Flash RAM. Some systems have transitory computer readable media such as Ethernet and PCI-Express connections.
- Some embodiments are small-scale devices that support single users. Some such embodiments are specialized personal computers, mobile phones, tablets, virtual assistant devices, and household automatons. Some such devices have keyboards, mice, liquid crystal displays, touch screens, accelerometers, microphones, speakers, and gestural limbs.
-
FIG. 10 shows some physical embodiments.FIG. 10(a) is a server farm.FIG. 10(b) is a mobile phone.FIG. 10(c) is a disk non-transitory computer readable medium.FIG. 10(d) is a Flash RAM non-transitory computer readable medium.FIG. 10(e) is the bottom side of a system-on-chip.FIG. 10(f) is the top side of a system-on-chip. -
FIG. 11 shows a software-implemented embodiment that runs on computer processors.Computer system 1100 comprisesparallel processors non-volatile memory 1105. Software running oncomputer system 1100 accesses the Internet throughnetwork interface 1108, provides a GUI throughdisplay controller 1106, and accepts user input through I/O controller 1107, all of which are also connected throughinterconnect 1103. In some embodiments, the processors are ARM instruction set processors. In some embodiments they are x86 processors. In some embodiments, memories, controllers, and interfaces are all on the same system-on-chip. In some embodiments, some elements are in different chips. In some embodiments, the non-volatile memory is a hard disk drive. In some embodiments, it is a solid-state drive. In some embodiments, the display controller connects to a local device display panel through a mobile industry processor interface (MIPI) display serial interface (DSI). In some embodiments, the display controller connects to a HDMI connector. In various embodiments, the I/O controller interfaces to touch screens, keyboards, mice, microphones, speakers, and USB connectors. In various embodiments, the network interface is an Ethernet cable interface, Wi-Fi interface, Bluetooth interface, and 5G LTE interface. - Various embodiments use suitable mapping functions on different combinations of user profile attributes, voice attributes, situational attributes, and semantic attributes to determine TTS voice parameters.
- Examples given in this disclosure are illustrative and do not define the limits of the invention. Particular embodiments described in this disclosure are representative of various aspects of the invention, but are not the only reasonable embodiments. Ordinarily skilled practitioners in the art will recognize various alternative arrangements of the elements described and equivalents thereof.
Claims (19)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/580,289 US20220148566A1 (en) | 2016-12-23 | 2022-01-20 | Text-to-Speech Adapted by Machine Learning |
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201662438873P | 2016-12-23 | 2016-12-23 | |
US15/406,213 US10586079B2 (en) | 2016-12-23 | 2017-01-13 | Parametric adaptation of voice synthesis |
US16/742,006 US11531819B2 (en) | 2016-12-23 | 2020-01-14 | Text-to-speech adapted by machine learning |
US17/580,289 US20220148566A1 (en) | 2016-12-23 | 2022-01-20 | Text-to-Speech Adapted by Machine Learning |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/742,006 Continuation US11531819B2 (en) | 2016-12-23 | 2020-01-14 | Text-to-speech adapted by machine learning |
Publications (1)
Publication Number | Publication Date |
---|---|
US20220148566A1 true US20220148566A1 (en) | 2022-05-12 |
Family
ID=62630419
Family Applications (3)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/406,213 Active 2037-06-23 US10586079B2 (en) | 2016-12-23 | 2017-01-13 | Parametric adaptation of voice synthesis |
US16/742,006 Active US11531819B2 (en) | 2016-12-23 | 2020-01-14 | Text-to-speech adapted by machine learning |
US17/580,289 Pending US20220148566A1 (en) | 2016-12-23 | 2022-01-20 | Text-to-Speech Adapted by Machine Learning |
Family Applications Before (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US15/406,213 Active 2037-06-23 US10586079B2 (en) | 2016-12-23 | 2017-01-13 | Parametric adaptation of voice synthesis |
US16/742,006 Active US11531819B2 (en) | 2016-12-23 | 2020-01-14 | Text-to-speech adapted by machine learning |
Country Status (1)
Country | Link |
---|---|
US (3) | US10586079B2 (en) |
Families Citing this family (19)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR102344832B1 (en) * | 2017-05-31 | 2021-12-28 | 에이조 가부시키가이샤 | Surgical instrument detection system and computer program |
US20190019497A1 (en) * | 2017-07-12 | 2019-01-17 | I AM PLUS Electronics Inc. | Expressive control of text-to-speech content |
WO2019017033A1 (en) * | 2017-07-19 | 2019-01-24 | ソニー株式会社 | Information processing device, information processing method, and program |
US10600404B2 (en) * | 2017-11-29 | 2020-03-24 | Intel Corporation | Automatic speech imitation |
JP7082357B2 (en) * | 2018-01-11 | 2022-06-08 | ネオサピエンス株式会社 | Text-to-speech synthesis methods using machine learning, devices and computer-readable storage media |
US11398218B1 (en) * | 2018-04-26 | 2022-07-26 | United Services Automobile Association (Usaa) | Dynamic speech output configuration |
US11134308B2 (en) * | 2018-08-06 | 2021-09-28 | Sony Corporation | Adapting interactions with a television user |
US10706347B2 (en) | 2018-09-17 | 2020-07-07 | Intel Corporation | Apparatus and methods for generating context-aware artificial intelligence characters |
US11195507B2 (en) * | 2018-10-04 | 2021-12-07 | Rovi Guides, Inc. | Translating between spoken languages with emotion in audio and video media streams |
CN109582271B (en) * | 2018-10-26 | 2020-04-03 | 北京蓦然认知科技有限公司 | Method, device and equipment for dynamically setting TTS (text to speech) playing parameters |
WO2020089961A1 (en) * | 2018-10-29 | 2020-05-07 | 健一 海沼 | Voice processing device and program |
US11854538B1 (en) * | 2019-02-15 | 2023-12-26 | Amazon Technologies, Inc. | Sentiment detection in audio data |
KR20200111853A (en) | 2019-03-19 | 2020-10-05 | 삼성전자주식회사 | Electronic device and method for providing voice recognition control thereof |
CN110634468B (en) * | 2019-09-11 | 2022-04-15 | 中国联合网络通信集团有限公司 | Voice wake-up method, device, equipment and computer readable storage medium |
CN112579746B (en) * | 2019-09-29 | 2024-06-18 | 京东科技控股股份有限公司 | Method and device for acquiring behavior information corresponding to text |
KR102680097B1 (en) * | 2019-11-01 | 2024-07-02 | 삼성전자주식회사 | Electronic apparatus and operating method thereof |
US11743380B2 (en) * | 2021-03-15 | 2023-08-29 | Avaya Management L.P. | System and method for context aware audio enhancement |
US12008289B2 (en) | 2021-07-07 | 2024-06-11 | Honeywell International Inc. | Methods and systems for transcription playback with variable emphasis |
CN116092479B (en) * | 2023-04-07 | 2023-07-07 | 杭州东上智能科技有限公司 | Text prosody generation method and system based on comparison text-audio pair |
Family Cites Families (25)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CA2119397C (en) * | 1993-03-19 | 2007-10-02 | Kim E.A. Silverman | Improved automated voice synthesis employing enhanced prosodic treatment of text, spelling of text and rate of annunciation |
US6035273A (en) * | 1996-06-26 | 2000-03-07 | Lucent Technologies, Inc. | Speaker-specific speech-to-text/text-to-speech communication system with hypertext-indicated speech parameter changes |
US7110951B1 (en) * | 2000-03-03 | 2006-09-19 | Dorothy Lemelson, legal representative | System and method for enhancing speech intelligibility for the hearing impaired |
US7277855B1 (en) * | 2000-06-30 | 2007-10-02 | At&T Corp. | Personalized text-to-speech services |
US6876968B2 (en) * | 2001-03-08 | 2005-04-05 | Matsushita Electric Industrial Co., Ltd. | Run time synthesizer adaptation to improve intelligibility of synthesized speech |
US6810378B2 (en) | 2001-08-22 | 2004-10-26 | Lucent Technologies Inc. | Method and apparatus for controlling a speech synthesis system to provide multiple styles of speech |
US7305340B1 (en) * | 2002-06-05 | 2007-12-04 | At&T Corp. | System and method for configuring voice synthesis |
US8768701B2 (en) | 2003-01-24 | 2014-07-01 | Nuance Communications, Inc. | Prosodic mimic method and apparatus |
US8340972B2 (en) * | 2003-06-27 | 2012-12-25 | Motorola Mobility Llc | Psychoacoustic method and system to impose a preferred talking rate through auditory feedback rate adjustment |
US7881934B2 (en) * | 2003-09-12 | 2011-02-01 | Toyota Infotechnology Center Co., Ltd. | Method and system for adjusting the voice prompt of an interactive system based upon the user's state |
US8886538B2 (en) | 2003-09-26 | 2014-11-11 | Nuance Communications, Inc. | Systems and methods for text-to-speech synthesis using spoken example |
US7865365B2 (en) * | 2004-08-05 | 2011-01-04 | Nuance Communications, Inc. | Personalized voice playback for screen reader |
US7490042B2 (en) | 2005-03-29 | 2009-02-10 | International Business Machines Corporation | Methods and apparatus for adapting output speech in accordance with context of communication |
US8694319B2 (en) | 2005-11-03 | 2014-04-08 | International Business Machines Corporation | Dynamic prosody adjustment for voice-rendering synthesized data |
TWI376925B (en) | 2007-08-28 | 2012-11-11 | Micro Star Int Co Ltd | Adjusting device and method of notification sound according to the environment |
US8374873B2 (en) * | 2008-08-12 | 2013-02-12 | Morphism, Llc | Training and applying prosody models |
US20100057465A1 (en) | 2008-09-03 | 2010-03-04 | David Michael Kirsch | Variable text-to-speech for automotive application |
US20120265533A1 (en) * | 2011-04-18 | 2012-10-18 | Apple Inc. | Voice assignment for text-to-speech output |
US8914290B2 (en) * | 2011-05-20 | 2014-12-16 | Vocollect, Inc. | Systems and methods for dynamically improving user intelligibility of synthesized speech in a work environment |
US9368104B2 (en) * | 2012-04-30 | 2016-06-14 | Src, Inc. | System and method for synthesizing human speech using multiple speakers and context |
CN103543979A (en) | 2012-07-17 | 2014-01-29 | 联想(北京)有限公司 | Voice outputting method, voice interaction method and electronic device |
US9728202B2 (en) * | 2013-08-07 | 2017-08-08 | Vonage America Inc. | Method and apparatus for voice modification during a call |
WO2015047248A1 (en) | 2013-09-25 | 2015-04-02 | Intel Corporation | Improving natural language interactions using emotional modulation |
US20160379638A1 (en) | 2015-06-26 | 2016-12-29 | Amazon Technologies, Inc. | Input speech quality matching |
US10091355B2 (en) * | 2016-02-19 | 2018-10-02 | International Business Machines Corporation | Virtual voice response agent individually configured for a user |
-
2017
- 2017-01-13 US US15/406,213 patent/US10586079B2/en active Active
-
2020
- 2020-01-14 US US16/742,006 patent/US11531819B2/en active Active
-
2022
- 2022-01-20 US US17/580,289 patent/US20220148566A1/en active Pending
Also Published As
Publication number | Publication date |
---|---|
US20180182373A1 (en) | 2018-06-28 |
US10586079B2 (en) | 2020-03-10 |
US20200151394A1 (en) | 2020-05-14 |
US11531819B2 (en) | 2022-12-20 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11531819B2 (en) | Text-to-speech adapted by machine learning | |
Wani et al. | A comprehensive review of speech emotion recognition systems | |
Hildebrand et al. | Voice analytics in business research: Conceptual foundations, acoustic feature extraction, and applications | |
US9916825B2 (en) | Method and system for text-to-speech synthesis | |
Piazza et al. | Mothers consistently alter their unique vocal fingerprints when communicating with infants | |
Martin et al. | Mothers speak less clearly to infants than to adults: A comprehensive test of the hyperarticulation hypothesis | |
Miyazawa et al. | Vowels in infant-directed speech: More breathy and more variable, but not clearer | |
Yilmazyildiz et al. | Review of semantic-free utterances in social human–robot interaction | |
Rachman et al. | DAVID: An open-source platform for real-time transformation of infra-segmental emotional cues in running speech | |
Holt et al. | Speech perception within an auditory cognitive science framework | |
Creel et al. | How talker identity relates to language processing | |
EP3151239A1 (en) | Method and system for text-to-speech synthesis | |
Gambi et al. | Prediction and imitation in speech | |
Johar | Emotion, affect and personality in speech: The Bias of language and paralanguage | |
Arias et al. | Beyond correlation: acoustic transformation methods for the experimental study of emotional voice and speech | |
Patnaik | Speech emotion recognition by using complex MFCC and deep sequential model | |
Caponetti et al. | Biologically inspired emotion recognition from speech | |
Schuller et al. | Synthesized speech for model training in cross-corpus recognition of human emotion | |
Postma-Nilsenová et al. | Auditory perception bias in speech imitation | |
Cave et al. | The use of speech recognition technology by people living with amyotrophic lateral sclerosis: a scoping review | |
Cohn et al. | Prosodic alignment toward emotionally expressive speech: Comparing human and Alexa model talkers | |
Larrouy-Maestri et al. | The Sound of Emotional Prosody: Nearly 3 Decades of Research and Future Directions | |
Rilliard et al. | Paradigmatic variation of vowels in expressive speech: Acoustic description and dimensional analysis | |
Chetouani et al. | Time-scale feature extractions for emotional speech characterization: applied to human centered interaction analysis | |
Chaurasiya | Cognitive hexagon-controlled intelligent speech interaction system |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: SOUNDHOUND, INC., CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:MONT-REYNAUD, BERNARD;ALMUDAFAR-DEPEYROT, MONIKA;SIGNING DATES FROM 20170201 TO 20170202;REEL/FRAME:058714/0607 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
AS | Assignment |
Owner name: ACP POST OAK CREDIT II LLC, TEXAS Free format text: SECURITY INTEREST;ASSIGNORS:SOUNDHOUND, INC.;SOUNDHOUND AI IP, LLC;REEL/FRAME:063349/0355 Effective date: 20230414 |
|
AS | Assignment |
Owner name: SOUNDHOUND AI IP HOLDING, LLC, CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:SOUNDHOUND, INC.;REEL/FRAME:064083/0484 Effective date: 20230510 |
|
AS | Assignment |
Owner name: SOUNDHOUND AI IP, LLC, CALIFORNIA Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:SOUNDHOUND AI IP HOLDING, LLC;REEL/FRAME:064205/0676 Effective date: 20230510 |
|
AS | Assignment |
Owner name: SOUNDHOUND AI IP, LLC, CALIFORNIA Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:ACP POST OAK CREDIT II LLC, AS COLLATERAL AGENT;REEL/FRAME:067698/0845 Effective date: 20240610 Owner name: SOUNDHOUND, INC., CALIFORNIA Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:ACP POST OAK CREDIT II LLC, AS COLLATERAL AGENT;REEL/FRAME:067698/0845 Effective date: 20240610 |
|
AS | Assignment |
Owner name: MONROE CAPITAL MANAGEMENT ADVISORS, LLC, AS COLLATERAL AGENT, ILLINOIS Free format text: SECURITY INTEREST;ASSIGNOR:SOUNDHOUND, INC.;REEL/FRAME:068526/0413 Effective date: 20240806 |