JP2013011902A - Method for generating speech by processing text by using non-language dependent rhythm markup and device for the same - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique for modeling a speech having desired rhythm characteristics.SOLUTION: A set of tags for defining rhythm characteristics is prepared, and a selected tag is arranged in a proper place of a text main body. Each tag imposes a constraint on the rhythm characteristics of a speech to be generated by processing the text. A set of equations to be solved so that a curve for defining the rhythm characteristics across a range of words and phrases can be generated and a set of equations to be solved so that a curve for defining the rhythm characteristics of the respective words in the words and phrases can be generated are generated by processing of the speech text and the tags. The data defined by the curve is used together with the text so that the speech having the rhythm characteristics defined by the tag can be generated. The set of tags is generated by the reading of the training text to be read by a target speaker, a training corpus on which the rhythm characteristics of the target speaker are reflected is generated, and then the training corpus is analyzed so that the tag for modeling the rhythm characteristics of the training corpus can be generated.

Description

This application takes advantage of US Patent Application No. 60 / 230,204 filed on September 5, 2000 and US Patent Application No. 60 / 236,002 filed on September 28, 2000. Both of which are hereby incorporated by reference in their entirety.
The present invention relates generally to improvements in the representation and modeling of phenomena that are continuous and subject to physiological constraints. In particular, the invention relates to the creation and use of a set of tags that define signal characteristics and tag processing to produce signals having the characteristics defined by the tags.

  A text-to-speech system takes text inputs, usually words and sentences, and converts these inputs into spoken words and sentences. The text-to-speech system employs a model of a particular speaker's speech to build a stock table of speech units and prosodic models that respond to each phonetic text unit. The prosodic features of speech are the rhythmic and intonation features of speech. The system then assembles the speech units, puts them in the order represented by the text, and plays back the speech units. A typical text-to-speech system analyzes text to predict phone sequences, performs duration modeling to predict the length of each phone, performs intonation modeling to predict pitch contours, Signal processing is performed to create speech speech by combining the results of different analyzes and modules.

  Many prior art text-to-speech systems infer prosodic information from text that produces synthetic speech. Prosodic information includes speech rhythm, pitch, accent, volume, and other features. Text usually contains only a small amount of information from which prosodic information can be estimated. Thus, prior art text-to-speech systems tend to be moderately designed. A moderately designed system generates ambiguous prosody based on the theory that if an accurate prosody cannot be determined, an ambiguous prosody wins over an inaccurate prosody. As a result, prosodic models tend to be moderately designed as well, and do not have the ability to model prosodic variations found in natural speech. These variations give natural speech the ability to match any given pitch contour or to convey a wide range of impressions, such as personal speech styles and emotions. This lack of variation in speech produced by prior art text-to-speech systems contributes significantly to the artificial speech produced by many such systems.

  In many applications, it is desirable to use a text-to-speech system that can perform interaction. For example, a text-to-speech system can be used to generate speech for a telephone menu system that provides a spoken response to customer input. Such a system may optionally include state information corresponding to concepts, goals, and intentions. For example, if the system generates a set of words that represent a single proper noun, such as “Wells Fargo Bank”, the synthesized speech should include an audio feature that conveys that the word set is a single noun. It is. In other cases, it may be necessary to convey by impression that a word is particularly important or that a word requires confirmation. In order to convey an accurate impression, the generated speech must have appropriate prosodic features. Prosodic features that can be advantageously defined for the speech that is generated include pitch, amplitude, and any other features that are necessary to give the speech a natural voice and convey the desired impression.

  Therefore, there is a need for a system of tags that can define prosodic features in sufficient detail to model speech having the desired features and systems that process the tags and generate speech having features defined by the tags. Is done.

  The present invention recognizes the need for a system that generates speech with desired prosodic features. To this end, the system includes the generation and processing of a set of tags that can be used for modeling phenomena that are continuous and subject to physiological constraints. Speech prosodic features are an example of such a phenomenon, and a set of tags can be created to represent a particular speaker's speech prosodic features or other desired prosodic features. These tags can be applied to the text at the appropriate place in the text and can define the prosodic features of the speech generated by processing the text. The set of tags defines the prosodic features in sufficient detail to be able to accurately model the speech having the prosodic features of the original speech from which the tag was created by processing the tag with the text. Including this level of detail can make the tag non-language dependent. This is because in other cases, information provided by knowledge of the prosodic features of the language used can be provided using tags. In this way, a text-to-speech system that employs a set of tags according to the present invention can generate accurate prosody for all languages and generate accurate prosody for mixed-language text. Can do. For example, a text-to-speech system that employs the teachings of the present invention can accurately process English text blocks containing French citations, as well as accurate prosodic features in the English portion of the speech, as well as the It is possible to generate a speech with accurate prosodic features in the French part of the speech.

  In order to provide an accurate representation of speech, it is preferable to include information that defines a compromise between tags, based on information in the tags and default information that defines how the tags relate to each other during tag processing A compromise is made. Many speech units affect the characteristics of other speech units. Adjacent units have a particular tendency to influence each other. If the tags used to define adjacent units, such as syllables, words, or word groups, contain competing instructions for prosodic feature assignment, depending on the priority of the information and how conflicts and compromises are handled Adjustments are made. For example, adjacent words or phrases are each adjusted. Alternatively, if the tag information indicates that one of the adjacent words or phrases is dominant, an appropriate adjustment will be made to the other word or phrase.

  A set of tags can be defined by training, i.e. by analyzing features of a training text corpus read by a particular speaker. Tags can be defined using the identified features. For example, if the training corpus shows that the speaker has a basic speech frequency of 150 Hz and the speaker's speech pitch is increased by 50 Hz at the end of the question, the fundamental frequency of the generated speech is set to 150 Hz. The tag can be defined to increase the pitch by 50 Hz at the end of the question.

  Once the tag is established, it can be entered into a text body where it is desirable to generate speech. This can be done by simply entering the appropriate tag into the text using an editor. For example, if you want to establish a basic frequency of 150Hz with the word “are” accented by text-to-speech processing for the sentence “You are the weakest link”, add a tag to the sentence as follows: Is possible. <setbase = 150 /> You <stress strength = 4 type = 0.5 pos = * shape = -0.2s.03, -.1s.03, 0s0, 0.1s-0.1, 0.2s-0.1 /> are <slope =- 0.8 /> the weakest link.

  This results in a phrase curve having a pitch centered around 150 Hz, accented on the word “are”, and decreasing in pitch from the end of the word “are” to the end of the sentence. When data defined by text and tags is provided to the synthesizer, the way the synthesizer pronounces the sentence reflects the characteristics defined by the phrase curve. Further aspects of the tag and its operation are described below.

  As an alternative to entering tags using an editor, tags can be automatically placed in speech according to a programmed rule set. An example rule set that defines the pitch of a plain text may be, for example, setting a slope that hangs down over the course of a sentence and using a descending accent for the last word of the sentence. When these rules are applied to the text body, an appropriate tag is established for each plain text in the text body. Additional rules may be employed to define other sentence types and functions. Other tags can be established and applied to the text, for example to define volume (amplitude) and accent (emphasis).

  When the text body is created using a set of tags, the tags are processed. First, the phrase curve is calculated. A phrase curve is a curve representing prosodic features such as pitch, calculated over a range of phrases. In processing text using the attached tabs according to the present invention, a phrase curve can be created as appropriate by processing one minor phase at a time. Here, a small phrase is a phrase, a subordinate clause, or a coordinate clause. A sentence typically includes one or more small words. A boundary is provided to limit the ability to affect the preceding subword of the tag to a single subword. Next, the prosody is calculated for the phrase curve. Prosodic features for each individual word are calculated and their effect on each word is calculated. This calculation models the effect of accented words that appear, for example, in phrases. After calculating the prosody for the phrase curve, mapping from linguistic attributes to observable acoustic features is performed. The acoustic features are then applied to the speech generated by processing the text. An acoustic feature can be suitably represented by a single curve or set of curves each having a specific value at a specific time and representing a function of time. Since the speech is generated by the machine, the time when each speech component is generated is known. Therefore, a prosodic feature appropriate to a specific speech component can be expressed as a value at a time when the speech component is known to occur. The speech component can be provided as an input to the synthesizer, and observable acoustic feature values are also provided to the synthesizer to control the speech features.

  A more complete understanding of the present invention, as well as further features and advantages of the present invention, will be apparent from the following detailed description and the accompanying drawings.

FIG. 3 shows a text-to-speech process according to the present invention. It is a figure which shows the accent curve produced | generated by the process of the tag by this invention. It is a graph which shows the effect | action of the <step> tag by this invention. It is a graph which shows the effect | action of the <step> tag by this invention. It is a graph which shows the effect | action of the <slope> tag by this invention. It is a graph which shows the effect | action of the <phrase> tag by this invention. It is a figure which shows the effect | action and correlation of the <stress> tag by this invention. It is a figure which shows the effect | action and correlation of the <stress> tag by this invention. It is a figure which shows the effect | action and correlation of the <stress> tag by this invention. It is a figure which shows the effect | action and correlation of the <stress> tag by this invention. It is a figure which shows the effect | action and correlation of the <stress> tag by this invention. 4 is a graph showing compromise between tags according to the present invention. It is a graph which shows the effect | action of the fluctuation | variation of the strength of the tag by this invention. Fig. 6 is a graph showing the effect of different values of the "pdroop" parameter used in a tag according to the present invention. 4 is a graph showing the effect of different values of the “adroop” parameter used in a tag according to the invention. 4 is a graph showing the effect of different values of the “smooth” parameter used in a tag according to the invention. 6 is a graph showing the effect of different values of the “jittercut” parameter used in a tag according to the present invention. FIG. 6 is a diagram showing steps of a tag processing process according to the present invention. 6 is a graph illustrating an example of mapping linguistic positions to observable acoustic features according to the present invention. It is a graph which shows the effect | action of the nonlinear transformation performed in the text-speech process by this invention. 4 is a graph showing the effect of different values of the “add” parameter used in a tag according to the invention. Fig. 6 is a graph illustrating exemplary data modeling using tags according to the present invention. Fig. 6 is a graph illustrating exemplary data modeling using tags according to the present invention. Fig. 6 is a graph illustrating exemplary data modeling using tags according to the present invention. Fig. 6 is a graph illustrating exemplary data modeling using tags according to the present invention. Fig. 6 is a graph illustrating exemplary data modeling using tags according to the present invention. Fig. 6 is a graph illustrating exemplary data modeling using tags according to the present invention. Fig. 6 is a graph illustrating exemplary data modeling using tags according to the present invention. Fig. 6 is a graph illustrating exemplary data modeling using tags according to the present invention. FIG. 4 illustrates a process for creating and using tags according to the present invention. FIG. 2 illustrates an exemplary text-to-speech system according to the present invention. FIG. 5 illustrates a process for generating and using tags for defining and generating movement according to the present invention.

  The following description describes techniques for identifying the prosodic features of speech according to the present invention. First, the overall process of text / speech processing will be described. Next, a set of tags used for specifying prosodic features will be described. After describing the general structure and grammar of the tag, the tag, parameter, and value categories used in the tag will be described. Next, the action of each of several exemplary tags will be described, showing the action of different parameters, a compromise between competing tags, and representative attributes of other tags. Next, the description of the processing of the text body including the tag according to the present invention, the description of how to create and use the tag to generate the speech having the prosody characteristics of the target speaker, and the text-to-speech processing system according to the present invention. The explanation continues.

  FIG. 1 illustrates a process 100 for text-to-speech processing of a text body containing tags in accordance with the present invention. In step 102, the text body is analyzed and tags are extracted. In step 104, the tag is processed to determine values of acoustic features defined as a function of time by the tag, such as pitch and volume. In step 106, the text and values determined for the acoustic features are converted into linguistic symbols that are provided to the synthesizer. In step 108, a linguistic symbol is provided as an input to the synthesizer to generate speech having acoustic features defined by the tag.

  Tags are placed in the text body, usually between words, to define the prosodic features desired for speech generated by processing the text. Each tag imposes a constraint set on the prosody. The <step> and <stress> tags include a “strength” parameter that defines a relationship to other tags. Tags often contain conflicting information, and the “strength” parameter determines how to resolve the conflict. Further details of the “strength” parameter and its operation will be described later.

  Tags can be appropriately defined in XML, an extensible markup language format. XML is a universal format for structured documents on the World Wide Web and is described at www.w3.org/XML. It will be clear to those skilled in the art that the tag need not be implemented in XML syntax. Tags can be delimited by any arbitrary string (unlike “<” and “>” used in XML), and the internal structure of the tag may not follow the XML format, The tag can be identified, and can have an arbitrary structure in which necessary attributes can be set. It will also be appreciated that text need not be interposed between tags in a single character stream. Tags and text can flow in two parallel data channels, for example, as long as there is a means to synchronize the tag to the location in the corresponding text sequence.

  Tags can also be used when there is no text and the input consists only of a series of tags. Such input is appropriate when using these tags, for example, to model muscle mechanics for computer graphics applications. As an example, a tag can be used to control the movement of a simulated goldfish fin. In such a case, it is not necessary to separate the tag from the non-existent text, and a tag delimiter is only necessary to separate the tag from the next tag.

  Finally, it will be appreciated that the tag need not be represented as a serial data stream, but can be represented as a data structure in the computer's memory instead of as a serial data stream. For example, in an interactive system in which a computer program is generating text and tags, passing pointers or references to data structures that describe the text (if any), tags, and temporary relationships between texts, Can be the most efficient. The data structure describing the tag then contains information equivalent to the XML description, possibly along with other information used, for example, for debugging, memory management, or other auxiliary purposes.

A set of tags according to the present invention is described below. In this description, alphabetic strings are enclosed in quotation marks. Like the standard in XML notation, “?” Represents an optional token, “*” represents zero or more occurrences of the token, and “+” represents one or more occurrences of the token. The tag syntax is represented in the following format:
Tag = ”<” tagname AttValue ”*” /> ”
An example tag is
<set base = ”200” />. This tag sets the speaker's fundamental frequency to 200 Hz. In this example, “” <”” indicates the start of the tag, “set” is the action to be taken, ie setting the value of the specified attribute, and “base” is the attribute whose value is to be set. , “200” is a value to which the attribute “base” should be set, and “”> ”indicates the end of the tag.

  Each tag includes two parts. The first part is an action and the second part is a set of attribute-value pairs that control the details of the tag's operation. Most tags are self-contained “point” tags. To take into account the accuracy in defining when a tag will operate, the tag may include a “move” attribute. This attribute allows a tag to be placed at the beginning of a word, but leaves the action anywhere within the word. The use and operation of the “move” attribute will be described in more detail below.

  Tags are in four categories: (1) tags that set parameters, (2) tags that define phrases or points that build phrase curves, (3) tags that define accents of words, and (4) Classified as one of the tags that mark the boundary.

A parameter is set by a <set> tag having a grammar <set Att = value>. Here, “Att” is an attribute controlled by the tag, and value is a numerical value of the attribute. The <set> tag allows the following attributes:
max = value. This attribute sets the maximum allowed value, for example, the maximum frequency to be generated when the pitch is appropriately controlled, in hertz.
min = value. This attribute sets the minimum allowable value, for example, the minimum frequency to be generated when the pitch is appropriately controlled, in hertz.
smooth = value. This controls the response time of the mechanical system being simulated. When the pitch is being controlled, this parameter sets the smoothing time of the pitch curve in seconds to set the pitch step width.
base = value. This sets the speaker's baseline, i.e. the frequency with no tags at all.
range = mvalue. This sets the speaker pitch range in Hz.
pdroop = value. This sets the droop of the word curve to the fundamental frequency and is expressed in units of droop per second.
adroop = value. This sets the droop rate of the pitch trajectory toward the phrase curve, and is expressed in units of the droop rate per second.
add = value. This sets up a non-linearity in the mapping between the pitch trajectory over the range of the phrase and the pitch trajectory of individual words that have a local effect on the phrase. If the value of “add” is equal to 1, linear mapping is performed. That is, it has the same effect on the pitch regardless of whether the accent is in the high pitch region or the low pitch region. When the value of “add” is equal to 0, the effect of the accent is logarithmic, and a small accent changes the frequency more greatly when on a high word curve. If the value of “add” is greater than 1, it is performed slower than linear mapping.
jitter = value. In this case, the magnitude of the root mean square (RMS) of pitch jitter is set as a decimal number with the speaker range set to 1. Jitter is the degree of random pitch variation introduced to give a natural speech with processed speech.
jittercut = value. This sets the time scale for pitch jitter in seconds. The pitch jitter is correlated (1 / f) noise at intervals shorter than “jittercut”, and is uncorrelated noise, ie, white noise, at intervals longer than “jittercut”. A large value of “jittercut” defines a longer and smooth pitch value, while a small value of “jittercut” defines a short and irregular pitch change.

  The arguments provided to the <set> tag are retained for each voice until the text-to-speech process is complete and across word boundaries.

The <step> tag takes several arguments and operates on a phrase curve. The <step> tag takes the form <step by = value | to = value | strength = value>. The attributes of the <step> tag are as follows.
by = value. This defines the size of each step as a decimal with 1 as the speaker range. The steps in the phrase curve are smoothed by “smooth” time. The parameter “smooth” is defined above.
to = value. This is the frequency at which the step approaches, and is expressed as a decimal with the speaker range set to 1.
strength = value. This attribute controls how a particular <step> tag interacts with its neighboring tags. When the value of “strength” is high, the tag is dominant over the adjacent tag, and when the value of “strength” is low, the adjacent tag is dominant over the tag.

  The <slope> tag takes one argument and operates on a phrase curve. <slope> tag is <slope rate = value ”%”? >. This sets an increase / decrease rate of a phrase expressed by a small number where the range of speakers per second is 1. When the symbol “%” is present, its value represents an increase or decrease with respect to the ratio of the small phrase to the range per unit length.

  The <stress> tag defines a prosody for the phrase curve. Each <stress> tag defines a preferred shape and preferred height for the phrase curve. However, <stress> tags often define competing characteristics. In processing the <stress> tag, the preferred shape and height defined by the <stress> tag will change due to the requirement that these characteristics can be compromised with each other and that the pitch curve must be smooth. Is done. The <stress> tag has a form of <stress shape = (point ”.”) * point | strength = value | type = value>.

  The “shape” parameter specifies the ideal shape of the accent curve in terms of a set of points when there is no compromise with other stress tags or constraints.

  The “strength” parameter defines the linguistic strength of the accent. A zero-strength accent has no effect on the pitch. Accents with a strength much greater than 1 are followed exactly if there are no adjacent tags that are comparable to or greater than that. If there is an adjacent tag that is comparable or greater in strength, the accent is compromised with the adjacent tag, or the adjacent tag becomes dominant over the tag, depending on the strength of the adjacent tag . An accent whose intensity is approximately equal to 1 is a pitch curve with a smoothed accent.

  The “type” parameter controls whether the accent is defined by the average value of the pitch curve or by its shape. The value of the “type” parameter works when accents need to be compromised with adjacent tags. If the accent is much stronger than the adjacent tag, both the pitch shape and the average value are retained.

  However, if compromise is required, “type” determines which characteristic to compromise. If “type” has a value of 0, the accent retains its shape at the expense of the average pitch. If “type” has a value of 1, the accent maintains its average pitch at the expense of shape. When the value of “type” is between 0 and 1, a compromise is made between the shape and the average pitch within the range of compromise determined by the actual value of “type”.

The “point” parameter in the argument “shape” of the <stress> tag follows the following syntax.
point = float (X "s" | X "p" | X "y" | X "w") value. A point on the accent curve is specified as a (time, frequency) pair whose frequency is represented by a decimal number with the speaker range being 1. X is measured in seconds (s), phonemes (p), syllables (y), or words (w). Since it is preferable to constrain the accent curve to be smooth, the accent curve need not be specified in detail.

  FIG. 2 shows an exemplary accent curve 202 described by a stress tag having the values <stress strength = 10 type = 0.5 shape = 0.3s0, 0.15s0.3, 0s0.5, 0.15s0, 0.25s0 />. It is the graph 200 shown. Processing the tag generates points 204-214 and a curve 202 that fits the points 204-214. The fit of curve 202 to points 204-214 is preferably designed to produce a smooth curve that reflects the natural speech that is truly human speech.

  In addition to the tags described above, a <phrase> tag that inserts a word boundary is implemented. The <phrase> tag is typically used to mark a small phrase or breath group. Advance planning beyond the phrase tag is not performed. The prosody defined before <phrase> is totally independent of any tags that occur after the <phrase> tag.

As described above, any tag can include a “move” attribute. The “move” attribute instructs the tag to defer the action up to the point specified by the “move” attribute. The “move” attribute follows the following syntax:
AttValue = position | other_attributes
However, position = ”move” ”=” move_value,
move_value = ”ell”? motion * and
motion = (float | "b" | "c" | "e") ("r" | "w" | "y" | "p" | "s") "*" | "?

  Motion is evaluated in order from left to right. The position is modeled as a cursor starting from a tag if the move_value does not start with "" ell "". When starting with "" ell "", the last cursor position from the preceding tag is used as the starting point. Usually, tags are placed in words and the “move” attribute is used to place accents in words. Motion can be specified in terms of subphrases (r), words (w), syllables (y), phonemes (p) or accents (*). When tags are gathered at the beginning of a phrase, it is useful to specify motion for sub-phrases and words. The rules for identifying motion are as follows. A motion specified for a small phrase skips every break between phrases. The motion specified for a word skips every pause between words. The motion specified for a syllable treats one pause as one syllable. The motion specified for phonemes treats a short pause as a phoneme. When using “b”, “c”, or “e” as motion, the pointer moves to the beginning, center, or end of the nearest phrase, word, syllable, or phoneme, respectively. A move specified for a second moves the pointer by that number of seconds. Motion “*” (stress is placed) moves the pointer to the center of the next syllable where the stress is placed. The question mark (?) Does not move the pointer and serves to limit the motion following the question mark from crossing word boundaries. If an argument violates the constraint, a warning message is generated or the violating tag is ignored.

An example of a tag containing a “move” command is as follows.
<step move = * 0.5p by = 1 />
The action of this tag is to place a step on the pitch curve that has the steepest part 0.5 phonemes after the center of the syllable where the stress is first placed after the tag. The “move” attribute causes the tag to act at the desired point, not the location of the tag itself.

  3A to 3I show the action of various tags. FIG. 3A shows one <step to> tag that sets a single frequency, two <step to> tags that respectively set the same frequency, two <step to> tags that respectively set different frequencies, 3 is a graph 300 showing curves 302 to 306 generated as a result of processing each of the above. Curve 302 results from the tag <step strength = 10 to = 0.5 />. Curve 304 results from a first tag <step strength = 10 to = 0.5 /> followed by intervening text, followed by a second tag <step strength = 10 to = 0.5 />. . Curve 306 results from the first tag <step strength = 10 to = 0.5 /> followed by intervening text, followed by the second tag <step strength = 10 to = 0 />. .

  The <step by> tag simply inserts a step into the pitch curve. The tag <step by = X /> instructs the pitch after the tag to be higher by X Hz than the pitch before the tag. The tag changes pitch but does not force the pitch on either side of the tag to take any particular value. Therefore, the <step by> tag does not tend to compete with other tags. For example, when <step by = -50 /> is followed by <step by = -50 />, the frequency before the <step by = -50> tag is 100 Hz, and the frequency after the tag is 50 Hz. .

  FIG. 3B is a graph 310 showing curves 312 and 314. Curve 312 results from a series of tags <step to = 0.1 strength = 10 /> ... <step by = 0.3 strength = 10 />. Curve 314 results from a series of tags <step to = 0.1 strength = 10 /> ... <step by = 0.3 strength = 10 /> ... <step by = 0.3 strength = 10 />. Since constraints on the pitch curve are not competing, no compromise is necessary in this example.

  The word curve also has a <slope> tag associated with it. The <slope> tag tilts the phrase curve up or down with respect to the left side of the tag, that is, the side temporally preceding the tag, depending on the argument. The slope tag replaces the current slope value. For illustration purposes, the result of a series of tags <slope rate = 1 /> ... <slope rate = 0 /> has a slope of zero. The tag <slope rate = 0 /> replaces the slope set by the tag <slope rate = 1 /> and any preceding tags.

  FIG. 3C is a graph 320 that includes curves 322-328. Curve 322 results from the tag <slope rate = 0.8 />. Curve 324 results from a series of tags <slope rate = 0.8 /> ... <step by = 0.1 strength = 10>. Curve 326 results from the tag ... <slope rate = 0.8>. Curve 328 results from a series of tags <slope rate = 0.8 /> ... <set slope = 0.1 />. Curves 322 to 328 represent the slope starting from the word boundary, the slope delayed by 0.25 seconds, the slope with a small step, and the slope going up and down, respectively. No compromise is necessary because the <slope> tag with the new value replaces any value imposed by the preceding <slope> tag.

  FIG. 3D shows the effect of the <phrase> tag. Graph 330 shows a curve 332 representing a flat tone. Curve 332 is followed by phrase boundary 334. The phrase boundaries are followed by curves 336-339 showing tones of various amplitudes. Graph 330 shows a series of tags <stress strength = 4 type = 0.8 shape = 0.1s0.3, 0.1s0.3 /> ... <phrase /> ... <stress strength = 4 type = 0.1 shape = various / The effect of> is shown. The <phrase> tag prevents the falling tone after 0.42 seconds from affecting the flat tone 0.42 seconds ago.

  The <phrase> tag marks the boundary where the pre-planning stops and is preferably placed at the small phrase boundary. A subphrase is usually a single phrase, or a subordinate or equal clause that has a smaller range than the full sentence. A typical human speech is characterized by planning or preparing a prosody, which is done a few syllables before it is generated. For example, by preparing, it is possible to prevent a speaker from smoothly compromising a difficult tone combination, exceeding a pleasant pitch range, or falling below it. The system for placing and processing tags according to the present invention can model this aspect of human speech generation, and controls the range to be prepared by using <phrase> tags. That is, the placement of the <phrase> tag controls the number of syllables that are compromised or otherwise prepared. The phrase tag acts as a one-way restriction element, so that the tag before the <phrase> tag can affect the destination, but the tag after the <pharse> tag does not affect it before that.

  3E-3I show the action of the <stress> tag. The <stress> tag allows you to accent words or syllables. The <stress> tag always includes at least the following three elements. The first element is the ideal “platon” shape of the accent, which is the shape that the accent has when there is no adjacent accent and when it is spoken very slowly. The second element is an accent type. The third element is the strength of the accent. Strong accents tend to retain their shape, while weak accents tend to be dominated by adjacent accents.

  The behavior of speaking is a compromise between these trends, and any system that seeks to model speech under these circumstances must have a way to compromise these trends. The argument “strength” of the <stress> tag controls the interaction between tags representing competing requirements. FIG. 3E is a graph 340 illustrating the interaction of a type 0.8 flat tone followed by a type 0 purely descending tone. The flat tone type is 0.8, i.e. the type value is close to 1, so it tends to keep its average pitch at the expense of shape. Since the descending tone type is 0, the shape is maintained at the expense of the average pitch. Curves 342A-342G are a series of tags <stress strength = 4 type = 0.8 shape = -0.1sY, 0.1sY /> ... <stress strength = 4 type = 0 shape = -0.2.03, -.1s.03 , 0s0, 0.1s-0.1, 0.2s-0.1 />. However, the value of Y changes from 0.1 to 0.5 in increments of 0.1.

  FIG. 3F is a graph 350 illustrating the interaction of a type 0.8 flat tone followed by a type 0.1 falling tone. The flat tone type is 0.8, i.e. the type value is close to 1, so it tends to keep its average pitch at the expense of shape. The descending tone type is 0.1, which indicates a slight tendency to compromise shape to maintain pitch. Curves 352A-352G are a series of tags <stress strength = 4 type = 0.8 shape = -0.1sY, 0.1sY /> ... <stress strength = 4 type = 0.1 shape = -0.2.03, -.1s.03 , 0s0, 0.1s-0.1, 0.2s-0.1 />. However, the value of Y changes from 0.1 to 0.5 in increments of 0.1. It can be seen that curves 352A-352G are slightly closer to a point in the area of the falling tone, as the tone indicates a slight pitch preference.

  FIG. 3G is a graph 360 illustrating the interaction of a type 0.8 flat tone followed by a type 0.5 falling tone. The flat tone type is 0.8, i.e. the type value is close to 1, so it tends to keep its average pitch at the expense of shape. Since the descending tone type here is 0.5, it shows a strong tendency to maintain that pitch, resulting in a compromise between pitch and shape. Curves 362A-362G are a series of tags <stress strength = 4 type = 0.8 shape = -0.1sY, 0.1sY /> ... <stress strength = 4 type = 0.5 shape = -0.2.03, -.1s.03 , 0s0, 0.1s-0.1, 0.2s-0.1 />. However, the value of Y changes from 0.1 to 0.5 in increments of 0.1. Although the curves 362A to 362G still maintain their shapes, it can be seen that both are strongly compressed to maintain the pitch.

  FIG. 3H is a graph 370 illustrating the interaction of a type 0.8 flat tone followed by a type 0.8 falling tone. The flat tone type is 0.8, i.e. the type value is close to 1, so it tends to keep its average pitch at the expense of shape. The descending tone type here is 0.8, so it shows a very strong tendency to maintain its pitch and only a weak tendency to maintain its shape. Curves 372A-372G are a series of tags <stress strength = 4 type = 0.8 shape = -0.1sY, 0.1sY /> ... <stress strength = 4 type = 0.8 shape = -0.2.03, -.1s.03 , 0s0, 0.1s-0.1, 0.2s-0.1 />. However, the value of Y changes from 0.1 to 0.5 in increments of 0.1. In curves 372A through 372G, it can be seen that the shape advantage can be forced to reduce the pitch near its midpoint, but the tendency to maintain the shape is significantly reduced. If the first tone, ie the flat tone, has a low pitch, the pitch curve has a strong tendency to rise between the two tones in order to maintain an accurate pitch in the middle of the second accent.

  FIG. 3I is a graph 380 illustrating the interaction of a type 0.8 flat tone followed by a type 1 falling tone. The flat tone type is 0.8, i.e. the type value is close to 1, so it tends to keep its average pitch at the expense of shape. Since the descending tone type is 1 here, the shape is compromised as necessary in order to maintain the pitch and maintain the pitch strictly. Curves 382A-382G are a series of tags <stress strength = 4 type = 0.8 shape = -0.1sY, 0.1sY /> ... <stress strength = 4 type = 1 shape = -0.2.03, -.1s.03 , 0s0, 0.1s-0.1, 0.2s-0.1 />. However, the value of Y changes from 0.1 to 0.5 in increments of 0.1. From curves 382A to 382G, it can be seen that the descending tone is here defined entirely by the pitch.

  You can see another example of compromise between tags when the accents are close together. The result of overlapping two accents is lower than the sum of both accents. Instead, a single accent is formed that is the same size and shape but has twice the strength of any individual accent.

  FIG. 4 is a graph 400 showing the results of a fixed accent curve 402 with a peak at 0.83 s and accent curves 404A-404E that move toward the curve 402 and gradually move until the curve 404F overlaps the curve 402. It is. Curve 402 and curves 404A-404E are a series of tags <stress strength = 4 shape =-. 15s0, -.1s0, -.05s.1, 0s.3, .05s.1, .1s0, .15s0 type = 0.5 /> ... <stress strength = 4 shape =-. 15s0, -.1s0,-. 05s.1, 0s.3, .05s.1, .1s0, .15s0 type = 0.5 /> . Curve 404F is the result of combining the curves represented by curve 402 and curve 404E. It can be seen that the peak of curve 404F is lower than the sum of the peaks of curve 402 and curve 404E.

  All accent tags include a “strength” parameter. A tag's “strength” parameter affects how the accent defined by the tag affects adjacent accents. In general, strong accents, ie, accents defined by tags with relatively high strength parameters, tend to retain their shape, while weak accents with relatively low strength parameters tend to be dominated by adjacent accents There is.

  FIG. 5 is a graph 500 illustrating the interaction between a descending tone, a preceding strong high tone, and a subsequent weakly high tone with varying strength of the descending tone. Curves 502-512 represent a sequence of tones where the strength of the falling tone is incremented by 1 from 0 to 5. Curves 502-512 are a series of tags <stress strength = 4 type = 0.3 shape = -0.1s0.3, 0.1s0.3 /> ... <stress strength = X type = 0.5 shape =-. 15s2,-. 1s.2, 0s0, .1s-.2, .15s-.2 /> ... By processing <stress strength = 2.5 type = 0.3, shape = -0.1s0.3, 0.1s0.3 /> Generated. However, X changes in increments of 1 from 0 to 5. Curve 514 shows a falling tone that follows a strong flat tone without following a weak flat tone. It can be seen that the descending tone having a strength of zero, shown by curve 502, is dominated by a fully adjacent tag. Curves 504-512 show how the falling tone tends to retain its shape as the strength increases, disturbing adjacent tags more and more. The shape of the falling tone shown by curve 512 is substantially the same as curve 514, and shows how the strength of the falling tone prevails over the subsequent weak flat tone.

  Another factor that affects the word curve is drooping, a regular drop in pitch that often occurs in words. This factor is represented by the parameter pdroop that sets the rate at which the phrase curve decays towards the speaker's fundamental frequency. Points near the <step to> tag are relatively unaffected, especially when they have high strength parameters. This is because the attenuation defined by the pdroop parameter acts with time, and a relatively slight attenuation occurs near the frequency setting. The farther away from the <set to> tag, the stronger the influence.

  Since the value of “pdroop” sets the attenuation rate of the phrase curve by an exponent, the step is attenuated by 1 / pdroop seconds. Typically, the speaker's pitch trajectory is planned in advance, ie, continuous or intermittent adjustments are made to achieve a smooth pitch trajectory. To model this pre-plan, the pdroop parameter has the ability to cause attenuation in the phrase curve regardless of whether the pdroop parameter is set before or after the <set to> tag.

  For example, FIG. 6 shows a graph 600 representing the occurrence of a positive <step to> tag 601 at the beginning of a phrase. The tag is <step to = 0.5 strength = 3 set pdroop = X />. However, X takes values of 0, 0.5, 1, and 2, and the results are the phrase curves 602 to 608, respectively. If the pdroop parameter used for the tags defining the curves 604 to 608 is not zero, the result is that as the value of pdroop increases, the curves 604 to 608 drop toward the fundamental frequency of 100 Hz with increasing droop rate. I can see that.

  A parameter similar to “pdroop” is “adroop”. The “adroop” parameter returns the pitch trajectory back to the word curve, which can limit the amount of pre-planning assumed during tag processing. An accent 1 / adroop seconds away from a given point has little effect on the local trajectory of the pitch around that point.

  Figure 7 shows a series of tags <set adroop = X /> ... <set smooth = 0.08 /> ... <step to = 0 strength = 3 /> ... <stress shape =-. 1s0. It is a graph 700 showing curves 702 to 708 generated by processing 05s0, .05s.3, .1s.3 strength = 3 type = .5 />. However, X takes values of 0, 1, 3, and 10, respectively. Here, the pitch curve is a constant 100 Hz, and the “adroop” parameter attenuates the curves 702-708 towards the pitch curve as the distance from the accent increases. The attenuation rate increases as the value of “adroop” increases.

  FIG. 8 is a graph 800 showing curves 802-808, representing accents with different smoothing times. Curves 802-808 are a series of tags <set smooth = X /> ... <stress strength = 4 shape =-. 15s0, -.1s0, -.05s.1, 0s.3, -15s0, .1s0, Generated by processing -05s.1 />. However, X takes the values of 0.004, 0.10, 0.14, and 0.2, respectively. The “smooth” parameter is preferably set to a time required for the speaker to change the normal pitch in order to intentionally change the pitch in the middle of the extended vowel, for example. Curve 808 with a “smooth” value of 0.2 is substantially too smooth with respect to the shape of the accent.

  FIG. 9 is a graph 900 illustrating the effect of the “jittercut” parameter. The “jittercut” parameter is used to introduce random variations into the phrase to produce a more realistic speech. Human speakers do not speak the same phrase or sentence exactly the same every time they say it. By using the “jittercut” parameter, it is possible to introduce some variation characteristics of human speakers.

  A graph 900 shows curves 902 to 906 in which the values of “jittercut” are set to 0.1, 0.3, and 1, respectively. The value of “jittercut” used to generate the curve 902 is approximately scaled by the average word length, thus causing significant variation within the word. The value of “jittercut” used to generate curve 906 is a measure of a phrase and produces variation over the range of the phrase, but produces little variation within the word.

  FIG. 10 shows a process 1000 for processing a tag and determining the value of an acoustic feature defined by the tag. Process 1000 may be employed as step 104 of process 100 of FIG. Process 1000 proceeds by building one or more linear equations for the pitch at each point in time and then solving the set of equations. Each tag represents a prosody constraint, and as each tag is processed, additional equations are added to the set of equations.

  In steps 1002-1008, the step and slope tags are processed to create a set of constraints on the phrase curve, each represented by a linear equation defined by the tags.

In step 1002, a linear equation is generated for each <step by> tag. In each equation, pt _{+ w} -pt _-w = stepsizet, w = 1 + [smooth / 2 [Delta] _t ] is half the smoothing width, and t is the tag position. Each <step to> tag adds one equation of the form p _t = target. Here, target is the value of the argument “to”.

In step 1004, a set of constraint equations is generated for each <slope> tag. One equation is added for each time t. The equation takes the form of P _{t + 1} −p _t = slope _t · Δt. Wherein, p _t is the phrase curve, slope _t is a preceding <slope> tag attributes "rate", Delta] t is the spacing prosody calculations, usually a 10 ms.

  The equation generated from the <slope> tag associates each point with an adjacent point. Solving the equation results in a continuous phrase curve, i.e., a phrase curve without steep steps or jumps. Such a continuous phrase curve reflects the actual human speech pattern, and the rate of change is continuous because the vocal cord muscles do not react immediately.

In step 1006, one equation is added to each point where "pdroop" is not zero. Each of these equations tends to pull the phrase curve to zero. Each droop equation has the form s ^[drop] = ploop · Δt. Each equation has a separate small effect, but the effect is cumulative, eventually bringing the phrase curve to zero.

In steps 1008-1012, the equations are solved. Overall, there are m + n equations (n is an unknown). The value of m is the number of step tags + (n−1). all values of p _t is unknown. Since there are more equations than unknowns, the equations lead to overdetermination of unknown values. It is therefore necessary to find one solution that properly solves all equations. Those familiar with the field of solving equations will recognize that this can be characterized as a “weighted least squares” problem with a standard algorithm in the solution.

In step 1008, the preferred implementation represents the equation in the form of a matrix with s · a · p = s · b. Here, s is the m × m diagonal matrix of strength, a (a is m × n) includes a coefficient of p _t in equation, b (which is m × 1) right-hand side of equation (constant) including. P is an m × 1 column vector. Next, in step 1010, the equation is transformed into the normal form of the solution, that is, at ^* s ² * a * p = a * s ² * b. The reason is that when doing so, the left side is because including narrow banded diagonal matrix of strip width ^{(a t · s 2 · a} ). The band width is usually w or less, which is much smaller than n or m. The cost of solving the equation is measured not as n ³ in the general case but as w ² n in the case of a band diagonal matrix, so it is important that the band width is narrow. In the present invention, this measurement reduces the computational cost by a factor of 1000 and ensures that the number of CPU cycles required to process each second of speech is constant. Finally, in step 1012, the equations are solved using matrix analysis. One skilled in the art will recognize that steps 1008-1012 may be replaced with other algorithms that may yield equivalent results.

To give an example, assume a sampling interval dt = 0.01 s, smooth = 0.04 s, ploop = 1, and the following tags:
<slope rate = 1 pos = 0s />
<step to = 0.3 strength = 2 pos = 0s />
<step by = 0.5 pos = 0.04 strength = 0.7 />
This results in the following set of equations: In the formula, “#” and the material in each subsequent line represent a comment and are not part of the equation.
1: p0 = 0.3; s1 = 2 # step to
2: p6-p2 = 0.5; s2 = 0.7 # step by
3: p1-p0 = 0.01; s3 = 1 # slope
4: p2-p1 = 0.01; s4 = 1 # slope
5: p3-p2 = 0.01; s5 = 1 # slope
6: p4-p3 = 0.01; s6 = 1 # slope
11: p0 = 0; sll = 0.01 # pdroop
12: p1 = 0; s12 = 0.01 # pdroop
13: p2 = 0; s13 = 0.01 # pdroop
The matrix “a” is as follows:
1 0 0 0 0 0 0 0 0 0
0 0 -1 0 0 0 1 0 0
-1 1 0 0 0 0 0 0 0 0
0 -1 1 0 0 0 0 0 0 0
0 0 -1 1 0 0 0 0 0 0
...
1 0 0 0 0 0 0 0 0 0
0 1 0 0 0 0 0 0 0 0
0 0 1 0 0 0 0 0 0 0
...
Here, each row corresponds to the left side of the above equation. Each column corresponds to a time value.

The right hand side of the above equation yields a “b” matrix. Each row of the “b” matrix corresponds to one right side of the above equation.
0.3
0.5
0.01
0.01
0.01
0.01
...
0
0
0
...
The diagonal elements of strength s _{i, i} are as follows.
[2 0.7 1 1 1 1 ... 0.01 0.01 0.01 ...]
Here, each entry corresponds to one equation.

  In order to achieve natural speech, it is important to achieve continuity between small phrases. This can be achieved by calculating the whole sentence at once. However, this approach leaves the tag at the beginning of the phrase to affect the pitch near the end of the preceding phrase, which is undesirable. In an actual human speech pattern, the pitch and accent at the beginning of a phrase do not affect the pitch near the end of the preceding phrase. Humans tend not to consider the pitch at the beginning of the next word, but to make any necessary pitch shift during the pause between words or at the beginning of the following word after the word is finished.

Thus, continuity is achieved by calculating the prosody of one small phrase at a time. However, completely separated rather than computing the phrases, looking back the value of p _t near the end of the preceding word, by substituting the equation them as known values to calculate the phrase.

  The next stage of tag processing is the calculation of the pitch curve. The pitch curve describes the pitch behavior of individual words and other smaller elements of the phrase rather than the entire phrase. The pitch trajectory is calculated based on the phrase curve and the <stress> tag. The algorithm described above with respect to process steps 1002-1012 is applied, but the set of equations is different.

In step _{1014, e t + 1 = e} t = 0 of the continuity equation is expressed in the form and _{-e t + 1 + 2e t -e} t + 1 = is represented by 0 mode, set the point of further equations representing the smoothness, Applies to Each equation has strength s ^[smooth] = π / 2 · smooth / Δt. The smoothness equation implies that there are no sharp corners in the pitch trajectory. Mathematically, the “smoothness” equation ensures that the second derivative remains small. This requirement is due to the physical constraint that the muscles used to perform the prosody all have non-zero mass, so it is smoothly accelerated and cannot respond seizureally.

In step 1016, a set of n "droop" equations is applied. These equations affect the pitch trajectory in the same way that the droop equation described above affects the phrase curve. Each “droop” equation has the form s ^[drop] = adloop · Δt. These equations cause the pitch trajectory to droop into the phrase curve, as opposed to the pdroop parameter described above, which tends to pull the phrase curve down to zero.

「type」が０から増大するにつれ、この方程式のstrengthもまた０（アクセントが平均ピッチを犠牲にして形状を保持することを意味する）から「strength」（アクセントが形状を犠牲にして平均ピッチを保持することを意味する）に増大することが見て取れる。 In steps 1018-1020, one equation is introduced for each <stress> tag. Each such equation places constraints on the shape of the pitch trajectory. In step 1018, the shape of the <stress> tag is first linearly interpolated to form a continuous set of targets. shape = t ₀ , x ₀ , t ₁ , x ₁ , t ₂ , x ₂ ,..., t _j , x _j are interpolated, and X _k , X _{k + 1} , X _{k + 2} ,. , X _j . However, k = t ₀ / Δt is the index of the first point of the accent shape, and J = t _j / Δt is the index of the end of the accent. The equation that constrains the average pitch of the accent is as follows, with s ^[pos] = strength · sin (type · π / 2).

As “type” increases from 0, the strength of this equation also changes from 0 (meaning that the accent retains shape at the expense of the average pitch) to “strength” (accent increases the average pitch at the expense of shape). Can be seen to increase).

式中、

は、アクセントにわたるピッチの軌跡の平均値であり、

はアクセントの形状である。平均を差し引くことで、これらの方程式によりアクセントが語句曲線の上にあるのか下にあるのかが制約されないようにする。方程式は、アクセントが語句曲線の上にあるのか下にあるのかを制約するのではなく、アクセントの形状のみを制約する。各アクセントは、ｓ^{［ｓｈａｐｅ］}＝ｓｔｒｅｎｇｔｈ・ｃｏｓ（ｔｙｐｅ・π／２）／（Ｊ−ｋ＋１）という「strength」値を有する。ステップ１０２２において、上記例において説明したものと同様の行列解析を用いて、該方程式を解く。 In step 1020, further equations are also generated for each point, i.e., for accents k through j. These equations define the shape of the accent and take the following form:

Where

Is the average pitch trajectory over the accent,

Is the shape of the accent. By subtracting the average, these equations do not constrain whether the accent is above or below the phrase curve. The equation does not constrain whether the accent is above or below the phrase curve, but only the shape of the accent. Each accent has a “strength” value of s ^[shape] = strength · cos (type · π / 2) / (J−k + 1). In step 1022, the equation is solved using matrix analysis similar to that described in the above example.

The constraint equation can be considered as an equivalent optimization problem. The equation e = (a · p−b) ^t · s ² · (a · p−b) gives the minimum value of e for the same value p that solves the constraint equation. Therefore, p can be determined by minimizing e. The equation e above can be divided into segments by selecting groups of rows a and b. These groups correspond to the groups of constraint equations, and e is the sum over groups of smaller versions of the same quadratic form. The continuity, smoothness, and droop equations can be placed in one group that can be understood as relating to the effort required to generate speech with the desired prosodic features. The constraint equations arising from the tags can be placed in another group that can be understood as preventing errors, i.e. clear and clearly related to the generation of speech. Then, the value of “e” can be understood as e = effort + error. Qualitatively, the term “effort” behaves like a physiological effort. Zero when the muscle is stationary in the neutral position and increases as muscle movement becomes faster and stronger. Similarly, the term “error” behaves like a transmission error rate. It is minimal when the prosody exactly matches the ideal target, and increases as the prosody departs from the ideal. As the prosody departs from ideal, it is expected that the opportunity for the listener to misidentify the accent or tone shape will increase. It is a reasonable assumption that human speech should represent an attempt to minimize the combination of speaking effort and potential misunderstanding. It is desirable to minimize the error rate (ie, the chance that speech is misinterpreted), and reducing speaking effort is also a desirable goal. The minimization of the value of “e” achieved by the technique of the present invention can be considered as reflecting true human speech trends and compromise characteristics.

After calculating the pitch curve, the process continues and linguistic concepts represented by the phrase curve and the pitch curve are mapped to observable acoustic features. The mapping can be achieved by assuming a statistical correlation between the predicted time variation enhancement _et and the observable features that can be generated in or for the speech signal. it can. Since e _t is usually vector can be accomplished by multiplying by the matrix M statistical correlation e _t.

In step 1024, the matrix M is derived from the tag <set range>. Next, in step 1026, _{e t} · M is computed. In step 1028, in order to adjust the prosodic characteristics defined by the tags for human perception and expected, the result of step 1026, i.e., performs non-linear conversion on e _t · M. A transformation is defined by a <set add> tag. The transformation is represented by the function f (x) = base · (1 + γ + x) ^{1 / add} , where γ = (1+ (rage / base)) ^add −1. The value of f (0) is equal to the value of “base”, and the value of f (1) is equal to the value of “base + range”.

The relationship between the pitch measured in frequency and the perceived strength of the accent need not be linear. Furthermore, the relationship between neural signals or muscle tension and pitch is not linear. When the perceptual effect is most important, and a human speaker adjusts the accent for proper speech, it is useful to see the change in pitch as the smallest change in frequency that can be detected. The minimum detectable frequency change value increases as the frequency increases. According to one opinion that widely accepted minimum frequency change detectable, the relationship between the frequency, given as ^DLαe √f. Where DL is the smallest detectable frequency change, e is the base of the natural logarithm, and f is the frequency or pitch. In the tag and tag processing system according to the present invention, this relationship is the frequency between the strength of the accent and the linear and exponent described by the <set add> tag with an “add” value of approximately 0.5. Corresponds to a certain relationship. On the other hand, when a system for modeling speech on the premise that the speaker does not suit the listener's convenience, other values of “add” can be considered, and a value of “add” exceeding 1 may be used. it can. For example, when muscle tension is added, the value of the pitch f0 is approximately equal to √ (tension).

  Each observable feature can have a different function that is controlled by the appropriate component of the <set add> tag. Amplitude perception is roughly similar to pitch perception in that both amplitude perception and pitch perception have a slowly increasing amount of reception as the underlying observable change. Both amplitude and pitch are expressed as inverse functions that increase approximately exponentially with the desired perceptual effect.

The above function, ie, f (x) = base · (1 + γ + x) ^{1 / add} smoothly describes linear behavior when the value of “add” is 1. If the value of “add” approaches 0, the function describes exponential behavior, and if the value of “add” is between 1 and 0, or close to 0, it is between linear and exponential Describe the behavior at.

  FIG. 11 shows an example of mapping linguistic coordinates to observable acoustic features, as described above in connection with steps 1024-1026 of FIG. Graph 1102 shows a curve 1104 depicting surprise versus emphasis. Graph 1106 shows a curve 1106 depicting pitch versus amplitude. Curve 1104 is mapped to curve 1106. This mapping is made possible by the matrix multiplication described above in connection with steps 1024-1026 of FIG.

  FIG. 12 is a graph 1200 showing the results of a linear transformation similar to that described in connection with step 1028 of FIG. Curves 1202 to 1208 represent the trajectories of the function f (x) having “add” values of 0.0, 0.5, 1.0, and 2.0, respectively. A curve 1202 with an “add” value of 0 indicates an exponential relationship, a curve 1206 with an “add” value of 1 indicates a linear relationship, and a curve 1208 with an “add” value of 2 indicates a logarithmic relationship.

  FIG. 13 is a graph 1300 showing the effect of accents on the pitch curve when the value of “add” is different. Curves 1302A, 1304A, and 1306A show the effect of a series of tags <set add = X /> ... <slope rate = 1 />. Here, the value of X is 0 for the curve 1302A, 0.5 for the curve 1304A, and 1 for the curve 1306A. It can be seen that curve 1302A exhibits an exponential relationship while curve 1306A exhibits a linear relationship. Since curve 1302A shows a logarithmic relationship between frequency and perceived pitch, the actual frequency increases linearly if a uniform slope of the perceived pitch is desired.

  Curves 1302B, 1304B, and 1306B are a series of tags <stress strength = 3 type = 0.5 shape = -0.1s0, 0.05s0, 0s0.1, 0.05s0, 0.1s0 /> ... <stress strength = 3 type = 0.5 shape = -0.1s0, 0.05s0, 0s0.1, 0.05s0, 0.1s0 /> to which a series of tags <set add = X /> ... <slope rate = 1 /> are shown. The value of X is 0 for the curve 1302B, 0.5 for the curve 1304B, and 1 for the curve 1306B. It can be seen that the effect of the initial accent is similar for curves 1302B, 1304B, and 1306B, respectively. This is because the different effects of different values of “add” are not particularly noticeable because the initial accent occurs at a relatively low frequency. The higher the “add” value, the more noticeable the action at high frequencies, but the action is not particularly noticeable at low frequencies. However, the second accent produces significantly different results for each of the curves 1302B, 1304B, and 1306B. It can be seen that as the frequency increases, the lower the value of “add”, the greater the accent is and the more the frequency is displaced.

The following example shows the generation of a standard Chinese sentence from the tag of the present invention. Mandarin Chinese is a tone language with four different vocabulary tones. A tone has strength and weakness, and the relative strength or weakness of a tone interacts with its shape and adjacent tone. FIGS. 14A-14H show how the pitch across sentences containing four different tones, respectively in strong and weak contexts, changes in eight situations. The interaction of tones with adjacent tones can be expressed using tags that control the strength of syllables in the sentence, as shown below.
Chinese word English translation Strength Type
Shou- radio 1.5 0.5
Yin- --- 1.0 0.2
Ji --- 1.0 0.3
Duo more 1.1 0.5
ying- should 0.8 0.2
gai --- 0.8 0.3
deng lamp 1.0 0.5
bi- comparatively 1.5 0.5
jiao --- 1.0 0.3
duo more 1.0 0.5

  The values of “strength” and “type” were derived from training sentences containing the words shou1 yin1 ji1. However, “1” indicates a standard Chinese tone 1, that is, a flat tone.

  These tags are used in the four diagrams of FIGS. 14E-14H (shou1 yin ji1) where four different tones are in the second syllable of the sentence. In the case of the short “Yan” sentence shown in FIGS. 14A-14D, the three-syllable word “shou1 yin / ying ji” is replaced with the single-syllable word “yan”. The rest of each sentence is the same. The tag of the syllable “Yan” is strength = 1.5, type = 0.5, which is the same as the strongest syllable “Shou” in the three-syllable word “Shou yin ji”.

  FIG. 14A is a graph 1400 showing a curve 1402 representing modeling of the word “Yan1” in a sentence, according to the use and processing of tags according to the present invention. “Yan1” is the word “Yan” spoken in tone 1, ie a flat tone. Curve 1404 represents data generated by a speaker speaking a sentence with the word “Yan1” at the beginning. The single syllable word “Yan1” has a strong strength, so the pitch curve shows little influence from other words in the vicinity.

  FIG. 14B is a graph 1410 illustrating a curve 1412 representing modeling of the word “Yan2” in a sentence, according to the use and processing of tags according to the present invention. “Yan2” is the word “Yan” spoken with tone 2, the rising tone. Curve 1414 represents data generated by a speaker speaking a sentence with the word “Yan2” at the beginning. The single syllable word “Yan2” has a strong strength, so the pitch curve shows little influence from other words in the vicinity.

  FIG. 14C is a graph 1420 illustrating a curve 1422 representing modeling of the word “Yan3” in a sentence, according to the use and processing of tags according to the present invention. “Yan3” is the word “Yan” spoken in tone 3, ie low tone. Curve 1424 represents data generated by a speaker speaking a sentence with the word “Yan3” at the beginning. The single syllable word “Yan3” has a strong strength, so the pitch curve shows little influence from other words in the vicinity.

  FIG. 14D is a graph 1430 illustrating a curve 1432 representing modeling of the word “Yan4” in a sentence, according to the use and processing of tags according to the present invention. “Yan4” is the word “Yan” spoken with tone 4, the descending tone. Curve 1434 represents data generated by a speaker speaking a sentence with the word “Yan4” at the beginning. The single syllable word “Yan4” has a strong strength, so the pitch curve shows little influence from other words in the vicinity.

  FIG. 14E is a graph 1440 illustrating a curve 1442 representing modeling of the word “Shou1 yin1 ji1” in a sentence, according to the use and processing of tags according to the present invention. “Yin1” is syllable “Yin” spoken with tone 1, that is, a flat tone. Curve 1444 represents data generated by a speaker speaking a sentence with the word “Shou1 yin1 ji1” at the beginning. The syllable “Yin1”, which is the middle syllable of the three syllable words, has a weak strength, so the pitch curve shows a strong influence from other words in the vicinity.

  FIG. 14F is a graph 1450 showing a curve 1452 representing modeling of the word “Shou1 yin2 ji1” in a sentence, according to the use and processing of tags according to the present invention. “Yin2” is the syllable “Yin” spoken with tone 2, the rising tone. Curve 1454 represents data generated by a speaker speaking a sentence with the word “Shou1 yin2 ji1” at the beginning. The syllable “Yin2”, the middle syllable of the three syllable words, has a weak strength, so the pitch curve shows a strong influence from other words in the vicinity.

  FIG. 14G is a graph 1460 illustrating a curve 1462 representing modeling of the word “Shou1 ying3 ji1” in a sentence, according to the use and processing of tags according to the present invention. “Ying3” is syllable “Ying” spoken in tone 3, ie, low tone. Curve 1464 represents data generated by a speaker speaking a sentence with the word “Shou1 ying3 ji1” at the beginning. The syllable “Ying3”, which is the middle syllable of the three syllable words, has a weak strength, so the pitch curve shows a strong influence from other nearby syllables.

  FIG. 14H is a graph 1470 illustrating a curve 1472 representing modeling of the word “Shou1 ying4 ji1” in a sentence, according to the use and processing of tags according to the present invention. “Ying4” is syllable “Ying” spoken with tone 4, that is, a descending tone. Curve 1474 represents data generated by a speaker speaking a sentence with the word “Shou1 ying4 ji1” at the beginning. The syllable “Ying4”, the middle syllable of the three syllable words, has a weak strength, so the pitch curve shows a strong influence from other nearby syllables.

  From the curves shown in FIGS. 14A-14H, it can be seen that the curves representing text processing modeling using tags according to the present invention provide a good approximation to the curves representing the words actually spoken.

  FIG. 15 shows the steps of a process 1500 for generating and using tags in accordance with the present invention. In step 1502, a training text body is selected. In step 1504, the target speaker reads the training text and generates a training corpus. At step 1506, the training corpus is analyzed to identify prosodic features of the training corpus. In step 1508, a set of tags that model the prosody features of the training corpus is generated, and the tags are placed in the training text to model the training corpus. At step 1510, the tag placement in the training text is analyzed and a rule set for tag placement in the text is generated to model the target speaker's prosodic features. In step 1512, tags are placed in the text body where it is desired to perform text-to-speech processing. Tag placement can be accomplished manually, for example through a text editor, or automatically using the rule set established in step 1510. Steps 1502-1510 are typically performed once or several times for each target speaker, but it is recognized that step 1512 is performed whenever it is desirable to prepare the text body for text-to-speech processing. Like.

  FIG. 16 illustrates a text-to-speech system 1600 according to the present invention. System 1600 includes a computer 1602 that includes a processing unit 1604 that includes memory 1606 and a hard disk 1608, a monitor 1610, a keyboard 1612, and a mouse 1614. Computer 1602 also includes a microphone 1616 and a loudspeaker 1618. Computer 1602 operates to implement text input interface 1620 and speech output interface 1622. Computer 1602 also provides a speech modeler 1624 adapted to receive text from text input interface 1620. In the text, tags generated by the present invention are arranged. The speech modeler 1624 processes the text and tags to generate speech having prosodic features defined by the tags, and outputs the speech to the loudspeaker 1618 using the speech output interface 1622. Speech modeler 1624 generates a set of tags and a prosody tag generation component 1626 adapted to generate rules for tag application to generate speech with prosodic features typical of the target speaker. It may be included as appropriate. To generate a set of tags, a prosody tag generation component 1626 analyzes a training corpus that represents the reading of the training text read by the target speaker, analyzes the prosodic features of the training corpus, and models the training corpus. Generate a set of tags that can be added to the training text. Next, prosody tag generation component 1626 places the tag in the training text, analyzes the tag placement, and creates a rule set for tag placement in the text to model the characteristics of the target speaker's speech.

  The speech modeler 1624 may also optionally include a prosodic evaluation component 1628 that is used to process tags placed in the text for which text-to-speech generation is desired. The prosody evaluation component 1628 generates a time series of pitch or amplitude values defined by the tag.

  The system for generating and processing the tags described above is a solution to one aspect of the more general problem. Speaking is a movement of the muscle that balances the two main goals of minimizing the effort required to move the muscle and minimizing the difference between the movement error, the desired movement and the movement actually made. Is the operation. The system for generating and processing the tags described above generally produces a smooth change in prosody, even when adjacent tag requirements are severely competing. The generation of smooth changes reflects the reality of how muscle movements are made, balancing effort and movement errors.

  It will be appreciated that in a system for generating and processing tags according to the present invention, a user can generate tags that define accents without limiting any shape or range to the accents being defined. Thus, the user has the freedom to create and place tags so as to define different language accent shapes as well as variations within the same language. It is also possible to define speaker-specific accents in speech. It is also possible to define decorative accents for music. The creation of a user's accent definition is not subject to shape or range constraints, so the definition may result in a target combination that is unlikely to be physiological. The system for generating and processing tags according to the present invention allows for competing specifications and returns an implementation of a smooth surface that satisfies all constraints.

  Creating a realization of a smooth surface while facing competing specifications helps to achieve real human speech accurately. The movement of the muscles that control the prosody with real human speech is smooth. This is because it takes time to move from one intended accent target to the next. Note also that if one section of speech material is not important, the speaker may not make much effort to achieve that target. Therefore, the realization of the prosodic surface can be presented as an optimization problem that minimizes the sum of two functions. The first function is a physiological constraint G, which imposes a smoothness constraint by minimizing the first and second derivatives of the identified pitch p. The second function is the communication constraint R, which minimizes the sum of errors τ between the realized pitch p and the target y. This constraint models requirements that require precision in speech for listener understanding.

Errors are weighted by the tag's strengthS _i indicating how important it is to meet the tag's specifications. If the strength of the tag is weak, physiological constraints prevail, and in such cases smoothness is more important than accuracy. S _i controls the interaction of accent tags with neighboring tags according to the smoothness requirement G. The stronger the tag, the stronger the effect on adjacent tags. The tag also includes parameters α and β, which control whether the most important is shape error or the average value of _pt . These parameters are derived from the “type” parameter. The target y can be represented by an accent component at the top of the phrase curve.

The values of G, R, and τ are given by

  Tags are generally processed to minimize the sum of G and R. The above equation shows the minimization of effort and movement combination errors in the processing of tags defining prosody.

  FIG. 17 shows a process 1700 for modeling a phenomenon of motion that is continuous and subject to constraints such as muscle mechanics. In step 1702, a set of tags defining the desired motion component is created. In step 1704, tags are selected and placed to define the desired movement. In step 1706, the tag is analyzed to determine the movement defined by the tag. In step 1708, a time series of motions that minimizes the combination of motion effort, ie, the effort required to generate motion, and motion error, ie, deviation from the motion defined by the tag, is identified. In step 1710, a time series of identified movements is generated. Step 1702 is performed relatively infrequently when a set of tags defining the motion to generate is generated, and steps 1704-1710 are more often employed when adopting tags to define and generate motion. It will be appreciated that this happens frequently.

  In the above description, techniques have been described for generating and using tags suitable for describing and modeling phenomena that are continuous and subject to physiological constraints. A widely used application where such techniques are useful is the description and modeling of speech prosodic features in text-to-speech generation, and a set of tags suitable for modeling such features has been described. An explanation of tag effects and techniques for processing tags were presented. A process for generating, selecting, placing and processing tags as well as a text-to-speech system using tags to generate speech with desired prosodic features was presented. Finally, the process of creating and using tags to define and generate a series of movements has been described.

  Although the present invention has been disclosed in the context of the presently preferred embodiment, it will be appreciated that those skilled in the art may employ a wide variety of implementations consistent with the above description and the appended claims.

Claims (9)

A method of text-to-speech processing,
Analyzing the training corpus representing the reading of the training text to create a set of tags from which to place in the text to define the prosodic features of the speech generated by processing the text;
Placing selected members of the set of tags in a text body in a desired sequence to generate speech features defined by the sequence of tags;
Processing the text body and the tag to generate speech having prosodic features defined by the tag;
The training corpus is formed by selecting a body of the training text and receiving speech representing reading of the training text by the target speaker.   Each tag imposes constraints on the prosodic features of the speech that are affected by the tag, or each tag specifies an action to be taken and attributes and associations that provide information about the action to be taken The method of claim 1, further comprising: a parameter that defines a value to be performed, or each of the tags may include a parameter that identifies a location where the influence of the tag appears.   Does the set of tags include tags that establish settings that remain unchanged unless modified by the next tag, or does the set of tags include members that define the pitch behavior of the speech across phrases? 3. The method of claim 2, wherein the tag set includes tags defining accents that define pitch behavior of local influences within a phrase.   Local effects within the phrase include individual words within the phrase, and the tag defining the accent includes a parameter defining the degree of non-linearity of the accent effect, with accents having higher non-linearity. , Accents appearing in higher pitch areas of the phrase than accents appearing in lower pitch areas of the phrase, and accents that have a linear effect are the same in each pitch area of the phrase 4. The method of claim 3, wherein the method has an effect.   Accents that have an impact that is lower than the linear effect have a lower impact in the higher region of the phrase pitch and a higher impact in the lower region of the phrase pitch, and the set of tags is the region that the tag affects A tag that defines the boundary of a phrase that marks the boundary between, the tag that defines the boundary of the phrase is a speech component in which the tag after the tag that marks the boundary is before the tag that marks the boundary The method of claim 3, wherein each of the tags can include a value defining a type and strength to define an interaction with the other tag of the tag. Processing the text body and tag comprises:
Extracting the tag from the text;
Creating a set of equations that define the phrase curve;
Solving the set of equations to generate the phrase curve;
Creating a set of equations that define the pitch curve;
Solving the set of equations to generate the pitch curve;
Mapping the linguistic concept appearing by the phrase curve and the pitch curve to an observable sound;
Performing a non-linear transformation to adjust the prosodic features defined by the tag to human perception and prediction. A method of defining a set of tags that identify a target speaker's prosodic features,
Selecting a training text body;
Receiving speech representing the reading of the training text by the target speaker and forming a training corpus;
Analyzing the training corpus to identify prosodic features of the training corpus;
Creating a set of tags defining prosodic features of the identified training corpus. A method for placing tags in text for text-to-speech processing,
Placing a tag in the body of the training text to model the prosodic features of the training corpus generated by reading the training text;
Analyzing the placement of the tag in the training text and creating a rule set for the placement of the tag in the text;
Applying the rules to text for which text-to-speech processing is desired and placing a tag in the text to generate speech with the desired prosodic features;
The training corpus is formed by selecting the training text body and receiving speech representing a reading of the training text by a target speaker. A text-to-speech system that receives text input including text to be processed to generate speech and tags that define the prosodic features of the speech to be generated, comprising:
A prosodic tag generation component that analyzes a training corpus to identify features indicated by one or more readings by one or more target speakers and to generate a set of tags that define the identified features; ,
A text input interface for receiving the text input;
A speech modeler operable to process the text input to generate speech having the prosodic features specified by the tag, wherein the speech generated by the speech modeler is the one or more targets Similar to that of the speaker,
A speech output interface for generating the speech output;
The training corpus is formed by selecting a body of training text and receiving speech representing the reading of the training text by the target speaker.

Images