JP5699496B2 - Stochastic model generation device for sound synthesis, feature amount locus generation device, and program - Google Patents

Description

  The present invention relates to generation of a probability model indicating a time series of acoustic feature quantities (for example, pitch and power) and generation of a time series of feature quantities using a probability model. The time series of feature amounts generated from the probability model is suitably used for synthesizing sounds such as singing sounds.

  It is possible to generate an acoustically natural synthesized sound by adding to the synthesized sound a variation in a feature amount that approximates the recorded sound (hereinafter referred to as “reference sound”). For example, Non-Patent Document 1 discloses a technique for generating a synthesized sound using a probability model (for example, HMM (Hidden Markov Model)) that represents a time series of the pitch of a reference sound. Specifically, the reference sound is divided into a plurality of note intervals for each note, and a probability model is generated for each note by a learning process for a time series of pitches in each note interval.

Shinji Sakaki Keijiro Saino Yoshihiko Minamikaku Keiichi Tokuda, Tadashi Kitamura, “Singing Voice Synthesis System with Automatic Voice Quality and Singing Style”, Information Processing Society of Japan [Music Information Science], 2008 (12), p.39−p. 44, February 2008

  FIG. 13 is a schematic diagram showing the relationship between the pitch P of the reference sound containing the singing sound of the music and the pitch (that is, the target value of the pitch P) of each note V (V1, V2, V3) of the music. . As shown in part (A) and part (B) of FIG. 13, the transition of the pitch P of the reference sound may differ depending on, for example, the singing expression even when the sequence of the note V is common. For example, in the part (A) of FIG. 13, the pitch P of the reference sound temporarily decreases before and after the boundary between the note V1 and the note V2 (so-called “shrimp” singing expression), whereas the part of FIG. In (B), the pitch P is maintained substantially constant from the note V1 to the note V2.

  In the technique of Non-Patent Document 1, a probability model is generated for each note by a learning process for a time series of pitches in each note interval in which notes are common among reference sounds. For example, in the case illustrated in FIG. 13, although the transition of the pitch P is different as described above, the generation of the probability model of the note V2 requires the note V2 in both the part (A) and the part (B). The pitch P in the section is applied. Therefore, a probability model that expresses an intermediate pitch P transition between the part (A) and the part (B) is generated. As described above, when a probabilistic model that does not faithfully reflect the characteristics of the actual reference sound is used, there is a problem that a synthetic sound that is audibly unnatural is generated.

  In the above description, the probability model expressing the transition of the pitch is exemplified, but the same problem may occur for the probability models of other feature quantities (for example, power). In view of the above circumstances, an object of the present invention is to generate a stochastic natural synthesized sound by generating a probability model that faithfully reflects the transition of the feature amount in the reference sound.

  Means employed by the present invention to solve the above problems will be described. In order to facilitate the understanding of the present invention, in the following description, the correspondence between the elements of the present invention and the elements of the embodiments described later will be indicated in parentheses, but the scope of the present invention will be exemplified in the embodiments. It is not intended to be limited.

  The sound synthesis probability model generation apparatus according to the first aspect of the present invention is a section setting means (for example, a section) that divides a reference sound into unit sections for each transition type according to a tendency of variation in a feature amount (for example, a reference pitch Pref). A setting unit 34) and a feature amount model (for example, feature amount model QA) for each transition type indicating the probability distribution of the feature amount for each of a plurality of states (for example, state St), and a unit section of the transition type in the reference sound And a probability model generation means (for example, a probability model generation unit 421) that generates from a time series of feature quantities. In the above configuration, since a feature amount model is generated for each reference sound transition type, a difference in variation tendency of the reference sound feature amount is faithfully reflected in the feature amount model. Therefore, for example, when compared with a configuration in which a feature model is generated from a reference sound without taking into account differences in transition types, a feature model that can generate an auditory natural synthesized sound that faithfully reflects the characteristics of the reference sound. There is an advantage that it can be generated.

  A sound synthesis probability model generation apparatus according to a preferred example of the first aspect classifies a plurality of feature amount models generated by the probability model generation means into a plurality of sets, and a feature amount determination tree (for example, feature amount) constructed by classification Feature amount classification means (for example, feature amount classification) that generates feature amount information including a decision tree TA) and a feature amount model (for example, feature amount model MA) generated for each set from the feature amount models classified into each set. Part 423). In the above configuration, since a feature amount model corresponding to the feature amount model in the set is generated for each of a plurality of sets into which the feature amount model generated by the probability model generating means is classified, a large number of features of the reference sound are generated. It is possible to generate a feature quantity model reflecting the quantity (ie, statistically valid). In addition, by applying the designated sound to be synthesized to the feature quantity decision tree constructed by the classification of the feature quantity model, there is also an advantage that an appropriate feature quantity model can be selected for the designated sound of the attribute that does not exist in the reference sound .

  The sound synthesis probability model generation apparatus according to the second aspect of the present invention is a section setting means (for example, a section) that divides a reference sound into unit sections for each transition type according to a tendency of fluctuation of a feature amount (for example, a reference pitch Pref). A setting section 34) and a duration model (eg, duration model QB) for each transition type indicating a probability distribution of duration for each of a plurality of states (eg, state St), and a unit section of the transition type of the reference sound And a probability model generation means (for example, a probability model generation unit 421) that generates from a time series of feature quantities. In the above configuration, since the duration model is generated for each transition type of the reference sound, the difference in the variation tendency of the reference sound feature amount is faithfully reflected in the duration model. Therefore, for example, when compared with a configuration in which a duration model is generated from a reference sound without taking into account differences in transition types, a duration model that can faithfully reflect the characteristics of the reference sound and generate an acoustically natural synthesized sound It is possible to generate.

  The sound synthesis probability model generation device according to a preferred example of the second aspect classifies a plurality of duration models generated by the probability model generation means into a plurality of sets, and a duration determination tree constructed by classification (for example, a duration length) A decision tree (for example, duration classification) that generates duration information including a duration model (for example, duration model MB) generated for each set from the duration model classified into each set. Part 425). In the above configuration, since a duration model corresponding to the duration model in the set is generated for each of a plurality of sets into which the duration model generated by the probability model generation unit is classified, a large number of features of the reference sound are generated. It is possible to generate a duration model that reflects the quantity (ie statistically relevant). In addition, there is an advantage that an appropriate duration model can be selected for a designated tone having an attribute that does not exist in the reference tone by applying the designated tone to be synthesized to the duration decision tree constructed by the duration model classification. .

  Note that the transition type (the tendency of fluctuation of the feature value) in each of the above forms means a trend (behavior) of the feature value over time such as increase / decrease or change / maintenance of the feature value. For example, a process in which the feature amount approaches the target value over time from the starting point of pronunciation (starting portion B), a steady process in which the feature amount is maintained substantially constant (steady portion S), or a feature from the end point of pronunciation. A process in which the amount changes from the target value over time (end E) can be exemplified as a typical example of the transition type.

The sound synthesis probability model generation apparatus according to the third aspect includes a transition arrangement model (for example, a transition arrangement model QC) that specifies a discrete probability that each arrangement appears in the note interval of each note for each of a plurality of types of transition types. ) Is generated from an array of transition types in a note interval corresponding to the note of the reference sound (for example, a transition array model generation unit 441). In the above configuration, since a transition array model indicating the appearance probability for each array of transition types in the note interval is generated, an appropriate transition array is determined for the designated sound to be synthesized and the probability corresponding to each transition type A model (feature model, duration model) can be selected. Therefore, it is possible to generate an acoustically natural synthesized sound that faithfully reflects the characteristics of the reference sound.

  The sound synthesis probability model generation apparatus according to a preferred example of the third aspect classifies a plurality of transition array models generated by the transition array model generation unit into a plurality of sets, and a transition sequence determination tree (for example, transitions) constructed by classification Transition sequence classification means (for example, transition sequence classification) that generates transition sequence information including a sequence determination tree TC) and a transition sequence model (for example, transition sequence model MC) generated for each set from the transition sequence model classified into each set Part 443). In the above configuration, a transition array model corresponding to the transition array model in the set is generated for each of a plurality of sets into which the transition array model generated by the transition array model is classified. It is possible to generate a transition sequence model that reflects (ie, statistically valid). In addition, by applying the designated sound to be synthesized to the transition sequence decision tree constructed by the classification of the transition sequence model, there is also an advantage that an appropriate transition sequence model can be selected even for the designated sound having an attribute that does not exist in the reference sound. .

  The present invention is also specified as a feature amount trajectory generation device that generates a time series of feature amounts using the transition arrangement model generated by the sound synthesis probability model generation device of the third aspect exemplified above. That is, the feature amount trajectory generating apparatus of the present invention has a plurality of transition array models (for example, the transition array model MC) indicating the probability that each array of transition types corresponding to the tendency of variation of the feature amount appears in the note interval of each note. ) And a transition type of each unit section of the specified sound according to the probability indicated by the transition array model corresponding to the note of the specified sound among the plurality of transition array models, Trajectory generating means (for example, a trajectory generating unit 52) is provided that generates a time series of feature values (for example, a combined pitch trajectory Psyn) so that the feature values in each unit section vary with a tendency corresponding to each transition type. In the above configuration, the transition type of each unit section of the designated sound is determined according to the probability indicated by the transition arrangement model corresponding to the note of the designated sound, and the feature amount in each unit section with a tendency according to each transition type A time series of feature values is generated such that fluctuates. Therefore, for example, when compared with a configuration in which a time series of feature values is generated without taking into account differences in transition types, the feature values of the feature values are generated so that a synthesized sound that is audibly natural and reflects the characteristics of the reference sound. It is possible to determine the trajectory.

  The present invention can also be specified as a sound synthesizer (for example, sound synthesizer 100) using the feature amount trajectory generator described above. The acoustic synthesizer of the present invention stores a plurality of transition array models (for example, a transition array model MC) indicating the probability that each array of transition types corresponding to the tendency of fluctuations in feature quantities will appear in the note interval of each note. The transition type of each unit section of the designated sound is determined according to the probability indicated by the storage means (for example, the storage device 14) and the transition arrangement model corresponding to the note of the designated sound among the plurality of transition arrangement models. A trajectory generating means (for example, the trajectory generating section 52) that generates a time series of feature quantities (for example, the combined pitch trajectory Psyn) so that the feature quantities in each unit section vary with a corresponding tendency, and a feature generated by the trajectory generating means. Synthesis processing means (for example, a synthesis processing unit 54) that processes the sound waveform data (for example, the sound waveform data ZA) to generate synthesized sound data (for example, the synthesized sound data Vout) so as to follow the time series of the quantity.

  The devices according to the above embodiments (sound synthesis probability model generation device, feature amount trajectory generation device, sound synthesis device) are realized by a dedicated electronic circuit such as a DSP (Digital Signal Processor), or a CPU (Central Processing). It is also realized by cooperation between a general-purpose arithmetic processing unit such as Unit) and a program. A program that causes a computer to function as an apparatus according to each of the above aspects is provided to a user in a form stored in a computer-readable recording medium and installed in the computer, or in a form distributed via a communication network. Provided by the server device and installed in the computer.

1 is a block diagram of a sound synthesizer according to an embodiment of the present invention. It is a block diagram of a 1st processing part. It is explanatory drawing which illustrates the fluctuation | variation of the reference pitch of a reference sound. It is explanatory drawing which illustrates the other fluctuation | variation of the reference pitch of a reference sound. It is a block diagram of the information production | generation part for a synthesis | combination. It is explanatory drawing of a feature-value model and a continuation length model. It is explanatory drawing of a feature-value decision tree. It is explanatory drawing of a continuation length decision tree. It is explanatory drawing of a transition arrangement | sequence model. It is explanatory drawing of a transition arrangement | sequence decision tree. It is a block diagram of a 2nd processing part. It is explanatory drawing of operation | movement of a locus | trajectory production | generation part. It is explanatory drawing of the problem of the production | generation of the probability model in background art.

<A: Embodiment>
FIG. 1 is a block diagram of a sound synthesizer 100 according to one embodiment of the present invention. The sound synthesizer 100 in FIG. 1 is a singing synthesizer that generates synthesized sound data Vout indicating the singing sound of a musical piece of desired notes and lyrics. As shown in FIG. This is realized by a computer system including the input device 16. The input device 16 (for example, a mouse or a keyboard) receives an instruction from the user.

  The storage device 14 stores a program PGM executed by the arithmetic processing device 12 and various data (reference information X, synthesis information Y, sound waveform information Z, score data SC) used by the arithmetic processing device 12. A known recording medium such as a semiconductor recording medium or a magnetic recording medium or a combination of a plurality of types of recording media is arbitrarily used as the storage device 14.

  The reference information X is composed of reference sound data XA and score data XB and is used for generating (learning) synthesis information Y. The reference sound data XA is a sample series that expresses a sound waveform in a time domain of a sound (reference sound) in which a specific singer (hereinafter referred to as “reference singer”) sings a song. The musical score data XB expresses the musical score of the music indicated by the reference sound data XA. That is, the musical score data XB designates a note (sound name, duration) and lyrics (phonetic characters) of the reference sound in time series.

  The synthesis information Y is generated according to the reference information X for each reference singer (or for each genre of the music sung by the reference singer), and the time series (trajectory) of the characteristic amount specific to the singing sound of the reference singer ) To identify. In the present embodiment, a pitch (fundamental frequency) is assumed as a feature amount specified from the synthesis information Y. The generation of the composition information Y using the reference information X will be described later.

  The sound waveform information Z includes a plurality of sound waveform data ZA. Each sound waveform data ZA is generated in advance for each speech unit uttered by the reference singer, and expresses a waveform characteristic of the speech unit (for example, a waveform in the time domain or a shape of a frequency spectrum). A phoneme segment is a phoneme chain that is a unit of phonemes or a plurality of phonemes that is the smallest unit that can be audibly distinguished.

  The musical score data SC designates notes (pitch names, duration) and lyrics (pronunciation characters) of each designated sound to be synthesized in time series. The musical score data SC is generated in response to an instruction from the user to the input device 16 (instruction for adding or editing each designated sound). Schematically, the pitch of the sound waveform data ZA corresponding to the notes and lyrics of each designated sound designated by the score data SC is a time series of pitches generated according to the synthesis information Y (hereinafter referred to as “synthetic pitch trajectory”). ), The synthesized sound data Vout is generated. That is, a singing expression (pitch fluctuation) peculiar to the reference singer is added to the synthesized sound represented by the synthesized sound data Vout.

  The arithmetic processing unit 12 in FIG. 1 executes a plurality of functions (a first processing unit 21 and a second processing unit 22) necessary for the generation (speech synthesis) of synthesized sound data Vout by executing the program PGM stored in the storage device 14. ). The first processing unit 21 generates the synthesis information Y using the reference information X, and the second processing unit 22 uses the synthesis information Y, the sound waveform information Z, and the score data SC to generate the synthesized sound. Data Vout is generated. A configuration in which each function of the arithmetic processing unit 12 is realized by a dedicated electronic circuit (DSP) or a configuration in which each function of the arithmetic processing unit 12 is distributed over a plurality of integrated circuits may be employed. The configuration and operation of the first processing unit 21 and the second processing unit 22 will be described sequentially.

(1) First processing unit 21
FIG. 2 is a block diagram of the first processing unit 21. As shown in FIG. 2, the first processing unit 21 includes a feature amount extraction unit 32, a section setting unit 34, and a composition information generation unit 36. The feature quantity extraction unit 32 sequentially detects the pitch of the reference sound (hereinafter referred to as “reference pitch”) Pref indicated by the reference sound data XA. A known technique is arbitrarily employed for detecting the reference pitch Pref. Note that the reference pitch Pref in a section where the harmonic structure does not exist in the reference sound (for example, a consonant section in which no pitch is detected) is set to a predetermined value (for example, an interpolated value of the preceding and following reference pitch Pref). In FIG. 3, the time series of the reference pitch Pref detected by the feature quantity extraction unit 32 and the time series of the designated sounds (V1, V2,...) Designated by the score data XB have a common time axis. It is shown in the figure.

  The section setting unit 34 in FIG. 2 divides the reference sound (time series of the reference pitch Pref) indicated by the reference sound data XA into a plurality of unit sections μ on the time axis. As shown in FIG. 2, the section setting unit 34 of the present embodiment includes a first section setting unit 341, a second section setting unit 343, and an identification information setting unit 345. As shown in FIG. 3, the first section setting unit 341 divides the time series of the reference pitch Pref detected by the feature amount extraction unit 32 into sections (hereinafter referred to as “note sections”) σ for each note. The musical score data XB of the reference information X is used for setting each note interval σ. That is, the first section setting unit 341 sets the time series of the reference pitch Pref to a plurality of note sections σ with the start point and end point of each designated sound (V1, V2,...) Designated by the musical score data XB for each note as a boundary. Break down.

  The second interval setting unit 343 in FIG. 2 divides each note interval σ in time series of the reference pitch Pref into unit intervals μ for each transition type. The transition type means a classification according to the tendency of fluctuation of the reference pitch Pref. In the present embodiment, as shown in FIG. 3, the start part B (Beginning), the steady part S (Sustain), and the end part E (End) are exemplified as transition types. The start part B means a section in which the reference pitch Pref fluctuates (e.g., rises) so as to approach the pitch of the note immediately after the sound of one note, and the stationary part S means the sound of one note. The reference pitch Pref means a section in which the reference pitch Pref is maintained substantially constant at the pitch of the note, and the end portion E changes the reference pitch Pref from the pitch of the note immediately before the end of the sounding of one note. It means a section (for example, a decrease).

  One or more transition types appear in each note interval σ. Moreover, the temporal relationship that the start part B is located in front of the steady part S and the end part E and the end part E is located behind the start part B and the steady part S is fixed. Accordingly, the transition type array patterns (hereinafter referred to as “transition arrays”) that may appear within one note interval σ are “BSE”, “BS”, “SE”, “B”. -E "," B "," S "," E "total 7 types. For example, the note interval σ corresponding to the designated sound V1 in FIG. 3 is divided into a unit interval μ of the stationary part S and a unit interval μ of the end part E (transition array “SE”), and the note interval of the designated sound V2 σ is divided into a unit interval μ of the start portion B, a unit interval μ of the stationary portion S, and a unit interval μ of the end portion E (transition array “BSE”), and the note interval σ of the designated sound V3 is stationary. The unit interval μ of the part S is set (transition array “S”).

  As described above, each note interval σ is divided into unit intervals μ for each transition type. Therefore, even when the note sequence of the reference sound (that is, the note of each designated note of the score data XB) is common, the note interval σ The mode of division (the number of unit sections μ and the time length) changes according to the mode of change (transition type) of the reference pitch Pref. For example, when the reference pitch Pref is temporarily lowered before and after the boundary between the designated sound V1 and the designated sound V2 as illustrated in FIG. 3 (that is, when the singing expression “sharsh” is given to the reference sound), Thus, the note interval σ corresponding to the designated sound V1 is divided into two unit intervals μ corresponding to the stationary part S and the end part E, and the note interval σ corresponding to the designated sound V2 is the start part B. Are divided into three unit sections μ corresponding to the stationary part S and the end part E. On the other hand, when the reference pitch Pref does not fluctuate before and after the boundary between the designated sound V1 and the designated sound V2 as shown in FIG. 4, the note interval σ corresponding to the designated sound V1 is set to one unit interval μ of the stationary part S. (Transition array “S”), the note interval σ corresponding to the designated sound V2 is divided into two unit intervals μ corresponding to the stationary part S and the end part E (transition array “SE”).

  Each unit interval μ is variably set according to an instruction from the user. For example, the user visually recognizes the time series of the reference pitch Pref displayed on the display device (not shown) (for example, time variation of the reference pitch Pref illustrated in FIG. 3) and reproduces it from the sound emitting device (for example, a speaker). Each unit section μ is designated by appropriately operating the input device 16 while estimating the transition type at each time point by listening to the reference sound. The second section setting unit 343 sets each unit section μ in accordance with an instruction from the user to the input device 16.

  The identification information setting unit 345 in FIG. 2 sets the identification information A for each unit section μ divided by the second section setting unit 343. The identification information A is an identifier (label) indicating the attribute of the unit section μ and includes a note attribute a1 and a transition type a2 as shown in FIG. The transition type a2 designates the transition type (any one of the start part B, the steady part S, and the end part E) of the unit section μ. The transition type a2 is designated by the user by operating the input device 16 when setting the unit interval μ, for example.

  The note attribute a1 is information indicating the attribute of a note (hereinafter referred to as “target note”) corresponding to the unit section μ, and includes variables p1 to p3 and variables d1 to d3. The variable p2 is set to the note name (note number) of the target note. The variable p1 is set to the pitch of the note immediately before the target note (relative value to the target note), and the variable p3 is set to the pitch of the note immediately after the target note. The variable d2 is set to the duration of the target note. The variable d1 is set to the duration of the note immediately before the target note, and the variable d3 is set to the duration of the note immediately after the target note. Each variable (p1 to p3, d1 to d3) of the note attribute a1 is specified from the score data XB. As can be understood from the above description, the identification information A is common to a plurality of unit sections μ having common musical conditions. Note that the content of the note attribute a1 is not limited to the above example. For example, information indicating what time signature the target note corresponds to in each measure of the music (first beat / second beat), or the position of the target note in the period corresponding to the breath of the reference sound (forward / backward) Any information that affects the time series of the pitch, such as information indicating), can be designated by the note attribute a1.

  2 generates the synthesis information Y by using the time series of the reference pitch Pref for each unit section μ set by the section setting section 34 (second section setting section 343). FIG. 5 is a block diagram of the synthesis information generation unit 36. As shown in FIG. 5, the synthesis information generation unit 36 includes a first information generation unit 42 that generates feature amount information YA and duration information YB, and a second information generation unit 44 that generates transition sequence information YC. To do. The feature amount information YA, the continuation length information YB, and the transition sequence information YC are stored in the storage device 14 as the combining information Y in FIG.

  As shown in FIG. 5, the first information generation unit 42 includes a probability model generation unit 421, a feature amount classification unit 423, and a duration classification unit 425. The probability model generation unit 421 generates a probability model Q expressing the appearance probability of the pitch P within one unit section μ corresponding to each transition type for each identification information A (for each combination of the note attribute a1 and the transition type a2). ) To generate. In the present embodiment, as shown in FIG. 6, HSMM (Hidden Semi Markov Model) defined by a plurality of (three in the example of FIG. 6) states St is exemplified as the probability model Q. The probability model Q includes a feature amount model QA and a duration model QB. The feature quantity model QA defines the probability distribution (output distribution) of the pitch P in the unit section μ and its time variation (differential value) ΔP for each state St, and the duration model QB is the state in the unit section μ. The probability distribution (continuation length distribution) of the continuation length D for each St is defined. A configuration in which the feature quantity model QA defines the probability distribution of the second-order differential value of the pitch P for each state St is also suitable.

  The probability model generation unit 421 in FIG. 5 executes the learning process (maximum likelihood estimation algorithm) on the time series of the reference pitch Pref in each unit section μ with which the identification information A is common, so that the identification information A A corresponding probability model Q is generated. Specifically, the probability model Q is generated so that the time series of the reference pitch Pref in each unit section μ appears with the maximum probability. The probability model Q is generated for each identification information A. That is, even when the note attribute a1 is common to a plurality of unit intervals μ, if the transition type a2 is different in each unit interval μ, a separate probability model Q is generated for each transition type a2.

  The feature quantity classification unit 423 in FIG. 5 classifies the feature quantity model QA generated by the probability model generation unit 421 for each piece of identification information A into a set of a plurality (number less than the total number of probability models Q). Although known machine learning can be arbitrarily employed for classification (clustering) of the feature quantity model QA, decision tree learning exemplified below is preferable.

  The feature quantity classifying unit 423 constructs the decision tree (hereinafter referred to as “feature quantity decision tree”) TA in FIG. 7 by sequentially determining the success or failure of a predetermined condition related to the identification information A for each feature quantity model QA. . As shown in FIG. 7, the feature quantity decision tree TA includes a start node (root node) serving as a starting point of classification, a plurality of intermediate nodes (intermediate nodes) corresponding to the determination of each condition, and each feature model QA. Is a classification tree composed of KA terminal nodes (leaf nodes) corresponding to the finally classified set. In the start node and each intermediate clause, for example, whether the duration d2 of the target note exceeds a threshold value, or whether the pitch p1 between the target note and the immediately preceding note (or the pitch p3 between the immediately following note) exceeds the threshold value. The success or failure of the condition is determined. The time point when the classification of each feature value model QA is stopped (the time point when the feature value determination tree TA is determined) is determined according to, for example, a minimum description length (MDL) standard.

  The feature quantity classification unit 423 generates, for each of the KA terminal clauses of the feature quantity decision tree TA, one feature quantity model MA corresponding to the plurality of feature quantity models QA classified into the terminal clause. Specifically, the feature quantity (pitch P) applied to the generation (learning) of the feature quantity model QA is used as a whole for a plurality of feature quantity models QA of one terminal clause, and corresponds to the terminal clause. One new feature model MA is re-estimated. For example, a weighted sum of a plurality of feature quantity models QA classified in each terminal clause is generated as the feature quantity model MA. As shown in FIG. 5, the feature quantity classification unit 423 stores feature quantity information YA including the feature quantity decision tree TA and KA feature quantity models MA generated by the above method in the storage device 14.

  Similar to the feature amount classification unit 423, the duration classification unit 425 in FIG. 5 classifies the duration model QB generated by the probability model generation unit 421 for each identification information A into a plurality of sets by decision tree learning. That is, the continuation length classifying unit 425 determines the success or failure of a predetermined condition related to the identification information A for each continuation length model QB, thereby determining the decision tree (hereinafter referred to as “continuation length decision tree”) TB in FIG. To construct. The conditions determined when the duration determination tree TB is constructed and the criteria for stopping the construction of the duration determination tree TB are the same as those when the feature amount decision tree TA is constructed. The continuation length classifying unit 425, for each of the KB terminal clauses of the confirmed continuation length decision tree TB, one continuation length model MB (for example, corresponding to a plurality of continuation length models QB classified into the ending clause) A weighted sum of a plurality of duration models QB) is generated. Then, the continuation length classifying unit 425 stores the continuation length information YB including the continuation length decision tree TB and KB continuation length models MB as shown in FIG.

  The second information generation unit 44 in FIG. 5 includes a transition array model generation unit 441 and a transition array classification unit 443. The transition array model generation unit 441 generates a transition array model QC for each note attribute a1 in the identification information A (for each note interval σ in which the note attribute a1 is common). As shown in FIG. 9, the transition array model QC corresponding to each note attribute a1 has a total of seven types of transition arrays (“BSE”, “BS”, “SE”, “B”). -E "," B "," S "," E ") is a probability model indicating the probability (discrete probability) that each of the note attributes a1 appears within the note interval σ.

  The transition arrangement model QC of the note attribute a1 is generated according to the frequency of appearance of each transition arrangement in each note interval σ with the common note attribute a1. That is, in the transition array model QC of each note attribute a1, the appearance probability of the transition array that frequently appears for each note section σ of the note attribute a1 is set to a larger numerical value. For example, the transition arrangement of one note interval σ of two note intervals σ having the same note attribute a1 is “BSE” as in the designated note V2 in FIG. When the transition array is “SE” as in the designated sound V2 of FIG. 4, the transition array “BSE” and the transition array “SE” of the transition attribute model QC of the note attribute a1 are shown. The appearance probability of each is set to 0.5, and the appearance probability of other transition arrays is set to 0.

  The transition array classifying unit 443 in FIG. 5 sets a plurality of transition array models QC generated by the transition array model generating unit 441 for each note attribute a1 (for each note interval σ) (a number less than the total number of transition array models QC). Classify. Although known machine learning can be arbitrarily employed for classification of the transition sequence model QC, decision tree learning described below is suitable as in the classification by the feature amount classification unit 423 and the duration classification unit 425.

  The transition sequence classifying unit 443 constructs the decision tree (hereinafter referred to as “transition sequence decision tree”) TC of FIG. 10 by sequentially determining success or failure of a predetermined condition related to the identification information A for each transition sequence model QC. . Similar to the feature value decision tree TA and the continuation length decision tree TB described above, the transition sequence decision tree TC is a KC corresponding to a starting node, a plurality of intermediate clauses, and a set in which each transition sequence model QC is finally classified. It is a classification tree composed of terminal clauses.

  The transition arrangement within the note interval σ is affected by the time length d2 of the target note, the pitch (p1, p2) between the target note and the preceding and following notes, and the like. For example, the total number of transition types in the note interval σ increases as the time length d2 of the target note increases, or the transition in the note interval σ increases as the pitch (pitch difference) between the target note and the preceding and following notes increases. There is a tendency for the total number of types to increase. In consideration of the above tendency, the transition arrangement classifying unit 443 determines whether the duration d2 of the target note exceeds a threshold value, for example, whether the duration of the target note exceeds the threshold, similarly to the feature amount classifying unit 423 and the duration classification unit 425. The success or failure of various conditions such as whether or not the pitch p1 with the note (or the pitch p3 with the immediately following note) exceeds a threshold value is determined in the starting clause and each intermediate clause. For example, a minimum description length (MDL) criterion is preferably applied to the determination of the time point when the classification of each transition sequence model QC is stopped (the time point when the transition sequence decision tree TC is determined).

  The transition array classification unit 443 generates, for each of the KC terminal nodes of the transition array determination tree TC, one transition array model MC corresponding to the plurality of transition array models QC classified into the terminal node. For example, a weighted sum of a plurality of transition array models QC classified into each terminal clause is generated as the transition array model MC. As shown in FIG. 5, the transition array classification unit 443 stores the transition array information YC including the transition array decision tree TC generated by the above method and KC transition array models MC in the storage device 14. The above is the configuration and operation of the first processing unit 21.

(2) Second processing unit 22
FIG. 11 is a block diagram of the second processing unit 22 that generates the synthesized sound data Vout. As shown in FIG. 11, the second processing unit 22 includes a trajectory generation unit 52 and a synthesis processing unit 54. The trajectory generator 52 generates a time series (synthetic pitch trajectory) Psyn of the pitches of the designated sounds designated by the score data SC from the synthesis information Y. The synthesis processing unit 54 generates synthesized sound data Vout of a singing sound whose pitch changes with time so as to follow the synthesized pitch trajectory Psyn generated by the trajectory generation unit 52. Specifically, the synthesis processing unit 54 acquires sound waveform data ZA corresponding to the lyrics of each designated sound indicated by the score data SC from the storage device 14, and the pitch changes with time along the synthesized pitch trajectory Psyn. In this way, the synthesized sound data Vout is generated by processing the sound waveform data ZA. Therefore, the reproduced sound of the synthesized sound data Vout is a singing sound to which a singing expression (pitch trajectory) peculiar to the reference singer who uttered the reference sound is added.

  FIG. 12 is an explanatory diagram of the operation of the trajectory generation unit 52. The process of FIG. 12 is started in response to a predetermined operation on the input device 16 (instruction to start generation of synthesized sound), and is sequentially executed for each designated sound of the score data SC.

  When the processing of FIG. 12 is started, the trajectory generator 52 determines the note attribute a1 (variables p1 to p3, variables d1 to d3) of the designated sound by referring to the score data SC (S11). The trajectory generation unit 52 then selects one transition array model corresponding to the note attribute a1 of the designated note among the KC transition array models MC in the transition array information YC of the synthesis information Y stored in the storage device 14. MC is selected (S12). For selection of the transition array model MC, the transition array decision tree TC in the transition array information YC is used. That is, the trajectory generating unit 52 applies the note attribute a1 of the designated sound to the transition arrangement determination tree TC (the success or failure of the conditions of the start node and each intermediate clause of the transition arrangement decision tree TC is sequentially determined for the note attribute a1 of the designated sound. The terminal section (set) to which the designated sound should belong is specified by determining), and the transition array model MC corresponding to the terminal section is selected from the transition array information YC. That is, the transition arrangement model MC generated from the note interval σ of the note attribute a1 similar to the note attribute a1 of the designated sound among the reference sounds is selected.

  The trajectory generator 52 determines the transition arrangement of the designated sound according to the transition arrangement model MC selected in the process S12 (S13). Specifically, the probability of determining each transition array in process S13 for each designated sound having the same note attribute a1 set in process S11 is approximated to the appearance probability defined for each transition array in the transition array model MC. In addition, the trajectory generator 52 determines the transition arrangement of the designated sound. That is, a transition sequence having a higher appearance probability defined by the transition sequence model MC is selected with a higher probability as a transition sequence of the designated sound. Then, the trajectory generating unit 52 sets the unit section μ of the designated sound for each transition type constituting the transition array determined in the process S13. For example, when the transition array selected in step S13 is “BSE”, three unit sections μ corresponding to each transition type are set for one designated sound. The trajectory generating unit 52 sets identification information A including the note attribute a1 of the designated sound and the transition type a2 determined for the unit section μ for each unit interval μ of the designated sound.

  The trajectory generating unit 52 selects a feature quantity model MA corresponding to the identification information A of the designated sound from the KA feature quantity models MA in the feature quantity information YA for each unit section μ of the designated sound (S14). Specifically, the trajectory generation unit 52 specifies the terminal clause (set) to which the specified sound identification information A should belong by applying the specified sound identification information A to the feature amount decision tree TA of the feature amount information YA. Then, one feature quantity model MA corresponding to the terminal clause is selected from the feature quantity information YA. Similarly, the trajectory generation unit 52 applies the specified sound identification information A to the continuous length decision tree TB of the continuous length information YB, so that the specified sound of the KB continuous length models MB in the continuous length information YB is selected. One duration model MB corresponding to the identification information A is selected for each unit section μ of the designated sound (S15).

  Then, the trajectory generating unit 52 generates a composite pitch trajectory Psyn within each unit section μ of the designated sound using the feature amount model MA selected in the process S14 and the duration model MB selected in the process S15 (S16). ). Specifically, the continuation length D of each state St in the unit section μ is determined according to the continuation length model MB, and the probability distribution of the pitch P and the probability distribution of the time change ΔP defined by the feature amount model MA A synthetic pitch trajectory Psyn for each unit section μ is generated so that the joint probability is maximized. The synthesized pitch trajectory Psyn of the designated sound is generated by connecting the synthesized pitch trajectory Psyn generated for each unit section μ with the above procedure on the time axis.

  In the above embodiment, the feature amount model QA and the duration model QB are generated for each transition type using the time series of the reference pitch Pref in the unit interval μ in which the reference sound is divided according to the transition type. Even when the notes of the sound (note attribute a1) are common, the difference in the variation of the reference pitch Pref is faithfully reflected in the feature amount model QA and the duration model QB (and also the feature amount model MA and the duration model MB). . Further, the transition type of each unit section μ of the designated sound in the musical score data SC is determined according to the transition array model MC, and the synthesized pitch trajectory Psyn corresponding to the feature amount model QA and the duration model QB corresponding to the transition type. Is generated for each unit interval μ of the designated sound. Therefore, compared to the case of using a probability model that does not faithfully reflect the difference in the reference pitch Pref of the reference sound, the synthesized sound that faithfully reflects the expression specific to the reference singer is more audible and natural. It is possible to generate while maintaining.

<B: Modification>
The above embodiment can be variously modified. Specific modifications are exemplified below. Two or more aspects arbitrarily selected from the following examples can be appropriately combined.

(1) Modification 1
The method of dividing the reference sound into note intervals σ and unit intervals μ is appropriately changed. For example, in the above-described embodiment, the reference sound is divided into the respective note intervals σ according to the score data XB, but a configuration in which each note interval σ is set according to an instruction from the user may be employed. For example, the user appropriately operates the input device 16 while estimating the boundary of each note by visually recognizing the waveform of the reference sound displayed on the display device and listening to the reference sound reproduced from the sound emitting device. To specify each note interval σ. The first section setting unit 341 sets each note section σ according to an instruction from the user. In a configuration in which the user designates each note interval σ, the score data XB can be omitted.

  In the above-described embodiment, each unit interval μ of the reference sound is set according to an instruction from the user. However, the second interval setting unit 343 automatically (that is, from the user) according to the reference sound data XA. A configuration in which each unit interval μ is set (without requiring an instruction) may be employed. For example, the second section setting unit 343 sets a section in which the reference pitch Pref changes immediately after the start point of the note section σ as the unit section μ of the start section B. Similarly, a section in which the reference pitch Pref is maintained substantially constant is set as the unit section μ of the stationary part S, and a section in which the reference pitch Pref varies toward the end point of the note section σ is set as the unit section μ of the end part E. The

(2) Modification 2
In the above-described embodiment, KC transition array models MC are generated in accordance with the result of classification of the transition array model QC. However, the unclassified transition array model QC is applied to the synthesis of the designated sound as the transition array information YC. (Hereinafter referred to as “Configuration A”) may also be employed. The transition arrangement of the designated sound is determined using the transition arrangement model QC corresponding to the note designated by the designated sound. In the configuration A, since the transition sequence classification unit 443 (the transition sequence model MC and the transition sequence determination tree TC) is omitted, there is an advantage that the configuration of the first processing unit 21 is simplified.

  However, in the configuration A, since the number of reference pitches Pref used for generating one transition array model QC is insufficient, it is difficult to ensure the statistical validity of the transition array model QC. In addition, since it is practically difficult to prepare transition arrangement models QC for all types of note attributes a1, there is also a problem that it is not possible to synthesize designated sounds of note attributes a1 for which transition arrangement models QC are not prepared. In the above-described embodiment, KC transition array models MC are generated according to the classification result of the transition array model QC (that is, a plurality of reference pitches are added to one transition array model MC compared to the transition array model QC). Preref is reflected), and it is possible to sufficiently ensure the statistical validity of the transition sequence model MC. In addition, since the designated sound transition array is determined by applying the designated sound to the transition array decision tree TC at the time of synthesis (S13), even for the designated sound of a note that does not exist in the reference sound, an acoustically natural synthesized sound. There is an advantage that an appropriate transition sequence capable of realizing the generation of can be selected.

  Similar to the configuration A described above, a configuration in which the feature amount model QA and the duration model QB are applied to the synthesis of the designated sound (a configuration in which the feature amount classification unit 423 and the duration classification unit 425 are omitted) can be adopted. From the viewpoint of synthesizing an acoustically natural synthesized sound while ensuring the statistical validity of the model Q, the feature amount model MA generated by the classification of the feature amount model QA as illustrated in the above embodiment. A configuration in which the duration model MB generated by the classification of the duration model QB is used for synthesizing the designated sound is particularly suitable.

(3) Modification 3
In the above-described embodiment, the feature amount extraction unit 32 extracts the reference pitch Pref from the reference sound data XA stored in the storage device 14. However, the time series of the reference pitch Pref extracted in advance from the reference sound is stored in the storage device 14. Also, the configuration stored in (and thus the feature quantity extraction unit 32 is omitted) may be employed. Further, a configuration in which the reference sound is divided into each note interval σ in advance and stored in the storage device 14 (therefore, the first interval setting unit 341 is omitted) may be employed.

(4) Modification 4
In the above-described embodiment, the acoustic synthesizer 100 including the first processing unit 21 and the second processing unit 22 is exemplified, but the synthesis information Y (feature information YA, duration information YB, transition sequence information YC) is used. A sound synthesis probability model generation device (a device in which the second processing unit 22 is omitted) including the first processing unit 21 to be generated, or a second processing unit 22 that generates the synthesized sound data Vout using the synthesis information Y. The present invention can also be implemented as a sound synthesizer (equipped with the first processing unit 21 omitted). In addition, a device including the storage device 14 that stores the synthesis information Y and the trajectory generation unit 52 of the second processing unit 22 (a configuration in which the synthesis processing unit 54 is omitted) is a time series (for example, a feature amount of synthesized sound) It can also be grasped as a feature amount trajectory generating device that generates a composite pitch trajectory Psyn).

(5) Modification 5
In the above-described embodiment, the synthesis information Y is generated from the time series of the reference pitch Pref of the reference sound and the synthesis pitch trajectory Psyn is generated from the synthesis information Y. However, the reference sound used to generate the synthesis information Y is generated. The feature amount of the designated sound generated from the feature amount and the synthesis information Y is not limited to the pitch (fundamental frequency). For example, a configuration may be employed in which the synthesis information Y is generated from the reference sound power time series and the specified sound power time series (synthesis power trajectory) is generated from the synthesis information Y. The present invention can also be applied to the generation of feature quantities such as the MFCC (Mel-Frequency cepstral coefficient) of the designated sound as in the above-described embodiment.

  The feature amount is not limited to a numerical value extracted directly from the reference sound. For example, a configuration in which the synthesis information Y is generated using a relative value of the feature amount of the reference sound with respect to a predetermined target value may be employed. Specifically, the synthesis information Y is generated from the relative value of the reference pitch reference pitch Pref with respect to a predetermined target value (for example, the pitch of the reference note), and the pitch information generated according to the synthesis information Y is generated. A configuration is employed in which the synthesized pitch trajectory Psyn is generated from the relative value and the pitch of the note of the designated sound.

(6) Modification 6
In the above-described embodiment, the synthesis of the singing sound is exemplified, but the range to which the present invention is applied is not limited to the synthesis of the singing sound. For example, when synthesizing musical instrument performance sounds (musical sounds), the present invention can be applied in the same manner as in the above-described embodiment.

DESCRIPTION OF SYMBOLS 100 ... Sound synthesizer, 12 ... Arithmetic processing unit, 14 ... Memory | storage device, 16 ... Input device, 21 ... 1st process part, 22 ... 2nd process part, 32 ... Feature-value extraction part, 34... Section setting section, 341... First section setting section, 343... Second section setting section, 345... Identification information setting section, 36. 421... Probability model generation unit 423... Feature amount classification unit 425... Continuous length classification unit 44... 2nd information generation unit 441 ... Transition sequence model generation unit 443. , 52... Locus generation unit, 54... Synthesis processing unit.

Claims (8)

Section setting means for dividing the reference sound into unit sections for each transition type according to the trend of fluctuation of the feature amount,
Probability model generation means for generating a feature amount model for each transition type indicating a probability distribution of a feature amount for each of a plurality of states from a time series of feature amounts in a unit section of the transition type in the reference sound;
For each of a plurality of types of transition types, a transition sequence model that specifies a discrete probability that the sequence appears in the note interval of each note, and an array of transition types in the note interval corresponding to the note among the reference sounds A sound synthesis probability model generation device comprising: a transition sequence model generation means for generating a sound sequence. The transition array model generation means includes a plurality of transition types of a start part of the note interval, a steady part whose feature value is constantly maintained immediately after the start part, and an end part immediately after the steady part. The sound synthesis probability model generation device according to claim 1, wherein the transition array model indicating the appearance probability is generated for each of the sequences and each of the start part, the steady part, and the end part. A transition array model that specifies a discrete probability that each array appears in a note interval of a single note for each of a plurality of types of arrays of transition types according to the tendency of variation in feature amount is 1 in a note interval for each note. A sound synthesis probability model generation device comprising transition array model generation means for generating from an array of transition types in a note interval corresponding to the one note among reference sounds including at least transition types . A plurality of transition array models generated by the transition array model generation means are classified into a plurality of sets, and are generated for each set from the transition array decision tree constructed by the classification and the transition array models classified into the respective sets. The sound synthesis probability model generation device according to any one of claims 1 to 3, further comprising transition sequence classification means for generating transition sequence information including a transition sequence model. Corresponding to the note of the specified note among a plurality of transition array models that specify discrete probabilities that the array appears in the note interval of each note for each of the plurality of types of transition types according to the tendency of the variation of the feature amount Determine the transition type of each unit section of the specified sound according to the discrete probability specified by the transition array model, and the time series of the feature quantity so that the feature quantity in each unit section varies with the tendency according to each transition type A feature amount trajectory generating apparatus comprising trajectory generating means for generating. Computer
Section setting means that divides the reference sound into unit sections for each transition type according to the tendency of fluctuation of the feature amount,
Probability model generation means for generating a feature amount model for each transition type indicating a probability distribution of a feature amount for each of a plurality of states from a time series of feature amounts in a unit section of the transition type in the reference sound; and
For each of a plurality of types of transition types, a transition sequence model that specifies a discrete probability that the sequence appears in the note interval of each note, and an array of transition types in the note interval corresponding to the note among the reference sounds A program that functions as a means for generating a transition sequence model generated from an object. Computer
A transition array model that specifies a discrete probability that each array appears in a note interval of a single note for each of a plurality of types of arrays of transition types according to the tendency of variation in feature amount is 1 in a note interval for each note. A program that functions as a transition array model generating unit that generates from an array of transition types in a note interval corresponding to the one note among reference sounds including at least transition types . Computer
Corresponding to the note of the specified note among a plurality of transition array models that specify discrete probabilities that the array appears in the note interval of each note for each of the plurality of types of transition types according to the tendency of the variation of the feature amount Determine the transition type of each unit section of the specified sound according to the discrete probability specified by the transition array model, and the time series of the feature quantity so that the feature quantity in each unit section varies with the tendency according to each transition type A program that functions as a trajectory generating means for generating

Images