JP5334142B2 - Method and system for estimating mixing ratio in mixed sound signal and method for phoneme recognition - Google Patents

Abstract

Provided are a mixed-sound signal mixture ratio estimating method and system for estimating a mixture ratio between a target sound signal and a noise signal in a mixed-sound signal. The gain of a target sound spectrum template, which constitutes a probabilistic spectrum template, and the gain of a noise spectrum template are modified to obtain a plurality of gain-modified spectrum templates. One of the gain-modified spectrum templates that exhibits the shortest distance from an observation spectrum is decided as a shortest-distance gain-modified spectrum template. The mixture ratio is estimated based on the gain of the shortest-distance gain-modified spectrum template and the gain of the noise spectrum template.

Description

  The present invention relates to a mixing ratio estimation method and system for a mixed sound signal for estimating a mixing ratio between a target sound signal and a noise signal in the mixed sound signal, and a phoneme recognition method.

  Conventionally, on the premise that the mixing ratio (S / N ratio) between the target sound signal and the noise signal in the mixed sound signal is known, the technology for recognizing the speech included in the acoustic signal, or the phoneme recognition technology Has proposed a technique for improving recognition accuracy (Non-patent Document 1).

Gales, MJF and Yound, S. `` An improved approach to the hidden Markov model decomposition of speech and noise '', Proceedings of the 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 1997), pp.835-838 ( 1997)

  Conventionally, since it is assumed that the mixing ratio (S / N ratio) is known, there is a problem that the estimation accuracy of the mixing ratio is deteriorated when the fluctuation amount of the noise signal included in the mixed sound signal is increased.

  An object of the present invention is to provide a mixed sound signal mixing ratio estimation method and system capable of estimating a mixing ratio between a target sound signal and a noise signal in a mixed sound signal.

  In addition to the above object, another object of the present invention is to provide a mixed sound signal mixing ratio estimation method capable of estimating the fundamental frequency F0 together when estimating the mixing ratio of the voiced sound signal. is there.

  Another object of the present invention is to provide a phoneme recognition method for performing phoneme recognition using an estimated mixing ratio.

  An object of the present invention is to improve a mixing ratio estimation method of a mixed sound signal by using a computer to estimate a mixing ratio of a target sound signal and a noise signal included in one frame signal obtained discretely from the mixed sound signal. To do. In the present specification, the target sound signal includes an audio signal (including a singing voice signal), an acoustic signal of a musical instrument, and the like. The noise signal is a signal other than the target sound signal included in the mixed sound signal. Further, “one frame signal obtained discretely” is a signal obtained from a mixed sound signal using a Hanning window having a predetermined time width as one frame.

  In the present invention, one or more target sound spectrum templates showing the relationship between the frequency components of one or more learning target sound signals and the probability distribution of the power spectrum are prepared. In addition, one or more noise spectrum templates indicating the relationship between the frequency component of one or more learning noise signals and the probability distribution of the power spectrum are prepared. Then, one or more stochastic spectrum templates are created by combining one or more target sound spectrum templates and one or more noise spectrum templates.

  In the present specification, a set of probability distributions in which a spectrum of a mixed sound signal including speech (including singing voice) and the like exists is called a probabilistic spectrum template (Probabilistic_Spectral Template).

  Here, the learning target sound signal is one or more learning sound signals collected according to the target sound. For example, when the target sound is a voice, a single sound signal such as a voiced sound such as a vowel or a consonant sound or an unvoiced sound becomes the learning target sound signal. In order to improve the accuracy, it is preferable to obtain a plurality of single sound signals as learning target sound signals from a plurality of human sound signals. Depending on the mixed sound signal to be observed, a plurality of types of learning target sound signals may be used, divided into types such as male audio signals, female audio signals, and child audio signals. When the target sound is a stringed instrument sound, a single tone signal of a certain stringed instrument becomes the learning target sound signal. When the target sound is a percussion instrumental sound, a single percussion instrument sound signal is used for learning. This is the target sound signal.

  In the specification of the present application, the learning noise signal is a sound signal other than the sound signal of the target sound included in the target mixed sound signal. If the music signal of the music including the singing voice is a mixed sound signal, the singing voice is the target sound and the background accompaniment sound is the noise sound. Therefore, the learning noise sound is appropriately selected in consideration of the type of noise sound included in the target mixed sound signal. If there is a sound signal only for the singing voice, the sound signal only for the singing voice becomes the learning target sound signal, and if there is a sound signal only for the accompaniment, the sound signal only for the accompaniment becomes the learning noise signal. Such a learning target sound signal and a learning noise signal are obtained individually.

  However, the learning target sound signal and the learning noise signal may not be easily available. Therefore, in such a case, both the target sound spectrum template of the learning target sound signal and the noise spectrum template of the learning noise signal may be estimated from the learning mixed signal. In this case, the learning mixed sound is configured by mixing a sound signal corresponding to the target sound and a sound signal corresponding to noise. For example, if the target sound is a singing voice, a certain sound signal including a singing voice and an accompaniment sound is a mixed sound signal, and if the target sound is a voice such as speech, the sound signal including background noise is mixed. It is a sound signal.

  If the mixed sound signal to be observed is a mixed sound signal including a female vocal singing voice, it is preferable to use a mixed sound signal including a female vocal singing voice as one or more learning mixed sound signals. Even if the sound signal is different in type from the mixed sound signal to be observed, a certain number of mixed sound signals are collected as learning mixed sound signals, and a plurality of target sound spectrum templates and If a plurality of noise spectrum templates are estimated, averaged learning data can be acquired, so that no major problem arises in the reduction in accuracy.

  In the method of the present invention, an observation spectrum in one frame is acquired from the mixed sound signal to be observed. An observation spectrum is a spectrum waveform showing the relationship between the frequency of a signal in one frame obtained from a mixed sound signal and the power spectrum. In the present invention, a plurality of gain change spectrum templates obtained by changing gains of one or more target sound spectrum templates and gains of one or more noise spectrum templates constituting one or more stochastic spectrum templates, and the observed spectrum The gain change spectrum template having the smallest distance is determined as the minimum distance gain change spectrum template. Then, the mixing ratio is estimated based on the gain of the minimum distance gain change spectrum template and the gain of the noise spectrum template.

  The quasi-Newton method can be used for optimization for determining the gain. Based on the gain Gs of the target sound spectrum template of the determined minimum distance gain change spectrum template and the gain Gn of the noise spectrum template, the mixing ratio (S / N ratio) of the mixed sound signal of one frame is estimated. Specifically, Gs / Gn is the mixing ratio of the mixed sound signal of one frame.

  According to the present invention, it is possible to recognize the mixing ratio as it is without separating the spectrum in a state where the target sound (speech, singing voice, etc.) is mixed with other noise (accompaniment sound, etc.). According to the present invention, since information about background noise is also used, the target sound and noise sound constituting the mixed sound are separated in order to recognize the mixed sound, and then compared with the conventional technique of recognizing the separated sound. Thus, the estimation accuracy can be improved. Further, according to the present invention, since the S / N ratio is estimated for each frame of the mixed sound signal, there is an advantage that it is robust against noise fluctuations.

  If the target sound signal is a sound signal having a harmonic structure such as a voiced sound signal, the target sound spectrum template is determined by the product of the driving sound source function and the voice envelope template. The driving sound source function is a filter that indicates a frequency component of a standard spectrum of a harmonic structure of a sound signal having a harmonic structure such as a voiced sound signal. When the driving sound source function is used, the fundamental frequency F0 of the driving sound source function is estimated at the same time when the minimum distance gain change spectrum template is determined. The above-described quasi-Newton method can also be used when estimating the fundamental frequency F0. When the driving sound source function is used, since the spectrum envelope of the spectrum of the target sound signal is not estimated, there is an advantage that a sound having a harmonic structure can be expressed as it is.

  If the target sound signal is a sound signal, the target sound spectrum template is a sound spectrum template. If the sound signal having the harmonic structure is a voiced sound signal, the target sound spectrum template is determined by the product of the driving sound source function indicating the frequency component of the standard spectrum of the harmonic structure of the voiced sound signal and the sound envelope template. It is done. If the target sound signal is an unvoiced sound signal, the target sound spectrum template is a voice envelope template. Here, the speech envelope template is a plurality of peaks in power included in a plurality of frequency spectrum waveforms indicating a relationship between frequency components and power obtained by frequency analysis of a learning sound signal collected for a target voiced or unvoiced sound. It is a template which shows the distribution state of the envelope which connects.

  The power spectrum probability distribution is preferably represented by a lognormal distribution at each frequency. If the logarithmic normal distribution is used, calculation for estimation becomes easy.

  If the type of the target sound is not known, a plurality of target sound spectrum templates may be prepared in advance.

  According to the present invention, it is possible to estimate the mixing ratio of one frame unit of the mixed sound signal to be observed with higher accuracy than before. If the driving sound source function is used when the target sound is a voiced sound, the fundamental frequency F0 of the driving sound source function can be estimated simultaneously when determining the minimum distance gain change spectrum template.

  A mixture ratio estimation system that implements the mixture ratio estimation method of the present invention includes a spectrum template storage unit, a stochastic spectrum template creation unit, an observed spectrum acquisition unit, a determination unit, and a mixture ratio estimation unit.

  The spectrum template storage unit includes one or more target sound spectrum templates indicating the relationship between the frequency distribution of one or more learning target sound signals and the probability distribution of the power spectrum, the frequency component of one or more learning noise signals, and the power spectrum. One or more noise spectrum templates indicating a probability distribution relationship are stored. The stochastic spectrum template creation unit creates one or more stochastic spectrum templates by combining one or more target sound spectrum templates and one or more noise spectrum templates. The observation spectrum acquisition unit acquires an observation spectrum in one frame from the mixed sound signal. Then, the determination unit has the smallest distance between the observed spectrum and a plurality of gain-change spectrum templates obtained by changing the gain of the target sound spectrum template and the gain of the noise spectrum template that respectively constitute one or more stochastic spectrum templates. The gain change spectrum template is determined as the minimum distance gain change spectrum template. The estimation unit estimates the mixture ratio based on the gain of the minimum distance gain change spectrum template and the gain of the noise spectrum template.

  The system of the present invention may include a template generation unit that generates one or more target sound spectrum templates and one or more noise spectrum templates. The template generation unit, when the target sound signal is a voiced sound signal having a harmonic structure, a target sound spectrum template, a driving sound source function indicating a frequency component of a standard spectrum of the harmonic structure of the voiced sound signal, and a sound If the target sound signal is determined as a product of the envelope template and the target sound signal is an unvoiced sound signal, the voice envelope template can be used as the target sound spectrum template.

  The template generation unit may be configured to estimate both the target sound spectrum template and the noise spectrum template from the learning mixed signal.

  In the phoneme recognition method of the present invention, the phoneme corresponding to the minimum distance gain change spectrum template obtained by the mixing ratio estimation method in the mixed sound signal is determined as one frame of phoneme. Then, the type of speech is determined based on the determined continuity of phonemes of a plurality of frames. Here, “continuity of phonemes in frames” means a property indicating the tendency of the same phonemes to appear continuously in a plurality of frames in an actual signal.

It is a block diagram which shows the structure of an example of the phoneme recognition system provided with embodiment of the mixing ratio estimation system of the mixed sound signal of this invention which implements the mixing ratio estimation method of the mixed sound signal of this invention. It is a flowchart which shows the algorithm of the program used when implement | achieving embodiment of FIG. 1 using a computer. (A) thru | or (c) is a figure used in order to demonstrate the production | generation process of the audio | voice spectrum template as an object sound spectrum template in case the object sound is an audio | voice. (A) to (d) show the process of generating the stochastic spectrum template Y _f based on the speech spectrum templates _{v and f} and the noise spectrum template, and the stochastic spectrum templates Y _{n and f} and the observed spectrum y (f It is a figure used in order to explain the process of calculating | requiring the distance (likelihood) between these. It is a figure which shows the outline | summary of the phoneme recognition method. It is a figure which shows an example of the algorithm of the program which calculates | requires the distance (likelihood) of gain change spectrum template _Y'f and observed spectrum y (f) using a computer. It is a figure which shows an example of the algorithm of estimation of the fundamental frequency F0 in step ST12 of FIG. It is a flowchart which shows an example of the algorithm of the program used when estimating phoneme using a computer. (A) thru | or (d) is a figure which shows the example of the estimation process of a parameter. It is a flowchart of the algorithm of the program used when parameter estimation is implemented with a computer. It is a flowchart which shows the algorithm for estimating a target sound spectrum template and a noise spectrum template from the mixed sound signal for learning. It is a figure which shows the concept of sampling typically.

  FIG. 1 is a block diagram showing a configuration of an example of a phoneme recognition system including an embodiment of a mixed sound signal mixing ratio estimation system of the present invention that implements a mixed sound signal mixing ratio estimation method of the present invention. FIG. 2 is a flowchart showing an algorithm of a program used when the embodiment of FIG. 1 is realized using a computer. FIG. 3 is a diagram used for explaining a generation process of a voice spectrum template as a target sound spectrum template when the target sound is voice. FIG. 4 illustrates a process of generating a stochastic spectrum template based on a speech spectrum template and a noise spectrum template and a process of obtaining a distance (likelihood) between the stochastic spectrum template and the observed spectrum. FIG.

  The mixture ratio estimation system 1 according to the present embodiment includes a template generation unit 2, a spectrum template storage unit 3, a stochastic spectrum template creation unit 9, an observed spectrum acquisition unit 14, a determination unit 15, and a mixture ratio estimation unit. 25. The template generation unit 2 generates a target sound spectrum template and a noise spectrum template. The template generation unit 2 employed in the present embodiment is configured to be able to perform either of the two generation methods. When the first generation method is performed, the template generation unit 2 adjusts the target sound signal. The target sound spectrum template is determined by the product of the driving sound source function indicating the frequency component of the standard spectrum of the harmonic structure of the voiced sound signal and the voice envelope template when the voice sound signal has a wave structure; If the sound signal is an unvoiced sound signal, a speech envelope template is used as the target sound spectrum template. When the second generation method is performed, the template generation unit 2 is configured to estimate both the target sound spectrum template and the noise spectrum template from the learning mixed signal. The first and second generation methods will be described in detail later.

The spectrum template storage unit 3 includes a target sound spectrum template storage unit 5 that stores the target sound spectrum template generated by the template generation unit 2 and a noise / spectrum template storage unit 7 that stores a noise / spectral template generated by the template generation unit 2. It consists of and. The target sound spectrum template storage unit 5 includes a plurality of target sound spectrum templates prepared in advance based on a plurality of learning target sound signals (specifically, “speech spectrum template _v in order to be used for phoneme recognition in the present embodiment). _{, F} "). For example, as shown in FIG. 3C, the target sound spectrum template is a probability distribution (probability density) of frequency components and power spectra of a plurality of learning target sound signals created based on a plurality of learning target sound signals. It is a template which shows the relationship. For example, when the target sound is an audio signal, the relationship between the obtained frequency component and the probability distribution (probability density) of the power spectrum is shown for a plurality of learning monotone signals such as vowels and consonant voiced and unvoiced sounds. The template is a plurality of target sound spectrum templates.

  Here, the one or more learning target sound signals are one or more learning sound signals collected according to the target sound. For example, when the target sound is speech, voiced sounds such as vowels and consonants, and unvoiced sounds Are obtained from a plurality of human voice signals. Depending on the mixed sound signal to be observed, the learning target sound signal may be divided into types such as a male voice signal, a female voice signal, and a child voice signal. The one or more learning noise signals are sound signals other than the sound signal of the target sound included in the target mixed sound signal. The learning noise sound is appropriately selected in consideration of the type of noise sound included in the target mixed sound signal. For example, if there is a sound signal only for a singing voice, the sound signal only for this singing voice becomes a learning target sound signal, and if there is a sound signal only for accompaniment, the sound signal only for this accompaniment becomes a learning noise signal.

  The learning mixed sound signal is formed by mixing a sound signal corresponding to the target sound and a sound signal corresponding to noise. For example, if the target sound is a singing voice, a certain sound signal including a singing voice and an accompaniment sound is a mixed sound signal, and if the target sound is a voice such as speech, the sound signal including background noise is mixed. It is a sound signal.

  If the mixed sound signal to be observed is a mixed sound signal including a female vocal singing voice, it is preferable to use a mixed sound signal including a female vocal singing voice as one or more learning mixed sound signals. However, even if the sound signal is different from the observed mixed sound signal, a certain number of mixed sound signals are collected as learning mixed sound signals, and a plurality of learning target sound signals are obtained from each learning mixed sound signal. If a plurality of learning noise signals are acquired and a plurality of target sound spectrum templates and a plurality of noise spectrum templates are prepared, averaged learning data can be acquired. .

If the target sound signal is a voiced sound signal, the template generation unit 2 sets the target sound spectrum template as the driving sound source function H (f; f ₀ ) shown in FIG. 3B and the voice envelope template shown in FIG. It is generated by the product of Y ′ _{v and f} . The driving sound source function (f; f ₀ ) is a filter indicating a frequency component of a standard spectrum of the harmonic structure of the voiced sound signal. The fundamental frequency F ₀ of the appropriate driving sound source function H (f; f ₀ ) is simultaneously determined when optimizing the gains of the speech spectrum templates Y _{v and f} and the noise spectrum template or the weight parameters g _v and g _n described later. Will be decided.

As shown in FIG. 3A _{, the} speech envelope template Y ′ _{v, f} includes frequency components obtained by frequency analysis of one or more learning target sound signals collected for the target sound (voiced sound or unvoiced sound). It is a template which shows the distribution state (probability density) of the envelope which connects the some peak in the power contained in the frequency spectrum waveform which shows the relationship of power. The shading shown in the speech envelope template Y ′ _{v, f} in FIG. 3A indicates the distribution state (probability density). The voice envelope template Y ′ _{v, f} is prepared for each target sound. In the case of phoneme recognition, a speech envelope template Y ′ _{v, f} is prepared for every phoneme to be recognized. As described above, when the target sound is a voiced sound, as shown in FIG. 3, the product of the driving sound source function H (f; f ₀ ) and the speech envelope template Y ′ _{v, f} shown in FIG. Is stored in the target sound spectrum template storage unit 5. Driving sound source function H (f; f ₀ ) and voice envelope template Y ′ _{v, f}
Is stored in an internal memory in the template generation unit 2, and the product of the two is executed by the calculation unit in the template generation unit 2.

When the target sound is an unvoiced sound, the voice envelope template Y ′ _{v, f} stored in the internal memory by the template generation unit 2 is stored in the target sound spectrum template storage unit 5 as the target sound spectrum template.

  The noise / spectrum template storage unit 7 stores one or more types of noise / spectrum templates [see FIG. 4B]. The noise spectrum template is a template indicating the relationship between the frequency component of the learning noise signal and the probability distribution of the power spectrum. Here, the learning noise signal is a sound signal other than the sound signal of the target sound included in the mixed sound signal to be observed. Noise also varies depending on the type of mixed sound signal. Therefore, the learning noise sound is appropriately selected in consideration of the type of noise sound included in the target mixed sound signal. That is, it is preferable to create a noise spectrum template according to the type of the mixed sound signal (according to the type of music such as a pop music signal, a classical music signal such as an opera). In the present embodiment, the template generator 2 creates a noise spectrum template based on the relationship between the frequency component of the learning noise signal and the probability distribution of the power spectrum, and stores it in the noise spectrum template storage unit 7. Let In the present embodiment, a plurality of types of noise / spectrum templates are stored in the noise / spectrum template storage unit 7 in accordance with the type of mixed sound signal to be observed. The shading shown in the noise spectrum template of FIG. 4B indicates the probability density.

The stochastic spectrum template creation unit 9 includes a combination unit 11 and a stochastic spectrum template storage unit 13. The combination unit 11 includes one or more target sound spectrum templates stored in the target sound spectrum template storage unit 5 and one or more types of noise spectrum templates stored in the noise spectrum template storage unit 7. One or more probabilistic spectral templates are created by combining them one by one. When there are 100 target sound spectrum templates (speech spectrum templates) and two noise spectrum templates, 200 stochastic spectrum templates are combined and combined by the combining unit 11. The 200 stochastic spectrum templates are stored in the stochastic spectrum template storage unit 13. FIG. 4 (c) shows an example of probabilistic spectral template Y _f.

  The observed spectrum acquisition unit 14 performs frequency analysis on one frame signal obtained discretely from the mixed sound signal to be observed, and observes the observed spectrum y () indicating the relationship between the frequency and the power spectrum as shown in FIG. f) is obtained. Specifically, a Hanning window having a predetermined time width is used as one frame to acquire one frame signal from the mixed sound signal, and frequency analysis is performed to acquire an observation spectrum.

The determination unit 15 includes a selection unit 17, a distance calculation unit 19, a temporary storage unit 21, and a determination unit 23. The selection unit 17 selects the stochastic spectrum templates from the stochastic spectrum template storage unit 13 in order. The distance calculation unit 19 obtains the gain Gs (weight parameter g _v ) of the target sound spectrum template and the gain G _n (weight parameter g _n ) of the noise spectrum template constituting one selected stochastic spectrum template. A distance (likelihood) between a plurality of gain change spectrum templates Y ′ _f and the observed spectrum y (f) is obtained, and a gain change spectrum template having the smallest distance is obtained as the minimum distance gain change spectrum template in the stochastic spectrum template. Y ′ is determined as _fmin . The temporary storage unit 21 stores the minimum distance gain change spectrum template Y ′ _fmin . After _{obtaining the} minimum distance gain change spectrum template Y ′ _fmin for all the stochastic spectrum templates stored in the stochastic spectrum template storage unit 13 and storing them in the temporary storage unit 21, the determination unit 23 performs a plurality of stochastic operations. The minimum distance gain change spectrum template Y ′ _fmin having the shortest distance among the plurality of minimum distance gain change spectrum templates determined for the spectrum templates and stored in the temporary storage unit 12 is determined. Then, the estimation unit 25 performs mixing based on the gain Gs (weight parameter g _v ) of the target sound spectrum template of the determined minimum distance gain change spectrum template Y ′ _fmin and the gain G _n (weight parameter g _n ) of the noise spectrum template. Estimate the ratio Gs / Gn. For example, if there are 100 target sound spectrum templates and two noise spectrum templates, there are 200 sets of stochastic spectrum templates, so that the target sound spectrum templates constituting each of these 200 sets of stochastic spectrum templates are present. 200 and the gain of the noise spectrum template are changed to determine 200 sets of the aforementioned candidates. Then, the one having the smallest distance from the observed spectrum among the 200 sets of candidates is determined as the minimum distance gain change spectrum template. The quasi-Newton method can be used for optimization for determining the gain.

  The mixing ratio Gs / Gn of the mixed sound signal for one frame estimated by the estimation unit 25 is stored in the estimation result storage unit 27 together with the identification information (information specifying the type of phoneme) of the target sound spectrum template. Based on the data stored in the estimation result storage unit 27, the phoneme recognition unit 29 determines a phoneme corresponding to the minimum distance gain change spectrum template as a phoneme of one frame. Then, the type of speech is determined based on the continuity of phonemes in the determined frame. Here, “continuity of phonemes in frames” means a property indicating the tendency of the same phonemes to appear continuously in a plurality of frames in an actual signal. For example, the length that one vowel continues in a singing voice may be 100 times or longer than one frame period.

  Therefore, when determining the phoneme of the singing voice based on the phoneme of the frame, all or most of the phonemes of a plurality of consecutive frames are always the same. Therefore, in the present embodiment, the type of speech is determined based on the continuity of phonemes in frames. In this way, phoneme recognition can be performed without extracting only the audio signal from the mixed sound signal.

  Next, the flowchart shown in FIG. 2 showing the algorithm of the program when the embodiment shown in FIG. 1 is implemented using a computer will be described. This flowchart is an example, and the present invention is not limited to this flowchart. First, in step ST1, a plurality of probability spectrum templates are created. Therefore, a stochastic spectrum template is created to execute step ST1. That is, a plurality of target sound spectrum templates prepared in advance based on a plurality of learning target sound signals and one or more types of noise spectrum templates prepared in advance based on a plurality of mixed sound signals for learning are combined and synthesized one by one. To create a plurality of stochastic spectral templates. Next, in step ST2, an observation spectrum in one frame is acquired from the mixed sound signal. In step ST3, for each of a plurality (or theoretically even one) of stochastic spectrum templates, a plurality of gains obtained by changing the gain of the target sound spectrum template and the gain of the noise spectrum template constituting the stochastic spectrum template. The gain change spectrum template that minimizes the distance between the gain change spectrum template and the observed spectrum is determined as the minimum distance gain change spectrum template. In step ST4, the gain of the target sound spectrum template and the noise spectrum template of the minimum distance gain change spectrum template having the smallest distance among the plurality of minimum distance gain change spectrum templates respectively determined for the plurality of stochastic spectrum templates. Based on the gain, the mixing ratio is estimated.

[Specific application examples]
Next, an embodiment in which the lyrics (phonemes) of the singing voice and the fundamental frequency (F0) in the mixed sound signal are simultaneously recognized using the mixing ratio estimation method and system of the above embodiment will be described. The lyrics express the content that the singer wants to convey with the singing voice, and the fundamental frequency F0 expresses the melody of the music and at the same time expresses the skill and expression of the singer, and both are important elements constituting the singing voice. Therefore, the technology for automatically recognizing these elements from the mixed sound can be applied to music information retrieval and the like and is an important basic technology. For example, by recognizing the lyrics, it is possible to search for songs with unknown lyrics using the lyrics as clues. In addition, the automatic phoneme recognition technology can be applied to temporal correspondence between lyrics and music, and can be applied to music players that display lyrics like karaoke, automatic creation of music video telops, and the like. The estimation of the singing voice fundamental frequency (F0) can be applied to automatic transcription of vocal parts, humming search, and the like. Furthermore, it has been reported that the accuracy of the Hamming search is improved by integrating the lyrics information into the Hamming search, and the range of application is further expanded by simultaneously estimating the lyrics and F0. However, the singing voice has more fluctuations due to vibrato, F0 variation range, singer's emotional expression, etc., and the accompaniment sound is superimposed at a louder volume than the spoken voice, so the singing voice (phoneme) is recognized Has a very difficult problem.

  The inventors have so far studied the temporal association method between music and lyrics (the following papers 1 and 2) and the F0 estimation method of the singing voice in the mixed sound (the following paper 3).

[Article 1]
Fujihara, H. and Goto, M., "Three Techniques for Improving Automatic Synchronization between Music and Lyrics: Fricative Sound Detection, Filler Model, and Novel Feature Vectors for Vocal Activity Detection", Proceedings of the 2008 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP2008), pp.69-72 (2008).
[Article 2]
"Automatic synchronization between lyrics and music CD recordings based on Viterbialignment of segregated vocal signals" by Fujihara, H, Goto, M., Ogata, J., Komatani, K., Ogata, T. and Okuno, HG, Proc. ISM Pp.257-264 (2006).
[Papers 3]
Hiromasa Fujiwara, Masataka Goto, and Hiroshi Okuno, “Pitch estimation method for vocal parts in multiple playing using statistical modeling of singing voice and Viterbi search”, Transactions of Information Processing Society of Japan, Vol.49, No.10 (2008 ).
In the methods described in the above paper, in common, the approach was to separate the sound from the mixed sound by using the harmonic structure as a clue, and identify it by a statistical method. Specifically, in the case of lyric temporal correspondence, it identifies which phoneme is the sound of the singing voice F0 estimated by the existing method, and in the case of the singing voice F0 estimation, the frequency component candidates at each time are determined. They identified whether it was a singing voice or other sounds. However, these methods have the following two problems.

[Separation issues]
The recognition performance of the singing voice largely depended on the performance of separation performed in the previous stage. For this reason, errors in F0 estimation and processing for selecting a harmonic component from a spectrum during separation have an adverse effect on performance. Also, background noise (sounds other than the sound to be separated) containing information such as the S / N ratio of singing voice and noise and the degree of distortion of the singing voice was thrown away in the separation process.

[Problem of spectral envelope estimation]
In the conventional method, the singing voice is recognized by estimating the spectral envelope from the harmonic structure of the separated singing voice and calculating the distance between the spectral envelopes. However, since each harmonic component of the harmonic structure can be considered as a sample of frequency components that are integer multiples of F0 from the original spectral envelope, the original spectral envelope must be uniquely restored from the given harmonic structure. Was impossible in principle. For this reason, it is difficult to accurately calculate the distance, for example, when the width of the valley of each harmonic component of the harmonic structure is wide, such as a sound with a high F0.

  In the present embodiment, the singing voice is not separated, and the spectral envelope is not estimated from a single harmonic structure, and an as-is shape in which accompaniment sounds are superimposed on the observed spectrum is probabilistically modeled. Furthermore, in the learning process, the spectral envelope is estimated more accurately by using a plurality of harmonic structures.

  Specifically, as shown in FIGS. 4C and 4D, it is assumed that the spectrum of a mixed sound signal including a singing voice is generated from a set of probability distributions. Here, it is assumed that the power appearing in each frequency bin (frequency analysis width) of the spectrum follows a certain probability distribution, and the probability distribution is different for each of the plurality of spectrum bins. Assuming that the spectrum is additive, the stochastic spectrum template includes a voice (singing voice) spectrum template expressing a singing voice (FIG. 4 (a)) and a noise spectrum template expressing a sound other than the singing voice (FIG. 4 (b)). Can be expressed by addition on the linear axis. Then, by introducing a weight parameter (gain adjustment) when adding these two spectrum templates and adding them with weights, it is possible to express spectra with various S / N ratios. Further, assuming a source filter model, the voice (singing voice) spectrum template represents the harmonic structure of the voice (singing voice) envelope template (FIG. 3A) expressing the spectral envelope and the driving source. It is considered to be generated by the product of the driving sound source function (Harmonic Filter) [FIG. 3 (b)]. The shape of the driving sound source function can be controlled using the value of the fundamental frequency F0 as a parameter.

If F0 of the driving sound source function, which is a parameter of the probability model, and the weights of the voice (singing voice) spectrum template and the noise spectrum template are determined, the likelihood (distance) for the probability model (probabilistic spectrum template) of the observed spectrum Can be calculated. When this model is used, as shown in FIG. 5, a speech (singing voice) envelope template Y ′ _{v, f} [phoneme / a /, phoneme / b /,... Phoneme / o /. Can be recognized in advance by selecting the most likely (closest distance) speech (singing voice) envelope template Y ′ _{v, f} with respect to the observed spectrum. F0 can be estimated by estimating the value of (close) F0. As described in the first embodiment described with reference to FIG. 3, a speech (singing voice) envelope template [phoneme / a /, phoneme / b /,... Phoneme / o / ..] expressing each phoneme. And a sound source function H (f _i , f ₀ ), and a plurality of speech (singing voice) spectrum templates (target sound spectrum templates) Y _{v, f} representing the spectrum template of each phoneme are created. Next, as shown in FIG. 4, a product of a plurality of speech (singing voice) spectrum templates Y _{v, f} and a noise spectrum template Y _{n, f} representing the spectrum template of each phoneme is taken (combined) to obtain a plurality of creating a plurality of probabilistic spectral template Y _f for voice (vocal) spectral template.

In order to determine the respective weights of the speech (singing voice) spectrum template and the noise spectrum template constituting the confirmation spectrum template of each phoneme, the gain Gs (weight parameter g _{v) of} the target sound spectrum template constituting each stochastic spectrum template ) And the gain Gn (weight parameter g _n ) of the noise spectrum template are changed to obtain a plurality of gain change spectrum templates Y ′ _f for each phoneme. Then, the distance (likelihood) between the plurality of gain change spectrum templates Y ′ _f and the observed spectrum y (f) for each phoneme is obtained, and the gain change spectrum template having the smallest distance is determined as the minimum in the stochastic spectrum template. The distance gain changing spectrum template Y ′ _fmin is determined. That is, the smallest distance gain change spectrum template Y ′ _fmin for the phoneme is the one with the smallest distance (likelihood) among the plurality of gain change spectrum templates Y ′ _f for each phoneme. 'Seeking _fmin, among a plurality of minimum distance gain changing spectral template determined, the distance is the smallest minimum distance gain changing spectral template Y' minimum distance gain changing spectral template Y Stochastic spectral template for all phonemes in _fmin The corresponding phoneme is determined as the recognized phoneme.

FIG. 6 shows an example of an algorithm of a program for obtaining the distance (likelihood) between the above-described gain-change spectrum template Y ′ _f and the observed spectrum y (f) using a computer. In this algorithm, the initial value of the fundamental frequency F0 is set in step ST11, the gain of the speech spectrum template is set, and the initial value of the gain of the noise spectrum template is set. In step ST12, the optimum gain and F0 are estimated by a nonlinear optimization method such as a quasi-Newton method. In step ST13, the obtained gain and likelihood for the F0 value are calculated.

  FIG. 7 shows an example of an algorithm for estimating the fundamental frequency F0 in step ST12. In this algorithm, a plurality of F0 candidates are estimated from the observed spectrum in step ST21. For estimation of this F0 candidate, a known estimation method such as a method using the value of the frequency peak of the observed spectrum or a method of estimating based on the response of the comb filter can be used.

  In step ST22, the following loop 1 is started for all the F0 candidates. In step ST23, the following loop 2 is started for all speech spectrum templates. In step ST24, the following loop 3 is started for all speech spectrum templates. In step ST25, using the value of the F0 candidate as an initial value, the optimum F0 is calculated and stored based on the likelihood of the speech spectrum template, noise spectrum template, and observed spectrum. The optimum F0 is calculated using Step 0 to Step 3 in the explanation of “parameter estimation” described later. At this time, the value of the F0 candidate is used as the initial value of F0 given in Step 0. In step ST26, the loop 3 is terminated, and in step ST27, the loop 2 is terminated. In step ST28, the F0 value and the likelihood when the likelihood is the greatest in the loop 2 and the loop 3 are stored. In step ST29, loop 1 is terminated, and in step ST30, F0 having the highest likelihood in loop 1 is output as an estimation result.

  FIG. 8 is a flowchart showing an example of a program algorithm used when phonemes are estimated using a computer. In this algorithm, the following loop 1 is started for all phonemes in step ST31. In step ST32, the following loop 2 is started for all speech spectrum templates of the phoneme. In step ST33, the following loop 3 is started for all noise spectrum templates. In step ST34, the likelihoods of the speech spectrum template, noise spectrum template, and observed spectrum are calculated and stored. In step ST35, the loop 3 is terminated, and in step ST36, the loop 2 is terminated. In step ST37, the value having the highest likelihood in loop 2 and loop 3 is stored as the likelihood of this phoneme. In step ST38, loop 1 is terminated, and in step ST39, the phoneme having the highest likelihood in loop 1 is output as an estimation result.

  According to this specific embodiment, a state in which noise (accompaniment sound) is mixed is expressed as it is without separating voice (singing voice). This specific embodiment is a natural method from the viewpoint of human perception, considering that humans can recognize voices without separating voices (singing voices). In the method of the present embodiment, since the S / N ratio between voice (singing voice) and noise (accompaniment sound) can be estimated for each frame, the system is robust against fluctuations in noise (accompaniment sound). Furthermore, the system can be made more robust by preparing a plurality of noise spectrum templates and selecting the most likely one.

  In this embodiment, since the spectral envelope is not estimated from a single harmonic structure, the system is robust even for sounds with high F0. Furthermore, in the present embodiment, it is possible to easily extend the sound (singing voice) spectrum template that does not use the driving sound source function for other sounds and sound sources such as unvoiced consonants without F0.

[Formulation]
A specific formulation of the method and system described above will be described below. In implementing the method of the present invention in a computer, the following three methods are embodied.

  (1) A method of expressing a stochastic spectrum template.

  (2) A method for calculating the addition of two spectral templates.

  (3) A method for optimizing F0 and gain as parameters.

  In order to embody the above three methods, the following approach is taken.

  (1) A lognormal distribution is used as the distribution of each frequency bin of the stochastic spectrum template.

  (2) It is assumed that a random variable obtained by adding random variables that follow a lognormal distribution follows a lognormal distribution.

  (3) The parameters are optimized by the quasi-Newton method.

[Probabilistic spectral template]
It is assumed that the spectrum y (f) of the mixed sound including voice (singing voice) is generated from the random variable Yf. Where f represents the frequency on the logarithmic axis, and s represents the power of the spectrum on the logarithmic axis. This random variable (set) Y _f is the aforementioned stochastic spectrum template.

Now assume that Y _f can be divided into two different spectral templates according to:

However, _{Yv, f} represents the spectrum of the voice (singing voice) and is the aforementioned voice (singing voice) spectrum template. Yn, f represents the spectrum of a sound (noise or accompaniment sound) other than voice (singing voice), and is the above-described noise spectrum template. gv and gn are weights of the voice spectrum template and the noise spectrum template, and the S / N ratio of the voice (singing voice) and other sounds can be changed by changing them. In equation (1), it is assumed that the spectrum is additive on the linear axis. It is assumed that Yv _{, f} and Yn, f follow a normal distribution (on the logarithmic frequency axis) as in the following equation.

Here, N (y; μ, σ ² ) is a normal distribution with mean μ and variance σ ² . Furthermore, assuming a source filter model, the voice (singing voice) Y _{v, f} having a harmonic structure is added on the logarithmic axis of the envelope probability model and the filter expressing the harmonic structure as shown in the following equation: (Fig. 3).

Here, Y ′ _{v, f} ˜N (y; μ ′ _{v, f} ; σ ² _{v, f} ) is a random variable representing the spectral envelope of speech (singing voice), and is the above-mentioned speech (singing voice) envelope template. is there. H (f; f ₀ ) represents a filter having a value F ₀ of f ₀ and is called a driving sound source function. The driving sound source function H (f; f ₀ ) is not a random variable. To summarize the above, the stochastic spectrum template Y _f expressing a spectrum in which voice (singing voice) and noise (accompaniment sound) are mixed is expressed as follows.

[Approximation of spectral template addition]
Since the stochastic spectrum template Y _f represented by the above equation (1) is difficult to calculate analytically, it is approximated using a normal distribution. Consider the following function _{l (x 1, x 2)} .

The second-order Taylor expansion of (x _1, x ₂ ) = (μ _{v, f} + g _v, μ _{n, f} + g _n ) in the above equation is

It is calculated as follows. However, C ₁ is a constant independent of x ₁ and x ₂ . Where the parameter
If g _v , g _n , and f ₀ are fixed, note that Equation (12) is a weighted addition of x ₁ and x ₂ , and the stochastic spectrum template Y _f is expressed as follows:

And Y _f is

It is expressed as

[Presumption of phonemes and F0]
In order to recognize phonemes and F0 using this model, it is first necessary to prepare a speech (singing voice) envelope template θ ⁱ _v and a noise spectrum template θ _n expressing each phoneme i. When the observation spectrum y (f) is given, the phonemes i and F0 included in y (f) can be estimated by the following equation.

However, u _f and σ ² f are defined by equations (16) and (17), respectively.

[Parameter optimization by quasi-Newton method]
The quasi-Newton method based on the BFGS (Broyden-Fletcher-Goldfarb-Shanno) formula is used for optimization of the parameter θ = (g _v , g _n, f ₀ ) for calculating the equation (19). The quasi-Newton method is a kind of hill-climbing method, and it updates parameters repeatedly. In this model, the objective function Q (θ) to be minimized is

It is represented by However, y (f) is an observed spectrum.

In Newton's method, the objective function is approximated by a quadratic Taylor expansion around the current parameter, and the parameter is updated sequentially. However, the Newton method assumes that the Hessian of the second derivative necessary for calculating the second-order Taylor expansion is positive definite, but this assumption does not necessarily hold. On the other hand, in the quasi-Newton method, stable optimization can be achieved by numerically approximating the change of the first derivative by updating the parameter without using the Hessian matrix directly, as shown in the following equation. is there.

  Here, k represents the number of iterations.

  The parameters can be optimized as follows:

Step 0: Set k = 0 and B ⁽⁰⁾ = I, and initialize θ ⁽⁰⁾ .

Step 1: Update θ ^{(k + 1)} by the following formula.

The value of α ^(k) is determined by linear search.

Step 2: Update B ^{(k + 1)} using equation (21).

  Step 3: Return to step 1.

[Singing voice envelope template estimation]
The voice (singing voice) envelope template Y _{v, f} and the noise spectrum template Y _{n, f} in the equation (4) are estimated from the learning data. In general, the spectrum of a voice having a harmonic structure (singing voice) can be considered as a sample of frequency component points that are integer multiples of the fundamental frequency with respect to the true spectral envelope. For this reason, since the observed spectrum (harmonic structure) and the spectrum envelope that is the source of the spectrum can be in a one-to-many relationship, it is difficult to estimate the true spectrum envelope from the harmonic structure of a single frame. Therefore, in the present embodiment, a highly reliable spectral envelope is estimated by using a harmonic structure of a plurality of frames having different F0 values. In addition, since the spectral envelope is not uniquely determined but estimated as a probability distribution, it is robust against fluctuations in singing voice and differences between learning data and test data. When estimating the original spectral envelope from a plurality of harmonic structures, it is necessary to take into account the difference in volume for each frame. For this reason, this embodiment solves this problem by introducing a parameter for normalizing the volume of each frame and estimating it as an unknown parameter.

[Mixed regression distribution]
A mixed regression model using linear regression as each regression element is introduced as a model expressing a spectrum template. This mixed regression model is described in, for example, Jacobs, RJ, Jordan, M., Nowlan, SJ and Hinton, GE, “Adaptive mixture of local experts”, Neural Computation, Vol. 3, pp. 79-87 (1991). Have been described. As described above, in this embodiment, the spectrum template needs to be defined using a model in which the distribution of logarithmic power at a certain frequency f is expressed by a normal distribution. Satisfies. In the mixed regression model, the mean μ _{v, f} and variance σ ² v, f of the spectral template are expressed as follows.

Here, Gm (f; ψ _m , μ _m , σ ² _m ) is the output of the gate function, and a normalized Gaussian function defined by the following equation was used. This normalized Gaussian function is described in Xu, L., Jordan, MI and Hinton, GE, “An alternative model for combination of experts”, Advances in Neural Information Processing Systems 7, pp. 633-640 (1994). ing.

In this model, the unknown parameters are {ψ _m , μ _m , σ ² _m , a _m , b _m , β ² _m } and can be estimated by the EM (Expectation and Maximization) method. However, ψ _m is ψ _m ≧ 0 and Σ _m ψ _m = 1.

[Parameter estimation]
The frequency f _{i, h} and its logarithmic power y _{i, h} of the harmonic structure si (i = 1, ..., I) of the harmonic structure si (i = 1, ..., I) given as training data is expressed as Suppose that

At this time, the likelihood function to be maximized is expressed by the following equation.

Here, k _i is an offset parameter that normalizes the volume of each harmonic structure. Since it is difficult to simultaneously optimize the parameters of the mixed regression model and k _i , they are updated iteratively.

  The parameters are estimated by the following procedure.

Step 0: Set k _i = 0 and give initial values for other parameters.

  Step 1: Estimate the parameters of the mixed regression model using the EM method.

Step 2: Update k _i by the following formula.

  Return to Step 3: 1.

FIG. 9 is an example of a parameter estimation process. FIG. 9 shows an example of the process of parameter estimation of the mixed regression model. The thick line at the center of each figure represents the average of the mixed regression model, and the two thin lines above and below it represent the standard deviation. The fine points on the background represent the harmonic components of the learning data, and the plurality of peaks at the bottom of the figure represent the gate function G _m (f; ψ _m , μ _m , σ ² _m ). From the figure, it can be seen that by repeating the update, the offset parameter k _i for each harmonic structure of the learning data is optimized, and a regression curve with less variance is estimated. The noise spectrum template can be estimated in the same way by considering s _i (i = 1, ..., I) not as a harmonic structure but as a spectrum itself.

  FIG. 10 shows a flowchart of a program algorithm used when this parameter estimation is executed by a computer. First, in step ST41, parameters are initialized. For initialization of parameters, learning data and a plurality of harmonic structures (each harmonic F0 and power) are used. Next, in step ST42, loop 1 is started with t = 1. In step ST43, using the current offset parameter and the parameters of each mixed regression model, the belonging probability for each mixed regression model of the harmonic structure of the learning data is calculated. In step ST44, the parameters of each mixed regression model are estimated by the EM algorithm using each mixed regression model using the current offset parameter and the belonging probability for the parameter of each mixed regression model. In step ST45, the offset parameter is updated. In step ST46, it is determined whether t has exceeded a certain number of times. If Yes, the process ends in step ST48, and if No, loop 1 is repeated.

  In the above embodiment, it is assumed that the learning target sound signal and the learning noise signal to be used are obtained separately. However, the learning target sound signal and the learning noise signal may not be easily available. Therefore, in such a case, both the target sound spectrum template of the learning target sound signal and the noise spectrum template of the learning noise signal can be estimated from the learning mixed signal. This estimation can be realized by changing the configuration of the template generation unit 2 in FIG. Note that the learning mixed sound is configured by mixing a sound signal of a type to which the target sound belongs and a sound signal corresponding to noise. If the mixed sound signal to be observed is a mixed sound signal including a female vocal singing voice, a mixed sound signal including a female vocal singing voice is used as one or more learning mixed sound signals.

Specifically, when a template is estimated from a learning mixed sound, it is necessary to simultaneously estimate a speech envelope template and a noise spectrum template. FIG. 11 shows an algorithm of a program used when the template generation unit 2 is realized using a computer. In step ST51, parameters are initialized. As a premise, it is assumed that I observation spectra y ₁ (f), ···, y _i (f), ···, y _I (f) are observed. The parameters of the target sound (speech) spectrum template to be estimated are θ _v = {ψ _{v, m} , μ _{v, m} , σ ² _{v, m} , a _{v, m} , b _{v, m} , β ² _{v, m} } The parameters of the noise spectrum template are θ _n = {ψ _{n, m} , μ _{n, m} , σ ² _{n, m} , an _{n, m} , b _{n, m} , β ² _{n, m} }. The target sound spectrum template after adding the driving sound source function in the i-th spectrum can be expressed as follows.

However, it is assumed that f ₀ (i) which is F0 of the i-th observed spectrum is known for all i.

In the previous embodiment, the addition of the lognormal distribution is approximately calculated using the first-order Taylor expansion. However, the obtained equations (15) to (17) have complicated shapes, and it is difficult to optimize the target sound (speech) spectrum template θ _v and the noise spectrum template θ _n . Therefore, in this embodiment, an approach is taken in which the parameter is approximately estimated after the addition of the lognormal distribution is strictly calculated according to the definition. If the probability density function of the spectrum template after synthesis is written as p _{i, f} (y; θ _v , θ _n , g _{i, v} , g _{i, n} ), [the probability density function of each observed spectrum number i is Since the shapes are different, the subscript i is added. ], The objective function L is expressed as follows.

Here, g _{i, v} and g _{i, n} are the offset parameters (weights) for normalizing the sound volume between frames in the same manner as the offset parameters k _i of the previous embodiment. Further, g _{i, v} and g _{i, n} also have a role of adjusting the SIR (Signal-to-Interference Ratio) of the voice (singing voice) envelope template and the noise spectrum template. In the actual implementation, the continuous wavelet transform is calculated discretely with respect to the frequency axis, so the integral over f is replaced with a sum operation.

The parameters to be estimated here are {g _{i, v} , g _{i, n} , θ _v , θ _n }. Since it is difficult to optimize all these parameters at the same time, they are optimized sequentially. First, in step ST52, the weight g _{i, n} and the noise spectrum template θ _n are fixed, and the weight g _{i, v} and the noise spectrum template θ _v are optimized by the above equation (31). In step ST56, Consider that the weight g _{i, v} and the target sound spectrum template θ _v are fixed and the optimization of the weight g _{i, n} and the target sound spectrum template θ _n according to the equation (32) is repeated alternately. First, in step ST52, if g _{i, n} and θ _n are fixed, the inside of the sum of equation (31) can be considered as the calculation of the expected value. Therefore, the calculation of the expected value of the sample U (the calculation including the integral of the normal distribution) is approximated by the calculation of the sum by sampling. Here, sampling means approximating the integral relating to the distribution with the sum of many points, as schematically shown in FIG. This sampling allows approximate optimization of g _{i, v} and θ _v . Specifically, the normal distribution N (U; μ _{n, f} + g _{i, n} , σ ² _{n, f} ) related to the learning noise sound is cut by U = y _i (f). R samples (U _{i, 1, f} ,..., U _{i, r, f} , U _{i, R, for} each i and f from a single-cut normal distribution with bounded upper bounds _{When f} ) is sampled, the objective function L 1 can be approximated as follows:

In a specific embodiment, the value of R is set to 300. Here, if weights g _{i, n} and noise spectrum template θ _n are fixed, π _{i, r, f} and (log (exp (y _i (f)) − exp (U _{i, r, f} )) are constants. since the, using equation (33), the weights g _{i, v} and the target sound spectrum template theta _v optimize (step ST51~ step ST55). the weight g _{i, v} and the target sound spectrum template theta _v Is the same, and the same expression as expression (33) is derived by sampling from expression (31), and the weights g _{i, n} and noise spectrum template θ _n are optimized (step ST56 to step ST59). .

However, since Equation (33) is in the form of the logarithm (log) of the sum (Σ), direct optimization is still difficult. Therefore, Equation (33) is iteratively optimized by an iterative method similar to the EM algorithm. For convenience, the parameter to be estimated is written as λ = {g _{i, v} , θ _v }. Also, let λ ′ be the parameter estimate in the previous iteration. First, consider the following variables z _{i, r, and f} .

Then, z _{i, r, f} calculated using λ ′ is set as z ′ _{i, r, f} (step ST4). At this time, z _{i, r, f} are fixed, and the following new objective function Q ₁ (λ | λ ′) is determined.

Then, the operation for optimizing the objective function with respect to λ 1 and the operation for recalculating z _{i, r, f} using the optimized λ 2 are repeated (repeatedly repeating steps ST53 to ST55). The function L can be maximized. The number of repetitions may be at least once. If you look closely at equation (36), you can see that π _{i, r, f} is irrelevant to optimization. Therefore, it can be seen that the optimization of the following function Q ₂ (λ | λ ′) is equivalent to the optimization of Q ₁ (λ | λ ′).

Furthermore, it can be seen that Q ₂ has the same form as equation (27) except for the existence of the constant term z. Therefore implementing optimization Q ₂ 'functions of the above formula (37) (step ST54). That is, as in the case of the template estimate from a single learning target sound signal described in the above embodiment and training noise signal, Q ₂ function it can be seen that the optimization.

The same operation as above, with the weights g _{i, v} and the target sound spectrum template θ _v fixed, the same expression as the expression (33) is derived by sampling from the expression (31), the weight g _{i, n} and the noise spectrum The template θ _n is optimized (step ST56 to step ST59). Then, when steps ST52 to ST59 are repeated a predetermined number of times (step ST60), the process ends. This iteration may be at least once.

  In summary, parameters are estimated by the following procedure.

Step ST51: g _{i, v} = 0, g _{i, n} = 0, and initial values are given to other parameters as described later.

Step ST52: g _{i, n} and θ _n are fixed, and U in Expression (31) is sampled.

Step ST53: Using the sampled U and the current parameters g _{i, v} , θ _v , z _{i, r, f} in equation (35) is calculated.

Step ST54: Step ST53 Using the calculated z _{i, r, f} , the Q ₂ function of Expression (37) is optimized. This optimization uses an iterative optimization method.

  Step ST55: When the repetition of step ST52 to step 54 exceeds the specified number of times, the process returns to step ST56, and otherwise, the process returns to step ST52.

Step ST56: G _{i, v} and θ _v are fixed, and U in Expression (31) is sampled.

Step ST57: Using the sampled U and the current parameters g _{i, n} , θ _n , z _{i, r, f} in equation (35) is calculated.

Step ST58: The calculated z _{i, r,} using _f, to optimize the Q ₂ function of Equation (37). An iterative optimization method is also used for this optimization.

  Step ST59: If the repetition of steps ST57 to ST58 has exceeded the specified number of times, the process returns to step ST60, and if not, the process returns to step ST57.

  Step ST60: When the repetition of steps ST52 to ST59 exceeds the specified number of times, the process ends. Otherwise, the process returns to step ST52.

  The initial value of the target sound spectrum template is obtained from the target target sound signal to be observed (for example, if the target sound is a song, it is obtained from the acoustic signal of a single singer different from the singer of the target sound. May use the values of the parameters estimated in the previous embodiment from the music acoustic signal (for example, karaoke track) that does not contain a singing voice.

  According to the present invention, a spectrum in a state where the target sound (speech, singing voice, etc.) is mixed with other noise (accompaniment sound, etc.) can be recognized as it is without being separated. Compared to the conventional technique of recognizing mixed sound, separating each sound that constitutes and then recognizing the separated sound, according to the present invention, since information on background noise is also utilized, Can also improve performance. Further, according to the present invention, since the S / N ratio is estimated for each frame of the mixed sound signal, there is an advantage that it is robust against noise fluctuations.

DESCRIPTION OF SYMBOLS 1 Mixing ratio estimation system 2 Template production | generation part 3 Spectrum template memory | storage part 5 Target sound spectrum template memory | storage part 7 Noise / spectrum template memory | storage part 9 Probabilistic spectrum template creation part 11 Combining part 13 Stochastic spectrum template memory | storage part 14 Observation spectrum acquisition part DESCRIPTION OF SYMBOLS 15 Determination part 17 Selection part 19 Distance calculation part 21 Temporary storage part 23 Determination part 25 Estimation part 27 Estimation result storage part 29 Phoneme recognition part

Claims (9)

A method for estimating a mixing ratio of a mixed sound signal by using a computer to estimate a mixing ratio between a target sound signal and a noise signal included in one frame signal obtained discretely from the mixed sound signal,
One or more target sound spectrum templates showing the relationship between the frequency components of one or more learning target sound signals and the probability distribution of the power spectrum, and the relationship between the frequency components of one or more learning noise signals and the probability distribution of the power spectrum. Prepare one or more noise spectrum templates,
Creating one or more stochastic spectrum templates by combining the one or more target sound spectrum templates and the one or more noise spectrum templates;
Obtaining an observation spectrum in the one frame from the mixed sound signal;
Distance between the observed spectrum and a plurality of gain-change spectrum templates obtained by changing the gain of the one or more target sound spectrum templates and the gain of the one or more noise spectrum templates constituting the one or more stochastic spectrum templates Is determined as a minimum distance gain change spectrum template, and the mixing ratio is estimated based on the gain of the minimum distance gain change spectrum template and the gain of the noise spectrum template,
When the target sound signal is a voiced sound signal having a harmonic structure, the target sound spectrum template includes a driving sound source function and a voice envelope template indicating frequency components of a standard spectrum of the harmonic structure of the voiced sound signal. Determined by the product of
If the target sound signal is an unvoiced sound signal, using the speech envelope template as the target sound spectrum template,
The speech envelope template is an envelope that connects a plurality of peaks in the power included in a frequency spectrum waveform indicating a relationship between frequency components and power obtained by frequency analysis of a learning sound signal for a target voiced or unvoiced sound. A method for estimating a mixing ratio in a mixed sound signal, which is a template indicating a distribution state of lines.   The method for estimating a mixture ratio in a mixed sound signal according to claim 1, wherein both the target sound spectrum template and the noise spectrum template are estimated from a learning mixed signal.   The method of estimating a mixing ratio in a mixed sound signal according to claim 1, wherein the fundamental frequency F0 of the driving sound source function is estimated when determining the minimum distance gain changing spectrum template. A method for estimating a mixing ratio of a mixed sound signal by using a computer to estimate a mixing ratio between a target sound signal and a noise signal included in one frame signal obtained discretely from the mixed sound signal,
One or more target sound spectrum templates showing the relationship between the frequency components of one or more learning target sound signals and the probability distribution of the power spectrum, and the relationship between the frequency components of one or more learning noise signals and the probability distribution of the power spectrum. Prepare one or more noise spectrum templates,
Creating one or more stochastic spectrum templates by combining the one or more target sound spectrum templates and the one or more noise spectrum templates;
Obtaining an observation spectrum in the one frame from the mixed sound signal;
Distance between the observed spectrum and a plurality of gain-change spectrum templates obtained by changing the gain of the one or more target sound spectrum templates and the gain of the one or more noise spectrum templates constituting the one or more stochastic spectrum templates Is determined as a minimum distance gain change spectrum template, and the mixing ratio is estimated based on the gain of the minimum distance gain change spectrum template and the gain of the noise spectrum template,
The minimum distance in determining the gain change spectral template, the mixing ratio estimation method in the mixed sound signals and estimates the fundamental frequency F0 of the drive dynamic sound function. The probability distribution of the power spectrum, the mixing ratio estimation method in the mixed sound signal according to claim 1, characterized in that the area is represented by the log-normal distribution at each frequency.   5. The method of estimating a mixing ratio in a mixed sound signal according to claim 3, wherein a quasi-Newton method is used for the optimization of the gain and the estimation of the fundamental frequency F0. The phoneme corresponding to the minimum distance gain change spectrum template obtained by the method for estimating the mixture ratio in the mixed sound signal according to any one of claims 1 to 6 is determined as the phoneme of the one frame. A phoneme recognition method, wherein a speech type is determined based on continuity of a plurality of one- frame phonemes. A mixing ratio estimation system for a mixed sound signal for estimating a mixing ratio between a target sound signal and a noise signal included in one frame signal obtained discretely from the mixed sound signal,
One or more target sound spectrum templates showing the relationship between the frequency components of one or more learning target sound signals and the probability distribution of the power spectrum, and the relationship between the frequency components of one or more learning noise signals and the probability distribution of the power spectrum. A spectrum template storage unit for storing one or more noise spectrum templates;
A stochastic spectrum template creating unit that creates one or more stochastic spectrum templates by combining the one or more target sound spectrum templates and the one or more noise spectrum templates;
An observation spectrum acquisition unit for acquiring an observation spectrum in the one frame from the mixed sound signal;
The distance between the observed spectrum and the plurality of gain-change spectrum templates obtained by changing the gain of the target sound spectrum template and the gain of the noise spectrum template respectively constituting the one or more stochastic spectrum templates is the smallest. A determination unit for determining the gain change spectrum template as a minimum distance gain change spectrum template;
An estimation unit that estimates the mixing ratio based on the gain of the minimum distance gain change spectrum template and the gain of the noise spectrum template;
A template generation unit that generates the one or more target sound spectrum templates and the one or more noise spectrum templates;
When the target sound signal is a voiced sound signal having a harmonic structure, the template generator is configured to convert the target sound spectrum template to a frequency component of a standard spectrum of the harmonic structure of the voiced sound signal. If the target sound signal is an unvoiced sound signal determined by a product of a wave drive sound source function and a voice envelope template, the voice envelope template is used as the target sound spectrum template,
The speech envelope template is an envelope that connects a plurality of peaks in the power included in a frequency spectrum waveform indicating a relationship between frequency components and power obtained by frequency analysis of a learning sound signal for a target voiced or unvoiced sound. A mixing ratio estimation system in a mixed sound signal, which is a template showing a distribution state of lines.   9. The mixing ratio estimation system in a mixed sound signal according to claim 8, wherein the template generation unit is configured to estimate both the target sound spectrum template and the noise spectrum template from a mixed signal for learning.