JP2008134606A - Automatic system and method for temporal alignment of music audio signal with lyric - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an automatic system for temporal alignment between a music audio signal and lyrics, capable of preventing accuracy for temporal alignment from being lowered due to the influence of non-vocal sections. <P>SOLUTION: An alignment means 17 includes a phone model 15 for singing voice that estimates phonemes corresponding to temporal-alignment features. The alignment means 17 receives temporal-alignment features output from a temporal-alignment feature extraction means 11, information on the vocal and non-vocal sections output from a vocal section estimation means 9, and a phoneme network SN, and performs an alignment operation on condition that no phoneme exists at least in non-vocal sections. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a system and method for automatically performing temporal association (alignment) between music acoustic signals and lyrics of music including singing voices and accompaniment sounds, and a program used in the system.

  Among digital music data (music acoustic signals) recorded on a recording medium such as a compact disc (CD), in particular, it is composed of human voice (for example, singing voice) and sound other than human voice (for example, accompaniment sound). 2. Description of the Related Art When reproducing digital music data, a technique for visually displaying the utterance content (that is, lyrics) of a person's voice while being synchronized with an accompanying sound in time is often used in a so-called karaoke apparatus.

  However, in the case of a conventional karaoke device, the accompaniment sound and the singer's singing voice are not accurately synchronized, and the lyrics of the music are only displayed sequentially on the screen at the tempo scheduled on the score. Absent. For this reason, the actual utterance timing and the display on the screen often deviate greatly. In addition, the synchronization of the accompaniment sound and the singing voice is performed through human hands and requires considerable human labor.

  By the way, as represented by so-called speech recognition technology, a technology for analyzing the content of a person's utterance is known. This technique recognizes the utterance content (lyrics) from digital music data of a singing voice without accompaniment sound (this is called “single singing”). Several studies have been reported on this. However, it is extremely difficult to directly apply a voice recognition technology that does not consider the influence of accompaniment sounds to digital music data distributed through a commercially available compact disc (CD) or an electric communication line such as the Internet.

  As a study on singing including an accompaniment sound, one that learns the length of time that each phoneme lasts and allocates a singing voice to a plurality of sections is known (see Non-Patent Document 1 below). This technology uses higher order information such as beat tracking and rust detection. However, this technique does not consider phonological features (for example, vowels and consonants) at all. Therefore, there is a problem that the accuracy rate is not so high. There is also a problem in that it cannot be applied to many types of music because the constraints on time signature and tempo are large.

  In Japanese Patent Laid-Open No. 2001-117582 (Patent Document 1), in a karaoke apparatus, a phoneme string of a singer (input person) and a phoneme string of a specific singer's singing voice are associated using an alignment means. Techniques to do this are disclosed. However, this publication does not disclose any technique for temporally associating music acoustic signals with lyrics.

  Japanese Patent Laid-Open No. 2001-125562 (Patent Document 2) predominates by estimating the pitch of the most dominant pitch including the singing voice at each time from the music sound signal of the mixed sound including the singing voice and the accompaniment sound. A technique for extracting an acoustic signal is disclosed. If this technique is used, the dominant sound signal which suppressed the accompaniment sound from the music sound signal can be extracted.

And a paper titled “Singer Name Identification Method Based on Accompaniment Suppression and Highly Reliable Frame Selection” by Hiromasa Fujiwara, Hiroshi Okuno, Masataka Goto and others [Information Processing Society Journal Vol.47 No.6 (announcement: 2006.6)] (Non-Patent Document 2) also discloses a technique for suppressing the accompaniment sound shown in Patent Document 2. This paper also discloses a technique for detecting a singing voice section and a non-singing voice section from a dominant sound signal by using two mixed Gaussian distributions (GMM) obtained by learning a singing voice and a non-singing voice. Furthermore, this paper discloses the use of LPC mel cepstrum as a feature quantity related to singing voice.
Ye Wang, et al .; LyricAlly: Automatic Synchronization of Acoustic Musical Signals and Textual Lyrics, Proceeding of the 12th ACM International Conference on Multimedia, October 10-15, 2004. A paper titled “Singer Name Identification Method Based on Accompaniment Suppression and Highly Reliable Frame Selection” by Hiromasa Fujiwara, Hiroshi Okuno, Masataka Goto et al. [Information Processing Society Journal Vol.47 No.6 (announcement: 2006.6) ]] JP 2001-117582 A JP 2001-125562 A

  In order to display lyrics in synchronism with accompaniment sound faithfully from music acoustic signals and lyrics information composed of human voice (for example, singing voice) and sound other than human voice (for example, accompaniment sound), time information is displayed. In other words, it is necessary to obtain lyrics with time information (in this specification, this is referred to as “time tag information”) indicating how many seconds after the start time of the performance, the lyrics will be spoken. .

  The lyrics themselves can be easily obtained as text data (text format digital information). Using this “lyric text data” and “musical sound signal (digital music data) with a singing voice that utters the lyrics”, it is practical to generate “lyrics with time tags”. There is an urgent need for technology that can be fully automated with accuracy (accuracy rate).

  A speech recognition technique is a useful technique for temporally corresponding music acoustic signals including accompaniment sounds and lyrics. However, the effect of the section where there is no singing voice (in this specification, it is called “non-voiced section” or “non-singing section”) may significantly reduce the accuracy (accuracy rate) of temporal correspondence. It became clear by research of inventors.

  An object of the present invention is to provide a system, method, and system for automatically performing temporal association between music acoustic signals and lyrics that can suppress a decrease in accuracy of temporal association due to the influence of a non-singing voice section. It is to provide a program to be used.

  The system for automatically associating a music acoustic signal with lyrics in accordance with the present invention includes a dominant sound signal extracting unit, a singing segment estimation feature amount extraction unit, a singing segment estimation unit, and a temporal correlation feature. A volume extraction unit, a phoneme network storage unit, and an alignment unit;

  The dominant sound signal extracting means extracts the dominant sound signal of the most dominant sound including the singing voice at each time (for example, every 10 msec) from the music sound signal of the music including the singing voice and the accompaniment sound. The dominant sound acoustic signal extraction technique is the same as the extraction technique used in Patent Document 2 and Non-Patent Document 2 described above.

  The feature extraction means for singing voice section estimation can be used to estimate a singing voice section including a singing voice and a non-singing voice section not including a singing voice from the dominant sound signal at each time (for example, every 10 msec). Extract features for singing voice segment estimation. The singing voice section estimation feature quantity that can be used here is a 13-dimensional feature quantity in a specific embodiment. More specifically, the LPC mel cepstrum and the differential coefficient ΔF0 of the fundamental frequency F0 can be used as the spectral feature quantity for identifying the singing voice state and the non-singing voice state.

  The singing voice section estimation means estimates a singing voice section and a non-singing voice section based on a plurality of singing voice section estimation feature quantities, and outputs information related to the singing voice section and the non-singing voice section.

  In addition, the temporal association feature amount extraction means is adapted to attach temporal correspondence between the lyrics of the singing voice and the dominant acoustic signal from the dominant acoustic signal at each time. To extract. In a specific embodiment, 25-dimensional feature quantities such as phoneme resonance characteristics are extracted as temporal association feature quantities.

  The results extracted by the singing voice segment estimation feature quantity extraction means and the temporal association feature quantity extraction means are provided with a storage unit in each means, and at least one song is stored in the storage unit. You may make it utilize in the case of this process.

  The phoneme network storage means stores a phoneme network composed of a plurality of phonemes and short pauses with respect to the lyrics of the music corresponding to the music acoustic signal. Such a phoneme network, for example, converts lyrics into phoneme strings, then converts phrase boundaries into multiple short poses, and converts word boundaries into a single short pose. In the case of lyrics, it is preferable to use a phoneme string consisting only of vowels and short pauses. For English lyrics, it is preferable to construct a phoneme network using phoneme strings consisting only of English phonemes and short pauses.

  The alignment means includes a singing voice acoustic model that estimates phonemes corresponding to the temporal association feature quantity based on the temporal association feature quantity. The alignment means executes an alignment operation for temporally associating the plurality of phonemes in the phoneme network with the priority sound signal. Specifically, the alignment means receives the temporal association feature amount output from the temporal association feature amount extraction means, the information about the singing voice section and the non-singing voice section, and the phoneme network as inputs. Using the acoustic model, alignment is executed under the condition that there is no phoneme at least in the non-singing voice section, and the time correspondence between the music acoustic signal and the lyrics is automatically performed.

  According to the present invention, a feature quantity suitable for use in estimating a singing voice section and a non-singing voice section (feature quantity for singing voice section estimation) and a feature quantity suitable for temporal association with lyrics (temporal) Since the feature amount for association) is extracted separately from the dominant sound signal, the estimation accuracy and temporal association accuracy of the singing voice section and the non-singing voice section can be increased. In particular, according to the present invention, the alignment means uses the singing voice acoustic model for estimating the phoneme corresponding to the temporal correspondence feature amount without using the speaking voice acoustic model. It is possible to estimate phonemes with higher accuracy considering characteristics of different singing voices. Furthermore, since the alignment means executes the alignment operation under the condition that there is no phoneme in at least the non-singing voice segment, a plurality of phonemes in the phoneme network and each time in a state where the influence of the non-singing voice segment is eliminated as much as possible. Can be temporally associated with the priority sound signal. Therefore, according to the present invention, the lyric data with the time tag synchronized with the music sound signal can be automatically obtained using the output of the alignment means.

  The configuration of the singing voice section estimation means is arbitrary as long as the estimation accuracy is high. For example, the singing voice section estimation means is provided with Gaussian distribution storage means for storing a plurality of mixed Gaussian distributions of singing voice and non-singing voice obtained by learning based on a plurality of learning songs in advance. And a singing voice section estimation means can be comprised so that a singing voice section and a non-singing voice section may be estimated based on the several singing voice area estimation feature-value obtained from the music acoustic signal, and several mixed Gaussian distribution. Based on the mixed Gaussian distribution obtained by prior learning in this way, estimating the singing voice section and the non-singing voice section can estimate the singing voice section and the non-singing voice section with high accuracy, and the alignment accuracy in the alignment means Can be high.

Such singing voice section estimation means includes log likelihood calculation means, log likelihood difference calculation means, histogram creation means, bias adjustment value determination means, estimation parameter determination means, weighting means, and maximum likelihood path. And a calculation means. The log likelihood calculating means calculates the singing voice log likelihood at each time and the non-log based on the feature value for estimating the singing voice interval at each time during the period from the beginning to the end of the music acoustic signal and the pre-stored mixed Gaussian distribution. The singing voice log likelihood is calculated. Then, the log likelihood difference calculating means calculates a log likelihood difference between the singing voice log likelihood and the non-singing voice log likelihood at each time. The histogram creation means creates a histogram related to a plurality of log likelihood differences obtained from the entire period of the priority sound signal in the pre-processing prior to estimation. The bias adjustment value determination means maximizes the interclass variance when the generated histogram is divided into a log likelihood difference class in the singing voice section and a log likelihood difference class in the non-singing voice section depending on the music. Is determined, and this threshold is determined as a music-dependent bias adjustment value. Further, the estimation parameter determination means adds a task-dependent value to the bias adjustment value to estimate the singing voice section in order to correct the bias adjustment value (in order to increase alignment accuracy or to adjust the singing voice section). The parameter for estimation used for is determined. Then, the weighting unit weights the singing voice log likelihood and the non-singing voice log likelihood at each time using the estimation parameters. Note that the singing voice log likelihood and the non-singing voice log likelihood used at this time may be those obtained at the time of the preprocessing, but it is of course possible to newly calculate them. When using the calculation result of the preprocessing, the log likelihood calculating means may have a storage function. The maximum likelihood path calculating means obtains a plurality of weighted singing voice log likelihoods and a plurality of weighted non-singing voice log likelihoods obtained from the entire period of the music acoustic signal, respectively, in the singing voice state (S _V ) of the hidden Markov model. Output probability and non-singing voice state (S _N ) output probability. The maximum likelihood path calculating means calculates the maximum likelihood path of the singing voice state and the non-singing voice state in the whole period of the music acoustic signal, and determines information on the singing voice section and the non-singing voice section in the whole period of the music acoustic signal from the maximum likelihood path. To do. The log likelihood difference determining means, the histogram creating means, the bias adjustment value determining means and the estimation parameter determining means are used for the music acoustic signal in the pre-processing before estimating the singing voice section in the system of the present invention. . When weighting by the weighting means using the estimation parameter obtained by the preprocessing is performed on the singing voice log likelihood and the non-singing log likelihood at each time, the singing voice section and the non-singing voice section in the maximum likelihood path calculation means later It is possible to appropriately adjust the adjustment of the boundary portion. At the time of the estimation operation, the singing voice log likelihood and the non-singing voice log likelihood calculated by the log likelihood calculation means from the singing voice section estimation feature quantity output at each time from the singing voice section estimation feature quantity extraction means, Direct weighting is performed to calculate the maximum likelihood path. When bias adjustment values (thresholds) of the singing voice log likelihood and the non-singing voice log likelihood are determined by using the histogram of the log likelihood difference by such preprocessing, the bias adjustment value suitable for the music acoustic signal is determined. be able to. This bias adjustment value (threshold value) determines the boundary between the singing voice state and the non-singing voice state. When weighting is performed using the estimation parameter determined by the bias adjustment value, the singing voice state and the non-singing voice state are matched to the tendency of the singing voice section estimation feature amount that appears due to the difference in the acoustic characteristics of the music acoustic signal for each song. The singing voice log likelihood and the non-singing voice log likelihood can be adjusted centering on the boundary portion between the singing voice section and the non-singing voice log likelihood, and the boundary between the singing voice section and the non-singing voice section can be appropriately set according to individual music.

The maximum likelihood route calculation means can calculate the maximum likelihood route as follows. That output probability logp singing condition _{(s V)} approximating the | | _{(s N} x) by the following equation (x _{s V)} and non-output probability singing condition _{(s N)} logp.

In the above equation, N _GMM (x; θ _V ) represents a probability density function of a mixed gaussian distribution (GMM) of singing voice, and N _GMM (x; θ _N ) represents a probability density function of a mixed Gaussian distribution (GMM) of non-singing voice. Represents. Θ _V and θ _N are parameters determined by learning in advance based on a plurality of learning songs, and η is an estimation parameter. The maximum likelihood path may be calculated using the following formula.

In the above formula, p (x | _{s t)} represents the output probability of the state _{S t.} P (s _{t + 1} | s _t ) represents the transition probability from the state s _t to the state s _{t + 1} .

  If the maximum likelihood path is calculated using the above formula, more accurate information about the singing voice section and the non-singing voice section in the entire period of the music acoustic signal can be obtained.

  As the alignment means, one configured to perform an alignment operation using a Viterbi alignment can be used. Here, “Viterbiar alignment” is known in the technical field of speech recognition, and is an optimal solution search method using a Viterbi algorithm that searches for a maximum likelihood path between an acoustic signal and a grammar (phoneme sequence for alignment). One. In the execution of the viterbi alignment, a condition that at least the non-singing voice section is in a short pause is defined as “a condition that no phoneme is present in the non-singing voice section”. In the short pause, the alignment operation is executed with the likelihood of other phonemes set to zero. In this way, since the likelihood of other phonemes becomes zero in the short pause section, the singing voice section information can be used, and high-precision alignment can be performed.

  Also, as the acoustic model for singing voice to be used, the acoustic model obtained by re-estimating (learning) the parameters of the acoustic model for speaking voice so as to recognize the phoneme of the singing voice in the music including the singing voice and the accompaniment sound is used. be able to. As the acoustic model for singing voice, it is ideal to use a model learned from a large amount of singing voice data in order to align the utterance contents (lyrics) of the singing voice. However, no such database has been constructed at this stage. Therefore, if the acoustic model obtained by re-estimating (learning) the parameters of the acoustic model for speaking voice so that it can recognize the phoneme of the singing voice in the music including the singing voice and the accompaniment sound, the acoustic model for speaking voice can be obtained. It becomes possible to recognize the phoneme of the singing voice with higher accuracy than in the case of using it.

  Note that, as such an acoustic model for singing voice, the parameters of the acoustic model for speaking voice are adapted using an adaptive music acoustic signal for single singing including only the singing voice and an adaptive phoneme label for the adaptive musical acoustic signal. It is possible to use an acoustic model for single singing obtained by re-estimation so that the phoneme of the singing voice can be recognized from the music audio signal for use. This acoustic model is suitable when there is no accompaniment sound or the accompaniment sound is smaller than the singing voice.

  The singing voice acoustic model includes a dominant sound acoustic signal of the most dominant sound including the singing voice extracted from the adaptive music acoustic signal including the accompaniment sound in addition to the singing voice, and an adaptive phoneme label for the dominant sound acoustic signal. It is possible to use an acoustic model for separated singing voice obtained by re-estimating the parameters of the acoustic model for single singing described above so that the phoneme from the dominant sound acoustic signal can be recognized. Such an acoustic model for a separated singing voice is suitable when the accompaniment sound is large like the singing voice.

  Furthermore, as the acoustic model for singing voice, the above-mentioned separated singing voice is used by using a plurality of temporal correlation feature quantities stored in the temporal correlation feature quantity storage means and the phoneme network stored in the phoneme network. The acoustic model for the specific singer obtained by estimating the parameters of the acoustic model for use so that the phoneme of the specific singer who sings the music of the music acoustic signal input to the acoustic signal extraction means can be recognized can be used. Since the acoustic model for the specific singer is an acoustic model specifying the singer, the alignment accuracy can be maximized.

  In a music sound signal reproducing apparatus that reproduces a music sound signal while displaying lyrics that are temporally associated with the music sound signal on the display screen, the music sound signal is temporally supported using the system of the present invention. When the attached lyrics are displayed on the display screen, the reproduced music and the lyrics displayed on the screen can be synchronized and displayed on the display screen.

  In the method of automatically performing temporal association between the music acoustic signal and the lyrics according to the present invention, temporal association is performed as follows. First, the dominant sound signal extraction means extracts the dominant sound signal of the most dominant sound including the singing voice at each time from the music sound signal of the music including the singing voice and the accompaniment sound (dominant sound signal extracting step). Next, a singing voice segment estimation feature quantity that can be used to estimate a singing voice section that includes a singing voice and a non-singing voice section that does not contain a singing voice from the dominant sound signal at each time. Extraction means extracts (singing voice segment estimation feature extraction step). Then, the singing voice section estimation means estimates the singing voice section and the non-singing voice section based on the plurality of singing voice section estimation feature quantities, and outputs information on the singing voice section and the non-singing voice section (singing voice section estimation step). Also, the temporal correlation feature quantity extraction means extracts a temporal correlation feature quantity suitable for providing a temporal correlation between the singing voice lyrics and the music acoustic signal from the dominant sound acoustic signal at each time. (Temporal association feature extraction step). Furthermore, the phoneme network storage unit stores a phoneme network in which a plurality of phonemes of lyrics corresponding to a music acoustic signal are connected so that a time interval between two phonemes adjacent to the plurality of phonemes can be adjusted ( Memory step). And a singing voice acoustic model for estimating a phoneme corresponding to the temporal association feature amount based on the temporal association feature amount, and temporally combining the plurality of phonemes and the priority sound acoustic signal in the phoneme network. The alignment means executes the alignment operation to be associated (alignment step). In this alignment step, the alignment means inputs the temporal association feature obtained in the temporal association feature extraction step, the information about the singing voice segment and the non-singing voice segment, and the phoneme network, and inputs the singing voice sound. Using the model, the alignment operation is executed under the condition that there is no phoneme at least in the non-singing voice section.

  Further, the present invention provides a computer that uses the above-described dominant acoustic signal extraction means and singing voice interval estimation in the case of using a computer for temporally associating music acoustic signals and lyrics of music including singing voices and accompaniment sounds. It can be specified as a program that functions as feature quantity extraction means, singing voice section estimation means, temporal association feature quantity extraction means, phoneme network storage means, and alignment means. Needless to say, this program may be recorded on a computer-readable recording medium.

  In the music acoustic signal reproducing apparatus for reproducing the music digital data while displaying the lyrics on the display screen, the music acoustic signal and lyrics temporal association program according to the present invention can be executed. In this case, after generating lyrics with time information in advance, the lyrics are displayed on the display screen. Then, with the lyrics displayed on the display screen, the display portion of the displayed lyrics is selected by the pointer. In this case, based on time information corresponding to a part of the selected lyrics, an acoustic music signal may be reproduced from that part. In addition, lyrics with time information generated in advance by the system of the present invention may be stored in a storage means such as a hard disk provided in the music sound signal reproducing apparatus, or stored in a server on the network. Also good. Then, in synchronism with the reproduction of the music digital data by the music acoustic signal reproduction device, the lyrics accompanied by the time information stored in the storage means or acquired from the server on the network may be displayed on the display screen.

  Hereinafter, an example of an embodiment of a system and method for automatically associating a time-synchronization of a music sound signal and lyrics according to the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram showing a configuration of function realizing means realized in a computer when an embodiment of a system 1 that automatically associates a music acoustic signal and lyrics with time is realized using a computer. . FIG. 2 is a flowchart showing steps when the embodiment of FIG. 1 is executed by executing a program on a computer. This system 1 includes a music sound control signal storage means 3, a dominant sound signal extraction means 5, a singing voice section estimation feature quantity extraction means 7, a singing voice section estimation means 9, and a temporal correlation feature quantity extraction means. 11, a phoneme network storage means 13, and an alignment means 17 including a singing voice acoustic model 15.

  The present invention performs the following three steps as a basic approach for effectively achieving the above technical problem.

Step 1: Accompaniment sound suppression Step 2: Singing voice segment detection Step 3: Alignment (temporal association)
In order to execute Step 1, the music acoustic signal storage means 3 stores music acoustic signals of a plurality of music pieces including the target singing voice and accompaniment sound. According to the flowchart shown in FIG. 3, the dominant sound signal extraction means 5 obtains the most dominant sound including the singing voice at each time (specifically, every 10 msec) from the music acoustic signal S1 of the music including the singing voice and the accompaniment sound. The dominant sound signal S2 is extracted. In the present embodiment, the dominant sound signal can be regarded as a signal in which the accompaniment sound is suppressed. The extraction technique of the dominant sound signal is the same as the extraction technique disclosed in Japanese Patent Laid-Open No. 2001-125562 (Patent Document 2) and Non-Patent Document 2. The signal waveform of the music acoustic signal S1 of the music including the singing voice and the accompaniment sound is, for example, a signal waveform as shown in FIG. 4A, and the accompaniment sound output by the dominant sound signal extraction means 5 is suppressed. The signal waveform of the acoustic signal S2 is a signal waveform as shown in FIG. Hereinafter, a method for extracting the dominant sound signal will be described.

  First, from a music acoustic signal of a tune (mixed sound) including a singing voice and an accompaniment sound, a singing voice section estimation feature amount and a temporal association feature amount [a feature amount representing a phonological feature of a melody (singing voice), etc.] In order to extract, it is necessary to obtain a dominant sound signal in which the influence of the accompaniment sound is reduced from the music sound signal. Therefore, the dominant sound signal extraction means 5 executes the following three steps shown in FIG.

  ST1: Estimate the fundamental frequency F0 of the melody (singing voice).

  ST2: Extract the harmonic structure of the melody (singing voice) based on the estimated fundamental frequency.

  ST3: Re-synthesize the extracted harmonic structure into the dominant sound signal.

  Note that the dominant sound acoustic signal may include an acoustic signal (accompaniment sound or silence) other than a singing voice in an interval such as an interlude. Therefore, in this embodiment, it is expressed as “reduction” of accompaniment sound, not “removal” of accompaniment sound. Hereinafter, steps ST1 to ST3 will be described.

(ST1: F0 estimation process)
Various methods are known for estimating the fundamental frequency of a melody (singing voice). For example, it is possible to use a method of estimating the fundamental frequency by a pitch estimation method (PreFEst) that does not assume the number of sound sources (for example, Masataka Goto, "Melody and bass pitch estimation for music acoustic signals", (See IEICE Transactions D-II, Vol.J84-D-II, No.1, pp.12-22, January 2001.) Here, PreFEst is known as a method for estimating the melody and bass fundamental frequency F0. This is a method of estimating the fundamental frequency F0 of the dominant sound having the most dominant harmonic structure (that is, the loudest sound) at each time in the limited frequency band. In this pitch estimation method (PreFEst), a probability distribution representing the shape of the harmonic structure is prepared for every pitch (fundamental frequency). Then, the input frequency component is modeled as a mixture distribution (weighted mixture = weighted sum).

The melody (singing voice) often has the most dominant harmonic structure at each time in the mid-high frequency band. Therefore, the fundamental frequency F0 of the melody (singing voice) can be estimated by appropriately limiting the frequency band. The outline of PreFEst will be described below. Note that x used in the following description is a frequency on the logarithmic frequency axis expressed in units of cent, and (t) represents time. In addition, cent is originally a scale representing a pitch difference (pitch), but in this specification, 440 × 2 { ^{(3/12) -5} } [Hz] is used as a reference, and the absolute value is as follows: It is used as a unit that represents a typical pitch.

For the power spectrum Ψ _p ^(t) (x), a band pass filter (Band Pass Filter) designed to pass most of the frequency components of the melody is used. For example, it is preferable to use a filter that passes a component of 4800 cent or more. The frequency component after passing through the filter is
BPF (x) · Ψ _p ^(t) (x)
It is expressed. Where BPF (x) is the frequency response of the filter. In order to enable the subsequent stochastic processing, the frequency component after passing through the filter is expressed as a probability density function (PDF) as follows.

Thereafter, a probability model in which the probability density function PDF of frequency components is a weighted sum of sound models (probability distributions) corresponding to all possible fundamental frequencies F0:

I think that it was generated from.

Here, p (x | F) is a sound model for each F0, Fh represents an upper limit value of F0 that can be taken, and Fl represents a lower limit value of F0 that can be taken. W ^(t) (F) is the weight of the sound model,

Meet. That is, the sound model is a probability distribution that represents a typical harmonic structure. Then, w ^(t) (F) is estimated using an EM (Expectation Maximization) algorithm, and the estimated w ^(t) (F) is interpreted as a probability density function (PDF) of the fundamental frequency F0. Finally, by tracking the trajectory of the dominant peak in w ^(t) (F) using a multi-agent model, an F0 sequence (F0 Estimation) of a melody (singing voice) is obtained. FIG. 4 shows the F0 sequence (F0 Estimation) acquired in this way.

(ST2: Harmonic structure extraction)
Based on the fundamental frequency F0 thus estimated, the power of each harmonic component of the harmonic structure of the melody is extracted. In extracting each frequency component, an error of about rcent before and after is allowed, and the peak with the largest power in this range is extracted. The power A _l and frequency F _l of the l-order overtone (l = 1,..., L) are expressed as follows.

  Here, S (F) represents a spectrum, and a symbol with a bar (−) above F represents a fundamental frequency F0 estimated by PreFEst. In the experiments by the inventors of the present application, the harmonic structure was extracted using 20 as the value of r, and the effect was confirmed as described later. FIG. 4C shows the harmonic structure of each extracted frequency component.

(ST3: Resynthesis)
By re-synthesizing the extracted harmonic structure based on the sine wave superposition model, the dominant sound signal of the most dominant sound including the singing voice is obtained at each time. Here, the frequency of the l-th overtone at time t is F ₁ ^(t) , and the amplitude is A ₁ ^(t) . The phase change is approximated by a quadratic function so that the frequency between frames (between time t and time t + 1) changes linearly. The change in amplitude between frames is approximated by a linear function. The re-synthesized dominant sound signal s (k) is expressed as follows. In the following expression, θ _l (k) is the phase of the l-order overtone at time k, and s _l (k) is the waveform of the l-order overtone at time k.

  Here, k represents time (unit: second), and k = 0 at time t. K represents a time difference between (t) and (t + 1), that is, a frame shift in units of seconds.

θ _{l, 0} ^(t) represents an initial value of the phase, and θ _{l, 0} ^(t) = 0 in the first frame of the input signal. In subsequent frames, θ _{l, 0} ^(t) uses the frequency F _l ^(t−1) of the l-order harmonic of the previous frame and the initial phase θ _{l, 0} ^(t−1).

Given in.

  Returning to FIG. 1, the singing voice section estimation feature quantity extraction means 7 includes a singing voice section including a singing voice from a dominant sound signal at each time (specifically, every 10 msec) and a non-singing voice. A feature amount for singing voice section estimation that can be used for estimating a singing voice section is extracted. In this embodiment, a 12-dimensional LPC mel cepstrum (LPMCC) and a one-dimensional fundamental frequency F0 differential coefficient (ΔF0) are used as singing voice section estimation feature quantities usable here. In this embodiment, the singing voice section estimation feature quantity extraction means 7 extracts the following two types of feature quantities as the singing voice section estimation feature quantity (spectrum feature quantity) in order to identify the singing voice and the non-singing voice. .

・ LPC Mel Cepstrum (LPMCC)
A 12-dimensional LPC mel cepstrum (LPMCC) is used as the first type of spectral feature. LPMCC is the mel cepstrum coefficient calculated from the LPC spectrum. According to the experiments by the present inventors, it has been confirmed that this feature value expresses the characteristics of the singing voice better than the mel frequency cepstrum coefficient (MFCC). In this embodiment, the LPC mel cepstrum LPMCC is extracted by calculating the mel frequency cepstrum coefficient MFCC from the LPC spectrum.

・ ΔF0 _s
A differential coefficient (ΔF0) of the fundamental frequency F0 is used as the second type of spectral feature quantity. This is useful for expressing the dynamic nature of the singing voice. Since the singing voice has more time variation due to vibrato, etc., compared to other songs, the differential coefficient ΔF0 representing the inclination of the trajectory of the fundamental frequency F0 is considered suitable for discriminating between singing voice and non-singing voice. is there.

In calculating ΔF0, a regression coefficient between 5 frames was used as in the following equation.

  Here, f [t] is a frequency (unit: cent) at time t.

  And in order to perform the above-mentioned step 2, the singing voice section estimation means 9 estimates the singing voice section and the non-singing voice section based on the plurality of singing voice section estimation feature values extracted at each time, and determines the singing voice section and non-singing voice section. Outputs information about the singing voice section. The singing voice section estimation means 9 of the present embodiment has the configuration shown in FIG. In the singing voice section estimation means 9 shown in FIG. 5, as shown in FIG. 2, a Gaussian distribution memory for storing a plurality of mixed Gaussian distributions of singing voices and non-singing voices obtained in advance based on a plurality of learning songs 8. Means 91 are provided. The singing voice section estimation means 9 estimates a singing voice section and a non-singing voice section based on a plurality of singing voice section estimation features and a plurality of mixed Gaussian distributions over the entire period of one music acoustic signal S1. Is output. Therefore, the singing voice section estimating means 9 further includes a log likelihood calculating means 92, a log likelihood difference calculating means 93, a histogram creating means 94, a bias adjustment value determining means 95, an estimation parameter determining means 96, and a weighting. Means 97 and maximum likelihood path calculation means 98 are provided. The log likelihood difference calculating means 93, the histogram creating means 94, the bias adjustment value determining means 95, and the estimation parameter determining means 96 are used in the pre-processing before estimating the singing voice section. FIG. 6 shows a flowchart in the case where the singing voice section estimating means 9 shown in FIG. 5 is realized by a computer by a program. FIG. 7 shows a flowchart when the detection of the singing voice section is realized by a program. FIG. 7 corresponds to details of step ST11 and step ST16 of FIG.

  The log likelihood calculation means 92 is a singing voice section estimation feature quantity (step ST11) extracted by the singing voice section estimation feature quantity extraction means 7 at each time during the period from the beginning to the end of the music acoustic signal S1. Based on the mixed Gaussian distribution previously stored in the Gaussian distribution storage unit 91 by preprocessing, the singing voice log likelihood and the non-singing voice log likelihood at each time are calculated.

Then, the log likelihood difference calculating means 93 calculates the log likelihood difference between the singing voice log likelihood and the non-singing voice log likelihood at each time (step ST12). This calculation is performed on the singing voice segment estimation feature quantity (feature vector sequence) extracted from the input music acoustic signal, as shown in the following equation, the log likelihood difference l between the singing voice log likelihood and the non-singing voice log likelihood. (X) is calculated.

  The function in front of the above formula shows the singing voice log likelihood, and the latter function shows the non-singing voice function likelihood. The histogram creating means 94 creates a histogram relating to a plurality of log likelihood differences obtained from the priority sound signal extracted from the whole period of the music sound signal (step ST13). FIG. 6 illustrates an example of a histogram created by the histogram creation means 94.

Then, the bias adjustment value determining unit 95 divides the variance between classes when the generated histogram is divided into two, depending on the music, into a log likelihood difference class in the singing voice section and a log likelihood difference class in the non-singing voice section. A threshold value that maximizes the threshold value is determined, and this threshold value is determined as a music-dependent bias adjustment value η _dyn. (Step ST14). FIG. 6 illustrates this threshold value. The estimation parameter determination means 96, in order to correct the bias adjustment value eta _dyn. (For adjustment to extend the order or vocal sections to increase the accuracy of the alignment), bias adjustment value eta _dyn. In a task-dependent value eta _fixed An estimation parameter η (= η _dyn. + Η _fixed ) used when estimating the singing voice interval by addition is determined (step ST15). Since the likelihood of the mixed Gaussian distribution (GMM) is biased by music, it is difficult to determine an estimation parameter η appropriate for all music. Therefore, in this embodiment, the estimation parameter η is divided into the bias adjustment value η _dyn. And the task-dependent value η _fixed . The task dependence value η _fixed is manually set in advance in consideration of the type of music. On the other hand, the bias adjustment value η _{dyn. May be} automatically set for each piece of music through the above-described steps or using a known threshold automatic setting method, or representative learning music depending on the type of music piece. You may set beforehand based on an acoustic signal.

The weighting means 97 weights the singing voice log likelihood and the non-singing voice log likelihood at each time using the estimation parameter η (step ST16A in FIG. 7). In this example, the singing voice log likelihood and the non-singing voice log likelihood used here are those calculated in the preprocessing. That is, the weighting means 97 approximates the output probabilities of the singing voice log likelihood and the non-singing voice log likelihood as shown in the following equation.

Here, N _GMM (x; θ) represents a probability density function of a mixed Gaussian distribution (GMM). Η is an estimation parameter for adjusting the relationship between the correct answer rate and the rejection rate. The parameter θ _v of the singing voice GMM and the parameter θ _N of the non-singing voice GMM are learned using the singing voice section and the non-singing voice section of the learning data, respectively. In the experiments conducted by the inventors of the present application, the effect was confirmed by using a GMM having a mixing number of 64 as described later.

The maximum likelihood path calculation means 98 obtains a plurality of weighted singing voice log likelihoods and a plurality of weighted non-singing voice log likelihoods obtained from the entire period of the music acoustic signal, respectively, as the singing state (S _V of the hidden Markov model). ) And non-singing voice state (S _N ) output probability (step ST16B in the figure). The maximum likelihood path calculation means 98 calculates the maximum likelihood path of the singing voice state and the non-singing voice state in the entire period of the music acoustic signal (step ST16C in FIG. 7), and the singing voice section in the entire period of the music acoustic signal from the maximum likelihood path. And information about non-singing voice segments. That is, for the detection of the singing voice, as shown in FIG. 8, a hidden Markov model (HMM) that goes back and forth between the singing voice state (S _v ) and the non-singing voice state (S _N ) is used. The singing voice state literally represents a “state where a singing voice exists”, and the “non-singing voice state” represents a state where no singing voice exists. The maximum likelihood path calculation means 98 is a maximum likelihood path in a singing voice / non-singing voice state with respect to the feature vector sequence extracted from the input acoustic signal as shown in the following equation.

Search for.

In the above equation, p (x | s _t ) represents the output probability of the state, and p (s _{t + 1} | s _t ) represents the transition probability from the state s _{t + 1} to the state _st .

In this singing voice section estimation means 9, during a normal estimation operation other than pre-processing, a log likelihood calculation means 92 is obtained from the singing voice section estimation feature quantity output at each time from the singing voice section estimation feature quantity extraction means 7. The maximum likelihood path is calculated by directly weighting the singing voice log likelihood and the non-singing voice log likelihood calculated by. When the bias adjustment value η _dyn (threshold value) of the singing voice log likelihood and the non-singing voice log likelihood is determined by using the histogram of the log likelihood difference by such preprocessing, the bias adjustment value η suitable for the music acoustic signal is determined. _dyn can be determined. Then, when weighting is performed using the estimation parameter η determined by the bias adjustment value η _dyn , the singing voice state and the singing voice state and The singing voice log likelihood and the non-singing voice log likelihood can be adjusted around the boundary with the non-singing voice state, and the boundary between the singing voice section and the non-singing voice section can be adjusted appropriately according to the music.

  Returning to FIG. 1, the temporal correspondence feature quantity extraction means 11 is suitable for attaching temporal correspondence between the lyrics of the singing voice and the dominant sound acoustic signal from the dominant sound acoustic signal at each time. A feature amount for association is extracted. In a specific embodiment, 25-dimensional feature quantities such as phoneme resonance characteristics are extracted as temporal association feature quantities. This processing corresponds to preprocessing necessary for the next alignment processing. Details will be described later with reference to the analysis conditions of the viterbi alignment shown in FIG. 9, but the feature quantities extracted in this embodiment are 12 dimensions MFCC, 12 dimensions ΔMFCC, and 25 dimensions of Δ power.

  The phoneme network storage unit 13 stores a phoneme network SN composed of a plurality of phonemes with respect to the lyrics of the music corresponding to the music acoustic signal. Such a phoneme network SN converts, for example, a Japanese lyrics into a phoneme string, and then converts a phrase boundary into a plurality of short pauses and a word boundary into one short pause. It is preferable to use a phoneme string consisting of only vowels and short pauses. Based on the text data of the given lyrics, a grammar used for the alignment (this is defined as “phoneme string for alignment”) is created.

  The phoneme string for alignment for Japanese lyrics consists of a short pause (sp), that is, only blanks, vowels and consonants. This is because unvoiced consonants do not have a harmonic structure and cannot be extracted by the accompaniment sound suppression method, and voiced consonants have short utterance lengths, making it difficult to stably estimate the fundamental frequency F0. Specifically, the lyrics are first converted into phoneme strings (substantially equivalent to the work of converting the words read aloud into Roman characters), and then the following two rules (Japanese grammar) ) To convert to a phoneme string for alignment.

  Rule 1: The boundaries of sentences and phrases in the lyrics are converted into multiple short pauses (sp).

  Rule 2: Convert word boundaries into one short pause.

  FIG. 10 shows an example of conversion from Japanese lyrics into a phoneme string (phoneme network) for alignment. First, text data A representing a phrase of original lyrics is converted into a phoneme string (Sequence of the phonemes) B. By applying the above “grammar” to the phoneme sequence B, the phoneme sequence B is converted into an “phoneme sequence for alignment” C composed only of vowels, consonants, and a short pause (sp).

  In this example, the lyrics A in Japanese, “When you stop and also look back,” A is converted to a phoneme sequence B, “tachidomaru toki mata futo furikaeru”, and then a phoneme and short pause (sp) containing vowels and consonants. A state in which the phoneme string C is converted into an alignment phoneme string C is shown. The phoneme string C for alignment is a phoneme network SN.

  Returning to FIG. 1, in order to execute the above-described step 3, the aligning means 17 uses the singing voice for estimating the phoneme corresponding to the temporal association feature amount based on the temporal association feature amount. An acoustic model 15 is provided. And the alignment means 17 performs the alignment operation | movement which matches a some phoneme in a phoneme network, and a priority sound acoustic signal temporally. Specifically, the alignment unit 17 includes the temporal association feature amount from the temporal association feature amount extraction unit 11, information about the singing voice segment and the non-singing voice segment from the singing voice segment estimation unit 9, and a phoneme network. Using the phoneme network from the storage means 13 as an input, using the acoustic model for singing voice 15, the alignment is executed under the condition that there is no phoneme at least in the non-singing voice section, and the time of the music acoustic signal and the lyrics Automatic association.

  The alignment means 17 of the present embodiment is configured to execute an alignment operation using viterbi alignment. Here, “Viterbi alignment” is known in the technical field of speech recognition, and is optimal using a Viterbi algorithm that searches for the maximum likelihood path between an acoustic signal and a grammar (phoneme sequence for alignment or phoneme network). This is one of the solution search methods. In the execution of the viterbi alignment, a condition that at least the non-singing voice section is set to a short pause (sp) is determined as a condition that no phoneme is present in the non-singing voice section. In the short pause (sp), the alignment operation is executed with the likelihood of other phonemes set to zero. In this way, in the short pause (sp) section, the likelihood of other phonemes becomes zero, so the singing voice section information can be used, and high-precision alignment can be performed.

  FIG. 11 is a flowchart showing an algorithm of a program when the alignment means 17 is realized by a computer using a Viterbi alignment called “frame synchronization Viterbi search”. In the following description of the alignment operation, a case where the lyrics are in Japanese will be described as an example. T = 1 in step ST101 is a frame in which the first temporal association feature amount (hereinafter simply referred to as feature amount in the description of FIG. 11) is input. In step ST102, an empty hypothesis with a score of 0 is created. Here, “hypothesis” means “line of phonemes” up to the present time. Therefore, creating an empty hypothesis means that there is no phoneme.

  Next, in step ST103, as loop 1, all hypotheses currently held are processed. Loop 1 is a loop that performs score calculation processing for each hypothesis possessed when the processing in the previous frame is completed. For example, it is assumed that a temporal correspondence with a phoneme network “ai-sp-ue ...” is attached. In this case, there are various hypotheses (arrangement of phonemes) when coming to the sixth frame (sixth phoneme), such as a hypothesis “aaaaaa”, a hypothesis “aaaiii”, and a hypothesis “aaiispu”. Can be hypothesized. During the search, the calculation process is executed while simultaneously holding the plurality of hypotheses. These hypotheses all have their own scores. Here, when the score is assumed to be up to 6 frames, the possibility (log likelihood) that each of the feature quantities from 1 to 6 frames is, for example, a sequence of phonemes “aaaiii” is represented by the feature quantity and the acoustic model. Is calculated by comparing. For example, when processing for the sixth frame (t = 6) is completed and processing for the seventh frame is started, calculation processing is performed for all currently held hypotheses. This process is the process of loop 1.

  In step ST104, the hypothesis is "one frame developed" based on the phoneme network. Here, “one frame expansion” means that the length of the hypothesis is extended by one frame. In the case of expansion, there is a possibility that a new phoneme is followed by a new phoneme by taking into consideration up to the frame of the next time, so that a plurality of new hypotheses can be formed. The phoneme network is referenced to find the next phoneme that may follow. For example, with regard to the hypothesis “aaaiii”, referring to the phoneme network, in the next frame, “i” continues like “aaaiii”, and “aaaiiisp” moves to a short pause sp. New hypotheses can be considered. In this case, when one hypothesis is “expanded into one frame”, two new hypotheses are taken into consideration until the next frame. In step ST105, as loop 2, scores are calculated for all new hypotheses generated by developing one frame for all hypotheses. The score calculation is the same as the score calculation in loop 1. The loop 2 is a loop that performs score calculation processing for each newly developed hypothesis, since several hypotheses are newly developed from each held hypothesis.

  Next, in step ST106, the singing voice section information from the singing voice section estimation means 9 is input, and it is determined whether the t-th frame is a singing voice section or whether the phoneme is in a short pause (sp). For example, it is assumed that there is singing voice section information that the seventh frame is a non-singing voice section. In this case, when the hypothesis is developed in the seventh frame, even though there is a hypothesis “aaaiiisp”, there is no hypothesis “aaaiii”. Such impossible hypotheses are rejected in step ST107. When there is singing voice section information in this way, an impossible hypothesis can be rejected through steps ST106 and ST107, so that alignment becomes easy. If YES is determined in step ST106, the process proceeds to step ST108.

  In step ST108, the acoustic score of the t-th feature amount is calculated using the input feature amount and the acoustic model, and is added to the hypothesis score. That is, the logarithmic likelihood (score) is calculated by comparing the t-th feature quantity with the acoustic model, and the calculated score is added to the hypothesis score. Eventually, the score is calculated by comparing the feature quantity with the acoustic model and calculating how much the feature quantity is similar to information about a plurality of phonemes in the acoustic model. Since the score is calculated logarithmically, if it is not similar at all, the value is −∞. In step ST108, scores are calculated for all hypotheses. When the calculation in step ST108 ends, the process proceeds to step ST109, and the hypothesis and score are held. In step ST110, loop 2 corresponding to step ST105 ends. In step ST111, loop 1 corresponding to step ST103 ends. Thereafter, in step ST112, the current processing target time is increased by 1 (t + 1), and the process proceeds to the next frame. Then, in step ST113, it is determined whether or not the frame is the end of a plurality of feature amounts input. Until all the feature values are input, each step from step ST103 to step ST112 is repeatedly executed. When the process is completed for all feature amounts, the process proceeds to step ST114. At this point, the comparison between the feature quantity and the acoustic model has reached the end of the phoneme network. Then, the hypothesis (phoneme arrangement) having the maximum total score is selected as the final determined hypothesis from a plurality of hypotheses reaching the end of the phoneme network. This finally determined hypothesis, that is, the phoneme sequence, is determined based on the feature quantity corresponding to the time. That is, the final determined phoneme arrangement is a sequence of phonemes synchronized with the music acoustic signal. Therefore, the lyrics data displayed on the basis of the sequence of the final determined phonemes becomes lyrics with a time tag (with time information for synchronizing with the music acoustic signal).

  FIG. 12A shows a phoneme network for the signal waveform S ′ of the dominant sound signal extracted from the music sound signal at the time using the Viterbi alignment (the sound waveform of the sound signal in which the accompaniment sound is suppressed). It shows how (grammar) is associated with time. After the alignment is completed, “lyric data with time tag” including the time information is finally obtained by returning to the lyrics from the phoneme string (grammar) for alignment accompanied by the time information. In FIG. 12A, only vowels are shown for simplicity of illustration.

  FIG. 12B shows a state in which the temporal association between the music acoustic signal S of the mixed sound including the accompaniment sound and the lyrics is completed by returning to the lyrics from the phoneme string (grammar) after the alignment is completed. . PA to PD are lyric phrases.

  Next, the singing voice acoustic model 15 used in the alignment means 17 will be described. As the singing voice acoustic model 15 to be used, it is ideal to use an acoustic model learned from a large amount of singing voice data in order to align the utterance contents (lyrics) of the singing voice. However, no such database has been constructed at this stage. Therefore, in this embodiment, the acoustic model obtained by re-estimating (learning) the parameters of the acoustic model for speaking voice so as to recognize the phoneme of the singing voice in the music including the singing voice and the accompaniment sound is used.

  There are three stages of methods (adaptation) for creating an acoustic model for singing voice based on an acoustic model for speaking voice. As a prior work, a step of preparing an “acoustic model for speaking voice” is necessary, but this point is well known and will be omitted.

  (1) Adapt an acoustic model for speaking voice to a single singing voice.

  (2) An acoustic model for single singing is adapted to the separated singing voice extracted by the accompaniment sound suppression method.

  (3) An acoustic model for separated singing voice is adapted to a specific music (specific singer) in the input music.

  These steps (1) to (3) all correspond to the “learning” process in FIG. 2 and are performed before the execution.

In the (1) stage adaptation, as shown in FIG. 2, the acoustic model 101 for speaking voice is adapted to a singing voice 103 with only a phoneme label 102 (teacher information) and a singing voice without accompaniment sounds, that is, a single singing voice 103. The acoustic model 104 is generated. In the adaptation of (2), the acoustic model 104 for single singing is adapted to the singing voice data 105 and the phoneme label 102 (teacher information) composed of the dominant sound acoustic signals extracted by the accompaniment sound suppression method, and is used for the separated singing voice. An acoustic model 106 is generated. In the adaptation of (3), the acoustic model 106 for the separated singing voice is adapted to the phoneme label (phoneme network) and the feature amount of the specific music of the input music to generate the acoustic model 107 for the specific singer. In the example of FIG. 2, a specific singer acoustic model 107 is used as the singing voice acoustic model 15 of FIG.

  Note that it is not always necessary to implement (1) to (3). For example, when only (1) is implemented (this is referred to as “one-stage adaptation”), and (1) and (2) are implemented. (This is called “two-stage adaptation”) and when all of (1) to (3) are implemented (this is called “three-stage adaptation”), etc. Acoustic model adaptation can be implemented.

  Here, the teacher information refers to time information (phoneme start time and end time) for each phoneme. Therefore, when adapting the acoustic model for speaking voice using the teacher information such as the single singing data 103 or the phoneme label 102, the adaptation is performed using the phoneme data accurately segmented by the time information.

  FIG. 13 shows an example of the adaptive phoneme label 102 in the case of Japanese lyrics with time information. Note that the phoneme label 102 in FIG. 13 was manually applied. For parameter estimation at the time of adaptation, maximum likelihood linear regression MLLR (Maximum Likelihood Linear Regression) and maximum posterior probability MAP (Maximum a Posterior) estimation can be combined. Note that the combination of MLLR and MAP means that the result obtained by the MLLR adaptation method is used as a prior distribution (such as an initial value) in the MAP estimation method.

  Hereinafter, a specific adaptation technique of the acoustic model will be described. FIG. 14 is a flowchart showing details of the one-stage adaptation described above. In the one-step adaptation, the singing voice acoustic model 15 divides the single singing data including only the singing voice, that is, the adaptation music acoustic signal 103 for each phoneme based on the adaptation phoneme label 102 for the adaptation music acoustic signal 103. . Then, using the data divided for each phoneme, the parameters of the acoustic model 101 for speaking voice are re-estimated so that the phoneme of the singing voice can be recognized from the adaptive music acoustic signal 103 to obtain the acoustic model 104 for single singing. This acoustic model 104 is suitable when there is no accompaniment sound or the accompaniment sound is smaller than the singing voice.

  FIG. 15 is a flowchart showing details of the two-stage adaptation described above. In the two-stage adaptation, the dominant sound acoustic signal 105 of the most dominant sound including the singing voice extracted from the adaptive music acoustic signal including the accompaniment sound in addition to the singing voice is divided for each phoneme based on the adaptive phoneme label 102. Then, using the data divided for each phoneme, the acoustic model 106 for separated singing voice obtained by re-estimating the parameters of the acoustic model 104 for single singing so that the phoneme of the singing voice can be recognized from the dominant sound acoustic signal 105. Get. Such an acoustic model 106 for a separated singing voice is suitable when the accompaniment sound is large like the singing voice.

  FIG. 16 is a flowchart showing details of the above-described three-stage adaptation. In the three-stage adaptation, the dominant sound signal S2 obtained by suppressing the accompaniment sound by the accompaniment sound suppression method from the music sound signal S1 including the singing voice and the accompaniment sound input when the system is executed is used. A plurality of temporal correspondence feature amounts extracted by the temporal correspondence feature amount extraction means 11 from the dominant sound acoustic signal S2 of the most dominant sound including the singing voice extracted from the music acoustic signal input to the system; Using the phoneme network SN for the input music acoustic signal, the parameters of the acoustic model 106 for the separated singing voice are estimated so that the phoneme of the specific singer who sings the music of the music acoustic signal can be recognized, and the acoustic model for the specific singer 107 is obtained. Since the acoustic model 107 for the specific singer is an acoustic model that identifies the singer, the alignment accuracy can be maximized.

  In a music sound signal reproducing apparatus that reproduces a music sound signal while displaying lyrics that are temporally associated with the music sound signal on the display screen, the music sound signal is temporally supported using the system of the present invention. When the attached lyrics are displayed on the display screen, the music to be played and the lyrics displayed on the screen can be synchronized and displayed on the display screen.

  A method for automatically associating a music acoustic signal with lyrics in accordance with the present invention will be described with reference to FIGS. First, the dominant sound signal extraction means 5 extracts the dominant sound signal S2 of the most dominant sound including the singing voice at each time from the music sound signal S1 of the music including the singing voice and the accompaniment sound (dominant sound signal extracting step). Next, singing voice section estimation features that can be used to estimate a singing voice section that includes a singing voice and a non-singing voice section that does not include a singing voice from the dominant sound signal S2 at each time. The quantity extraction means 7 extracts (singing voice segment estimation feature quantity extraction step). Then, the singing voice section estimation means estimates the singing voice section and the non-singing voice section based on the plurality of singing voice section estimation feature quantities, and outputs information on the singing voice section and the non-singing voice section (singing voice section estimation step). The temporal association feature quantity extraction means 11 uses the temporal association feature quantity suitable for providing temporal correspondence between the lyrics of the singing voice and the music acoustic signal from the dominant sound acoustic signal S2 at each time. Extract (temporal association feature extraction step). Further, a phoneme network SN configured by connecting a plurality of phonemes of the lyrics of the music corresponding to the music acoustic signal S1 so that the time interval between two phonemes adjacent to the plurality of phonemes can be adjusted is stored in the phoneme network storage unit 13. Store (memory step). A singing voice acoustic model 15 for estimating a phoneme corresponding to the temporal association feature amount based on the temporal association feature amount is provided, and a plurality of phonemes in the phoneme network SN and the priority sound acoustic signal S1 are provided. The alignment means 17 performs the alignment operation | movement matched temporally (alignment step). In this alignment step, the alignment means 17 receives as input the temporal association feature amount obtained in the temporal association feature extraction step, information about the singing voice segment and the non-singing voice segment, and the phoneme network SN, and singing voice The alignment operation is executed using the acoustic model 15 under the condition that there is no phoneme at least in the non-singing voice section.

  In general, the detection of singing voice is evaluated by a correct rate (hit rate) and a correct rejection rate. However, the correct answer rate refers to the ratio that can be correctly detected as a singing voice section in the area that actually contains singing voice, and the rejection rate means the percentage that can be correctly rejected as a non-singing voice section in the area that does not actually contain singing voice. Shall point to. In addition, the singing voice section estimation means 9 employ | adopted by this said embodiment has a mechanism which can adjust the balance of a correct answer rate and a rejection rate. The reason why such a mechanism is necessary is that the standard of the correct answer rate and the rejection rate are in a trade-off relationship, and the appropriate relationship varies depending on, for example, the use. Since the estimation of the singing voice detection section has a meaning as a pre-processing of the Viterbi alignment, generally it is possible to detect without any omission if there is a possibility of including a singing voice by keeping the correct answer rate high to some extent. desirable. However, on the other hand, when used for applications such as singer name identification, only the portion containing the singing voice should be detected reliably by keeping the rejection rate high. Incidentally, none of the conventional techniques related to singing voice detection can adjust the balance between the correct answer rate and the rejection rate.

  Next, the evaluation result of the embodiment to which the present invention is applied will be described.

  The method according to the present invention was applied to digital music data and lyrics data that were actually marketed, and the display of lyrics synchronized with playback was confirmed by experiments. As a result, according to the method of the present invention, it has been confirmed that the lyrics can be temporally associated with music acoustic signals in the real world including various accompaniment sounds. Hereinafter, the evaluation experiment method will be described.

(experimental method)
10 popular singers (5 male and 5 female singers) from the popular music database (RWC-MDB-P-2001) registered in the RWC research music database, which is one of the public research music databases. ) Were randomly extracted.

  Most of the songs are sung in Japanese, but some are sung in English. In this experiment, English phonemes were approximated using an acoustic model of similar Japanese phonemes. These songs were evaluated by the 5fold cross-validation method for each gender. In other words, when evaluating a song sung by a singer, the acoustic model was adapted using another song sung by a singer of the same gender as the singer.

  19 pieces of music composed of 11 singers selected at random were used as learning data for the singing voice section detection method. These music pieces were also extracted from “RWC Music Database: Popular Music (RWC-MDB-P-2001)”.

Moreover, since these 11 singers are data for learning, they are not included in the 10 singers used for evaluation. The accompaniment sound suppression method was also applied to the learning data of the singing voice interval detection method. The value of η _fixed was set to 15.

  FIG. 9 described above shows the analysis conditions for the Viterbi alignment. As the initial acoustic model, the gender-independent monophone model in the CSRC software was used. In addition, to convert phoneme strings from lyrics, we used the Japanese morphological analysis system ChaSen, and used the reading information output at that time. Hidden Markov Toolkit (HTK) was used for adaptation of the acoustic model.

  Evaluation was performed based on the phrase unit alignment. In this experiment, the phrase means a passage separated by spaces or line breaks in the original lyrics.

  FIG. 17 is a diagram for explaining the evaluation criteria. First, as shown in FIG. 17, “the section that was correctly answered” refers to the time during which the correct answer label and the output result overlap, and the other is assumed to be “incorrect answer”. The sum of the length of the correct answer section to the total length of the music (the sum of the length of the correct answer section and the incorrect answer section) is expressed as “correct answer rate” [= total length of correct answer sections (Length of “correct” regions) / music Total length of the song]. For example, in the example of FIG. 10, “when you stop” and “turn back again” each form one phrase.

  And as a whole evaluation standard, the ratio of the section in which the phrase unit label was correct in the overall length of the music was calculated. When the accuracy exceeded 90%, it was judged that the music was correctly aligned.

(Evaluation of the entire system)
In order to evaluate the performance of the proposed method as a whole, an experiment was conducted by the method according to the invention.

  18A and 18B show the results of an evaluation experiment for confirming the effect of the present invention. As shown in FIG. 18 (A), an alignment accuracy rate of 90% or more was achieved with 8 of 10 songs except for 2 songs of # 007 and # 013. FIG. 18B is a list showing the results of the average error of the phrase start time for each music piece.

  These results show that the temporal correspondence can be estimated with sufficient accuracy for 8 out of 10 songs by this method. It can also be seen that the accuracy of male voice is higher than that of female. This is because a female voice generally has a higher F0 than a male voice, so it is difficult to extract a spectral feature amount such as MFCC. A typical error occurred in the part where humming etc. not written in the lyrics were sung.

(Confirmation of effects of acoustic model adaptation)
An alignment experiment was performed under the following four conditions for the purpose of confirming the effect of adapting the acoustic model.

  (I) No adaptation: Acoustic model adaptation was not performed.

  (Ii) One-step adaptation: The acoustic model for speaking voice was directly adapted to the separated singing voice. There was no unsupervised adaptation to specific singers.

  (Iii) Two-stage adaptation: First, an acoustic model for speaking voice was adapted to a single singing voice and then adapted to a separated singing voice. There was no unsupervised adaptation to specific singers.

  (Iv) Three-stage adaptation (proposed method): First, an acoustic model for speaking voice was adapted to a single singing voice and then adapted to a separated singing voice. Finally, unsupervised adaptation of the input acoustic signal to a specific singer was performed. In this experiment, accompaniment sound suppression (step 1) and singing voice segment detection (step 2) were applied for all conditions (i) to (iv).

  FIGS. 19A and 19B show the results of the experiment under the conditions (i) to (iv). Among these, FIG. 19 (A) shows the result of examining the accuracy rate of the alignment for each musical piece for each condition. FIG. 19B summarizes the accuracy rates in a list with numerical values.

  These results show that all the songs have a certain effect. In particular, it can be seen that condition (iv) has the highest accuracy rate. In this sense, it can be said that the condition (iv) is the best mode for carrying out the invention.

(Evaluation of singing voice segment detection)
Next, for the purpose of confirming the effectiveness of the singing voice section detection described in Step 2, the correct rate (hit rate) and the rejection rate (correct rejection rate) of the singing voice section detection for each musical composition were examined.

  At the same time, the performance of singing voice section detection itself was also evaluated. About this, it experimented on two kinds of conditions, the case where it does not use the case where singing voice area detection is used. In this experiment, the adaptation process uses all three stages (steps 1 to 3) of adaptation methods.

  FIG. 20A shows the correct answer rate (hit rate) and correct rejection rate (single voice segment detection) for each song. FIG. 20B shows a comparison of the correct answer rate of the alignment for each music piece with and without the singing voice section detection.

  From these results, on average, it can be evaluated that the accuracy rate of the alignment has been improved by applying the singing voice section detection. In particular, as is apparent from the results of FIG. 20B, it can be seen that when the singing voice section detection is applied to a music with relatively low accuracy, the accuracy rate of the alignment is particularly improved. However, with respect to # 007 and # 013, the effect of the singing voice section detection method is weak despite being applied to a music with low accuracy. The reason for this is considered to be that these music pieces could not sufficiently remove the non-singing voice section because the rejection rate of the singing voice section detection was not high as seen in FIG.

  Further, when the singing voice section detection is performed on a song whose original correct rate is high, such as # 012 and # 037, it can be seen that the correct rate is slightly reduced. This is considered to be because a singing voice section that has been erroneously removed (rejected) by singing voice section detection is always an incorrect answer during alignment.

  As described above, in the present invention, the operation was confirmed by performing experiments using Japanese lyrics. However, for English music, by creating a phoneme network by converting English phonemes to the nearest Japanese phonemes, the temporal correspondence can be estimated with relatively high accuracy for English music. It was confirmed. If an appropriate acoustic model and acoustic model adaptation data can be prepared according to the language of the target music, higher accuracy and temporal association can be estimated for music in other languages including English.

  Furthermore, it is considered that more advanced music and lyrics can be temporally associated with each other by using high-order music structure information such as partial repetitions and tempos included in the music.

  The method for temporally associating a music acoustic signal and lyrics according to the present invention is composed of independent programs in which each step is distributed in the form of a tool kit or the like at the present time. For example, it may be implemented in the form of a single computer program. The following application examples are conceivable as specific application examples of the present invention.

(Application example 1) Display of lyrics synchronized with playback This is an application of displaying lyrics synchronized with playback. The inventors of the present invention are simultaneously developing software for reproducing digital music data that changes the color of the lyrics in synchronization with the playback of music based on the time-tagged lyrics, thereby synchronizing with the singing voice being played in time. I succeeded in changing the color of the lyrics, and confirmed that the accuracy rate of the alignment was as described above.

  The movement of the lyrics displayed on the displayed screen and the color changing with the singing voice looks like a so-called karaoke at first glance, but the phrase and the lyrics are very accurate and the music can be enjoyed more fully. I got the impression that Moreover, it is completely different from the conventional one in that it is automatically associated by a program without human intervention.

(Application example 2) Cueing of music using lyrics When time information is obtained in the lyrics by the method according to the present invention, the lyrics are displayed in advance, and when a part of the lyrics is clicked, the performance starts from there. It is also possible to program as follows.

  The inventors of the present invention have succeeded in starting performance by clicking on the lyrics by adding a function to the music digital data reproduction software developed by the inventors. This operation is a function that has not been realized so far, and it can be said that a new music appreciation method has been realized in that music can be appreciated while actively selecting a user's favorite part.

  In the above application examples 1 and 2, the music digital data playback software originally developed by the inventors is used, but the present invention is not limited to this, and other music digital data playback software may be used. Of course it is good.

  The present invention is expected to be applied to industrial application fields such as music appreciation support technology or search technology, and in particular, with the recent spread of digital music data distribution services, its importance increases further. It is thought that.

It is a block diagram which shows the structure of the function implementation | achievement means implement | achieved in a computer, when implement | achieving embodiment of the system which performs a time correlation with a music acoustic signal and a lyrics automatically using a computer. It is a flowchart which shows the step in the case of implementing embodiment of FIG. 1 by running a program with a computer. It is a figure which shows the process sequence about an accompaniment sound suppression process. (A) thru | or (D) is a wave form diagram used in order to demonstrate the assumption which extracts a dominant sound sound signal from a music sound signal. It is a block diagram which shows the specific structure of a singing voice area estimation means. It is a flowchart in the case of implement | achieving the singing voice area estimation means shown in FIG. 5 with a program. It is a flowchart at the time of implement | achieving the detection of a singing voice area by a program. It is a figure used in order to demonstrate using the hidden Markov model (HMM) which goes back and forth between a singing voice state ( _Sv ) and a non-singing voice state ( _SN ). It is a figure which shows the analysis conditions of viterbi alignment. It is a figure which shows the example of the conversion from the lyrics to the phoneme string for alignment. It is a flowchart which shows the algorithm of a program when the alignment means is implement | achieved by a computer with a program. (A) is a figure which shows a mode that the phoneme network was temporally matched with respect to the signal waveform of the dominant sound acoustic signal extracted from the music acoustic signal at the time using Viterbi alignment, and (B). It is a figure which shows a mode that the time correlation of the music acoustic signal of the mixed sound containing an accompaniment sound and a lyrics was completed by returning to a lyrics from a phoneme sequence after alignment was completed. It is a figure which shows an example of the phoneme label for adaptation with time information. It is a flowchart which shows the flow in the case of producing an acoustic model. It is a flowchart which shows the flow in the case of producing an acoustic model. It is a flowchart which shows the flow in the case of producing an acoustic model. It is a figure for demonstrating evaluation criteria. (A) And (B) has shown the result of the evaluation experiment for confirming the effect of this invention. (A) and (B) show the results of experiments under conditions (i) to (iv). Among these, FIG. 19 (A) shows the result of examining the accuracy rate of the alignment for each musical piece for each condition. FIG. 19B summarizes the accuracy rates in a list in numerical values. (A) shows the correct answer rate (hit rate) and correct rejection rate of singing voice section detection for each music piece. (B) shows a comparison of the correct answer rate of the alignment for music with and without singing voice section detection.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 The system which performs the time correlation of a music acoustic signal and a lyrics automatically 3 The music sound control signal memory | storage means 5 The dominant sound sound signal extraction means 7 The feature-value extraction means for singing voice area estimation 9 The singing voice area estimation means 11 For time correlation Feature extraction means 13 Phoneme network storage means 15 Acoustic model for singing voice 17 Alignment means

Claims (13)

A dominant sound sound signal extracting means for extracting a dominant sound sound signal of the most dominant sound including the singing voice at each time from a music sound signal of a song including a singing voice and an accompaniment sound;
A singing voice section that extracts a singing voice section estimation feature quantity that can be used to estimate a singing voice section that includes the singing voice and a non-singing voice section that does not include the singing voice from the dominant sound signal at each time. A feature extraction means for estimation;
Based on a plurality of singing voice section estimation features, the singing voice section and the non-singing voice section are estimated, and a singing voice section estimation means for outputting information on the singing voice section and the non-singing voice section;
Temporal association feature extraction for extracting temporal association features suitable for temporal association between the lyrics of the singing voice and the music acoustic signal from the dominant sound acoustic signal at each time Means,
Phoneme network storage means for storing a phoneme network composed of a plurality of phonemes and short pauses for the lyrics of the music corresponding to the music acoustic signal;
A singing voice acoustic model for estimating a phoneme corresponding to the temporal association feature amount based on the temporal association feature amount, and a plurality of phonemes in the phoneme network and the priority sound acoustic signal are temporally combined. Alignment means for executing an alignment operation that is associated with each other, and the alignment means includes the temporal association feature quantity output from the temporal association feature quantity extraction means, the singing voice section, and the non-singing voice. Temporal correspondence between music audio signal and lyrics, wherein the alignment operation is executed under the condition that at least the non-singing voice segment does not exist, with the information about the segment and the phoneme network as inputs A system that performs automatic attachment. The singing voice section estimation means includes Gaussian distribution storage means for storing a plurality of mixed Gaussian distributions of singing voice and non-singing voice obtained by learning based on a plurality of learning songs in advance.
The singing voice section estimation means is configured to estimate the singing voice section and the non-singing voice section based on a plurality of singing voice section estimation feature quantities and the plurality of mixed Gaussian distributions. The system which performs the time correlation of the music acoustic signal of 1 and a lyrics automatically. The singing voice section estimating means includes
Log likelihood calculating means for calculating a singing voice log likelihood and a non-singing log log likelihood at each time based on the singing voice section estimation feature value and the mixed Gaussian distribution at each time;
Log likelihood difference calculating means for calculating a log likelihood difference between the singing voice log likelihood and the non-singing log likelihood at each time;
Histogram creation means for creating a plurality of log likelihood difference histograms obtained from the whole period of the music acoustic signal;
A threshold that maximizes the interclass variance is determined when the histogram is divided into the log likelihood difference class in the singing voice section and the log likelihood difference class in the non-singing voice section depending on the music. Bias adjustment value determining means for determining the threshold as a music-dependent bias adjustment value;
In order to correct the bias adjustment value, an estimation parameter determination means for determining an estimation parameter used when estimating a singing voice section by adding a task-dependent value to the bias adjustment value;
Weighting means for weighting the singing voice log likelihood and the non-singing voice log likelihood at each time using the estimation parameters;
A weighted plurality of singing voice log likelihoods and a plurality of weighted non-singing log likelihoods obtained from the whole period of the music acoustic signal are respectively expressed as the output probability of the singing voice state (s _V ) of the hidden Markov model and Considering the output probability of the non-singing voice state (s _N ), the maximum likelihood path of the singing voice state and the non-singing voice state in the whole period of the music acoustic signal is calculated, and the whole period of the music acoustic signal is calculated from the maximum likelihood path. The system according to claim 2, further comprising a maximum likelihood path calculation means for determining information related to the singing voice section and the non-singing voice section. The weighting means approximates the output probability logp (x | s _V ) of the singing voice state (s _V ) and the output probability logp (x | s _N ) of the non-singing voice state (s _N ) by the following equation:
In the above equation, N _GMM (x; θ _V ) represents a probability density function of a mixed gaussian distribution (GMM) of singing voice, and N _GMM (x; θ _N ) represents a probability density function of a mixed Gaussian distribution (GMM) of non-singing voice. Θ _V and θ _N are parameters previously determined by learning based on the plurality of learning songs, η is the estimation parameter,
The maximum likelihood path calculating means calculates the maximum likelihood path using the following equation:
In the above formula, p (x | s _t ) represents the output probability of the state s _t , and p (s _{t + 1} | s _t ) represents the transition probability from the state s _t to the state s _{t + 1} . A system that automatically associates the described music acoustic signal and lyrics with time. The alignment means is configured to perform the alignment operation using a Viterbi alignment;
In the execution of the Viterbi alignment, as a condition that there is no phoneme in the non-singing voice section, a condition for setting at least the non-singing voice section as a short pause is determined, and in the short pause, the likelihood of other phonemes is set to zero. The system according to claim 1, wherein the alignment operation is executed automatically.   The acoustic model for singing voice is an acoustic model obtained by re-estimating parameters of the acoustic model for speaking voice so that the phoneme of the singing voice in a song including a singing voice and accompaniment sounds can be recognized. A system that automatically associates music audio signals with lyrics in time.   The acoustic model uses a music acoustic signal for adaptation of a single singing that includes only a singing voice and a phoneme label for adaptation to the music acoustic signal for adaptation, and parameters of the acoustic model for speech are used as the acoustic music signal for adaptation. 7. The system for automatically associating the time relationship between the music acoustic signal and the lyrics according to claim 6, which is an acoustic model for single singing obtained by re-estimation so that the phoneme of the singing voice can be recognized. The acoustic model is
Using the adaptive music acoustic signal of a single singing that includes only the singing voice and the adaptive phoneme label for the adaptive music acoustic signal, the parameters of the acoustic model for speaking voice are derived from the adaptive music acoustic signal and the phoneme of the singing voice. Prepare an acoustic model for single singing obtained by re-estimation so that it can be recognized,
Using the dominant sound sound signal of the most dominant sound including the singing voice extracted from the adaptive music sound signal including the accompaniment sound in addition to the singing voice, and the adaptive phoneme label for the dominant sound sound signal, the single singing The time of the music acoustic signal and the lyrics according to claim 6, which is an acoustic model for a separated singing voice obtained by re-estimating the parameters of the acoustic model for the voice so that the phoneme of the singing voice can be recognized from the dominant sound acoustic signal. A system that automatically performs automatic association. The acoustic model is
Using the adaptive music acoustic signal of a single singing that includes only the singing voice and the adaptive phoneme label for the adaptive music acoustic signal, the parameters of the acoustic model for speaking voice are derived from the adaptive music acoustic signal and the phoneme of the singing voice. Prepare an acoustic model for single singing obtained by re-estimation so that it can be recognized,
Next, using the dominant sound acoustic signal of the most dominant sound including the singing voice extracted from the adaptive music acoustic signal including the accompaniment sound in addition to the singing voice, and the adaptive phoneme label for the dominant sound acoustic signal, Preparing an acoustic model for a separated singing voice obtained by re-estimating the parameters of the acoustic model for a single singing so that the phoneme of the singing voice can be recognized from the dominant sound acoustic signal;
Next, using the plurality of temporal association feature quantities stored in the temporal association feature quantity storage means and the phoneme network stored in the phoneme network, the sound for the separated singing voice is used. The acoustic model for a specific singer obtained by estimating model parameters so that a phoneme of a specific singer who sings the song of the music acoustic signal input to the acoustic signal extraction unit can be recognized. A system that automatically associates music audio signals and lyrics with time. In a music acoustic signal reproduction apparatus that reproduces the music acoustic signal while displaying lyrics associated with the music acoustic signal in time on a display screen,
The music acoustic signal reproducing apparatus, wherein the lyrics associated with the music acoustic signal in time are displayed on the display screen using the system according to claim 1. The dominant sound acoustic signal extraction step in which the dominant sound signal extraction means extracts the dominant sound acoustic signal of the most dominant sound including the singing voice at each time from the music acoustic signal of the song including the singing voice and the accompaniment sound;
A singing voice section estimation feature amount that can be used to estimate a singing voice section that includes the singing voice and a non-singing voice section that does not include the singing voice from the dominant sound signal at each time. A feature extraction step for singing voice estimation extracted by the feature extraction means;
Based on a plurality of singing voice section estimation features, the singing voice section estimating means estimates the singing voice section and the non-singing voice section, and outputs information about the singing voice section and the non-singing voice section; and
Temporal association feature quantity extraction means suitable for assigning temporal correspondence between the lyrics of the singing voice and the music acoustic signal from the dominant sound acoustic signal at each time. A temporal feature extraction step for extracting,
Storing a phoneme network composed of a plurality of phonemes and short pauses with respect to the lyrics of the music corresponding to the music acoustic signal in a phoneme network storage means;
A singing voice acoustic model for estimating a phoneme corresponding to the temporal association feature amount based on the temporal association feature amount, and a plurality of phonemes in the phoneme network and the priority sound acoustic signal are temporally combined. An alignment step in which the alignment means executes an alignment operation to be associated with each other,
In the alignment step, the alignment means inputs the temporal association feature amount obtained in the temporal association feature extraction step, information on the singing voice segment and the non-singing voice segment, and the phoneme network. As a method for automatically performing temporal association between a music acoustic signal and lyrics, the alignment operation is performed under the condition that no phoneme is present in at least the non-singing voice section. In order to make temporal correspondence between the music acoustic signal of the music including the singing voice and the accompaniment sound and the lyrics,
From the music acoustic signal, the dominant sound acoustic signal extraction means for extracting the dominant sound acoustic signal of the most dominant sound including the singing voice at each time;
A singing voice section that extracts a singing voice section estimation feature quantity that can be used to estimate a singing voice section that includes the singing voice and a non-singing voice section that does not include the singing voice from the dominant sound signal at each time. A feature extraction means for estimation;
Based on a plurality of singing voice section estimation features, the singing voice section and the non-singing voice section are estimated, and a singing voice section estimation means for outputting information on the singing voice section and the non-singing voice section;
Temporal association feature quantity for extracting temporal association feature quantity suitable for providing temporal correspondence between the lyrics of the singing voice and the dominant acoustic signal from the dominant sound acoustic signal at each time Extraction means;
Phoneme network storage means for storing a phoneme network composed of a plurality of phonemes and short pauses for the lyrics of the music corresponding to the music acoustic signal;
A singing voice acoustic model for estimating a phoneme corresponding to the temporal association feature amount based on the temporal association feature amount, and a plurality of phonemes in the phoneme network and the priority sound acoustic signal are temporally combined. Function as an alignment means for performing an alignment operation that is automatically associated,
The alignment means is inputted with the temporal association feature quantity output from the temporal association feature quantity extraction means, information on the singing voice section and the non-singing voice section, and the phoneme network, and at least A program for temporally associating a music acoustic signal and lyrics for executing the alignment operation under the condition that no phoneme is present in the non-singing voice section.   A computer-readable recording medium on which the program according to claim 12 is recorded.