JP2016156938A - Singing voice signal separation method and system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a singing voice signal separation method and system capable of improving accuracy in separating a singing voice signal from a musical acoustic signal including the singing voice signal and an accompaniment sound signal as compared with the prior arts.SOLUTION: Even for a spectrum bin judged as a spectrum bin including a singing voice if judged from a harmonic structure or even for a spectrum bin judged not as a spectrum bin including a singing voice if judged from an appearance possibility and a spectrum bin judged as a spectrum bin including a singing voice if judged from the appearance possibility, a time frequency mask is prepared that includes a function for masking the spectrum bin that is judged not as a spectrum bin including a singing voice if judged from the harmonic structure, from music spectrogram. Next, the time frequency mask is applied to the music spectrogram, thereby generating singing voice spectrogram for separation. Based on the singing voice spectrogram for separation, the singing voice signal is separated.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a singing voice signal separation method and system for separating a singing voice signal from a music acoustic signal including a singing voice signal and an accompaniment sound signal.

  Non-Patent Document 1 [Combining Modeling of Singing Voice and Background Music for Automatic Separation of Musical Mixtures, ISMIR (2013)] discloses an example of a conventional technique for separating singing voice and accompaniment.

  For example, Non-Patent Document 2 [Mixture of Gaussian Process Experts for Predicting Sung Melodic Contour with Expressive Dynamic Fluctuations, ICASSP (2014)] expresses the F0 trajectory of a singing voice as a superposition of discontinuous score components and fine fluctuation components. We have proposed a method for generating an F0 trajectory of a singing voice from an arbitrary score using a probability model. A similar model is also applied to the volume trajectory of singing voices in Non-Patent Document 3 [Generation process model of singing voice volume trajectory based on mixed Gaussian process, Information Processing Research Report (2013)].

Rafii, Z., Germain, F. G., Sun, D. L., and Mysore, G. J .: Combining Modeling of Singing Voice and Background Music for Automatic Separation of Musical Mixtures, ISMIR (2013) Ohishi, Y., Mochihashi, D., Kameoka, H., and Kashino, K .: Mixture of Gaussian Process Experts for Predicting Sung Melodic Contour with Expressive Dynamic Fluctuations, ICASSP (2014) Yasutoshi Oishi, Daichi Mochihashi, Hirokazu Kameoka, Kunio Kanno: Model of generation process of singing voice volume trajectory based on mixed Gaussian process, Information Processing Research Report (2013)

  In order to realize an editing system for a singing voice in a mixed sound, high-precision singing voice / accompaniment sound separation and F0 estimation of the singing voice are necessary. However, none of the conventional techniques can improve the accuracy at once in consideration of the interdependency of both tasks.

  The objective of this invention is providing the singing voice signal separation method and system which can improve the precision which isolate | separates a singing voice signal from the music acoustic signal containing a singing voice signal and an accompaniment sound signal conventionally.

  Another object of the present invention is to improve the accuracy of separating the singing voice signal from the music acoustic signal including the singing voice signal and the accompaniment sound signal in consideration of the interdependence of the singing voice / accompaniment sound separation and the F0 estimation of the singing voice. To provide a singing voice signal separation method and system that can be improved at once.

  The present invention aims to improve a singing voice signal separation method and system for separating a singing voice signal from a music acoustic signal including a singing voice signal and an accompaniment sound signal. In the method of the present invention, a music acoustic signal is first converted into a music spectrogram by performing time-frequency analysis (conversion step). Further, even if the spectrum bin is determined to be a spectrum bin including a singing voice as determined from the harmonic structure, the spectrum bin is determined not to include a singing voice as determined from the appearance possibility, and from the appearance possibility. Even if it is a spectrum bin that is judged to be a spectrum bin that includes a singing voice when judged, a time frequency that has a function of masking a spectrum bin that is judged not to contain a singing voice from the harmonic structure from a music spectrogram A mask is prepared (mask preparation step).

  Next, this time frequency mask is applied to the music spectrogram to generate a separating singing voice spectrogram (masking step). Then, the singing voice signal is separated and generated based on the singing voice spectrogram for separation (separation generation step). When the time frequency mask as described above is used, the singing voice signal can be separated from the music acoustic signal including the singing voice signal and the accompaniment sound signal with higher accuracy than in the past.

  In the mask preparation step, specifically, a time-frequency mask is prepared as follows. First, the music spectrogram is decomposed into a low rank matrix and a sparse matrix using robust principal component analysis. Next, based on the comparison of the low rank matrix and the sparse matrix, the music spectrogram has a low rank that includes a sparse matrix that includes a spectral bin where a singing voice is likely to appear and a spectral bin that is unlikely to have a singing voice A first type time-frequency mask having a function of separating into a matrix is generated. A first type of time frequency mask is then applied to the music spectrogram to isolate a singing spectrogram that includes spectral bins where a singing voice is likely to appear. Then, the singing voice fundamental frequency F0 is estimated for the separated singing voice spectrogram to estimate the singing voice fundamental frequency F0 locus. Next, based on the singing voice fundamental frequency F0 trajectory, a second type time frequency mask having a function of masking spectrum bins other than the singing voice fundamental frequency F0 and the overtone vicinity is generated. Here, “the singing voice fundamental frequency F0 and the harmonic overtone” is a frequency that falls within a predetermined frequency width around the peak of the singing voice fundamental frequency F0 and the peak of the harmonic overtone. This frequency width can also be determined automatically from the shape of the spectrum of the singing voice fundamental frequency F0 and its harmonics.

  Finally, the first type time frequency mask and the second type time frequency mask are integrated, and the second type time frequency mask is a spectrum bin that is determined to be a spectrum bin containing a singing voice. However, it is a spectrum bin that is determined not to include a singing voice spectrum in the first type time frequency mask and a spectrum bin that is determined to be a spectral bin including singing voice in the first type time frequency mask. However, a third type time frequency mask having a function of masking a spectrum bin that is determined not to be a spectrum bin including a singing voice is prepared as a time frequency mask. In this specific method, basically, singing voice and accompaniment sound separation is performed on a spectrogram by using Robust Principal Component Analysis (RPCA). If F0 information of singing voice is used, an unnecessary accompaniment sound can be suppressed. On the other hand, it is much easier to perform F0 estimation on a separated singing voice than to perform F0 estimation on a mixed sound.

  The integration of the first type time frequency mask and the second type time frequency mask means that the functions of both masks can be used together with superior functions. For example, the first type time frequency mask is used. The integration of the mask and the second type time frequency mask is the logical product of the selection area of the first type time frequency mask and the selection area of the second type second time frequency mask. Thus, both masks can be integrated.

  In another example of integration, a temporal integration of a selection region of the first type time frequency mask and a selection region of the second type time frequency mask is performed to generate a temporary integration time frequency mask, and provisional integration is performed. A third type time frequency mask can be obtained by estimating a section without a song from the time frequency mask and setting all elements of the estimated time frame to zero. In this time-frequency mask, since a section without a song is estimated and all elements of the estimated time frame are set to 0, the separation accuracy can be further improved.

  In another example of integration, a temporal integration of the selection area of the first type time frequency mask and the selection area of the second type time frequency mask is performed to generate a provisional integration time frequency mask, and provisional integration is performed. An interval without a song is estimated from the time-frequency mask, all elements of the estimated time frame are set to 0, and an element that allows consonants to pass is obtained from the first-type time-frequency mask, and the element is temporarily integrated. Reflected in the frequency mask. In this way, the separation accuracy can be further increased.

  The first type time frequency mask, the second type time frequency mask, and the third type time frequency mask are each preferably a binary mask. When a binary mask is used, since the mask is composed of a combination of 1 and 0, the singing voice and accompaniment are clearly separated, and there is almost no possibility that the singing voice remains on the accompaniment sound side.

  In the separation generating step, the singing voice signal can be separated and generated by inversely transforming the separating singing voice spectrogram into the time domain. Each step can then be performed by one or more processors.

  The singing voice signal separation system of the present invention that implements the method of the present invention includes a time frequency analysis unit, a masking unit, and a signal separation generation unit.

  The time frequency analysis unit performs time frequency analysis on the music acoustic signal and converts it into a music spectrogram. Even if a spectrum bin is judged to be a spectrum bin that includes a singing voice as judged from the harmonic structure, the masking unit may appear as a spectrum bin that is judged as not a spectrum bin that contains a singing voice based on the possibility of appearance. Even if it is a spectrum bin that is judged to be a spectrum bin that contains a singing voice as judged from gender, it has a function of masking a spectrum bin that is judged not to contain a singing voice from a harmonic structure from the music spectrogram. Using the temporal frequency mask, the temporal frequency mask is applied to the music spectrogram to generate a separating singing voice spectrogram. The signal separation / generation unit separates and generates a singing voice signal based on the singing voice spectrogram for separation.

  The time frequency mask is generated by a mask generation system. The mask generation system includes a first type time frequency mask generation unit, a first type time frequency mask storage unit, a singing spectrogram separation unit, an F0 trajectory estimation unit, and a second type time frequency mask generation. , A second type time frequency mask storage unit, and a mask integration unit.

  The first type temporal frequency mask generator divides the music spectrogram into a low rank matrix and a sparse matrix using robust principal component analysis, and sings the music spectrogram based on the comparison of the low rank matrix and the sparse matrix. A first type time-frequency mask having a function of separating a sparse matrix including spectral bins that are likely to appear and a low-rank matrix including spectral bins that are less likely to have a singing voice To do. The first type time frequency mask storage unit stores a first type time frequency mask. The singing voice spectrogram separating unit applies the first type time frequency mask to the music spectrogram to separate the singing voice spectrogram including the spectral bins where the singing voice is likely to appear. The F0 trajectory estimation unit estimates the singing voice fundamental frequency F0 with respect to the separated singing voice spectrogram to estimate the singing voice fundamental frequency F0 trajectory. The second type time frequency mask generation unit generates a second type time frequency mask having a function of masking spectrum bins other than the singing voice fundamental frequency F0 and overtones based on the singing voice fundamental frequency F0 locus. The second type time frequency mask storage unit stores a second type time frequency mask. The mask integration unit is a spectrum bin that is determined to be a spectrum bin including a singing voice in the second type time frequency mask based on the first type time frequency mask and the second type time frequency mask. However, it is a spectrum bin that is determined not to include a singing voice spectrum in the first type time frequency mask and a spectrum bin that is determined to be a spectral bin including singing voice in the first type time frequency mask. However, the third type time frequency mask having the function of masking the spectrum bin that is determined not to be a spectrum bin containing a singing voice is integrated as the time frequency mask.

  The signal separation generation unit separates and generates a singing voice signal by inversely transforming the separating singing voice spectrogram into the time domain. Note that the above-described configuration requirements are preferably realized by one or more processors and memories.

It is a block diagram which shows the structure of an example of the singing voice signal separation system which implements the singing voice signal separation method of this invention. It is a flowchart which shows the algorithm of the software used when implementing the singing voice signal separation system of embodiment of FIG. 1 with a computer (1 or more processors and 1 or more memories are included). It is a figure which shows an example of the music spectrogram obtained by carrying out time frequency analysis of the input music sound signal. An example of a result obtained by analyzing a music spectrogram into a sparse matrix and a low rank matrix by robust principal component analysis, and an example of a binary mask as a first type time frequency analysis mask obtained by comparing values of each element of both matrices FIG. In order to enhance the understanding of the display contents of FIG. 4, a part of the music spectrogram is enlarged, a part of the sparse matrix and the low rank matrix is enlarged, and a binary mask is used as a first type of time frequency analysis mask. The figure which expanded the part is shown. It is a figure which shows an example of the process which produces | generates the said 2nd type time frequency mask (binary mask) from the singing voice fundamental frequency F0 locus | trajectory estimated by the F0 locus | trajectory estimation part. The figure which shows an example in the case of integrating a 1st type time frequency mask (binary mask by robust principal component analysis) and a 2nd type time frequency mask (binary mask of the harmonic structure by singing voice fundamental frequency F0). It is. FIG. 8 is an enlarged view showing a part of the plurality of images shown in FIG. 7 in order to enhance understanding of the image of FIG. 7. It is a figure for showing processing in a masking part with an image. FIG. 10 is an enlarged view showing a part of the plurality of images shown in FIG. 9 in order to enhance understanding of the image of FIG. 9. (A) thru | or (D) is a wave form diagram which shows the condition of the masking process by a masking part. It is a figure used in order to explain resynthesis of a singing voice signal. It is a conceptual diagram which shows the other example of integration of a mask. It is a conceptual diagram which shows the further another example of integration of a mask.

  Hereinafter, an example of an embodiment of a singing voice signal separation method and system according to the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram showing the configuration of an example of a singing voice signal separation system for implementing the singing voice signal separation method of the present invention. FIG. 2 is a flowchart showing an algorithm of software used when the singing voice signal separation system of the embodiment of FIG. 1 is implemented by a computer (including one or more processors and one or more memories).

  The singing voice signal separation system of the present invention for implementing the method of the present invention includes a time frequency analysis unit 2, a music spectrogram storage unit 3, a third type time frequency mask storage unit 4, a masking unit 5, and a signal separation. And a generation unit 6. FIG. 1 also shows a mask generation system 7 for generating a time frequency mask to be stored in the third type time frequency mask storage unit 4. For the convenience of explanation, the mask generation system 7 will also be explained during the explanation of the present embodiment.

  The time frequency analysis unit 2 performs time frequency analysis on the music acoustic signal 1 including the singing voice signal and the accompaniment sound signal and converts it into a music spectrogram (matrix) (step ST1). First, a time-frequency analysis of an input music acoustic signal is performed using a short-time Fourier transform (STFT) or a constant Q transform. The constant Q transform is described in detail in “Schorkhuber, C. and Klapuri, A .: Constant-Q Transform Toolbox for Music Processing, SMC Conference (2010)”.

  In practice, logarithmic spectral bins are not obtained for all time samples n, but are cut out with a time width of 10 [msec], for example. In the following, for ease of understanding, the time index and the frequency index are described as t and f, respectively, and the music spectrogram is described as X (t, f). FIG. 3 shows an example of a music spectrogram obtained by time-frequency analysis of an input music sound signal.

  The third type time frequency mask storage unit 4 stores a third type time frequency mask as a time frequency mask (integrated mask) created by the mask generation system 7. The mask generation system 7 includes a first type time frequency mask generation unit 71, a first type time frequency mask storage unit 72, a singing spectrogram separation unit 73, an F0 trajectory estimation unit 74, and a second type. The time frequency mask generation unit 75, the second type time frequency mask storage unit 76, and the mask integration unit 77. The first type time frequency mask generation unit 71 decomposes the music spectrogram in the music spectrogram stored in the music spectrogram storage unit 3 into a low rank matrix and a sparse matrix using robust principal component analysis (step ST2). Based on the comparison of low-rank matrix and sparse matrix, the music spectrogram is a low-rank matrix containing sparse matrix containing spectral bins where singing voice is likely to appear and spectral bins containing less likely singing voice A first type time-frequency mask having a function of separating into two is generated (step ST3). The first type time frequency mask is stored in the first type time frequency mask storage unit 72.

  Robust principal component analysis is a technique for decomposing a given matrix (two-dimensional array) into a low rank matrix and a sparse matrix, and is formulated by the following equation.

Where X, L, and S are the input matrix, low rank matrix, and sparse matrix, respectively, ‖ · ‖ * and ‖ · ‖1 are the nuclear norm and L1 norm, and λ is the tradeoff between low rank and sparseness, respectively. Represents a parameter. In general, a time-varying data set or the like is input, and frequent components (components that appear repeatedly in each frame) are decomposed into low rank matrices, and other components (components that rarely appear in each frame) are decomposed into sparse matrices. .

  When the robust spectrogram analysis is applied with the music spectrogram regarded as the input matrix X, the spectrum bins of accompaniment sounds (drums and guitars) that appear many times due to repeated performance are moved to the low rank matrix L, and other singing voices, etc. Spectral bins with large temporal variations are decomposed into sparse matrix S. In the present embodiment, a binary mask is created as a first type time-frequency analysis mask from the analysis result.

The singing spectrogram can be separated by applying the first type of time frequency analysis mask comprising the binary mask to the music spectrogram X (t, f).

  FIG. 4 shows an example of the result of analyzing a sparse matrix (singing voice) and a low rank matrix (accompaniment) as a result of analysis by a robust principal component analysis from a music spectrogram, and comparing values of each element of both matrices. An example of a binary mask is shown as the obtained first type time-frequency analysis mask. FIG. 5 expands a part of the music spectrogram, expands a part of the sparse matrix (singing voice) and the low rank matrix (accompaniment) in order to improve the understanding of the display contents of FIG. The figure which expanded a part of binary mask as a time frequency analysis mask of this is shown.

  The singing voice spectrogram separating unit 73 applies the first type of time frequency mask (binary mask) to the music spectrogram, and separates the singing voice spectrogram including the spectral bins where the singing voice is likely to appear (step ST4). The F0 trajectory estimation unit 74 estimates the singing voice fundamental frequency F0 from the separated singing voice spectrogram to estimate the singing voice fundamental frequency F0 trajectory (step ST5).

Specifically, the F0 trajectory of the singing voice is estimated from the singing voice spectrogram Xs ^rpca (t, f) separated by the robust principal component analysis using the Subharmonic Summation (SHS). For SHS, see “Hermes, DJ: Measurement of pitch by subharmonic summation, J. Acoust. Soc. Am., Vol. 83, No. 1, pp.257-264 (online), DOI: 10.1121 / 1.396427 (1988). Is described in detail. SHS is a pitch estimation method that combines low calculation cost and robustness to noise. For each frequency bin of the spectrum bin, the power of the frequency bin corresponding to the harmonic overtone when the bin is assumed to be F0. By adding together with weights, the likelihood that F0 exists in the bin is calculated. The calculation of the pitch likelihood function is formulated as follows on a logarithmic frequency scale.

Here, t and s represent the time index and the logarithmic frequency [cents], respectively, and P (t, s) is the amplitude of the input spectrogram at the time frame t and the frequency s [cents]. N is the number of overtones to be added, hn is a weight function of each overtone, and in this embodiment, it is 15 and 0.86 ⁿ⁻¹ , respectively. In order to consider the non-linearity of human auditory characteristics, A characteristic correction is applied to the input spectrum bin before applying SHS.

  The singing voice pitch F (t) is calculated from the pitch likelihood function H (t, s) by SHS by the following formula.

Here, cl ^(t) and ch ^(t) are the lower limit and the upper limit ([cents]) of the pitch search frequency range in the time frame t, respectively.

  FIG. 6 shows an example of a process of generating a second type of time frequency mask (binary mask) from the singing voice fundamental frequency F0 locus estimated by the F0 locus estimation unit 74. The second type time frequency mask generation unit 75 generates a second type time frequency mask having a function of masking spectrum bins other than the singing voice fundamental frequency F0 and the harmonic overtone based on the singing voice fundamental frequency F0 locus. (Step ST6). The generated second type time frequency mask is stored in the second type time frequency mask storage unit 76.

  In conventional singing voice separation using robust principal component analysis, bass and drums that appear only in a part of the song, and back chorus that sings and shifts the pitch from the main vocal are also separated as singing voices. Singing voice / accompaniment sound separation and singing voice F0 estimation are interdependent. That is, if the singing voice F0 trajectory is given, it can be used for singing voice separation, while if the singing voice is separated, it is relatively easy to estimate the F0 trajectory. Therefore, precise singing voice separation is performed on the input acoustic signal by using an integrated mask (third type time-frequency mask) for singing voice separation using this complementary relationship. Therefore, the second type temporal frequency mask generation unit 75 uses the singing voice fundamental frequency F0 trajectory to perform more accurate singing voice separation from the singing voice fundamental frequency F0 trajectory. A harmonic mask that masks power other than the surroundings is generated as a second type of time frequency. Here, “the singing voice fundamental frequency F0 and the harmonic overtone” is a frequency that falls within a predetermined frequency width around the peak of the singing voice fundamental frequency F0 and the peak of the harmonic overtone. This frequency width can also be automatically determined from the shape of the spectrum of the singing voice fundamental frequency F0 and its harmonics.

Here, Ft represents F0 [cents] in the time frame t, C (f) represents the logarithmic frequency [cents] corresponding to the frequency bin f, H represents the number of overtones, and w represents the width [cents] for masking each overtone. . The final singing voice and accompaniment spectrograms Xs (t, f) and Xm (t, f) are obtained as follows using a binary mask and a harmonic mask by robust principal component analysis.

The mask integration unit 77 integrates the first type time frequency mask and the second type time frequency mask to create a third type time frequency mask as a time frequency mask (integrated mask) (step ST7). ). The temporal frequency mask (integrated mask) composed of the third type temporal frequency mask is a spectral bin that is determined to be a spectral bin including a singing voice as judged from the harmonic structure in terms of a superordinate concept. Judging from the harmonic structure, even if it is a spectrum bin that is judged not to be a spectrum bin containing a singing voice from the appearance possibility and a spectrum bin that is judged to be a spectrum bin containing a singing voice from the possibility of appearance It has a function of masking a spectrum bin determined not to be a spectrum bin containing a singing voice from a music spectrogram. More specifically, even if the third type temporal frequency mask (integrated mask) is a spectral bin that is determined to be a spectral bin containing a singing voice in the second type temporal frequency mask, Even if the spectrum bin is determined to be a spectrum bin including a singing voice according to the first type of time frequency mask, the second spectrum bin is determined to be a spectrum bin including a singing voice. This type of time frequency mask has a function of masking spectrum bins that are determined not to be spectrum bins containing singing voices. The time frequency mask (integrated mask) of the third type time frequency mask is stored in the third type time frequency mask storage unit 4.

  FIG. 7 shows an example in which the first type time frequency mask (binary mask based on robust principal component analysis) and the second type time frequency mask (binary mask with harmonic structure based on singing voice fundamental frequency F0) are integrated. This is shown in the image. FIG. 8 is an enlarged view showing a part of the plurality of images shown in FIG. 7 in order to enhance understanding of the image of FIG. In this example, the integration of the first type of time frequency mask (binary mask with robust principal component analysis) and the second type of time frequency mask (binary mask or harmonic mask with singing voice fundamental frequency F0) is performed as the first. The third type time-frequency mask (integrated binary mask) is obtained by integrating the two masks by ANDing the selected region of the time-frequency mask of this type and the selected region of the second-type time-frequency mask. )

The masking unit 5 applies the third type time frequency mask (integrated binary mask) to the music spectrogram to generate a separating singing voice spectrogram (step ST8). Of course, the separating spectrogram output from the masking unit 5 may be stored in the storage unit. FIG. 9 is a diagram for showing the processing in the masking unit 5 as an image. FIG. 10 is an enlarged view showing a part of the plurality of images shown in FIG. 9 in order to improve the understanding of the image of FIG. FIGS. 11A to 11D are waveform diagrams showing the state of masking processing by the masking unit 5. Note that in FIGS. 11A to 11D, the spectrum for one frame included in the spectrogram is an object to be illustrated. FIG. 11A shows a mixed sound spectrum X (f) included in the music spectrogram. FIG. 11B shows a first type of time frequency mask (Mb (f) included in the robust principal component analysis mask (Mb (f)) corresponding to the mixed sound spectrum X (f). (C) is a frequency mask [harmonic mask Mh (f)] for one frame included in the second type temporal frequency mask corresponding to the mixed sound spectrum X (f), and FIG. , First type time frequency mask (frequency mask for one frame included in robust principal component analysis mask (Mb (f) and frequency mask for one frame included in second type time frequency mask [harmonic a third type of one frame of the frequency mask time included in the frequency mask [integrated mask Mb (f) ^* mass by Mh (f)] to mask Mh (f)] is generated integrated Separated voice spectrum obtained is ring ^{[X (f) * Mb (} f) * Mh (f)] is. As can be seen from FIG. 11 (D), and performs masking using the integrated mask separation You can see that the accuracy is high.

  And the signal separation production | generation part 6 isolate | separates and produces | generates a singing voice signal based on the singing voice spectrogram for isolation | separation. Specifically, as shown in FIG. 12, the signal separation generation unit 6 separates and generates a singing voice signal by inversely transforming the separation singing voice spectrogram output from the masking unit 5 into the time domain (step ST9).

  In order to confirm the effect of the present embodiment, NSDR (determining the separation accuracy based on the distortion of the target sound source when the singing voice signal is separated by the robust principal component analysis and when the singing voice signal is separated by the present embodiment. Normalized Signal-to-Distortion Ratio [dB]), we compared the results of separating singing voice signals from 110 music audio signals. As a result, the separation accuracy of singing voice is 5.06 [dB] in this embodiment, 2.09 [dB] in robust principal component analysis, and the separation accuracy of accompaniment is 6.21 in this embodiment. dB], and the robust principal component analysis gave a result of 1.71 [dB]. In both singing voice separation and accompaniment separation, it was confirmed that the present embodiment has higher accuracy than RPCA.

  Each of the above constituent elements is preferably realized by one or more processors and memories. The mask generation system 4 need not be configured together with the singing voice signal separation system of the present embodiment. That is, the third type time frequency mask (integrated mask) may of course be generated in advance by a mask generation system provided separately from the singing voice signal separation system.

  In the above embodiment, the logical product (AND) is used to integrate the two binary masks. However, the mask integration in the present invention is not limited to the above embodiment. FIG. 13 is a conceptual diagram showing another example of mask integration. In this example, the logical product of the selection region of the first type time frequency mask (binary mask by RPCA) and the selection region of the second type time frequency mask (binary mask of harmonic structure by singing voice fundamental frequency F0). Then, a temporary integrated time frequency mask is generated. Then, from this temporary integrated time frequency mask, a section without a song is estimated, and all elements of the estimated time frame are set to 0, whereby a third type time frequency mask (integrated binary mask) can be obtained. . In this time-frequency mask, since a section without a song is estimated and all elements of the estimated time frame are set to 0, the separation accuracy can be further improved.

  FIG. 14 is a conceptual diagram showing still another example of mask integration. In this example, a selection area of a first type time frequency mask (binary mask by robust principal component analysis) and a selection area of a second type time frequency mask (binary mask of harmonic structure by singing voice fundamental frequency F0), And a temporary integrated time frequency mask is generated. Then, from this temporary integrated time frequency mask, a section without a song is estimated, all elements of the estimated time frame are set to 0, and consonants are derived from the first type time frequency mask (binary mask by robust principal component analysis). By obtaining an element that passes through and reflecting the element in the temporary integrated time frequency mask, a third type time frequency mask (integrated binary mask) can be obtained. In this way, the separation accuracy can be further increased.

  In recent years, active music appreciation systems that allow users to edit and process existing music to their liking have been extensively studied. Among them, editing singing voices in mixed sounds is one of the most difficult issues to realize, and techniques to directly convert the voice quality of existing singing voices to the voice quality of other singers have been proposed. Pitch pitch, i.e., a technique for editing a singing expression has not been realized, but according to the present invention, even if it is a spectrum bin that is determined to be a spectrum bin that includes a singing voice as judged from the harmonic structure, Judging from the harmonic structure even if it is a spectrum bin that is judged not to be a spectrum bin containing a singing voice from the appearance possibility and a spectrum bin that is judged to be a spectrum bin containing a singing voice from the appearance possibility Then, use a time-frequency mask that has a function to mask spectrum bins that are judged not to contain singing voices from the music spectrogram. Allows separation accurately than the conventional singing voice signal from the music audio signal including a voice signal and accompaniment tone signal.

DESCRIPTION OF SYMBOLS 1 Music acoustic signal 2 Time frequency analysis part 3 Music spectrogram memory | storage part 4 Time frequency mask memory | storage part 5 Masking part 6 Signal separation production | generation part 7 Mask generation system 71 Time frequency mask generation part 72 Time frequency mask memory | storage part 73 Singing voice spectrogram separation part 74 F0 locus estimation unit 75 time frequency mask generation unit 76 time frequency mask storage unit 77 mask integration unit

Claims (13)

A singing voice signal separating method for separating the singing voice signal from a music acoustic signal including a singing voice signal and an accompaniment sound signal,
Converting the music acoustic signal into a music spectrogram by performing time-frequency analysis;
Even if the spectrum bin is determined to be a spectrum bin that includes a singing voice as determined from the harmonic structure, it is determined from the spectrum bin that is determined not to be a spectrum bin that includes a singing voice as determined from the appearance possibility and the appearance possibility. Then, even if the spectrum bin is determined to be a spectrum bin including a singing voice, it has a function of masking the spectrum bin determined to be not a spectrum bin including a singing voice from the music spectrogram even if determined from the harmonic structure. A mask preparation step for preparing a frequency mask;
A masking step of applying the time frequency mask to the music spectrogram to generate a separating singing voice spectrogram;
A singing voice signal separation method comprising: a separation generating step of separating and generating the singing voice signal based on the separating singing voice spectrogram. In the mask preparation step,
The music spectrogram is decomposed into a low rank matrix and a sparse matrix using robust principal component analysis,
Based on the comparison between the low rank matrix and the sparse matrix, the music spectrogram includes a sparse matrix including a spectrum bin where a singing voice is likely to appear and a low spectrum pattern including a spectrum bin where a singing voice is unlikely to appear. Generating a first type of time frequency mask having the function of separating into rank matrices;
Applying a first type of time frequency mask to the music spectrogram to isolate a singing spectrogram including spectral bins where a singing voice is likely to appear;
A singing voice fundamental frequency F0 is estimated for the separated singing voice spectrogram to estimate a singing voice fundamental frequency F0 locus,
Based on the singing voice fundamental frequency F0 trajectory, a second type time frequency mask having a function of masking spectral bins other than the singing voice fundamental frequency F0 and overtones,
The first type time frequency mask and the second type time frequency mask are integrated, and the second type time frequency mask is a spectrum bin that is determined to be a spectrum bin containing a singing voice. However, a spectrum bin that is determined not to include a singing voice in the first type time-frequency mask and a spectrum bin that is determined to be a spectral bin that includes a singing voice in the first type time-frequency mask. Even so, a third type time frequency mask having a function of masking a spectrum bin that is determined not to be a spectrum bin containing a singing voice in the second type time frequency mask is prepared as the time frequency mask. The method of separating a singing voice signal according to claim 1.   The integration of the first type time frequency mask and the second type time frequency mask includes a selection region of the first type time frequency mask and a second type second time frequency mask. The singing voice signal separation method according to claim 2, wherein a logical product with the selected area is taken.   The integration of the first type time frequency mask and the second type time frequency mask includes a selection region of the first type time frequency mask and a selection region of the second type time frequency mask. And a temporary integrated time frequency mask is generated, and a section without a song is estimated from the temporary integrated time frequency mask, and all elements of the estimated time frame are set to zero. 3. The singing voice signal separation method according to claim 1, wherein the time frequency mask is a type 3 time frequency mask.   The integration of the first type time frequency mask and the second type time frequency mask includes a selection region of the first type time frequency mask and a selection region of the second type time frequency mask. And a temporary integrated time frequency mask is generated, a section without a song is estimated from the temporary integrated time frequency mask, all elements of the estimated time frame are set to 0, and the first The singing voice signal separation method according to claim 1, wherein an element that allows consonants to pass is obtained from a temporal frequency mask of the following type, and the element is reflected in the temporary integrated temporal frequency mask.   6. The singing voice signal separation method according to claim 3, 4 or 5, wherein the first type time frequency mask, the second type time frequency mask and the third type time frequency mask are binary masks, respectively.   2. The singing voice signal separating method according to claim 1, wherein, in the separating and generating step, the singing voice signal is separated and generated by inversely transforming the separating singing voice spectrogram into a time domain.   The singing voice signal separation method according to claim 1, wherein each step is performed by one or more processors. A singing voice signal separating system for separating the singing voice signal from a music acoustic signal including a singing voice signal and an accompaniment sound signal,
A time-frequency analysis unit that converts the music acoustic signal into a music spectrogram by performing time-frequency analysis;
Even if the spectrum bin is determined to be a spectrum bin that includes a singing voice as determined from the harmonic structure, it is determined from the spectrum bin that is determined not to be a spectrum bin that includes a singing voice as determined from the appearance possibility and the appearance possibility. Then, even if the spectrum bin is determined to be a spectrum bin including a singing voice, it has a function of masking the spectrum bin determined to be not a spectrum bin including a singing voice from the music spectrogram even if determined from the harmonic structure. Using a frequency mask, a masking unit that applies the temporal frequency mask to the music spectrogram to generate a separating singing voice spectrogram;
A singing voice signal separation system comprising: a signal separation / generation unit that separates and generates the singing voice signal based on the singing voice spectrogram for separation. The music spectrogram is decomposed into a low rank matrix and a sparse matrix using robust principal component analysis, and a singing voice may appear in the music spectrogram based on the comparison between the low rank matrix and the sparse matrix. A first type of time frequency generating a first type of time frequency mask having a function of separating a sparse matrix including high spectral bins and a low rank matrix including spectral bins that are less likely to have a singing voice A mask generation unit;
A first type time frequency mask storage unit for storing the first type time frequency mask;
A singing spectrogram separation unit that applies the first type time frequency mask to the music spectrogram to separate a singing spectrogram including a spectral bin that is likely to have a singing voice;
An F0 trajectory estimator for estimating a singing voice fundamental frequency F0 trajectory by estimating a singing voice fundamental frequency F0 with respect to the separated singing voice spectrogram;
A second type time frequency mask that is generated based on the singing voice fundamental frequency F0 trajectory and generates a second type time frequency mask having a function of masking spectrum bins other than the singing voice fundamental frequency F0 and surrounding harmonics. A generator,
A second type time frequency mask storage unit for storing the second type time frequency mask;
A spectrum that is created by integrating the first type time frequency mask and the second type time frequency mask and is determined to be a spectrum bin containing a singing voice in the second type time frequency mask. Even a bin is determined to be a spectrum bin that is determined not to be a spectrum bin that includes a singing voice in the first type time frequency mask and a spectrum bin that includes a singing voice in the first type time frequency mask. A third type time frequency mask having a function of masking a spectrum bin that is determined not to be a spectrum bin containing a singing voice in the second type time frequency mask. The time frequency mask by a mask generation system comprising: Song word signal separation system according to claim 9 is one that was created.   The singing voice signal separation system according to claim 10, wherein each of the first type time frequency mask, the second type time frequency mask, and the third type time frequency mask is a binary mask.   The singing voice signal separation system according to claim 9, wherein the signal separation / generation unit separates and generates a singing voice signal by inversely transforming the separating singing voice spectrogram into a time domain.   The singing voice signal separation system according to any one of claims 9 to 12, wherein the constituent elements are realized by one or more processors and a memory.