JP2003005790A - Method and device for voice separation of compound voice data, method and device for specifying speaker, computer program, and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method and a device for separating compound voice data where voice data of several speakers mixedly exist into the voice of every speaker and to provide a method and a device for accurately and quickly specifying the speaker of each separated voice data. SOLUTION: The method for separating compound voice data where voice data of several speakers mixedly exist into the voice data of every speaker has a step (1) where correlation elimination processing is performed to eliminate correlation between the compound voice data and a step (2) where independent component separation processing is performed to separate data subjected to the correlation elimination processing into independent components.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method of separating voices of composite voice data of a plurality of speakers, a method of specifying a speaker of each of the separated voice data, and a voice of composite voice data of a plurality of speakers. Device, a computer program, and a recording medium that specify a speaker of each separated audio data.

[0002]

2. Description of the Related Art A technique for accurately separating composite voice data in a voice recording medium, in which voices of a plurality of speakers are mixed and recorded, for each speaker is desired. Specifically, there is a need for a technology that can automatically create minutes of a conference by separating and identifying composite voice data for each speaker simultaneously with the input of voice. .

Conventionally, in order to create a minutes of a meeting over a long period of time, a person in charge of minutes recording re-listens and summarizes the audio data of the meetings recorded in various audio recording devices. I was creating. This work needs to be performed while repeating the reproduction and the temporary stop of the voice recording device, which is troublesome and time-consuming.

[0004] Another problem is that it is difficult to identify the speaker. If the person who attended the meeting was the person who attended the meeting, it was very difficult for the person who did not attend the meeting to make the minutes, and it was very difficult to determine which voice was attributed by which speaker.

Conventionally, although there are several techniques relating to voice separation from mixed voice data and speaker identification, even if voices and noises of a plurality of persons are mixed and input to one microphone, the separation is performed. Accurate identification and further high-speed separation / identification processing simultaneously with the input of complex speech is effective for segmentation of temporally continuous phoneme data.
And it was a very difficult task in terms of articulation.

[0006] Japanese Patent Laid-Open No. 2001-27895 describes a signal separation method for separating acoustic signals from a plurality of signal sources and combining and outputting a desired signal. This invention
Time / frequency analysis is performed on the mixed voice / acoustic signal to be analyzed to obtain a harmonic composition of frequency components. Among the overtone frequency components, whether at least one of the rise time and the fall time is common is used to identify whether or not the frequency components are from the same signal source. A signal from a single signal source is separated by extracting and reconstructing the frequency component.

Since the present invention does not consider the correlation and independence of mixed signals, it is difficult to separate mixed signals belonging to the same frequency band or mixed signals existing in the same time band. is there.

Further, in the sound source signal estimating device described in Japanese Patent Laid-Open No. 2000-97758, when a plurality of acoustic signals are mixed and input through a plurality of channels,
Based on a mixing process model in which each sound source signal is subjected to inner product calculation with the mixing coefficient vector and added to other sound source signals, the separation coefficient vector corresponding to the mixing coefficient vector is obtained by sequentially correcting, and the sound source is calculated using this separation coefficient vector. In estimating and separating signals (the ICA method), each signal is estimated from a signal in which a speech signal for normalizing a correction vector used for successive correction of a separation coefficient vector and a signal other than that are mixed together. When separating, it is said that it is possible to reduce the influence on the estimation and separation due to the fluctuation of each signal power, and further to increase the convergence coefficient, which enables stable and high-speed signal separation. .

Since the present invention is performed while sequentially correcting the separation coefficient vector based on the independent component analysis (ICA), it is possible to reduce the influence of fluctuations in signal power and realize high-speed separation. The sound source signals from the sources do not always maintain independence from each other. In general, even source signals from independent signal sources often have correlation when mixed, but that point is not taken into consideration.

Japanese Unexamined Patent Publication No. 9-258788 discloses a voice separation method for properly separating and separating mixed voices having fundamental frequencies close to each other so as to obtain high-quality separated voices regardless of the number of sound sources. And the device is described. In this invention, the voiced sound portion of the voice signal included in the input acoustic signal and the unvoiced sound portion are individually extracted while taking into consideration the information of the sound source direction of the voiced sound, and the extracted voiced sound portion is extracted. Divided into multiple voiced sounds and extracted as a group of voiced sounds,
The unvoiced sound part of the voice signal is extracted as a component of the acoustic signal corresponding to the unvoiced sound of each voiced sound group from the residuals extracted by subtracting the voiced sound part from the input acoustic signal, and then extracted into each separately extracted voiced sound group. The above object is realized by supplementing unvoiced sound and extracting a voice signal.

The present invention has a sound source localization section for extracting information on the direction of the sound source, but if different voices are emitted from the same direction, separation becomes difficult. Also, when multiple speakers make the same vowel or voiced sound, it seems difficult to separate them.

[0012]

SUMMARY OF THE INVENTION In order to solve the various problems of the prior art as described above, the present invention provides mixed voice data in which voice data of a plurality of speakers are mixed, with voices for each speaker. It is a principal object of the present invention to provide a method and apparatus for separating into two parts, and a method and device capable of accurately and speedily specifying the speaker of each separated audio data.

[0013]

In order to solve the above-mentioned problems, a first invention of the present application is to provide mixed voice data in which voice data of a plurality of speakers are mixed, to voice data for each speaker. In the audio data separation method for separating into (1)
Performing a decorrelation process for decorrelating the mixed speech data with each other; and (2) performing an independent component separation process for separating the decorrelated data into independent components. And a voice separation method. According to the first aspect of the invention as described above, a plurality of voice data and mixed noise are considered in consideration of both the correlation and the independence of each voice data included in the input mixed voice data (raw data). Even if the correlation or independence of the above changes temporally or spatially, it is possible to accurately separate the voices of each speaker.

A second invention according to the present application is the speech separation method according to the first invention, wherein the separability is sufficient when the separability of the data subjected to the independent component separation is insufficient. Until the above, the speech separation method is characterized in that the decorrelation process and the independent component separation process are repeatedly performed on the data subjected to the independent component separation process. According to the second aspect, the mixed voice data can be sufficiently separated into the voice data for each sound source.

A third invention according to the present application is the speech separation method according to the first or second invention, wherein the non-Gaussian data is separated into independent components as the independent component separation processing. Gaussian independent component separation processing, non-stationary independent component separation processing for separating nonstationary data into independent components, and colored independent component separation processing for separating colored data into independent components A voice which is prepared and performs one of the non-Gaussian independent component separation process, the non-stationary independent component separation process, and the chromatic independent component separation process depending on the nature of the data. It is a separation method. According to the third aspect, the optimum independent component separation process can be performed according to the property of the data subjected to the decorrelation process, so that the mixed voice data is more effective as the voice data for each sound source. Can be separated into

The fourth invention according to the present application is the speech separation method according to the third invention, wherein the independent component separation process performed first is a non-separation process for separating non-Gaussian data into independent components. A speech separation method characterized by a Gaussian independent component separation process. Since the non-Gaussian independent component separation processing is more susceptible to the decorrelation processing as its preprocessing than other independent component processing methods, according to the fourth invention, the non-Gaussian independent component separation processing is performed first. By performing the independent component separation processing, it becomes possible to effectively evaluate whether or not the decorrelation processing was successfully executed by the non-Gaussian independent component separation processing subsequent to the decorrelation processing.

Further, a fifth invention according to the present application is characterized in that, in the speech separation method according to the first to fourth inventions, the decorrelation processing performs at least a principal component analysis and a factor analysis. This is a voice separation method. According to the fifth aspect of the invention, the number of principal component data to be adopted is obtained by obtaining the contribution rate of each principal component and setting the number of components where the cumulative contribution rate exceeds a predetermined threshold as the order. It is possible to effectively perform the decorrelation process after determining the (order).

Further, a sixth invention according to the present application is to separate mixed voice data in which voice data of a plurality of speakers are mixed into voice data for each speaker, and to speak for voice data for each speaker. In the speaker identification method for identifying the person,
(1) Separating mixed voice data in which voice data of a plurality of speakers are mixed into voice data for each speaker by the voice separation method according to any one of the first to fifth inventions,
(2) preparing a specific parameter for specifying the speaker for each speaker, and (3) specifying the speaker by referring to the specific parameter for the separated voice data for each speaker. The method for identifying a speaker is characterized by the following steps. According to the sixth aspect of the invention, for example, mixed voice data including voices and noises of a plurality of speakers recorded in recorded data of a conference is separated for each sound source, and each separated. By specifying the speaker of the voice data, the conference record data can be automatically created, for example.

Further, a seventh invention according to the present application is the speaker identifying method according to the sixth invention, wherein the specific parameter is a formant frequency when the speaker produces a vowel, and the separated parameter is separated. The speaker specifying method is characterized in that a formant frequency is obtained for voice data of each speaker, and the speaker is specified by referring to the formant frequency as the specific parameter with respect to the obtained formant frequency. According to the seventh aspect of the invention, it is possible to easily identify the speaker of each separated voice data by using the formant frequency which is a feature amount that can be extracted by a simple process such as Fourier transform.

An eighth invention according to the present application is the speaker specifying method according to the seventh invention, wherein the specifying parameter is the first formant frequency and the second formant frequency when the speaker pronounces a vowel. The first formant frequency and the second formant frequency are obtained for the separated voice data of each speaker, and the first formant frequency as the specific parameter is obtained with respect to the obtained first formant frequency and the second formant frequency. And a second formant frequency to identify a speaker, which is a speaker identification method. According to the eighth aspect, the speaker can be specified using the two formant frequencies that are the first and second spectral peaks, so that the speaker can be specified easily and more accurately. .

Further, a ninth invention according to the present application is the speaker identifying method according to any one of the sixth invention to the eighth invention, wherein the voice data of each of the speakers separated is identified. When the speaker cannot be specified in the step of specifying the speaker by referring to the parameter, the formant frequencies at a plurality of time points are obtained from the voice data, and the specified parameter is determined with respect to the obtained formant frequencies at the plurality of time points. The speaker identification method is characterized by identifying the speaker by referring to the formant frequencies at a plurality of times. According to the ninth aspect of the invention, the speaker can be specified more accurately by considering the temporal variation of the formant frequency, which is a feature amount for specifying the speaker of a certain voice. You can

The tenth invention of the present application is the sixth invention.
In the speaker identification method according to any one of claims 9 to 9, the speaker is identified in the step of identifying the speaker with respect to the separated voice data for each speaker by referring to the identification parameter. If not possible, the voiced sound data is separated from the voice data, the formant frequency is obtained for the voiced sound data, and the obtained formant frequency is referred to by referring to the formant frequency as the specific parameter. It is a speaker identification method characterized by identifying. Since the identification of the speaker by the formant frequency is effective for distinguishing voiced sounds, especially vowels,
According to such a tenth invention, it is possible to more accurately identify various voices including unvoiced sounds. Here, while the voice data before the decorrelation process and the independent component separation process is the data in which the voices of a plurality of people are mixed, two processes of the decorrelation process and the independent component separation process are performed. The separated voice data separated by is the data in which a certain person's voice is extracted. Therefore, voiced sound can be extracted with high accuracy from the separated voice data separated by such two processes.

The eleventh invention of the present application is the first invention.
In the speaker identification method according to the invention of 0, if the speaker cannot be specified in the step of identifying the speaker by referring to the specific parameter in the separated voice data for each speaker, The voiced sound data is separated from the data, the first formant frequency and the second formant frequency are obtained for the voiced sound data, and the first formant frequency as the specific parameter with respect to the obtained first formant frequency and the second formant frequency. And a second formant frequency to identify a speaker, which is a speaker identification method. According to the eleventh aspect, the speaker can be identified more accurately by using the two formant frequencies, which are the first and second spectral peaks, for the separated voiced sound data. You can specify.

The twelfth invention of the present application is the first invention.
In the speaker identification method of the 0th invention or the 11th invention, the speaker cannot be identified in the step of identifying the speaker with respect to the separated voice data for each speaker by referring to the identification parameter. For the voiced sound data, the formant frequencies at a plurality of time points are determined, and with respect to the obtained formant frequencies at a plurality of time points, the speaker is identified by referring to the formant frequencies at the plurality of time points as the specific parameters. This is a characteristic speaker identification method. According to such a twelfth invention, by taking into consideration the temporal variation of the formant frequency, which is the feature amount for specifying the speaker, with respect to the separated voiced sound data,
The speaker can be specified more accurately.

The thirteenth invention of the present application is the first invention.
In the speaker identification method of the 0th invention or the 11th invention,
When separating the voiced sound data from the voice data,
The speaker identifying method is characterized in that an independent component separation process for separating the voice data into independent components is performed. Since the voiced sound is accompanied by the vibration of the vocal cord, according to the thirteenth invention, by performing the independent component separation processing on the voice data, the unvoiced sound without the vibration of the vocal cord and the voiced sound with the vibration of the vocal cord are generated. Can be easily separated.

Further, a fourteenth invention according to the present application is a minutes creating method for creating a minutes from mixed voice data in which voice data of a plurality of speakers are mixed. According to the speaker identifying method of any of the inventions, a step of identifying a speaker in the separated voice data for each speaker, the identified speaker, and the statement of the speaker are recorded in association with each other. And a step of creating a minutes by outputting the minutes to the minutes. According to the fourteenth invention, since the speaker is specified automatically and accurately,
It is convenient because the minutes of a long meeting can be created automatically.

The fifteenth invention of the present application is to provide mixed voice data in which voice data of a plurality of speakers are mixed,
In a voice data separation device for separating the voice data for each speaker, a decorrelation process is performed in order to decorrelate the mixed voice data with each other, and the data subjected to the decorrelation process is separated into independent components. In order to achieve this, an audio component separation device is characterized by performing independent component separation processing. According to the fifteenth aspect, a plurality of voice data and mixed noise are considered in consideration of both the correlation and the independence of each voice data included in the input mixed voice data (raw data). It is possible to realize a voice separation device capable of accurately separating the voice of each speaker even when the correlation or independence of the above changes temporally and spatially. According to a fifteenth aspect of the invention, in the speech separation device according to the fifteenth aspect, when the separability of the data on which the independent component separation is performed is insufficient, the independent component separation processing is performed until the separability is sufficient. The speech separation apparatus is characterized in that the decorrelation processing and the independent component separation processing are repeatedly performed on the obtained data. According to such a sixteenth invention, it is possible to realize a voice separation device capable of sufficiently separating mixed voice data into voice data for each sound source.

The seventeenth invention of the present application is the first invention.
In the speech separation device according to the fifth or sixteenth aspect of the invention, depending on the nature of the data, the non-Gaussian independent component separation process for separating non-Gaussian data into independent components and the non-stationary One of non-stationary independent component separation processing for separating data into independent components and colored independent component separation processing for separating chromatic data into independent components It is a voice separation device. According to the seventeenth aspect, the optimum independent component separation process can be performed according to the property of the data subjected to the decorrelation process, so that the mixed voice data is more effective as the voice data for each sound source. It is possible to realize a voice separation device that can be separated into two.

The eighteenth invention of the present application is the first invention.
In the speech separation device according to the invention of claim 7, the independent component separation process performed first is a non-Gaussian independent component separation process for separating non-Gaussian data into independent components. Is. The non-Gaussian independent component separation process is more susceptible to the decorrelation process as its preprocessing than other independent component processing methods. By implementing the independent component separation processing, it is possible to realize a speech separation device capable of effectively evaluating whether or not the decorrelation processing has been successfully executed by the non-Gaussian independent component separation processing subsequent to the decorrelation processing. it can.

The nineteenth invention of the present application is the first invention.
The speech separation apparatus according to any one of the fifth through eighteenth inventions is characterized in that the decorrelation processing performs at least a principal component analysis and a factor analysis. According to the nineteenth aspect, the number of principal component data to be adopted is obtained by calculating the contribution ratio of each principal component and setting the number of components where the cumulative contribution ratio exceeds a predetermined threshold as the order. It is possible to realize a voice separation device capable of effectively performing decorrelation processing after determining (order).

The twentieth invention of the present application is to provide mixed voice data in which voice data of a plurality of speakers are mixed,
In a speaker identifying device that separates voice data for each speaker and identifies a speaker for the voice data for each speaker, the voice separating device according to any one of the fifteenth to nineteenth inventions The mixed voice data in which the data is mixed is separated into the voice data for each speaker, and the separated voice data for each speaker is referred to a specific parameter for specifying the speaker for each speaker. The speaker identifying apparatus is characterized in that the speaker is identified by According to such a twentieth invention, for example, mixed voice data containing voices and noises of a plurality of speakers, which is recorded in recorded data of a conference, is separated for each sound source, and each separated. By specifying the speaker of the voice data, it is possible to realize a speaker specifying device capable of automatically creating conference record data.

The twenty-first invention of the present application is the second invention.
In the speaker identifying apparatus according to the invention of 0, the specific parameter is a formant frequency when the speaker produces a vowel, and the formant frequency is calculated for each of the separated voice data of each speaker. Regarding the formant frequency, the speaker specifying device is characterized in that the speaker is specified by referring to the formant frequency as the specifying parameter. According to the twenty-first aspect of the invention, by using the formant frequency, which is a feature amount that can be extracted by a simple process such as Fourier transform, it is possible to easily specify the speaker of each separated voice data. The person identification device can be realized.

The 22nd invention of the present application is the 2nd invention.
In the speaker identifying device according to the first aspect of the present invention, the specific parameters are a first formant frequency and a second formant frequency when a speaker produces a vowel, The first formant frequency and the second formant frequency are obtained, and the first formant frequency and the second formant frequency as the specific parameters are determined with respect to the obtained first formant frequency and the second formant frequency.
It is a speaker specifying device characterized in that a speaker is specified by referring to a formant frequency. According to the twenty-second aspect, the first and second spectral peaks of 2
By specifying the speaker using one formant frequency, a speaker specifying device capable of specifying the speaker easily and more accurately can be realized.

The 23rd invention of the present application is the 2nd invention.
In the speaker identifying apparatus according to any one of 0th to 22nd inventions, when the speaker cannot be specified by referring to the specific parameter in the separated voice data for each speaker, A speaker characterized in that formant frequencies at a plurality of time points are obtained from the voice data, and the obtained formant frequencies at a plurality of time points are referred to by referring to the formant frequencies at a plurality of time points as the specific parameters. It is a specific device. Such a second
According to the invention of claim 3, by taking into consideration the temporal variation of the formant frequency, which is a feature amount for specifying the speaker of a certain voice, a statement that enables the speaker to be specified more accurately The person identification device can be realized.

The twenty-fourth invention of the present application is the second invention.
In the speaker identifying device according to any one of 0th to 23rd inventions, when the speaker cannot be specified by referring to the specific parameter for the separated voice data for each speaker, The voiced sound data is separated from the voice data, the formant frequency is obtained for the voiced sound data, and the speaker is specified by referring to the formant frequency as the specific parameter with respect to the obtained formant frequency. It is a speaker identification device that does. Since the speaker identification by the formant frequency is effective for identifying voiced sounds, particularly vowels, according to the twenty-fourth aspect, it is possible to more accurately identify various voices including unvoiced sounds. Here, while the voice data before the decorrelation process and the independent component separation process is the data in which the voices of a plurality of people are mixed, two processes of the decorrelation process and the independent component separation process are performed. The separated voice data separated by is the data in which a certain person's voice is extracted. Therefore, voiced sound can be extracted with high accuracy from the separated voice data separated by such two processes.

The 25th invention of the present application is the second invention.
In the speaker identifying device according to the invention of claim 4, in the case where the speaker cannot be specified by referring to the specific parameter for the separated voice data for each speaker, the voiced sound data is separated from the voice data. , The first formant frequency and the second formant frequency of the voiced sound data,
With respect to the obtained first formant frequency and second formant frequency, the speaker specifying device is characterized in that the speaker is specified by referring to the first formant frequency and the second formant frequency as the specifying parameters.
According to such a twenty-fifth aspect, the first and second spectral peaks of the separated voiced sound data are 2
By specifying the speaker using one formant frequency, a speaker specifying device capable of specifying the speaker more accurately can be realized.

The 26th invention of the present application is the second invention.
In the speaker identifying apparatus according to the fourth invention or the twenty-fifth invention, regarding the separated voice data for each speaker, if the speaker cannot be identified by referring to the specific parameter, the voiced sound data is analyzed. , Formant frequencies at a plurality of time points, with respect to the obtained formant frequencies at a plurality of time points, with reference to the formant frequencies at a plurality of time points as the specific parameter, a speaker specifying device characterized by specifying a speaker is there. According to the twenty-sixth aspect, the speaker's identification can be performed more accurately by considering the temporal variation of the formant frequency, which is the feature amount for the speaker identification, with respect to the separated voiced sound data. It is possible to realize a speaker identification device capable of performing.

The twenty-seventh invention of the present application is the second invention.
In the speaker identifying apparatus according to the fourth invention or the twenty-fifth invention,
When separating the voiced sound data from the voice data,
The speaker identifying device is characterized in that an independent component separation process for separating the voice data into independent components is performed. Since the voiced sound is accompanied by vibration of the vocal cord, according to the twenty-seventh aspect of the invention, by performing independent component separation processing on the voice data, unvoiced sound without vibration of the vocal cord and voiced sound with vibration of the vocal cord are obtained. It is possible to realize a speaker identification device that can easily separate the speakers.

The twenty-eighth invention of the present application is a minutes-creating apparatus for creating a minutes from mixed voice data in which voice data of a plurality of speakers are mixed.
Through the speaker identifying apparatus of any one of the twenty-seventh invention,
It is possible to create a minutes by specifying a speaker in the separated voice data for each speaker and outputting the specified speaker and the statement of the speaker in association with each other on a recording medium. It is a characteristic minutes creating device. According to the twenty-eighth invention described above, since the speaker is automatically and accurately specified, the minutes creating apparatus capable of automatically creating the minutes of the conference for a long time can be realized.

A computer program for causing a voice separation device to execute the voice separation method of any of the first to fifth inventions can also be realized.

A computer program for causing the speaker identifying apparatus to execute the speaker identifying method according to any one of the sixth to thirteenth inventions can be realized.

A computer-readable recording medium recording such a computer program can also be realized.

[0043]

BEST MODE FOR CARRYING OUT THE INVENTION == Voice Separation of Mixed Voice Data =
= Hereinafter, more specific embodiments of the present invention will be described in detail with reference to the drawings. First, the voice separation step of mixed voice data, which is the first half of the method of the present invention, will be described.

In this embodiment, two microphones (microphone 1 and microphone 2) pick up voice data of the content of a statement made by a two-person conference. FIG. 1 shows a waveform of audio data (raw data) X input from the microphone 1. The mixed voice data may include not only voice data of a plurality of speakers but also music and noise. The two utterances will be referred to as sound sources S1 and S2, respectively.

FIG. 2 is a diagram showing a cycle of voice separation processing. The mixed voice data input from the microphone 1 and the microphone 2 is first subjected to the decorrelation processing W1. The audio data passed to the decorrelation processing W1 is [1] and [2] in FIG.
It is segmented like this and passed one by one. For maximum efficiency, the segments overlap each other by ½ cycle.

In FIG. 2, the IC tuner, which is the next step of the decorrelation processing W1, is an independent component analysis (IC
This is a tuner for selecting the method A) from three types. Independent component separation process W which is the next step
2 performs one of three types of separation processing W2 (α) based on non-Gaussianity, separation processing W2 (β) based on non-stationarity, and separation processing W (γ) based on chromaticity. . The evaluator E in the step after W2 evaluates the separability of the data separated in W2. Until the voice separation performance of mixed voice data input from the microphone becomes sufficient,
The above cycle of W1 → IC tuner → W2 → E is repeated. However, in the first cycle, as the independent component separation processing W2, the independent component separation processing W2 (α) based on the non-Gaussian property is performed, and in the second and subsequent cycles, W2 (α), W2
An appropriate component independent component separation process is performed from among the three types of (β) and W2 (γ).

FIG. 3 shows the first voice separation cycle. Mixed voice data x1 and x2 from the microphone 1 and the microphone 2 in the time segment [1] in FIG.
Is first input to the decorrelation processing W1.

7 and 8 show x1 and x2 digitized waveform diagram data (the vertical axis represents the sound intensity and the unit is millivolts). FIG. 9 is a scatter plot of the x1 and x2 data at each time point with the abscissa representing the intensity of x1 and the ordinate representing the intensity of x2. The scatter plot exhibits a slightly linear distribution from the first quadrant to the third quadrant, indicating that the data of x1 and x2 are correlated with each other. When these raw data x1 and x2 are subjected to the decorrelation processing W1, they are converted into data f1 and f2 having no correlation with each other.

A scatter plot of f1 and f2 is shown in FIG. The horizontal axis of FIG. 10 represents the first factor f1 of the factor score F, and the vertical axis represents the second factor f2 of the factor score F. 9 is distributed in a parallelogram shape that is distorted with respect to the axis, it is distributed in a rhombus shape that is straight and has a regular shape with respect to the axis.
It can be seen that and f2 are no longer correlated with each other.

Here, the contents of the decorrelation process will be described. FIG. 6 shows a flowchart of an example of the decorrelation process W1. First, the raw audio data x1 and x2 shown in FIGS. 7 and 8 are standardized by the equation (1). As a result of standardization, the data has an average of 0 and a standard deviation of 1.

[Equation 1]

The correlation matrix (vector C) of the raw data x1 and x2 is obtained from the equation (2). In formula (2), (x1,
x2) represents the dot product of the vectors.

[Equation 2]

The eigenvalue λi and eigenvector A for the above correlation matrix are obtained from (3).

[0053]

[Equation 3]

Now, by factor analysis, we are trying to obtain mutually uncorrelated factor scores. At that time, it is important to start with the first factor and up to which factor. When adopting up to the mth factor,
Call it m-dimensional. By the eigenvector A obtained earlier,
The principal component Z is obtained by the equation (4).

[Equation 4]

Next, a factor analysis is performed on the m factors by the definition equation of the form (5). E in equation (5)
Is called a special factor.

[Equation 5]

This factor model takes the expression (6).
The factor load bij and the factor score F in the equation (6) are
It is determined by the equations (7) and (8). Then, in the final step of the flowchart of FIG. 6, the raw audio data is eventually converted into factor scores (vector F) that are uncorrelated with each other.

[Equation 6]

[Equation 7]

[Equation 8]

The main feature of W1 described above is that principal component analysis and factor analysis are combined. The effect is that the contribution ratio of each principal component can be obtained at the same time by executing the principal component analysis. Therefore, for example, the main contribution until the cumulative contribution ratio from the first-order principal component to the m-th-order principal component exceeds 80%. By adopting the component, the order m is determined. The raw audio data to be separated has a large temporal variation, and the degree of correlation due to mixing greatly changes.
How many factors are adopted is an important point in decorrelation processing.

When the number of speakers is known in advance, the order m may be fixed to the number of speakers, but when the number of speakers is unknown, for example, the cumulative contribution rate is a predetermined threshold. Let m be the number of principal components when the value is exceeded. As a method of determining the order m, it is preferable to prepare various methods according to the system and to change (tune) flexibly. Next, an example of this tuning will be described in detail.

In FIG. 23, the order m is calculated by the method according to the system.
It is a flow chart which shows the procedure which determines. FIG. 23
Here, RK0 is the initial threshold value of the cumulative contribution rate, M is the maximum order (upper threshold value of the order) that can be adopted, and ΔRK is the change amount of the cumulative contribution rate. When the principal component analysis is performed, a graph showing the relationship between the degree m (indicating that the m-th principal component is adopted) and its cumulative contribution rate is obtained as shown in FIG. FIG. 21 shows an example of three types of graphs of A, B, and C.

First, as a first processing step, a threshold value RK0 (80% in this embodiment) is set in the cumulative contribution rate RK, and an order m exceeding this threshold value RK0 is obtained. However, if the order is too large, the subsequent processing becomes too complicated. Therefore, the upper limit M of the order is determined in advance. In the example of FIG. 21, assuming that M = 4, in the case of A, the order m = 2 that exceeds the threshold value RK0, so that m = 2.
<4 = M, and the order m is determined to be 2. In the example of B, the degree m exceeding RK0 is 5, so that m = 5> 4 =
It becomes M, and the order m is not determined yet. Similarly, in the case of C, the order m is not determined.

In such a case, the second step shown in FIG. 22 is executed. That is, for an increase in the order m,
Check the difference change amount ΔRK of RK. In short, this is a processing method in which the order m that maximizes the change in the cumulative contribution rate is the order to be adopted. In this embodiment, B
In the example, ΔRK takes the maximum value when m = 2 and in the example C, m = 4. Also in this case, if the order m is lower than the upper limit value M, the order m is adopted, but if it is higher than M, the processing is sent to the next step.

If the order m exceeds the upper limit M even in the second step, the cumulative contribution ratio threshold R
K0 is lowered to, for example, 60% (= RK1), and the comparison is performed in the same manner as the first step. If the order above the new threshold value RK1 is M = 4 or less,
This is adopted as the order m, and when it exceeds M, the values of RK2, RK3, ... RKn are sequentially decreased with a predetermined decrease amount. However, if the cumulative contribution rate RK is less than 50%, it means that half or more of the information is lost, so the lower limit value of RKn is set to 50%.

When the order m is RKn = 50% or more and is not found with a value less than or equal to M, the same process as in the second step above is performed again, that is, the order at which ΔRK becomes maximum is obtained, and the value is It will be adopted as the order m. This is based on the idea that since the cumulative contribution ratio changes significantly, more information is stored before and after the order, and therefore it is desirable to adopt at least up to that order.

As described above, in FIG. 3, the decorrelated data f1 and f2 are immediately sent to the independent component separating process W2. In the first speech separation cycle, the independent component separation process W2 (α) based on non-Gaussianity is executed on these decorrelated data f1 and f2.

As described above, the separation signals a and b are obtained by the processing of W1 and W2 (α) in FIG. 3, and their separation characteristics (whether or not they are sufficiently separated) are evaluated by the evaluator E and separated. When the value is insufficient (* 1 in the figure), the second cycle is executed for the data of a and b.

An example of the second cycle is shown in FIG. Figure 3
Similar to the first cycle shown in, but with the addition of processing in the IC tuner. Independent component separation process W2
Before performing, the signal characteristics of the data f1 ′ and f2 ′ subjected to the second decorrelation processing by the IC tuner are analyzed, and the processing W2 (α) based on non-Gaussianity and the processing W2 based on non-stationarity are analyzed.
Either (β) or the process W2 (γ) based on chromaticity is selected as W2. In this example, W2 (β) is executed. The separability of the data y1 and y2 after the processing W2 (β) is evaluated by the evaluator E, and when it is insufficient (* 2 in FIG. 4), the third cycle is executed.

Here, the function of the IC tuner will be described. The IC tuner evaluates the Gaussianity, stationarity, and chromaticity of the decorrelation-processed input data as follows, and selects the optimum independent component separation process from the three types.

First, the IC tuner evaluates the Gaussian property of two input data. Specifically, for each input data, it is checked whether the frequency distribution of the input time series data is the Gaussian function (normal distribution function) type or the non-Gaussian function type.
If the input data is gs and the Gaussian function is g0, the absolute value of the difference between the two, that is, | gs-g0 |, integrated in the interval Δg is a non-Gaussian type if the value Δg is larger than a predetermined threshold δg. If it is small, it is evaluated as Gaussian. If none of the input data subjected to the decorrelation process is non-Gaussian type, the IC tuner selects the process W2 (α) based on non-Gaussian property as the independent component separation process W2.

If any of the decorrelated input data is evaluated as Gaussian, then the IC tuner evaluates the stationarity of the two input data. In this evaluation, a set average of a plurality of irregular waveforms is taken, and attention is paid to the time change of the set average. If the collective average is constant with respect to the time axis, it is “completely stationary”. If it fluctuates with time, the probability density distribution in a certain time width is obtained, and the nonstationarity is quantified from the variance, skewness, and kurtosis. The strength of non-stationarity is the magnitude of variance, the magnitude of skewness,
Since the influence of the degree of kurtosis is strongly influenced, it is preferable to evaluate after weighting according to the strength. When any of the input data subjected to the decorrelation is evaluated to have non-stationarity, the IC tuner selects the non-stationarity-based process W2 (β) as the independent component separation process W2.

If any of the decorrelated input data is evaluated as having stationarity, then the IC tuner evaluates the chromaticity of the two input data. To evaluate chromaticity, an autocorrelation function of irregular waveform is obtained. A graph of the autocorrelation function with respect to the magnitude of the time lag τ is obtained, and how much the center of gravity of the graph deviates from the origin (τ = 0) is checked. When the position of the center of gravity deviates from the origin (τ = 0) by a predetermined value or more, it is evaluated as having color. In the case of white noise,
The autocorrelation function has a value only at τ = 0. When all of the input data subjected to the decorrelation are evaluated to have chromaticity, the IC tuner selects the chromaticity-based process W2 (γ) as the independent component separation process W2.

FIG. 5 shows the third cycle. Each process is similar to the second cycle, but in the third independent component separation process, W2 (γ) based on the chromaticity is executed in this example.

Here, the above-described three types of independent separation processing W2
The contents of (α), W2 (β), and W2 (γ) will be described in more detail. First, the signal source estimation procedure by the independent component separation processing W2 (α) based on non-Gaussianity
First, the separation coefficient (matrix) Wt is appropriately assumed (the initial value is W0).

Next, the signal source y (t) for the data F (t) after the decorrelation processing is estimated as in the equation (9).

[Equation 9] Using this y (t) and Wt, ΔWt is obtained from the equation shown in equation (10).

[Equation 10]

From equation (11), Wt + 1 in the next convergence calculation step is obtained. The above steps are repeated with this Wt + 1 as a new Wt. Then, y (t) at the time when ΔWt becomes almost zero, that is, when Wt is considered to have sufficiently converged is the mixed voice raw data x (t).
It becomes an estimated signal of the signal source s (t) obtained from

[Equation 11]

Secondly, the signal source estimation procedure by the independent component separation process W2 (β) based on the non-stationarity is as follows. ² Find the initial value of the moving average Φ of (t).
Further, y (t) is calculated by the equation (12). In Expression (12), I is an identity matrix.

[Equation 12]

Next, Φ is obtained by solving the differential equation shown in equation (12). In the equation (13), T'is a time constant of the system.

[Equation 13]

Next, Φ, Ct, y in the equation (12)
A new Ct + 1 is obtained from (t) using the differential equation shown in equation (14). In equation (14), T is the time constant of the system.

[Equation 14] The obtained Ct + 1 and the data F (t) after the decorrelation process
From this, y (t) in the next step is estimated using the equation (15).

[Equation 15] The above steps are repeated using this y (t) and Ct + 1. Then, y (t) at the time when Ct is considered to have sufficiently converged becomes an estimated signal of the signal source s (t) obtained from the mixed voice raw data x (t).

Thirdly, regarding the signal source estimation procedure by the independent component separation processing W2 (γ) based on chromaticity, first, the separation coefficient matrix Ct and the initial values of Ψ1 and Ψ2 are given. here,
Ψ1 and Ψ2 are two products (y1 * y1 ^T ) and (y2 * y2 ^T ) made from y (t), which are two types of linear filters y1 (t) and y2 (t). It is a time average. In addition, y (t) is the data F after decorrelation processing.
Estimate from (t) using equation (16).

[Equation 16] By applying two types of linear filters G1 and G2 to this y (t), y1 (t) and y2 (t) are obtained by the equation (17).

[0079]

[Equation 17] From the above initial values of Ψ1 and Ψ2, and y1 and y2,
Ψ1 and Ψ2 are newly added using the differential equation shown in equation (18).
Ask for.

[0080]

[Equation 18] From Ct, Ψ1, Ψ2, a new Ct + 1 is obtained by the equation (19).

[0081]

[Formula 19]

From this Ct + 1 and the data F (t), a new y (t) is obtained by the above equation (16).
Then, this change in Ct, that is, the change in y (t) becomes sufficiently small, and y
(T) becomes an estimated signal of the signal source s (t) obtained from the mixed voice raw data x (t). If it has not yet converged, y1 (t) and y2 (t) are obtained from the equation (17), and the above steps are repeated.

Returning to FIG. 5, the evaluator E judges that the output data y1 ', y2' of the third separation cycle here have sufficient separability. That is, y1 ',
It is considered that y2 ′ corresponds to the voice of either the sound source S1 or S2, respectively. Digitized waveform diagrams of these data are shown in FIGS. 11 and 12. When the points whose amplitude is below a certain level are analyzed as noise instead of utterance, y1 ′
The voice data of "a" (-) and "ka" (-) can be seen in. Similarly, voice data of "shi" (-) is seen in y2 '.

FIG. 13 is a scatter diagram in which the magnitudes of y1 'and y2' are plotted on the horizontal axis and the vertical axis, respectively. As can be seen from this figure, the values of ,,,,, have y2 ′ values of almost zero, and conversely, the points of ,, have y1 ′ values of almost zero. It can be seen that the sound from was separated exactly.

In the evaluator E, in order to evaluate the separability of the data after executing the process W2, points such as in the graph of FIG. 13 may be examined. That is, in the scatter diagram, a point that is most distant from the horizontal axis or the vertical axis is selected, and if the distance to the axis is a certain value or more, the separability is still insufficient, and again, in FIG. 4 and FIG. Such a separation cycle is executed.

== Identification of Speaker of Separated Audio Data == Next, the step of identifying the speaker of each separated audio data, which is the latter half of the present invention, will be described. Figure 1
4 is the separated data y1 obtained in the voice separation step.
2A and 2B are a waveform diagram of 'and a spectrum distribution diagram by its Fourier transform. Here, as the method of obtaining the spectral distribution,
A filter bank, LPC method, or the like can be used.

Similarly, FIG. 15 is a waveform diagram of the separated data y2 'and a spectrum distribution diagram by its Fourier transform. In this embodiment, since two speakers (A and B) are assumed to be speakers, these two waveform data y
It was separated into 1'and y2 ', but which is A at this point?
With Mr.'s voice, I do not know which is Mr. B's voice. We will identify it from now on.

First, as a first method for specifying a speaker, a method of using a formant frequency as a speaker specifying parameter is executed. Fo1 and f in FIG.
o2 is the first formant frequency and the second formant frequency of the y1 ′ data, and go1 and go2 in FIG.
Are the first formant frequency and the second formant frequency of the y2 ′ data. Conference participants A and B in advance
The first and second formant frequency data of Mr. is prepared in the database as specific parameters for speaker identification. Then, the speaker of each separated voice data is specified by referring to the formant frequencies of the separated data y1 'and y2'.

FIG. 16 is a process for matching the formant frequencies of all five vowels of Mr. A and Mr. B, which are specific parameters, with the first and second formant frequencies of the obtained separated speech data y1 'and y2'. It is a conceptual diagram of.
The horizontal axis represents the first formant frequency, and the vertical axis represents the second formant frequency. First, the spread area of the formant frequency of the pronunciation of Mr. A's vowel (the area surrounded by the solid line in the figure) and the spread area of the formant frequency of the pronunciation of Mr. B's vowel (the area surrounded by the dotted line in the figure) are shown. On top of FIG. 14 and FIG.
The formant frequencies of the separated speech data of No. 5 are plotted.

The formant frequencies of y1 'and y2' are
If it falls within the formant frequency range of Mr. A or Mr. B, it can be said that the speaker can be identified by this. However, if it does not fit into any of the formant frequency regions of Mr. A and Mr. B (portion C in FIG. 16), or if it falls within the overlapping portion D of the regions of Mr. A and Mr. B, this first Since the speaker cannot be specified by the method, the second specifying method or the third specifying method described below is executed.

The second speaker identification method is a method of
Method of using cloak frequency as speaker identification parameter
Is. FIG. 17 shows a speech separation stream which is the first half of the present invention.
Some audio data separated by a step (sound of "A"
Voice) divided into n sampling times and Fourier transformed
It is a figure which shows that it decomposed into a spectrum. Each
The first formant frequency which is the first and second peaks with respect to
Number (f1¹ , F1^Two, ... f1ⁿ ) And the second forma
Frequency (f2¹ , F2^Two , ... f2 ⁿ )
Meru.

Next, principal component analysis is performed on these formant frequency data to obtain principal component scores Z1, Z2 ,.
..Zn is obtained and used as the feature amount of the voice of the speaker. Therefore, as speaker identification parameters prepared in advance in the database, the principal component scores Z1, Z2, ... Of various voices (all vowels, etc.) of conference participants are prepared.

FIG. 18 is a graph showing the result of the second speaker identification method. FIG. 18A shows a result obtained by the first speaker identification method provided for comparison. FIG.
In (a), the first and second formant frequencies at the five sampling times are plotted for the sound of "A", but it cannot be said to belong to the region of Mr. A or the region of Mr. B. .

On the other hand, FIG. 18B is a distribution diagram in which the first and second principal component scores Z1 and Z2 are two-dimensional coordinate axes according to the second method. First, as in the example of this figure, the area of Mr. A (solid line in the figure) and the area of Mr. B (dotted line in the figure) are often clearly separated on this principal component score plane, so judgment is easy. is there. When the principal component scores (Z1, Z2) of the separated data obtained from the results of FIG. 17 are plotted, it is clearly close to the area of Mr. A, so “a” in this case
It is understood that is the pronunciation of Mr. A.

The third speaker identification method is to further separate only voiced sound data from the voice separation data and examine the formant frequency as a speaker identification parameter. FIG. 19 is a diagram in which the separated voice data y1 ′ is separated into the voiced sound component A1 and the other component A2, and the respective spectrum distributions are obtained by Fourier transform. In order to separate the voiced sound and the other data, the IC tuner used in the voice separation step in the first half of the present invention and the three types of independent component separation processing are used.

FIG. 20 shows the first and second formant frequencies f1 ⁽¹⁾ and f of A1 and A2 in FIG. 19, respectively.
2 is a graph in which 2 ⁽¹⁾ , f1 ⁽²⁾ , and f2 ⁽²⁾ are plotted on the first and second formant frequency planes. According to the graph, the first voiced sound component A1 of y1 ′,
Second formant frequency (f1 ⁽¹⁾ , f2 ⁽¹⁾ )
Is well within the "A" region of Mr. A, but the formant frequency (f1 ⁽²⁾ ,
f2 ⁽²⁾ ) does not fit into any of the areas of Mr. A and Mr. B. In this way, by separating only voiced sound data and examining the formant frequency, the speaker can be identified more accurately than in the case where voice data other than voiced sound is mixed.

The second and third speaker identification methods, that is, the method using the formant frequencies at a plurality of time points as the identification parameters and the method for separating the voiced sound data,
You may change the order. That is, when the speaker cannot be specified by the first speaker specifying method, that is, the method using the formant frequency at a single time point as the specifying parameter, the third method is used when the speaker cannot be specified by the second method.
The method may be performed in the order of using the method, or the third method may be used in the order of using the second method when the method cannot be specified.

Thus, the time segment in FIG.
The mixed voice data of [1] was separated, and the speaker of each separated data could be specified. If similar processing is performed for time segments [2] and [3] and thereafter, all continuous mixed voice data can be separated and specified as voices for each speaker.

== Variations of the Invention and Specific Applications == In the above embodiment, the number of participants in the conference was two for the sake of convenience in drawing the graph, but in the case of three or more participants. Even if there is, the voice can be separated in exactly the same way and the speaker can be specified.

Further, as another embodiment of the present invention, regarding the processing method of the above-mentioned independent component separation processing, when the obtained observation data is non-linearly mixed with a plurality of voices and various sounds from the surroundings, By executing the independent component separation process based on the non-linear mixture model, it is possible to obtain the same effect based on the same basic configuration as this embodiment.

As one of the specific applications of the present invention, the identified speaker and the speech of the speaker are associated with each other, converted into character data using various known voice recognition software, There is automatic minutes creation by outputting to a recording medium. It is easy to create minutes of a long-term meeting, and the speaker can be identified automatically and accurately.

In addition, the identification of the speaker of a mobile phone call under poor sound quality, the identification of the speaker in CTI (Computer Telephony Integrated),
It can be applied to various applications such as car navigation in a noisy car and voice input to a personal computer in a situation where a microphone cannot be installed in the mouth and identification of the inventor. Furthermore, information appliances, mobile terminals such as mobile phones and PDAs,
In addition, application to a voice input means such as a wearable computer (Wearable Computer) which can be worn and carried is also considered.

[0103]

According to the voice separation method and speaker identification method of the composite voice data of the present invention, the separation and the speaker identification of the mixed voice data in which the voice data of a plurality of speakers are mixed can be accurately and quickly performed. Can be done.

The present invention as described above can be applied to the production of conference minutes, which is simultaneous and automatic with the input of voice data, and various voice input interfaces in a real environment.

[Brief description of drawings]

FIG. 1 is a diagram showing a waveform of audio data (raw data) X input from a microphone 1.

FIG. 2 is a diagram showing a cycle of voice separation processing.

FIG. 3 is a flowchart showing a first voice separation cycle.

FIG. 4 is a flowchart showing a second voice separation cycle.

FIG. 5 is a flowchart showing a third voice separation cycle.

FIG. 6 is a flowchart of an example of decorrelation processing W1.

FIG. 7 is a graph of x1 digitized waveform diagram data.

FIG. 8 is a graph of x2 digitized waveform diagram data.

FIG. 9 is a scatter graph of x1 and x2 data, where the horizontal axis represents the intensity of x1 and the vertical axis represents the intensity of x2.

FIG. 10 is data f1 and f2 having no correlation with each other.
It is a graph of a scatter diagram of.

FIG. 11 is a graph of digitized waveform diagram data of y1 ′.

FIG. 12 is a graph of y2 ′ digitized waveform diagram data.

FIG. 13 shows the sizes of y1 ′ and y2 ′ on the horizontal axis,
It is a scatter diagram plotted on the vertical axis.

FIG. 14: Separation data y obtained in the voice separation step
FIG. 1 is a waveform chart of 1 ′ and a spectrum distribution chart by Fourier transform thereof.

FIG. 15: Separation data y obtained in the voice separation step
FIG. 2 is a waveform diagram of 2 ′ and a spectrum distribution diagram by its Fourier transform.

FIG. 16 is a conceptual diagram of matching processing between a formant frequency as a specific parameter and a formant frequency of separated audio data.

FIG. 17 is a spectrum distribution diagram in which certain separated audio data is divided into n sampling times.

FIG. 18A is a distribution diagram of matching processing by formant frequencies according to the first speaker identification method provided for comparison. FIG. 7B is a distribution diagram in which the first and second principal component scores Z1 and Z2 are two-dimensional coordinate axes according to the second speaker identification method.

FIG. 19 is a spectrum distribution diagram in which the separated data y1 ′ is separated into a voiced sound component A1 and other components A2.

20 is a first diagram of each of A1 and A2 in FIG.
And the second formant frequencies f1 ⁽¹⁾ , f2 ⁽¹⁾ ,
And f1 ⁽²⁾ and f2 ⁽²⁾ are graphs plotted on the first and second formant frequency planes.

FIG. 21 is a graph showing the relationship between the degree m and its cumulative contribution rate.

FIG. 22 is a graph showing the relationship between the degree m and the amount of change in the cumulative contribution rate.

FIG. 23 is a flowchart showing a procedure for determining the order m by a method according to the system.

─────────────────────────────────────────────────── ───

[Procedure amendment]

[Submission date] January 25, 2002 (2002.1.2
5)

[Procedure Amendment 1]

[Document name to be amended] Statement

[Correction target item name] Full text

[Correction method] Change

[Correction content]

[Document name] Statement

Title: Speech separation method for composite voice data, speaker identification method, voice separation apparatus for composite voice data, speaker identification apparatus, computer program, and recording medium

[Claims]

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method of separating voices of composite voice data of a plurality of speakers, a method of specifying a speaker of each of the separated voice data, and a voice of composite voice data of a plurality of speakers. Device, a computer program, and a recording medium that specify a speaker of each separated audio data.

[0002]

2. Description of the Related Art A technique for accurately separating composite voice data in a voice recording medium, in which voices of a plurality of speakers are mixed and recorded, for each speaker is desired. Specifically, there is a need for a technology that can automatically create minutes of a conference by separating and identifying composite voice data for each speaker simultaneously with the input of voice. .

Conventionally, in order to create a minutes of a meeting over a long period of time, a person in charge of minutes recording re-listens and summarizes the audio data of the meetings recorded in various audio recording devices. I was creating. This work needs to be performed while repeating the reproduction and the temporary stop of the voice recording device, which is troublesome and time-consuming.

[0004] Another problem is that it is difficult to identify the speaker. If the person who attended the meeting was the person who attended the meeting, it was very difficult for the person who did not attend the meeting to make the minutes, and it was very difficult to determine which voice was attributed by which speaker.

Conventionally, although there are several techniques relating to voice separation from mixed voice data and speaker identification, even if voices and noises of a plurality of persons are mixed and input to one microphone, the separation is performed. Accurate identification and further high-speed separation / identification processing simultaneously with the input of complex speech is effective for segmentation of temporally continuous phoneme data.
And it was a very difficult task in terms of articulation.

[0006] Japanese Patent Laid-Open No. 2001-27895 describes a signal separation method for separating acoustic signals from a plurality of signal sources and combining and outputting a desired signal. This invention
Time / frequency analysis is performed on the mixed voice / acoustic signal to be analyzed to obtain a harmonic composition of frequency components. Among the overtone frequency components, whether at least one of the rise time and the fall time is common is used to identify whether or not the frequency components are from the same signal source. A signal from a single signal source is separated by extracting and reconstructing the frequency component.

Since the present invention does not consider the correlation and independence of mixed signals, it is difficult to separate mixed signals belonging to the same frequency band or mixed signals existing in the same time band. is there.

Further, in the sound source signal estimating device described in Japanese Patent Laid-Open No. 2000-97758, when a plurality of acoustic signals are mixed and input through a plurality of channels,
Based on a mixing process model in which each sound source signal is subjected to inner product calculation with the mixing coefficient vector and added to other sound source signals, the separation coefficient vector corresponding to the mixing coefficient vector is obtained by sequentially correcting, and the sound source is calculated using this separation coefficient vector. In estimating and separating signals (the ICA method), each signal is estimated from a signal in which a speech signal for normalizing a correction vector used for successive correction of a separation coefficient vector and a signal other than that are mixed together. When separating, it is said that it is possible to reduce the influence on the estimation and separation due to the fluctuation of each signal power, and further to increase the convergence coefficient, which enables stable and high-speed signal separation. .

Since the present invention is performed while sequentially correcting the separation coefficient vector based on the independent component analysis (ICA), it is possible to reduce the influence of fluctuations in signal power and realize high-speed separation. The sound source signals from the sources do not always maintain independence from each other. In general, even source signals from independent signal sources often have correlation when mixed, but that point is not taken into consideration.

Japanese Unexamined Patent Publication No. 9-258788 discloses a voice separation method for properly separating and separating mixed voices having fundamental frequencies close to each other so as to obtain high-quality separated voices regardless of the number of sound sources. And the device is described. In this invention, the voiced sound portion of the voice signal included in the input acoustic signal and the unvoiced sound portion are individually extracted while taking into consideration the information of the sound source direction of the voiced sound, and the extracted voiced sound portion is extracted. Divided into multiple voiced sounds and extracted as a group of voiced sounds,
The unvoiced sound part of the voice signal is extracted as a component of the acoustic signal corresponding to the unvoiced sound of each voiced sound group from the residuals extracted by subtracting the voiced sound part from the input acoustic signal, and then extracted into each separately extracted voiced sound group. The above object is realized by supplementing unvoiced sound and extracting a voice signal.

The present invention has a sound source localization section for extracting information on the direction of the sound source, but if different voices are emitted from the same direction, separation becomes difficult. Also, when multiple speakers make the same vowel or voiced sound, it seems difficult to separate them.

[0012]

SUMMARY OF THE INVENTION In order to solve the various problems of the prior art as described above, the present invention provides mixed voice data in which voice data of a plurality of speakers are mixed, with voices for each speaker. It is a principal object of the present invention to provide a method and apparatus for separating into two parts, and a method and device capable of accurately and speedily specifying the speaker of each separated audio data.

[0013]

In order to solve the above-mentioned problems, a first invention of the present application is to provide mixed voice data in which voice data of a plurality of speakers are mixed, to voice data for each speaker. In the audio data separation method for separating into (1)
Performing a decorrelation process for decorrelating the mixed speech data with each other; and (2) performing an independent component separation process for separating the decorrelated data into independent components. And if the separability of the data on which the independent component separation is performed is insufficient, until the separability is sufficient, the data on which the independent component separation processing is performed, the decorrelation process and In the speech separation method, the independent component separation process is repeatedly performed. This is a method for separating voice. According to the first aspect of the invention as described above, a plurality of voice data and mixed noise are considered in consideration of both the correlation and the independence of each voice data included in the input mixed voice data (raw data). Even if the correlation or independence of the above changes temporally or spatially, it is possible to accurately separate the voices of each speaker. In addition, according to the first aspect of the invention, the mixed voice data can be sufficiently separated into the voice data for each sound source.

A second invention according to the present application is, in the speech separation method according to the first invention, a non-Gaussian independent for separating non-Gaussian data into independent components as the independent component separation processing. Prepare component separation processing, non-stationary independent component separation processing for separating non-stationary data into independent components, and colored independent component separation processing for separating chromatic data into independent components, Depending on the nature of the data, any one of the non-Gaussian independent component separation process, the non-stationary independent component separation process, and the chromatic independent component separation process is performed. is there. According to the second aspect of the invention as described above, the optimum independent component separation process can be performed according to the property of the data that has been subjected to the decorrelation process. Can be separated into

The third invention according to the present application is the speech separation method according to the second invention, wherein the independent component separation processing performed first is a non-separation process for separating non-Gaussian data into independent components. A speech separation method characterized by a Gaussian independent component separation process. Since the non-Gaussian independent component separation processing is more susceptible to the decorrelation processing as its preprocessing than other independent component processing methods, according to such a third invention, the non-Gaussian independent processing is first performed. By performing the independent component separation processing, it becomes possible to effectively evaluate whether or not the decorrelation processing was successfully executed by the non-Gaussian independent component separation processing subsequent to the decorrelation processing.

Further, a fourth invention according to the present application is characterized in that, in the speech separation method according to the first to third inventions, the decorrelation processing performs at least a principal component analysis and a factor analysis. This is a voice separation method. According to the fourth invention, the number of principal component data to be adopted is obtained by obtaining the contribution ratio of each principal component and setting the number of components where the cumulative contribution ratio exceeds a predetermined threshold as the order. It is possible to effectively perform the decorrelation process after determining the (order).

Further, a fifth invention according to the present application is to separate mixed voice data in which voice data of a plurality of speakers are mixed into voice data for each speaker, and speak for the voice data for each speaker. In the speaker identification method for identifying the person,
(1) Separating mixed voice data in which voice data of a plurality of speakers are mixed into voice data for each speaker by the voice separation method according to any one of the first to fourth inventions,
(2) preparing a specific parameter for specifying the speaker for each speaker, and (3) specifying the speaker by referring to the specific parameter for the separated voice data for each speaker. The method for identifying a speaker is characterized by the following steps. According to such a fifth invention, for example, mixed voice data containing voices and noises of a plurality of speakers, which is recorded in recorded data of a conference or the like, is separated for each sound source, and separated. By specifying the speaker of the voice data, the conference record data can be automatically created, for example.

A sixth invention according to the present application is the speaker specifying method according to the fifth invention, wherein the specifying parameter is a formant frequency when the speaker pronounces a vowel, and the separated parameters are separated. The speaker specifying method is characterized in that a formant frequency is obtained for voice data of each speaker, and the speaker is specified by referring to the formant frequency as the specific parameter with respect to the obtained formant frequency. According to the sixth aspect, it is possible to easily identify the speaker of each separated voice data by using the formant frequency which is a feature amount that can be extracted by a simple process such as Fourier transform.

A seventh invention according to the present application is the speaker identifying method according to the sixth invention, wherein the specific parameter is the first formant frequency and the second formant frequency when the speaker produces a vowel. The first formant frequency and the second formant frequency are obtained for the separated voice data for each speaker, and the first formant frequency as the specific parameter is obtained with respect to the obtained first formant frequency and the second formant frequency. And a second formant frequency to identify a speaker, which is a speaker identification method. According to the seventh aspect of the invention, the speaker can be identified using the two formant frequencies that are the first and second spectrum peaks, so that the speaker can be identified easily and more accurately. .

Further, an eighth invention according to the present application is the speaker identifying method according to any one of the fifth invention to the seventh invention, in which the voice data for each speaker separated is identified. When the speaker cannot be specified in the step of specifying the speaker by referring to the parameter, the formant frequencies at a plurality of time points are obtained from the voice data, and the specified parameter is determined with respect to the obtained formant frequencies at the plurality of time points. The speaker identification method is characterized by identifying the speaker by referring to the formant frequencies at a plurality of times. According to the eighth aspect of the invention, the speaker can be specified more accurately by considering the temporal variation of the formant frequency, which is a feature amount for specifying the speaker of a certain voice. You can

Further, a ninth invention according to the present application is that, from mixed voice data in which voice data of a plurality of speakers are mixed,
A minutes creating method for creating minutes; a step of specifying a speaker in the voice data separated for each speaker by the speaker specifying method according to any one of the fifth invention to the eighth invention; The minutes creating method is characterized by comprising the step of creating the minutes by outputting the specified speaker and the statement of the speaker in association with each other on the recording medium. According to the ninth aspect of the invention, since the speaker is automatically and accurately identified, the minutes of the conference over a long time can be automatically created, which is convenient.

The tenth invention of the present application is to provide mixed voice data in which voice data of a plurality of speakers are mixed,
In a voice data separation device for separating the voice data for each speaker, a decorrelation process is performed in order to decorrelate the mixed voice data with each other, and the data subjected to the decorrelation process is separated into independent components. Independent component separation processing is performed in order to separate the independent component data, and if the separability of the data is insufficient, the independent component separation processing is performed until the separability becomes sufficient. It is a voice separation device characterized in that the decorrelation process and the independent component separation process are repeatedly performed. According to the tenth aspect, a plurality of audio data and mixed noise are considered in consideration of both the correlation and independence of each audio data included in the input mixed audio data (raw data). It is possible to realize a voice separation device capable of accurately separating the voice of each speaker even if the correlation or independence of the above changes with time and space. In addition, such a first
According to the invention of No. 0, it is possible to realize a voice separation device capable of sufficiently separating mixed voice data into voice data for each sound source.

The eleventh invention of the present application is the first invention.
In the speech separation apparatus according to the invention of No. 0, depending on the nature of the data, as the independent component separation processing, the non-Gaussian independent component separation processing for separating the non-Gaussian data into independent components and the non-stationary data independent A voice separation device characterized by performing any one of a non-stationary independent component separation process for separating components and a chromatic independent component separation process for separating chromatic data into independent components. Is. According to such an eleventh invention, since optimum independent component separation processing can be performed according to the property of the data subjected to decorrelation processing, mixed speech data can be more effectively converted into speech data for each sound source. It is possible to realize a voice separation device that can be separated into two parts.

The twelfth invention of the present application is the first invention.
In the speech separation apparatus according to the first aspect of the invention, the first independent component separation process is a non-Gaussian independent component separation process for separating non-Gaussian data into independent components. Is. Since the non-Gaussian independent component separation processing is more susceptible to the decorrelation processing as its preprocessing than other independent component processing methods, according to such a twelfth invention, the non-Gaussian independent processing is first performed. By implementing the independent component separation processing, it is possible to realize a speech separation device capable of effectively evaluating whether or not the decorrelation processing has been successfully executed by the non-Gaussian independent component separation processing subsequent to the decorrelation processing. it can.

The thirteenth invention of the present application is the first invention.
The speech separation apparatus according to any one of claims 0 to 12, wherein the decorrelation processing performs at least a principal component analysis and a factor analysis. According to such a thirteenth invention, by calculating the contribution rate of each principal component and setting the number of components where the cumulative contribution rate exceeds a predetermined threshold as the order, the number of principal component data to be adopted It is possible to realize a voice separation device capable of effectively performing decorrelation processing after determining (order).

The fourteenth invention of the present application is to provide mixed voice data in which voice data of a plurality of speakers are mixed,
In a speaker identifying device for separating voice data for each speaker and specifying a speaker for the voice data for each speaker, the voice separating device according to any one of the tenth to thirteenth inventions, The mixed voice data in which the data is mixed is separated into the voice data for each speaker, and the separated voice data for each speaker is referred to a specific parameter for specifying the speaker for each speaker. The speaker identifying apparatus is characterized in that the speaker is identified by According to such a fourteenth invention, for example, mixed voice data including voices and noises of a plurality of speakers, which is recorded in recorded data of a conference, is separated for each sound source, and each separated. By specifying the speaker of the voice data, it is possible to realize a speaker specifying device capable of automatically creating conference record data.

The fifteenth invention of the present application is the first invention.
In the speaker identifying device according to the fourth aspect of the present invention, the specific parameter is a formant frequency when the speaker produces a vowel, and the formant frequency is calculated for each of the separated voice data of each speaker. Regarding the formant frequency, the speaker specifying device is characterized in that the speaker is specified by referring to the formant frequency as the specifying parameter. According to such a fifteenth aspect, by using the formant frequency, which is a feature amount that can be extracted by a simple process such as Fourier transform, it is possible to easily specify the speaker of each separated voice data. The person identification device can be realized.

The sixteenth invention of the present application is the first invention.
In the speaker identifying device according to the invention of claim 5, the specific parameters are a first formant frequency and a second formant frequency when a speaker produces a vowel, and the specific parameter is the first formant frequency and the second formant frequency. The first formant frequency and the second formant frequency are obtained, and the first formant frequency and the second formant frequency as the specific parameters are determined with respect to the obtained first formant frequency and the second formant frequency.
It is a speaker specifying device characterized in that a speaker is specified by referring to a formant frequency. According to the sixteenth aspect, the first and second spectral peaks of 2
By specifying the speaker using one formant frequency, a speaker specifying device capable of specifying the speaker easily and more accurately can be realized.

The seventeenth invention of the present application is the first invention.
In the speaker identifying device according to any one of claims 4 to 16, when the speaker cannot be identified by referring to the specific parameter in the separated voice data for each speaker, A speaker characterized in that formant frequencies at a plurality of time points are obtained from the voice data, and the obtained formant frequencies at a plurality of time points are referred to by referring to the formant frequencies at a plurality of time points as the specific parameters. It is a specific device. Such a first
According to the invention of claim 7, it is possible to more accurately specify the speaker by considering the temporal variation of the formant frequency, which is a feature amount for specifying the speaker of a certain voice. The person identification device can be realized.

The eighteenth invention of the present application is the meeting minutes creating apparatus for creating a meeting minutes from mixed sound data in which sound data of a plurality of speakers are mixed.
Through the speaker identifying apparatus according to any one of the seventeenth invention,
It is possible to create a minutes by specifying a speaker in the separated voice data for each speaker and outputting the specified speaker and the statement of the speaker in association with each other on a recording medium. It is a characteristic minutes creating device. According to such an eighteenth invention, since the speaker is automatically and accurately specified, the minutes preparation apparatus capable of automatically creating the minutes of the conference for a long time can be realized.

A computer program for causing a voice separation device to execute the voice separation method according to any one of the first to fourth inventions can also be realized.

A computer program for causing the speaker identifying apparatus to execute the speaker identifying method according to any one of the fifth to eighth inventions can be realized.

A computer-readable recording medium recording such a computer program can also be realized.

[0034]

BEST MODE FOR CARRYING OUT THE INVENTION == Voice Separation of Mixed Voice Data =
= Hereinafter, more specific embodiments of the present invention will be described in detail with reference to the drawings. First, the voice separation step of mixed voice data, which is the first half of the method of the present invention, will be described.

In this embodiment, two microphones (microphone 1 and microphone 2) pick up voice data of the contents of a statement made by a two-person conference. FIG. 1 shows a waveform of audio data (raw data) X input from the microphone 1. The mixed voice data may include not only voice data of a plurality of speakers but also music and noise. The two utterances will be referred to as sound sources S1 and S2, respectively.

FIG. 2 is a diagram showing a cycle of voice separation processing. The mixed voice data input from the microphone 1 and the microphone 2 is first subjected to the decorrelation processing W1. The audio data passed to the decorrelation processing W1 is [1] and [2] in FIG.
It is segmented like this and passed one by one. For maximum efficiency, the segments overlap each other by ½ cycle.

In FIG. 2, the IC tuner, which is the next step of the decorrelation processing W1, is an independent component analysis (IC
This is a tuner for selecting the method A) from three types. Independent component separation process W which is the next step
2 performs one of three types of separation processing W2 (α) based on non-Gaussianity, separation processing W2 (β) based on non-stationarity, and separation processing W (γ) based on chromaticity. . The evaluator E in the step after W2 evaluates the separability of the data separated in W2. Until the voice separation performance of mixed voice data input from the microphone becomes sufficient,
The above cycle of W1 → IC tuner → W2 → E is repeated. However, in the first cycle, as the independent component separation processing W2, the independent component separation processing W2 (α) based on the non-Gaussian property is performed, and in the second and subsequent cycles, W2 (α), W2
An appropriate component independent component separation process is performed from among the three types of (β) and W2 (γ).

FIG. 3 shows the first voice separation cycle. Mixed voice data x1 and x2 from the microphone 1 and the microphone 2 in the time segment [1] in FIG.
Is first input to the decorrelation processing W1.

FIGS. 7 and 8 show digitized waveform diagram data of x1 and x2, respectively (the vertical axis represents the sound intensity and the unit is millivolts). FIG. 9 is a scatter plot of the x1 and x2 data at each time point with the abscissa representing the intensity of x1 and the ordinate representing the intensity of x2. The scatter plot exhibits a slightly linear distribution from the first quadrant to the third quadrant, indicating that the data of x1 and x2 are correlated with each other. When these raw data x1 and x2 are subjected to the decorrelation processing W1, they are converted into data f1 and f2 having no correlation with each other.

A scatter diagram of f1 and f2 is shown in FIG. The horizontal axis of FIG. 10 represents the first factor f1 of the factor score F, and the vertical axis represents the second factor f2 of the factor score F. 9 is distributed in a parallelogram shape that is distorted with respect to the axis, it is distributed in a rhombus shape that is straight and has a regular shape with respect to the axis.
It can be seen that and f2 are no longer correlated with each other.

Here, the contents of the decorrelation processing will be described. FIG. 6 shows a flowchart of an example of the decorrelation process W1. First, the raw audio data x1 and x2 shown in FIGS. 7 and 8 are standardized by the equation (1). As a result of standardization, the data has an average of 0 and a standard deviation of 1.

[Equation 1]

The correlation matrix (vector C) of the raw data x1 and x2 is obtained from the equation (2). In formula (2), (x1,
x2) represents the dot product of the vectors.

[Equation 2]

The eigenvalue λi and the eigenvector A for the above correlation matrix are obtained from (3).

[Equation 3]

Now, by factor analysis, we are trying to obtain mutually uncorrelated factor scores, but at that time, it is important to start with the first factor and up to what factor. When adopting up to the mth factor,
Call it m-dimensional. By the eigenvector A obtained earlier,
The principal component Z is obtained by the equation (4).

[Equation 4]

Next, a factor analysis is performed on the m factors by the definition equation of the form (5). E in equation (5)
Is called a special factor.

[Equation 5]

This factor model takes the expression (6).
The factor load bij and the factor score F in the equation (6) are
It is determined by the equations (7) and (8). Then, in the final step of the flowchart of FIG. 6, the raw audio data is eventually converted into factor scores (vector F) that are uncorrelated with each other.

[0047]

[Equation 6]

[Equation 7]

[Equation 8]

The main feature of W1 described above is that principal component analysis and factor analysis are combined. The effect is that the contribution ratio of each principal component can be obtained at the same time by executing the principal component analysis. Therefore, for example, the main contribution until the cumulative contribution ratio from the first-order principal component to the m-th-order principal component exceeds 80%. By adopting the component, the order m is determined. The raw audio data to be separated has a large temporal variation, and the degree of correlation due to mixing greatly changes.
How many factors are adopted is an important point in decorrelation processing.

When the number of speakers is known in advance, the order m may be fixed to the number of speakers. When the number of speakers is unknown, for example, the cumulative contribution ratio is a predetermined threshold. Let m be the number of principal components when the value is exceeded. As a method of determining the order m, it is preferable to prepare various methods according to the system and to change (tune) flexibly. Next, an example of this tuning will be described in detail.

FIG. 21 shows the order m according to the system.
It is a flow chart which shows the procedure which determines. Figure 21
Here, RK0 is the initial threshold value of the cumulative contribution rate, M is the maximum order (upper threshold value of the order) that can be adopted, and ΔRK is the change amount of the cumulative contribution rate. When the principal component analysis is executed, a graph showing the relationship between the degree m (indicating that the m-th principal component is adopted) and its cumulative contribution rate is obtained as shown in FIG. FIG. 19 shows an example of three types of graphs of A, B, and C.

First, as a first processing step, a threshold value RK0 (80% in this embodiment) is set in the cumulative contribution rate RK, and an order m exceeding this threshold value RK0 is obtained. However, if the order is too large, the subsequent processing becomes too complicated. Therefore, the upper limit M of the order is determined in advance. In the example of FIG. 19, assuming that M = 4, in the case of A, the degree m that exceeds the threshold RK0 is m = 2, so m = 2
<4 = M, and the order m is determined to be 2. In the example of B, the degree m exceeding RK0 is 5, so that m = 5> 4 =
It becomes M, and the order m is not determined yet. Similarly, in the case of C, the order m is not determined.

In such a case, the second step shown in FIG. 20 is executed. That is, for an increase in the order m,
Check the difference change amount ΔRK of RK. In short, this is a processing method in which the order m that maximizes the change in the cumulative contribution rate is the order to be adopted. In this embodiment, B
In the example, ΔRK takes the maximum value when m = 2 and in the example C, m = 4. Also in this case, if the order m is lower than the upper limit value M, the order m is adopted, but if it is higher than M, the processing is sent to the next step.

If the order m exceeds the upper limit M even in the second step, the cumulative contribution ratio threshold R
K0 is lowered to, for example, 60% (= RK1), and the comparison is performed in the same manner as the first step. If the order above the new threshold value RK1 is M = 4 or less,
This is adopted as the order m, and when it exceeds M, the values of RK2, RK3, ... RKn are sequentially decreased with a predetermined decrease amount. However, if the cumulative contribution rate RK is less than 50%, it means that half or more of the information is lost, so the lower limit value of RKn is set to 50%.

When the order m is RKn = 50% or more and is not found with a value less than or equal to M, the same process as in the second step above is performed again, that is, the order in which ΔRK is maximized is obtained and the value is set. It will be adopted as the order m. This is based on the idea that since the cumulative contribution ratio changes significantly, more information is stored before and after the order, and therefore it is desirable to adopt at least up to that order.

As described above, in FIG. 3, the decorrelated data f1 and f2 are immediately sent to the independent component separation processing W2. In the first speech separation cycle, the independent component separation process W2 (α) based on non-Gaussianity is executed on these decorrelated data f1 and f2.

As described above, the separation signals a and b are obtained by the processing of W1 and W2 (α) in FIG. 3, and their separation characteristics (whether or not they are sufficiently separated) are evaluated by the evaluator E and separated. When the value is insufficient (* 1 in the figure), the second cycle is executed for the data of a and b.

An example of the second cycle is shown in FIG. Figure 3
Similar to the first cycle shown in, but with the addition of processing in the IC tuner. Independent component separation process W2
Before performing, the signal characteristics of the data f1 ′ and f2 ′ subjected to the second decorrelation processing by the IC tuner are analyzed, and the processing W2 (α) based on non-Gaussianity and the processing W2 based on non-stationarity are analyzed.
Either (β) or the process W2 (γ) based on chromaticity is selected as W2. In this example, W2 (β) is executed. The separability of the data y1 and y2 after the processing W2 (β) is evaluated by the evaluator E, and when it is insufficient (* 2 in FIG. 4), the third cycle is executed.

Here, the function of the IC tuner will be described. The IC tuner evaluates the Gaussianity, stationarity, and chromaticity of the decorrelation-processed input data as follows, and selects the optimum independent component separation process from the three types.

First, the IC tuner evaluates the Gaussian property of two input data. Specifically, for each input data, it is checked whether the frequency distribution of the input time series data is the Gaussian function (normal distribution function) type or the non-Gaussian function type.
If the input data is gs and the Gaussian function is g0, the absolute value of the difference between the two, that is, | gs-g0 |, integrated in the interval Δg is a non-Gaussian type if the value Δg is larger than a predetermined threshold δg. If it is small, it is evaluated as Gaussian. If none of the input data subjected to the decorrelation process is non-Gaussian type, the IC tuner selects the process W2 (α) based on non-Gaussian property as the independent component separation process W2.

If any of the decorrelated input data is evaluated as Gaussian, then the IC tuner evaluates the stationarity of the two input data. In this evaluation, a set average of a plurality of irregular waveforms is taken, and attention is paid to the time change of the set average. If the collective average is constant with respect to the time axis, it is “completely stationary”. If it fluctuates with time, the probability density distribution in a certain time width is obtained, and the nonstationarity is quantified from the variance, skewness, and kurtosis. The strength of non-stationarity is the magnitude of variance, the magnitude of skewness,
Since the influence of the degree of kurtosis is strongly influenced, it is preferable to evaluate after weighting according to the strength. When any of the input data subjected to the decorrelation is evaluated to have non-stationarity, the IC tuner selects the non-stationarity-based process W2 (β) as the independent component separation process W2.

If any of the decorrelated input data is evaluated as having stationarity, then the IC tuner evaluates the chromaticity of the two input data. To evaluate chromaticity, an autocorrelation function of irregular waveform is obtained. A graph of the autocorrelation function with respect to the magnitude of the time lag τ is obtained, and how much the center of gravity of the graph deviates from the origin (τ = 0) is checked. When the position of the center of gravity deviates from the origin (τ = 0) by a predetermined value or more, it is evaluated as having color. In the case of white noise,
The autocorrelation function has a value only at τ = 0. When all of the input data subjected to the decorrelation are evaluated to have chromaticity, the IC tuner selects the chromaticity-based process W2 (γ) as the independent component separation process W2.

FIG. 5 shows the third cycle. Each process is similar to the second cycle, but in the third independent component separation process, W2 (γ) based on the chromaticity is executed in this example.

Here, the above-described three types of independent separation processing W2
The contents of (α), W2 (β), and W2 (γ) will be described in more detail. First, the signal source estimation procedure by the independent component separation processing W2 (α) based on non-Gaussianity
First, the separation coefficient (matrix) Wt is appropriately assumed (the initial value is W0).

Next, the signal source y (t) for the data F (t) after the decorrelation processing is estimated as in the equation (9).

[Equation 9] Using this y (t) and Wt, ΔWt is obtained from the equation shown in equation (10).

[Equation 10]

From equation (11), Wt + 1 in the next convergence calculation step is obtained. The above steps are repeated with this Wt + 1 as a new Wt. Then, y (t) at the time when ΔWt becomes almost zero, that is, when Wt is considered to have sufficiently converged is the mixed voice raw data x (t).
It becomes an estimated signal of the signal source s (t) obtained from

[Equation 11]

Secondly, the signal source estimation procedure by the independent component separation process W2 (β) based on non-stationarity is as follows. First, y2 in the time of the order of the separation coefficient (matrix) Ct and the system time constant T '. The initial value of the moving average Φ of (t) is calculated. Further, y (t) is calculated by the equation (12). In Expression (12), I is an identity matrix.

[Equation 12] Next, the differential equation shown in the equation (12) is solved to obtain Φ. In the equation (13), T'is a time constant of the system.

[Equation 13] Next, from Φ, Ct, and y (t) in the equation (12), (1
A new Ct + 1 is obtained by using the differential equation shown in equation 4). In equation (14), T is the time constant of the system.

[Equation 14] The obtained Ct + 1 and the data F (t) after the decorrelation process
From this, y (t) in the next step is estimated using the equation (15).

[Equation 15] The above steps are repeated using this y (t) and Ct + 1. Then, y (t) at the time when Ct is considered to have sufficiently converged becomes an estimated signal of the signal source s (t) obtained from the mixed voice raw data x (t).

Third, regarding the signal source estimation procedure by the independent component separation processing W2 (γ) based on chromaticity, first, the separation coefficient matrix Ct and initial values of Ψ1 and Ψ2 are given. here,
Ψ1 and Ψ2 are time averages of two products (y1 * y1T) and (y2 * y2T) made from y (t) obtained by applying two types of linear filters y1 (t) and y2 (t). Is. In addition, y (t) is the data F after decorrelation processing.
Estimate from (t) using equation (16).

[Equation 16] By applying two types of linear filters G1 and G2 to this y (t), y1 (t) and y2 (t) are obtained by the equation (17).

[Equation 17]

Initial values of Ψ1, Ψ2 and y1, y
2 and Ψ1 and Ψ2 are newly obtained by using the differential equation shown in Expression (18).

[Equation 18] From Ct, Ψ1, Ψ2, a new Ct + 1 is obtained by the equation (19).

[0069]

[Formula 19] From this Ct + 1 and the data F (t), the above (16)
A new y (t) is calculated by the formula. Then, this change in Ct, that is, the change in y (t) becomes sufficiently small, and y (t) at the time when it is considered that the convergence has occurred is the signal source s (t) obtained from the mixed voice raw data x (t). )
It becomes the estimated signal of. If it has not converged, (1
Y1 (t) and y2 (t) are calculated by the equation 7), and the above steps are repeated.

Returning to FIG. 5, the evaluator E has determined that the output data y1 ', y2' of the third separation cycle have sufficient separability here. That is, y1 ',
It is considered that y2 ′ corresponds to the voice of either the sound source S1 or S2, respectively. Digitized waveform diagrams of these data are shown in FIGS. 11 and 12. When the points whose amplitude is below a certain level are analyzed as noise instead of utterance, y1 ′
The voice data of "a" (-) and "ka" (-) can be seen in. Similarly, voice data of "shi" (-) is seen in y2 '.

FIG. 13 is a scatter diagram in which the sizes of y1 'and y2' are plotted on the horizontal axis and the vertical axis, respectively. As can be seen from this figure, the values of ,,,,, have y2 ′ values of almost zero, and conversely, the points of ,, have y1 ′ values of almost zero. It can be seen that the sound from was separated exactly.

In the evaluator E, in order to evaluate the separability of the data after executing the process W2, points such as in the graph of FIG. 13 may be examined. That is, in the scatter diagram, a point that is most distant from the horizontal axis or the vertical axis is selected, and if the distance to the axis is a certain value or more, the separability is still insufficient, and again, in FIG. 4 and FIG. Such a separation cycle is executed.

== Identification of Speaker of Separated Voice Data == Next, the step of identifying the speaker of each separated voice data, which is the latter half of the present invention, will be described. Figure 1
4 is the separated data y1 obtained in the voice separation step.
2A and 2B are a waveform diagram of 'and a spectrum distribution diagram by its Fourier transform. Here, as the method of obtaining the spectral distribution,
A filter bank, LPC method, or the like can be used.

Similarly, FIG. 15 is a waveform diagram of the separated data y2 'and a spectrum distribution diagram by its Fourier transform. In this embodiment, since two speakers (A and B) are assumed to be speakers, these two waveform data y
It was separated into 1'and y2 ', but which is A at this point?
With Mr.'s voice, I do not know which is Mr. B's voice. We will identify it from now on.

First, as a first method for specifying a speaker, a method of using a formant frequency as a speaker specifying parameter is executed. Fo1 and f in FIG.
o2 is the first formant frequency and the second formant frequency of the y1 ′ data, and go1 and go2 in FIG.
Are the first formant frequency and the second formant frequency of the y2 ′ data. Conference participants A and B in advance
The first and second formant frequency data of Mr. is prepared in the database as specific parameters for speaker identification. Then, the speaker of each separated voice data is specified by referring to the formant frequencies of the separated data y1 'and y2'.

FIG. 16 is a process for matching the formant frequencies of all five vowels of Mr. A and Mr. B, which are specific parameters, with the first and second formant frequencies of the obtained separated speech data y1 'and y2'. It is a conceptual diagram of.
The horizontal axis represents the first formant frequency, and the vertical axis represents the second formant frequency. First, the spread area of the formant frequency of the pronunciation of Mr. A's vowel (the area surrounded by the solid line in the figure) and the spread area of the formant frequency of the pronunciation of Mr. B's vowel (the area surrounded by the dotted line in the figure) are shown. On top of FIG. 14 and FIG.
The formant frequencies of the separated speech data of No. 5 are plotted.

The formant frequencies of y1 'and y2' are
If it falls within the formant frequency range of Mr. A or Mr. B, it can be said that the speaker can be identified by this. However, if it does not fit into any of the formant frequency regions of Mr. A and Mr. B (portion C in FIG. 16), or if it falls within the overlapping portion D of the regions of Mr. A and Mr. B, this first Since the speaker cannot be specified by the method, the second specifying method described below is executed.

The second speaker identification method is a method of using formant frequencies at a plurality of time points as speaker identification parameters. FIG. 17 is a diagram showing that certain voice data (“a” voice) separated by the voice separating step in the first half of the present invention is divided into n sampling times and subjected to Fourier transform and spectral decomposition. is there. The first and second peakant frequencies (f11, f12, ... F1n) and the second formant frequencies (f21, f22 ,.

Next, principal component analysis is performed on these formant frequency data to obtain principal component scores Z1, Z2 ,.
..Zn is obtained and used as the feature amount of the voice of the speaker. Therefore, as speaker identification parameters prepared in advance in the database, the principal component scores Z1, Z2, ... Of various voices (all vowels, etc.) of conference participants are prepared.

FIG. 18 is a graph showing the result of the second speaker identification method. FIG. 18A shows a result obtained by the first speaker identification method provided for comparison. FIG.
In (a), the first and second formant frequencies at the five sampling times are plotted for the sound of "A", but it cannot be said to belong to the region of Mr. A or the region of Mr. B. .

On the other hand, FIG. 18B is a distribution chart in which the first and second principal component scores Z1 and Z2 are two-dimensional coordinate axes by the second method. First, as in the example of this figure, the area of Mr. A (solid line in the figure) and the area of Mr. B (dotted line in the figure) are often clearly separated on this principal component score plane, so judgment is easy. is there. When the principal component scores (Z1, Z2) of the separated data obtained from the results of FIG. 17 are plotted, it is clearly close to the area of Mr. A, so “a” in this case
It is understood that is the pronunciation of Mr. A.

As described above, the time segment in FIG.
The mixed voice data of [1] was separated, and the speaker of each separated data could be specified. If similar processing is performed for time segments [2] and [3] and thereafter, all continuous mixed voice data can be separated and specified as voices for each speaker.

== Variations and Specific Applications of the Invention == In the above embodiment, the number of participants in the conference was two for the sake of convenience in drawing a graph, but in the case of three or more participants. Even if there is, the voice can be separated in exactly the same way and the speaker can be specified.

As one of the specific applications of the present invention, the specified speaker and the speech of the speaker are associated with each other, converted into character data using various known voice recognition software, There is automatic minutes creation by outputting to a recording medium. It is easy to create minutes of a long-term meeting, and the speaker can be identified automatically and accurately.

In addition, the identification of the speaker of a mobile phone call under poor sound quality, the identification of the speaker in CTI (Computer Telephony Integrated),
It can be applied to various applications such as car navigation in a noisy car and voice input to a personal computer in a situation where a microphone cannot be installed in the mouth and identification of the inventor. Furthermore, information appliances, mobile terminals such as mobile phones and PDAs,
In addition, application to a voice input means such as a wearable computer (Wearable Computer) which can be worn and carried is also considered.

[0086]

According to the voice separation method and speaker identification method of the composite voice data of the present invention, the separation and the speaker identification of the mixed voice data in which the voice data of a plurality of speakers are mixed can be accurately and quickly performed. Can be done.

The present invention as described above can be applied to the conference minutes preparation which is simultaneous and automatic with voice data input, and various voice input interfaces in a real environment.

[Brief description of drawings]

FIG. 1 is a diagram showing a waveform of audio data (raw data) X input from a microphone 1.

FIG. 2 is a diagram showing a cycle of voice separation processing.

FIG. 3 is a flowchart showing a first voice separation cycle.

FIG. 4 is a flowchart showing a second voice separation cycle.

FIG. 5 is a flowchart showing a third voice separation cycle.

FIG. 6 is a flowchart of an example of decorrelation processing W1.

FIG. 7 is a graph of x1 digitized waveform diagram data.

FIG. 8 is a graph of x2 digitized waveform diagram data.

FIG. 9 is a scatter graph of x1 and x2 data, where the horizontal axis represents the intensity of x1 and the vertical axis represents the intensity of x2.

FIG. 10 is data f1 and f2 having no correlation with each other.
It is a graph of a scatter diagram of.

FIG. 11 is a graph of digitized waveform diagram data of y1 ′.

FIG. 12 is a graph of y2 ′ digitized waveform diagram data.

FIG. 13 shows the sizes of y1 ′ and y2 ′ on the horizontal axis,
It is a scatter diagram plotted on the vertical axis.

FIG. 14: Separation data y obtained in the voice separation step
FIG. 1 is a waveform chart of 1 ′ and a spectrum distribution chart by Fourier transform thereof.

FIG. 15: Separation data y obtained in the voice separation step
FIG. 2 is a waveform diagram of 2 ′ and a spectrum distribution diagram by its Fourier transform.

FIG. 16 is a conceptual diagram of matching processing between a formant frequency as a specific parameter and a formant frequency of separated audio data.

FIG. 17 is a spectrum distribution diagram in which certain separated audio data is divided into n sampling times.

FIG. 18A is a distribution diagram of matching processing by formant frequencies according to the first speaker identification method provided for comparison. FIG. 7B is a distribution diagram in which the first and second principal component scores Z1 and Z2 are two-dimensional coordinate axes according to the second speaker identification method.

FIG. 19 is a graph showing the relationship between the degree m and its cumulative contribution rate.

FIG. 20 is a graph showing the relationship between the degree m and the amount of change in the cumulative contribution rate.

FIG. 21 is a flowchart showing a procedure for determining the order m by a method according to the system.

[Procedure Amendment 2]

[Document name to be corrected] Drawing

[Name of item to be corrected] Figure 3

[Correction method] Change

[Correction content]

[Figure 3]

[Procedure 3]

[Document name to be corrected] Drawing

[Name of item to be corrected] Figure 5

[Correction method] Change

[Correction content]

[Figure 5]

[Procedure amendment 4]

[Document name to be corrected] Drawing

[Name of item to be corrected] Fig. 19

[Correction method] Change

[Correction content]

FIG. 19

[Procedure Amendment 5]

[Document name to be corrected] Drawing

[Name of item to be corrected] Fig. 20

[Correction method] Change

[Correction content]

FIG. 20

[Procedure correction 6]

[Document name to be corrected] Drawing

[Name of item to be corrected] Fig. 21

[Correction method] Change

[Correction content]

FIG. 21

[Procedure Amendment 7]

[Document name to be corrected] Drawing

[Correction target item name] Fig. 22

[Correction method] Delete

[Procedure Amendment 8]

[Document name to be corrected] Drawing

[Correction target item name] Fig. 23

[Correction method] Delete ───────────────────────────────────────────── ────────

[Procedure amendment]

[Submission date] September 9, 2002 (2002.9.9)

[Procedure Amendment 1]

[Document name to be amended] Statement

[Correction target item name] 0013

[Correction method] Change

[Correction content]

[0013]

In order to solve the above-mentioned problems, a first invention of the present application is to provide mixed voice data in which voice data of a plurality of speakers are mixed, to voice data for each speaker. In the audio data separation method for separating into (1)
Performing a decorrelation process for decorrelating the mixed speech data with each other; and (2) performing an independent component separation process for separating the decorrelated data into independent components. And if the separability of the data on which the independent component separation is performed is insufficient, until the separability is sufficient, the data on which the independent component separation processing is performed, the decorrelation process and In the speech separation method, the independent component separation process is repeatedly performed. According to the first aspect of the invention as described above, a plurality of voice data and mixed noise are considered in consideration of both the correlation and the independence of each voice data included in the input mixed voice data (raw data). Even if the correlation or independence of the above changes temporally or spatially, it is possible to accurately separate the voices of each speaker. In addition, according to the first aspect of the invention, the mixed voice data can be sufficiently separated into the voice data for each sound source.

[Procedure Amendment 2]

[Document name to be amended] Statement

[Correction target item name] 0064

[Correction method] Change

[Correction content]

Next, the signal source y (t) for the data F (t) after the decorrelation processing is estimated as in the equation (9).

[Equation 9] Using this y (t) and Wt, ΔWt is obtained from the equation shown in equation (10).

[Equation 10]

─────────────────────────────────────────────────── ─── Continued Front Page (51) Int.Cl. ⁷ Identification Code FI Theme Coat (Reference) G10L 9/08 301A 3/02 301E 301C

Claims (31)

[Claims] 1. A voice data separation method for separating mixed voice data in which voice data of a plurality of speakers are mixed into voice data for each speaker, (1) for making the mixed voice data mutually uncorrelated And a step (2) of performing independent component separation processing for separating the data subjected to the decorrelation processing into independent components. . 2. The audio separation method according to claim 1, wherein when the separability of the data on which the independent component separation is performed is insufficient, the independent component separation processing is performed until the separability becomes sufficient. A speech separation method, wherein the decorrelation processing and the independent component separation processing are repeatedly performed on the performed data. 3. The speech separation method according to claim 1, wherein the independent component separation process is a non-Gaussian independent component separation process for separating non-Gaussian data into independent components, and Non-stationary independent component separation processing for separating stationary data into independent components, and colored independent component separation processing for separating colored data into independent components are prepared. Non-Gaussian independent component separation processing,
A voice separation method, wherein any one of the non-stationary independent component separation process and the colored independent component separation process is performed. 4. The speech separation method according to claim 3, wherein the independent component separation process performed first is a non-Gaussian independent component separation process for separating non-Gaussian data into independent components. Characteristic voice separation method. 5. The speech separation method according to claim 1, wherein the decorrelation processing performs at least a principal component analysis and a factor analysis. 6. A speaker identifying method for separating mixed voice data in which voice data of a plurality of speakers are mixed into voice data for each speaker, and identifying a speaker for the voice data for each speaker. (1) Separating mixed voice data, in which voice data of a plurality of speakers are mixed, into voice data for each speaker by the voice separation method according to any one of claims 1 to 5. 2) preparing a specific parameter for specifying the speaker for each speaker, and (3) specifying the speaker by referring to the specific parameter for the separated voice data for each speaker. A method for identifying a speaker, comprising: 7. The speaker identification method according to claim 6, wherein the identification parameter is a formant frequency when the speaker produces a vowel, and the formant frequency is set for each of the separated voice data of each speaker. The speaker specifying method is characterized in that, with respect to the obtained formant frequency, the speaker is specified by referring to the formant frequency as the specifying parameter. 8. The speaker identification method according to claim 7, wherein the identification parameter is a first formant frequency and a second formant frequency when the speaker produces a vowel, and each of the separated speakers is separated. The first formant frequency and the second formant frequency are obtained for the audio data of, and the first formant frequency and the second formant frequency as the specific parameters are referred to with respect to the obtained first formant frequency and the second formant frequency, A speaker identification method characterized by identifying a speaker. 9. The speaker identifying method according to claim 6, further comprising the step of identifying the speaker by referring to the specific parameter for the separated voice data of each speaker. If the speaker cannot be identified by the above, the formant frequencies at a plurality of time points are obtained from the voice data, and the obtained formant frequencies at a plurality of time points are
A speaker specifying method characterized in that a speaker is specified by referring to formant frequencies at a plurality of times as the specifying parameter. 10. The speaker identifying method according to claim 6, further comprising the step of identifying a speaker by referring to the specific parameter for the separated voice data of each speaker. If the speaker cannot be specified by separating the voiced sound data from the voice data, a formant frequency is obtained for the voiced sound data, and the obtained formant frequency is referred to the formant frequency as the specific parameter. Then, a speaker specifying method characterized by specifying a speaker. 11. The speaker identifying method according to claim 10, wherein the speaker cannot be identified in the step of identifying the speaker with respect to the separated voice data of each speaker by referring to the specific parameter. In this case, the voiced sound data is separated from the voice data, the first formant frequency and the second formant frequency are obtained for the voiced sound data, and the obtained first formant frequency and the second formant frequency are obtained.
Regarding a formant frequency, a speaker specifying method characterized in that a speaker is specified by referring to a first formant frequency and a second formant frequency as the specifying parameter. 12. The speaker identifying method according to claim 10 or 11, wherein the speaker is identified in the step of identifying the speaker with respect to the separated voice data of each speaker by referring to the specific parameter. If it is not possible to specify, the formant frequencies at a plurality of time points are obtained for the voiced sound data, and the obtained formant frequencies at a plurality of time points are referred to by referring to the formant frequencies at a plurality of time points as the specific parameters. A speaker identification method characterized by identifying a speaker. 13. The speaker identifying method according to claim 10 or 11, wherein when separating the voiced sound data from the voice data,
A speaker identifying method, characterized in that an independent component separation process for separating the voice data into independent components is performed. 14. A minutes creating method for creating a minutes from mixed audio data in which audio data of a plurality of speakers are mixed, according to the speaker identifying method according to any one of claims 6 to 13. A step of identifying a speaker in the separated voice data for each speaker, and outputting the recorded minutes to the recording medium in association with the identified speaker and the statement of the speaker The method of creating a minutes, comprising: 15. A voice data separating device for separating mixed voice data in which voice data of a plurality of speakers are mixed into voice data for each speaker, in order to decorrelate the mixed voice data with each other A speech separation apparatus, characterized by performing an independence component separation process in order to separate the data subjected to the decorrelation process into independent components. 16. The speech separation apparatus according to claim 15, wherein when the separability of the data on which the independent component separation is performed is insufficient, the independent component separation processing is performed until the separability becomes sufficient. A speech separation apparatus, wherein the decorrelation processing and the independent component separation processing are repeatedly performed on the performed data. 17. The speech separation apparatus according to claim 15 or 16, wherein the non-Gaussian independent component for separating non-Gaussian data into independent components is used as the independent component separation processing depending on the nature of the data. Any one of separation processing, non-stationary independent component separation processing for separating non-stationary data into independent components, and chromatic independent component separation processing for separating chromatic data into independent components A voice separation device characterized by performing. 18. The speech separation apparatus according to claim 17, wherein the independent component separation process performed first is a non-Gaussian independent component separation process for separating non-Gaussian data into independent components. Characteristic audio separation device. 19. The speech separation apparatus according to claim 15, wherein the decorrelation processing performs at least a principal component analysis and a factor analysis. 20. Separating mixed voice data in which voice data of a plurality of speakers are mixed into voice data for each speaker,
A speaker identification device for identifying a speaker for audio data of each speaker, wherein the audio separation device according to any one of claims 15 to 19 mixes audio data of a plurality of speakers. The data is separated into voice data for each speaker, and the speaker is specified by referring to the separated voice data for each speaker by referring to a specific parameter for specifying the speaker for each speaker. Characterized speaker identification device. 21. The speaker identifying apparatus according to claim 20, wherein the specific parameter is a formant frequency when a speaker produces a vowel, and the formant frequency is set for each of the separated voice data of each speaker. And a speaker specifying device that specifies the speaker with reference to the formant frequency as the specifying parameter with respect to the obtained formant frequency. 22. The speaker identifying apparatus according to claim 21, wherein the specific parameter is a first formant frequency and a second formant frequency when a speaker produces a vowel, and each of the separated speakers is separated. The first formant frequency and the second formant frequency are obtained for the audio data of, and the first formant frequency and the second formant frequency as the specific parameters are referred to with respect to the obtained first formant frequency and the second formant frequency, A speaker specifying device characterized by specifying a speaker. 23. The speaker identifying apparatus according to claim 20, wherein the speaker cannot be specified by referring to the specific parameter for the separated voice data of each speaker. In this case, the formant frequencies at a plurality of time points are obtained from the audio data, and the obtained formant frequencies at a plurality of time points are
A speaker identifying apparatus for identifying a speaker by referring to formant frequencies at a plurality of times as the specific parameters. 24. The speaker identifying apparatus according to claim 20, wherein the speaker cannot be specified by referring to the specific parameter for the separated voice data of each speaker. In this case, the voiced sound data is separated from the voice data, the formant frequency is obtained for the voiced sound data, and the speaker is identified by referring to the formant frequency as the specific parameter with respect to the obtained formant frequency. A speaker identification device characterized by the above. 25. The speaker identifying apparatus according to claim 24, wherein, with respect to the separated voice data for each speaker, when the speaker cannot be identified by referring to the specific parameter, the voice data is output. Voiced sound data is separated from the voiced sound data, a first formant frequency and a second formant frequency are obtained for the voiced sound data, and the obtained first formant frequency and second formant frequency are obtained.
Regarding a formant frequency, a speaker identifying device characterized in that a speaker is identified by referring to a first formant frequency and a second formant frequency as the specific parameters. 26. The speaker identifying apparatus according to claim 24 or 25, wherein the speaker cannot be specified by referring to the specific parameter for the separated voice data of each speaker. The voiced sound data, formant frequencies at a plurality of time points are obtained, and with respect to the obtained formant frequencies at a plurality of time points, the speaker is identified by referring to the formant frequencies at a plurality of time points as the specific parameters. Speaker identification device. 27. The speaker identifying apparatus according to claim 24 or 25, wherein when separating the voiced sound data from the voice data,
An apparatus for identifying a speaker, wherein an independent component separation process for separating the voice data into independent components is performed. 28. A minutes creating device for creating a minutes from mixed voice data in which voice data of a plurality of speakers are mixed, comprising: the speaker identifying device according to claim 20. Creating a minutes by specifying a speaker in the separated voice data for each speaker and outputting the specified speaker and the statement of the speaker in association with each other on a recording medium. A minutes preparation device. 29. A computer program for causing a voice separation device to execute the voice separation method according to claim 1. Description: 30. A computer program for causing a speaker identifying device to execute the speaker identifying method according to claim 6. 31. A computer-readable recording medium having the computer program according to claim 29 or 30 recorded therein.