JP7138824B2 - Sound source separation model learning device, sound source separation device, program, sound source separation model learning method, and sound source separation method - Google Patents

Description

The present disclosure relates to a sound source separation model learning device, a sound source separation device, a program, a sound source separation model learning method, and a sound source separation method.

In recent years, a technique based on a neural network (hereinafter referred to as NN) has been used as a technique for separating only a desired sound source signal from a mixed signal composed of a plurality of sound sources. In Non-Patent Document 1, sound source separation is achieved by passing a mixed signal in which a plurality of sounds are mixed through a sound source separation device using a neural network.

Z. Q. Wang et al. , Alternative Objective Functions for Deep Clustering, Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018

Like the conventional technique, in the NN-based sound source separation method, the input feature amount to the NN is generated from the acquired sound source signal and applied to the NN.

On the other hand, there are other signal processing processes such as separating a desired sound source and suppressing signals coming from unnecessary sound sources. For example, signal processing such as beam forming processing using a microphone array, spectral subtraction processing for suppressing noise, or adaptive filtering processing for noise canceling or the like.

In the learning stage of Non-Patent Document 1, it is not assumed that the above signal processing is performed between the acquisition of the sound source signal and the generation of the input feature amount. For this reason, even if a mixed signal that has undergone signal processing is input to the NN during sound source separation, the NN cannot cope with changes in acoustic characteristics that occur with the signal processing, and sufficient sound source separation performance cannot be obtained. can't Here, it is assumed that the variation of the acoustic characteristics is, for example, that the scale, delay, reverberation, frequency characteristics, etc. of the signal change.

Accordingly, it is an object of one or more aspects of the present disclosure to enable sound source separation by machine learning to function effectively even when acoustic characteristics fluctuate.

A sound source separation model learning device according to a first aspect of the present disclosure performs predetermined processing on a mixed learning signal indicating at least a plurality of target sounds, thereby obtaining a plurality of target sounds derived from the target sounds. using a learning-side signal processing unit that generates a processed learning mixture signal representing at least a processed target sound, and a learning-side sound source separation model for extracting the plurality of processed target sounds, the processed learning mixture a learning-side model inference unit that extracts a sound from a signal to represent the extracted sound and generates a plurality of training extraction signals each corresponding to each of the plurality of processed target sounds; To bring said one target sound closer to one of said plurality of processed target sounds corresponding to said one target sound, in response to a signal indicating said one target sound among said target sounds. a signal transforming unit for generating a plurality of transformed target sound signals each representing a plurality of transformed target sounds each derived from each of the plurality of target sounds; and the plurality of learning extraction signals. and a model updating unit that updates the learning-side sound source separation model using the plurality of modified target sound signals so that the extracted sound approaches the plurality of modified target sounds. do.

A sound source separation model learning device according to a second aspect of the present disclosure performs predetermined processing on a mixed learning signal indicating at least a plurality of target sounds, thereby obtaining a plurality of signals derived from the plurality of target sounds. a learning-side signal processing unit that generates a processed learning mixed signal indicating at least a processed target sound; a learning-side feature quantity extraction unit for generating learning feature data, which is time-series data of the extracted learning sound feature quantity, and the plurality of processed target sounds for extracting the plurality of A learning side model that generates a plurality of learning masks for extracting each of the plurality of processed target sounds from the learning feature data, using a learning side sound source separation model that indicates a weight for each of the components of A sound is extracted from the learning feature data using the inference unit and the plurality of learning masks, and a plurality of a learning-side signal extracting unit for generating a learning extraction signal of; a plurality of deformations each representing a plurality of deformed target sounds each derived from each of the plurality of target sounds by performing deformation processing for approximating one processed target sound corresponding to the one target sound; a signal transforming unit that generates a target sound signal; and a training method that uses the plurality of learning extraction signals and the plurality of transformed target sound signals so that the extracted sound approaches the plurality of transformed target sounds. and a model updating unit that updates the side sound source separation model.

A sound source separation device according to a first aspect of the present disclosure performs predetermined processing on a target mixed signal indicating at least a plurality of target sounds, thereby obtaining a plurality of processed target sounds derived from the plurality of target sounds. a utilization-side signal processing unit that generates a processed target mixed signal that at least indicates a sound; a learning side for generating a processed mixed learning signal indicating at least a plurality of processed target sounds derived from a plurality of target sounds derived from a learning side, and extracting a plurality of processed target sounds indicated by the processed mixed learning signal; By extracting sounds from the processed mixed learning signal using a sound source separation model, the extracted sounds are represented, and each of the plurality of processed target sounds represented by the processed mixed learning signals generates a plurality of learning extraction signals corresponding to the one target sound for a signal indicating one of a plurality of target sounds represented by the learning mixed signal, the one target sound to the processed learning By performing deformation processing to approximate one of the processed target sounds indicated by the mixed signal for learning to one of the processed target sounds corresponding to the one target sound, the plurality of target sounds indicated by the mixed signal for learning generating a plurality of modified target sound signals each representing a plurality of modified target sounds each derived from each of the target sounds, and using the plurality of learning extraction signals and the plurality of modified target sound signals, for extracting a plurality of processed target sounds represented by the processed target mixed signal, generated by updating the learning-side sound source separation model so that the sounds approach the plurality of deformed target sounds ; extracting sounds from the processed target mixed signal using the utilization-side sound source separation model to represent the sounds extracted from the processed target mixed signal; and a utilization-side model inference unit that generates a plurality of utilization extraction signals each corresponding to each of the target sounds.

A sound source separation device according to a second aspect of the present disclosure performs predetermined processing on a target mixed signal indicating at least a plurality of target sounds, thereby obtaining a plurality of processed target sounds derived from the plurality of target sounds. A utilization-side signal processing unit that generates a processed target mixed signal that indicates at least a sound; a utilization-side feature quantity extraction unit that generates utilization feature data that is time-series data of the extracted utilization acoustic feature quantity ; generating a processed learning mixed signal indicating at least a plurality of processed target sounds derived from the plurality of target sounds indicated by the learning mixed signal; and generating a predetermined sound from the processed learning mixed signal By extracting the acoustic feature amount for learning which is the feature amount in a plurality of components, learning feature data which is time-series data of the extracted acoustic feature amount for learning is generated, and the processed mixed signal for learning is used as using a learning source separation model that indicates weights for each of a plurality of components in the learning feature data to extract a plurality of processed target sounds indicated from the learning feature data to the processed training mixture signal generating a plurality of learning masks for extracting each of a plurality of processed target sounds represented by and extracting a sound from the learning feature data using the plurality of learning masks, generating a plurality of learning extraction signals each representing the extracted sound and corresponding to each of a plurality of processed target sounds represented by the processed mixed learning signals; corresponding to the one target sound among the plurality of processed target sounds indicated by the processed mixed signal for learning. a plurality of modified target sounds each representing a plurality of modified target sounds each derived from each of the plurality of target sounds represented by the learning mixed signal by performing a deformation process for approximating one processed target sound to the generating a sound signal, and using the plurality of learning extraction signals and the plurality of modified target sound signals, and adjusting the learning-side sound source separation model so that the extracted sound approaches the plurality of modified target sounds; using a utilization-side sound source separation model that indicates a weight for each of a plurality of components in the utilization feature data for extracting a plurality of processed target sounds indicated by the processed target mixed signal generated by updating , the activity a utilization-side model inference unit for generating a plurality of utilization masks for extracting each of the plurality of processed target sounds indicated by the processed target mixed signal from the target sound feature data, and using the plurality of utilization masks, a plurality of processed target sounds each corresponding to each of the plurality of processed target sounds indicated by the processed target mixed signal, which at least represent the sounds extracted from the utilized feature data by extracting sounds from the utilized feature data; and a utilization-side signal extraction unit that generates a utilization extraction signal.

A program according to a first aspect of the present disclosure causes a computer to perform predetermined processing on a mixed learning signal indicating at least a plurality of target sounds, thereby performing a plurality of processing derived from the plurality of target sounds. a learning-side signal processing unit that generates a processed learning mixed signal representing at least a processed target sound; and a learning-side sound source separation model for extracting the plurality of processed target sounds from the processed learning mixed signal. a learning-side model inference unit that extracts a sound to generate a plurality of learning extraction signals that indicate the extracted sound and that correspond to each of the plurality of processed target sounds; deformation processing for approximating said one target sound to one of said plurality of processed target sounds corresponding to said one target sound, with respect to a signal indicating one target sound in to generate a plurality of modified target sound signals each representing a plurality of modified target sounds each derived from each of the plurality of target sounds; and the plurality of learning extraction signals and the It functions as a model updating unit that updates the learning-side sound source separation model using a plurality of modified target sound signals so that the extracted sound approaches the plurality of modified target sounds.

A program according to a second aspect of the present disclosure causes a computer to perform predetermined processing on a mixed learning signal indicating at least a plurality of target sounds, thereby performing a plurality of processing derived from the plurality of target sounds. a learning-side signal processing unit that generates a processed learning mixed signal that at least indicates the target sound; By doing so, a learning-side feature quantity extraction unit that generates learning feature data that is time-series data of the extracted learning acoustic feature quantity, and extracts the plurality of components to extract the plurality of processed target sounds. a learning-side model inference unit that generates a plurality of learning masks for extracting each of the plurality of processed target sounds from the learning feature data, using a learning-side sound source separation model that indicates a weight for each;
a plurality of learning extractions each representing the extracted sound and corresponding to each of the plurality of processed target sounds by extracting sounds from the learning feature data using the plurality of learning masks; a learning-side signal extraction unit that generates a signal;
for a signal indicating one of the plurality of target sounds, one of the plurality of processed target sounds corresponding to the one target sound. a signal transforming unit that generates a plurality of transformed target sound signals each representing a plurality of transformed target sounds each derived from each of the plurality of target sounds by performing transformation processing to bring the plurality of transformed target sounds closer to A model updating unit that updates the learning-side sound source separation model using the learning extraction signal and the plurality of modified target sound signals so that the extracted sound approaches the plurality of modified target sounds. It is characterized by

A program according to a third aspect of the present disclosure causes a computer to perform predetermined processing on a target mixed signal indicating at least a plurality of target sounds, thereby generating a plurality of processed signals derived from the plurality of target sounds. a utilization-side signal processing unit that generates a processed target mixed signal that indicates at least a target sound; for generating a processed mixed learning signal indicating at least a plurality of processed target sounds derived from a plurality of target sounds indicated by and extracting a plurality of processed target sounds indicated by the processed mixed learning signal each of a plurality of processed target sounds represented by the processed mixed signal for learning by extracting sounds from the processed mixed signal for learning using the learning-side sound source separation model; and generating a plurality of learning extraction signals each corresponding to a signal indicating one target sound among a plurality of target sounds indicated by the learning mixed signal, the one target sound being subjected to the processing By performing deformation processing to approximate one of the processed target sounds indicated by the mixed learning signal to one of the processed target sounds corresponding to the one target sound, the target sound indicated by the mixed learning signal generating a plurality of modified target sound signals each representing a plurality of modified target sounds each derived from each of the plurality of target sounds; using the plurality of learning extraction signals and the plurality of modified target sound signals, Extracting a plurality of processed target sounds represented by the processed target mixed signal generated by updating the learning-side sound source separation model so that the extracted sounds approximate the plurality of deformed target sounds . By extracting sounds from the processed target mixed signal using the utilization-side sound source separation model for It is characterized by functioning as a utilization-side model inference section that generates a plurality of utilization extraction signals each corresponding to each of the processed target sounds.

A program according to a fourth aspect of the present disclosure causes a computer to perform predetermined processing on a target mixed signal indicating at least a plurality of target sounds, thereby generating a plurality of processed signals derived from the plurality of target sounds. A utilization-side signal processing unit that generates a processed target mixed signal that indicates at least a target sound, and extracts, from the processed target mixed signal, a utilization acoustic feature quantity, which is a predetermined acoustic feature quantity, in a plurality of components, By performing a predetermined process on a learning mixed signal indicating at least a plurality of target sounds, a utilization-side feature quantity extraction unit that generates utilization feature data that is time-series data of the extracted utilization acoustic feature quantity, generating a processed learning mixed signal indicating at least a plurality of processed target sounds derived from the plurality of target sounds indicated by the learning mixed signal; and obtaining a predetermined acoustic feature from the processed learning mixed signal. By extracting the learning acoustic feature amount, which is a quantity, in a plurality of components, learning feature data, which is time-series data of the extracted learning acoustic feature amount, is generated and represented by the processed learning mixed signal. using a learning-side sound source separation model that indicates a weight for each of a plurality of components in the learning feature data in order to extract a plurality of processed target sounds from the learning feature data with the processed learning mixture signal generating a plurality of learning masks each for extracting each of the plurality of processed target sounds shown, and using the plurality of learning masks to extract a sound from the learning feature data, generating a plurality of learning extraction signals each representing the extracted sound and corresponding to each of the plurality of processed target sounds represented by the processed learning mixed signals; For a signal indicating one target sound among the target sounds, the one target sound corresponds to the one target sound among the plurality of processed target sounds indicated by the processed mixed signal for learning. A plurality of modified target sounds each representing a plurality of modified target sounds each derived from each of the plurality of target sounds represented by the learning mixed signal by performing transformation processing to approximate one processed target sound. generating a signal, and using the plurality of learning extraction signals and the plurality of modified target sound signals, updating the learning-side sound source separation model so that the extracted sound approaches the plurality of modified target sounds. using a utilization-side sound source separation model that indicates a weight for each of the plurality of components in the utilization feature data in order to extract a plurality of processed target sounds indicated by the processed target mixed signal generated by a utilization-side model inference unit for generating a plurality of utilization masks, each of which extracts a plurality of processed target sounds indicated by the processed target mixed signal from the utilization feature data; Extracting sounds from the inflectional feature data using a mask to indicate at least the sounds extracted from the inflectional feature data , each of a plurality of processed target sounds indicated by the processed target mixed signal having It is characterized by functioning as a utilization-side signal extraction unit that generates a plurality of corresponding utilization extraction signals.

A sound source separation model learning method according to a first aspect of the present disclosure performs predetermined processing on a mixed learning signal indicating at least a plurality of target sounds, thereby obtaining a plurality of signals derived from the plurality of target sounds. generating a processed training mixture signal indicative of at least a processed target sound, and extracting sounds from the processed training mixture signal using a learning-side source separation model for extracting the plurality of processed target sounds. to generate a plurality of learning extraction signals each representing the extracted sound and corresponding to each of the plurality of processed target sounds, and a signal representing one of the plurality of target sounds , the one target sound is deformed to bring it closer to one of the plurality of processed target sounds corresponding to the one target sound, thereby obtaining the plurality of purposes generating a plurality of modified target sound signals each indicating a plurality of modified target sounds each derived from each of the sounds, and using the plurality of learning extraction signals and the plurality of modified target sound signals, The learning-side sound source separation model is updated so that the sound approaches the plurality of modified target sounds.

A sound source separation model learning method according to a second aspect of the present disclosure performs predetermined processing on a mixed learning signal indicating at least a plurality of target sounds, thereby obtaining a plurality of signals derived from the plurality of target sounds. generating a processed learning mixed signal indicating at least the processed target sound, and extracting, from the processed learning mixed signal, learning acoustic feature amounts, which are predetermined acoustic feature amounts, in a plurality of components, A learning-side sound source separation model that generates learning feature data, which is time-series data of the extracted learning acoustic feature amount, and indicates a weight for each of the plurality of components in order to extract the plurality of processed target sounds. to generate a plurality of learning masks for extracting each of the plurality of processed target sounds from the learning feature data, and using the plurality of learning masks to generate the learning feature data generating a plurality of training extraction signals each representing the extracted sound and corresponding to each of the plurality of processed target sounds; Transforming a signal indicating a target sound so as to bring the one target sound closer to one of the plurality of processed target sounds corresponding to the one target sound generating a plurality of modified target sound signals each indicating a plurality of modified target sounds each derived from each of the plurality of target sounds, and using the plurality of learning extraction signals and the plurality of modified target sound signals; and updating the learning-side sound source separation model so that the extracted sound approaches the plurality of modified target sounds.

A sound source separation method according to a first aspect of the present disclosure performs predetermined processing on a target mixed signal indicating at least a plurality of target sounds, thereby obtaining a plurality of processed target sounds derived from the plurality of target sounds. generating a processed target mixed signal representing at least a sound, and performing predetermined processing on a learning mixed signal representing at least a plurality of target sounds, thereby obtaining a plurality of target sounds represented by the learning mixed signal; using a learning-side sound source separation model for generating a processed learning mixture signal indicating at least a plurality of processed target sounds from which the learning-side sound source separation model is extracted, and extracting the plurality of processed target sounds indicated by the processed learning mixture signal; , extracting sounds from the processed mixed signal for learning, indicating the extracted sound, and a plurality of learning signals each corresponding to each of a plurality of processed target sounds indicated by the processed mixed signal for learning generating an extraction signal for learning, and for a signal indicating one target sound among a plurality of target sounds indicated by the mixed learning signal, the one target sound indicated by the processed mixed learning signal Each of the plurality of target sounds represented by the learning mixed signal is transformed by performing deformation processing to approximate one of the plurality of processed target sounds to one of the processed target sounds corresponding to the one target sound. generating a plurality of modified target sound signals each representing a plurality of modified target sounds derived from the A utilization-side sound source separation model for extracting a plurality of processed target sounds indicated by the processed target mixed signal generated by updating the learning-side sound source separation model so as to approach the modified target sound of to extract a sound from the processed target mixed signal using generates a plurality of corresponding leveraged extraction signals.

A sound source separation method according to a second aspect of the present disclosure performs predetermined processing on a target mixed signal indicating at least a plurality of target sounds, thereby obtaining a plurality of processed target sounds derived from the plurality of target sounds. generating a processed target mixed signal indicating at least a sound, and extracting a plurality of components from the processed target mixed signal, which is a predetermined acoustic feature quantity, to utilize the extracted utilized sound; By generating utilization feature data, which is time-series data of feature amounts, and performing predetermined processing on a learning mixed signal indicating at least a plurality of target sounds, a plurality of purposes indicated by the learning mixed signal are obtained. generating a processed learning mixed signal indicating at least a plurality of processed target sounds derived from sounds, and obtaining a learning acoustic feature amount, which is a predetermined acoustic feature amount, as a plurality of components from the processed learning mixed signal; to generate learning feature data, which is time-series data of the extracted learning acoustic feature amount, and to extract a plurality of processed target sounds indicated by the processed learning mixed signal Each of a plurality of processed target sounds represented by the processed mixed signal for learning is obtained from the feature data for learning using a learning side sound source separation model that indicates a weight for each of the plurality of components in the feature data for learning. generates a plurality of learning masks for extraction, and extracts sounds from the learning feature data using the plurality of learning masks, thereby indicating the extracted sounds and the processed learning generating a plurality of learning extraction signals each corresponding to each of the plurality of processed target sounds indicated by the mixed signal, and a signal indicating one of the plurality of target sounds indicated by the mixed learning signal; , transforming the one target sound closer to one processed target sound corresponding to the one target sound among the plurality of processed target sounds indicated by the processed mixed signal for learning. to generate a plurality of modified target sound signals each representing a plurality of modified target sounds each derived from each of the plurality of target sounds represented by the mixed learning signals, and the plurality of learning extracted signals and the processed target mixture generated by updating the learning-side sound source separation model using the plurality of modified target sound signals so that the extracted sound approaches the plurality of modified target sounds. with the processed target mixture signal from the utilized feature data using a utilized-side sound source separation model that indicates a weight for each of the plurality of components in the utilized feature data to extract a plurality of processed target sounds indicated by a signal Show generating a plurality of inflection masks for extracting each of a plurality of processed target sounds, and extracting sounds from the inflection feature data using the plurality of inflection masks, thereby obtaining the inflection feature data and generating a plurality of utilized extraction signals each corresponding to each of the plurality of processed target sounds indicated by the processed target mixed signal .

According to one or more aspects of the present disclosure, sound source separation by machine learning can function effectively even when acoustic characteristics fluctuate.

1 is a block diagram schematically showing the configuration of a sound source separation system; FIG. 1 is a block diagram schematically showing the configuration of a sound source separation model learning device; FIG. 3 is a block diagram schematically showing the configuration of a signal transforming section according to Embodiment 1; FIG. 2 is a block diagram schematically showing the hardware configuration of a sound source separation model learning device; FIG. 1 is a block diagram schematically showing the configuration of a sound source separation device; FIG. 1 is a block diagram schematically showing the hardware configuration of a sound source separation device; FIG. 4 is a flowchart showing the operation of the sound source separation model learning device; 4 is a flow chart showing the operation of a signal transforming unit according to Embodiment 1; 4 is a flow chart showing the operation of the sound source separation device; FIG. 4 is a conceptual diagram showing the operation of the sound source separation model learning device; (A) and (B) are schematic diagrams for explaining an operation example of the sound source separation device. It is a schematic diagram showing an example of use of the sound source separation device. FIG. 10 is a block diagram schematically showing the configuration of a signal transforming section according to Embodiment 2; FIG. 10 is a flow chart showing the operation of a signal transforming unit according to Embodiment 2. FIG.

Embodiment 1.
FIG. 1 is a block diagram schematically showing the configuration of a sound source separation system 100 according to Embodiment 1. As shown in FIG.
A sound source separation system 100 separates a sound source separation model learning device 110 that generates a sound source separation model from a learning signal, and a target sound emitted from each sound source contained in a target mixed signal using the sound source separation model. and a sound source separation device 130 for outputting the target sound.

Here, the target sound refers to a sound to be separated and extracted using the sound source separation device 130 , and the non-target sound refers to a sound that does not need to be extracted using the sound source separation device 130 . In other words, the target sound refers to the sound that should be extracted by the sound source separation device 130 , and the non-target sound refers to the sound that should not be extracted by the sound source separation device 130 .

The sound source separation model learning device 110 and the sound source separation device 130 can exchange data. For example, although not shown, the sound source separation model learning device 110 and the sound source separation device 130 are connected to a network.

The sound source separation model learning device 110 generates a sound source separation model based on the learning signal. The generated sound source separation model is provided to the sound source separation device 130 .
The sound source separation device 130 uses the sound source separation model to extract multiple target sounds from a mixed signal containing multiple target sounds emitted from multiple sound sources.

A sound source separation model is a learning model in the NN that is used when the sound source separation device 130 separates sound sources. The sound source separation model includes, for example, information for defining the wiring structure of the NN and parameters storing weights in each wiring of the NN. The wiring structure of the sound source separation model includes, for example, fully connected NN, convolutional NN (CNN), recurrent NN (RNN), long short-term memory (LSTM), and gated regression. It may be a type unit (gated recurrent unit: GRU) or a combination thereof.

FIG. 2 is a block diagram schematically showing the configuration of the sound source separation model learning device 110. As shown in FIG.
The sound source separation model learning device 110 includes a learning side input unit 111, a mixed signal generation unit 112, a learning side signal processing unit 113, a learning side feature quantity extraction unit 114, a learning side sound source separation model storage unit 115, and a learning side input unit 111. A side model inference unit 116 , a learning side signal extraction unit 117 , a signal transformation unit 118 , a model update unit 119 , and a learning side communication unit 120 are provided.

Learning-side input unit 111 receives an input of a learning signal. The input learning signal is provided to the mixed signal generation section 112 and the signal modification section 118 .
The learning signal is, for example, a target sound such as a voice individually emitted from a plurality of speakers, a piece of music individually played by a plurality of musical instruments, or a noise individually emitted from a plurality of noise sources. and non-target sound recording data signals.

The mixed signal generation unit 112 acquires the signals of the target sound and the non-target sound as the learning signal, and for example, by adding these signals, a mixed signal in which a plurality of target sounds and the non-target sounds are mixed is obtained. Generate a training mixture signal. The mixed signal for learning is given to the learning-side signal processing section 113 .
Here, the learning mixed signal includes two or more target sounds. Also, the learning mixed signal may or may not include one or more non-target sounds. The learning mixed signal may be, for example, a signal obtained by simply adding two or more signals acquired as learning signals. In other words, the training mixed signal is a signal that at least represents a plurality of target sounds.

The mixed signal generator 112 may include, for example, a process of simulating a target mixed signal, which is a mixed signal input to the sound source separation device 130 . For example, if the target mixed signal is a multi-channel signal recorded by a microphone array, the mixed signal generator 112 may include processing that simulates observation by the microphone array by convolving the impulse response of the microphone array.

The learning-side signal processing unit 113 generates a processed learning mixed signal representing at least a plurality of processed target sounds derived from the plurality of target sounds by performing predetermined processing on the learning mixed signal. . The processed learning mixed signal is provided to the learning-side feature amount extraction unit 114 and the signal transformation unit 118 .
For example, the learning-side signal processing unit 113 applies various signal processing to the learning mixed signal given from the mixed signal generation unit 112 to easily extract the target sound. to generate
Specifically, the predetermined process may be a process other than machine learning, or may be a process using machine learning.
Moreover, it is desirable that the predetermined process be a process that makes it easier to extract a plurality of target sounds.
Furthermore, it is desirable that the predetermined processing be processing for emphasizing a plurality of target sounds.

The learning-side signal processing unit 113 performs the same processing as the processing performed in the sound source separation device 130 . For example, classical signal processing, processing using machine learning, unknown signal processing, or the like is performed. The unknown signal processing may include classical signal processing or processing using machine learning.

Specifically, the processing performed by the learning-side signal processing unit 113 includes beamforming processing for suppressing a noise signal or a signal indicating a sound other than the target sound from the input mixed signal for learning. good too. Further, the processing performed by the learning-side signal processing unit 113 may include processing for suppressing reverberation. Furthermore, in the processing performed by the learning-side signal processing unit 113, when a reference signal of a non-target sound existing in the mixed signal for learning is given, a reference signal of the non-target sound represented by an echo canceller or the like is used. A process of adaptively transforming the signal into a form contained in the training mixture signal and subtracting it from the training mixture signal to remove components derived from non-target sounds from the training mixture signal may be included.

Note that the details of the processing performed by the learning-side signal processing unit 113 may change over time. The learning mixed signal input to the learning side signal processing unit 113 is, for example, a multi-channel signal recorded by a microphone array, and the output processed learning mixed signal is, for example, a single channel signal. However, the requirements for the number of channels are not limited to this.

The learning-side feature quantity extraction unit 114 extracts an acoustic feature quantity from the processed mixed signal for learning provided from the learning-side signal processing unit 113, and extracts learning feature data, which is time-series data of the extracted acoustic feature quantity. to generate
For example, the learning-side feature amount extracting unit 114 extracts learning acoustic feature amounts, which are predetermined acoustic feature amounts, from a plurality of components from the processed learning mixed signal, thereby extracting the extracted learning acoustic features. Generate feature data for learning, which is time-series data of quantity.
Here, the acoustic feature amount is, for example, a complex spectrum obtained by performing Fast Fourier Transform (FFT) processing on the processed mixed signal for learning. The feature data for learning is given to the learning side model inference section 116 and the learning side signal extraction section 117 .

The learning-side sound source separation model storage unit 115 stores a learning-side sound source separation model that is a sound source separation model used in the sound source separation model learning device 110 . The learning-side sound source separation model indicates, for example, a weight parameter for each component in the learning feature data.

The learning-side model inference unit 116 performs learning-side sound source separation by extracting learning-separation feature values, which are separation feature values necessary for performing sound source separation, from the learning-use feature data provided from the learning-side feature value extraction unit 114 . Extract using a model. The time-series data of the learning separation feature quantity extracted by the learning-side model inference unit 116 is, for example, time-series data called “mask”. A mask is a filter for extracting only the component of each sound source from the acoustic feature quantity extracted by the learning-side feature quantity extraction unit 114 . The mask is given, for example, by determining the ratio of the components of the sound source to be separated and extracted among the components of the acoustic feature quantity extracted by the learning-side feature quantity extraction unit 114 . The mask generated here is given to learning-side signal extraction section 117 as a learning mask.
That is, in order to extract a plurality of processed target sounds, the learning-side model inference unit 116 uses a learning-side sound source separation model that indicates the weight for each of the plurality of components that constitute the learning feature data, and extracts the learning feature data. A learning mask for extracting one processed target sound from the data is generated for each target sound. Here, since the learning mixed signal includes a plurality of target sounds, a plurality of learning masks are generated.

The learning-side signal extraction unit 117 extracts learning feature data, which is time-series data of the acoustic feature quantity extracted by the learning-side feature quantity extraction unit 114, and the learning separation feature quantity estimated by the learning-side model inference unit 116. Acoustic signals to be extracted are extracted using learning masks, which are time-series data.
For example, the learning-side signal extracting unit 117 uses each of a plurality of learning masks given from the learning-side model inference unit 116 to extract sounds from the learning feature data, thereby learning at least the extracted sounds. Generate an extraction signal for

Specifically, the learning-side signal extraction unit 117 multiplies the learning separation feature amount and the learning acoustic feature amount for each component, and then performs Inverse Fast Fourier Transform (IFFT) processing. By doing so, a learning extraction signal, which is a signal obtained by extracting the target sound to be extracted, is restored. Since a plurality of learning masks are used here, a plurality of learning extraction signals each corresponding to each of the plurality of learning masks are restored.

The signal transformation unit 118 transforms each of the plurality of target sounds included in the learning signal by using the learning signal supplied from the learning-side input unit 111 and the processed learning mixed signal supplied from the learning-side signal processing unit 113. , a modified target sound signal is generated by performing a deformation process for approximating each target sound included in the processed mixed signal for learning to a sound corresponding to the target sound. The generated modified target sound signal is given to the model updating unit 119 .
For example, the signal transformation unit 118 may perform transformation processing on a signal indicating one target sound among a plurality of target sounds so as to bring the one target sound closer to one corresponding processed target sound. Then, a modified target sound signal indicating one modified target sound derived from the one target sound is generated for each target sound. Here, since a plurality of target sounds exist, a plurality of modified target sound signals each corresponding to each of the plurality of target sounds are generated.

Specifically, when the learning signal includes three components of a first target sound, a second target sound, and a non-target sound, the signal transforming unit 118 transforms the signal representing the first target sound. and a transformation f2 for transforming the signal representing the second target sound. Then, the signal transforming unit 118 determines that the difference between the processed learning mixed signal given from the learning-side signal processing unit 113 and the signal obtained by adding the signal representing the first target sound and the signal representing the second target sound is By determining the transformation f1 and the transformation f2 so as to be the smallest, each of the first target sound and the second target sound is brought closer to the sound derived from each of the target sounds included in the processed mixed signal for learning. be able to. Thus, by applying the transformation f1 to the signal representing the first target sound, the modified target sound signal corresponding to the first target sound can be generated, and the transformation f2 to the signal representing the second target sound. can generate a modified target sound signal corresponding to the second target sound.

Here, it is assumed that the first target sound, the second target sound, and the non-target sound have statistically different properties, in other words, have no correlation. For this reason, for example, the difference between the processed mixed signal for learning given from the learning-side signal processing unit 113 and the signal indicating the first target sound and the signal indicating the second target sound is expressed as a squared error By calculating , each of the first target sound and the second target sound can be approximated to the sound derived from each of the target sounds included in the processed mixed signal for learning. A specific structure of the signal transforming unit 118 will be described later.

The model update unit 119 updates the learning-side sound source separation model storage unit 115 with the plurality of learning extraction signals provided from the learning-side signal extraction unit 117 and the plurality of modified target sound signals provided from the signal modification unit 118. Update the weight parameter included in the stored training sound source separation model.
For example, the model update unit 119 uses a plurality of learning extraction signals and a plurality of deformed target sound signals to convert the sound extracted by the learning-side signal extraction unit 117 into one target sound corresponding to one target sound to be extracted. The learning-side sound source separation model is updated so as to approach the modified target sound.
Specifically, the model updating unit 119 updates the learning-side sound source separation model so that the difference between the plurality of learning extraction signals and the plurality of deformation target sound signals becomes smaller.

For updating the weight parameter, for example, the result of calculating the difference between the output of the signal transforming unit 118 and the output of the learning-side signal extracting unit 117 and, for example, stochastic gradient descent (SGD) or Adam A known optimization technique such as the method is used.

The learning side communication unit 120 uses the learning sound source separation model stored in the learning side sound source separation model storage unit 115 as a utilization side sound source separation model, which is a sound source separation model used in the sound source separation device 130, to the sound source separation device 130. send to

Note that the learning-side feature quantity extraction unit 114 and the learning-side signal extraction unit 117 may be configured without both.
In this case, the learning-side model inference unit 116 uses a learning-side sound source separation model for extracting a plurality of processed target sounds included in the processed learning mixed signal given from the learning-side signal processing unit 113. Then, by extracting sounds from the processed mixed signal for learning, a learning extraction signal indicating the extracted sounds is generated.
In addition, the signal transforming unit 118 converts a signal indicating one target sound corresponding to one processed target sound among a plurality of target sounds indicated by the learning signal to one target sound. A modified target sound signal representing one modified target sound derived from one target sound is generated for each target sound by performing deformation processing to approximate the processed target sound.
Then, the model updating unit 119 uses the plurality of learning extraction signals and the plurality of deformation target sound signals to update each of the plurality of sounds extracted by the learning-side model inference unit 116 to one of the plurality of deformation target sounds. The learning-side sound source separation model is updated so as to approach the corresponding modified target sound.

FIG. 3 is a block diagram schematically showing the configuration of signal transforming section 118 according to the first embodiment.
The signal transformation unit 118 includes a mixed signal block division unit 118a, a learning signal block division unit 118b, a filter estimation unit 118c, a filter application unit 118d, and a block combination unit 118e.

The mixed signal block division unit 118a generates a mixed block signal, which is a signal obtained by dividing the processed learning mixed signal given from the learning side signal processing unit 113 into blocks that are appropriate sections. This is the block division part.
For example, the mixed signal block division unit 118a generates a plurality of mixed block signals by dividing the processed learning mixed signal into a plurality of blocks.
The mixed block signal is provided to filter estimator 118c.

The division into blocks may be performed, for example, at regular time intervals.
Alternatively, the blocks may be divided such that overlapping sections are generated between the blocks.
However, the length of each block corresponding to the number of samples must be set to exceed the length required for deriving the filter in the filter estimation unit 118c.

The learning signal block division unit 118b extracts a signal of the target sound from the learning signal given from the learning side input unit 111, and divides the signal of the target sound into appropriate sections. It is a second block division unit that generates a sound block signal.
For example, the learning signal block dividing unit 118b generates a plurality of target sound block signals by dividing a signal representing one target sound into a plurality of blocks.
The target sound block signal is provided to the filter estimation section 118c and the filter application section 118d. The division method into blocks is the same as the division method in mixed signal block division section 118a.

The filter estimation unit 118c estimates a filter for approximating the sound indicated by each of the plurality of target sound block signals to the sound corresponding to one target sound to be extracted among the sounds indicated by the plurality of mixed block signals. to estimate multiple filters.
For example, the filter estimation unit 118c divides the mixed block signal divided into blocks by the mixed signal block division unit 118a and the target sound block signal divided into blocks by the learning signal block division unit 118b into blocks. And, for each target sound, a deformation parameter, which is a parameter of a filter approximating the conversion of the sound indicated by the target sound block signal to the sound indicated by the mixed block signal, is generated. As the filter, for example, an FIR (Finite Impulse Response) filter, an IIR (Infinite Impulse Response) filter, or a filter in the frequency domain using FFT may be used.
Note that the deformation parameters may differ, for example, for each block.

The filter application unit 118d generates a plurality of modified block signals by applying each of the plurality of filters estimated by the filter estimation unit 118c to each of the plurality of target sound block signals.
For example, the filter application unit 118d converts a signal obtained by applying a deformation parameter corresponding to the target sound block signal estimated by the filter estimation unit 118c to the target sound block signal supplied from the learning signal block division unit 118b into a deformation block. Generate as a signal. The deformed block signal is provided to the block combiner 118e.

The block combiner 118e generates a transformed target sound signal that is a signal obtained by combining the transformed block signals supplied from the filter application unit 118d. The modified target sound signal is provided to the model updating unit 119 shown in FIG.
Note that when the mixed signal block division unit 118a and the learning signal block division unit 118b perform division so that overlapping sections occur between a plurality of blocks, the block combination unit 118e performs weighting, for example. Duplicates may be resolved by calculating the sum.

The mixed signal block dividing section 118a, the learning signal block dividing section 118b, and the block combining section 118e may be configured without these. That is, the entire signal may be treated as a single block.
In such a case, the filter estimating unit 118c associates one target sound indicated by the learning signal with one target sound among the plurality of processed target sounds indicated by the processed mixed signal for learning. Estimate a filter for approximating one processed target sound.
Then, the filter application unit generates a modified target sound signal by applying the filter estimated by the filter estimation unit 118c to the signal indicating one target sound among the learning signals.

FIG. 4 is a block diagram schematically showing the hardware configuration of the sound source separation model learning device 110. As shown in FIG.
The sound source separation model learning device 110 can be configured by a computer 150 having a storage device 151 , a memory 152 , a processor 153 and a communication interface (hereinafter referred to as communication I/F) 154 .

The storage device 151 stores programs and data necessary for processing performed by the sound source separation model learning device 110 .
Memory 152 provides a work area in which processor 153 works.
The processor 153 expands the programs and data stored in the storage device 151 into the memory 152 and executes processing.
Communication I/F 154 communicates with sound source separation device 130 .

For example, the mixed signal generation unit 112, the learning side signal processing unit 113, the learning side feature amount extraction unit 114, the learning side model inference unit 116, the learning side signal extraction unit 117, the signal transformation unit 118, and the model update unit 119 are the processor 153 can be realized by expanding the program and data stored in the storage device 151 into the memory 152 and executing the program.
The learning-side sound source separation model storage unit 115 can be realized by the storage device 151 .
The learning side input section 111 and the learning side communication section 120 can be realized by the communication I/F 154 .

The program as described above may be provided through a network, or may be provided by being recorded on a recording medium. That is, such programs may be provided as program products, for example.
Sound source separation model learning device 110 may be realized by a program as described above, or may be realized by forming a circuit for each function executed by sound source separation model learning device 110 and connecting the circuits. may
In other words, the sound source separation model learning device 110 can also be realized by a processing circuit network.

FIG. 5 is a block diagram schematically showing the configuration of the sound source separation device 130. As shown in FIG.
The sound source separation device 130 includes a utilization-side communication unit 131, a utilization-side sound source separation model storage unit 132, a utilization-side input unit 133, a utilization-side signal processing unit 134, a utilization-side feature amount extraction unit 135, and a utilization-side model It includes an inference unit 136 , a utilization-side signal extraction unit 137 , and a utilization-side output unit 138 .

The utilization side communication unit 131 communicates with the sound source separation model learning device 110 . For example, the utilization-side communication unit 131 receives the utilization-side sound source separation model from the sound source separation model learning device 110 and stores the utilization-side sound source separation model in the utilization-side sound source separation model storage unit 132 .

The utilization-side sound source separation model storage unit 132 stores the utilization-side sound source separation model.
Utilization side input unit 133 receives input of the target mixed signal. The input target mixed signal is provided to the utilization side signal processing section 134 .
The target mixed signal may be stored in advance in the sound source separation device 130, may be acquired by an audio device such as a microphone described later, or may be acquired from a telephone line or the like via a communication I/F. In such a case, the utilization side input unit 133 can be omitted.

The utilization-side signal processing unit 134 performs a predetermined process on the target mixed signal indicating at least a plurality of target sounds, thereby obtaining a processed target signal indicating at least a plurality of processed target sounds derived from the plurality of target sounds. Generate a mixed signal.
For example, the utilization-side signal processing unit 134 generates a processed target mixed signal obtained by applying various signal processing to the target mixed signal given from the utilization-side input unit 133 in order to easily extract the target sound. do. The processing performed here is the same as the processing performed in the learning-side signal processing unit 113 of the sound source separation model learning device 110 . The processed target mixed signal is provided to the utilization side feature quantity extraction unit 135 .

The utilization-side feature amount extraction unit 135 extracts acoustic feature amounts from the processed target mixed signal given from the utilization-side signal processing unit 134, and generates utilization feature data that is time-series data of the extracted acoustic feature amounts. do.
For example, the utilizing-side feature amount extraction unit 135 extracts, from the processed target mixed signal, the utilized acoustic feature amount, which is a predetermined acoustic feature amount, in a plurality of components, and extracts the extracted utilized acoustic feature amount. Generate utilization feature data, which is time-series data.
The processing performed here is the same as the processing performed by the learning-side feature quantity extraction unit 114 of the sound source separation model learning device 110 . The utilization feature data is given to the utilization side model inference section 136 .

The utilizing-side model inference unit 136 converts the utilizing-side separation feature quantity, which is a separation feature quantity required for performing sound source separation, from the utilization feature data given from the utilizing-side feature quantity extraction unit 135 to the utilizing-side sound source separation model. Extract using The processing performed here is the same as the processing performed in the learning side model inference section 116 of the sound source separation model learning device 110 .
Then, the utilizing-side model inference unit 136 gives the mask, which is the extracted time-series data of the utilized separating feature quantity, to the utilizing-side signal extraction unit 137 as a utilized mask.
In other words, in order to extract a plurality of processed target sounds, the utilizing-side model inference unit 136 uses the utilizing-side sound source separation model, which indicates the weight for each of the plurality of components of the utilizing feature data, to extract one from the utilizing feature data. A utilization mask for extracting two processed target sounds is generated for each target sound. Therefore, a plurality of utilization masks each corresponding to each of a plurality of target sounds are generated.

The utilizing-side signal extraction unit 137 extracts the utilization-side feature data, which is the time-series data of the acoustic feature quantity extracted by the utilization-side feature quantity extraction unit 135, and the utilization-side separation feature quantity estimated by the utilization-side model inference unit 136. An acoustic signal to be extracted is extracted using a utilization mask, which is series data.
For example, the utilization-side signal extraction unit 137 extracts a sound from the utilization feature data using the utilization mask, thereby generating a utilization extraction signal indicating at least the extracted sound.
The processing performed here is the same as the processing performed by the learning-side signal extraction unit 117 of the sound source separation model learning device 110 . Then, the utilization-side signal extraction unit 137 provides the utilization-side extraction signal, which is the extracted acoustic signal, to the utilization-side output unit 138 as an output signal.

The utilization-side output section 138 outputs the output signal given from the utilization-side signal extraction section 137 .
For example, one or both of the utilization-side feature amount extraction unit 135 and the utilization-side signal extraction unit 137 may be omitted. For example, when neither the utilizing-side feature quantity extraction unit 135 nor the utilizing-side signal extraction unit 137 is included, the utilizing-side model inference unit 136 processes the processed target mixed signal output from the utilizing-side signal processing unit 134. , which functions to directly output the signal of the separated sound. In other words, the utilizing-side model inference unit 136 uses the utilizing-side sound source separation model for extracting a plurality of processed target sounds indicated by the processed target mixed signal given from the utilizing-side signal processing unit 134 to extract the processed target sound. A sound is extracted from the target mixed signal to generate a leveraged extraction signal indicative of the extracted sound.

FIG. 6 is a block diagram schematically showing the hardware configuration of the sound source separation device 130. As shown in FIG.
The sound source separation device 130 can be configured by a computer 160 having a storage device 161 , a memory 162 , a processor 163 , a communication I/F 164 and an acoustic interface (hereinafter referred to as acoustic I/F) 165 .

The storage device 161 stores programs and data necessary for processing performed by the sound source separation device 130 .
Memory 162 provides a work area in which processor 163 works.
The processor 163 develops the programs and data stored in the storage device 161 in the memory 162 and executes processing.
Communication I/F 164 communicates with sound source separation model learning device 110 .
Acoustic I/F 165 receives an input of a target mixed signal. The target mixed signal may be generated by an audio device that collects sounds including the target sound and generates the target phonetic signal.

For example, the utilizing-side signal processing unit 134, the utilizing-side feature amount extracting unit 135, the utilizing-side model inference unit 136, the utilizing-side signal extracting unit 137, and the utilizing-side output unit 138 are executed by the processor 163 as a program stored in the storage device 161 and It can be realized by deploying the data in the memory 162 and executing the program.
The utilization-side sound source separation model storage unit 132 can be realized by the storage device 161 .
The utilization side input unit 133 can be realized by the acoustic I/F 165 .
The utilization side communication unit 131 can be realized by the communication I/F 154 .

The program as described above may be provided through a network, or may be provided by being recorded on a recording medium. That is, such programs may be provided as program products, for example.
Note that the sound source separation device 130 may be realized by a program as described above, or may be realized by forming a circuit for each function executed by the sound source separation device 130 and connecting the circuits.
In other words, the sound source separation device 130 can also be realized by processing circuitry.

Next, operation will be described. First, the operation of the sound source separation model learning device 110 will be described.
FIG. 7 is a flowchart showing the operation of the sound source separation model learning device 110. As shown in FIG.

First, the mixed signal generator 112 creates a learning mixed signal, which is a mixed signal used for learning, from the learning signal (S10). The learning mixed signal is created by simulating the active mixed signal input to the active signal processing unit 134 of the sound source separation device 130 . The learning mixed signal may be generated, for example, by simply adding a plurality of target sound signals and non-target sound signals as learning signals. In addition, in order to simulate recording by a microphone array, the mixed signal for training is processed by convoluting the impulse response of the microphone array for each signal acquired from the training signal, and then adding the output signals. may be generated by

Next, the learning-side signal processing unit 113 applies various signal processing to the learning mixed signal provided from the mixed signal generation unit 112 (S11). The processing content here is the same as the processing content in the active-side signal processing unit 134 of the sound source separation device 130 .

Next, the signal transforming unit 118 transforms the target sound obtained from the learning signal into a form that imitates the target sound included in the processed learning mixed signal given from the learning-side signal processing unit 113. Thus, a modified target sound signal is generated for each target sound (S12). Details of the processing in step S12 will be described later.

Next, the learning-side feature amount extraction unit 114 extracts learning acoustic feature amounts, which are acoustic feature amounts, from the processed learning mixed signal given from the learning-side signal processing unit 113, and converts them into time-series data. Learning feature data is generated (S13). As the acoustic feature quantity, for example, a complex spectrum obtained by applying FFT to the processed learning mixed signal from the utilization-side signal processing unit 134 is used. The processing content here is the same as the processing content in the utilization side feature amount extraction unit 135 of the sound source separation device 130 .

Next, the learning-side model inference unit 116 uses the learning sound source separation model to separate and synthesize each sound source signal from the acoustic feature quantity extracted by the learning-side feature quantity extraction unit 114. A feature amount for learning separation, which is a feature amount for learning, is extracted, and a mask, which is time-series data of the feature amount for learning separation, is generated (S15). A mask is generated for each sound source signal, in other words, for each target sound. The processing content here is the same as the processing content in the utilizing-side model inference section 136 of the sound source separation device 130 .

Next, the learning-side signal extraction unit 117 uses the acoustic feature amount extracted by the learning-side feature amount extraction unit 114 and the learning separation feature amount extracted by the learning-side model inference unit 116, An extraction signal for learning, which is a sound signal obtained by processing the target sound contained in the mixed signal for learning, is extracted (S15). For example, the learning-side signal extraction unit 117 multiplies the learning separation feature amount and the learning acoustic feature amount for each component, and then performs an inverse Fourier transform process to obtain a sound originating from the target sound to be extracted. is restored for each target sound. The processing content here is the same as the processing content in the learning-side signal extraction unit 117 of the sound source separation device 130 .

Next, the model updating unit 119 calculates the error between the plurality of modified target sound signals provided by the signal modifying unit 118 and the plurality of learning extraction signals provided by the learning-side signal extracting unit 117, and then calculates the error. A weight parameter provided in the learning sound source separation model is updated so as to correct it (S16).

Next, the operation of the signal transforming section 118 will be described.
FIG. 8 is a flow chart showing the operation of signal transforming section 118 according to the first embodiment.
First, the mixed signal block division unit 118a generates a mixed block signal by dividing the processed learning mixed signal given from the learning side signal processing unit 113 into one or more blocks on the time axis (S20 ).

Next, the learning signal block dividing unit 118b generates a target sound block signal by dividing the learning signal given from the learning side input unit 111 into one or more blocks on the time axis (S21 ). The signal division method in the learning signal block division unit 118b is the same as the division method performed by the mixed signal block division unit 118a in step S20.

Next, the filter estimation unit 118c estimates a filter (S22).
Here, the processed mixed signal for learning and the signal for learning are all single-channel acoustic signals, and the mixed signal generation unit 112 acquires signals indicating n target sounds as the learning signal to create a mixed signal. A case will be described as an example. Here, n is an integer of 1 or more.

Let y(t) be the mixed block signal obtained from the mixed signal block division unit 118a. Here, t is an integer that satisfies t=0, .
Let s _i (t) be the target sound block signal of the i-th target sound obtained from the learning signal block division unit 118b. Here, i is an integer that satisfies 1≤i≤n.
Furthermore, when the filter calculated by the filter estimator 118c is an FIR filter of length L, let h _i (τ) be the coefficient of the FIR filter for the i-th target sound. Here, τ is an integer that satisfies τ=0, . . . , L−1.
At this time, the mixed block signal y(t) is approximated by the following equation (1).

Here, let us consider the case where the approximation of equation (1) holds best with the squared error criterion.
That is, consider the case where h _i (τ) minimizes the error function of the following equation (2).

As means for obtaining such h _i (τ), first, a matrix S _i εR ^{((T−L+1)×L)} shown in the following equation (3) is defined.

At this time, equation (2) can be expressed in a matrix form as shown in equation (4) below.

Here, _y is represented by the following formula (5), hi is represented by the following formula (6), S is represented by the following formula (7), and h is represented by the following formula (8).

At this time, the filter h _i that best approximates y with the least square error criterion is the solution of the optimization problem expressed by the following equation (9).

The solution of the optimization problem of formula (9) is given by formula (10) below.

By such a procedure, the FIR filter coefficients h _i (t) that closely approximate y(t) are obtained.

Note that the matrix S ^T S often has a large number of conditions, and there is a possibility that the optimization problem cannot be solved numerically stably. Therefore, a modified optimization problem may be solved as shown in equation (11) below.

The solution of the optimization problem represented by equation (11) is represented by equation (12) below.

where λ is an arbitrary hyperparameter and INL is an identity matrix of size _NL .
Comparing the matrix _STS with the matrix ^STS + ^λINL , the latter has a smaller condition number and can stably calculate the inverse matrix.

Note that y(t) and s _i (τ) in the above are the training signal and the processed mixed training signal, for example, a signal acquired from a single sound device such as one microphone. Although it was assumed to be a single-channel signal, Embodiment 1 is not limited to such an example.
For example, the training signal and the processed mixed training signal may be multi-channel signals acquired using a microphone array comprising a plurality of microphones. In this case, when the filter estimator 118c receives multi-channel target sound block signals, it may select the target sound block signals of representative channels and calculate the above filter coefficients. Also, even when the filter estimator 118c receives multi-channel mixed block signals, it suffices to select a representative mixed block signal and perform the above calculation of the filter coefficients.

Next, the filter application unit 118d applies the filter estimated for each block in step S22 to the target sound block signal generated in step S20, thereby generating a modified block signal (S23).

Finally, the block combiner 118e joins the deformed block signals divided into blocks to generate a deformed target sound signal (S24).

FIG. 9 is a flowchart showing the operation of the sound source separation device 130. As shown in FIG.
First, the utilization-side signal processing unit 134 applies various types of signal processing to the input target mixed signal to generate a processed target mixed signal (S30).

Next, the utilization-side feature amount extraction unit 135 extracts acoustic feature amounts from the processed target mixed signal given from the utilization-side signal processing unit 134, and extracts the utilization feature data, which is time-series data of the extracted acoustic feature amounts. Generate (S31).

Next, the utilization-side model inference unit 136 uses the utilization-side sound source separation model to extract the sound feature values extracted by the utilization-side feature value extraction unit 135, and the separation parameters necessary for separating and synthesizing each sound source signal. A utilization mask, which is time-series data of feature amounts, is generated for each target sound (S32).

Next, the utilizing-side signal extracting unit 137 uses the utilizing acoustic feature amount extracted by the utilizing-side feature amount extracting unit 135 and the separation feature amount extracted by the utilizing-side model inference unit 136 to extract the object An output signal, which is the signal of the target sound contained in the mixed signal, is generated for each target sound (S33).

Next, an operation example of the sound source separation model learning device 110 will be described.
FIG. 10 is a conceptual diagram showing the operation of the sound source separation model learning device 110. As shown in FIG.
A first signal 170 is a signal indicating the first target sound obtained from the learning signal, a second signal 171 is a signal indicating the second target sound obtained from the learning signal, and a third A signal 172 is a signal indicating the non-target sound acquired from the learning signal.

The mixed signal generator 112 creates a pseudo learning mixed signal 173 by simply adding the first signal 170 , the second signal 171 and the third signal 172 , for example.
The learning mixed signal 173 includes a first component 170#1 derived from the first signal 170, a second component 171#1 derived from the second signal 171, and a third component 171#1 derived from the third signal 172. A third component 172#1 is included.

By passing learning mixed signal 173 through learning-side signal processing section 113, processed learning mixed signal 173# is obtained. At this time, the first component 170#1 derived from the first target sound becomes the fourth component 170#2, and the second component 171#1 derived from the second target sound becomes the fifth component 170#2. A third component 172#1 derived from a non-target sound, such as the component 171#2 of , appears in the processed mixed training signal 173#, such as the sixth component 172#2.

In order to extract sounds corresponding to the first target sound and the second target sound from the processed learning mixed signal 173#, the learning side feature quantity extraction unit 114, the learning side model inference unit 116, and the learning side By applying the processing in the signal extraction unit 117, a first extraction signal for learning 174 corresponding to the first target sound and a second extraction signal for learning 175 corresponding to the second target sound are obtained.

Further, the signal transforming unit 118 uses the first signal 170 and the second signal 171, and the processed learning mixed signal 173# to change the first signal 170 into a fourth component 170#2, a filter and a Assume a filter that transforms the second signal 171 into the fifth component 171#2. Then, the signal transforming unit 118 applies respective filters to the first signal 170 and the second signal 171 to generate a first transform target sound signal 176 and a second transform target sound signal 177 .

The model update unit 119 performs Update the parameters of the training sound source separation model.

Next, an operation example of the sound source separation device 130 when using the sound source separation model learned by the sound source separation model learning device 110 will be described.
11A and 11B are schematic diagrams for explaining an operation example of the sound source separation device 130. FIG.

FIG. 11A is a conceptual diagram showing how the waveform of the input target mixed signal is changed by the sound source separation device 130. FIG.
The target mixed signal 180 shown in FIG. 11A includes a first component 181 derived from the first target sound, a second component 182 derived from the second target sound, and A third component 183 derived from is included.

When target mixed signal 180 passes through utilization-side signal processing section 134, processed target mixed signal 180# is obtained. The processed target mixed signal 180# includes a fourth component 181# derived from the first component 181, a fifth component 182# derived from the second component, and a third component 182# derived from the third component 183#. 6 component 183#.

Since processing for suppressing the non-target sound is performed in the utilization-side signal processing unit 134, the volume of the sixth component 183# is lowered compared to the third component 183 derived from the non-target sound. Also, compared to the first component 181 derived from the first target sound and the second component 182 derived from the second target sound, the fourth component 181# and the fifth component 182# are emphasized. It is Further, the fourth component 181# and the fifth component 182# have their volume and waveform shape (frequency characteristics) changed due to signal processing, and also have a delay caused by the utilization side signal processing section 134. As a result, target mixed signal 180 and processed target mixed signal 180# are out of time synchronization.

A first output signal 184 and a second An output signal 185 of is obtained. The first output signal 184 is obtained by extracting the component corresponding to the first target sound, and the second output signal 185 is obtained by extracting the component corresponding to the second target sound.

FIG. 11B is a conceptual diagram showing a case where similar signal processing is applied to a target mixed signal 186 different from the target mixed signal 180. FIG.
Comparing the processed target mixed signal 180# and the processed target mixed signal 186#, changes in waveform and volume are different. Therefore, the waveform and volume of the first output signal 187 and the second output signal 188 are also different from the first output signal 184 and the second output signal 185 .

In this way, due to changes in the characteristics of the target mixed signal input to the utilization-side signal processing unit 134, changes in the processing contents of the utilization-side signal processing unit 134, etc., the characteristics of the processed target mixed signal also change, and after signal processing Sound sources can be separated with high accuracy by using a learning model generated considering the state of .

In addition, in the sound source separation model learning device 110, a configuration can be considered in which the learning-side signal processing unit 113 is omitted and the signal transformation unit 118 does not transform the learning signal. Such a sound source separation model learning device and learning method are conventionally known.

In this case, the learning-side model inference unit 116 extracts the first signal 170 of the first target sound and the second Learning is performed so as to obtain a separating feature amount for separating the signal 171 of No. 2. FIG.
However, when operating the sound source separation device 130, as shown in FIG.
Various characteristics are different between the feature amount extracted from the learning mixed signal 173 and the feature amount extracted from the processed target mixed signal 180#. Since the sound source separation model is not trained on the assumption that the feature amount extracted from the processed target mixed signal 180# is input, the separation performance deteriorates.

Further, in the sound source separation model learning apparatus 110, it is conceivable to adopt a configuration in which the learning signal processing unit 113 is not omitted, but the signal transforming unit 118 does not transform the learning signal.
In this case, the learning-side model inference unit 116 generates the first signal 170 of the first target sound and the second signal 171 of the second target sound from the feature quantity extracted from the processed learning mixed signal 173#. Learning is performed so as to obtain a separation feature amount for separating the . Learning-side model inference section 116 learns the sound source separation model on the premise that the feature amount extracted from processed mixed signal for learning 173# is input, so the above-described problem can be solved. .

However, the fourth component 181# and the fifth component 182# included in the processed target mixed signal 180# shown in FIG. 11(A) and the elephant mixed signal 180 shown in FIG. The included first component 181 and second component 182 differ in characteristics such as volume, frequency characteristics and delay.
For this reason, the sound source separation model is learned so as to cancel such changes in various characteristics and output the original signal. However, as described above, such changes in characteristics change depending on what kind of signal is input to the utilization-side signal processing unit 134 or as time passes. It is difficult to train a sound source separation model so as to absorb such various characteristic changes.

In the sound source separation model learning device 110, the learning-side signal processing unit 113 and the signal transforming unit 118 function together, and the sound source separation model separates the first deformed target sound signal 176 and the second deformed target sound signal 177. By learning to output the feature amount for the sound source separation model, it is sufficient to learn to output the result that takes into account the characteristic change.
In the case of a configuration in which the signal transformation unit 118 does not transform the learning signal, it was necessary to train the sound source separation model so as to cancel out the characteristic change and output the restored result. This eliminates the need for learning to output the result of canceling the characteristic change, which simplifies the learning process and, as a result, improves the quality of the sound source separation output.

FIG. 12 is a schematic diagram showing a usage example of the sound source separation device 130. As shown in FIG.
FIG. 12 shows a sound uttered by a speaker 192 in the driver's seat, a sound uttered by a speaker 193 in the passenger's seat, and noise 194 emitted from a car running sound or a car stereo, etc., in microphones 191A, 191B, and 191C installed in a vehicle 190. are observed at the same time. At this time, the case where the sound source separation device 130 is used to extract the voice uttered by the driver's seat speaker 192 and the voice uttered by the passenger's seat speaker 193 will be described.

The voice uttered by the driver's seat speaker 192 is the first component 181 of the first target sound shown in FIG. , and various noises 194 correspond to the third component 183 of the non-target sound. Signals recorded by the microphones 191A, 191B, and 191C correspond to the target mixed signal 180. FIG.
In the sound source separation device 130, the sixth component 183# corresponding to the noise 194 is suppressed in the processed target mixed signal 180# output from the utilization-side signal processing unit 134. FIG.

After applying the utilization-side sound source separation model, the results extracted by the utilization-side signal extraction unit 137 correspond to the first output signal 184 and the second output signal 185 . In these signals, the voices of the driver's seat and passenger's seat are emphasized.

The utilization-side sound source separation model is a modified first sound source obtained when the sound source separation model learning device 110 performs signal processing for suppressing the noise 194 for the voices of the speakers on the driver's seat side and the passenger's seat side. Since the modified target sound signal 176 and the second modified target sound signal 177 are taken into consideration, the driver's seat and the passenger's seat are generated from a state in which the voice of the driver's seat, the voice of the passenger's seat, and the noise 194 are actually mixed. It is possible to properly separate the voices of two speakers sitting on the same floor.

In addition, not only in a vehicle but also in the case of retrieving attendees' utterances from recording memory during a meeting, if the sound source separation model learning device learns the attendee's voice and generates a sound source separation model, If the sound source separation model is used after performing signal processing to remove irrelevant surrounding noise, the voice of each attendee can be separated.

As described above, according to Embodiment 1, when the sound source separation apparatus 130 performs sound source separation using a sound source separation model, sound source separation is performed according to changes in acoustic characteristics that occur with the utilization-side signal processing unit 134 . The model corresponds, and as a result, the quality of the separated sound output from the sound source separation device 130 is improved.

Further, as an effect of providing the mixed signal block dividing unit 118a, the learning signal block dividing unit 118b, and the block combining unit 118e, by outputting different filter parameters for each block, it is possible to cope with time-series changes. become.

Embodiment 2.
In Embodiment 1, the filter estimation unit 118c estimates a filter for each block divided by the mixed signal block division unit 118a and the learning signal block division unit 118b. In Embodiment 2, a different filter is estimated not for each block but for each time in one block, in other words, by sequentially updating the filter, it is possible to cope with time-series changes in the block. to

As shown in FIG. 1 , a sound source separation system 200 according to Embodiment 2 includes a sound source separation model learning device 210 and a sound source separation device 130 .
The sound source separation device 130 according to the second embodiment is the same as the sound source separation device 130 according to the first embodiment.

As shown in FIG. 2, the sound source separation model learning apparatus 210 according to Embodiment 2 includes a learning side input unit 111, a mixed signal generation unit 112, a learning side signal processing unit 113, and a learning side feature amount extraction unit. a learning-side sound source separation model storage unit 115; a learning-side model inference unit 116; a learning-side signal extraction unit 117; a signal transformation unit 218; .
Learning-side input unit 111, mixed signal generation unit 112, learning-side signal processing unit 113, learning-side feature amount extraction unit 114, learning-side source separation model storage unit 115, learning-side model inference unit 116, and learning-side in Embodiment 2 The signal extraction unit 117, the model update unit 119, and the learning side communication unit 120 are the same as the learning side input unit 111, the mixed signal generation unit 112, the learning side signal processing unit 113, the learning side feature amount extraction unit 114, and the learning side communication unit 120 in Embodiment 1. It is the same as the side sound source separation model storage unit 115 , the learning side model inference unit 116 , the learning side signal extraction unit 117 , the model updating unit 119 and the learning side communication unit 120 .

FIG. 13 is a block diagram schematically showing the configuration of signal transforming section 218 according to the second embodiment.
The signal transformation unit 218 includes a mixed signal block division unit 118a, a learning signal block division unit 118b, a filter application unit 218d, a block combination unit 118e, a filter parameter storage unit 218f, and a filter update unit 218g.

The mixed signal block dividing unit 118a, the learning signal block dividing unit 118b, and the block combining unit 118e in Embodiment 2 are the mixed signal block dividing unit 118a, the learning signal block dividing unit 118b, and the block combining unit 118e in Embodiment 1. is similar to

The filter parameter storage unit 218f stores filter parameters used by the filter application unit 218d.
For example, the filter parameter storage unit 218f stores filter parameters for each sample corresponding to a predetermined period.

The filter application unit 218d applies the filter parameters stored in the filter parameter storage unit 218f to a plurality of target sound block signals, thereby generating processed sample signals at times corresponding to the filter parameters. The processed sample signal is provided to filter updater 218g. In other words, the filter application unit 218d generates the processed sample signal by applying the filter parameter to the portion selected from the plurality of target sound block signals for each sample.

Further, the filter applying unit 218d generates a plurality of modified block signals by combining the generated processed sample signals with each of the plurality of target sound block signals. A plurality of modified block signals are provided to the block combiner 118e.

The filter updating unit 218g updates the filter parameters stored in the filter parameter storage unit 218f so that the processed sample signal provided from the filter applying unit 218d approaches the corresponding portion of the processed mixed signal for learning.

FIG. 14 is a flow chart showing the operation of signal transforming section 218 according to the second embodiment.
Among the steps included in the flowchart shown in FIG. 14, the steps that perform the same processing as the steps included in the flowchart shown in FIG. The same reference numerals as the steps included in the flow chart are attached.

The processes in steps S20 and S21 included in the flowchart shown in FIG. 14 are the same as the processes in steps S20 and S21 included in the flowchart shown in FIG. However, in FIG. 14, after the process of step S21, the process proceeds to step S40.

In step S40, the filter applying unit 218d selects one unselected target sound block signal from the plurality of target sound block signals received from the learning signal block dividing unit 118b.

Next, the filter updating unit 218g determines the initial values of the filter parameters and stores the initial values in the filter parameter storage unit 218f (S41). When the filter used by the filter applying unit 218d is an FIR filter, the filter updating unit 218g performs, for example, the same processing as in step S22 of the flowchart shown in FIG. An initial value should be estimated.

Next, the filter application unit 218d selects the top sample among the samples for which processed sample signals have not yet been generated, among the target sound block signals selected in step S40 (S42).

Next, the filter application unit 218d reads the filter parameters stored in the filter parameter storage unit 218f, and applies the read filter parameters to the portion corresponding to the selected sample in the target sound block signal. By doing so, a processed sample signal is generated (S43). The generated processed sample signal is provided to the filter updating section 218g.

Next, the filter updating unit 218g uses the processed sample signal from the filter applying unit 218d, the mixed block signal from the mixed signal block dividing unit 118a, and the target sound block signal from the learning signal block dividing unit 118b. , the filter parameters stored in the filter parameter storage unit 218f are updated (S44). For example, when the filter is an FIR filter, a known NLMS (Normalized Least Mean Square) algorithm, RLS (Recursive Least Square) algorithm, or the like can be used as a method for updating the filter parameters. When the filter updating unit 218g performs updating, there are cases where processing by the filter applying unit 218d is required.

Next, the filter application unit 218d determines whether or not a processed sample signal has been generated from all the samples included in the selected target sound block signal (S45). If processed sample signals have been generated from all the samples (Yes at S45), the process proceeds to step S46. , the process returns to step S42.

In step S46, the filter application unit 218d generates a modified block signal by concatenating the processed sample signals generated for each sample. The deformed block signal is provided to the block combiner 118e.

Next, the filter application unit 218d determines whether or not all target sound block signals given from the learning signal block division unit 118b have been selected (S47). If all the target sound block signals have been selected (Yes at S47), the process proceeds to step S24. Return to S40.

Then, the block combiner 118e joins the deformed block signals divided for each block to generate a deformed target sound signal (S24), similarly to the processing in step S24 of FIG.

As described above, according to Embodiment 2, since the filters are sequentially updated, even when the learning-side signal processing unit 113 and the utilization-side signal processing unit 134 perform adaptive processing, the learning-side signal processing It is possible to cope with time-series changes in the section 113 and the utilization-side signal processing section 134 .

In Embodiment 2, the filter updating unit 218g and the filter applying unit 218d update the filter for each sample to generate the deformed block signal. 118b and block coupling portion 118e may not be provided.
In such a case, the filter application unit 218d applies the filter parameters stored in the filter parameter storage unit 218f to the signal indicating the target sound to be extracted, thereby obtaining the time corresponding to each filter parameter. Generate a processed sample signal at .
The filter updating unit 218g updates the filter parameters so that the processed sample signal approaches the corresponding portion of the processed mixed signal for learning.
Then, the filter application unit 218d combines the generated processed sample signals to generate a modified target sound signal.

On the other hand, by providing the mixed signal block dividing unit 118a, the learning signal block dividing unit 118b, and the block combining unit 118e, the filter application processing can be performed in parallel on a block basis to improve the processing speed, or the filter parameter can be set on a block basis. parameter extraction speed can be improved by creating a candidate group and searching for parameters from the group when extracting parameters for each sample.

For example, if an FIR filter is used, the length of each block should be set longer than the length of the filter during block division in order to estimate the filter. Therefore, when estimating a filter for each block as in Embodiment 1, at least the length of the FIR filter is added to the time-series changes of the learning-side signal processing unit 113 and the utilization-side signal processing unit 134. If it is not a unit, it cannot be tracked. On the other hand, by estimating the filter for each sample as in Embodiment 2, the time-series changes in the learning-side signal processing unit 113 and the utilization-side signal processing unit 134 can be handled more efficiently in units of time for each sample. It can be followed closely.

Further, by providing the filter parameter storage unit 218f as in the second embodiment, the filter updating unit 218g retains the immediately preceding filter estimation result in the filter parameter storage unit 218f, and then adds a new sample. When obtained, the filter parameters recorded in the filter parameter storage unit 218f can be applied after being slightly modified according to the selected sample.

The sound source separation model learning devices 110 and 210 described above combine a sound source separation method based on NN and a signal processing method based on classical signal processing, processing using machine learning, or unknown signal processing. This has the effect of promoting the learning of the sound source separation model and improving the sound source separation performance when configuring the separation device 130 . Therefore, for example, in a device for recognizing speech in a noisy environment, classical signal processing and NN-based source separation can be combined to extract the uttered speech of the target speaker. The unknown signal processing may include classical signal processing or processing using machine learning.

Embodiments 1 and 2 described above are composed of two devices, the sound source separation model learning devices 110 and 210 and the sound source separation device 130. Not limited. For example, the sound source separation model learning devices 110 and 210 and the sound source separation device 130 may be configured as one device, for example, one sound source separation learning device. In such a case, the learning-side communication unit 120 and the utilization-side communication unit 131 are unnecessary, and the learning-side sound source separation model storage unit 115 and the utilization-side sound source separation model storage unit 132 are used to store sound source separation models. It can be integrated as a separate model store.

100, 200 sound source separation system 110, 210 sound source separation model learning device 111 learning side input unit 112 mixed signal generation unit 113 learning side signal processing unit 114 learning side feature amount extraction unit 115 learning side source separation model storage 116 learning-side model inference unit 117 learning-side signal extraction unit 118, 218 signal transformation unit 118a mixed signal block division unit 118b learning signal block division unit 118c filter estimation unit 118d, 218d filter application unit 118e block combining unit 218f filter parameter storage unit 218g filter update unit 119 model update unit 120 learning side communication unit 130 sound source separation device 131 utilization side communication unit 132 utilization side sound source separation model storage unit 133 utilization side Input section 134 Utilization side signal processing section 135 Utilization side feature value extraction section 136 Utilization side model inference section 137 Utilization side signal extraction section 138 Utilization side output section.

Claims (19)

performing predetermined processing on a mixed learning signal representing at least a plurality of target sounds to generate a processed mixed learning signal representing at least a plurality of processed target sounds derived from the plurality of target sounds; a learning-side signal processing unit;
extracting sounds from the processed learning mixed signal using a training-side sound source separation model for extracting the plurality of processed target sounds, thereby representing the extracted sounds and the plurality of processed target sounds; a learning-side model inference unit that generates a plurality of training extraction signals each corresponding to each sound;
for a signal indicating one of the plurality of target sounds, one of the plurality of processed target sounds corresponding to the one target sound. a signal transformation unit that generates a plurality of transformed target sound signals each representing a plurality of transformed target sounds each derived from each of the plurality of target sounds by performing transformation processing to approximate to
a model updating unit that updates the learning-side sound source separation model using the plurality of learning extraction signals and the plurality of modified target sound signals so that the extracted sound approaches the plurality of modified target sounds; A sound source separation model learning device comprising: performing predetermined processing on a mixed learning signal representing at least a plurality of target sounds to generate a processed mixed learning signal representing at least a plurality of processed target sounds derived from the plurality of target sounds; a learning-side signal processing unit;
for learning, which is time-series data of the extracted learning acoustic feature quantity, by extracting a learning acoustic feature quantity, which is a predetermined acoustic feature quantity, in a plurality of components from the processed mixed signal for learning; a learning-side feature quantity extraction unit that generates feature data;
Each of the plurality of processed target sounds is extracted from the learning feature data using a learning-side sound source separation model that indicates a weight for each of the plurality of components for extracting the plurality of processed target sounds. a learning-side model inference unit that generates a plurality of learning masks for
a plurality of learning extractions each representing the extracted sound and corresponding to each of the plurality of processed target sounds by extracting sounds from the learning feature data using the plurality of learning masks; a learning-side signal extraction unit that generates a signal;
for a signal indicating one of the plurality of target sounds, one of the plurality of processed target sounds corresponding to the one target sound. a signal transformation unit that generates a plurality of transformed target sound signals each representing a plurality of transformed target sounds each derived from each of the plurality of target sounds by performing transformation processing to approximate to
a model updating unit that updates the learning-side sound source separation model using the plurality of learning extraction signals and the plurality of modified target sound signals so that the extracted sound approaches the plurality of modified target sounds; A sound source separation model learning device comprising: 3. The sound source separation model learning device according to claim 1, wherein the predetermined process is a process for facilitating extraction of the plurality of target sounds. The sound source separation model learning device according to any one of claims 1 to 3, wherein the predetermined process is a process of emphasizing the plurality of target sounds. The signal transforming unit
a filter estimation unit for estimating a filter for approximating the one target sound to the one processed target sound;
5. The method according to any one of claims 1 to 4, further comprising a filter applying unit that generates the modified target sound signal by applying the filter to the signal indicating the one target sound. Sound source separation model learning device. The signal transforming unit
a first block dividing unit that generates a plurality of mixed block signals by dividing the processed learning mixed signal into a plurality of blocks;
a second block dividing unit that generates a plurality of target sound block signals by dividing the signal indicating the one target sound into a plurality of blocks;
estimating a filter for approximating the sound indicated by each of the plurality of target sound block signals to the sound corresponding to the one target sound among the sounds indicated by the plurality of mixed block signals; a filter estimation unit that estimates a filter;
a filter application unit that generates a plurality of modified block signals by applying each of the plurality of filters to each of the plurality of target sound block signals;
The sound source separation model learning device according to any one of claims 1 to 4, further comprising a block combiner configured to generate the deformed target sound signal by combining the plurality of deformed block signals. . The signal transforming unit
a filter parameter storage unit that stores filter parameters for each sample corresponding to a predetermined period;
applying the filter parameters to selected portions of the signal representing the one target sound for each of the samples to generate a processed sample signal; a filter application unit that generates a sound signal;
5. The filter updating unit that updates the filter parameter so that the processed sample signal approaches the corresponding part of the processed mixed signal for learning. The sound source separation model learning device according to the item. The signal transforming unit
a first block dividing unit that generates a plurality of mixed block signals by dividing the processed learning mixed signal into a plurality of blocks;
a second block dividing unit that generates a plurality of target sound block signals by dividing the signal indicating the one target sound into a plurality of blocks;
a filter parameter storage unit that stores filter parameters for each sample corresponding to a predetermined period;
generating a processed sample signal by applying the filter parameter to a selected portion of the signal representing the one target sound for each of the samples, and applying the processed sample signal to the plurality of target sound blocks; a filter applicator configured to combine on each of the signals to generate a plurality of deformed block signals;
a filter updating unit that updates the filter parameter so that the processed sample signal approaches the corresponding portion of the processed mixed signal for learning;
The sound source separation model learning device according to any one of claims 1 to 4, further comprising a block combiner configured to generate the deformed target sound signal by combining the plurality of deformed block signals. . 9. The model updating unit updates the learning-side sound source separation model so that a difference between the plurality of learning extraction signals and the plurality of deformation target sound signals becomes smaller. The sound source separation model learning device according to any one of 1. A utilization side that generates a processed target mixed signal representing at least a plurality of processed target sounds derived from the plurality of target sounds by performing predetermined processing on a target mixed signal representing at least a plurality of target sounds. a signal processing unit;
A processed mixed signal indicating at least a plurality of target sounds is subjected to predetermined processing, thereby indicating at least a plurality of processed target sounds derived from the plurality of target sounds indicated by the mixed learning signal. generating a training mixture signal and extracting sounds from the processed training mixture signal using a training-side source separation model for extracting a plurality of processed target sounds indicated by the processed training mixture signal; a plurality of learning extraction signals each representing the extracted sound and corresponding to each of a plurality of processed target sounds indicated by the processed learning mixed signal; for a signal indicating one target sound out of the plurality of target sounds indicated, the one target sound is converted to the one target sound out of the plurality of processed target sounds indicated by the processed mixed signal for learning A plurality of deformed target sounds each representing a plurality of deformed target sounds each derived from each of the plurality of target sounds represented by the learning mixed signal by performing deformation processing for approximating one processed target sound corresponding to the sound. and using the plurality of learning extraction signals and the plurality of deformation target sound signals, the learning-side sound source is adjusted so that the extracted sound approaches the plurality of deformation target sounds Extracting sounds from the processed target mixed signal using a utilization-side source separation model for extracting a plurality of processed target sounds indicated by the processed target mixed signal generated by updating the separation model. By doing so, the utilization side generates a plurality of utilization extraction signals each representing a sound extracted from the processed target mixed signal and corresponding to each of the plurality of processed target sounds indicated by the processed target mixed signal. A sound source separation device comprising: a model inference unit; A utilization side that generates a processed target mixed signal representing at least a plurality of processed target sounds derived from the plurality of target sounds by performing predetermined processing on a target mixed signal representing at least a plurality of target sounds. a signal processing unit;
Generating utilized feature data, which is time-series data of the extracted utilized acoustic features, by extracting utilized acoustic features, which are predetermined acoustic features, from the processed target mixed signal in a plurality of components. a utilization-side feature quantity extraction unit that
A processed mixed signal indicating at least a plurality of target sounds is subjected to predetermined processing, thereby indicating at least a plurality of processed target sounds derived from the plurality of target sounds indicated by the mixed learning signal. generating a learning mixed signal, and extracting a learning acoustic feature that is a predetermined acoustic feature from the processed learning mixed signal in a plurality of components, thereby obtaining the extracted learning acoustic feature to generate learning feature data that is time-series data of and indicate weights for each of a plurality of components in the learning feature data for extracting a plurality of processed target sounds indicated by the processed learning mixed signal generating a plurality of learning masks for extracting each of a plurality of processed target sounds indicated by the processed mixed learning signal from the learning feature data using the learning-side sound source separation model; By extracting sounds from the learning feature data using a plurality of learning masks, the extracted sounds are indicated, and each of the plurality of processed target sounds indicated by the processed learning mixed signal generates a plurality of learning extraction signals corresponding to the one target sound for a signal indicating one of a plurality of target sounds represented by the learning mixed signal, the one target sound to the processed learning By performing deformation processing to approximate one of the processed target sounds indicated by the mixed signal for learning to one of the processed target sounds corresponding to the one target sound, the plurality of target sounds indicated by the mixed signal for learning generating a plurality of modified target sound signals each representing a plurality of modified target sounds each derived from each of the target sounds, and using the plurality of learning extraction signals and the plurality of modified target sound signals, for extracting a plurality of processed target sounds represented by the processed target mixed signal generated by updating the learning-side sound source separation model so that the sounds approach the plurality of deformed target sounds Each of the plurality of processed target sounds represented by the processed target mixed signal is extracted from the utilized feature data using a utilized sound source separation model indicating a weight for each of the plurality of components in the utilized feature data. an exploiting-side model inference unit that generates a plurality of exploitation masks for
By extracting sounds from the utilization feature data using the plurality of utilization masks, at least the sounds extracted from the utilization feature data are represented, and a plurality of processed target sounds indicated by the processed target mixed signal are obtained. and a utilization-side signal extraction unit that generates a plurality of utilization extraction signals corresponding to each of the utilization-side signal extraction units. the computer,
performing predetermined processing on a mixed learning signal representing at least a plurality of target sounds to generate a processed mixed learning signal representing at least a plurality of processed target sounds derived from the plurality of target sounds; learning side signal processing unit,
extracting sounds from the processed learning mixed signal using a training-side sound source separation model for extracting the plurality of processed target sounds, thereby representing the extracted sounds and the plurality of processed target sounds; a learning-side model inference unit that generates a plurality of learning extraction signals each corresponding to each sound;
for a signal indicating one of the plurality of target sounds, one of the plurality of processed target sounds corresponding to the one target sound. a signal transformation unit that generates a plurality of transformed target sound signals each representing a plurality of transformed target sounds each derived from each of the plurality of target sounds by performing transformation processing to approximate to
a model updating unit that updates the learning-side sound source separation model using the plurality of learning extraction signals and the plurality of modified target sound signals so that the extracted sound approaches the plurality of modified target sounds; A program characterized by functioning as the computer,
performing predetermined processing on a mixed learning signal representing at least a plurality of target sounds to generate a processed mixed learning signal representing at least a plurality of processed target sounds derived from the plurality of target sounds; learning side signal processing unit,
for learning, which is time-series data of the extracted learning acoustic feature quantity, by extracting a learning acoustic feature quantity, which is a predetermined acoustic feature quantity, in a plurality of components from the processed mixed signal for learning; a learning-side feature quantity extraction unit that generates feature data;
Each of the plurality of processed target sounds is extracted from the learning feature data using a learning-side sound source separation model that indicates a weight for each of the plurality of components for extracting the plurality of processed target sounds. a learning-side model inference unit that generates a plurality of learning masks for
a plurality of learning extractions each representing the extracted sound and corresponding to each of the plurality of processed target sounds by extracting sounds from the learning feature data using the plurality of learning masks; a learning-side signal extraction unit that generates a signal;
for a signal indicating one of the plurality of target sounds, one of the plurality of processed target sounds corresponding to the one target sound. a signal transformation unit that generates a plurality of transformed target sound signals each representing a plurality of transformed target sounds each derived from each of the plurality of target sounds by performing transformation processing to approximate to
a model updating unit that updates the learning-side sound source separation model using the plurality of learning extraction signals and the plurality of modified target sound signals so that the extracted sound approaches the plurality of modified target sounds; A program characterized by functioning as the computer,
A utilization side that generates a processed target mixed signal representing at least a plurality of processed target sounds derived from the plurality of target sounds by performing predetermined processing on a target mixed signal representing at least a plurality of target sounds. a signal processing unit, and
A processed mixed signal indicating at least a plurality of target sounds is subjected to predetermined processing, thereby indicating at least a plurality of processed target sounds derived from the plurality of target sounds indicated by the mixed learning signal. generating a training mixture signal and extracting sounds from the processed training mixture signal using a training-side source separation model for extracting a plurality of processed target sounds indicated by the processed training mixture signal; a plurality of learning extraction signals each representing the extracted sound and corresponding to each of a plurality of processed target sounds indicated by the processed learning mixed signal; for a signal indicating one target sound out of the plurality of target sounds indicated, the one target sound is converted to the one target sound out of the plurality of processed target sounds indicated by the processed mixed signal for learning A plurality of deformed target sounds each representing a plurality of deformed target sounds each derived from each of the plurality of target sounds represented by the learning mixed signal by performing deformation processing for approximating one processed target sound corresponding to the sound. and using the plurality of learning extraction signals and the plurality of deformation target sound signals, the learning-side sound source is adjusted so that the extracted sound approaches the plurality of deformation target sounds Extracting sounds from the processed target mixed signal using a utilization-side source separation model for extracting a plurality of processed target sounds indicated by the processed target mixed signal generated by updating the separation model. By doing so, the utilization side generates a plurality of utilization extraction signals each representing a sound extracted from the processed target mixed signal and corresponding to each of the plurality of processed target sounds indicated by the processed target mixed signal. A program characterized by functioning as a model inference part. the computer,
A utilization side that generates a processed target mixed signal representing at least a plurality of processed target sounds derived from the plurality of target sounds by performing predetermined processing on a target mixed signal representing at least a plurality of target sounds. signal processor,
Generating utilized feature data, which is time-series data of the extracted utilized acoustic features, by extracting utilized acoustic features, which are predetermined acoustic features, from the processed target mixed signal in a plurality of components. Utilization side feature value extraction unit,
A processed mixed signal indicating at least a plurality of target sounds is subjected to predetermined processing, thereby indicating at least a plurality of processed target sounds derived from the plurality of target sounds indicated by the mixed learning signal. generating a learning mixed signal, and extracting a learning acoustic feature that is a predetermined acoustic feature from the processed learning mixed signal in a plurality of components, thereby obtaining the extracted learning acoustic feature to generate learning feature data that is time-series data of and indicate weights for each of a plurality of components in the learning feature data for extracting a plurality of processed target sounds indicated by the processed learning mixed signal generating a plurality of learning masks for extracting each of a plurality of processed target sounds indicated by the processed mixed learning signal from the learning feature data using the learning-side sound source separation model; By extracting sounds from the learning feature data using a plurality of learning masks, the extracted sounds are indicated, and each of the plurality of processed target sounds indicated by the processed learning mixed signal generates a plurality of learning extraction signals corresponding to the one target sound for a signal indicating one of a plurality of target sounds represented by the learning mixed signal, the one target sound to the processed learning By performing deformation processing to approximate one of the processed target sounds indicated by the mixed signal for learning to one of the processed target sounds corresponding to the one target sound, the plurality of target sounds indicated by the mixed signal for learning generating a plurality of modified target sound signals each representing a plurality of modified target sounds each derived from each of the target sounds, and using the plurality of learning extraction signals and the plurality of modified target sound signals, for extracting a plurality of processed target sounds represented by the processed target mixed signal generated by updating the learning-side sound source separation model so that the sounds approach the plurality of deformed target sounds Each of the plurality of processed target sounds represented by the processed target mixed signal is extracted from the utilized feature data using a utilized sound source separation model indicating a weight for each of the plurality of components in the utilized feature data. a exploiting-side model inference unit that generates a plurality of exploitation masks for
By extracting sounds from the utilization feature data using the plurality of utilization masks, at least the sounds extracted from the utilization feature data are represented, and a plurality of processed target sounds indicated by the processed target mixed signal are obtained. A program characterized by functioning as a utilization-side signal extraction unit that generates a plurality of utilization extraction signals corresponding to each. performing predetermined processing on a mixed learning signal indicating at least a plurality of target sounds to generate a processed mixed learning signal indicating at least a plurality of processed target sounds derived from the plurality of target sounds; ,
extracting sounds from the processed learning mixed signal using a training-side sound source separation model for extracting the plurality of processed target sounds, thereby representing the extracted sounds and the plurality of processed target sounds; generating a plurality of training extracts, each corresponding to each of the sounds;
for a signal indicating one of the plurality of target sounds, one of the plurality of processed target sounds corresponding to the one target sound. generating a plurality of deformed target sound signals each representing a plurality of deformed target sounds each derived from each of the plurality of target sounds by performing deformation processing to approximate to
using the plurality of learning extraction signals and the plurality of modified target sound signals to update the learning-side sound source separation model so that the extracted sound approaches the plurality of modified target sounds. sound source separation model learning method. performing predetermined processing on a mixed learning signal indicating at least a plurality of target sounds to generate a processed mixed learning signal indicating at least a plurality of processed target sounds derived from the plurality of target sounds; ,
for learning, which is time-series data of the extracted learning acoustic feature quantity, by extracting a learning acoustic feature quantity, which is a predetermined acoustic feature quantity, in a plurality of components from the processed mixed signal for learning; Generate feature data,
Each of the plurality of processed target sounds is extracted from the learning feature data using a learning-side sound source separation model that indicates a weight for each of the plurality of components for extracting the plurality of processed target sounds. Generate multiple training masks for
a plurality of learning extractions each representing the extracted sound and corresponding to each of the plurality of processed target sounds by extracting sounds from the learning feature data using the plurality of learning masks; generate a signal,
for a signal indicating one of the plurality of target sounds, one of the plurality of processed target sounds corresponding to the one target sound. generating a plurality of deformed target sound signals each representing a plurality of deformed target sounds each derived from each of the plurality of target sounds by performing deformation processing to approximate to
using the plurality of learning extraction signals and the plurality of modified target sound signals to update the learning-side sound source separation model so that the extracted sound approaches the plurality of modified target sounds. sound source separation model learning method. performing predetermined processing on a target mixed signal indicating at least a plurality of target sounds to generate a processed target mixed signal indicating at least a plurality of processed target sounds derived from the plurality of target sounds;
A processed mixed signal indicating at least a plurality of target sounds is subjected to predetermined processing, thereby indicating at least a plurality of processed target sounds derived from the plurality of target sounds indicated by the mixed learning signal. generating a training mixture signal and extracting sounds from the processed training mixture signal using a training-side source separation model for extracting a plurality of processed target sounds indicated by the processed training mixture signal; a plurality of learning extraction signals each representing the extracted sound and corresponding to each of a plurality of processed target sounds indicated by the processed learning mixed signal; for a signal indicating one target sound out of the plurality of target sounds indicated, the one target sound is converted to the one target sound out of the plurality of processed target sounds indicated by the processed mixed signal for learning A plurality of deformed target sounds each representing a plurality of deformed target sounds each derived from each of the plurality of target sounds represented by the learning mixed signal by performing deformation processing for approximating one processed target sound corresponding to the sound. and using the plurality of learning extraction signals and the plurality of deformation target sound signals, the learning-side sound source is adjusted so that the extracted sound approaches the plurality of deformation target sounds Extracting sounds from the processed target mixed signal using a utilization-side source separation model for extracting a plurality of processed target sounds indicated by the processed target mixed signal generated by updating the separation model. generating a plurality of exploited extraction signals each representing a sound extracted from the processed target mixed signal and corresponding to each of a plurality of processed target sounds indicated by the processed target mixed signal ; A sound source separation method characterized by: performing predetermined processing on a target mixed signal indicating at least a plurality of target sounds to generate a processed target mixed signal indicating at least a plurality of processed target sounds derived from the plurality of target sounds;
Generating utilized feature data, which is time-series data of the extracted utilized acoustic features, by extracting utilized acoustic features, which are predetermined acoustic features, from the processed target mixed signal in a plurality of components. death,
A processed mixed signal indicating at least a plurality of target sounds is subjected to predetermined processing, thereby indicating at least a plurality of processed target sounds derived from the plurality of target sounds indicated by the mixed learning signal. generating a learning mixed signal, and extracting a learning acoustic feature that is a predetermined acoustic feature from the processed learning mixed signal in a plurality of components, thereby obtaining the extracted learning acoustic feature to generate learning feature data that is time-series data of and indicate weights for each of a plurality of components in the learning feature data for extracting a plurality of processed target sounds indicated by the processed learning mixed signal generating a plurality of learning masks for extracting each of a plurality of processed target sounds indicated by the processed mixed learning signal from the learning feature data using the learning-side sound source separation model; By extracting sounds from the learning feature data using a plurality of learning masks, the extracted sounds are indicated, and each of the plurality of processed target sounds indicated by the processed learning mixed signal generates a plurality of learning extraction signals corresponding to the one target sound for a signal indicating one of a plurality of target sounds represented by the learning mixed signal, the one target sound to the processed learning By performing deformation processing to approximate one of the processed target sounds indicated by the mixed signal for learning to one of the processed target sounds corresponding to the one target sound, the plurality of target sounds indicated by the mixed signal for learning generating a plurality of modified target sound signals each representing a plurality of modified target sounds each derived from each of the target sounds, and using the plurality of learning extraction signals and the plurality of modified target sound signals, for extracting a plurality of processed target sounds represented by the processed target mixed signal generated by updating the learning-side sound source separation model so that the sounds approach the plurality of deformed target sounds Each of the plurality of processed target sounds represented by the processed target mixed signal is extracted from the utilized feature data using a utilized sound source separation model indicating a weight for each of the plurality of components in the utilized feature data. generate multiple conjugation masks for
By extracting sounds from the utilization feature data using the plurality of utilization masks, at least the sounds extracted from the utilization feature data are represented, and a plurality of processed target sounds indicated by the processed target mixed signal are obtained. A method of sound source separation, characterized by generating a plurality of exploited extracted signals, one for each.

Images