JP2022186212A - Extraction device, extraction method, learning device, learning method, and program - Google Patents

Abstract

To extract a target voice from a mixed voice accurately and easily.SOLUTION: A generation unit 131a of an extraction device 10 generates data of a predetermined format from activity information representing a time at which a target sound source emits sound, and a mixed sound. The generation unit 131a generates a weighted sum of an output obtained by inputting the mixed sound into auxiliary NN and the activity information, a coupling vector obtained by coupling the activity information and the mixed sound, and the like. An extraction unit 131b extracts the sound of the target sound source from the mixed sound using the data generated by the generation unit 131a and an extraction network.SELECTED DRAWING: Figure 1

Description

There is an application for the application of Article 30, Paragraph 2 of the Patent Act. arXiv website https://arxiv. org/ (top page) https://arxiv. org/abs/2101.05516 (paper page) https://arxiv. org/pdf/2101.05516. pdf (Paper PDF) Posted on website January 14, 2021

The present invention relates to an extraction device, an extraction method, a learning device, a learning method, and a program.

SpeakerBeam is known as a technique for extracting the speech of a target speaker from a mixed speech signal obtained from speeches of a plurality of speakers (see, for example, Non-Patent Document 1).

For example, the method described in Non-Patent Document 1 converts a mixed speech signal into the time domain, and extracts the speech of the target speaker from the mixed speech signal in the time domain. The output of the auxiliary NN is input to an adaptation layer provided in the intermediate part of the main NN to extract the target speech contained in the mixed speech signal in the time domain. It estimates and outputs the speech signal of the person.

FIG. 13 is a diagram illustrating a conventional speaker beam. As shown in FIG. 13, in a conventional speaker beam, mixed speech _yt is input to the extraction network. Also, the target _speaker 's speech at is input to the auxiliary network.

Furthermore, the time average e (equation (1)) of the output of the auxiliary network is input between the first extraction block and the second extraction block of the extraction network. Then, the target speech ̂x _t is extracted from the mixed speech y _t by the mask ̂m _t (̂ just above m) output from the extraction network.

Marc Delcroix, et al. “Improving speaker discrimination of target speech extraction with time-domain SpeakerBeam”,https://arxiv.org/pdf/2001.08378.pdf

However, the conventional method has a problem that it may not be possible to accurately and easily extract the target speech from the mixed speech.

For example, the method described in Non-Patent Document 1 requires the voice of the target speaker to be registered in advance. In addition, for example, due to fatigue due to a long meeting, changes in physical condition, etc., the voice characteristics of the target speaker may deviate from the pre-registered voice characteristics.

In order to solve the above-described problems and achieve the object, an extraction device includes: a generation unit that generates data in a predetermined format from mixed sound and activity information representing a time at which a target sound source produced a sound; and an extraction unit that extracts the sound of the target sound source from the mixed sound using the data generated by and an NN (neural network) for extraction.

The learning device also includes a generation unit that generates data in a predetermined format from the mixed sound and activity information representing the time when the target sound source emitted the sound; network) to extract the sound of the target sound source from the mixed sound, and the loss function calculated based on the sound extracted by the extraction unit is optimized. and an updating unit that updates parameters of the neural network.

According to the present invention, it is possible to accurately and easily extract target speech from mixed speech.

FIG. 1 is a diagram showing a configuration example of an extraction device according to the first embodiment. FIG. 2 is a diagram for explaining diarization. FIG. 3 is a diagram for explaining activity information. FIG. 4 is a diagram illustrating a configuration example of a model. FIG. 5 is a diagram showing a configuration example of a model. FIG. 6 is a diagram illustrating a configuration example of a model. FIG. 7 is a diagram showing a configuration example of a learning device according to the first embodiment. FIG. 8 is a flow chart showing the processing flow of the extraction device according to the first embodiment. FIG. 9 is a flow chart showing the flow of processing of the learning device according to the first embodiment. FIG. 10 is a diagram showing experimental results. FIG. 11 is a diagram showing experimental results. FIG. 12 is a diagram illustrating an example of a computer that executes programs. FIG. 13 is a diagram illustrating a conventional speaker beam.

Embodiments of an extraction device, an extraction method, a learning device, a learning method, and a program according to the present application will be described below in detail with reference to the drawings. In addition, this invention is not limited by embodiment described below.

[First embodiment]
FIG. 1 is a diagram showing a configuration example of an extraction device according to the first embodiment. As shown in FIG. 1 , the extraction device 10 has an interface section 11 , a storage section 12 and a control section 13 .

The extraction device 10 accepts input of mixed speech including speech from a plurality of sound sources. Further, the extraction device 10 extracts the sound of the target sound source from the mixed sound and outputs it.

In this embodiment, the sound source is assumed to be the speaker. In this case, the mixed speech is a mixture of speech uttered by multiple speakers. For example, mixed speech can be obtained by recording the speech of a conference involving multiple speakers with a microphone. "Sound source" in the following description may be appropriately replaced with "speaker".

Note that this embodiment can handle not only voice uttered by a speaker, but also sound from any sound source. For example, the extraction device 10 can receive an input of mixed sound whose sound source is a sound event such as the sound of a musical instrument or the sound of a car siren, and extract and output the sound of the target sound source. Also, "voice" in the following description may be appropriately replaced with "sound".

The interface unit 11 is an interface for inputting and outputting data. For example, the interface unit 11 is a NIC (Network Interface Card). Further, the interface unit 11 may be connected to an output device such as a display and an input device such as a keyboard.

The storage unit 12 is a storage device such as a HDD (Hard Disk Drive), an SSD (Solid State Drive), an optical disc, or the like. Note that the storage unit 12 may be a rewritable semiconductor memory such as a RAM (Random Access Memory), a flash memory, or an NVSRAM (Non Volatile Static Random Access Memory). The storage unit 12 stores an OS (Operating System) and various programs executed by the extraction device 10 .

As shown in FIG. 1 , the storage unit 12 stores model information 121 . The model information 121 is parameters and the like for constructing a model. For example, the model information 121 is weights, biases, etc. for constructing each neural network described later.

The control unit 13 controls the extraction device 10 as a whole. The control unit 13 includes, for example, electronic circuits such as CPU (Central Processing Unit), MPU (Micro Processing Unit), GPU (Graphics Processing Unit), ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), etc. It is an integrated circuit. The control unit 13 also has an internal memory for storing programs defining various processing procedures and control data, and executes each processing using the internal memory.

The control unit 13 functions as various processing units by running various programs. For example, the control section 13 has a signal processing section 131 .

The signal processing unit 131 uses a model constructed from the model information 121 to extract the target speech from the mixed speech. It is also assumed that the model constructed from the model information 121 is a model trained by a learning device. The signal processor 131 has a generator 131a and an extractor 131b.

The generation unit 131a generates data in a predetermined format from the mixed sound and the activity information representing the time when the target sound source emitted the sound.

Here, activity information will be described. The activity information is information representing the time when the target sound source emitted the sound.

For example, activity information can be obtained by known techniques such as visual-based voice activity detection (VAD) [1], personal VAD [2], diarization [3-5].
Reference 1: P. Liu and Z. Wang, “Voice activity detection using visual information,” in Proc. of ICASSP'04, 2004, vol. 1, pp. I-609.
Reference 2: S. Ding, Q. Wang, S.-Y. Chang, L. Wan, and I. Lopez Moreno, “Personal VAD: Speaker-Conditioned Voice Activity Detection,” in Proc. of Odyssey'20, 2020 , pp. 433-439.
Reference 3: D. Garcia-Romero, D. Snyder, G. Sell, D. Povey, and A. McCree, “Speaker diarization using deep neural network embeddings,” Proc. of ICASSP'17, pp. 4930-4934, 2017.
Reference 4: Z. Huang, S. Watanabe, Y. Fujita, P. Garcia, Y. Shao, D. Povey, and S. Khudanpur, “Speaker diarization with region proposal network,” Proc. of ICASSP'20, pp. 6514-6518, 2020.
Reference 5: I. Medennikov, M. Korenevsky, T. Prisyach, YY Khokhlov, M. Korenevskaya, I. Sorokin, TV Timofeeva, A. Mitrofanov, A. Andrusenko, I. Podluzhny, A. Laptev, and A. Romanenko , “Target-speaker voice activity detection: a novel approach for multi-speaker diarization in a dinner party scenario,” ArXiv, vol. abs/2005.07272, 2020.

For example, if there is an image of the target speaker, visual-based VAD is effective. Also, personal VAD is effective when there is the voice of the target speaker. Diarization is also used when neither video nor audio of the target speaker is available.

In particular, by combining the recent high-precision diarization and this embodiment, practical and simple speech recognition accuracy can be improved in situations such as meetings where overlap occurs.

Here, the overlap is a state in which the target speaker's voice and the voices other than the target speaker overlap in the mixed voice. In the activity information, there are cases where the target speaker is active and where there is an overlap, and there are cases where there is no overlap.

FIG. 2 is a diagram for explaining diarization. As shown in FIG. 2, according to diarization, it is possible to identify the time period during which each of speaker A, speaker B, and speaker C spoke.

FIG. 3 is a diagram for explaining activity information. As shown in FIG. 3, the activity information may take 1 in the time interval when the sound source is emitting sound and take 0 in the time interval when the sound source is not emitting sound. In this case, p _t ε{0,1}.

The extraction unit 131b uses the data generated by the generation unit 131a and the extraction network to extract the sound of the target sound source from the mixed sound.

Here, a model constructed based on the model information 121 may be called an ADEnet (a speaker activity driven speech extraction neural network). The generating unit 131a and the extracting unit 131b perform processing using ADEnet. Several variations of ADEnet are described below.

Although the ADEnet of this embodiment receives activity information as an input, the method of acquiring activity information is not limited to a specific method, and any method may be used.

[ADEnet auxiliary]
In ADEnet-auxiliary, the generator 131a generates a weighted sum of the output obtained by inputting the mixed sound to the auxiliary network and the activity information. Also, the extraction unit 131b extracts the sound of the target sound source from the mixed sound by inputting the mixed sound and the weighted sum to the extraction network.

FIG. 4 is a diagram illustrating a configuration example of a model. The model in FIG. 4 (ADEnet-auxiliary) uses an extraction network and an auxiliary network. Both the extraction network and the auxiliary network are neural networks.

As shown in FIG. 4, the generator 131a inputs the mixed speech _yt to the auxiliary network. Then, the generation unit 131a calculates the weighted sum e of the output of the auxiliary network using the activity information p _t as the weight by the equation (2).

Furthermore, the extraction unit 131b inputs the mixed speech _yt and the weighted sum e to the extraction network to obtain a mask ^ _mt . Then, the extraction unit 131b calculates the product of each element of the mixed speech _yt and the mask ^ _mt as the target speech ^ _xt .

[ADEnet-input]
In ADEnet-input, the generation unit 131a generates a combined vector that combines the activity information and the mixed sound, both of which are represented by vectors. Also, the extraction unit 131b extracts the sound of the target sound source from the mixed sound by inputting the connection vector to the extraction network.

FIG. 5 is a diagram showing a configuration example of a model. The model (ADEnet-input) in FIG. 5 uses an extraction network.

As shown in FIG. 5, the generator 131a concatenates the mixed speech ^yt and the activity information ^pt to obtain [ _ytT , _{pt ] T} .

Then, the extraction unit 131b inputs [y _t ^T , p _t ] ^T to the extraction network to obtain the mask ^m _t . Then, the extraction unit 131b calculates the product of each element of the mixed speech _yt and the mask ^ _mt as the target speech ^ _xt .

[ADEnet-mix]
In ADEnet-mix, the generating unit 131a further generates a weighted sum of the activity information and the output obtained by inputting the mixed sound to the auxiliary network, in addition to the connection vector. Also, the extraction unit 131b extracts the sound of the target sound source from the mixed sound by inputting the connection vector and the weighted sum to the extraction network.

FIG. 6 is a diagram illustrating a configuration example of a model. The model (ADEnet-mix) in FIG. 6 uses an extraction network and an auxiliary network.

As shown in FIG. 6, the generator 131a concatenates the mixed speech ^yt and the activity information ^pt to obtain [ _ytT , _{pt ] T} .

Then, the extraction unit 131b inputs the weighted sum e and [y _t ^T , p _t ] ^T to the extraction network to obtain the mask ^m _t . Then, the extraction unit 131b calculates the product of each element of the mixed speech _yt and the mask ^ _mt as the target speech ^ _xt .

As described herein, the extractor 131b can use the mask to calculate the target speech. On the other hand, the extraction unit 131b may directly calculate the target speech using an extraction network or the like without using the mask.

Here, the configuration of the learning device will be described with reference to FIG. FIG. 7 is a diagram showing a configuration example of a learning device according to the first embodiment. As shown in FIG. 7, the learning device 20 has an interface section 21, a storage section 22 and a control section 23. FIG.

The interface unit 21 is an interface for inputting and outputting data. For example, the interface unit 21 is a NIC. Further, the interface unit 21 may be connected to an output device such as a display and an input device such as a keyboard.

The storage unit 22 is a storage device such as an HDD, an SSD, an optical disc, or the like. Note that the storage unit 22 may be a rewritable semiconductor memory such as a RAM, a flash memory, or an NVSRAM. The storage unit 22 stores the OS and various programs executed by the learning device 20 .

As shown in FIG. 7, the storage unit 22 stores model information 221 . The model information 221 is parameters and the like for constructing a model. For example, the model information 221 is weights and biases for constructing each neural network.

The model information 221 may be updated by the learning device 20 and passed to the extraction device 10 . The delivered model information 221 is stored as the model information 121 by the extraction device 10 .

The control unit 23 controls the learning device 20 as a whole. The control unit 23 is, for example, an electronic circuit such as a CPU, MPU, or GPU, or an integrated circuit such as an ASIC or FPGA. The control unit 23 also has an internal memory for storing programs defining various processing procedures and control data, and executes each processing using the internal memory.

The control unit 23 functions as various processing units by running various programs. For example, the controller 23 has a signal processor 231 , a loss calculator 232 and an updater 233 .

The signal processing unit 231 uses a model constructed from the model information 221 to extract the target speech from the mixed speech. The signal processor 231 has a generator 231a and an extractor 231b.

The generation unit 231a generates data in a predetermined format from the activity information representing the time when the target sound source emitted the sound and the mixed sound. Also, the extraction unit 231b uses the data generated by the generation unit 231a and the extraction network to extract the sound of the target sound source from the mixed sound.

The generation unit 231a and the extraction unit 231b perform the same processing as the generation unit 131a and the extraction unit 131b, respectively.

The loss calculator 232 calculates a loss function based on the sounds extracted by the extractor 231b. For example, the loss function is a signal-to-noise ratio ( _SiSNR ) between the sound ^ _xt extracted by the extraction unit 231b and the correct answer xt included in the learning data. Also, the loss function is not limited to the signal-to-noise ratio, and may be a signal-to-distortion ratio (SDR), MSE (Mean square error), or the like.

The updating unit 233 updates parameters of the NN for extraction so that the loss function calculated based on the sound extracted by the extracting unit 231b is optimized. The updating unit 233 can update the parameters by a known technique such as error backpropagation.

[Processing flow of the first embodiment]
FIG. 8 is a flow chart showing the processing flow of the extraction device according to the first embodiment. As shown in FIG. 8, first, the extraction device 10 generates data in a predetermined format from synthesized speech and activity information (step S101).

Next, the extraction device 10 extracts the target speech from the mixed speech using the generated data and the extraction network (step S102).

FIG. 9 is a flow chart showing the flow of processing of the learning device according to the first embodiment. As shown in FIG. 9, first, the learning device 20 generates data in a predetermined format from synthesized speech and activity information (step S201).

Next, the learning device 20 extracts the target speech from the mixed speech using the generated data and the extraction network (step S202).

Here, the learning device 20 calculates a loss function that optimizes the network (step S203). The learning device 20 then updates the network parameters so that the loss function is optimized (step S204).

When the learning device 20 determines that the parameters have converged (step S205, Yes), the processing ends. On the other hand, when the learning device 20 determines that the parameters have not converged (step S205, No), the learning device 20 returns to step S201 and repeats the process.

[Effects of the first embodiment]
As described above, the generation unit 131a generates data in a predetermined format from the activity information representing the time when the target sound source emitted the sound and the mixed sound. Also, the extraction unit 131b uses the data generated by the generation unit 131a and the extraction network to extract the sound of the target sound source from the mixed sound.

Thus, the extraction device 10 extracts the target sound using the activity information. Therefore, for example, in the extraction process by the extraction device 10, it is not necessary to register the speech of the target speaker in advance. In addition, the extraction processing by the extraction device 10 can reduce the influence of changes in the voice features of the target speaker. As a result, according to this embodiment, it is possible to accurately and easily extract the target speech from the mixed speech.

Conventionally, there have been various forms of data that serve as clues for speaker separation. For example, the speaker's voice for about 10 seconds, a picture of the speaker's face, a combination of the voice and the video, and the like are used as clues. Then, it is necessary to prepare a model according to each aspect.

In contrast, in the present embodiment, a model corresponding to activity information may be prepared regardless of the format of the data that serves as a clue.

The generation unit 131a generates a weighted sum of the output obtained by inputting the mixed sound to the auxiliary network and the activity information. Also, the extraction unit 131b extracts the sound of the target sound source from the mixed sound by inputting the mixed sound and the weighted sum to the extraction network. This method corresponds to ADEnet-auxiliary in FIG.

ADEnet-auxiliary can obtain the same performance as removing the discrepancy between the mixed speech in the speaker beam and the pre-registered target speaker's speech, especially if the overlap can be eliminated.

The generation unit 131a generates a combined vector that combines the activity information and the mixed sound, both of which are represented by vectors. Also, the extraction unit 131b extracts the sound of the target sound source from the mixed sound by inputting the connection vector to the extraction network. This method corresponds to ADEnet-input in FIG.

Since ADEnet-input does not require an auxiliary network, it can realize a simpler configuration than ADEnet-auxiliary.

The generating unit 131a further generates a weighted sum of the output obtained by inputting the mixed sound to the auxiliary network and the activity information. The extraction unit 131b extracts the sound of the target sound source from the mixed sound by inputting the connection vector and the weighted sum to the extraction network. This method corresponds to ADEnet-mix in FIG.

ADEnet-mix can reflect features of both joint vectors and weighted sums in extraction results.

[Experimental result]
An experiment conducted using this embodiment will be described. In the experiment, activity information for learning was prepared by adding noise to oracle speaker activity estimated based on teacher data. Note that the oracle speaker activity is an activity of a target speech estimated (extracted) from correct speech data (teacher data) using a speech segment extraction method (for example, Reference 6).
Reference 6: “https://github.com/wiseman/py-webrtcvad”

Specifically, overlapping time intervals were removed from the oracle speaker activity, and the start and end points of each time interval in which the target speaker was active were corrected with values sampled from the range of -1 second to 1 second.

First, FIG. 10 shows a comparison result of SDR (signal to distortion ratio) when oracle speaker activity is used as it is and when noise is added. FIG. 10 is a diagram showing experimental results.

Noisy activity training indicates whether or not noise is added to the training data. Also, Activity signal at test time is SDR depending on the presence or absence of overlap at the time of extraction and whether or not noise is added (Oracle or +Noise).

As shown in FIG. 10, some models of this embodiment may exceed the SDR of the speaker beam. Note that the SDR of the speaker beam is 9.4. Also, in the experiment of FIG. 10, LibriSpeech corpus (V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an asr corpus based on public domain audio books,” in Proc. of ICASSP'15, 2015, pp. 5206-5210.) are used.

Also, from FIG. 10, it can be said that adding noise to the training data (check Noisy activity training) improves the robustness of the model.

Next, we present data simulating a conference situation (Z. Chen, T. Yoshioka, L. Lu, T. Zhou, Z. Meng, Y. Luo, J. Wu, and J. Li, “Continuous speech separation: Dataset and analysis,” Proc. of ICASSP'20, pp. 7284-7288, 2020.) are shown in FIG. FIG. 11 is a diagram showing experimental results.

In the experiment of FIG. 11, activity information by TS-VAD described in reference 5 was used. FIG. 11 also shows cpWER (concatenated minimum-permutation word error rate), that is, evaluation values including errors due to diarization.

From FIG. 11, it can be said that adoption of ADEnet is more advantageous than adoption of ADEnet as the ratio of overlap increases.

[System configuration, etc.]
Also, each component of each device illustrated is functionally conceptual, and does not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution and integration of each device is not limited to the illustrated one, and all or part of them can be functionally or physically distributed and integrated in arbitrary units according to various loads and usage conditions. Can be integrated and configured. Furthermore, all or any part of each processing function performed by each device is realized by a CPU (Central Processing Unit) and a program analyzed and executed by the CPU, or hardware by wired logic can be realized as

Further, among the processes described in the present embodiment, all or part of the processes described as being automatically performed can be performed manually, or the processes described as being performed manually can be performed manually. All or part of this can also be done automatically by known methods. In addition, information including processing procedures, control procedures, specific names, and various data and parameters shown in the above documents and drawings can be arbitrarily changed unless otherwise specified.

[program]
As one embodiment, the extraction device 10 and the learning device 20 can be implemented by installing a program for executing the above-described audio signal extraction processing or learning processing as package software or online software on a desired computer. For example, the information processing device can function as the extraction device 10 by causing the information processing device to execute a program for the extraction process described above. The information processing apparatus referred to here includes a desktop or notebook personal computer. In addition, information processing devices include mobile communication terminals such as smart phones, mobile phones and PHS (Personal Handyphone Systems), and slate terminals such as PDAs (Personal Digital Assistants).

The extraction device 10 and the learning device 20 can also be implemented as a server device that uses a terminal device used by a user as a client and provides the client with a service related to the above-described audio signal extraction processing or learning processing. For example, the server device is implemented as a server device that receives a mixed speech signal as an input and provides a service of extracting the speech signal of the target speaker. In this case, the server device may be implemented as a web server, or may be implemented as a cloud that provides services through outsourcing.

FIG. 12 is a diagram illustrating an example of a computer that executes programs. The computer 1000 has a memory 1010 and a CPU 1020, for example. Computer 1000 also has hard disk drive interface 1030 , disk drive interface 1040 , serial port interface 1050 , video adapter 1060 and network interface 1070 . These units are connected by a bus 1080 .

The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012 . The ROM 1011 stores a boot program such as BIOS (Basic Input Output System). Hard disk drive interface 1030 is connected to hard disk drive 1090 . A disk drive interface 1040 is connected to the disk drive 1100 . A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100 . Serial port interface 1050 is connected to mouse 1110 and keyboard 1120, for example. Video adapter 1060 is connected to display 1130, for example.

The hard disk drive 1090 stores an OS 1091, application programs 1092, program modules 1093, and program data 1094, for example. That is, a program that defines each process of the extraction device 10 is implemented as a program module 1093 in which computer-executable code is described. Program modules 1093 are stored, for example, on hard disk drive 1090 . For example, the hard disk drive 1090 stores a program module 1093 for executing processing similar to the functional configuration of the extraction device 10 . Note that the hard disk drive 1090 may be replaced by an SSD.

Also, setting data used in the processing of the above-described embodiment is stored as program data 1094 in the memory 1010 or the hard disk drive 1090, for example. Then, the CPU 1020 reads out the program module 1093 and the program data 1094 stored in the memory 1010 and the hard disk drive 1090 to the RAM 1012 as necessary and executes them.

The program modules 1093 and program data 1094 are not limited to being stored in the hard disk drive 1090, but may be stored in a removable storage medium, for example, and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, program modules 1093 and program data 1094 may be stored in another computer connected via a network (LAN (Local Area Network), WAN (Wide Area Network), etc.). Program modules 1093 and program data 1094 may then be read by CPU 1020 through network interface 1070 from other computers.

10 extraction device 20 learning device 11, 21 interface unit 12, 22 storage unit 13, 23 control unit 121, 221 model information 131, 231 signal processing unit 131a, 231a generation unit 131b, 231b extraction unit

Claims (8)

a generating unit that generates data in a predetermined format from the mixed sound and activity information representing the time when the target sound source emitted the sound;
an extraction unit that extracts the sound of the target sound source from the mixed sound using the data generated by the generation unit and a neural network (NN) for extraction;
An extraction device comprising: The generating unit generates a weighted sum of an output obtained by inputting the mixed sound to the auxiliary NN and the activity information,
2. The extractor according to claim 1, wherein the extraction unit extracts the sound of the target sound source from the mixed sound by inputting the mixed sound and the weighted sum to the extraction neural network. Device. The generating unit generates a combined vector combining the activity information and the mixed sound, both of which are represented by vectors,
2. The extracting apparatus according to claim 1, wherein the extraction unit extracts the sound of the target sound source from the mixed sound by inputting the combination vector to the extraction neural network. The generating unit further generates a weighted sum of an output obtained by inputting the mixed sound to the auxiliary NN and the activity information,
4. The extracting device according to claim 3, wherein the extracting unit extracts the sound of the target sound source from the mixed sound by inputting the combination vector and the weighted sum to the NN for extraction. . An extraction method performed by an extraction device, comprising:
a generating step of generating data in a predetermined format from the mixed sound and the activity information representing the time when the target sound source emitted the sound;
an extracting step of extracting the sound of the target sound source from the mixed sound using the data generated in the generating step and a neural network (NN) for extraction;
An extraction method comprising: a generating unit that generates data in a predetermined format from the mixed sound and activity information representing the time when the target sound source emitted the sound;
an extraction unit that extracts the sound of the target sound source from the mixed sound using the data generated by the generation unit and a neural network (NN) for extraction;
an updating unit that updates parameters of the NN for extraction so that a loss function calculated based on the sound extracted by the extracting unit is optimized;
A learning device characterized by comprising: A learning method performed by a learning device, comprising:
a generating step of generating data in a predetermined format from the mixed sound and the activity information representing the time when the target sound source emitted the sound;
an extracting step of extracting the sound of the target sound source from the mixed sound using the data generated in the generating step and a neural network (NN) for extraction;
an update step of updating parameters of the NN for extraction so that a loss function calculated based on the sound extracted by the extraction step is optimized;
A learning method comprising: A program for causing a computer to function as the extraction device according to any one of claims 1 to 4 or the learning device according to claim 6.

Images