JP2022063080A - Computer and voice processing method - Google Patents

Abstract

To extract voice of a speaker included in sound while suppressing the influence of noise by using the sound and an image.SOLUTION: A computer stores input sound acquired from a sound collection device and an input image acquired from an imaging device in a storage device, calculates a speaker characteristic amount showing characteristics of voice of a target person by using the input sound, extracts face area images including the face of the target person by using the input image, specifies an utterance period estimated to be a period in which the target person utters by using a plurality of face area images, and extracts estimated voice of the target person in the utterance period from the input sound by using the speaker characteristic amount and the utterance period.SELECTED DRAWING: Figure 3

Description

The present invention relates to a voice processing technique for extracting a speaker's voice included in a sound while suppressing the influence of noise by using the input sound and an image.

In voice recognition in which transcription is performed from voice data collected using a microphone or the like, it is desirable to remove noise components from the input voice as much as possible in order to improve accuracy. On the other hand, the technique described in Patent Document 1 is known.

In Patent Document 1, the open / closed state of the speaker's mouth is identified from the image acquired by using the camera, and the voice at the time of opening the speaker (mouth opening / closing operation period) is treated as a signal voice including the speaker's voice. , A control program that treats the voice of the speaker when the mouth is closed as noise voice is described. By distinguishing the speaker's voice and noise using images, highly accurate voice recognition is realized.

Japanese Unexamined Patent Publication No. 2019-8134

However, the voice at the time of opening the speaker may contain noise other than the speaker. In this case, it is difficult to extract the speaker's voice without noise from the speaker's opening sound.

The present invention has been made to solve the above problems, and provides a voice processing system that accurately extracts a speaker's voice from the sound during the opening / closing operation period of the speaker's mouth.

A typical example of the invention disclosed in the present application is as follows. That is, it is a computer that extracts the voice of the target person from the input sound collected by the sound collector, and includes a calculation device, a storage device connected to the calculation device, and a connection interface connected to the calculation device. , The sound collecting device and the image pickup device that acquires the image of the target person are connected via the connection interface, and the arithmetic device stores the input sound and the image in the storage device, and stores the input sound and the input sound. Using, the speaker characteristic amount indicating the voice characteristic of the target person is calculated, the face area image including the face of the target person is extracted using the image, and the plurality of face area images are used. The speech period estimated to have been spoken by the target person is specified, and the estimated voice of the target person in the speech period is extracted from the input sound by using the speaker feature amount and the speech period. The estimated voice of the target person is stored in the storage device.

According to the present invention, the speaker's voice can be accurately extracted from the sound during the opening / closing operation period of the speaker's mouth. Issues, configurations and effects other than those mentioned above will be clarified by the description of the following examples.

It is a figure which shows an example of the structure of the voice processing system of Example 1. FIG. It is a figure which shows an example of the functional structure of the server of Example 1. FIG. It is a figure which shows an example of the functional structure of the server of Example 1. FIG. It is a figure which shows an example of the detailed structure of the speech enhancer of Example 1. FIG. It is a figure which shows the image of the time series data of the complex feature quantity output by the feature coupling part of Example 1. FIG. It is a flowchart explaining an example of the learning processing executed by the voice processing system of Example 1. FIG. It is a flowchart explaining an example of the estimation processing performed by the voice processing system of Example 1. FIG.

Hereinafter, examples of the present invention will be described with reference to the drawings. However, the present invention is not limited to the description of the examples shown below. It is easily understood by those skilled in the art that a specific configuration thereof can be changed without departing from the idea or purpose of the present invention.

In the configuration of the invention described below, the same or similar configurations or functions are designated by the same reference numerals, and duplicate description will be omitted.

The notations such as "first", "second", and "third" in the present specification and the like are attached to identify the components, and are not necessarily limited in number or order.

The position, size, shape, range, etc. of each configuration shown in the drawings and the like may not represent the actual position, size, shape, range, etc., in order to facilitate understanding of the invention. Therefore, the present invention is not limited to the position, size, shape, range, etc. disclosed in the drawings and the like.

FIG. 1 is a diagram showing an example of the configuration of the voice processing system 100 of the first embodiment.

The voice processing system 100 includes a server 101, an image pickup device 102, and a sound collector 103.

The server 101, the image pickup device 102, and the sound collector 103 are connected to each other directly or via a network. The network is, for example, WAN (Wide Area Network) and LAN (Local Area Network).

The server 101 may include the image pickup device 102 and the sound collecting device 103. The configuration of the voice processing system 100 shown in FIG. 1 is an example and is not limited thereto.

The voice processing system 100 acquires sounds and images from the space in which the speaker 110 exists, and extracts the voice of the speaker 110 from the acquired sounds. In addition to the speaker 110, there is a noise sound source 111 that emits a sound (noise) different from the voice of the speaker 110 in the space. The sound emitted from the noise sound source 111 is, for example, a car engine sound, an environmental sound, a voice of a person different from the speaker 110, or the like.

The image pickup device 102 is a device for acquiring an image, and is, for example, a camera, a depth measuring instrument, or the like. The image pickup apparatus 102 of the first embodiment acquires the face area image 112 including the mouth area of the speaker 110. The face area image 112 is, for example, an RGB image, a depth map, or the like. The voice processing system 100 may have a plurality of image pickup devices 102 that acquire face region images 112 having different characteristics.

The sound collecting device 103 is a device that collects sound in the installed space, and is, for example, a monaural microphone, a microphone array, or the like. The sound collecting device 103 of the first embodiment collects sounds (mixed sounds) emitted from the speaker 110 and the noise sound source 111.

The server 101 uses the face area image 112 and the mixed sound to extract only the voice of the speaker 110 from the sound collection voice. In the present specification, the function of extracting the voice of the speaker 110 from the mixed sound is described as a speech enhancement function.

The server 101 has a processor 120, a storage device 121, and a connection interface 122. The hardware configurations are connected to each other via the internal bus 123.

The processor 120 is an arithmetic unit such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit), and executes a program stored in the storage device 121. The processor 120 operates as a module that realizes a speech enhancement function by executing processing according to a program. In the following description, when the process is described with the module as the subject, it is shown that the processor 120 is executing the program that realizes the module.

The storage device 121 stores a program executed by the processor 120 and information used by the program. The storage device 121 is also used as a working area for the processor 120. The storage device 121 may be a non-temporary storage device or a temporary storage device. The storage device 121 is, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), an HDD (Hard Disk Drive), a flash memory, or the like.

The storage device 121 of the first embodiment stores a program that realizes the estimation unit 130 and the learning unit 131. The learning unit 131 learns the speech enhancer 202 (see FIG. 2) incorporated in the estimation unit 130 by using machine learning and deep learning. The estimation unit 130 is a functional unit that realizes a speech enhancement function. The estimation unit 130 extracts the voice of the target person (speaker) from the sound input by using the speech enhancer 202 learned by the learning unit 131.

The connection interface 122 is an interface for connecting to an external device such as an image pickup device 102 and a sound collecting device 103. The connection interface 122 is, for example, a network interface and an IO interface.

2A and 2B are diagrams showing an example of the functional configuration of the server 101 of the first embodiment.

FIG. 2A shows the details of the functional configuration of the estimation unit 130. The estimation unit 130 accepts the continuous image data 211 and the sound data 212 as inputs, and outputs the sound data 213 including only the estimated voice of the speaker 110 and the estimated speaker ID 214 which is the identification information of the speaker 110. Here, the continuous image data 211 is time-series data composed of a plurality of face region images 112 acquired by the image pickup apparatus 102 at regular intervals. The sound data 212 is data related to the sound collected by the sound collecting device 103.

It is assumed that the sampling rate of the face area image 112 included in the continuous image data 211 and the sampling rate of the sound data 212 are different, but are synchronized in time.

The continuous image data 211 and the sound data 212 do not have to be separate data. For example, moving image data including images and sounds may be used.

The estimation unit 130 includes an image preprocessing unit 200, a sound preprocessing unit 201, and a speech enhancer 202.

The image preprocessing unit 200 extracts an image including the mouth region or the face region of the speaker 110 from each face region image 112 included in the continuous image data 211. The image preprocessing unit 200 outputs the time series data of the extracted image to the speech enhancer 202. The image preprocessing unit 200 may normalize the pixel value of the extracted image.

The sound preprocessing unit 201 calculates sound features such as a voice spectrum and a merkepstrum by executing arithmetic processing such as a short-time Fourier transform on the sound data 212. Here, the sound feature amount for each sampling rate is calculated. The sound preprocessing unit 201 outputs sound feature amount data including the sound feature amount of each sampling rate to the speech enhancer 202. The sound preprocessing unit 201 may normalize the sound feature amount.

The speech enhancer 202 outputs the sound data 213 and the estimated speaker ID 214 by using the time series data of the image output from the image preprocessing unit 200 and the sound feature amount data output from the sound preprocessing unit 201. The speech enhancer 202 may output only the sound data 213 as the final output. The speech enhancer 202 is a model defined from a plurality of learnable parameters, and model information (not shown) for storing the parameters is stored in the storage device 121.

FIG. 2B shows the details of the functional configuration of the learning unit 131. The learning unit 131 receives the sample 220 included in the learning data as an input, and learns the parameters that define the speech enhancer 202.

The sample 220 includes continuous image data 221, sound data 223, and speaker ID 224.

The continuous image data 221 is time-series data composed of images 222 at regular intervals. Image 222 is an image including a speaker's face area or mouth area. The image 222 may be an image acquired from the speaker in advance, or may be an image acquired by the image pickup apparatus 102 during the operation of the voice processing system 100.

The sound data 223 is data including the voice of the speaker. The sound data 223 may include only the voice of the speaker, or may include the voice of the speaker and other sounds. The sound data 223 may be speaker voice data collected in an environment with little noise, or single channel voice data collected using a monaural microphone.

It is assumed that the sampling rate of the image 222 included in the continuous image data 221 and the sampling rate of the sound data 223 are different, but are synchronized in time.

The continuous image data 221 and the sound data 223 do not have to be separate data. For example, moving image data including images and sounds may be used.

The speaker ID 224 is speaker identification information. In the first embodiment, it is assumed that a number is assigned as the speaker ID 224. In this case, different numbers are assigned to each speaker.

The training data includes a plurality of samples 220 of the data structure as described above. For example, a sample 220 having a number of 1000 units or 10000 units is included in the training data.

The learning unit 131 includes an image preprocessing unit 200, a sound preprocessing unit 201, a speech enhancer 202, a mixed speech generation unit 203, a voice error calculation unit 204, a voice error reflection unit 205, a speaker error calculation unit 206, and a speaker. The error reflection unit 207 is included.

The image preprocessing unit 200, the sound preprocessing unit 201, and the speech enhancement unit 202 are the same as the modules included in the estimation unit 130.

The mixed voice generation unit 203 generates sound data 227 related to the mixed voice by using the sound data 223 related to the speaker's voice and the sound data 225 related to the interfering voice. For example, the mixed voice generation unit 203 generates mixed voice by adding or weighting the speaker's voice and the interfering voice.

The mixed voice generation unit 203 randomly selects one sample 220 from the samples 220 including the speaker ID 224 different from the speaker ID 224 of the input sample 220, and interferes with the sound data 223 included in the selected sample 220. It is used as voice sound data 225. The number of samples 220 to be selected may be two or more. In this case, the mixed voice includes the voices of a plurality of people. Note that sound data related to the interference sound may be input as data different from the learning data. For example, sound data including environmental sounds can be considered.

In speech enhancement, a sound including the voice of the speaker 110 and various noises is collected, and it is necessary to extract the voice of the speaker 110 from the sound. Therefore, in order to improve the voice extraction accuracy of the speaker 110, a mixed voice is generated, and the speech enhancer 202 is learned using the mixed voice. A method for generating mixed speech during learning of the speech enhancer 202 is a known method.

The voice error calculation unit 204 calculates the error of the sound data 223 and the sound data 213. When the sound data 213 includes a voice waveform, the voice error calculation unit 204 calculates an error between the two voice waveforms based on a known error scale such as a square error or a norm error. When the sound data 213 includes a voice spectrum, the voice error calculation unit 204 converts the voice included in the sound data 223 into a voice spectrum and two voice spectra based on a known error scale such as a square error. Calculate the error of.

The voice error reflection unit 205 updates the parameters of the speech enhancer 202 so that the error calculated by the voice error calculation unit 204 becomes smaller by using a known method such as an error back propagation method.

The speaker error calculation unit 206 calculates an error between the speaker ID 224 and the estimated speaker ID 214 based on a known error scale such as a cross entropy error.

The speaker error reflection unit 207 updates the parameters of the speech enhancer 202 so that the error calculated by the speaker error calculation unit 206 becomes small by using a known method such as an error back propagation method.

FIG. 3 is a diagram showing an example of a detailed configuration of the speech enhancer 202 of the first embodiment.

The speech enhancer 202 includes an image feature extraction unit 300, a sound feature extraction unit 301, a feature coupling unit 302, a synchronization estimation unit 303, a speaker voice estimation unit 304, a sound feature conversion unit 305, and a speaker identification unit 306.

The image feature extraction unit 300 extracts an image feature amount (feature amount, feature expression, etc.) from each image included in the time-series data of the image processed by the image preprocessing unit 200. The image feature extraction unit 300 outputs a temporally continuous image feature amount to the feature coupling unit 302. The image feature extraction unit 300 is configured by using, for example, a CNN (Convolutional Neural Network) or the like. The CNN constituting the image feature extraction unit 300 includes parameters to be learned.

The sound feature extraction unit 301 extracts a sound feature amount (feature amount, feature expression, etc.) from the sound feature amount data processed by the sound preprocessing unit 201. The sound feature extraction unit 301 outputs data including temporally continuous sound feature amounts (time-series data of sound feature amounts) to the feature coupling unit 302. The sound feature extraction unit 301 is configured by using CNN, RNN (Recurrent Neural Network) or the like. The CNN or RNN constituting the sound feature extraction unit 301 includes parameters to be learned.

The feature combination unit 302 generates a composite feature amount by combining the image feature amount and the sound feature amount in a time-synchronized format. Specifically, the feature combining unit 302 generates a composite feature amount by combining an image feature amount and a sound feature amount at predetermined time intervals (time steps). The feature coupling unit 302 outputs the composite feature data including the temporally continuous composite feature data to the synchronization estimation unit 303 and the sound feature conversion unit 305.

In general, the sampling rate of an image is sparser than the sampling rate of audio, so it is necessary to devise in order to combine the image feature amount and the sound feature amount in a time-synchronized format. In this embodiment, temporal synchronization is realized by widening the convolution area in the time direction of the CNN constituting the sound feature extraction unit 301.

The sound feature conversion unit 305 calculates an index indicating the content of the speaker's voice in the sound input to the sound preprocessing unit 201 based on the time-series data of the sound feature amount extracted by the sound feature extraction unit 301. , Identify the speech status based on the index. Further, the sound feature conversion unit 305 converts the input time-series data of the sound feature amount into the time-series data of the sound feature amount reflecting the utterance situation. The sound feature conversion unit 305 includes a time step similarity calculation unit 310, an utterance status identification unit 311, a weight calculation unit 312, and a weight reflection unit 313.

The time-step similarity calculation unit 310 calculates the similarity of the complex features between the time steps. The similarity of the complex features between the time steps is used as an index for identifying the utterance situation. The time step similarity calculation unit 310 is configured by using, for example, a linear layer or the like. The linear layer constituting the time-step similarity calculation unit 310 includes parameters to be learned. When the composite feature quantity is a vector, the time step similarity calculation unit 310 calculates the inner product of the vectors as the similarity.

The utterance status identification unit 311 identifies the utterance status of the speaker at each time step based on the similarity between the time steps calculated by the time step similarity calculation unit 310. In the first embodiment, the utterance status identification unit 311 includes a case where only the speaker's voice is included (first case), a case where the speaker's voice and noise are included (second case), and a case where each time step includes the speaker's voice and noise. Identify which case corresponds to the case containing only noise (third case). When the second case is applicable, the utterance status identification unit 311 may identify the mixture ratio of the speaker's voice and noise together. The utterance situation identification unit 311 is configured by using, for example, a linear layer or the like. The linear layer constituting the utterance status identification unit 311 includes parameters to be learned.

The weight calculation unit 312 calculates the weight of the sound feature amount of each time step based on the identification result of the utterance status identification unit 311. The weight calculation unit 312 is configured by using, for example, a linear layer or the like. The linear layer constituting the weight calculation unit 312 includes parameters to be learned. In the first embodiment, the weight calculation unit 312 calculates a large weight for the time step corresponding to the first case, calculates a medium-sized weight for the time step corresponding to the second case, and obtains a third weight. Calculate a small weight for the time step that corresponds to the case.

The weight reflection unit 313 generates time-series data of the weighted sound feature amount by using the weight calculated by the weight calculation unit 312 and the time-series data of the sound feature amount output from the sound feature extraction unit 301. .. For example, the weight reflection unit 313 calculates the time-series data of the weighted sound feature amount by multiplying the time direction of the sound feature amount by the weight of each time step. The weight reflection unit 313 outputs the time-series data of the weighted sound feature amount to the speaker identification unit 306. The weighting target may be a composite feature amount.

The speaker identification unit 306 identifies the speaker based on the time-series data of the weighted sound feature amount, and outputs the estimated speaker ID 214 as the identification result. The speaker identification unit 306 includes a speaker feature extraction unit 320 and a speaker estimation unit 321.

The speaker feature extraction unit 320 extracts the speaker feature amount 330 from the time-series data of the weighted sound feature amount, and outputs the speaker feature amount 330 to the speaker estimation unit 321 and the speaker voice estimation unit 304. The speaker feature extraction unit 320 is configured by using, for example, CNN, RNN, or the like. The CNN and RNN constituting the speaker feature extraction unit 320 include parameters to be learned.

The time-series data of the weighted sound feature amount is the time-series data of the feature amount in which the sound feature amount of the time step in which the ratio of the speaker's voice included in the sound is high is emphasized. Therefore, the speaker feature amount 330 extracted using the time-series data of the weighted sound feature amount has a better voice component of the speaker than the speaker feature amount extracted using the time-series data of the sound feature amount. It is expected that the features will be reflected.

The speaker estimation unit 321 estimates the speaker using the speaker feature amount 330, and outputs the estimated speaker ID 214 as the estimation result. The speaker estimation unit 321 is composed of, for example, a linear layer or the like. The linear layer constituting the speaker estimation unit 321 includes parameters to be learned.

The synchronous estimation unit 303 estimates the period (estimated period) in which the mouth opening / closing operation accompanying the speaker's speech is performed based on the time-series data of the image feature amount included in the time-series data of the composite feature amount. In addition, the feature amount related to the opening / closing operation of the mouth during the estimated period is calculated. The synchronization estimation unit 303 extracts the sound feature amount of the period that matches or synchronizes with the estimation period from the time series data of the sound feature amount. The synchronous estimation unit 303 outputs the extracted sound feature amount to the speaker voice estimation unit 304. The synchronization estimation unit 303 is configured by using, for example, an RNN or the like. The RNN constituting the synchronization estimation unit 303 includes parameters to be learned.

The speaker voice estimation unit 304 extracts the estimated voice of the speaker by using the sound feature amount of the estimation period input from the synchronous estimation unit 303 and the speaker feature amount 330 input from the speaker identification unit 306. Sound data 213 is output as the extraction result. The speaker voice estimation unit 304 is configured by using, for example, an RNN and a linear layer. The RNN and the linear layer constituting the speaker voice estimation unit 304 include parameters to be learned.

A conventional speech enhancer includes only an image feature extraction unit, a sound feature extraction unit, a feature combination unit, a synchronization estimation unit, and a speaker voice estimation unit, and estimates the voice of a speaker during a period synchronized with the opening / closing operation of the mouth. Output as audio.

On the other hand, in the speech enhancer 202 of this embodiment, in addition to the image feature extraction unit 300, the sound feature extraction unit 301, the feature coupling unit 302, the synchronization estimation unit 303, and the speaker voice estimation unit 304, the sound feature conversion unit 305 And the speaker identification unit 306. The sound feature conversion unit 305 gives weights so as to emphasize the amount of sound features during the period (time step) in which the speaker's voice is highly pure. This functions as a means for selecting the extraction section of the sound feature amount. The speaker identification unit 306 can extract the speaker feature amount 330 with high accuracy by using the time-series data of the weighted sound feature amount. Since the speaker voice estimation unit 304 can also identify the sound quality (height, speaking speed, tone color, etc.) of the speaker's voice based on the speaker feature amount 330, the speaker excluding the interference sound from the sound during the estimation period. Sound can be extracted. As described above, the speaker voice estimation unit 304 can more accurately extract the speaker voice from the sound during the period synchronized with the opening / closing operation of the mouth by using the speaker feature amount 330 as a filter.

Therefore, the estimation accuracy of the voice processing system 100 of the first embodiment can be expected to be higher than the estimation accuracy of the conventional system.

For each module included in the speech enhancer 202, a plurality of modules may be combined into one module, or one module may be divided into a plurality of modules for each function.

Next, the details of the processing of the sound feature conversion unit 305 will be described with reference to FIG. FIG. 4 is a diagram showing an image of time-series data of the complex feature amount output by the feature coupling portion 302 of the first embodiment.

The first line of the time-series data 400 of the composite feature amount shown in FIG. 4 represents the time step, the second line represents the sound feature amount of the time step, and the third line represents the image feature amount of the time step.

In FIG. 4, the time step is set as a numerical value from “1” to “8” indicating the order of the complex feature amount in the time series data 400.

The sound features of each time step are actually given as a vector representation, but for the sake of explanation, they show the qualitative properties of the vector representation. The sound features of the time steps "1" to "3" indicate that the features include only the voice of the speaker. The sound features of the time steps "4" to "6" indicate that the features include the speaker's voice and the interference sound. Further, it is shown that the sound feature amounts of the time steps "7" and "8" are feature amounts including only the interference sound. It may be a feature amount representing the ratio of the speaker's voice and the interference sound.

The image features of each time step are actually given as a vector representation, but for the sake of simplicity, they show the qualitative properties that the vector representation represents. The image feature amounts of the time steps "1" to "6" indicate that the feature amount indicates that the speaker's mouth is opened and closed with the utterance. The image feature quantities of the time steps "7" and "8" indicate that the speaker is closed because he / she is not speaking.

The time-series data 400 of the composite feature amount including the sound feature amount and the image feature amount as shown in FIG. 4 can be divided into the following cases.

In the time steps "1", "2", and "3", only the voice of the speaker is present, and the speaker's mouth is opened and closed. Therefore, the time steps "1", "2", and "3" are classified into the first case. In the time steps "4", "5", and "6", the speaker's voice and the interference sound are present, and the speaker's mouth is opened and closed. Therefore, the time steps "4", "5", and "6" are classified into the second case. In the time steps "7" and "8", only the interference sound is present and the speaker's mouth is closed. Therefore, the time steps "7" and "8" are classified into the third case.

The sound feature conversion unit 305 classifies each time step into one of the first case, the second case, and the third case by using the time series data of the complex feature amount. Specifically, the sound feature conversion unit 305 classifies case transitions between time steps based on the degree of similarity of complex features (sound features and image features) between time steps.

The similarity between the sound features and the image features calculated by the time step similarity calculation unit 310 is as follows.

In the case of the transition between the first case and the third case, the sound of each time step is different, so that the similarity of the sound features is low. In the case of the transition between the first case and the second case, the voice of the speaker is included in both cases, but the interference sound is also included in the second case, so that the similarity of the sound features is medium. In the case of the transitions of the second case and the third case, the interference sound is included in both cases, but since the speaker's voice is also included in the second case, the similarity of the sound features is medium. In the case of transitions in the same case, the similarity of sound features is assumed to be high.

In the case of the transition of the first case and the second case, since the mouth is opened and closed, the similarity of the image feature quantities is high. In the case of the transition of the first case and the third case, and the transition of the second case and the third case, the mouth is opened and closed in the first case and the second case, and the mouth is closed in the third case. , The similarity of image features is low. In the case of transitions in the same case, the similarity of image features is high.

Summarizing the above, the characteristics are as follows. In the case of the transition between the first case and the third case, the similarity of the sound features and the similarity of the image features are both low. In the case of the transition between the first case and the second case, the similarity of the sound features is medium, and the similarity of the image features is high. In the case of the transition between the second case and the third case, the similarity of the sound features is medium, and the similarity of the image features is low.

The utterance status identification unit 311 classifies the cases of each time step based on the nature of the similarity in the transition between the cases as described above.

The following classification method may be adopted. The utterance status identification unit 311 identifies a time step in which the mouth is closed based on the image feature amount, and classifies the time step into the third case. Next, the utterance status identification unit 311 sets the sound feature amount of the time step classified in the third case as the reference sound feature amount. When there are a plurality of time steps classified into the third case, it is conceivable to set a statistical value such as an average value of the sound features of each time step as the reference sound feature. Next, the utterance status identification unit 311 calculates the degree of similarity between the sound feature amount of the unclassified time step and the reference sound feature amount. Next, the utterance status identification unit 311 classifies the unclassified time step based on the comparison result of the similarity and the threshold value. For example, the utterance status identification unit 311 classifies a time step having a similarity smaller than the threshold value into the first case, and a time step having a similarity equal to or higher than the threshold value into the second case.

The utterance status identification unit 311 may be configured by using a learnable linear layer. In this case, the utterance status identification unit 311 converts the composite feature amount into a feature amount for which the case can be easily identified, and classifies the cases of each time step based on the feature amount. For example, the utterance status identification unit 311 classifies the cases of each time step by using the arrangement of the converted features in the feature space and the norm distance. The norm distance of the same case is small, and the norm distance of different cases is large. In addition, the features of the same case are concentrated in a specific area of the feature space.

The weight calculation unit 312 calculates a large weight for the sound feature amount or the composite feature amount of the time step classified in the first case, and calculates the sound feature amount or the composite feature amount of the time step classified in the second case. On the other hand, a medium weight is calculated, and a small weight is calculated for the sound feature amount or the compound feature amount of the time step classified in the third case.

The classification of the utterance situation cases shown in FIG. 4 is an example and is not limited to this.

The sound feature conversion unit 305 emphasizes the feature amount of the time step including the speaker's voice, and suppresses the feature amount of the time step not including the speaker's voice in a time series of the sound feature amount. Convert the data. As a result, the speaker identification unit 306 can extract the speaker feature amount 330 that well reflects the voice of the speaker.

Next, the learning process and the estimation process executed by the speech processing system 100 will be described.

FIG. 5 is a flowchart illustrating an example of learning processing executed by the voice processing system 100 of the first embodiment.

When the learning unit 131 receives the execution instruction or the learning data is input, the learning unit 131 starts the learning process described below.

The learning unit 131 accepts the input of learning data (step S501). For example, the learning unit 131 accepts input of learning data from a user terminal connected via the connection interface 122. The learning unit 131 stores the received learning data in the storage device 121.

Next, the learning unit 131 reads one sample 220 from the learning data (step S502). At this time, the learning unit 131 reads out the sound data 223 included in the other sample 220 as the sound data 225 of the interference voice. The sample 220 may be randomly selected or may be selected based on a preset policy.

Next, the learning unit 131 executes preprocessing on the continuous image data 221 and the sound data 223 (step S503).

Specifically, the image preprocessing unit 200 executes preprocessing on the image 222 included in the continuous image data 221. Further, the mixed voice generation unit 203 generates the sound data 227 of the mixed voice using the sound data 223 and 225, and the sound preprocessing unit 201 executes preprocessing on the sound data 227.

Next, the learning unit 131 outputs the sound data 213 and the estimated speaker ID 214 using the continuous image data 221 and the sound data 223 on which the preprocessing has been executed (step S504). Specifically, the speech enhancer 202 executes the following processing.

The feature combining unit 302 has a time-series synchronized format of the time-series data of the image feature amount extracted by the image feature extraction unit 300 and the time-series data of the sound feature amount extracted by the sound feature extraction unit 301. By combining, time-series data of complex features are generated.

The sound feature conversion unit 305 outputs the time-series data of the weighted sound features using the time-series data of the composite features.

The speaker identification unit 306 extracts the speaker feature amount 330 as an intermediate output from the time-series data of the weighted sound feature amount. Further, the speaker identification unit 306 estimates the speaker based on the speaker feature amount 330, and outputs the estimated speaker ID 214 as the estimation result.

The synchronization estimation unit 303 extracts the sound feature amount of the time step corresponding to the period during which the mouth opening / closing operation is performed. The speaker voice estimation unit 304 uses the speaker feature amount 330 to extract and output sound data 213 from the sound feature amount of an arbitrary time step extracted by the synchronous estimation unit 303.

The above is a description of the process executed by the speech enhancer 202.

Next, the learning unit 131 calculates the error of the sound data 213 and the sound data 223 and the error of the estimated speaker ID 214 and the speaker ID 224 (step S505).

Specifically, the voice error calculation unit 204 calculates the error of the sound data 213 and the sound data 223, and the speaker error calculation unit 206 calculates the error of the estimated speaker ID 214 and the speaker ID 224.

Next, the learning unit 131 reflects each error on the speech enhancer 202 (step S506).

Specifically, the voice error reflecting unit 205 updates the learning target parameters of each module of the voice enhancer 202 based on the errors of the sound data 213 and the sound data 223, and the speaker error reflecting unit 207 estimates. Based on the error of the speaker ID 214 and the speaker ID 224, the parameters to be learned of each module of the voice enhancer 202 are updated.

In the first embodiment, the image feature extraction unit 300, the sound feature extraction unit 301, the synchronous estimation unit 303, the speaker voice estimation unit 304, and the sound feature conversion unit 305 (time step similarity calculation unit 310, utterance status identification unit 311, And the parameters of the weight calculation unit 312) and the speaker identification unit 306 (speaker feature extraction unit 320 and speaker estimation unit 321) are updated.

Next, the learning unit 131 determines whether or not to end the learning (step S507).

For example, when the reduction width of the error becomes smaller than the threshold value and the error cannot be reduced any more, the learning unit 131 ends the learning. The user may determine the end of learning.

If it is determined that the learning is not completed, the learning unit 131 returns to step S502 and executes the same process.

When it is determined that the learning is finished, the learning unit 131 outputs the learning result to the estimation unit 130 (step S508), and then ends the learning process.

Specifically, the learning unit 131 outputs the parameters of each module of the speech enhancer 202 to the estimation unit 130.

By the learning process, the error of the sound data 213 and the sound data 223 output by the speech enhancer 202 and the error of the estimated speaker ID 214 and the speaker ID 224 are reduced. As described above, in the learning process, the parameters of the synchronous estimation unit 303, the speaker voice estimation unit 304, the sound feature conversion unit 305, and the speaker identification unit 306 are updated together.

FIG. 6 is a flowchart illustrating an example of estimation processing executed by the voice processing system 100 of the first embodiment.

When the estimation unit 130 receives the execution instruction or the data is input, the estimation unit 130 starts the estimation process described below. It is assumed that the voice processing system 100 is operating in the environment as shown in FIG.

The estimation unit 130 acquires continuous image data 211 from the image pickup device 102, and also acquires sound data 212 from the sound collector 103 (step S601). The estimation unit 130 stores the continuous image data 211 and the sound data 212 in the storage device 121.

Next, the estimation unit 130 executes preprocessing on the continuous image data 211 and the sound data 212 (step S602). The process of step S602 is the same as the process of step S503.

Next, the estimation unit 130 outputs the sound data 213 and the estimated speaker ID 214 using the continuous image data 211 and the sound data 212 for which the preprocessing has been executed (step S603), and then ends the estimation process.

Specifically, the speech enhancer 202 reflecting the learning result outputs the sound data 213 and the estimated speaker ID 214. The process of step S603 is the same process as the process of step S504.

The estimated voice included in the sound data 213 is a voice very similar to the voice of the speaker 110 in which the sound of the noise sound source 111 is suppressed. The estimation unit 130 may perform transcription by inputting the sound data 213 into a known voice recognizer.

The present invention is not limited to the above-described embodiment, and includes various modifications. Further, for example, the above-described embodiment describes the configuration in detail in order to explain the present invention in an easy-to-understand manner, and is not necessarily limited to the one including all the described configurations. Further, it is possible to add, delete, or replace a part of the configuration of each embodiment with other configurations.

Further, each of the above configurations, functions, processing units, processing means and the like may be realized by hardware by designing a part or all of them by, for example, an integrated circuit. The present invention can also be realized by a software program code that realizes the functions of the examples. In this case, a storage medium in which the program code is recorded is provided to the computer, and the processor included in the computer reads out the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the function of the above-described embodiment, and the program code itself and the storage medium storing it constitute the present invention. Examples of the storage medium for supplying such a program code include a flexible disk, a CD-ROM, a DVD-ROM, a hard disk, an SSD (Solid State Drive), an optical disk, a magneto-optical disk, a CD-R, and a magnetic tape. Non-volatile memory cards, ROMs, etc. are used.

In addition, the program code that realizes the functions described in this embodiment can be implemented in a wide range of programs or script languages such as assembler, C / C ++, perl, Shell, PHP, Python, and Java (registered trademark).

Further, by distributing the program code of the software that realizes the functions of the embodiment via the network, the program code is stored in a storage means such as a hard disk or a memory of a computer or a storage medium such as a CD-RW or a CD-R. The processor included in the computer may read and execute the program code stored in the storage means or the storage medium.

In the above-described embodiment, the control lines and information lines show what is considered necessary for explanation, and do not necessarily indicate all the control lines and information lines in the product. All configurations may be interconnected.

100 Voice processing system 101 Server 102 Image pickup device 103 Sound collector 110 Speaker 111 Noise sound source 112 Face area image 120 Processor 121 Storage device 122 Connection interface 123 Internal bus 130 Estimator unit 131 Learning unit 200 Image preprocessing unit 201 Sound preprocessing unit 202 Voice enhancer 203 Mixed voice generation unit 204 Voice error calculation unit 205 Voice error reflection unit 206 Speaker error calculation unit 207 Speaker error reflection unit 211, 221 Continuous image data 212, 213, 223, 225, 227 Sound data 214 Estimate Speaker ID
220 Sample 222 Image 224 Speaker ID
300 Image feature extraction unit 301 Sound feature extraction unit 302 Feature coupling unit 303 Synchronous estimation unit 304 Speaker voice estimation unit 305 Sound feature conversion unit 306 Speaker identification unit 310 Time step similarity calculation unit 311 Speech status identification unit 312 Weight calculation Part 313 Weight reflection part 320 Speaker feature extraction part 321 Speaker estimation part 330 Speaker feature amount

Claims (14)

It is a computer that extracts the voice of the target person included in the sound collected by the sound collector.
It includes an arithmetic unit, a storage device connected to the arithmetic unit, and a connection interface connected to the arithmetic unit.
The sound collector and the image pickup device that acquires an image of the target person are connected via the connection interface.
The arithmetic unit is
The input sound acquired from the sound collecting device and the input image acquired from the imaging device are stored in the storage device.
Using the input sound, a speaker feature amount indicating the characteristics of the voice of the target person is calculated.
Using the input image, a face area image including the face of the target person is extracted.
Using the plurality of facial area images, the utterance period estimated to have been spoken by the target person was specified, and the utterance period was specified.
Using the speaker feature amount and the utterance period, the estimated voice of the target person in the utterance period is extracted from the input sound, and the extracted estimated voice of the target person is stored in the storage device. Characterized computer. The computer according to claim 1.
The arithmetic unit is
Using the input sound and the plurality of input images, the utterance status of the target person at each time step in the input sound is identified.
Based on the utterance status of the target person in each time step, the input sound is converted so as to emphasize the time step including the voice of the target person.
A computer characterized in that the speaker feature amount is calculated using the converted input sound. The computer according to claim 2.
The arithmetic unit is
From the input sound, a first time-series data consisting of sound features for each time step is generated.
From the plurality of face area images, a second time-series data consisting of an image feature amount for each time step is generated.
Using the first time-series data and the second time-series data, an index indicating the content of the voice of the target person in the input sound is calculated.
Based on the index, the utterance status of the target person at each time step is identified.
The weight according to the utterance situation of the target person in each time step is calculated.
Using the weights, the first time series data is transformed.
A computer characterized in that the speaker feature amount is calculated using the converted first time-series data. The computer according to claim 3.
The arithmetic unit is
The similarity between the sound features and the image features between the time steps was calculated as the index.
A computer characterized by identifying the utterance status of the target person in each time step based on the similarity of the image features between the time steps and the similarity of the sound features between the time steps. The computer according to claim 3.
The arithmetic unit is
Based on the image feature amount, the reference time step in which the target person is closed is specified.
Based on the sound feature amount corresponding to the reference time step, the reference sound feature amount is calculated.
The degree of similarity between the reference sound feature amount and the sound feature amount of each time step is calculated as the index.
A computer characterized by identifying the utterance status of the target person in each time step based on the degree of similarity between the reference sound feature amount and the sound feature amount in each time step. The computer according to claim 3.
The storage device includes a first model that identifies a speaker who is speaking from an input sound, a second model that identifies an utterance status for each time step, a third model that specifies the utterance period, and the target. Stores information that defines a fourth model that extracts the voice of a person,
The arithmetic unit is
By inputting the first time-series data and the second time-series data into the second model, the converted input sound is calculated.
From the first model in which the converted input sound is input, the speaker feature amount of the target person is extracted.
By inputting the first time-series data and the second time-series data into the third model, the utterance period is calculated.
A computer characterized in that the voice of the target person is extracted from the input sound by inputting the speaker feature amount and the utterance period into the fourth model. The computer according to claim 6.
The arithmetic unit is
Learning data including a plurality of samples composed of a plurality of learning images, learning voices, and speaker identification information is received and stored in the storage device.
Using the learning data, an error in the speaker identification result output from the first model and the speaker identification information included in the sample, and the voice of the target person output from the fourth model. A computer characterized by learning the first model, the second model, the third model, and the fourth model so that the error of the learning voice included in the sample is small. It is a voice processing method executed by a computer to extract the voice of the target person included in the sound.
The calculator
It has an arithmetic unit, a storage device connected to the arithmetic unit, and a connection interface connected to the arithmetic unit.
A sound collecting device that collects sounds in the space where the target person exists and an imaging device that acquires an image of the target person are connected via the connection interface.
The voice processing method is
The first step in which the arithmetic unit stores the input sound acquired from the sound collecting device and the input image acquired from the image pickup device in the storage device.
A second step in which the arithmetic unit calculates a speaker feature amount indicating the characteristics of the voice of the target person using the input sound.
A third step in which the arithmetic unit uses the input image to extract a face region image including the face of the target person.
A fourth step in which the arithmetic unit uses a plurality of the face area images to identify an utterance period in which the target person is presumed to have spoken.
The arithmetic unit uses the speaker feature amount and the utterance period to extract the estimated voice of the target person in the utterance period from the input sound, and the extracted estimated voice of the target person is stored in the storage device. And the fifth step to store in
A voice processing method characterized by including. The voice processing method according to claim 8.
The second step is
A sixth step in which the arithmetic unit identifies the utterance status of the target person for each time step in the input sound using the input sound and the plurality of input images.
A seventh step in which the arithmetic unit converts the input sound so as to emphasize the time step including the voice of the target person based on the utterance status of the target person in each time step.
A voice processing method comprising the eighth step of calculating the speaker feature amount using the converted input sound. The voice processing method according to claim 9.
The first step is
A step in which the arithmetic unit generates first time-series data consisting of sound features for each time step from the input sound.
The arithmetic unit includes a step of generating a second time-series data including an image feature amount for each time step from the plurality of face area images.
The sixth step is
A ninth step in which the arithmetic unit uses the first time-series data and the second time-series data to calculate an index indicating the content of the voice of the target person in the input sound.
The arithmetic unit includes a tenth step of identifying the utterance status of the target person in each time step based on the index.
The seventh step is
A step in which the arithmetic unit calculates a weight according to the utterance status of the target person in each time step, and a step.
The arithmetic unit includes a step of converting the first time series data using the weights.
The eighth step is a voice processing method, wherein the arithmetic unit includes a step of calculating the speaker feature amount using the converted first time series data. The voice processing method according to claim 10.
The ninth step includes a step in which the arithmetic unit calculates the similarity of the sound feature amount and the similarity of the image feature amount between the time steps as the index.
In the tenth step, the arithmetic unit of the target person in each time step is based on the similarity of the image feature amount between the time steps and the similarity of the sound feature amount between the time steps. A voice processing method comprising a step of identifying an utterance situation. The voice processing method according to claim 10.
The ninth step is
A step in which the arithmetic unit identifies a reference time step in which the target person is closed based on the image feature amount, and a step.
A step in which the arithmetic unit calculates a reference sound feature amount based on the sound feature amount corresponding to the reference time step.
The arithmetic unit includes a step of calculating the similarity between the reference sound feature amount and the sound feature amount of each time step as the index.
In the tenth step, the arithmetic unit identifies the utterance status of the target person in each time step based on the degree of similarity between the reference sound feature amount and the sound feature amount in each time step. A voice processing method characterized by including steps. The voice processing method according to claim 10.
The storage device includes a first model that identifies a speaker who is speaking from an input sound, a second model that identifies an utterance status for each time step, a third model that specifies the utterance period, and the target. Stores information that defines a fourth model that extracts the voice of a person,
The second step is
A step of calculating the converted input sound by the arithmetic unit inputting the first time series data and the second time series data into the second model.
The arithmetic unit includes a step of extracting the speaker feature amount of the target person from the first model in which the converted input sound is input.
The fourth step includes a step in which the arithmetic unit calculates the utterance period by inputting the first time series data and the second time series data into the third model.
The fifth step includes a step in which the arithmetic unit extracts the voice of the target person from the input sound by inputting the speaker feature amount and the utterance period into the fourth model. A voice processing method characterized by that. The voice processing method according to claim 13.
A step in which the arithmetic unit receives learning data including a plurality of samples composed of a plurality of learning images, learning voices, and speaker identification information via the connection interface and stores the learning data in the storage device.
Using the learning data, the arithmetic unit outputs the speaker identification result output from the first model, the error of the speaker identification information included in the sample, and the fourth model. A step of learning the first model, the second model, the third model, and the fourth model so that the error between the voice of the target person and the voice for learning included in the sample is small. A voice processing method characterized by including.

Images