JP6961545B2 - Sound signal processor, sound signal processing method, and program - Google Patents

Description

Embodiments of the present invention relate to sound signal processing devices, sound signal processing methods, and programs.

A technique for emphasizing a target sound signal included in a sound signal of a sound emitted from a plurality of sound sources is known. For example, a technique is disclosed in which an SN ratio maximizing beamformer calculated based on a feature amount of a sound signal observed by a microphone is used as a filter for emphasizing a target sound signal included in the sound signal. As the feature quantity, a vector representing the speaker direction and the difference in voice arrival time between microphones is used.

Conventionally, a feature amount is extracted from an observed sound signal, and a filter for emphasizing the target sound signal is calculated from the feature amount, and it may be difficult to emphasize the target sound signal with high accuracy.

Japanese Patent No. 4891801 Japanese Patent No. 50444581

An object to be solved by the present invention is to provide a sound signal processing device, a sound signal processing method, and a program capable of emphasizing a target sound signal with high accuracy.

The sound signal processing device of the embodiment includes a coefficient deriving unit that derives a spatial filter coefficient for emphasizing the target sound signal included in the first sound signal based on the emphasized sound signal that emphasizes the target sound signal, and the above-mentioned A detection unit that detects a target sound section based on the emphasized sound signal, and a first spatial correlation matrix of the target sound section in the first sound signal based on the target sound section and the first sound signal. The first sound signal includes a second space correlation matrix of a non-target sound section other than the target sound section, and a correlation derivation unit for deriving the first sound signal, and the coefficient derivation unit includes the first space correlation matrix and the first space correlation matrix. The spatial filter coefficient is derived based on the two-spatial correlation matrix, and the detection unit uses the second sound signal whose power ratio of the non-target sound signal to the target sound signal is larger than that of the first sound signal and the emphasis. The target sound section is detected based on the sound signal .

Schematic diagram of a sound signal processing system. The schematic diagram of the functional configuration of a sound signal processing unit. Flowchart of sound signal processing. Schematic diagram of a sound signal processing system. The schematic diagram of the functional configuration of a sound signal processing unit. Flowchart of sound signal processing. Schematic diagram of a sound signal processing system. Explanatory drawing of hardware configuration.

The details of the present embodiment will be described below with reference to the accompanying drawings.

(First Embodiment)
FIG. 1 is a schematic diagram showing an example of the sound signal processing system 1 of the present embodiment.

The sound signal processing system 1 includes a sound signal processing device 10, a first microphone 14, and a second microphone 16. The sound signal processing device 10 and the first microphone 14 and the second microphone 16 are connected so as to be able to exchange data and signals.

The sound signal processing device 10 processes the sound signal of the sound emitted from one or a plurality of sound sources 12.

The sound source 12 is a sound source. The sound source 12 is, for example, a living thing such as a human being and an animal other than a human being, or a non-living body such as a musical instrument, but is not limited thereto. In the present embodiment, the case where the sound source 12 is a person will be described as an example. Therefore, in the present embodiment, the case where the sound is voice will be described as an example. The type of sound is not limited. In the following, a person may be referred to as a speaker.

In the present embodiment, the sound signal processing device 10 processes a sound signal including sounds emitted from a plurality of sound sources 12 and emphasizes a target sound signal included in the sound signal. The plurality of sound sources 12 are classified into a target sound source 12A and a non-purpose sound source 12B. The target sound source 12A is a sound source 12 that emits a target sound. The target sound is a sound to be emphasized. The target sound signal is a signal indicating a target sound. The target sound signal is represented by, for example, a spectrum. The non-purpose sound source 12B is a sound source 12 that emits a non-purpose sound. A non-purpose sound is a sound other than the target sound.

In the present embodiment, it is assumed that the target sound source 12A and the non-purpose sound source 12B, which are two speakers, talk with each other across the table T. In the present embodiment, for example, the non-purpose sound source 12B is a clerk, the target sound source 12A is a customer, and the target sound signal of the target sound source 12A, which is one speaker, is obtained from the sound signals indicating the conversations of these speakers. The explanation will be made assuming the intended use. The number of sound sources 12 and the arrangement of the sound sources 12 are not limited to these. Moreover, the assumed environment is not limited to this environment.

The first microphone 14 and the second microphone 16 collect sound. In the present embodiment, the first microphone 14 and the second microphone 16 collect the sound emitted from the sound source 12 and output the sound signal to the sound signal processing device 10.

The first microphone 14 is a microphone for collecting sound including at least a target sound. In other words, the first microphone 14 is a microphone for collecting at least the target sound emitted from the target sound source 12A.

The first microphone 14 outputs a third sound signal to the sound signal processing device 10 as a sound signal indicating the collected sound. The third sound signal is a sound signal including a non-purpose sound signal and a target sound signal. The non-purpose sound signal is a signal indicating a non-purpose sound. The non-target sound signal is represented by, for example, a spectrum. The first microphone 14 is arranged in advance at a position where the sound emitted from the sound source 12 (target sound source 12A, non-purpose sound source 12B) can be collected and the third sound signal can be output to the sound signal processing device 10. In the present embodiment, it is assumed that the first microphone 14 is arranged on the table T.

In the present embodiment, the sound signal processing system 1 includes a plurality of first microphones 14 (first microphones 14A to 14D). Therefore, a plurality of third sound signals are output to the sound signal processing device 10 from the plurality of first microphones 14. A sound signal obtained by combining a plurality of third sound signals into one will be referred to as a first sound signal.

The number of the first microphones 14 may be equal to or greater than the number of sound sources 12 to be collected. As described above, in the present embodiment, the sound signal processing system 1 assumes one target sound source 12A and one non-purpose sound source 12B, for a total of two sound sources 12. In this case, the number of the first microphones 14 may be 2 or more. In the present embodiment, the case where the sound signal processing system 1 includes four first microphones 14 (first microphones 14A to 14D) will be described as an example.

The plurality of first microphones 14 have different sound arrival time differences from each of the plurality of sound sources 12. That is, the arrangement positions of the plurality of first microphones 14 are adjusted in advance so that the sound arrival time differences are different from each other.

The second microphone 16 is a microphone for collecting at least non-purpose sounds. In other words, the second microphone 16 is a microphone for collecting at least the non-purpose sound emitted from the non-purpose sound source 12B.

The second microphone 16 outputs the second sound signal to the sound signal processing device 10 as a sound signal indicating the collected sound. The second sound signal is a sound signal in which the ratio of the power of the non-target sound signal to the target sound signal is larger than that of the first sound signal (third sound signal). In the second sound signal, the ratio of the power of the non-target sound signal to the target sound signal is larger than that of the first sound signal (third sound signal), and the power of the non-target sound signal is higher than the power of the target sound signal. It is preferably a loud sound signal.

In the present embodiment, the second microphone 16 is arranged at a position closer to the non-purpose sound source 12B than the first microphone 14. For example, the second microphone 16 is a headset microphone or a pin microphone. In the present embodiment, the second microphone 16 is attached to the non-purpose sound source 12B so that the voice can be collected at the mouth of the speaker, which is the non-purpose sound source 12B.

The sound signal processing device 10 includes an AD conversion unit 18, a sound signal processing unit 20, and an output unit 22. The sound signal processing device 10 may be configured to include at least the sound signal processing unit 20, and at least one of the AD conversion unit 18 and the output unit 22 may be configured as a separate body.

The AD conversion unit 18 receives a plurality of third sound signals from the plurality of first microphones 14. Further, the AD conversion unit 18 receives the second sound signal from the second microphone 16. The AD conversion unit 18 converts each of the plurality of third sound signals and the second sound signal into digital signals and outputs them to the sound signal processing unit 20.

The sound signal processing unit 20 uses the plurality of third sound signals and the second sound signals received from the AD conversion unit 18, and the sound signal processing unit 20 uses the plurality of third sound signals to be combined into one target sound included in the first sound signal. The signal is emphasized, and the emphasized sound signal is output to the output unit 22.

The output unit 22 is a device that outputs an emphasis sound signal received from the sound signal processing unit 20. The output unit 22 is, for example, a speaker, a communication device, a display device, a recording device, a recording device, and the like. The speaker outputs the sound represented by the emphasis signal. The communication device transmits an emphasis sound signal to an external device or the like via a network or the like. The display device displays information indicating an emphasis signal. The recording device stores the emphasis signal. The recording device is, for example, an IC recorder, a personal computer, or the like. The recording device is a device that converts the sound indicated by the emphasis sound signal into text by a known method and records it. The output unit 22 may output, transmit, store, or record the emphasized sound signal received from the sound signal processing unit 20 after converting it into an analog signal.

Next, the sound signal processing unit 20 will be described in detail.

FIG. 2 is a schematic diagram showing an example of the functional configuration of the sound signal processing unit 20.

The sound signal processing unit 20 includes a conversion unit 20A, a conversion unit 20B, a detection unit 20C, a correlation derivation unit 20D, a first correlation storage unit 20E, a second correlation storage unit 20F, a coefficient derivation unit 20G, and the like. A generation unit 20H and an inverse conversion unit 20I are provided.

The conversion unit 20A, the conversion unit 20B, the detection unit 20C, the correlation derivation unit 20D, the coefficient derivation unit 20G, the generation unit 20H, and the inverse conversion unit 20I are realized by, for example, one or more processors. For example, each of the above-mentioned parts may be realized by causing a processor such as a CPU (Central Processing Unit) to execute a program, that is, by software. Each of the above-mentioned parts may be realized by a processor such as a dedicated IC (Integrated Circuit), that is, hardware. Each of the above-mentioned parts may be realized by using software and hardware together. When a plurality of processors are used, each processor may realize one of each part, or may realize two or more of each part.

The first correlation storage unit 20E and the second correlation storage unit 20F store various types of information. The first correlated storage unit 20E and the second correlated storage unit 20F may be composed of any commonly used storage medium such as an HDD (Hard Disk Drive), an optical disk, a memory card, and a RAM (Random Access Memory). can. Further, the first correlation storage unit 20E and the second correlation storage unit 20F may be physically different storage media, or may be realized as different storage areas of physically the same storage medium. Further, each of the first correlation storage unit 20E and the second correlation storage unit 20F may be realized by a plurality of physically different storage media.

The conversion unit 20A performs a short-time Fourier transform (STFT: Short-Time Fourier Transform) of the second sound signal received from the second microphone 16 via the AD conversion unit 18, and is represented by a frequency spectrum X ₁ (f, n). The second sound signal to be generated is output to the detection unit 20C. In addition, f represents the frequency bin number, and n represents the frame number.

For example, the sampling frequency is set to 16 kHz, the frame length is set to 256 samples, and the frame shift is set to 128 samples. In this case, the conversion unit 20A converts the second sound signal into a frequency spectrum by performing a fast Fourier transform (FFT) after applying a Hanning window of 256 samples to the second sound signal. Then, in consideration of the symmetry of the low frequency band and the high frequency frequency spectrum, the complex numerical values of 129 points in the range where f is 0 or more and 128 or less in the frequency spectrum are set to the nth frame in the second sound signal. Calculated as the frequency spectrum X ₁ (f, n). Then, the conversion unit 20A _{outputs the second sound signal represented by the frequency spectrum X 1} (f, n) to the detection unit 20C.

The conversion unit 20B performs short-time Fourier transform (STFT) on each of the plurality of third sound signals received from the plurality of first microphones 14 (first microphones 14A to 14D) via the AD conversion unit 18. , Frequency spectrum X _2,1 (f, n), Frequency spectrum X _2,2 (f, n), Frequency spectrum X _2,3 (f, n), Frequency spectrum X _2,4 (f, n), respectively. Generates a plurality of third sound signals represented by.

The frequency spectrum X _{2, 1} (f, n) is a short-time Fourier transform of the third sound signal received from the first microphone 14A. The frequency spectra X _{2, 2} (f, n) are short-time Fourier transforms of the third sound signal received from the first microphone 14B. The frequency spectra X2, ₃ (f, n) are short-time Fourier transforms of the third sound signal received from the first microphone 14C. The frequency spectra X _{2, 4} (f, n) are short-time Fourier transforms of the third sound signal received from the first microphone 14D.

It should be noted that a multidimensional vector (four-dimensional vector in the present embodiment) that summarizes the plurality of frequency spectra indicating each of the plurality of third sound signals is used below, and the frequency spectrum X ₂ (f) indicating the first sound signal is described below. , N). In other words, the first sound signal is _{represented by the frequency spectrum X 2} (f, n). _{The frequency spectrum X 2} (f, n) showing the first sound signal is represented by the following equation (1).

The conversion unit 20B outputs the frequency spectrum X ₂ (f, n) indicating the first sound signal to the correlation derivation unit 20D and the generation unit 20H.

The first correlation storage unit 20E stores the first spatial correlation matrix _φxx (f, n). The first spatial correlation matrix phi _xx, showing the spatial correlation matrix of the target sound section in the first sound signal. The target sound section indicates a section including the target sound in the first sound signal. The interval indicates a specific period in the time series direction.

As described above, in the present embodiment, the first sound signal is represented by the frequency spectrum X ₂ (f, n) showing a four-dimensional vector. Therefore, the first spatial correlation matrix _φxx (f, n) is represented by a matrix of 4 × 4 complex numbers for each frequency bin.

In the initial state, the first correlation storage unit 20E stores _{the first spatial correlation matrix φxx} (f, n) _{initialized with the zero matrix (φxx} (f, 0) = 0). The first spatial correlation matrix _φxx (f, n) is updated by the correlation deriving unit 20D described later.

The second correlation storage unit 20F stores the second spatial correlation matrix φ _NN (f, n). The second spatial correlation matrix φ _NN indicates the spatial correlation matrix of the non-purpose sound section in the first sound signal. The non-target sound section indicates a section other than the target sound section in the first sound signal.

Similar to the first spatial correlation matrix φ _xx (f, n), in the present embodiment, the second spatial correlation matrix φ _NN (f, n) is represented by a 4 × 4 complex matrix for each frequency bin. NS.

In the initial state, the second correlation storage unit 20F stores _{the second spatial correlation matrix φ NN} (f, n) _{initialized with the zero matrix (φ NN} (f, 0) = 0). The second spatial correlation matrix φ _NN (f, n) is updated by the processing of the correlation deriving unit 20D described later.

Next, the detection unit 20C, the correlation derivation unit 20D, the coefficient derivation unit 20G, the generation unit 20H, and the inverse conversion unit 20I will be described. In the present embodiment, the sound signal processing unit 20 executes the steady processing after performing the initial processing at the start of the sound signal processing. The correlation derivation unit 20D, the coefficient derivation unit 20G, and the generation unit 20H execute different processes during the initial process and the steady process.

First, the functions of the correlation derivation unit 20D, the coefficient derivation unit 20G, and the generation unit 20H in the initial processing will be described.

The initial processing is a process executed by the sound signal processing unit 20 at the start of the sound signal processing. In the initial processing, the sound signal processing section 20, stored in the first correlation storage portion 20E and the second correlation storage unit 20F, initialized with that first spatial correlation matrix phi _xx zero matrix _(f, n) And by updating the second spatial correlation matrix φ _NN (f, n), initial values are set in these spatial correlation matrices.

The coefficient derivation unit 20G derives the spatial filter coefficient F (f, n) for emphasizing the target sound signal included in the first sound signal. The coefficient derivation unit 20G derives the spatial filter coefficient F (f, n) based on _{the first spatial correlation matrix φ xx} (f, n) and the second spatial correlation matrix φ _{NN (f, n).}

As described above, in the present embodiment, the first sound signal is represented by the frequency spectrum X ₂ (f, n) showing a four-dimensional vector. Therefore, the coefficient derivation unit 20G is based on the first spatial correlation matrix φ _xx (f, n) and the second spatial correlation matrix φ _NN (f, n), and is a complex four-dimensional vector spatial filter coefficient F ( Calculate f, n). The spatial filter coefficient F (F, n) is represented by the following equation (2).

However, in the initial processing, the coefficient deriving unit 20G derives the spatial filter coefficient F (f, n) = [0,0,0,1] as the spatial filter coefficient F (f, n).

The generation unit 20H uses the spatial filter coefficient F (f, n) derived by the coefficient derivation unit 20G to generate a target sound signal included in the first sound signal represented by the _{frequency spectrum X 2 (f, n).} Generates an emphasized, emphasized sound signal.

Specifically, the generation unit 20H uses the following equation (3) to generate an emphasis sound signal represented by the output spectrum Y (f, n).

That is, the generation unit 20H calculates the product of the frequency spectrum X ₂ (f, n) and the transposed matrix in which the spatial filter coefficient F (f, n) is Hermitian transposed, and outputs the output spectrum Y (f,) showing the emphasized sound signal. Generate as n). In the initial processing, the generation unit 20H outputs an emphasis sound signal in which _{Y (f, n) = X 2,4 (f, n).} That is, in the initial processing, the generation unit 20H outputs the frequency spectrum of the third sound signal collected by the first microphone 14D as an emphasis sound signal. The first microphone 14 used as the emphasis sound signal in the initial processing may be the first microphone 14 among the plurality of first microphones 14 (first microphones 14A to 14D), and may be the first microphone. Not limited to 14D.

The generation unit 20H outputs the emphasis signal represented by the output spectrum Y (f, n) to the inverse conversion unit 20I and the detection unit 20C.

The detection unit 20C detects a target sound section based on the emphasis sound signal. In the present embodiment, the detection unit 20C detects a target sound section based on the second sound signal and the emphasis sound signal.

Specifically, the detection unit 20C includes a _{second sound signal represented by the frequency spectrum X 1} (f, n) and an emphasis sound signal represented by the output spectrum Y (f, n) received from the generation unit 20H. , Detects the target sound section.

_{The target sound section is represented by a function u 2} (n) indicating whether or not the target sound source 12A is emitting sound for each frame number.

u ₂ (n) = 1 indicates that the target sound source 12A is emitting sound in the nth frame. The nth frame indicates the nth frame. u ₂ (n) = 0 indicates that the target sound source 12A does not emit sound in the nth frame.

Specifically, the function u ₂ is represented by the following equation (4).

In the formula (4), p _Y (n) and p _X (n) are represented by the following formulas (5) and (6). That is, p _Y (n) and p _X (n) are an emphasis sound signal represented by the output spectrum Y (f, n) and a second sound signal represented by the _{frequency spectrum X 1 (f, n).} Depends on the power of each of.

Here, at the initial processing stage, p _Y (n) includes spectra corresponding to the sounds of both the target sound source 12A and the non-purpose sound source 12B. Therefore, in the equation (4), the threshold value t ₁ is set in advance so as to satisfy the relationship of _{t 1} <P _Y (n) when the sound is emitted from the target sound source 12A or the non-purpose sound source 12B. ..

Further, when the non-purpose sound source 12B is emitting sound among the target sound source 12A and the non-purpose sound source 12B, p _X (n) is _{relatively larger than py} (n). Thus, in the formula (4), the threshold value _{t 2,} when the non-target sound source 12B is emitting a _sound, so as to satisfy the relationship _{p X (n) -p y (} n) ≧ t 2, preset do.

With these settings, the function u ₂ (n) shows a value of "1" in the nth frame in which only the target sound source 12A emits sound. Then, the function u ₂ shows a value “0” in the nth frame in which the target sound source 12A does not emit sound.

Therefore, the detection unit 20C detects the section _{represented by u 2} (n) = 1 as the target sound section, and detects the section represented by u ₂ (n) = 0 as the non-target sound section.

The correlation derivation unit 20D has a first spatial correlation based on the target sound section detected by the detection unit 20C and the first sound signal received from the first microphone 14 via the conversion unit 20B and the AD conversion unit 18. The matrix φ _xx (f, n) and the second spatial correlation matrix φ _NN (f, n) are derived. Then, the correlation derivation unit 20D _{updates the first space correlation matrix φxx} (f, n) by _{storing the derived first space correlation matrix φxx} (f, n) in the first correlation storage unit 20E. .. Similarly, the correlation derivation unit 20D _{updates the second space correlation matrix φ NN} (f, n) by _{storing the derived second space correlation matrix φ NN} (f, n) in the second correlation storage unit 20F. do.

Specifically, the correlation deriving unit 20D uses the following _{equation (7) to obtain a} first spatial correlation matrix φxx (f, n) for the interval (nth frame) represented by _{u 2 (n) = 1.} Derived and updated, and the second spatial correlation matrix φ _NN (f, n) is not updated.

On the other hand, the correlation derivation unit 20D derives _{the second spatial correlation matrix φ NN} (f, n) by the following equation (8) for the interval (nth frame) represented by _{u 2 (n) = 0.} It is updated, and the first spatial correlation matrix _φxx (f, n) is not updated.

In the formula (7) and the formula (8), α is a value of 0 or more and less than 1. As the value of α is a value close to 1, the weight of the first spatial correlation matrix phi _xx derived in the past _(f, n) is larger than the latest first spatial correlation matrix φ _{xx (f,} n) Means things. The value of α may be set in advance. α may be, for example, 0.95.

That is, the correlation derivation unit 20D transposes the first sound signal and the first sound signal of the first sound signal of the target sound section, which has been derived in the past, into the first space correlation matrix _φxx (f, n) by Hermitian. A new first spatial correlation matrix φ _xx (f, n) is derived _{by correcting with the latest first spatial correlation matrix φ xx} (f, n) represented by the multiplication value with the transposed signal. The first sound signal in the target section means the sound signal in the target section in the first sound signal.

Correlation derivation unit 20D includes first spatial correlation matrix phi _xx of already stored in the first correlation storage section 20E of the (f, n), may be used as the first spatial correlation matrix phi _xx derived in the past (f, n) .. _{Only one first spatial correlation matrix φxx} (f, n) is stored in the first correlation storage unit 20E, and is sequentially updated by the correlation derivation unit 20D.

_{Further, the correlation derivation unit 20D uses the second spatial correlation matrix φ NN} (f, n) derived in the past for the first sound signal in the non-purpose sound section, and the first sound signal and the first sound signal as Elmeet. A new second spatial correlation matrix φ _NN (f, n) is derived _{by correcting with the latest second spatial correlation matrix φ NN} (f, n) represented by the multiplication value with the transposed transposed signal. The first sound signal in the non-purpose section means the sound signal in the non-purpose section in the first sound signal.

Correlation derivation unit 20D, the second spatial correlation matrix phi _NN of already stored in the second correlation storing section 20F to (f, n), may be used as the second spatial correlation matrix phi _NN derived in the past (f, n) .. It is assumed that only one second spatial correlation matrix φ _NN (f, n) is stored in the second correlation storage unit 20F, and is sequentially updated by the correlation derivation unit 20D.

Next, the functions of the correlation derivation unit 20D, the coefficient derivation unit 20G, and the generation unit 20H in the steady processing will be described. The steady-state process is a process executed after the initial process.

The sound signal processing unit 20 may shift to the steady processing after executing the initial processing for a predetermined time, or may shift to the first spatial correlation matrix _φxx (f, n) and the second spatial correlation matrix φ _NN (f, NN). When n) is updated a predetermined number of times, the process may shift to routine processing.

First, the function of the coefficient derivation unit 20G in the steady processing will be described. In the initial processing, the coefficient derivation unit 20G derives the spatial filter coefficient F (f, n) = [0,0,0,1] as the spatial filter coefficient F (f, n).

In the steady processing, the coefficient derivation unit 20G derives the spatial filter coefficient F (f, n) for emphasizing the target sound signal included in the first sound signal based on the emphasized sound signal emphasizing the target sound signal. ..

As described above, the first sound signal includes a plurality of third sound signals acquired from the plurality of first microphones 14. Therefore, the coefficient deriving unit 20G has a spatial filter coefficient F based on the emphasized sound signal emphasizing the target sound signal included in the first sound signal composed of the plurality of third signals output from the plurality of first microphones 14. (F, n) is derived.

Specifically, the coefficient deriving unit 20G is based on the first spatial correlation matrix φ _xx (f, n) and the second spatial correlation matrix φ _NN (f, n) updated by the correlation deriving unit 20D. F (f, n) is derived.

_{The coefficient derivation unit 20G reads the first spatial correlation matrix φxx} (f, n) and the second spatial correlation matrix φ _NN (f, n) from the first correlation storage unit 20E and the second correlation storage unit 20F, and performs a spatial filter. It may be used for deriving the coefficient F (f, n).

Here, at the stage of steady processing, the first spatial correlation matrix φ _xx (f, n) and the second spatial correlation matrix φ _NN (f,) stored in the first correlation storage unit 20E and the second correlation storage unit 20F. n) is a spatial correlation matrix updated by the correlation derivation unit 20D. That is, these spatial correlation matrices are spatial correlation matrices updated by the correlation derivation unit 20D using the target sound section detected based on the emphasized sound signal. Therefore, the coefficient deriving unit 20G derives the spatial filter coefficient F (f, n) based on the emphasized sound signal.

Specifically, the coefficient derivation unit 20G is the maximum eigenvalue of the matrix represented by the product of _{the first spatial correlation matrix φxx} (f, n) and the _{inverse matrix of the second spatial correlation matrix φ NN (f, n).} The eigenvector F _SNR (f, n) corresponding to is derived. Then, the coefficient deriving unit 20G derives the eigenvector F _SNR (f, n) as the spatial filter coefficient F (f, n) (F (f, n) = F _SNR (f, n)).

The eigenvector F _SNR (f, n) constitutes a MAX-SNR (Maximum Signal-to-Noise) beamformer that maximizes the power ratio between the target sound and the non-target sound.

The coefficient derivation unit 20G adds a post filter w (f, n) that improves sound quality by adjusting the power of each frequency bin, and uses the following equation (9) to create a spatial filter coefficient F (f, n). n) may be derived.

The post filter w (f, n) is represented by the following equation (10).

Next, the generation unit 20H will be described. _{In the steady processing, the generation unit 20H is represented by the frequency spectrum X 2} (f, n) using the spatial filter coefficient F (f, n) derived by the coefficient derivation unit 20G as in the initial processing. An emphasized sound signal that emphasizes the target sound signal included in the first sound signal is generated. That is, the generation unit 20H uses the above equation (3) to generate an emphasis sound signal represented by the output spectrum Y (f, n).

The generation unit 20H outputs the generated emphasis sound signal to the inverse conversion unit 20I and the detection unit 20C.

The inverse transform unit 20I performs an inverse short-time Fourier transform (ISTFT: Inverse Short-Time Fourier Transform) of the emphasized sound signal received from the generation unit 20H and outputs it to the output unit 22.

That is, the inverse conversion unit 20I converts the emphasized signal in which the target sound signal of the target sound emitted from the target sound source 12A is emphasized and the non-target sound signal is suppressed into a sound wave shape in the time domain.

Specifically, the inverse transform unit 20I generates a spectrum of 256 points from the output spectrum Y (f, n) by using the symmetry of the output spectrum Y (f, n) indicating the emphasized signal, and performs the inverse Fourier transform. I do. Next, the inverse transformation unit 20I may generate a sound wave shape by applying a composite window function and superimposing it on the output waveform of the previous frame by shifting the frame shift.

Next, the detection unit 20C will be described. At the time of initial processing, the detection unit 20C detected the target sound section.

At the time of steady processing, the detection unit 20C detects a target sound section and an overlapping section based on the emphasis sound signal and the second sound signal. The overlapping section indicates a section in which sound is emitted from both the target sound source 12A and the non-purpose sound source 12B. For example, the overlapping section indicates a section in which a plurality of speakers are speaking.

Specifically, the detection unit 20C, in addition to the function _u 2 (n), for detecting the function _u 1 (n).

The function u ₁ (n) is a function indicating a second non-purpose sound section. Specifically, the function u ₁ (n) is a function indicating whether or not the non-purpose sound source 12B is emitting sound for each frame number. The second non-purpose sound section is a section in which the non-purpose sound source 12B emits sound.

Here, at the stage of steady processing, the power due to the non-purpose sound emitted from the non-purpose sound source 12B included in the emphasis sound signal represented by the output spectrum Y (f, n) is suppressed. _{Therefore, p Y} (n) represented by the above equation (5) can be approximately regarded as the power generated by the target sound emitted from the target sound source 12A. Therefore, at the stage of steady processing, the second non-purpose sound section represented by _{u 1 (n) and the target sound section represented by u 2} (n) are the following equations (11) and (12). Represented by.

Note that u ₂ (n) = 1 indicates that the target sound source 12A is emitting sound in the nth frame. u ₂ (n) = 0 indicates that the target sound source 12A does not emit sound in the nth frame. Further, u ₁ (n) = 1 indicates that the non-purpose sound source 12B is emitting sound in the nth frame. u ₁ (n) = 0 indicates that the non-purpose sound source 12B does not emit sound in the nth frame.

_{Therefore, the threshold values t 3} and t ₄ in the equations (11) and (12) may be set in advance so that u ₁ (n) and u ₂ (n) are equations indicating the above conditions. ..

The detection unit 20C detects a section in which u ₂ (n) = 1 and u ₁ (n) = 0 as a target sound section. Further, the detection unit 20C detects a section where u ₂ (n) = 0 as a non-purpose sound section. Further, the detection unit 20C detects a section in which u ₂ (n) = 1 and u ₁ (n) = 1 as an overlapping section in which sound is emitted from both the target sound source 12A and the non-purpose sound source 12B. do. Then, the detection unit 20C outputs the detection result to the correlation derivation unit 20D. In the present embodiment, the detection unit 20C outputs u ₁ (n) and u ₂ (n) to the correlation derivation unit 20D as the detection result.

_{The correlation derivation unit 20D has a first spatial correlation matrix φxx} (f, n) and a second spatial correlation matrix based on the target sound section, the overlapping section, and the first sound signal detected by the detection unit 20C. Derivation of φ _NN (f, n).

The correlation derivation unit 20D uses a section in which u ₂ (n) = 1 and u ₁ (n) = 0 as a target sound section, and uses the following equation (13) for the first spatial correlation matrix. _{Derivation and update of φxx} (f, n). The _{correlation derivation unit 20D derives and updates the second spatial correlation matrix φ NN} (f, n) for the target sound section in which _{u 2} (n) = 1 and u _{1 (n) = 0.} No.

On the other hand, in the correlation derivation unit 20D _{, a section where u 2} (n) = 0 is set as a non-purpose sound section, and for this section, the second spatial correlation matrix φ _NN (f, n) is used by using the following equation (14). ) Is derived and updated. Note that the correlation deriving unit 20D does not derive or update _{the first spatial correlation matrix φxx (f, n) for the interval where u 2 (n) = 0.}

Further, the correlation derivation unit 20D has a _{first spatial correlation matrix φ xx} (f, n) and a second spatial correlation matrix φ _{for an interval in which u 2} (n) = 1 and u ₁ (n) = 1. _{Both NN} (f, n) are not derived or updated. As described above, the section in which u ₂ (n) = 1 and u ₁ (n) = 1 is an overlapping section in which sound is emitted from both the target sound source 12A and the non-purpose sound source 12B.

_{That is, in the steady processing, the correlation derivation unit 20D uses the first space correlation matrix φxx} (f, n) and the second space for the overlapping section in which sound is emitted from both the target sound source 12A and the non-target sound source 12B. Both of the correlation matrix φ _NN (f, n) are not derived or updated.

As described above, for the overlapping section in which both the target sound source 12A and the non-purpose sound source 12B emit sound at the same time, the first spatial correlation matrix φ _xx (f, n) and the second spatial correlation matrix φ _NN (f, n) ) Will not be updated. With this configuration, it is possible to suppress a decrease in the emphasis accuracy of the target sound signal due to the use of the overlapping section in which both the target sound source 12A and the non-purpose sound source 12B emit sound at the same time.

Next, the procedure of sound signal processing executed by the sound signal processing device 10 of the present embodiment will be described.

FIG. 3 is a flowchart showing an example of a sound signal processing procedure executed by the sound signal processing device 10 of the present embodiment.

The conversion unit 20B performs a short-time Fourier transform on the third signal received from the plurality of first microphones 14 to _{acquire the first sound signal represented by the frequency spectrum X 2} (f, n) (step S100). The conversion unit 20B outputs the acquired first sound signal to the correlation derivation unit 20D and the generation unit 20H (step S102).

Next, the conversion unit 20A performs a short-time Fourier transform on the second sound signal received from the second microphone 16 to _{acquire the second sound signal represented by the frequency spectrum X 1} (f, n) (step S104). .. The conversion unit 20A outputs the acquired second sound signal to the detection unit 20C (step S106).

The processes of steps S100 to S106 may be executed by the conversion unit 20A and the conversion unit 20B in parallel, and are not limited to the order shown in FIG. Further, it is assumed that the processes of steps S100 to S106 are continuously and repeatedly executed until the sound signal processing is completed.

Then, the sound signal processing device 10 executes the initial processing (steps S108 to S120).

Specifically, first, the coefficient deriving unit 20G receives the first spatial correlation matrix φ _xx (f, n) and the second spatial correlation matrix φ _NN (f, from the first correlation storage unit 20E and the second correlation storage unit 20F). n) is read (step S108). As described above, in the initial state, the first spatial correlation matrix _φxx (f, n) and the second spatial correlation matrix φ _NN (f, n) are initialized with a zero matrix.

Next, the coefficient deriving unit 20G uses the first spatial correlation matrix φ _xx (f, n) and the second spatial correlation matrix φ _NN (f, n) read in step S108 to use the spatial filter coefficient F (f, n). n) is derived (step SS110). As described above, in the initial state, the coefficient deriving unit 20G derives the spatial filter coefficient F (f, n) = [0,0,0,1] as the spatial filter coefficient F (f, n).

Next, the generation unit 20H uses the spatial filter coefficient F (f, n) derived in step S110 to obtain the first sound signal represented by the _{frequency spectrum X 2 (f, n) acquired in step S110.} Generates an emphasized sound signal that emphasizes the target sound signal included in (step S112).

Next, the inverse transform unit 20I performs inverse short-time Fourier transform on the emphasized sound signal represented by the output spectrum Y (f, n) generated in step S112 and outputs it to the output unit 22 (step S114).

Next, the detection unit 20C detects the target sound section represented by the _{function u 2} (n) by using the emphasis sound signal and the second sound signal generated in step S112 (step S116).

Next, the correlation derivation unit 20D uses the target sound section detected in step S116 and the first sound signal to form the first spatial correlation matrix φ _xx (f, n) and the second spatial correlation matrix φ _NN (f,). n) is derived (step S118).

Next, the correlation derivation unit 20D uses the first spatial correlation matrix φ _xx (f, n) and the second spatial correlation matrix φ _NN (f, n) derived in step S118 as the first correlation storage unit 20E and the second. These spatial correlation matrices are updated by storing in each of the correlation storage units 20F (step S120).

Next, the sound signal processing unit 20 determines whether or not to shift from the initial processing to the steady processing (step S122). For example, the sound signal processing unit 20 determines whether or not to shift to the steady processing by determining whether or not the initial processing has been executed for a predetermined time. Further, the sound signal processing unit 20 _{determines whether or not the first spatial correlation matrix φxx} (f, n) and the second spatial correlation matrix φ _NN (f, n) have been updated a predetermined number of times, thereby performing steady processing. It may be determined whether or not to shift to.

If a negative determination is made in step S122 (step S122: No), the process returns to step S108. On the other hand, if an affirmative determination is made in step S122 (step S122: Yes), the sound signal processing unit 20 executes steady processing (steps S124 to S138).

In the steady processing, the coefficient deriving unit 20G receives the first spatial correlation matrix φ _xx (f, n) and the second spatial correlation matrix φ _NN (f, n) from the first correlation storage unit 20E and the second correlation storage unit 20F. Is read (step S124). That is, the coefficient deriving unit 20G reads the latest first spatial correlation matrix φ _xx (f, n) and the second spatial correlation matrix φ _NN (f, n) updated by the correlation deriving unit 20D.

Next, the coefficient deriving unit 20G has a spatial filter coefficient F (f, n) based on _{the first spatial correlation matrix φ xx} (f, n) and the second spatial correlation matrix φ _{NN (f, n) read in step S124.} n) is derived (step S126).

Next, the generation unit 20H uses the spatial filter coefficient F (f, n) derived in step S126 to emphasize the target sound signal included in the first sound signal received from the conversion unit 20B, and emphasizes the emphasis sound signal. Is generated (step S128).

Next, the inverse transform unit 20I performs inverse short-time Fourier transform on the emphasized sound signal generated in step S128 and outputs it to the output unit 22 (step S130).

Next, the detection unit 20C detects a target sound section and an overlapping section using the second sound signal and the emphasis sound signal generated in step S128 (step S132).

Next, the correlation derivation unit 20D is based on the target sound section detected in step S132, the overlapping section, and the first sound signal received from the first microphone 14 via the conversion unit 20B and the AD conversion unit 18. Then, the first spatial correlation matrix φ _xx (f, n) and the second spatial correlation matrix φ _NN (f, n) are derived (step S134). Then, the correlation derivation unit 20D uses the derived first space correlation matrix _φxx (f, n) and the second space correlation matrix φ _NN (f, n) of the first correlation storage unit 20E and the second correlation storage unit 20F. By storing in each, these spatial correlation matrices are updated (step S136).

Next, the sound signal processing unit 20 determines whether or not to end the sound signal processing (step S138). If a negative determination is made in step S138 (step S138: No), the process returns to step S124. If an affirmative judgment is made in step S138 (step S138: Yes), this routine ends.

As described above, the sound signal processing device 10 of the present embodiment includes a coefficient deriving unit 20G. The coefficient derivation unit 20G derives the spatial filter coefficient F (f, n) for emphasizing the target sound signal included in the first sound signal based on the emphasized sound signal that emphasizes the target sound signal. Therefore, by using the derived spatial filter coefficient F (f, n) to generate an emphasis sound signal that emphasizes the target sound signal, the target sound signal can be emphasized with high accuracy.

Here, conventionally, when a plurality of speakers speak at the same time, the emphasis accuracy of the target sound may decrease. For example, there is a conventional method in which a vector representing the speaker direction and the arrival time difference between microphones is used as a feature amount of a sound signal, and a filter for emphasizing the target sound signal included in the sound signal is generated based on the feature amount. Are known.

However, in such a conventional method, when a plurality of speakers speak at the same time, a distribution of features different from the features of each speaker is obtained, so that the accuracy of the filter for emphasizing the target sound signal is lowered. There was a case. Further, even in the case where a plurality of speakers speak in order, the accuracy of the filter for emphasizing the target sound signal may be lowered because a section in which the speakers speak at the same time due to an aizuchi or the like is generated.

On the other hand, in the sound signal processing device 10 of the present embodiment, the spatial filter coefficient F (f, n) for emphasizing the target sound signal included in the first sound signal is based on the emphasized sound signal emphasizing the target sound signal. ) Is derived. Therefore, by applying the derived spatial filter coefficient F (f, n) to the first sound signal, the target sound signal is emphasized with high accuracy by generating an emphasized sound signal that emphasizes the target sound signal. Can be done.

Therefore, the sound signal processing device 10 can emphasize the target sound signal with high accuracy.

Further, in the sound signal processing device 10 of the present embodiment, the detection unit 20C sets the ratio of the power of the non-target sound signal to the target sound signal to be larger than that of the first sound signal, that is, the second sound signal and the emphasized sound signal. Based on this, the target sound section is detected. Therefore, the detection unit 20C can detect the target sound section with high accuracy. Then, the coefficient derivation unit 20G is derived based on the target sound section and the first sound signal detected with high accuracy, and the first spatial correlation matrix _φxx (f, n) and the second spatial correlation matrix φ _NN ( The spatial filter coefficient F (f, n) is derived based on f, n).

Therefore, the sound signal processing device 10 can emphasize the target sound signal with higher accuracy.

Further, in the present embodiment, the detection unit 20C detects the target sound section based on the second sound signal and the emphasis sound signal. Therefore, the sound signal processing device 10 of the present embodiment suppresses the non-purpose sound of the non-purpose sound source 12B and emphasizes the target sound signal of the target sound source 12A regardless of the positions of the target sound source 12A and the non-purpose sound source 12B. As such, the spatial filter coefficient F (f, n) can be derived. Therefore, the sound signal processing device 10 can derive a spatial filter coefficient (f, n) for emphasizing the target sound signal included in the first sound signal with higher accuracy.

Further, in the present embodiment, the detection unit 20C detects an overlapping section in which the target sound and the non-target sound overlap and a target sound section based on the emphasized sound signal. _{Then, the correlation derivation unit 20D calculates the first spatial correlation matrix φ xx} (f, n) and the second spatial correlation matrix φ _NN (f, n) based on the target sound section, the overlapping section, and the first sound signal. Derived.

Then, the correlation derivation unit 20D _{does not derive and update the first spatial correlation matrix φ xx} (f, n) and the second spatial correlation matrix φ _NN (f, n) for the overlapping interval. Therefore, the coefficient deriving unit 20G does not derive the spatial filter coefficient F (f, n) for the overlapping section. Therefore, the sound signal processing device 10 of the present embodiment can emphasize the target sound signal with high accuracy even in the section of the first sound signal in which sounds are simultaneously emitted from the plurality of sound sources 12.

<Modification example 1>
In the above, the detection unit 20C is based on the power of the emphasis sound signal represented by the output spectrum Y (f, n) and the second sound signal represented by the _{frequency spectrum X 1 (f, n).} The target sound section and the overlapping section were detected.

However, the detection unit 20C may detect the target sound section and the overlapping section by other methods using the output spectrum Y (f, n) and the frequency spectrum X _{1 (f, n).}

_{For example, a model for estimating the function u 1} (n) and the function u ₂ (n) with the output spectrum Y (f, n) and the frequency spectrum X ₁ (f, n) as inputs is determined by a decision tree or near k. It may be learned by a method, a support vector machine, a neural network, or the like.

As an example, learning of a model using a neural network will be described.

In this case, the detection unit 20C collects training data for training the model. For example, by mounting the sound signal processing unit 20 of the present embodiment on the learning device and executing the above processing using the sound signal processing unit 20, the frequency spectrum X ₁ (f, n) and the frequency spectrum X ₁ A plurality of training data including the output spectrum Y (f, n) derived from (f, n) are recorded. Next, the target sound of the target sound source 12A is collected and recorded by the first microphone 14D. _{Then, the function u 1} (n) and the function u ₂ (n) are determined by determining the sound source 12 that emits sound in each frame by viewing the target sound by the user or visually observing the waveform of the target sound by the user. ) Correct answer data c ₁ (n) and c ₂ (n) are created.

Further, the detection unit 20C uses the vector v (n) represented by the following equation (15) as the input feature amount.

The vector v (n) represented by the equation (15) is a 516-dimensional vector obtained by concatenating the logarithms of the absolute values of the spectra of the frame and the immediately preceding frame. Therefore, the detection of the target sound section and the overlapping section is formulated from the vector v (n) to the estimation of the _{two-dimensional vector c (n) = [c 1} (n), c _{2 (n)] representing the correct answer data.} can do.

Here, the configuration of the neural network model is defined by the following equations (16) to (20).

When the number of nodes of the intermediate layer is 100, the size of the matrix _{W i} and the matrix _{W 0} are each a 100 × 516,2 × 100. Therefore, the sizes of the matrix W ₁ , the matrix W ₂ , and the matrix W ₃ are all 100 × 100.

Further, the function sigmoid () in the equations (16) to (20) represents an operation in which the sigmoid function represented by the following equation (21) is applied to each element of the vector.

Then, the objective function L is defined by the cross entropy represented by the following equation (22).

Then, the detection unit 20C _{obtains the parameter sequences Wi} , W _o , W ₁ , W ₂ , and W ₃ that maximize the objective function L by learning.

As the learning method, an existing method such as the stochastic gradient descent method may be used. _{The function u 1} (n) and the function u ₂ (n) derived using this model are continuous values between 0 and 1. Therefore, for example, 0.5 may be set as a threshold value, and if it is more than that, it may be binarized to 1 (target sound section), and if it is less than that, it may be binarized to 0 (non-target sound section).

As described above, the detection unit 20C uses the output spectrum Y (f, n) and the frequency spectrum X ₁ (f, n) to set the target sound section and the overlapping section by a method different from that of the first embodiment. It may be detected.

(Second Embodiment)
In the present embodiment, a mode in which sound signal processing is performed using the first audio signal acquired from the first microphone 14 without using the second audio signal acquired from the second microphone 16 will be described.

FIG. 4 is a schematic diagram showing an example of the sound signal processing system 2 of the present embodiment.

The sound signal processing system 2 includes a sound signal processing device 11 and a plurality of first microphones 14. The sound signal processing device 11 and the plurality of first microphones 14 are connected so as to be able to exchange data and signals.

That is, the sound signal processing system 1 of the first embodiment is provided with the sound signal processing device 11 instead of the sound signal processing device 10 and is not provided with the second microphone 16. Is similar to.

In the present embodiment, the sound signal processing system 2 assumes a plurality of target sound sources 12A as the sound source 12. In FIG. 4, as the plurality of target sound sources 12A, the target sound sources 12A1 to 12A3, which are three speakers, are shown as an example. The target sound source 12A is, for example, a person (speaker). In the present embodiment, it is assumed that one speaker (target sound source 12A1, target sound source 12A2, target sound source 12A3) sits and talks on each of the three sides of the rectangular table T. In this embodiment, it is assumed that the positions of the plurality of target sound sources 12A do not move significantly during the sound signal processing by the sound signal processing device 11. The number of target sound sources 12A is not limited to 3, and may be 2 or 4 or more.

Similar to the first embodiment, the sound signal processing system 2 includes a plurality of first microphones 14. In the present embodiment, as an example, four first microphones 14 of the first microphone 14A to the first microphone 14D are shown.

Similar to the first embodiment, the plurality of first microphones 14 have different sound arrival time differences from each of the plurality of target sound sources 12A. That is, the arrangement positions of the plurality of first microphones 14 are adjusted in advance so that the sound arrival time differences are different from each other.

Further, the number of the plurality of first microphones 14 provided in the sound signal processing system 2 may be equal to or greater than the number of the sound sources 12 of the present embodiment. Therefore, in the present embodiment, the number of the first microphones 14 may be 3 or more. As the number of the first microphones 14 increases, the accuracy of emphasizing the target sound can be improved.

As an example, a mode in which the sound signal processing system 2 includes four first microphones 14 (first microphones 14A to 14D) will be described.

Similar to the first embodiment, the third signal is output from each of the plurality of first microphones 14, so that the sound signal processing device 11 outputs the plurality of third signals. Similar to the first embodiment, a sound signal obtained by combining a plurality of third sound signals into one will be referred to as a first sound signal.

The sound signal processing device 11 includes an AD conversion unit 18, a sound signal processing unit 30, and an output unit 22. The AD conversion unit 18 and the output unit 22 are the same as those in the first embodiment. The sound signal processing device 11 is the same as that of the first embodiment except that the sound signal processing unit 30 is provided instead of the sound signal processing unit 20. The sound signal processing device 11 may be configured to include at least the sound signal processing unit 30, and at least one of the AD conversion unit 18 and the output unit 22 may be configured as a separate body.

The sound signal processing unit 30 receives a plurality of third sound signals via the AD conversion unit 18. The sound signal processing unit 30 emphasizes the target sound signal included in the first sound signal that combines the received plurality of third sound signals into one, and outputs the emphasized sound signal to the output unit 22.

The sound signal processing unit 30 will be described in detail.

FIG. 5 is a schematic diagram showing an example of the functional configuration of the sound signal processing unit 30.

The sound signal processing unit 30 includes a conversion unit 30B, a separation unit 30J, a detection unit 30C, a correlation derivation unit 30D, a plurality of third correlation storage units 30E, a fourth correlation storage unit 30F, and a plurality of addition units. It includes 30K, a plurality of coefficient derivation units 30G, a plurality of generation units 30H, and a plurality of inverse conversion units 30I.

The conversion unit 30B, the separation unit 30J, the detection unit 30C, the correlation derivation unit 30D, the plurality of coefficient derivation units 30G, the plurality of addition units 30K, the plurality of generation units 30H, and the plurality of inverse conversion units 30I may be, for example, one or more. It is realized by the processor of. For example, each of the above-mentioned parts may be realized by causing a processor such as a CPU to execute a program, that is, by software. Each of the above-mentioned parts may be realized by a processor such as a dedicated IC, that is, hardware. Each of the above-mentioned parts may be realized by using software and hardware together. When a plurality of processors are used, each processor may realize one of each part, or may realize two or more of each part.

The third correlation storage unit 30E and the fourth correlation storage unit 30F store various types of information. The third correlation storage unit 30E and the fourth correlation storage unit 30F can be configured by any commonly used storage medium such as an HDD, an optical disk, a memory card, and a RAM. Further, the third correlation storage unit 30E and the fourth correlation storage unit 30F may be physically different storage media, or may be realized as different storage areas of physically the same storage medium. Further, each of the third correlated storage unit 30E and the fourth correlated storage unit 30F may be realized by a plurality of physically different storage media.

The sound signal processing unit 30 is provided with a third correlation storage unit 30E, a coefficient derivation unit 30G, an addition unit 30K, a generation unit 30H, and an inverse conversion unit 30I corresponding to each of the plurality of target sound sources 12A. There is. As described above, in the present embodiment, three target sound sources 12A (target sound source 12A1 to target sound source 12A3) are assumed.

Therefore, in the present embodiment, the sound signal processing unit 30 has three third correlation storage units 30E (third correlation storage units 30E1 to third correlation storage unit 30E3) and three coefficient derivation units 30G (coefficient derivation). Units 30G1 to coefficient derivation unit 30G2), three addition units 30K (addition unit 30K1 to addition unit 30K3), three generation units 30H (generation unit 30H1 to generation unit 30H3), and three inverse conversion units 30I (inverse conversion unit). 30I1 to the inverse conversion unit 30I3) are provided.

The number of target sound sources 12A assumed by the sound signal processing system 2 is not limited to three. For example, the number of target sound sources 12A assumed by the sound signal processing system 2 may be 1, 2, or 4 or more. Then, the sound signal processing unit 30 includes the third correlation storage unit 30E, the coefficient derivation unit 30G, the three addition units 30K, the generation unit 30H, and the inverse conversion unit 30I in the same number as the plurality of target sound sources 12A. It may be configured as such.

The conversion unit 30B receives from the plurality of first microphones 14 (first microphones 14A to 14D) via the AD conversion unit 18 as in the case of the conversion unit 20B of the first embodiment. Each of the sound signals is short-time Fourier transform (STFT), and the frequency spectrum X ₁ (f, n), the frequency spectrum X ₂ (f, n), the frequency spectrum X ₃ (f, n), and the frequency spectrum X ₄ (f). , N) generate a plurality of third sound signals represented by each of.

The frequency spectrum X ₁ (f, n) is a short-time Fourier transform of the third sound signal received from the first microphone 14A. The frequency spectrum X ₂ (f, n) is a short-time Fourier transform of the third sound signal received from the first microphone 14B. Frequency spectrum X _{3 (f,} n) is obtained by the third sound signal received from the first microphone 14C and the short-time Fourier transform. The frequency spectrum X ₄ (f, n) is a short-time Fourier transform of the third sound signal received from the first microphone 14D.

In the present embodiment, a multidimensional vector (four-dimensional vector in the present embodiment) that summarizes the plurality of frequency spectra indicating each of the plurality of third sound signals is used as a frequency spectrum X indicating the first sound signal. This will be described as (f, n). In other words, in this embodiment, the first sound signal is represented by the frequency spectrum X (f, n). The frequency spectrum X (f, n) showing the first sound signal is represented by the following equation (23).

The conversion unit 30B outputs the frequency spectrum X (f, n) indicating the first sound signal to each of the separation unit 30J and the plurality of generation units 30H (generation unit 30H1 to generation unit 30H3).

The third correlation storage unit 30E stores the third spatial correlation matrix. The third spatial correlation matrix shows the spatial correlation matrix of the target sound component in the first sound signal.

As described above, the sound signal processing unit 30 includes three third correlation storage units 30E (third correlation storage units 30E1 to third correlation storage units 30E3) corresponding to each of the plurality of target sound sources 12A.

The third correlation storage unit 30E1 is a third correlation storage unit 30E corresponding to the target sound source 12A1. The third correlation storage unit 30E1 stores the third spatial correlation matrix _φxxa (f, n). The third spatial correlation matrix _φxxa (f, n) shows the spatial correlation matrix of the target sound component of the target sound source 12A1 in the first sound signal. The target sound component of the target sound source 12A1 indicates a component (that is, a spectrum) of the target sound emitted from the target sound source 12A1 included in the first sound signal. The target sound component is separated from the first sound signal by the separation unit 30J described later (details will be described later).

As described above, in the present embodiment, the first sound signal is represented by the frequency spectrum X (f, n) showing the four-dimensional vector. Therefore, the third spatial correlation matrix _φxxa (f, n) is represented by a matrix of 4 × 4 complex numbers for each frequency bin.

Similarly, the third correlation storage unit 30E2 is the third correlation storage unit 30E corresponding to the target sound source 12A2. The third correlation storage unit 30E2 stores the third spatial correlation matrix _φxxb (f, n). The third spatial correlation matrix _φxxb (f, n) shows the spatial correlation matrix of the target sound component of the target sound source 12A2 in the first sound signal. The target sound component of the target sound source 12A2 indicates a component of the target sound emitted from the target sound source 12A2 included in the first sound signal. Third space correlation matrix φ _xxa (f, n) similarly to the third spatial correlation matrix φ _xxb (f, n) is represented by a complex matrix of 4 × 4 for each frequency bin.

The third correlation storage unit 30E3 is a third correlation storage unit 30E corresponding to the target sound source 12A3. The third correlation storage unit 30E3 stores the third spatial correlation matrix _φxxc (f, n). The third spatial correlation matrix _φxxc (f, n) shows the spatial correlation matrix of the target sound component of the target sound source 12A3 in the first sound signal. The target sound component of the target sound source 12A3 indicates a component of the target sound emitted from the target sound source 12A3 included in the first sound signal. The third spatial correlation matrix _φxxc (f, n) is represented by a matrix of 4 × 4 complex numbers for each frequency bin.

The fourth correlation storage unit 30F stores the fourth spatial correlation matrix φ _NN (f, n). The fourth spatial correlation matrix φ _NN (f, n) shows the spatial correlation matrix of the non-objective sound component in the first sound signal. The non-target sound component refers to a component other than the target sound component emitted from each of the target sound sources 12A (target sound source 12A1 to target sound source 12A3) included in the first sound signal. The non-purpose sound component is separated from the first sound signal by the separation unit 30J described later (details will be described later).

_{In the initial state, the fourth spatial correlation matrix φ NN} (f, n) initialized with a zero matrix (φ _NN (f, 0) = 0) is stored in advance in the fourth correlation storage unit 30F as an initial value. Has been done.

On the other hand, in the initial state, the third correlation storage unit 30E1, the third correlation storage unit 30E2, and the third correlation storage unit 30E3 are emitted at the respective positions of the target sound source 12A1, the target sound source 12A2, and the target sound source 12A3, respectively. _{The third spatial correlation matrix φxxa} (f, n), the third spatial correlation matrix _φxxb (f, n), and the third spatial correlation matrix _φxxc (f, n) showing the spatial correlation matrix of the obtained target sound are It is stored in advance as an initial value.

The initial values of the third spatial correlation matrix _φxxa (f, n), the third spatial correlation matrix _φxxb (f, n), and the third spatial correlation matrix _φxxc (f, n) are set to It may be obtained in advance by simulation from the arrangement of each of the plurality of first microphones 14 (first microphones 14A to 14D) and the positions of the plurality of target sound sources 12A (target sound sources 12A to 12A3). Further, the initial values of the third space correlation matrix _φxxa (f, n), the third space correlation matrix _φxxb (f, n), and the third space correlation matrix _φxxc (f, n) are a plurality of target sound sources 12A. The target sounds emitted by each of the target sound sources 12A1 to 12A3 at the positions of the target sound sources 12 are collected in advance by a plurality of first microphones 14 (first microphones 14A to 14D), and the sounds are collected. It may be derived in advance from the obtained target sound signal.

Specifically, the sound signal processing unit 30 transmits the target sounds emitted from the respective positions of the target sound sources 12A1 to 12A3 to a plurality of first microphones 14 (first microphones 14A to first microphones) on the table T. The initial value of each third space correlation matrix may be derived in advance from the target sound signal obtained by collecting the sound in 14D).

For example, speakers are arranged at each position of the target sound source 12A1 to the target sound source 12A3 to reproduce white noise, and the spectrum of the sound collected by the plurality of first microphones 14 (first microphone 14A to first microphone 14D) is displayed. It is assumed that the four-dimensional vector to be represented is represented by Na (f, n), Nb (f, n), and Nc (f, n). In this case, the sound signal processing unit 30 derives the initial value of each third spatial correlation matrix in advance using the following equations (24) to (26), and the third correlation storage unit 30E1 to the third correlation storage. Each of them may be stored in advance in the unit 30E3.

Next, the addition unit 30K1, the coefficient derivation unit 30G1, the generation unit 30H1, and the inverse conversion unit 30I1 corresponding to the target sound source 12A1 will be described.

The addition unit 30K1 is an addition unit 30K corresponding to the target sound source 12A1. _{The addition unit 30K1 is a third space correlation matrix (third space correlation matrix φ xxb} (f, n), third space correlation matrix φ) of the target sound source 12A (target sound source 12A2, target sound source 12A3) other than the corresponding target sound source 12A1. _xxx (f, n)) and the fourth spatial correlation matrix φ _NN (f, n) are added and output to the coefficient derivation unit 30G. Specifically, the addition unit 30K1 derives the sum of the spatial correlation matrix by the following equation (27) and outputs it to the coefficient derivation unit 30G1.

The coefficient derivation unit 30G1 is a coefficient derivation unit 30G corresponding to the target sound source 12A1. Coefficient deriving unit 30G1 is included in the first sound signal, corresponding spatial filter coefficient for emphasizing the target sound signal of the target sound source 12A1 to F _{a (f,} n) to derive a. Specifically, the coefficient derivation unit 30G1 determines _{the spatial filter coefficient F a} (f, n) _{based on the third spatial correlation matrix φ xxa} (f, n) and the fourth spatial correlation matrix φ _NN (f, n). Derived.

Specifically, the coefficient derivation unit 30G1 is the maximum of the matrix represented by the product of _{the third spatial correlation matrix φxxa} (f, n) and _{the inverse matrix of the sum φ SS (f, n) of the spatial correlation matrices. The eigenvector F SNR} (f, n) corresponding to the eigenvalue is derived.

The coefficient deriving unit 30G1 is the eigenvector _F SNR (f, n), the spatial filter coefficients _F a (f, n) corresponding to the target sound source 12A derived as _{(Fa (f, n) =} F SNR (f , N)). The coefficient deriving unit 30G1, like the first embodiment, by adding the post filter w (f, n) and may derive the spatial filter coefficients _F a (f, n).

The generation unit 30H1 is a generation unit 30H corresponding to the target sound source 12A1. Generator 30H1, using the spatial filter coefficients derived by the coefficient deriving unit _{30G1 F a (f, n)} , the frequency spectrum X (f, n) included in the first sound signal, represented by, the target sound source 12A1 Generates an emphasized sound signal that emphasizes the target sound signal of.

In particular, generator 30H1, using the following equation (28), to produce an enhanced sound signal represented by the output spectrum _Y a (f, n). Emphasized sound signal represented by the output spectrum Y _{a (f,} n) is the first sound signal, a sound signal emphasizing a target sound signal of the target sound source 12A.

That is, the generating unit 30H1 the frequency spectrum X (f, n) and the spatial filter coefficients _F a (f, n) and transposed matrix obtained by Hermitian transpose of a product, the output spectrum _Y a (f showing the emphasized sound signal , N).

Generator 30H1 the output spectrum _Y a (f, n) the emphasized sound signal represented by, and outputs it to the inverse transform unit 30I1 and the detection unit 30C. That is, the generation unit 30H1 outputs the emphasized sound signal of the target sound signal of the target sound source 12A to the inverse conversion unit 30I1 and the detection unit 30C.

The inverse conversion unit 30I1 is an inverse conversion unit 30I corresponding to the target sound source 12A1. Inverse transform unit 30I, like the inverse transform unit 20I of the first embodiment, the output spectrum _Y a (f, n) indicating the enhancement signal using the symmetry of the output spectrum _Y a (f, n) A spectrum of 256 points is generated from the above, and an inverse Fourier transform is performed. Next, the inverse transformation unit 30I1 may generate a sound wave shape by applying a composite window function and superimposing it on the output waveform of the previous frame by shifting the frame shift. Then, the inverse transformation unit 30I1 outputs the emphasis sound signal of the target sound source 12A represented by the generated sound wave shape to the output unit 22.

Next, the addition unit 30K2, the coefficient derivation unit 30G2, the generation unit 30H2, and the inverse conversion unit 30I2 corresponding to the target sound source 12A2 will be described. Further, the addition unit 30K3, the coefficient derivation unit 30G3, the generation unit 30H3, and the inverse conversion unit 30I3 corresponding to the target sound source 12A3 will be described.

The points that the information corresponding to the corresponding target sound source 12A is different between the addition unit 30K2, the addition unit 30K3, the coefficient derivation unit 30G2, the coefficient derivation unit 30G3, the generation unit 30H2, the generation unit 30H3, the inverse conversion unit 30I2, and the inverse conversion unit 30I3. Except for the above, the same processing as that of the addition unit 30K1, the coefficient derivation unit 30G1, the generation unit 30H1, and the inverse conversion unit 30I1 is performed.

Specifically, the addition unit 30K2 includes a third spatial correlation matrix _φxxa (f, n), a third spatial correlation matrix _φxxc (f, n), and a fourth spatial correlation matrix φ _NN (f, n). The sum of the spatial correlation matrices of, φ _SS (f, n) is derived and output to the coefficient derivation unit 30G2. This sum φ _SS (f, n) is expressed by the following equation (29).

Then, the coefficient deriving unit 30G2 derives the spatial filter coefficient F _b (f, n) _{based on φ XX b (f, n) and φ SS} (f, n) represented by the equation (29). do. Therefore, the generation unit 30H2 outputs the emphasized sound signal (output spectrum Y _b (f, n)) of the target sound signal of the target sound source 12A2 to the inverse conversion unit 30I1 and the detection unit 30C.

The addition unit 30K3 is a spatial correlation between the third spatial correlation matrix φ _xxa (f, n), the third spatial correlation matrix φ _{xx b} (f, n), and the fourth spatial correlation matrix φ _NN (f, n). The sum of the matrices φ _SS (f, n) is derived and output to the coefficient derivation unit 30G3. This sum φ _SS (f, n) is expressed by the following equation (30).

Then, the coefficient deriving unit 30G3 derives the spatial filter coefficient F _c (f, n) _{based on φXXc (f, n) and φ SS (f, n) represented by the equation (29).} .. Thus, generator 30H3 is an enhanced emphasized sound signal of the target sound signal of the target sound source 12A3 (output spectrum _{Y c (f, n))} , and outputs it to the inverse transform unit 30I2 and the detection unit 30C.

Next, the detection unit 30C will be described. The detection unit 30C detects a target sound section based on the emphasis sound signal. In the present embodiment, the detection unit 30C uses a plurality of emphasized sound signals corresponding to the plurality of target sound sources 12A (target sound sources 12A to 12A3), and the detection unit 30C uses the plurality of emphasized sound signals corresponding to the plurality of target sound sources 12A (target sound sources 12A1 to target sound sources 12A3), and the detection unit 30C uses the plurality of emphasized sound signals emitted from each of the plurality of target sound sources 12A. Detects the target sound section of the sound.

Specifically, the detection unit 30C, from the generation unit 30h1, the output spectrum _Y a (f, n) is represented by, it accepts emphasized sound signal emphasizing a target sound signal of the target sound source 12A1. Further, the detection unit 30C receives an emphasis sound signal emphasizing the target sound signal of the target sound source 12A2 represented by _{the output spectrum Y b (f, n) from the generation unit 30H2.} Further, the detection unit 30C receives an emphasis sound signal emphasizing the target sound signal of the target sound source 12A3 represented by _{the output spectrum Y c (f, n) from the generation unit 30H3.}

Then, the detection unit 30C is based on these emphasized sound signals (output spectrum Y _a (f, n), output spectrum Y _b (f, n), output spectrum Y _c (f, n)), and the target sound source 12A1. -Detects each target sound section of the target sound source 12A3.

Similar to the first embodiment, the target sound section is represented by a function u (n) indicating whether or not the target sound source 12A is emitting sound for each frame number. In this embodiment, the target sound section of the target sound for each target sound source 12A1~ target sound source 12A3, represented by the function _u a (n), the function _u b (n), the function _u c (n). When these functions show a value of "1", they indicate that the target sound source 12A is emitting sound in the nth frame. Further, when the value "0" is shown, it means that the target sound source 12A does not emit sound in the nth frame.

Detector 30C, these functions _u a (n), the function _u b (n), using a function _u c (n), by performing the threshold processing expressed by the equation (31) to (33) Then, the target sound section of the target sound of each target sound source 12A is detected.

In the above equations (31) to (33), t is a threshold value representing the power of the boundary between the target sound and the non-target sound. In the formula (31) to Formula _{(33), P a, P} b, P c are each represented by the following formula (34) to (36).

The detection unit 30C outputs the detection result of the target sound section of each target sound of the plurality of target sound sources 12A (target sound source 12A1 to target sound source 12A3) to the correlation derivation unit 30D.

Next, the separation unit 30J will be described. The separation unit 30J separates the first sound signal into a target sound component and a non-target sound component.

The separation unit 30J receives the frequency spectrum X (f, n) indicating the first sound signal from the conversion unit 30B. As described above, in the present embodiment, the frequency spectrum X (f, n) showing the first sound signal is represented by the above equation (23). Further, in the present embodiment, the frequency spectrum X (f, n) indicates each of the four third sound signals received from each of the four first microphones 14A (first microphones 14A to 14D). It is represented by a four-dimensional vector that summarizes the frequency spectrum.

The separation unit 30J separates the first sound signal represented by the frequency spectrum X (f, n) into the target sound component S (f, n) and the non-target sound component N (f, n). The target sound component S (f, n) is represented by the following formula (37). The non-purpose sound component _Ni (f, n) is represented by the following equation (38).

Then, the separation unit 30J uses a known sound section detection technique, and when the nth frame is the target sound section for all frequencies f, S (f, n) = X (f, n), N (f, n) = [0,0,0,0] is calculated. Further, the separation unit 30J has S (f, n) = [0,0,0,0], N (f, n) when the nth frame is a non-purpose sound section for all frequencies f. = X (f, n) may be set.

Then, the separation unit 30J outputs the target sound component S (f, n) and the non-target sound component N (f, n) separated from the first sound signal to the correlation derivation unit 30D.

The correlation derivation unit 30D has a third spatial correlation matrix of the target sound component in the first sound signal and a non-purpose sound in the first sound signal based on the target sound section, the target sound component, and the non-target sound component. The fourth spatial correlation matrix of the components is derived.

Specifically, the correlation derivation unit 30D receives the target sound component S (f, n) and the non-target sound component N (f, n) from the separation unit 30J. Moreover, the correlation derivation unit 30D, receives from the detector 30C, as a function indicating the target sound section, the function _u a (n), the function _u b (n), the function _u c a (n).

Then, the correlation derivation unit 30D is the target sound component S (f, n), a non-target sound component N (f, n), the function _u a (n), the function _u b (n), the function _u c (n) Based on, the third spatial correlation matrix φ _xxa (f, n), the third spatial correlation matrix φ _xxb (f, n), the third spatial correlation matrix φ _xxx (f, n), and the fourth spatial correlation matrix φ _NN. (F, n) is derived. Then, the correlation derivation unit 30D includes the derived third space correlation matrix _φxxa (f, n), the third space correlation matrix _φxxb (f, n), the third space correlation matrix _φxxc (f, n), and By storing the fourth space correlation matrix φ _NN (f, n) in the third correlation storage unit 30E1, the third correlation storage unit 30E2, the third correlation storage unit 30E3, and the fourth correlation storage unit 30F, respectively. Update these spatial correlation matrices.

Correlation derivation unit 30D _is, u a (n) = 1, and _u b (n) = 0, for and _u c (n) = 0 in the interval (n-th frame), the third space by the following equation (39) The correlation matrix _φxxa (f, n) is derived and updated, and the third spatial correlation matrix _φxxb (f, n) and the third spatial correlation matrix _φxxc (f, n) are not updated.

Moreover, the correlation derivation unit 30D _is, u a (n) = 0, and _u b (n) = 1, for and _u c (n) = 0 in the interval (n-th frame), the the following equation (40) 3 The spatial correlation matrix _φxxb (f, n) is derived and updated, and the third spatial correlation matrix _φxxa (f, n) and the third spatial correlation matrix _φxxc (f, n) are not updated.

Moreover, the correlation derivation unit 30D _is, u a (n) = 0, and _u b (n) = 0, for and _u c (n) = 1 of the section (n-th frame), the the following equation (41) 3 The spatial correlation matrix _φxxc (f, n) is derived and updated, and the third spatial correlation matrix _φxxa (f, n) and the third spatial correlation matrix _φxxb (f, n) are not updated.

Further, the correlation derivation unit 30D _{derives and updates the fourth spatial correlation matrix φ NN} (f, n) by the following equation (42).

In formulas (39) to (42), α is a value of 0 or more and less than 1. The closer the value of α is to 1, the larger the weight of the spatial correlation matrix derived in the past is larger than that of the latest spatial correlation matrix. The value of α may be set in advance. α may be, for example, 0.95.

That is, the correlation derivation unit 30D is the latest third space correlation matrix represented by the multiplication value of the third space correlation matrix derived in the past with the transposed component obtained by Hermitian transposition of the target sound component S (f, n). By correcting, a new third spatial correlation matrix is derived.

_{The correlation derivation unit 30D includes a third spatial correlation matrix φxxa} (f, n) and a third spatial correlation stored in the third correlation storage unit 30E (third correlation storage unit 30E1 to third correlation storage unit 30E3). The matrix _φxxb (f, n) and the third spatial correlation matrix _φxxc (f, n) may be used as the third spatial correlation matrix derived in the past. Further, only one third spatial correlation matrix is stored in each of these third correlation storage units 30E, and the correlation derivation unit 30D sequentially updates them.

Further, the correlation derivation unit 30D transposes the fourth spatial correlation matrix φ _NN (f, n) derived in the past into the non-objective sound component N (f, n) and the non-objective sound component N (f, n) by Hermitian. A new fourth spatial correlation matrix φ _NN (f, n) is derived _{by correcting with the latest fourth spatial correlation matrix φ NN} (f, n) represented by the multiplication value with the transposed component.

Incidentally, the correlation derivation unit 30D is used fourth space correlation matrix φ _NN (f, n) of the already stored in the fourth correlation storage unit 30F, and a fourth spatial correlation matrix phi _NN derived in the past (f, n) Just do it. Further, only one fourth spatial correlation matrix φ _NN (f, n) is stored in the fourth correlation storage unit 30F, and is sequentially updated by the correlation derivation unit 30D.

Next, the procedure of sound signal processing executed by the sound signal processing device 11 of the present embodiment will be described.

FIG. 6 is a flowchart showing an example of a sound signal processing procedure executed by the sound signal processing device 11 of the present embodiment.

The conversion unit 30B performs a short-time Fourier transform on the third signal received from the plurality of first microphones 14 via the AD conversion unit 18 to acquire the first sound signal represented by the frequency spectrum X (f, n). (Step S200). The conversion unit 30B outputs the acquired first sound signal to each of the separation unit 30J and the generation unit 30H (generation unit 30H1 to generation unit 30H3) (step S202).

Next, the separation unit 30J separates the first sound signal into the target sound component S _i (f, n) and the non-target sound component _Ni (f, n) (step S204). Then, the separation unit 30J outputs the target sound component S _i (f, n) and the non-target sound component _Ni (f, n) to the correlation derivation unit 30D.

Next, in the sound signal processing unit 30, the addition unit 30K, the coefficient derivation unit 30G, the generation unit 30H, and the inverse conversion unit 30I1 corresponding to each of the target sound source 12A1 to the target sound source 12A3 perform the processing of steps S206 to S212. To execute. The processes of steps S206 to S212 are executed in parallel between the functions corresponding to each of the plurality of target sound sources 12A (target sound sources 12A to 12A3).

First, the addition unit 30K adds the third space correlation matrix of the target sound source 12A other than the corresponding target sound source 12A and the fourth space correlation matrix φ _NN (f, n), and the coefficient of the corresponding target sound source 12A. Output to the lead-out unit 30G (step S2206).

The coefficient derivation unit 30G reads the third space correlation matrix and the fourth space correlation matrix φ _NN (f, n) of the corresponding target sound source 12A from the third correlation storage unit 30E and the fourth correlation storage unit 30F ( Step S208).

Then, the coefficient derivation unit 30G derives the spatial filter coefficient based on the third spatial correlation matrix and the fourth spatial correlation matrix φ _{NN (f, n) read in step S208 (step S210).}

Next, the generation unit 30H uses the spatial filter coefficient derived in step S210 to generate an emphasis sound signal that emphasizes the target sound signal of the corresponding target sound source 12A included in the first sound signal (step S212). ..

Then, the inverse conversion unit 30I1 outputs the emphasis sound signal generated in step S212 to the output unit 22 (step S214).

The addition unit 30K, the coefficient derivation unit 30G, the generation unit 30H, and the inverse conversion unit 30I1 corresponding to each of the target sound source 12A1 to the target sound source 12A3 execute the processes of steps S206 to S212 to start from the target sound source 12A1. An emphasized sound signal that emphasizes the emitted target sound signal, an emphasized sound signal that emphasizes the target sound signal emitted from the target sound source 12A2, and an emphasized sound signal that emphasizes the target sound signal emitted from the target sound source 12A3. Is output to the detection unit 30C and the inverse conversion unit 30I.

Therefore, the output unit 22 that outputs the emphasis sound signals received from each of the inverse conversion units 30I1 to the inverse conversion unit 30I3 outputs a plurality of emphasis sound signals that emphasize each target sound of the plurality of target sound sources 12A. can do.

Next, the detection unit 30C detects a target sound section of each target sound of the plurality of target sound sources 12A by using a plurality of emphasized sound signals received from the generation unit 30H (generation unit 30H1 to generation unit 30H3). (Step S216).

Next, the correlation derivation unit 30D includes the target sound component S (f, n) and the non-target sound component N (f, n) separated in step S204, and the target sound of each target sound of the plurality of target sound sources 12A. function indicating the section based on the _{(u a (n), u} b (n), u c (n)), the third space correlation matrix corresponding to each of the plurality of target sound source 12A (third space correlation matrix phi _xxa (F, n), the third spatial correlation matrix φ _xxb (f, n), the third spatial correlation matrix φ _xxx (f, n)), and the fourth spatial correlation matrix φ _NN (f, n) are derived ( Step PS218).

Then, the correlation derivation unit 30D includes the derived third space correlation matrix _φxxa (f, n), the third space correlation matrix _φxxb (f, n), the third space correlation matrix _φxxc (f, n), and By storing the fourth space correlation matrix φ _NN (f, n) in the third correlation storage unit 30E1, the third correlation storage unit 30E2, the third correlation storage unit 30E3, and the fourth correlation storage unit 30F, respectively. These spatial correlation matrices are updated (step S220).

Next, the sound signal processing unit 30 determines whether or not to end the sound signal processing (step S222). If a negative determination is made in step S222 (step S222: No), the process returns to step S200. On the other hand, if an affirmative judgment is made in step S222 (step S222: Yes), this routine ends.

As described above, in the sound signal processing device 11 of the present embodiment, the separation unit 30J separates the first sound signal into a target sound component and a non-target sound component. The detection unit 30C detects a target sound section based on the emphasis sound signal. The correlation derivation unit 30D has a third spatial correlation matrix of the target sound component in the first sound signal and a non-purpose sound in the first sound signal based on the target sound section, the target sound component, and the non-target sound component. The fourth spatial correlation matrix of the components is derived. Then, the coefficient derivation unit 30G derives the spatial filter coefficient based on the third spatial correlation matrix and the fourth spatial correlation matrix.

As described above, in the sound signal processing device 11 of the present embodiment, the spatial filter coefficient is used by using the first audio signal acquired from the first microphone 14 without using the second audio signal acquired from the second microphone 16. Is derived. Therefore, in the present embodiment, the target sound signal emitted from the target sound source 12A is highly accurate without preparing the second microphone 16 for collecting the sounds of the non-purpose sound source 12B other than the target sound source 12A. Can be emphasized.

Further, in the sound signal processing device 11 of the present embodiment, the target sound signals of the target sounds of the plurality of target sound sources 12A can be separated and emphasized.

Further, in the sound signal processing device 11 of the present embodiment, the correlation derivation unit 30D sequentially updates the third space correlation matrix and the fourth space correlation matrix. Therefore, even if the positional relationship between the target sound source 12A and the first microphone 14 is assumed to be the third spatial correlation matrix stored as the initial value in the third correlation storage unit 30E, it is actually It will be updated so that it gradually converges to the spatial correlation matrix according to the positional relationship of.

Therefore, the sound signal processing device 11 of the present embodiment can effectively emphasize the target sound signal emitted from the target sound source 12A and suppress the non-target sound signal.

Further, the sound signal processing device 11 of the present embodiment separates the first sound signal into a target sound component and a non-target sound component, and uses them for deriving a spatial correlation matrix. Therefore, the sound signal processing device 11 can generate an emphasis sound signal that effectively suppresses non-purpose sounds such as noise. Therefore, the sound signal processing device 11 can provide a highly accurate emphasized sound signal.

<Modification 2>
The separation unit 30J may separate the first sound signal into a target sound component and a non-target sound component by using a method different from the method shown in the second embodiment.

For example, the separation unit 30J may determine whether the sound is a target sound or a non-target sound for each frequency bin, and may separate the first sound signal into a target sound component and a non-target sound component using the determination result.

For example, the separation unit 30J separates the first sound signal into a target sound component and a non-target sound component by using a neural network.

In this case, the separation unit 30J, using a neural network, for each frame and for each frequency bin, voice mask indicates a value "0" or the value _{"1" M S (f,} n) and non-speech mask _M N (f , N) is estimated. _{Then, the separation unit 30J derives the target sound component S i} (f, n) and the non-target sound component _Ni (f, n) using the following equations (43) and (44).

The separation unit 30J uses the frequency spectrum Xi (f, n) of one channel as the input of the neural network. Then, the separation unit 30J, to the input of each channel, estimates the speech mask _M S (f, n) and non-speech mask _M N (f, n).

Then, the separation unit 30J, using the majority or the like of the estimation results of all the channels, the channel common voice mask M _{S (f,} n) and non-speech mask M _{N (f,} n) may be estimated.

The separation unit 30J may generate training data of the neural network in advance by a simulation using a clean target sound signal that does not contain noise and a non-target sound signal that does not contain the target sound.

The spectrum of a clean target sound signal is represented by _St (f, n), and the spectrum of a non-target sound signal that does not include the target sound is _{represented by N t} (f, n). _{Then, the spectrum X t} (f, n) of the sound on which the non-purpose sound such as noise is superimposed _{, the correct answer data M tS} (f, n) of the voice mask, and the correct answer data M _tN (f, n) of the non-voice mask are obtained. , Derived from the following equations (45) to (47).

In the formulas (46) and (47), t _S and t _N indicate the threshold value of the power ratio between the target sound and the non-target sound.

As the input feature amount, the vector v (n) represented by the following equation (48) is used.

The vector v (n) represented by the equation (48) is a 516-dimensional vector in which the logarithms of the absolute values of the spectra of the frame and the immediately preceding frame are concatenated. The audio mask _M S (f, n) estimation and non-speech mask _M N (f, n) from the vector v (n) indicating the input feature value, 258-dimensional vector c representing the correct answer data (n) As a problem to be estimated, it can be formulated into the following equation (49).

Therefore, the configuration of the neural network model can be defined by the following equations (50) to (54).

Here, it is assumed that the number of nodes in the intermediate layer is 200. Then, the matrix _{W i} size becomes 200 × 516, the size of the matrix _{W o} becomes 258 × 200. Therefore, the sizes of the matrix W ₁ , the matrix W ₂ , and the matrix W ₃ are all 200 × 200.

Here, the objective function L is defined by the cross entropy represented by the following equation (55).

Then, the separation unit 30J _{derives the parameter sequences Wi} , W _o , W ₁ , W ₂ , and W ₃ that maximize the objective function L by learning. As the learning method, a known method such as a stochastic gradient descent method may be used.

_{Then, the separation unit 30J estimates that mi} (n), (i = 1, ..., 258) in the above equation (55) are between 0 and 1, estimated using the model generated by the above method. It becomes a continuous value of. Therefore, for example, the separation unit 30J sets a threshold value of, for example, "0.5", and binarizes the value to "1" if the threshold value is equal to or higher than the threshold value, and to "0" if the threshold value is less than the threshold value. and frequency bins for each voice mask _M S (f, n) and non-speech mask _M n (f, n) to estimate. _{Then, the separation unit 30J may derive the target sound component S i} (f, n) and the non-target sound component _Ni (f, n) by using the above formula (43) and the above formula (44).

In the first modification and the second modification, the case where the component of the neural network is a fully connected network having three intermediate layers has been described as an example. However, the components of the neural network are not limited to this.

For example, if sufficient training data can be prepared, the accuracy may be improved by further increasing the number of layers and the number of nodes in the intermediate layer. Moreover, the bias term may be used. Further, as the activation function, various functions such as relu and tanh can be used in addition to sigmoid. In addition to the fully connected layer, various configurations such as a convolutional neural network and a recurrent neural network can be used. Further, although the FFT power spectrum has been described as the feature amount to be input to the neural network, various feature amounts such as a mel filter bank and a mel cepstrum and a combination thereof can be used.

(Third Embodiment)
The sound signal processing device 10 and the sound signal processing device 11 may be configured to include a recognition unit instead of the sound signal processing system 2.

FIG. 7 is a schematic diagram showing an example of the sound signal processing system 3 of the present embodiment.

The sound signal processing system 3 includes a sound signal processing device 13 and a plurality of first microphones 14. The sound signal processing device 13 and the plurality of first microphones 14 are connected so as to be able to exchange data and signals. That is, the sound signal processing system 3 includes a sound signal processing device 13 instead of the sound signal processing device 10.

The sound signal processing device 13 includes an AD conversion unit 18, a sound signal processing unit 20, and a recognition unit 24. The AD conversion unit 18 and the sound signal processing unit 20 are the same as those in the first embodiment. That is, the sound signal processing device 13 is the same as the sound signal processing device 10 except that the recognition unit 24 is provided instead of the output unit 22.

The recognition unit 24 recognizes the emphasis sound signal received from the sound signal processing unit 20.

Specifically, the recognition unit 24 is a device that analyzes the emphasis sound signal. The recognition unit 24 recognizes, for example, the emphasis signal represented by the output spectrum Y (f, n) by a known analysis method, and outputs the recognition result. This output may be text data or symbolized information such as recognized word IDs. A known recognition device may be used for the recognition unit 24.

(Scope of application)
The sound signal processing device 10, the sound signal processing device 11, and the sound signal processing device 13 described in the above-described embodiments and modifications can be applied to various devices and systems that emphasize the target sound signal.

Specifically, the sound signal processing device 10, the sound signal processing device 11, and the sound signal processing device 13 are various systems and devices that collect and process sound in an environment in which one or more speakers output sound. Can be applied to.

For example, the sound signal processing device 10, the sound signal processing device 11, and the sound signal processing device 13 can be applied to a conference system, a lecture system, a customer service system, a smart speaker, an in-vehicle system, and the like.

The conference system is a system that processes the sound collected by the microphones arranged in the space where one or more speakers speak. The lecture system is a system that processes the sound collected by the microphone arranged in the space where at least one of the lecturer and the student speaks. The customer service system is a system that processes the sound collected by the microphone arranged in the space where the clerk and the customer speak interactively. A smart speaker is a speaker that can use an AI (Artificial Intelligence) assistant that supports interactive voice operations. The in-vehicle system is a system that collects and processes the sounds spoken by the occupants in the vehicle and uses the processing results for driving control of the vehicle and the like.

Next, the hardware configurations of the sound signal processing device 10, the sound signal processing device 11, and the sound signal processing device 13 of the above-described embodiment and modification will be described.

FIG. 8 is an explanatory diagram showing a hardware configuration example of the sound signal processing device 10, the sound signal processing device 11, and the sound signal processing device 13 of the above-described embodiment and modification.

The sound signal processing device 10, the sound signal processing device 11, and the sound signal processing device 13 of the above-described embodiments and modifications include a control device such as a CPU (Central Processing Unit) 51, a ROM (Read Only Memory) 52, and a RAM. It includes a storage device such as (Random Access Memory) 53, a communication I / F 54 that connects to a network for communication, and a bus 61 that connects each unit.

The program executed by the sound signal processing device 10, the sound signal processing device 11, and the sound signal processing device 13 of the above-described embodiment and the modified example is provided by being incorporated in the ROM 52 or the like in advance.

The programs executed by the sound signal processing device 10, the sound signal processing device 11, and the sound signal processing device 13 of the above-described embodiment and modifications are CD-ROMs (Compact) in an installable format or an executable format file. It is configured to be recorded on a computer-readable recording medium such as a Disk Read Only Memory), a flexible disk (FD), a CD-R (Compact Disk Recordable), or a DVD (Digital Versaille Disk) and provided as a computer program product. You may.

Further, the programs executed by the sound signal processing device 10, the sound signal processing device 11, and the sound signal processing device 13 of the above-described embodiment and the modified example are stored on a computer connected to a network such as the Internet, and the network. It may be configured to be provided by downloading via. Further, the program executed by the sound signal processing device 10, the sound signal processing device 11, and the sound signal processing device 13 of the above-described embodiment and the modified example may be provided or distributed via a network such as the Internet. good.

The programs executed by the sound signal processing device 10, the sound signal processing device 11, and the sound signal processing device 13 of the above-described embodiment and the modified example use the computer as the sound signal processing device 10 of the above-described embodiment and the modified example. It can function as each part of the sound signal processing device 11 and the sound signal processing device 13. This computer can read a program from a computer-readable storage medium onto the main storage device and execute the program by the CPU 51.

Although some embodiments and modifications of the present invention have been described, these embodiments and modifications are presented as examples and are not intended to limit the scope of the invention. These novel embodiments and modifications can be implemented in various other embodiments, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are also included in the scope of the invention described in the claims and the equivalent scope thereof.

10, 11, 13 Sound signal processing device 12 Sound source 12A, 12A1, 12A2, 12A3 Target sound source 12B Non-purpose sound source 14, 14A, 14B, 14C, 14D 1st microphone 16 2nd microphone 20, 30 Sound signal processing unit 20C Detection unit 20D Correlation Derivation Unit 20G Coefficient Derivation Unit 20H Generation Unit 24 Recognition Unit 30C Detection Unit 30D Correlation Derivation Unit 30G, 30G1, 30G2, 30G3 Coefficient Derivation Unit 30H, 30H1, 30H2, 30H3 Generation Unit 30J Separation Unit

Claims (14)

Based on the emphasized sound signal the target sound signal emphasizing a coefficient deriving unit that derives the spatial filter coefficients for emphasizing the target sound signal contained in the first sound signal,
A detection unit that detects a target sound section based on the emphasized sound signal,
Based on the target sound section and the first sound signal, the first spatial correlation matrix of the target sound section in the first sound signal and the non-purpose sound section other than the target sound section in the first sound signal. The second space correlation matrix, the correlation derivation unit that derives the second space correlation matrix, and
With
The coefficient derivation unit derives the spatial filter coefficient based on the first spatial correlation matrix and the second spatial correlation matrix.
The detection unit
The target sound section is detected based on the second sound signal in which the power ratio of the non-target sound signal to the target sound signal is larger than that of the first sound signal and the emphasized sound signal.
Sound signal processing device. The coefficient derivation unit is
The spatial filter coefficient is derived based on the emphasized sound signal that emphasizes the target sound signal included in the first sound signal acquired from a plurality of microphones.
The sound signal processing device according to claim 1. The detection unit
Based on the emphasized sound signal, the target sound section and the overlapping section where the target sound signal and the non-target sound signal overlap are detected.
The correlation derivation unit
Based on the target sound section, the overlapping section, and the first sound signal, the first space correlation matrix and the second space correlation matrix are derived.
The sound signal processing device according to claim 1 or 2. The correlation derivation unit
The latest obtained by multiplying the first spatial correlation matrix derived in the past with respect to the first sound signal in the target sound section by the first sound signal and the transposed signal obtained by Hermit translocation of the first sound signal. By correcting with the first space correlation matrix of the above, a new first space correlation matrix is derived.
The second spatial correlation matrix derived in the past for the first sound signal in the non-purpose sound section is represented by a multiplication value of the first sound signal and the transposed signal obtained by Hermit translocation of the first sound signal. A new second space correlation matrix is derived by correcting with the latest second space correlation matrix.
The sound signal processing apparatus according to any one of claims 1 to 3. The coefficient derivation unit is
The eigenvector corresponding to the maximum eigenvalue of the product of the first spatial correlation matrix and the inverse matrix of the second spatial correlation matrix is derived as the spatial filter coefficient.
The sound signal processing device according to claim 3 or 4. Based on the emphasized sound signal the target sound signal emphasizing a coefficient deriving unit that derives the spatial filter coefficients for emphasizing the target sound signal contained in the first sound signal,
A detector that detects the target sound section based on the emphasized sound signal,
A separation unit that separates the first sound signal into a target sound component and a non-target sound component,
Based on the target sound section, the target sound component, and the non-target sound component, the third spatial correlation matrix of the target sound component in the first sound signal and the non-purpose sound component in the first sound signal. A correlation derivation unit that derives the fourth spatial correlation matrix of sound components, and
With
The coefficient derivation unit derives the spatial filter coefficient based on the third spatial correlation matrix and the fourth spatial correlation matrix.
Sound signal processing device. The correlation derivation unit
The third spatial correlation matrix derived in the past is newly corrected by the latest third spatial correlation matrix represented by the multiplication value of the target sound component and the transposed component obtained by Hermitian transposition of the target sound component. Derivation of the third spatial correlation matrix
By correcting the fourth spatial correlation matrix derived in the past with the latest fourth spatial correlation matrix represented by the multiplication value of the non-objective sound component and the transpose of the non-objective sound component Hermitian transposed. , Derivation of the new fourth spatial correlation matrix,
The sound signal processing device according to claim 6. The coefficient derivation unit is
The eigenvector corresponding to the maximum eigenvalue of the product of the third spatial correlation matrix and the inverse matrix of the fourth spatial correlation matrix is derived as the spatial filter coefficient.
The sound signal processing device according to claim 7. Based on the emphasized sound signal the target sound signal emphasizing a coefficient deriving step of deriving the spatial filter coefficients for emphasizing the target sound signal contained in the first sound signal,
A detection step for detecting a target sound section based on the emphasized sound signal, and
Based on the target sound section and the first sound signal, the first spatial correlation matrix of the target sound section in the first sound signal and the non-purpose sound section other than the target sound section in the first sound signal. The second spatial correlation matrix, the correlation derivation step for deriving, and
Including
The coefficient derivation step derives the spatial filter coefficient based on the first spatial correlation matrix and the second spatial correlation matrix.
The detection step
The target sound section is detected based on the second sound signal in which the power ratio of the non-target sound signal to the target sound signal is larger than that of the first sound signal and the emphasized sound signal.
Sound signal processing method. A coefficient derivation step for deriving a spatial filter coefficient for emphasizing the target sound signal included in the first sound signal based on the emphasized sound signal emphasizing the target sound signal, and a coefficient derivation step.
A detection step for detecting a target sound section based on the emphasized sound signal, and
Based on the target sound section and the first sound signal, the first spatial correlation matrix of the target sound section in the first sound signal and the non-purpose sound section other than the target sound section in the first sound signal. The second spatial correlation matrix, the correlation derivation step for deriving, and
Is a program that allows a computer to execute
The coefficient derivation step derives the spatial filter coefficient based on the first spatial correlation matrix and the second spatial correlation matrix.
The detection step
The target sound section is detected based on the second sound signal in which the power ratio of the non-target sound signal to the target sound signal is larger than that of the first sound signal and the emphasized sound signal.
Program. A coefficient derivation unit that derives a spatial filter coefficient for emphasizing the target sound signal included in the first sound signal based on the emphasized sound signal that emphasizes the target sound signal, and a coefficient derivation unit.
A generation unit that uses the spatial filter coefficient to generate the emphasized sound signal that emphasizes the target sound included in the first sound signal.
A recognition unit that recognizes the emphasis signal and
A detection unit that detects a target sound section based on the emphasized sound signal,
Based on the target sound section and the first sound signal, the first spatial correlation matrix of the target sound section in the first sound signal and the non-purpose sound section other than the target sound section in the first sound signal. The second space correlation matrix, the correlation derivation unit that derives the second space correlation matrix, and
With
The coefficient derivation unit derives the spatial filter coefficient based on the first spatial correlation matrix and the second spatial correlation matrix.
The detection unit
The target sound section is detected based on the second sound signal in which the power ratio of the non-target sound signal to the target sound signal is larger than that of the first sound signal and the emphasized sound signal.
Sound signal processing device. Based on the emphasized sound signal the target sound signal emphasizing a coefficient deriving step of deriving the spatial filter coefficients for emphasizing the target sound signal contained in the first sound signal,
A detection step for detecting a target sound section based on the emphasized sound signal, and
A separation step for separating the first sound signal into a target sound component and a non-target sound component, and
Based on the target sound section, the target sound component, and the non-target sound component, the third spatial correlation matrix of the target sound component in the first sound signal and the non-purpose sound component in the first sound signal. A correlation derivation step for deriving the fourth spatial correlation matrix of sound components, and
Including
The coefficient derivation step derives the spatial filter coefficient based on the third spatial correlation matrix and the fourth spatial correlation matrix.
Sound signal processing method. A coefficient derivation step for deriving a spatial filter coefficient for emphasizing the target sound signal included in the first sound signal based on the emphasized sound signal emphasizing the target sound signal, and a coefficient derivation step.
A detection step for detecting a target sound section based on the emphasized sound signal, and
A separation step for separating the first sound signal into a target sound component and a non-target sound component, and
Based on the target sound section, the target sound component, and the non-target sound component, the third spatial correlation matrix of the target sound component in the first sound signal and the non-purpose sound component in the first sound signal. A correlation derivation step for deriving the fourth spatial correlation matrix of sound components, and
Is a program that allows a computer to execute
The coefficient derivation step derives the spatial filter coefficient based on the third spatial correlation matrix and the fourth spatial correlation matrix.
Program. A coefficient derivation unit that derives a spatial filter coefficient for emphasizing the target sound signal included in the first sound signal based on the emphasized sound signal that emphasizes the target sound signal, and a coefficient derivation unit.
A generation unit that uses the spatial filter coefficient to generate the emphasized sound signal that emphasizes the target sound included in the first sound signal.
A recognition unit for recognizing the emphasized sound signal,
A detector that detects the target sound section based on the emphasized sound signal,
A separation unit that separates the first sound signal into a target sound component and a non-target sound component,
Based on the target sound section, the target sound component, and the non-target sound component, the third spatial correlation matrix of the target sound component in the first sound signal and the non-purpose sound component in the first sound signal. A correlation derivation unit that derives the fourth spatial correlation matrix of sound components, and
With
The coefficient derivation unit derives the spatial filter coefficient based on the third spatial correlation matrix and the fourth spatial correlation matrix.
Sound signal processing device.

Images