JP2017195581A - Signal processing device, signal processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a signal processing device achieving both of the clear sound reproduction of a point sound source and the reproduction of an omnidirectional and uniform atmospheric sound.SOLUTION: In a signal processing device 1, an acoustic signal input unit 2 acquires each directional sound associated with a plurality of orientations from acoustic signals which are collected by a plurality of microphone elements 3. A signal analysis processing unit 13, in order to achieve both of the sound reproduction of the point source and the omnidirectional and uniform atmospheric sound, optimizes the orientation of the acquired directional sound. Among the values of beam pattern gains each indicating the directivity of the directional sound associated with each of the plurality of orientations, the signal analysis processing unit 13 sets a ratio of a maximum gain value in the detected point source direction to other values to be larger than a predetermined value. The signal analysis processing unit 13 sets a deviation among the gains of synthetic beam patterns, obtained by synthesizing the directional beam patterns associated with the plurality of orientations, to be smaller than a predetermined value. The signal analysis processing unit 13 controls each orientation in a manner to satisfy these two conditions.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a signal processing device, a signal processing method, and a program for acquiring a direction sound from an acoustic signal.

A technique is known in which sound signals of a plurality of channels are acquired from sound collected by a microphone array including a plurality of microphone elements, and sound for each direction (hereinafter referred to as “direction sound”) is acquired from the sound signals. ing. If the obtained omnidirectional sounds in all directions can be presented to the user so that they are reproduced from the respective directions, a high sense of presence as if they were present can be realized.
As a method for acquiring a direction sound, there is a method based on filtering in addition to a method for acquiring a direction sound from an acoustic signal of sound collected using a microphone array including a microphone element having directivity. In this method, a multi-channel acoustic signal obtained by picking up sound with an omnidirectional microphone array is filtered by a directivity-forming filter coefficient corresponding to a desired directivity direction, so that an arbitrary directivity direction can be obtained. Directional sound can be acquired.
In the technique disclosed in Patent Literature 1, the directivity is rotated by switching the filter coefficient of the directivity forming filter, and the direction in which the sound source exists is estimated. Then, a filter coefficient corresponding to the estimated direction of the sound source is selected, and sound is collected with directivity directed toward the direction of the sound source.

Japanese Patent No. 4898907

Directional sound includes sound of a sound source in a specific direction (hereinafter referred to as “point sound source”) and sound of a non-directional diffused sound source (hereinafter referred to as “atmosphere sound”). . In order to present to the user that each omnidirectional sound is reproduced from each direction, in addition to clearly reproducing the sound of the point sound source, the ambient sound should be reproduced evenly in all directions. Is also important.
Clear point source sound reproduction contributes to the direction of the sound, whereas omnidirectional atmosphere sound reproduction in all directions contributes to the "wrapping feeling" that feels like being wrapped in the sound field. . These senses of direction and envelopment are important elements of the realism of sound.

However, with the technique disclosed in Patent Document 1, it is considered that sound reproduction of a clear point sound source can be realized by collecting sound with directivity in the direction in which the sound source exists. No consideration is given to the reproduction of atmospheric sounds without noise.
The present invention has been made in order to solve the above-described problems, and an object of the present invention is to achieve both the reproduction of a clear point sound source sound and the reproduction of atmospheric sound without any unevenness in all directions.

  The signal processing apparatus according to the present invention includes an acquisition means for acquiring a direction sound for each of a plurality of directional directions from an acoustic signal collected by a plurality of microphone elements, and a detection for detecting a direction of a point sound source from the acoustic signal. And a gain value of a beam pattern indicating the directivity of the directional sound corresponding to each of the plurality of directivity directions, wherein the gain value in the direction of the point sound source detected by the detection means is the largest. Control means for controlling the respective directivity directions so that the ratio with other ones is greater than a predetermined value and the deviation of the gain of the combined beam pattern obtained by combining a plurality of directivity beam patterns is smaller than the predetermined value. And comprising.

  According to the present invention, it is possible to achieve both the reproduction of a clear point sound source sound and the reproduction of an atmospheric sound without any unevenness in all directions.

1 is a block diagram showing a configuration of a signal processing system according to Embodiment 1 of the present invention. FIG. 2 is a block diagram illustrating a configuration of a computer according to the first embodiment. 3 is a flowchart showing a flow of signal analysis processing according to the first embodiment. Explanatory drawing of directivity direction control. Explanatory drawing of directivity direction control. Explanatory drawing of directivity direction control. The flowchart which shows the flow of the signal analysis process which concerns on Embodiment 2 of this invention.

  DESCRIPTION OF EMBODIMENTS Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the accompanying drawings. The embodiment described below is an example as means for realizing the present invention, and should be appropriately modified or changed according to the configuration and various conditions of the apparatus to which the present invention is applied. It is not limited to the embodiment.

<Embodiment 1>
(overall structure)
FIG. 1 is a block diagram showing the configuration of the signal processing system according to the first embodiment of the present invention.
This signal processing system includes a signal processing device 1 that performs signal processing on an acoustic signal, an acoustic signal input unit 2 that inputs the acoustic signal to the signal processing device 1, and a microphone array 3 that collects the sound and outputs the acoustic signal. And a headphone 4 for reproducing a sound corresponding to the supplied acoustic signal. The signal processing device 1 includes a system control unit 11 that controls the entire device, a storage unit 12 that stores various data such as an acoustic signal, a signal analysis processing unit 13 that performs an analysis process of the acoustic signal, and an acoustic signal. Is output to the headphone 4. The acoustic signal input unit 2 is connected to the signal processing device 1 wirelessly or by wire. The signal processing device 1 is connected to the headphones 4 wirelessly or by wire.
Although FIG. 1 shows a configuration in which the headphones 4 are provided outside the signal processing device 1, the configuration may include the headphones 4 in the signal processing device 1, or the signal processing device 1 may be included in the headphones 4. It is good also as a structure to incorporate.

  The microphone array 3 includes, for example, M (six) microphone elements 3a, 3b, 3c, 3d, 3e, and 3f. If at least two microphone elements are provided, directionality can be obtained by forming directivity in an arbitrary directivity direction by selecting a filter coefficient in filter processing to be described later. (M) is not limited to six. Each of the microphone elements 3 a to 3 f collects surrounding sounds, generates an analog sound signal, and outputs the analog sound signal to the sound signal input unit 2. Ambient sounds include point sound and atmosphere sounds. The sound of the point sound source includes, for example, sounds of people, animals, vehicles, musical instruments and the like, and sounds when a ball is attacked in sports such as volleyball. The ambient sound includes background sounds such as reflection / reverberation sound indoors and environmental sound outdoors.

The acoustic signal input unit 2 performs amplification processing, A / D conversion processing, and the like on the six-channel acoustic signals from the respective microphone elements 3a to 3f, and is a digital acoustic signal with a period corresponding to a predetermined acoustic sampling rate. A 6-channel signal is generated. The acoustic signal input unit 2 inputs the generated digital acoustic signal to the storage unit 12.
The headphone 4 includes a reproduction unit 4R that reproduces sound for the right ear and a reproduction unit 4L that reproduces sound for the left ear. The headphones 4 reproduce sound corresponding to the acoustic signal supplied from the acoustic signal output unit 14.
The storage unit 12 stores 6-channel acoustic signals input from the acoustic signal input unit 2. In addition, the storage unit 12 stores a head-related transfer function (HRTF) for the left and right ears and a filter coefficient that forms directivity for obtaining a directional sound.

(Reproduction operation overview)
The signal analysis processing unit 13 detects the direction of the point sound source from the six-channel acoustic signals stored in the storage unit 12 by signal analysis processing described later, and acquires the direction sound to be acquired according to the detected direction of the point sound source. Control the directivity direction. Further, the signal analysis processing unit 13 acquires direction sounds in the respective directional directions from the six-channel acoustic signals by signal analysis processing described later, and reproduces the sound signals (hereinafter referred to as “headphone reproduction”) from the acquired direction sounds using the headphones 4. Signal "). In this generation, the signal analysis processing unit 13 sets a virtual speaker in the directivity direction of each directional sound acquired from the 6-channel acoustic signal stored in the storage unit 12. Further, in consideration of the HRTF stored in the storage unit 12, the signal analysis processing unit 13 adds a sound signal corresponding to the direction sound in the direction of each virtual speaker to generate a headphone reproduction signal. The signal analysis processing unit 13 inputs the generated headphone reproduction signal to the acoustic signal output unit 14.

The acoustic signal output unit 14 performs DA conversion and amplification on the headphone reproduction signal input from the signal analysis processing unit 13 and supplies the resultant signal to the headphone 4. The headphone 4 reproduces a sound corresponding to the headphone reproduction signal supplied from the acoustic signal output unit 14.
As described above, the sound is reproduced by the headphone reproduction signal generated in accordance with the sound of each direction, so that the sound (direction sound) of each channel is reproduced by arranging an actual speaker around the user. Can be presented.

(Hardware configuration)
Each functional block shown in FIG. 1 is stored as a program in a storage unit such as a ROM 22 described later, and is executed by the CPU 21. Note that at least a part of the functional blocks shown in FIG. 1 may be realized by hardware. When realized by hardware, for example, a dedicated circuit may be automatically generated on the FPGA from a program for realizing each step by using a predetermined compiler. FPGA is an abbreviation for Field Programmable Gate Array. Further, a Gate Array circuit may be formed in the same manner as an FPGA and realized as hardware. Further, it may be realized by an ASIC (Application Specific Integrated Circuit).

FIG. 2 shows an example of the hardware configuration of the signal processing apparatus 1. The signal processing device 1 includes a CPU 21, a ROM 22, a RAM 23, an external memory 24, an input unit 25, and an output unit 26.
The CPU 21 performs various operations and controls each part constituting the signal processing device 1 in accordance with the input signal and program. Specifically, the CPU 21 performs detection of the direction of a point sound source, control of a directivity when acquiring a direction sound, acquisition of a direction sound, generation of a headphone reproduction signal, and the like. The functional blocks in FIG. 1 described above illustrate functions executed by the CPU 21.

The RAM 23 stores temporary data and is used for the work of the CPU 21. The ROM 22 stores a program for executing each functional unit shown in FIG. 1 and various setting information. The external memory 24 is a detachable memory card, for example, and can be loaded into a PC (personal computer) or the like to read data.
A predetermined area of the RAM 23 or the external memory 24 is used as the storage unit 12.
The input unit 25 stores the acoustic signal input from the acoustic signal input unit 2 in an area used as the storage unit 12 of the RAM 23 or the external memory 24. The output unit 26 supplies the headphone reproduction signal generated by the CPU 21 to the headphones 4.

(Signal analysis processing)
The flowchart of FIG. 3 is processed by the CPU 21 included in the signal processing device 1 executing a program stored in the ROM 22 or the like. This process is, for example, a process executed when a headphone reproduction signal is generated from an acoustic signal stored in the storage unit 12 and reproduced by the headphones 4, and is a process started in response to a start instruction from the user. is there.
Hereinafter, the signal analysis processing of this embodiment will be described along the flowchart of FIG. Note that the processing of the flowchart of FIG. 3 is performed by the signal analysis processing unit 13 unless otherwise specified.
Before starting the processing of FIG. 3, the sound signals of M channels (6 channels in the configuration of FIG. 1) collected by M (six in the case of FIG. 1) microphone elements 3a to 3f are stored. It is assumed that it is stored in the unit 12. The signal analysis processing unit 13 acquires D directional sounds from the M acoustic signals stored in the storage unit 12 and generates a headphone reproduction signal from the D directional sounds by the process of FIG. In addition, since the directivity in an arbitrary directivity direction can be formed by selecting the filter coefficient in the filter processing for directivity formation, and the direction sound can be acquired, the number D of directional sounds is the number of channels M of the acoustic signal. The number may be the same or different.

(Detection of the direction of a point sound source)
In S <b> 1, the signal analysis processing unit 13 acquires the M channel acoustic signal held in the storage unit 12 and performs Fourier transform for each channel to obtain z (f) that is frequency domain data (Fourier coefficient). obtain. Here, z (f) of each frequency is a vector having M elements.
In S2, the signal analysis processing unit 13 calculates a spatial spectrum P (f, θ) that forms a peak of sensitivity in the point sound source direction in order to detect the direction of the point sound source from the acoustic signal in S3. In this calculation, the signal analysis processing unit 13 includes a spatial correlation matrix R (f) of Equation (1), which is a statistic representing the spatial properties of the acoustic signal, a sound source in each direction (azimuth angle θ), and each microphone element. An array manifold vector a (f, θ) which is a transfer function between 3a to 3f is used.
R (f) = E [z (f) z ^H (f)] (1)
Here, E represents the expected value, and the superscript H represents the complex conjugate transpose. Further, a (f, θ) is frequency domain data (Fourier coefficient), and is composed of M elements.
For example, the spatial spectrum P _MV (f, θ) based on the minimum variance method is obtained by the equation (2).

In addition, a matrix in which elements corresponding to the noise subspace among M eigenvectors of the spatial correlation matrix R (f) are arranged is _denoted by En. At this time, considering the orthogonality with the array manifold vector a (f, θ) belonging to the signal subspace, the spatial spectrum P _MU (f, θ) of the MUSIC (Multiple Signal Classification) method is expressed by Equation (3). can get.

P (f, θ) = P _MV (f, θ) [Expression (2)] or P (f, θ) while changing θ of a (f, θ), for example, in increments of 1 ° from −180 ° to 180 °. By calculating as θ) = P _MU (f, θ) [Expression (3)], a spatial spectrum in all horizontal directions can be obtained. Depending on the structure of the microphone array 3 used to collect sound corresponding to the acoustic signal, the array manifold vector a (f, θ) can be calculated at an arbitrary resolution by a theoretical formula such as free space or a hard sphere. .
In S3, the signal analysis processing unit 13 detects the direction of the point sound source from the acoustic signal based on the spatial spectrum calculated in S2. Specifically, the signal analysis processing unit 13 calculates the average spatial spectrum P _mean (θ) by averaging the spatial spectrum P (f, θ) for each frequency in a range of, for example, f _{min to} f _max . Further, the signal analysis processing unit 13 detects the direction in which the average spatial spectrum P _mean (θ) is a peak (maximum value) and sets it as the point sound source direction θ _sq [q = 1 to Q]. Here, f _min and f _max are lower and upper limit frequencies to be detected in the direction of the point sound source, and Q is the number of detected point sound sources.

(Optimization of directivity)
In S4 to S12, the signal analysis processing unit 13 optimizes the directivity direction of the directional sound to be acquired in order to achieve both the reproduction of a clear point sound source sound and the reproduction of an atmospheric sound with no unevenness in all directions.
The signal analysis processing unit 13 is a beam pattern gain value indicating the directivity of the directional sound corresponding to each of the plurality of directivity directions, and the gain value in the direction of the detected point sound source is the maximum. The ratio with the other is made larger than a predetermined value. In addition, the signal analysis processing unit 13 makes the deviation of the gain of the combined beam pattern obtained by combining the directional beam patterns corresponding to the plurality of directivity directions smaller than a predetermined value. The signal analysis processing unit 13 controls each directivity direction so as to satisfy these two conditions.

If it is only intended to reproduce the atmosphere sound without any unevenness in all directions, for example, as shown in FIG. 4, the directivity directions 31 to 36 of the sound in each direction are all around (−180 ° to 180 °). What is necessary is just to arrange | position equally. FIG. 4 shows a case where the number D of directivity directions of directional sounds is 6, which is the same as the number M of the microphone elements 3a to 3f in FIG. If the number M of microphone elements is two or more, the directivity in an arbitrary directivity direction can be formed by selecting a filter coefficient in the filter processing for forming directivity. Therefore, the number D of directivity directions is equal to the number M of microphone elements. And may be different.
In FIG. 4, the radius of the outermost solid circle and the circle having the same center as the circle correspond to the relative gain of the beam pattern. The circumferential positions of these circles correspond to the orientation of the microphone array 3 from a predetermined reference direction (0 °). Thick one-dot chain lines 31 to 36 indicate the directing direction (main lobe direction) of each direction sound. Thick broken circles 31a to 36a indicate beam patterns indicating the directivity of directional sounds corresponding to the directivity directions 31 to 36, respectively.
In this way, by arranging the directivity directions 31 to 36 of each directional sound evenly, the combined beam pattern 37 obtained by synthesizing the directional sound beam patterns 31a to 36a in each directional direction becomes a substantially circular shape. Reproduction of atmosphere sound without unevenness can be realized.

  By the way, if the directivity direction (main lobe direction) of a certain directivity is in the direction of the point sound source, the energy of the sound captured as the directional sound corresponding to that directivity is the directional sound corresponding to the other directivity. It is considerably larger than the energy of sound that can be perceived as. If the headphone reproduction signal is generated from the direction sound acquired in such a state, the sound of the point sound source is mainly reproduced from the virtual speaker arranged in the direction of the point sound source. For this reason, the sound of the point sound source reproduced from the virtual speaker arranged in the direction of the point sound source is louder than the sound of the point sound source reproduced from the virtual speaker arranged in the other direction. Sound is reproduced clearly.

On the other hand, if the directivity direction does not face the direction of the point sound source, and there is a point sound source between the adjacent directivity directions, it can be captured as a direction sound corresponding to the multiple directivities. There is not much difference in sound energy.
Here, it is assumed that there is a point sound source in the point sound source directions 30a and 30b in FIG. In this case, for example, the ratio of the values of the beam pattern 32a and the beam pattern 33a in the point sound source direction 30a is small. For this reason, the sound of the point sound source corresponding to the point sound source direction 30a is not so much in the direction sound energy 42 in the pointing direction corresponding to the beam pattern 32a and the direction sound energy 43 in the pointing direction corresponding to the beam pattern 33a. There is no difference. When the headphone reproduction signal is generated according to the directional sound acquired in such a state, the difference between the volume of the virtual speaker arranged in the directing direction 32 and the volume of the virtual speaker arranged in the directing direction 33 becomes small. End up.

For this reason, the sound reproduced by the headphones 4 also has a small volume difference between the directivity direction 32 and the directivity direction 33, and the sound reproduction of the point sound source becomes unclear.
For this reason, in the signal processing system of the present embodiment, the directivity directions 32 and 35 closest to the point sound source directions 30a and 30b are directed by the control from the signal analysis processing unit 13, respectively, for example, as shown in FIG. ', 35'. Thereby, for example, the ratio of the gain value of the beam pattern 32a ′ and the beam pattern 33a in the point sound source direction 30a and the gain value of the other beam pattern becomes larger than that in the case of FIG. That is, in the point sound source direction 30a, the ratio between the maximum gain value (beam pattern 32a ′) and the other gain value (beam pattern) is larger than a predetermined value. Therefore, for the point sound source in the point sound source direction 30a, the difference between the direction sound energy 42 'in the directivity direction corresponding to the beam pattern 32a' and the direction sound energy 43 in the directivity direction corresponding to the beam pattern 33a is shown in FIG. It becomes bigger than the case. When a headphone reproduction signal is generated according to the direction sound acquired in such a state and the sound is reproduced by the headphone 4, the sound of the point sound source is reproduced according to the direction sound acquired in the state shown in FIG. It becomes clearer than when the headphone playback signal is generated.

However, when such an arrangement change in the directivity direction is performed, disturbances such as bulges 51 and 53 and dents 52 and 54 occur in the combined beam pattern 37 ′. That is, the combined beam pattern 37 ′ is not substantially circular. For this reason, the atmosphere sound reproduced based on all directional sounds is also uneven.
In addition, when the synthesized beam pattern 37 ′ is disturbed, the following problem occurs in addition to the unevenness of the atmospheric sound. For example, for the point sound source corresponding to the point sound source direction 30a, the sound acquired as the direction sound in the directivity direction corresponding to the beam pattern 32a ′ is reproduced from the virtual speaker arranged in the same directivity direction 32 ′ as the point sound source direction 30a. The In addition to this, the sound acquired as the direction sound in the directivity direction corresponding to the beam pattern 33 a is reproduced from the virtual speaker arranged in the directivity direction 33. In this case, since there is a bulge 51 of the composite beam pattern 37 ′, the direction of the point sound source may be perceived as being shifted toward the bulge 51 of the composite beam pattern 37 ′.
For this reason, in the signal processing system of this embodiment, in order to achieve both the reproduction of a clear point sound source sound and the reproduction of an atmospheric sound with no unevenness in all directions, the signal analysis processing unit 13 is represented by Equation (4). The directivity direction θ _d [d = 1 to D] for obtaining the directional sound is optimized so as to solve the optimization problem.

ここで、σ_ｂｓｕｍ（θ_ｄ）は合成ビームパターンの乱れの目安である標準偏差である。また、点音源のインデックスｑの関数であるｄｍｉｎ（ｑ）は、式（５）のように点音源方向θ_ｓｑ［ｑ＝１〜Ｑ］に最も近い指向方向を示すインデックスである。

Here, σ _bsum (θ _d ) is a standard deviation that is a measure of the disturbance of the combined beam pattern. Further, dmin (q), which is a function of the point sound source index q, is an index indicating the directivity direction closest to the point sound source direction θ _sq [q = 1 to Q] as shown in Equation (5).

Equation (4) is obtained by changing Q directivity directions θ _{dmin (q)} [q = 1 to Q] out of D directivity directions θ _d [d = 1 to D] as point sound source directions θ _sq [q = 1 to 1]. Q] means to minimize the disturbance of the combined beam pattern and to optimize the directivity direction under the constraint condition. The disturbance of the combined beam pattern is evaluated based on a gain deviation of the combined beam pattern, for example, a standard deviation σ _bsum (θ _d ) of the gain. That is, in this optimization problem, each directivity direction becomes an optimization variable, and the standard deviation σ _bsum (θ _d ) becomes an evaluation function. The signal analysis processing unit 13 solves an optimization problem using the thus defined pointing direction as an optimization variable, and determines the solution as each pointing direction.

  Based on the above, the processing of S4 to S12 for optimizing the pointing direction will be described. Note that the optimization of the directivity direction may be considered for at least the representative frequency (for example, 1 kHz) among all frequencies. The frequency index f in the description of S4 to S12 below represents such a representative frequency. The representative frequency may be, for example, the center frequency of a band with high intensity in the acoustic signal.

In S4, the signal analysis processing unit 13 initializes the directivity direction θ _d [d = 1 to D] of each directivity. First, in order to cover all horizontal directions with a plurality of directivities, each directivity is indicated by the number of directivity directions D (= 6) as shown in FIG. 4 with reference to 0 ° in the coordinate system of the microphone array that picks up the sound. The directivity directions 31 to 36 of the sex are evenly arranged. That is, θ ₁ = 0 ° in the pointing direction 31, θ ₂ = 60 ° in the pointing direction 32, θ ₃ = 120 ° in the pointing direction 33, θ ₄ = 180 ° in the pointing direction 34, θ _{5 in the} pointing direction 35 = −. 120 ° and θ _{6 in the} directivity direction 36 = −60 °. Note that if the number of directivity directions D is small, the composite beam pattern will be recessed evenly arranged, so it is preferable to use at least D that starts to be substantially circular (for example, about 6 in the case shown in FIG. 4).

Here, it is assumed that, for example, Q = 2 point sound sources are detected in S3, and θ _s1 = 85 ° in the point sound source direction 30a and θ _s2 = −148 ° in the point sound source direction 30b. In this case, none of the evenly directed directivity directions θ _d [d = 1 to D] is directed to the point sound source direction θ _sq [q = 1 to Q], and thus does not satisfy the constraint condition of Expression (4). . Therefore, the signal analysis processing unit 13 directs the directivity directions 32 and 35 closest to the point sound source directions 30a and 30b to the point sound source direction, and sets the directivity directions 32 ′ and 35 ′ as shown in FIG. The directivity direction θ _d [d = 1 to D] is initialized so as to satisfy the constraint condition 4). Accordingly, θ ₁ = 0 ° in the pointing direction 31, θ ₂ = 85 ° in the pointing direction 32 ′, θ ₃ = 120 ° in the pointing direction 33, θ ₄ = 180 ° in the pointing direction 34, θ in the pointing direction 35 ′. ₅ = −148 ° and θ _{6 in the} directivity direction 36 = −60 °.

Subsequent S5 to S11 are processes related to iterative optimization calculation. The signal analysis processing unit 13 repeatedly executes the processes of S5 to S11 in the optimization loop. S5 to S6 are processing for each directivity in which the directivity direction is initialized in S4. The signal analysis processing unit 13 repeatedly executes the processes of S5 to S6 in the directivity loop.
In S5, the signal analysis processing unit 13 acquires a filter coefficient for forming the directivity targeted in the current directivity loop. Here, the filter coefficient w _d (f) corresponding to the directivity direction θ _d is acquired from the filter coefficient stored in the storage unit 12. Here, the filter coefficient w _d (f) is vector data (Fourier coefficient) in the frequency domain, and is composed of M elements. In addition, since the filter coefficients differ when the configuration of the microphone array 3 is different, a type ID indicating the type of the microphone array 3 used for sound collection may be recorded as additional information of the acoustic signal. In this case, the signal analysis processing unit 13 may acquire the filter coefficient of the microphone array 3 corresponding to the type ID from the storage unit 12 and use it in the processing of this step.

The array manifold vector a (f, θ) is generally used for calculating the directivity forming filter coefficient. As a method of forming a directional main lobe in the directivity direction θ _d , for example, in the case of the delay sum method, the array manifold vector a _d (f) in the θ _d direction is used, and w _d (f) = _ad (f) filter coefficients are obtained as ^{_{/ (a d H (f)}} a d (f)).
In S6, the signal analysis processing unit 13 calculates a directivity beam pattern using the directivity forming filter coefficient w _d (f) acquired in S5 and the array manifold vector a (f, θ). . The value b _d (f, θ) in the azimuth angle θ direction of the beam pattern is obtained by Expression (6).
b _d (f, θ) = w _d ^H (f) a (f, θ) (6)

By calculating b _d (f, θ) while changing θ of the array manifold vector a (f, θ) in increments of 1 ° from −180 ° to 180 °, for example, a beam pattern in all horizontal directions is obtained. It is done. When the microphone elements are isotropically arranged, such as a circular equidistant microphone array, the beam pattern b ₁ (f, θ) when the directing direction is 0 ° in front is sequentially rotated. Thus, it is possible to obtain beam patterns b _d (f, θ) [d = 2 to] having other directivities.
In S7, the signal analysis processing unit 13 synthesizes the beam patterns b _d (f, θ) [d = 1 to D] of the directivities calculated in S6, so that the combined beam pattern is expressed by Expression (7). b _sum (f, θ) is calculated.

In S8, the signal analysis processing unit 13 converts the combined beam pattern b _sum (f, θ) into, for example, decibel [dB] display to calculate the standard deviation σ _bsum (θ _d ), and optimizes the expression (4). An evaluation function for the optimization problem. Here, since the standard deviation is a function of the directivity direction θ _d [d = 1 to D], it is expressed as σ _bsum (θ _d ), and the frequency index f is omitted.
In S9, the signal analysis processing unit 13 determines whether or not the optimization in the optimization loop has converged. If it has converged, the process proceeds to S12, and if it has not converged, the process proceeds to S10. The determination as to whether or not it has converged is, for example, the amount of decrease with respect to the value at the time of execution of the optimization loop before the evaluation function value [standard deviation σ _bsum (θ _d ) in the case of equation (4)] is less than a predetermined value. This is done by determining whether or not. Alternatively, the convergence may be determined depending on whether the difference between the pointing direction θ _d [d = 1 to D], which is an optimization variable, and the value obtained in the previous optimization loop is less than a predetermined value. . Or you may determine with having converged when the evaluation function value of the present optimization loop becomes less than predetermined value. In this case, since the standard deviation is used as the evaluation function, the pointing direction is controlled so as to make the standard deviation of the combined beam pattern smaller than a predetermined value by executing the process of the optimization loop until convergence.

In S10, the signal analysis processing unit 13 determines whether or not the number of updates of the directivity direction θ _d [d = 1 to D] in the optimization loop has reached a predetermined upper limit value. If reached, the process proceeds to S12. If not, the process proceeds to S11.
In S11, the signal analysis processing unit 13 updates the directivity direction. That is, in the state where Q directivity directions θ _{dmin (q)} [q = 1 to Q] are fixed (constrained) to the point sound source directions θ _sq [q = 1 to Q] based on the constraint condition of Expression (4), (DQ) number of directivity directions are updated in a direction in which the standard deviation σ _bsum (θ _d ) of the combined beam pattern decreases. If an optimization problem is defined by a mathematical expression as in Expression (4) (hereinafter referred to as “formulation”), various known optimization algorithms can be used for updating the pointing direction as an optimization variable. Can be applied. Alternatively, the pointing direction may be updated by full search or random search instead of the optimization algorithm.

When the optimization converges or the number of updates in the optimization loop reaches the upper limit value, the signal analysis processing unit 13 optimizes the evaluation function (in this case, the standard deviation σ _bsum (θ _d )) in _S12. The directivity direction θ _d [d = 1 to D] is selected. Thereby, the directivity direction at the time of acquiring a direction sound is optimized. That is, the deviation of the gain of the combined beam pattern obtained by combining a plurality of directional beam patterns is minimized under the constraint condition of Expression (4).
For example, FIG. 6 shows the result of optimizing the pointing direction initialized as shown in FIG. Here, the directivity directions 31, 33, 34, and 36 in FIG. 5 are optimized to the directivity directions 31 ′, 33 ′, 34 ′, and 36 ′ in FIG. 6, respectively. That is, θ ₁ in the pointing direction 31 ′ = − 31.5 ° [−31.5 °], θ _{3 in the} pointing direction 33 ′ = 26.9 ° [−93.1 °], and θ _{4 in the} pointing direction 34 ′. = 148.6 ° [-31.4 °], θ _{6 in} the pointing direction 36 ′ = − 89.9 ° [−29.9 °]. Note that the numerical value in the brackets [] is the update amount from the initial value in the directivity direction, and is schematically represented by an arrow in FIG.

  For example, with respect to the point sound source direction 30a, the ratio of the gain value of the beam pattern 32a ′ having the largest directivity beam pattern gain value to the gain values of the other beam patterns 33a ′ and 34a ′ is shown in FIG. It is getting bigger. For the point sound source in the point sound source direction 30a, the directional sound energy 42 'in the directional direction corresponding to the beam pattern 32a' is the directional sound energy 43 ', 44' in the directional direction corresponding to the beam patterns 33a 'and 34a'. It will be considerably larger than In other words, the ratio of the gain value of the directivity beam pattern in the direction of the point sound source to the maximum value is set larger than the predetermined value. Note that this ratio may be maximized.

  Similarly, the pointing direction is optimized for the point sound source direction 30b. As a result, for each of the detected direction of the plurality of point sound sources, the gain value of the directional beam pattern, and the ratio of the gain value in the direction of the point sound source to the maximum value is the predetermined value. It is set larger. In the state shown in FIG. 6, if a direction sound is acquired from an acoustic signal, a headphone reproduction signal is generated according to the acquired direction sound, and sound is reproduced by the headphones 4 according to the headphone reproduction signal, Sound can be reproduced clearly.

In addition, in the state shown in FIG. 6, since the synthesized beam pattern 37 ″ is substantially circular, it is possible to reproduce the atmospheric sound without any unevenness in all directions. Specific of the standard deviation σ _bsum (θ _d ) of the synthesized beam pattern For example, the combined beam pattern 37 in FIG. 4 is 0.21 dB, the combined beam pattern 37 ′ in FIG. 5 is 1.65 dB, and the combined beam pattern 37 ″ in FIG. 6 is 0.29 dB. That is, in the state of FIG. 6 in which the pointing direction is optimized, the disturbance of the combined beam pattern can be suppressed to the same level as in the case of the uniform arrangement in FIG. 4 while directing the pointing direction toward the point sound source direction. That is, by the above processing, the difference between the gain when reproducing the direction sound of the acoustic signal corresponding to the direction of the point sound source and the gain when reproducing the direction sound other than the direction of the point sound source is smaller than a predetermined value. Thus, the directivity direction is set. In such a state, if direction sound is acquired from the acoustic signal, a headphone reproduction signal is generated according to the acquired direction sound, and sound is reproduced by the headphone 4 according to the headphone reproduction signal, unevenness is obtained in all directions. No atmosphere sound can be reproduced. Therefore, it is possible to reproduce both the sound of a clear point sound source and the atmosphere sound without any unevenness in all directions.
Here, if the microphone elements are arranged isotropically, such as a circular equidistant microphone array, and the number of point sound sources Q = 1, any one of the directivity directions θ _d [d = 1 to D]. _Is directed to the direction of the point sound source θ _s1 , and the other directivity direction θ _d of equal arrangement may be rotated by the same angle.

  In contrast, the arrangement of microphone elements is not isotropic, the shape of the beam pattern that can be formed differs depending on the direction of the direction, and there are multiple point sound sources. When it becomes complicated, an expression such as Expression (4) is required to derive an appropriate pointing direction. This is because, in the example of FIG. 4 where there are two point sound sources, the amount of update from the initial value of the optimized pointing direction of FIG. 6 is different. In particular, the pointing direction 33 ′ is different from the pointing direction 31 ′. It can also be seen from the result of being in the direction 32 ′.

In addition to formula (4), various formulas for defining an optimization problem in order to achieve both the reproduction of a clear point sound source sound and the reproduction of an atmospheric sound with no unevenness in all directions are conceivable. For example, the equation (8) indicates that the difference between the directivity direction θ _{dmin (q)} [q = 1 to Q] closest to the point sound source direction and the point sound source direction θ _sq [q = 1 to Q] is the threshold Δθ _q [q = 1 to Q] This is an expression that minimizes the disturbance of the combined beam pattern as an evaluation function under the constraint condition.

Here, the threshold value Δθ _q [q = 1 to Q] may be changed for each point sound source. For example, a point sound source having a larger average spatial spectrum peak (maximum value) in S3 is prioritized and the threshold value Δθ _q is set smaller. You may make it do. As a result, point sound sources with high priority can be clarified by directing the pointing direction accurately, and point sources with low priority can allow slight deviations in the pointing direction and suppress the disturbance of the combined beam pattern accordingly. it can. As a result, even if the expression (4) is the same as the evaluation function, the constraint condition can be described more flexibly.
Also, in the optimization problem, it is possible to define such that the constraint condition is incorporated into the evaluation function. For example, the equation (9) indicates that the sum of the differences between the directivity direction θ _{dmin (q)} [q = 1 to Q] closest to the point sound source direction and the point sound source direction θ _sq [q = 1 to Q] and the combined beam pattern This is an expression that minimizes the weighted sum of the disturbance as an evaluation function.

Here, λ _q [q = 1 to Q] is a weight representing the priority of the point sound source. For example, a point sound source having a larger average spatial spectrum peak (maximum value) in S3 is prioritized and λ _q is set larger. You may make it do. Β _θ adjusts the trade-off (priority) between the omnidirectional atmosphere sound according to the first term of the equation (9) and the sound reproduction of the clear point sound source according to the second term. The weight to be. Note that, for example, the user may be able to indirectly adjust this trade-off via a GUI (Graphical User Interface) unit (not illustrated) controlled by the system control unit 11. By using the GUI unit, for example, a slider bar in which both ends are emphasized with a sense of direction and a feeling of emphasis is displayed, and a user instruction is input. Signal analyzing processing section 13, in accordance with an instruction of the user, may give priority to reproduction of apparent point source of sound bars positions by increasing the extent beta _theta closer to sense of direction emphasis. Then, as the position of the bar is wrapped and the feeling is closer to emphasis, β _θ may be decreased to give priority to an atmosphere sound that is uniform in all directions.

  In the equation constraint equation such as Equation (4), for example, even when two point sound sources are close to each other, the two pointing directions are directed to the respective point sound sources. It will swell. On the other hand, if optimization is performed using equations such as Equation (8) and Equation (9), two point sound sources are covered with one directivity while allowing a slight deviation in the directivity direction. A result that suppresses the disturbance of the combined beam pattern can be expected.

(Directional sound acquisition and headphone playback signal generation)
S13 to S16 (precisely, processing in a frequency loop including these) indicate processing for acquiring a direction sound from the M channel acoustic signal stored in the storage unit 12 and generating a headphone reproduction signal.
Since S13 to S15 are processes for each frequency, the signal analysis processing unit 13 repeatedly executes the processes of S13 to S15 in the frequency loop. Moreover, since the process of S13-S15 is also the process for every directivity which determined the directivity direction by S12, the signal analysis process part 13 repeatedly performs the process of S13-S15 in a directivity loop.
In S13, the signal analysis processing unit 13 acquires the filter coefficient w _d (f) for forming the directivity targeted in the current directivity loop, as in S5. That is, the signal analysis processing unit 13 acquires the filter coefficient w _d (f) corresponding to the directivity direction θ _d from the directivity forming filter coefficient held in the storage unit 12.

In S14, the signal analysis processing unit 13 performs a filtering process on the Fourier coefficient z (f) of the M-channel acoustic signal acquired in S1 using the directivity-forming filter coefficient w _d (f) acquired in S13. As a result, the signal analysis processing unit 13 generates a direction sound Y _d (f) in the directivity direction θ _d corresponding to the current directivity loop as shown in Expression (10). Y _d (f) is frequency domain data (Fourier coefficient).
Y _d (f) = w _d ^H (f) z (f) (10)
In S15, the signal analysis processing unit 13 adds the left and right ear HRTFs [H _L (f, F, F) in the same direction as the directivity direction θ _d to the Fourier coefficient Y _d (f) of the direction sound in the directivity direction θ _d acquired in S14. θ _d ), H _R (f, θ _d )]. Further, the signal analysis processing unit 13 adds the multiplication result to the left and right headphone reproduction signals X _L (f) and X _R (f) as shown in Expression (11).

Here, X _L (f) and X _R (f) are frequency domain data (Fourier coefficients). In addition, what is necessary is just to acquire and use what is stored in the memory | storage part 12 for HRTF. Performing this step in the directivity loop is equivalent to sequentially arranging virtual speakers that reproduce direction sounds in each directivity direction around the user.
Thereafter, in S16, the signal analysis processing unit 13 performs inverse Fourier transform on each of the Fourier coefficients X _L (f) and X _R (f) of the headphone reproduction signal generated in the processes in S13 to S15, and the headphones having a time waveform. Reproduction signals x _L (t) and x _R (t) are generated. Further, the signal analysis processing unit 13 inputs the generated headphone reproduction signals x _L (t) and x _R (t) to the acoustic signal output unit 14.
In addition, you may perform the process of S13-S15 not in the frequency domain but in the time domain, and in that case, the inverse Fourier transform of this step becomes unnecessary.

In S <b> 17, the acoustic signal output unit 14 performs DA conversion and amplification on the headphone reproduction signals x _L (t) and x _R (t) input from the signal analysis processing unit in S <b> 16 and supplies them to the headphones 4. The headphone 4 reproduces a sound corresponding to the supplied headphone reproduction signal.
In FIG. 1, the acoustic input unit 2 is illustrated separately from the microphone array 3, but may be provided integrally with the microphone array 3. Alternatively, the sound input unit 2 may be provided so as to be included in the signal processing device 1.

(effect)
As described above, the signal analysis processing unit 13 minimizes the ratio between the maximum beam pattern in each pointing direction in the direction of the detected point sound source and the other beam pattern, and combines the beam patterns in each pointing direction. Each directivity direction is determined so as to minimize pattern disturbance.
The signal analysis processing unit 13 acquires each directional sound according to each directional direction determined as described above, and generates a headphone reproduction signal using the acquired directional sound. Furthermore, by reproducing the sound with the headphones 4 in accordance with the headphone reproduction signal, the signal processing system according to the present embodiment makes it possible to reproduce both the sound of a clear point sound source and the reproduction of the atmospheric sound without any unevenness in all directions. be able to.

<Embodiment 2>
In the first embodiment, the direction of the point sound source is detected from the acoustic signal of the sound collected by the plurality of microphone elements, but the present invention is not limited to such an embodiment. For example, the direction of the point sound source may be detected based on the video. An embodiment in which the direction of a point sound source is detected based on video will be described below as a second embodiment. In the following description, differences from the first embodiment will be mainly described. The same reference numerals as those in the first embodiment are used for the same configurations as those in the first embodiment. The headphones 4 according to the first embodiment are replaced with, for example, a head mounted display (HMD).

(Constitution)
In addition to the configuration of the signal processing system of the first embodiment shown in FIG. 1, the signal processing system of the second embodiment captures an object and outputs an image signal such as a camera (not shown) and an image from the imaging unit. A video signal input unit that inputs a signal to the storage unit 12. The video signal input unit performs processing such as AD conversion and encoding on the video signal input from the imaging unit, and inputs the video signal to the storage unit 12 as a digital video signal. By providing the imaging unit and the video signal input unit, the video signal can be acquired simultaneously with the acquisition of the acoustic signal and stored in the storage unit 12. In this signal processing system, the direction of a point sound source is detected from a video signal.

In the first embodiment, by optimizing the directivity direction, the ratio of the energy of the sound captured as a directional sound in a certain directional direction in the direction of the point sound source to the energy of the sound captured as a directional sound in other directional directions is calculated. Was indirectly maximizing.
On the other hand, in the second embodiment, the energy ratio of the sound captured as the direction sound in each directivity direction is directly maximized. For this purpose, first, from the value b _d (f, θ _sq ) [d = 1 to D] in the point sound source direction of the directional beam pattern in each directional direction, the energy ratio r _q ( θ _d ) [q = 1 to Q] is determined.

Here, since the energy ratio is a function of the directivity direction θ _d [d = 1 to D], it is expressed as r _q (θ _d ), and the frequency index f is omitted. Further, dmax (f, θ _sq ) is an index in the directivity direction as expressed by the equation (13), and b _dmax (f, θ _sq ) is each of the point sound source directions θ _sq [q = 1 to Q]. This is the maximum value of the beam pattern value b _d (f, θ _sq ) [d = 1 to D].

The signal analysis processing unit 13 achieves both the reproduction of a clear point sound source by maximizing the energy ratio and the reproduction of atmospheric sound without any unevenness in all directions by minimizing the disturbance of the combined beam pattern. For this purpose, the signal analysis processing unit 13 optimizes the directivity direction θ _d [d = 1 to D] of each directivity so as to be a solution to the optimization problem defined by Expression (14).

Equation (14) minimizes, as an evaluation function, a weighted sum of the sign inversion value of the sum of the energy ratios r _q (θ _d ) [q = 1 to Q] in the point sound source direction and the disturbance of the combined beam pattern. It is a formula. Here, the sign inversion value is used to convert the energy ratio maximization problem into a minimization problem. Further, μ _q [q = 1 to Q] is a weight representing the priority of the point sound source, and μ _q is set to be larger as the point sound source has a higher priority. In addition, β _r is a trade-off (priority) between the reproduction of the atmospheric sound having no unevenness in all directions according to the first term of the formula (14) and the reproduction of the sound of the clear point sound source according to the second term. Is a weight to adjust.
Note that, similarly to the first embodiment, for example, the user may be able to indirectly adjust this trade-off via a GUI unit (not shown) controlled by the system control unit 11.

(Signal analysis processing)
Hereinafter, the signal analysis processing of this embodiment will be described along the flowchart of FIG. As in the first embodiment, the signal analysis processing unit 13 performs the processing of the flowchart in FIG. 7 unless otherwise specified.
Note that before starting the processing of FIG. 7, M-channel (six channels in the configuration of FIG. 1) sound signals and video collected by M (six in the case of FIG. 1) microphone elements 3a to 3f. It is assumed that the video signal input from the signal input unit is stored in the storage unit 12.

Since the process of S21 is the same as the process of S1 of Embodiment 1 of FIG. 3, description is abbreviate | omitted.
In S <b> 22, the signal analysis processing unit 13 acquires the video signal held by the storage unit 12, executes video recognition processing, and detects a subject (object) that can be a point sound source. Specifically, for example, the signal analysis processing unit 13 executes processes such as known face recognition and mouth recognition (speech recognition), or uses a known machine learning technique, so that humans, animals, An object that can emit sound such as a vehicle or a musical instrument is detected. Further, the signal analysis processing unit 13 may detect, for example, a ball at the moment of attack in sports such as volleyball as an object from the inversion of the motion vector detected from the video signal.
In S23, the signal analysis processing unit 13 calculates the direction of the point sound source from the Q objects detected in S22. In a coordinate system having the origin of the center of the video signal (which coincides with 0 ° front of the microphone array coordinate system), if the horizontal pixel coordinate of the object (for example, the center of the object detection frame) is U, a point sound source The direction θ _sq [q = 1 to Q] can be calculated by the following equation (15).

Here, V is the horizontal shooting angle of view of the video signal, and B is the number of horizontal pixels of the video signal.
(Priority setting example)
Note that the signal analysis processing unit 13 may set the priority of the point sound source according to the object detected in S22.
Specifically, for example, the priority of the point sound source may be set according to the size of the detection frame of the object. Alternatively, for example, a point sound source having a smaller detection frame (the number of horizontal pixels) has a narrower horizontal range on the video signal, so that the priority may be increased because it is necessary to clarify.

In addition, when the video signal is, for example, an omnidirectional video signal, and only a part of the video signal is displayed according to the head movement by the head-mounted HMD, the point sound source is selected according to the display range displayed on the HMD. The priority may be set. Alternatively, for example, the priority may be increased as the point sound source is closer to the center of the display range. In addition, when the point sound source that the user wants to show as a video is out of the display range, the effect of gaze guidance may be aimed at by increasing the priority and clarifying.
Also, since the priority of the point sound source is related to the loudness, the point sound source priority is determined from both the sound and the video in combination with the detection of the point sound source by the acoustic signal as in the first embodiment. Also good.

In S24, the signal analysis processing unit 13 initializes the directivity direction θ _d [d = 1 to D] of each directivity. However, unlike the equation (4) of the first embodiment in which the directivity direction constraint is present, the equation (14) of the present embodiment has no constraint condition, and therefore the directivity directions 31 to 36 arranged as shown in FIG. May be the initial value.
S25 to S32 are processes related to iterative optimization calculation, and the processes of S25 to S32 are repeatedly executed in the optimization loop.
Since the process of S25-S27 is the same as the process of S5-S7 of Embodiment 1, description is abbreviate | omitted.

In S28, the signal analysis processing unit 13 calculates the energy ratio r _q (θ _d ) [q = 1 to Q] in the point sound source direction as shown in Expression (12). In the case of FIG. 4, for the point sound source [q = 1] corresponding to the point sound source direction 30a, the sound energy 42 captured as the direction sound corresponding to the beam pattern 32a is the largest, and then the direction sound corresponding to the beam pattern 33a. The sound energy 43 that is captured as Moreover, the energy of the sound perceived as the directional sound corresponding to the beam patterns 31a, 34a to 36a is relatively small. Therefore, in this case, the energy ratio r ₁ (θ _d ) in the point sound source direction 30 a is approximately the ratio of the sound energy 42 and 43.
In S29, the signal analysis processing unit 13 and the standard deviation σ _bsum (θ _d ) of the combined beam pattern calculated in S27 and the energy ratio r _q (θ _d ) [q = 1 to Q in the point sound source direction calculated in S28. ], An evaluation function for the optimization problem shown in Expression (14) is calculated.
Since the process of S30-S38 is the same as the process of S9-S17 of Embodiment 1, description is abbreviate | omitted.

In addition to the equation (14), an equation that defines an optimization problem for achieving both the sound reproduction of a clear point sound source and the reproduction of atmosphere sound that is uniform in all directions can be considered. For example, the following equation (16) is obtained by combining the beam ratio under the constraint that the energy ratio r _q (θ _d ) [q = 1 to Q] in the direction of the point sound source is greater than or equal to the threshold Δr _q [q = 1 to Q]. This is an expression that minimizes pattern disturbance as an evaluation function.

Here, the threshold value Δr _q [q = 1 to Q] may be changed for each point sound source, or Δr _q may be set larger for a point sound source having a higher priority.
As described above, in the signal processing system according to the present embodiment, by reproducing the direction of the directivity for obtaining the direction sound, it is possible to reproduce the sound of a clear point sound source and the atmosphere sound that is uniform in all directions. Can be compatible.
In the signal processing system according to the second embodiment, the head-mounted head mounted display is used. However, any device other than the head mounted display can be used as long as it can receive a reproduction signal from the signal processing device 1. Also good.

<Other embodiments>
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.
In the above-described embodiment, processing such as input of an acoustic signal and generation of a headphone reproduction signal is realized by one signal processing system. However, each processing can be configured as a separate signal processing system or signal processing device. .
Further, various data that the storage unit 12 holds in advance in the above-described embodiment may be input from the outside via a data input / output unit (not shown) controlled by the system control unit 11. .

Moreover, it is good also as a structure with which the headphone 4 is equipped with the sensor which can detect a user's head movement, for example. In such a configuration, for example, for each predetermined length of an acoustic signal (hereinafter referred to as “acoustic frame”), the HRTF used in S15 or S36 is switched according to the user's head movement detected by the sensor. Head tracking processing may be performed.
Needless to say, if the processing of each of the above-described embodiments is performed for each acoustic frame of the acoustic signal, it can be applied to a moving point sound source. That is, while tracking the moving point sound source, the directivity direction for acquiring each direction sound is sequentially controlled so as to minimize the disturbance of the combined beam pattern. At this time, it is preferable to use the optimization result in the previous acoustic frame as the initial value of the pointing direction in each acoustic frame.

  Instead of the microphone array 3, a microphone array in which omnidirectional microphone elements are arranged in a matrix may be used. When such a microphone array is used, an acoustic signal in a desired direction can be generated by performing filter processing for forming directivity on the acoustic signal collected by each microphone element. The direction sound in each directivity direction can be acquired by adjusting the filter coefficient in the filter processing for forming directivity. For this reason, even in such a configuration, by optimizing the directivity direction, it is possible to achieve both reproduction of a clear point sound source sound and reproduction of atmosphere sound without any unevenness in all directions.

Further, instead of the headphones 4, sound signals may be reproduced by a plurality of speakers arranged around the user. In this case, instead of generating the headphone playback signal in S13 to S15 or S34 to S36, a playback signal for each speaker is generated from the acquired direction sound. Furthermore, the sound signal may be reproduced by an ambient speaker system having a mechanism capable of controlling the arrangement direction of each speaker. In this case, in consideration of mechanical constraints, a constraint condition may be imposed so that the order of each directivity direction is not changed in the optimization problem.
The signal processing device 1 itself may be configured to have functions such as sound collection (microphone array), shooting (camera), and display (display). Further, a remote live system can be realized by separating the function for photographing / sound collection and the function for displaying / reproducing and so as to operate synchronously in a remote place.

  In each of the above-described embodiments, the sound reproduction of a clear point sound source in all horizontal directions and the reproduction of the atmosphere sound without unevenness in all directions are compatible, but the target direction range is arbitrarily set. May be. For example, not only the horizontal direction but also all directions including the elevation direction may be set as the target direction range, or may be limited to the horizontal front half, the angle of view range of the captured video signal, and the like. In this case, for example, the standard deviation, which is an indication of the disturbance of the combined beam pattern, is calculated from the combined beam pattern in the target direction range instead of all horizontal directions.

Further, a directional microphone array having a mechanism capable of controlling the axial direction of each microphone element having directivity may be used. In this case, a constraint condition may be imposed so that the order of each directivity direction is not changed in the optimization problem in consideration of the constraints on the mechanism.
Further, in each of the above-described embodiments, the case where the disturbance of the combined beam pattern is evaluated based on the standard deviation of the gain around the combined beam pattern has been described. However, the evaluation is performed using other deviations. Also good. For example, the difference between the gain value and a predetermined value may be obtained in all directions of the synthesized beam pattern, and the disturbance of the synthesized beam pattern may be evaluated by the sum of the obtained differences. Thereby, similarly to each embodiment, the deviation of the gain of the combined beam pattern can be made smaller than a predetermined value.

  DESCRIPTION OF SYMBOLS 1 ... Signal processing apparatus, 2 ... Acoustic signal input part, 3a, 3b, 3c, 3d, 3e, 3f ... Microphone element, 4 ... Headphone, 11 ... System control part, 12 ... Memory | storage part, 13 ... Signal analysis process part, 14 ... Acoustic signal output section

Claims (21)

Obtaining means for obtaining directional sound for each of a plurality of directional directions from an acoustic signal of sound collected by a plurality of microphone elements;
Detecting means for detecting the direction of the point sound source from the acoustic signal;
A beam pattern gain value indicating the directivity of the directional sound corresponding to each of the plurality of directivity directions, the gain value in the direction of the point sound source detected by the detection means being the largest and other values Control means for controlling each directivity direction so as to make a deviation of a gain of a combined beam pattern obtained by combining a plurality of beam patterns having a ratio larger than a predetermined value and a combined beam pattern smaller than a predetermined value;
A signal processing apparatus comprising: The detection means detects directions of the plurality of point sound sources from the acoustic signal,
The control means, for each of the direction of the plurality of point sound sources detected by the detection means, is a beam pattern gain value indicating the directivity of the directional sound corresponding to each of the plurality of directivity directions, 2. The signal processing according to claim 1, wherein each of the directivity directions is controlled so that a ratio of a gain value in the direction of the point sound source to a maximum value is larger than a predetermined value. apparatus.   3. The signal processing apparatus according to claim 1, wherein the acquisition unit acquires the directional sound by performing filter processing for forming directivity corresponding to the directivity direction on the acoustic signal. 4. .   The signal processing apparatus according to claim 1, wherein the control unit solves an optimization problem using the directivity direction as an optimization variable and sets the solution as the directivity direction.   The optimization problem is minimization of a deviation in gain of the combined beam pattern under a constraint that restricts the directivity of the number of point sound sources to the direction of the point sound sources. 5. The signal processing device according to 4.   The optimization problem is that a minimum deviation in gain of the combined beam pattern under the constraint that a difference between a directivity direction closest to the direction of the point sound source and a direction of the point sound source is equal to or less than a first threshold value. The signal processing apparatus according to claim 4, wherein   The optimization problem is minimization of a weighted sum of a sum of differences between the pointing direction closest to the direction of the point sound source and a direction of the point sound source and a gain deviation of the combined beam pattern. 5. The signal processing apparatus according to claim 4, wherein   5. The optimization problem is minimization of a deviation in gain of the combined beam pattern under a constraint that makes the ratio in the direction of the point sound source equal to or greater than a second threshold. A signal processing device according to 1.   5. The optimization problem according to claim 4, wherein the optimization problem is minimization of a weighted sum of a sign inversion value of the sum of the ratios in the direction of the point sound source and a deviation in gain of the combined beam pattern. Signal processing equipment.   The signal processing apparatus according to claim 7, further comprising adjustment means for adjusting a weight in the weighted sum.   The detecting means obtains a spatial spectrum using a spatial correlation matrix of the acoustic signal, calculates an average spatial spectrum by averaging, and detects a direction of the point sound source from a maximum value of the average spatial spectrum. The signal processing device according to claim 1.   The signal processing apparatus according to claim 11, wherein the control unit gives priority to a point sound source having a large maximum value of the average spatial spectrum.   13. The input device according to claim 1, further comprising an input unit configured to input a video signal, wherein the detection unit detects a direction of the point sound source according to a position of an object detected from the video signal. The signal processing device according to item.   The signal processing apparatus according to claim 13, wherein the control unit sets the priority of the point sound source according to the size of the detected object.   15. The signal processing apparatus according to claim 13, wherein the control unit sets the priority of the point sound source according to a display range of the video signal.   The signal processing apparatus according to claim 1, further comprising a generation unit that generates a reproduction signal from the directional sound acquired by the acquisition unit.   The signal processing apparatus according to claim 16, wherein the generation unit generates the reproduction signal using a head-related transfer function.   The signal processing apparatus according to claim 1, further comprising head-mounted display means for displaying an image. Detection means for detecting the direction of a point sound source based on acoustic signals collected by a plurality of microphone elements;
Control means for setting the plurality of directivity directions in order to reproduce the direction sound of the acoustic signal by output for each of the plurality of directivity directions, and the direction of the point sound source detected by the detection means is set to the plurality of directivity The direction of the sound signal corresponding to the direction of the point sound source detected by the detection means and the gain for reproducing the direction sound of the acoustic signal detected by the detection means, and the direction detected by the detection means Control means for setting another directivity direction among the plurality of directivity directions so that a difference from a gain when reproducing the direction sound of the acoustic signal corresponding to a direction different from the direction of the point sound source is smaller than a predetermined value. When,
A signal processing apparatus comprising: Detecting the direction of the point source;
The ratio of the maximum value to the other value in the direction of the detected point sound source of each directional beam pattern for acquiring the directional sound in each directional direction is increased, and the directional beam patterns are synthesized. Controlling the directivity direction so as to reduce disturbance of the combined beam pattern;
Obtaining a directional sound for each directional direction from acoustic signals collected by a plurality of microphone elements;
A signal processing method characterized by comprising:   A computer program for causing a computer to function as the signal processing device according to any one of claims 1 to 19.

Images