KR20180003384A - System and method for capturing 3d sound - Google Patents

Abstract

A system for capturing a stereophonic sound is disclosed. The system for capturing a stereophonic sound includes: a main microphone to which a sound signal is inputted; at least two or more auxiliary microphones which are arranged at positions different from each other around the main microphone and to which the sound signal is inputted; a directivity extracting module for extracting directional information of the sound signal from the sound signal inputted to the auxiliary microphones; and a sound signal integration module for integrating the sound signal with the directional information of the sound extracted from the directivity extracting module. Accordingly, the present invention can provide a vivid stereophonic sound to a user by combining the sound signal combined with the extracted directional information.

Description

[0001] SYSTEM AND METHOD FOR CAPTURING 3D SOUND [0002]

The present invention relates to a stereophonic capturing system and method, and more particularly to a system and method capable of three-dimensionally capturing sound in a variety of electronic devices.

Recently, various electronic devices such as head-mount display (HMD), smart glass, smart phone, smart pad, and mobile camera are available for use in a mobile environment. These electronic devices include functions for vividly displaying pictures or moving pictures of actual images. These functions focus on the image from the viewpoint that the view is the most sensitive and can accept a large amount of information from the human senses.

However, in order to provide more vivid information, it is required to provide sound information together. When you provide sound information together, you can feel the vivid feeling of being in the field because of the various sounds heard around you, including the human voice.

Most of the electronic devices that have been released so far focus on image transmission and use a single microphone to transmit only the user's voice. Recently, a technology for providing a microphone for separately capturing a noise sound to remove ambient noise has been developed, but there is a limitation in providing stereo information in a three-dimensional manner.

The present invention provides a stereophonic sound capturing system and method capable of stereoscopically providing a user's voice signal and a surrounding acoustic signal.

A stereophonic sound capturing system according to the present invention includes: a main microphone for receiving an acoustic signal; At least two auxiliary microphones arranged at positions different from each other around the main microphone and receiving the acoustic signals; A direction extracting module for extracting direction information of the acoustic signal from the acoustic signal input to the auxiliary microphones; And a sound signal integration module for integrating the direction information of the sound extracted by the direction extraction module and the sound signal.

In addition, the auxiliary microphones may be arranged on the same plane as the main microphone.

The auxiliary microphones may be divided into at least two auxiliary microphones arranged on the first plane with the main microphones and at least two auxiliary microphones arranged on a second plane different from the first plane .

The acoustic signal filtering module may further include a noise canceling filter for extracting an ambient noise signal included in the acoustic signal and removing the ambient noise signal through reverse phase filtering. An acoustic echo cancellation filter for eliminating an echo signal included in the acoustic signal; And a speech extraction filter for extracting a speech signal included in the acoustic signal.

The directional extraction module may calculate an input time difference of the acoustic signal input to each of the auxiliary microphones, calculate an incident angle of the acoustic signal to the auxiliary microphones according to the input time difference, Can be extracted.

In addition, the main microphones and the auxiliary microphones may be attached to electronic devices.

A method for capturing stereophonic sound according to the present invention includes: inputting a sound signal through a main microphone and at least two auxiliary microphones arranged around the main microphone; Extracting directional information of the acoustic signal from the acoustic signal input to the auxiliary microphones; And integrating the direction information with the acoustic information input to the main microphone.

In addition, the auxiliary microphones may be positioned on the same plane as the main microphone and may input the acoustic signals.

The acoustic signal filtering step may further include an acoustic signal filtering step of filtering signals included in the acoustic signal input to the main microphones and the auxiliary microphones, wherein the acoustic signal filtering step comprises filtering the ambient noise signals included in the acoustic signals by anti- Lt; / RTI > Removing an echo signal included in the acoustic signal; And extracting a voice signal included in the acoustic signal.

The step of extracting the directional information of the acoustic signal may further include calculating an input time difference of the acoustic signal input to each of the auxiliary microphones and calculating an incident angle of the acoustic signal to the auxiliary microphones corresponding to the input time difference, And the directionality of the sound can be extracted.

According to the present invention, it is possible to provide the user with a vivid stereo sound by extracting the direction information of the sound signal and providing the sound signal combined with the extracted direction information.

1 is a diagram of a stereophonic sound capturing system according to an embodiment of the present invention.
2 is a diagram showing microphone array characteristic parameters.
FIG. 3 is a diagram illustrating a problem that may occur when microphones are arranged in a linear array.
4 is a view illustrating a microphone array according to an embodiment of the present invention.
5 is a view showing a microphone array disposed in an electronic device according to the criteria of FIG.
6 is a view illustrating a microphone array according to another embodiment of the present invention.
FIGS. 7 and 8 are views respectively showing electronic devices to which the microphone array of FIG. 6 is coupled.
9 is a view showing a microphone array mounted on a glass.
10 is a diagram showing a microphone array arranged in a mobile device.
11 is a view illustrating a microphone array arranged in a mobile device according to another embodiment of the present invention.
12 is a view showing a microphone array arrangement according to another embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the technical spirit of the present invention is not limited to the embodiments described herein but may be embodied in other forms. Rather, the embodiments disclosed herein are provided so that the disclosure can be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

In this specification, when an element is referred to as being on another element, it may be directly formed on another element, or a third element may be interposed therebetween. Further, in the drawings, the thicknesses of the films and regions are exaggerated for an effective explanation of the technical content.

Also, while the terms first, second, third, etc. in the various embodiments of the present disclosure are used to describe various components, these components should not be limited by these terms. These terms have only been used to distinguish one component from another. Thus, what is referred to as a first component in any one embodiment may be referred to as a second component in another embodiment. Each embodiment described and exemplified herein also includes its complementary embodiment. Also, in this specification, 'and / or' are used to include at least one of the front and rear components.

The singular forms "a", "an", and "the" include plural referents unless the context clearly dictates otherwise. It is also to be understood that the terms such as " comprises "or" having "are intended to specify the presence of stated features, integers, Should not be understood to exclude the presence or addition of one or more other elements, elements, or combinations thereof. Also, in this specification, the term "connection " is used to include both indirectly connecting and directly connecting a plurality of components.

In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear.

1 is a diagram of a stereophonic sound capturing system according to an embodiment of the present invention.

1, a stereophonic capturing system 10 includes a microphone array 110, a multiple input signal input interface module 120, an acoustic signal filtering module 130, a directional extraction module 140, a sound field parameter extraction module 140, (150), and a sound signal integration module (160).

The microphone array 110 is attached to an electronic device as means by which a sound signal is input. The acoustic signal includes a voice signal of a user of the electronic device, a voice signal of the people around the user, background noise, peripheral noise, and the like. Electronic devices can include various mobile devices such as smart phones, smart pads, and mobile cameras, and various wearable devices such as HMDs (head mounted displays) and smart glasses.

The microphone array 110 includes a main microphone 210 and a secondary microphone 220. The main microphone 210 can mainly receive the voice signal of the user of the electronic device. At least two auxiliary microphones 220 are provided and arranged at different positions around the periphery of the main microphone 210. The audio signal of the people around the user, the background noise, the noise of the peripheral device, and the like can be mainly input to the auxiliary microphone 220.

The multiple input signal input interface module 120 receives the acoustic signals input simultaneously through the main microphone 210 and the auxiliary microphones 220. When the main microphone 210 and the auxiliary microphone 220 are analog microphones, the multiple input signal input interface module 120 can convert an analog input sound signal into a digital signal. If the main microphone 210 and the auxiliary microphone 220 are microphones that support I2S, PDM, and PCM interfaces, they perform the interface function of the corresponding protocol.

The acoustic signal filtering module 130 removes or reduces the intensity of acoustic signals outside the area of interest, such as background noise and peripheral noise, from the acoustic signal passed through the multiple input signal input interface module 120. Specifically, the acoustic signal filtering module 130 includes a noise removing filter 131, an acoustic echo removing filter 132, and an acoustic characteristic extracting filter 133.

The noise reduction filter 131 extracts the background noise in the space and the acoustic characteristics of the peripheral noise, and removes it through the anti-phase filter, thereby minimizing the distortion of the speech signal and reducing the noise.

The acoustic echo cancellation filter 132 extracts and removes the frequency characteristic of the generated echo in real time in order to remove the echo generated between the microphones 210 and 220 and the speaker when using the speaker system.

The acoustic characteristic extraction filter 133 extracts only a voice signal in a noisy environment and suppresses noise through a voice detector algorithm.

The microphone array characteristic parameter 111 defines a characteristic parameter of the microphone array 110. [

2 is a diagram showing microphone array characteristic parameters.

Referring to FIG. 2, the microphone array characteristic parameter 111 includes a near field sound field model / far field model 112 and a microphone directivity pattern model 113.

The near field field model / far field field model 112 may be classified into a near field field model or a far field field model according to the distance between the installation position of the microphone array 110 and the sound source of a voice signal The sound field model defines the characteristic parameters of the microphones.

The microphone directivity pattern model 113 defines a directional pattern of each microphone 210, 220 for improved accuracy with respect to directional capture of the microphone array 110. The microphones 210 and 220 may be an omnidirectional microphone or a directional microphone.

The directional extraction module 140 extracts the directional information of the voice signal and tracks the position of the sound source of the voice signal. The directional extraction module 140 may track the location of the source of sound through DOA (Direction of Arrival) and beamforming for three-dimensional capturing of the audio signal and directional acoustic rendering. The directional extraction module 140 may obtain the delay time according to the arrival direction of the voice signal through the DOA and then calculate the incident angle by cross-correlation calculation to extract the directionality. Then, the beam focus is formed through the beam forming technique in the extracted direction to give the directional characteristic, thereby minimizing the collection of the acoustic information for the user voice and other sound sources. Also, if there are a large number of users, directionality of multiple sound sources can be simultaneously extracted through DOA and beamforming.

The sound field parameter extraction module 150 models the room sound and integrates the modeled sound field parameters and the captured sound signal in real time. The modeling of the room acoustic model models the room acoustic characteristics through the impulse responses generated in the space and stores the corresponding impulse responses to define the sound field parameters. Then, by integrating the modeled sound field parameters and the captured sound signals in real time, the sound signals are converted into sound signals reflecting the room acoustic characteristics of the room and reproduced.

The acoustic signal integration module 160 integrates the direction information of the sound source extracted by the directional extraction module 140 and the sound field parameters defined by the sound field parameter extraction module 140 into the sound signals passed through the sound signal filtering module 130, Thereby generating a stereo sound signal.

The number of microphones, the geometric shape of the microphone array, and the directional pattern of the microphone are important factors in generating the above-mentioned stereo sound signal. In order to determine the geometry of the microphone array, the range of the microphone array's direction of sound reception and the frequency range of the main source must be considered.

The microphone array 110 can arrange the main microphone 210 and the auxiliary microphones 220 in at least one of a linear array, a semi-circular array, and a rectangular array.

Linear arrays are defined to have a range of morbidity direction of +/- 45 degrees from the vertical line.

Semicircular arrays are defined in the range of the mute direction to a range of ± 90 ° from the vertical line of the main microphone.

Rectangular arrays are necessary to overcome the spatial aliasing problem of linear arrays.

FIG. 3 is a diagram illustrating a problem that may occur when microphones are arranged in a linear array.

Referring to FIGS. 3 (a) and 3 (b), in the case of a linear array, when the phase difference between specific frequencies arriving at two microphones at 0 ° and 45 ° is 0 or 2π, °, as shown in FIG. Also, when it is derived as 45 ° as shown in FIG. 3 (c), it can not be accurately determined whether the position of the actual sound source (source 1, source 2) is the front face or the rear face of the microphone.

In contrast, square arrays can improve errors arising in a linear array through directional results derived from a combination of various spacing and placement between each microphone.

In order to extract the directionality of a sound source, two or more microphones capable of covering the entire frequency band are required. In addition, the definition of proper spacing between microphones is required.

The spacing of the microphones is determined in terms of the target frequency range. When the use of a microphone array is mainly for voice transmission, the target frequency range is set to 300 to 3400 Hz, and accordingly, the distance between the microphones (d _sep = λ / 2) is required to be at least 5.0 to 56.6 cm.

In the case of a linear array, the number of microphones to be arrayed can be determined by the following equation (1).

d _max / d _min = M (M-1) / 2 Equation (1)

here,

d _max : Maximum distance between microphones

d _min : Minimum distance between microphones

M: Number of microphones

According to one example,

? = 2 * 0.05 = 0.10

f = c /? = 343.5 / 0.1 (at 20 ° C) → about 3400 Hz

(λ: wavelength, c = speed of sound, f: frequency)

In the case of a square array, the number of microphones required can be reduced by the combination of the microphones, as compared with the case of the linear array.

The direction in which the microphone array inputs a voice signal, that is, the array directivity (D), is obtained by dividing the acoustic direction intensity in a specific direction by the isotropic radiant intensity (P _total ) Can be defined.

식 (2)

Equation (2)

Then, the total isotropic sound radiation intensity (P _total ) can be defined as the sum of the values obtained by equally dividing the complete sphere surrounding the microphone array into the intensities of the uniform angle grid, as shown in equation (3).

식 (3)

Equation (3)

In equation (3), M is the number of grid points at high angles and N is the number of grid points at azimuths.

Since the radiated acoustic intensity is divided into beam patterns, the direction of sound reception of the microphone array can be expressed in the form of the direction (B (θ, φ)) of the beam pattern as shown in Equation (4) below.

식(4)

Equation (4)

Hereinafter, the directional extraction method using the microphone array will be described.

The directional extraction module 140 uses the time interval (τ _d ) and the sound velocity (ν) of the acoustic signal arriving at each of the microphones, The directionality of the sound source can be extracted

식 (5)

Equation (5)

In detail, the following equation (6) shows the cross-correlation calculation, and the direction of the sound source can be extracted through the relationship between microphone signals of one phase.

식(6)

Equation (6)

Equation (7) can be derived by applying the frequency-weighted phase transformation weighting (PHAT) function (ψ) to Eq. (6) and the more accurate sound source direction in the reverberation environment and noise environment Can be extracted.

식 (7)

Equation (7)

이며,here,

Lt;

The frequency weighted PHAT function (ψ) can be defined by the following equations (8) and (9).

식 (8)

Equation (8)

식(9)

Equation (9)

Hereinafter, various embodiments of the microphone array arranged in the electronic apparatus according to the above-described microphone array placement criterion will be described.

4 is a view illustrating a microphone array according to an embodiment of the present invention.

Referring to FIG. 4, a plurality of auxiliary microphones 221 to 228 are arranged three-dimensionally around the main microphone 210. Some of the auxiliary microphones 221 to 228 may be positioned on the same plane 11 as the main microphone 210 and the remaining ones of the auxiliary microphones 221 to 228 may be positioned on a plane different from that of the main microphone 210 12). ≪ / RTI > These planes 11 and 12 may be arranged side by side.

According to the embodiment, eight auxiliary microphones 211 to 218 are provided. The four microphones 221 to 224 are positioned on the XY plane 11 with the main microphone 210. The four microphones 225 to 228 are arranged in a rectangular shape around the main microphone 210. The main microphone 210 is located at the center of the interval ([Delta] x, [Delta] y) between the auxiliary microphones 211 to 214 in the X axis direction and the Y axis direction.

The remaining four microphones 215 to 218 are located at different heights from the main microphone 210 in the Z-axis direction. The remaining four microphones 215 to 218 are located at corresponding positions with the auxiliary microphones 211 to 214 located on the same plane as the main microphone 210 in the Z axis direction.

5 is a view showing a microphone array disposed in an electronic device according to the criteria of FIG.

Referring to FIG. 5, the electronic device 300 may be provided with an HMD. It is possible to extract acoustic direction information in various frequency ranges according to the interval between the auxiliary microphones 221 to 228 disposed in the HMD 300. [ According to the embodiment, it is possible to extract acoustic directional information of a minimum interval of 5 cm to 3400 Hz between the auxiliary microphones 221 to 228, and auxiliary microphones 221 to 228 of the interval between the maximum diagonal length of 26 cm of the HMD 300 It is possible to extract acoustic directional information of 650 Hz or more.

6 is a view illustrating a microphone array according to another embodiment of the present invention.

Referring to FIG. 6, the auxiliary microphones 221 to 217 are arranged in a semicircular shape around the main microphone 210. The auxiliary microphones 221 to 227 are arranged on the same plane 11 as the main microphone 210.

6 may be applied to the HMDs 310 and 320 provided with curved front surfaces as shown in FIGS. 7 and 8.

The auxiliary microphones 221 to 227 are arranged in the Uniform Semi-Circular Array form and arranged at the same angular intervals in the left-right direction centering on the front center. According to the embodiment, the auxiliary microphones 221 to 223 and 225 to 227 may be arranged at an angle of 30 ° or 45 ° in the lateral direction with respect to the auxiliary microphone 224 located at the front center.

The angular intervals between the auxiliary microphones 221 to 227 can be set to be narrow to increase the beam forming accuracy and to be wide to decrease the precision.

9 is a view showing a microphone array mounted on a glass. Referring to FIG. 9, the main microphone 210 may be attached to the front center of the glass, and the auxiliary microphones 221 to 224 may be attached to the main microphone 210 at predetermined intervals along the left and right sides.

In addition, the combination of the main microphone 210 and the auxiliary microphones 221 to 224 can be attached so as to be arranged in a straight line.

10 is a diagram showing a microphone array arranged in a mobile device. 10 (a) is a perspective view showing a mobile device to which a microphone array is attached, (b) shows a front surface of the mobile device, and FIG. 10 (c) shows a rear surface of the mobile device.

10, the microphone array 110 may be arranged over a front surface 341 with a display portion 343 of the mobile device 340 and a rear surface 342 opposite thereto.

According to the embodiment, the main microphone 210 is arranged at the center of the rear surface 342 of the mobile device 340, and a plurality of auxiliary microphones 221 to 228 are arranged around the main microphone 210. The auxiliary microphones 222, 224, 225, and 227 may be attached to points symmetrical with respect to the main microphone 210 in the up, down, left, and right directions. And auxiliary microphones 221, 223, 226, and 228 may be attached to the points symmetrical to each other in the diagonal direction. A plurality of auxiliary microphones 229 to 232 are arranged in the outer edge area of the display unit 343 on the front surface 341.

11 is a view illustrating a microphone array arranged in a mobile device according to another embodiment of the present invention. 11 shows the rear surface of the mobile device.

11, the main microphone 210 is attached to the center of the rear surface 342 of the mobile device 340 and the auxiliary microphones 221 to 224 are attached to the side surface 343 area of the mobile device 340 have. The auxiliary microphones 221 to 224 may be attached to points symmetrical with respect to the main microphone 210 in the up, down, left, and right directions.

In the case of the microphone arrays 221 to 224 attached to the mobile device 340 described above, stereophonic sound can be supported in real time during a video call through the mobile device 340.

12 is a view showing a microphone array arrangement according to another embodiment of the present invention.

Referring to FIG. 12, a panoramic camera 350 is recently available that can simultaneously photograph 360 ° images after being worn on a user's head. In the case of such a wearable electronic device 350, the microphone arrays 221 to 228 can be arranged in a uniform circular array form. The microphones 221 to 228 may constitute an array at the same angular interval in positions symmetrical to each other with respect to the center of the wearable electronic device 350. [

In this embodiment, eight microphones 221 to 228 are shown as being arrayed at intervals of 45 [deg.], But the present invention is not limited thereto and may be variously changed. For example, if the four microphones constitute the array, the interval between the microphones may be 90 [deg.].

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the scope of the present invention is not limited to the disclosed exemplary embodiments. It will also be appreciated that many modifications and variations will be apparent to those skilled in the art without departing from the scope of the invention.

10: Stereophonic Capturing System
110: microphone array
120: Multiple input signal input interface module
130: acoustic signal filtering module
131: Noise reduction filter
132: Acoustic Echo Removal Filter
133: Acoustic characteristic extraction filter
140: directional extraction module
150: sound field parameter extraction module
160: Acoustic signal integration module
210: Main microphone
220: Secondary microphone

Claims (10)

A main microphone for receiving an acoustic signal;
At least two auxiliary microphones arranged at positions different from each other around the main microphone and receiving the acoustic signals;
A direction extracting module for extracting direction information of the acoustic signal from the acoustic signal input to the auxiliary microphones; And
And a sound signal integration module for integrating the sound signal with the directional information of the sound extracted from the directional extraction module. The method according to claim 1,
Wherein the auxiliary microphones are arranged coplanar with the main microphone. The method according to claim 1,
The auxiliary microphones,
At least two auxiliary microphones arranged on the first plane with the main microphone,
And at least two auxiliary microphones arranged on a second plane different from the first plane. The method according to claim 1,
Further comprising an acoustic signal filtering module for filtering the signals included in the acoustic signal,
The acoustic signal filtering module
A noise canceling filter for extracting a surrounding noise signal included in the sound signal and removing the peripheral noise signal through reverse phase filtering;
An acoustic echo cancellation filter for eliminating an echo signal included in the acoustic signal; And
And a speech extraction filter for extracting the speech signal contained in the acoustic signal. The method according to claim 1,
The directional extraction module calculates an input time difference of the acoustic signal input to each of the auxiliary microphones, calculates an incident angle of the acoustic signal to the auxiliary microphones according to the input time difference, A stereo audio capturing system. The method according to claim 1,
Wherein the main microphone and the auxiliary microphones are attached to an electronic device. Inputting a sound signal through a main microphone and at least two auxiliary microphones arranged around the main microphone;
Extracting directional information of the acoustic signal from the acoustic signal input to the auxiliary microphones; And
And integrating the direction information with the acoustic information input to the main microphone. 8. The method of claim 7,
Wherein the auxiliary microphones are positioned coplanar with the main microphone to input the acoustic signal. 8. The method of claim 7,
Further comprising: an acoustic signal filtering step of filtering signals included in the acoustic signal input to the main microphones and the auxiliary microphones,
The acoustic signal filtering step
Removing an ambient noise signal included in the acoustic signal through anti-phase filtering;
Removing an echo signal included in the acoustic signal;
And extracting a voice signal included in the acoustic signal. 8. The method of claim 7,
The step of extracting the directional information of the acoustic signal
A stereo sound capturing unit for calculating an input time difference of the acoustic signal input to each of the auxiliary microphones and calculating an incident angle of the acoustic signal to the auxiliary microphones according to the input time difference, Way.

Images