KR102261905B1 - Apparatus, Method or Computer Program for Generating a Sound Field Description - Google Patents

Abstract

An apparatus for generating a sound field description having a representation of a sound field component includes: a direction determiner (102) for determining one or more sound directions for each time-frequency tile of a plurality of time-frequency tiles of a plurality of microphone signals; a space-based function evaluator (103) for evaluating, for each time-frequency tile of the plurality of time-frequency tiles, one or more spatial-based functions using one or more sound directions; and, for each time-frequency tile of the plurality of time-frequency tiles, one or more sound field components corresponding to one or more spatial-based functions evaluated using one or more sound directions and a reference signal for the corresponding time-frequency tiles. a calculating sound component calculator 201, the reference signal being derived from one or more microphone signals of a plurality of microphone signals.

Description

Apparatus, Method or Computer Program for Generating a Sound Field Description

The present invention relates to an apparatus, method, or computer program for generating a Sound Field Description and also for the synthesis of (higher order) Ambisonics signals in the time-frequency domain using sound direction information. will be.

The present invention relates to the field of spatial sound recording and reproduction. Spatial sound recording aims to capture a sound field using multiple microphones so that, on the playback side, the listener perceives the sound image as if it were at the recording location. Standard approaches for spatial sound recording generally use spaced omni-directional microphones (eg, in AB stereophonic sound) or coherent directional microphones (eg, in intensity stereophonic sound). The recorded signal can be played back in a standard stereo loudspeaker setup to obtain a stereo sound image. For surround sound playback using, for example, a 5.1 loudspeaker setup, a similar recording technique can be used (eg 5 cardioid microphones [ArrayDesign] directed towards the loudspeaker position). Recently, 3D sound reproduction systems have emerged, such as a 7.1+4 loudspeaker setup that uses 4 height speakers to reproduce elevated sound. Signals for these loudspeaker setups can be recorded with very specific spaced 3D microphone setups [MicSetup3D]. All these recording techniques are designed for a specific loudspeaker setup and therefore have a common application, limiting their practical applicability, for example, if the recorded sound needs to be reproduced in different loudspeaker configurations.

Rather than directly recording the signal for a specific loudspeaker setup, more flexibility is gained when recording signals in an intermediate format that can generate arbitrary loudspeaker setup signals on the playback side. In practice, such a well-established intermediate format is represented by (higher-order) ambisonics [Ambisonics]. From the ambisonics signal, any desired loudspeaker setup signal can be generated, including binaural signals for headphone reproduction. This requires a specific renderer that applies to the Ambisonics signal, such as the classic Ambisonics renderer [Ambisonics], Direcalal Audio Coding (DirAC) [DirAC], or HARPEX [HARPEX].

An Ambisonics signal represents a multi-channel signal, where each channel (called an Ambisonics component) is equal to a coefficient of a so-called space-based function. A weighted sum of these spatially based functions (with weights corresponding to coefficients) can be used to reproduce the original sound field at the recording location [FourierAcoust]. Thus, the spatially based function coefficients (ie, the ambisonics component) represent a concise description of the sound field at the recording location. There are various types of spatially based functions, such as the spherical harmonic (SH) [FourierAcoust] or the cylindrical harmonic (CH) [FourierAcoust]. CH can be used to describe the sound field in 2D space (eg 2D sound reproduction) while SH can be used to describe the sound field in 2D and 3D space (eg 2D and 3D sound reproduction).

모드가 존재하는데, 여기서 m 및 l은 범위가

및

인 정수이다. 대응하는 공간 기반 함수의 예가 도 1a에 도시되며, 이는 상이한 차수 l 및 모드 m에 대한 구면 고조파 함수를 나타낸다. 차수 l은 때로는 레벨이라고 불리며, 모드 m은 차수라고도 지칭될 수 있음에 유의한다. 도 1a에서 알 수 있는 바와 같이, 0차(제0 레벨) l=0의 구면 고조파는 레코딩 위치에서의 전 방향 음압을 나타내고, 반면 1차(제1 레벨) l=1의 구면 고조파는 데카르트 좌표계의 3차원을 따른 다이폴 성분을 나타낸다. 이는 특정 차수(레벨)의 공간 기반 함수는 차수 l의 마이크폰의 지향성을 기술함을 의미한다. 다시 말해, 공간 기반 함수의 계수는 차수(레벨) l 및 모드 m의 마이크로폰의 신호에 대응한다. 서로 다른 차수와 모드의 공간 기반 함수는 서로 직교함에 유의한다. 이는 예를 들어 순수한 확산 음장에서 모든 공간 기반 함수의 계수는 서로 상관 관계가 없음을 의미한다.Spatial-based functions exist for different orders l, and mode m exists in the case of 3D spatial-based functions (eg SH). In the latter case, for each order l

Modes exist, where m and l are the ranges

and

is an integer. An example of a corresponding spatially based function is shown in FIG. 1A , which represents a spherical harmonic function for different orders l and mode m. Note that order l is sometimes referred to as a level, and mode m may also be referred to as an order. As can be seen from Fig. 1a, the spherical harmonic of the 0th (0th level) l=0 represents the omnidirectional sound pressure at the recording location, whereas the spherical harmonic of the 1st (1st level) l=1 is in the Cartesian coordinate system represents the dipole component along the three dimensions of . This means that a spatial-based function of a certain order (level) describes the directivity of a microphone of order l. In other words, the coefficients of the spatially based function correspond to the signal of the microphone of order (level) l and mode m. Note that spatial-based functions of different orders and modes are orthogonal to each other. This means that, for example, in a purely diffuse sound field, the coefficients of all spatially based functions are uncorrelated with each other.

인 앰비소닉스 신호는 고차 앰비소닉스(higher-order 앰비소닉스, HOA)라고 지칭된다. 더 높은 차수 l을 사용하여 음장을 기술하는 경우, 공간 해상도는 더 높아진다, 즉 보다 정확하게 음장을 기술하거나 재현할 수 있다. 따라서 정확도가 낮고(데이터가 적음) 더 적은 수의 차수만으로 음장을 기술하거나 더 높은 정확도(그리고 더 많은 데이터)로 이어지는 더 높은 차수를 사용할 수 있다.As described above, each Ambisonics component of an Ambisonics signal corresponds to a spatial-based function coefficient of a particular level (and mode). For example, if we describe a sound field as level l=1 using SH as a spatial-based function, (since it has one mode for order l=0 and three modes for l=1), the ambisonics signal has four It will contain an Ambisonics component. Ambisonics signals of highest order 1 are referred to as first-order ambisonics (FOA) in the following, whereas maximum order ambisonics (FOA) is

In Ambisonics signals are referred to as higher-order ambisonics (HOA). When a higher order l is used to describe the sound field, the spatial resolution is higher, that is, the sound field can be described or reproduced more accurately. Thus, it is possible to describe the sound field with less accuracy (less data) and with fewer orders, or use higher orders that lead to higher accuracy (and more data).

Different but closely related mathematical definitions exist for different spatially based functions. For example, you can calculate complex-valued square harmonics as well as real-valued square harmonics. Also, the spherical harmonics can be calculated using other normalization terms such as SN3D, N3D, or N2D normalization. Other definitions can be found in [Ambix]. Some specific examples will be shown later in conjunction with the description and examples of the invention.

The desired ambisonics signal can be determined from recordings with multiple microphones. A simple way to obtain an ambisonics signal is to directly compute the ambisonics components (space-based function coefficients) from the microphone signal. This approach requires measuring the sound pressure at a very specific location, for example a circle or a sphere. After that, the spatial fundamental function coefficients are [FourierAcoust, p. 218], it can be calculated by integrating over the measured sound pressure as described for example. This direct approach requires a specific microphone setup, for example a circular array or a spherical array of omni-directional microphones. Two typical examples of commercially available microphone setups are the SoundField ST350 microphone or EigenMike® [EigenMike]. Unfortunately, the geometry requirements of a particular microphone greatly limit practical applicability, for example when the microphone needs to be integrated into a small device or a microphone array needs to be combined with a video camera. In addition, determining high-dimensional spatial coefficients using this direct approach requires a relatively large number of microphones to ensure sufficient robustness to noise. Therefore, a direct approach to acquiring an ambisonics signal is often very expensive.

It is an object of the present invention to provide an improved concept for creating a sound field description with representations of sound field components.

This object is achieved by an apparatus according to claim 1 , a method according to claim 23 or a computer program according to claim 24 .

The present invention relates to an apparatus or method or computer program for generating a sound field description having representations of sound field components. In the direction determiner, one or more sound directions for each time-frequency tile of the plurality of time-frequency tiles of the plurality of microphone signals are determined. The spatial-based function evaluator evaluates, for each time-frequency tile of the plurality of time-frequency tiles, one or more spatially-based functions using one or more sound directions. Further, the sound field component calculator corresponds to, for each time-frequency tile of the plurality of time-frequency tiles, one or more spatially based functions evaluated using one or more sound directions and a reference signal for the corresponding time-frequency tiles. calculates one or more sound field components, wherein the reference signal is derived from one or more microphone signals of the plurality of microphone signals.

The present invention is based on the discovery that a sound field description describing any complex sound field can be derived in an efficient manner from a plurality of microphone signals within a time-frequency representation made up of time-frequency tiles. These time-frequency tiles are used, on the one hand, to refer to a plurality of microphone signals and, on the other hand, to determine the sound direction. Thus, sound direction determination takes place within the spectral domain using time-frequency tiles of a time-frequency representation. Then, the main part of the subsequent processing is preferably performed within the same time-frequency representation. To this end, evaluation of the spatially based function is performed using one or more sound directions determined for each time-frequency tile. Spatial-based functions depend on sound direction but independent of frequency. Thus, the evaluation of the spatial based function by means of a frequency domain signal, ie a signal in a time-frequency tile, is applied. Within the same time-frequency representation, one or more sound field components corresponding to the one or more spatial-based functions evaluated using the one or more sound directions are computed together with the reference signal also present in the same time-frequency representation.

These one or more sound field components for each block of the signal and each frequency bin, ie for each time-frequency tile, may be the final result, or alternatively, one or more time domain sound fields corresponding to one or more spatially based functions. A transformation may be performed back to the time domain to obtain the component. Depending on the implementation, the one or more sound field components may be direct sound field components determined within a time-frequency representation using time-frequency tiles or may be diffuse sound field components determined generally in addition to the direct sound field components. The final sound field component having the direct part and the diffuse part may be obtained by combining the direct sound field component and the diffuse sound field component, where this combination may be performed in the time domain or the frequency domain according to actual implementation.

Several procedures may be performed to derive a reference signal from one or more microphone signals. Such a procedure may include direct selection of a particular microphone signal from a plurality of microphone signals or an advanced selection based on one or more sound directions. The advanced reference signal determination selects a particular microphone signal from among the microphones from which the microphone signal is derived, a plurality of microphone signals from a microphone located closest to the sound direction. Another alternative is to apply a multi-channel filter to two or more microphone signals to jointly filter these microphone signals so that a common reference signal for all frequency tiles of the time block is obtained. Alternatively, different reference signals for different frequency tiles within a time block may be derived. Naturally, different reference signals for different time blocks as well as different reference signals for the same frequency in different time blocks can also be generated. Thus, depending on the implementation, the reference signal for the time-frequency tile may be freely selected or derived from a plurality of microphone signals.

In this context, it should be emphasized that the microphone may be disposed at any position. Microphones may have different directional characteristics. Also, the plurality of microphone signals are not necessarily signals recorded by actual actual microphones. Instead, the microphone signal can be an artificially generated microphone signal in a specific sound field using specific data processing operations that mimic a real microphone.

For determining diffuse sound field components in certain embodiments, different procedures are possible and useful in certain embodiments. Typically, the diffuse part is derived as a reference signal from a plurality of microphone signals, and this order (diffusion) reference signal is of a particular order (or level and/or mode) to obtain a diffuse sound component for this order or level or mode. is processed along with the average response of the spatially based function. Therefore, a direct sound component is calculated by evaluating a specific spatial-based function in a specific direction of arrival, and the diffuse sound component is of course not calculated using a specific direction of arrival, but using a spreading reference signal, and a spreading reference signal and a specific order or level or It is calculated by combining the average response of the spatial-based functions of the modes into a specific function. This functional combination may be, for example, a multiplication that may be performed in the calculation of the direct sound component, or this combination may be, for example, a weighted multiplication or addition or subtraction if the calculation in the logarithmic domain is performed. Other combinations different from multiplication or addition/subtraction are performed using additional non-linear or linear functions, where non-linear functions are preferred. Following generation of the direct sound field component and the diffuse sound field component of any order, combining may be performed by combining the direct sound field component and the diffuse sound field component in the spectral domain for each individual time/frequency tile. Alternatively, it may also be performed that the diffuse sound field component and the direct sound field component for a specific order are transformed from the frequency domain to the time domain and then are transformed into a time domain combination of the direct time domain component and the diffuse time domain component of the specific order. .

Additional decorrelators may be used to decorrelating diffuse sound field components depending on the situation. Alternatively, the decorrelated diffuse sound field component can be obtained by using different time/frequency bins for different microphone signals or different diffuse field components of different orders, or by using different microphone signals for the calculation of the direct sound field components and using the diffuse sound field components. can be generated by using another microphone signal for the calculation of

In a preferred embodiment, the spatially based function is a spatially based function associated with a specific level (order) and mode of the well-known Ambisonics sound field technique. A sound field component of a specific order and a specific mode corresponds to an ambisonics sound field component associated with a specific level and specific mode. Typically, the first sound field component is a sound field component associated with an omnidirectional spatial-based function as shown in FIG. 1A when the order is l=0 and the mode is m=0.

The second sound field component may for example relate to a spatially based function having a maximum directivity in the x direction corresponding to the order l = 1 and the mode m = -1 with respect to FIG. 1A . The third sound field component may be, for example, a spatially based function directional in the y direction corresponding to the mode m = 0 and the order l = 1 in FIG. 1A , and the fourth sound field component may be, for example, the mode m = 1 and It may be a space-based function directional in the z-direction corresponding to order l=1.

However, of course, other sound field techniques different from Ambisonics are of course well known to the person skilled in the art, and those other sound field components that depend on different spatially based functions from Ambisonics spatial based functions are also advantageous within the time-frequency domain representation as described above. can be calculated

The following inventive embodiment describes a practical method for acquiring an Ambisonics signal. Unlike the state-of-the-art approach described above, this approach can be applied to any microphone setup with two or more microphones. In addition, high-order ambisonics components can be computed using only a relatively small number of microphones. Therefore, the method is relatively inexpensive and practical. In the proposed embodiment, the ambisonics component is not calculated directly from the sound pressure information along a specific surface as in the state-of-the-art approach described above, but is synthesized based on a parametric approach. For this purpose, a rather simple sound field model similar to that used for example in DirAC [DirAC] is assumed. More precisely, the sound field at the recording location consists of one or several direct sounds arriving from a specific sound direction and diffuse sounds arriving from all directions. Based on this model and using parametric information about the sound field, such as the sound direction of direct sound, it is possible to synthesize ambisonics components or other sound field components with only a few measurements of sound pressure. This approach is described in detail in the next section.

도 1a는 상이한 차수 및 모드에 대한 구면 고주파 함수를 도시한다;

도 1b는 도착 방향 정보에 기초하여 기준 마이크로폰을 선택하는 방법의 일 예를 도시한다;

도 1c는 음장 기술을 생성하기 위한 장치 또는 방법의 바람직한 구현 예를 도시한다;

도 1d는 예시적인 마이크로폰 신호의 시간-주파수 변환을 도시하며, 여기서 한편으로는 주파수 빈 (10) 및 시간 블록 (1)에 대한 특정 시간-주파수 타일 (10, 1) 및 주파수 빈 (5) 및 시간 블록 (2)에 대한 (5,2)이 구체적으로 식별된다;

도 1e는 식별된 주파수 빈 (10, 1) 및 (5, 2)에 대한 사운드 방향을 사용하는 예시적인 4개의 공간 기반 함수의 평가를 도시한다;

도 1f는 2개의 빈 (10, 1) 및 (5, 2) 및 후속하는 주파수-시간 변환 및 크로스-페이드/중첩-가산 처리에 대한 음장 성분의 계산을 도시한다;

도 1g는 도 1f의 처리에 의해 획득되는 예시적인 4개의 음장 성분(b1 내지 b4)의 시간 도메인 표현을 도시한다;

도 2a는 본 발명의 일반적인 블록 기법을 도시한다;

도 2b는 역 시간-주파수 변환이 결합기 전에 적용되는 본 발명의 일반적인 블록 기법을 도시한다;

도 3a는 원하는 레벨 및 모드의 앰비소닉스 성분이 기준 마이크로폰 신호 및 사운드 방향 정보로부터 계산되는 본 발명의 실시예를 도시한다;

도 3b는 기준 마이크로폰이 도착 방향 정보에 기초하여 선택되는 본 발명의 실시예를 도시한다;

도 4는 다이렉트 사운드 앰비소닉스 성분 및 확산 사운드 앰비소닉스 성분이 계산되는 본 발명의 실시예를 도시한다;

도 5는 확산 사운드 앰비소닉스 성분이 상관 해제되는 본 발명의 실시예를 도시한다;

도 6은 다이렉트 사운드 및 확산 사운드가 다수의 마이크로폰 및 사운드 방향 정보로부터 추출되는 본 발명의 실시예를 도시한다;

도 7은 확산 사운드이 다수의 마이크로폰으로부터 추출되고 확산 사운드 앰비소닉스 성분이 상관 해제되는 본 발명의 실시예를 도시한다; 그리고

도 8은 이득 평활화가 공간 기반 함수 응답에 적용되는 본 발명의 실시예를 도시한다.A preferred embodiment of the present invention will be described below with reference to the accompanying drawings, wherein:

1A shows spherical high-frequency functions for different orders and modes;

1B shows an example of a method for selecting a reference microphone based on arrival direction information;

1C shows a preferred embodiment of an apparatus or method for generating a sound field description;

1d shows a time-frequency transformation of an exemplary microphone signal, wherein specific time-frequency tiles (10, 1) and frequency bins (5) and frequency bins (10) and frequency bins (5) for frequency bin 10 and time block 1 on the one hand (5,2) for time block (2) is specifically identified;

1E shows the evaluation of four example spatially based functions using sound directions for the identified frequency bins (10, 1) and (5, 2);

Fig. 1f shows the calculation of sound field components for two bins (10, 1) and (5, 2) and subsequent frequency-time transformation and cross-fade/superposition-add processing;

Fig. 1G shows a time domain representation of four exemplary sound field components (b1 to b4) obtained by the processing of Fig. 1F;

Figure 2a shows a general block scheme of the present invention;

Figure 2b shows a general block scheme of the present invention in which an inverse time-frequency transform is applied before the combiner;

Figure 3a shows an embodiment of the present invention in which the ambisonics component of a desired level and mode is calculated from a reference microphone signal and sound direction information;

Figure 3b shows an embodiment of the present invention in which a reference microphone is selected based on arrival direction information;

Figure 4 shows an embodiment of the present invention in which a direct sound ambisonics component and a diffuse sound ambisonics component are calculated;

Figure 5 shows an embodiment of the present invention in which diffuse sound ambisonics components are decorrelated;

Fig. 6 shows an embodiment of the present invention in which direct sound and diffuse sound are extracted from multiple microphones and sound direction information;

7 shows an embodiment of the present invention in which diffuse sound is extracted from multiple microphones and diffuse sound ambisonics components are decorrelated; And

8 shows an embodiment of the present invention in which gain smoothing is applied to a spatially based functional response.

A preferred embodiment is shown in Fig. 1c. 1C illustrates an embodiment of an apparatus or method for generating a sound field description 130 having a time domain representation of a sound field component or a frequency domain representation of a sound field component, an encoded or decoded representation, or a representation of a sound field component, such as an intermediate representation. shows

To this end, the direction determiner 102 determines one or more sound directions 131 for each time-frequency tile of the plurality of time-frequency tiles of the plurality of microphone signals.

Thus, the direction determiner receives at least two different microphone signals at input 132 , and for each of these two different microphone signals, a time-frequency representation is available, typically consisting of subsequent blocks of spectral bins, where the spectrum A block of bins is associated with a specific time index n, where the frequency index is k. The block of frequency bins for the time index represents the spectrum of the time domain signal for the block of time domain samples generated by the feature windowing operation.

The sound direction 131 is used by the spatial based function evaluator 103 for evaluating one or more spatial based functions for each time-frequency tile of the plurality of time-frequency tiles. Accordingly, the result of the processing at block 103 is one or more evaluated spatially based functions for each time-frequency tile. Preferably, two or more different spatially based functions are used, such as four spatially based functions as discussed in relation to FIGS. 1E and 1F . Accordingly, at output 133 of block 103 , evaluated spatially based functions of different orders and modes for different time-frequency tiles of the time-spectral representation are available and input to sound field component calculator 201 . The sound field component calculator 201 additionally uses a reference signal 134 generated by a reference signal calculator (not shown in Fig. 1C). The reference signal 134 is derived from one or more microphone signals of a plurality of microphone signals and is used by the sound field component calculator within the same time/frequency representation.

Accordingly, the sound field component calculator 210 calculates, for each time-frequency tile of the plurality of time-frequency tiles, the evaluated using one or more sound directions, with the aid of one or more reference signals for the corresponding time-frequency tiles. and calculate one or more sound field components corresponding to the one or more spatially based functions.

Depending on the implementation, the spatially based function evaluator 103 uses a parameterized representation for the spatially based function, wherein the parameter of the parameterized representation is a sound direction, the sound direction being one-dimensional or three-dimensional in a two-dimensional situation. In the context of two-dimensionality - it is configured to insert parameters corresponding to sound directions into the parameterized representation to obtain evaluation results for each spatially based function.

Alternatively, the spatially based function evaluator is configured to use a lookup table for each spatially based function having a spatial based function identification and sound direction at an input and an evaluation result as an output. In this situation, the spatial based function evaluator is configured to determine a corresponding sound direction of the lookup table input for one or more sound directions determined by the direction determiner 102 . Typically, the different directional inputs are quantized in such a way that there are a certain number of table inputs, for example 10 different sound directions.

The spatial based function evaluator 103 is configured to determine a corresponding lookup table input for a particular sound direction that does not immediately match the sound direction input into the lookup table. This may be done, for example, by using the next highest sound direction or the next lowest sound direction entered into the lookup table for a particular determined sound direction. Alternatively, the table is used in such a way that a weighted average between two neighboring lookup table inputs is computed. Thus, the procedure would be that the table output for the next lower directional input is determined. Also, the lookup table output for the next highest input is determined, and then the average between these values is calculated.

This average may be a simple average obtained by adding the two outputs and dividing the result by two, or it may be a weighted average according to the location of the sound direction determined for the next highest table output and the next lowest table output. . Thus, illustratively, the weighting factor will depend on the difference between the determined sound direction and the corresponding next higher/next lower input to the lookup table. For example, if the measured direction approaches the next lower input, the lookup table result for the next lower input is multiplied by the higher weight factor compared to the weighting factor, and the lookup table output for the next higher input is weighted Thus, for small differences between the determined direction and the next lower input, the output of the lookup table for the next lower input is used to weight the output of the lookup table corresponding to the next higher lookup table input for the direction of sound. will be weighted with a higher weighting factor compared to the weighting factor used.

1D-1G are then discussed to illustrate in more detail examples for specific calculations of different blocks.

The top view of FIG. 1d shows a schematic microphone signal. However, the actual amplitude of the microphone signal is not shown. Instead, windows, particularly windows 151 and 152 are shown. The window 151 defines the first block 1 , and the window 152 identifies and determines the second block 2 . Accordingly, the microphone signal is preferably treated as an overlapping block with an overlap of 50%. However, higher or lower overlap may also be used, and no overlap may be possible at all. However, overlap processing is performed to avoid blocking artifacts.

Each block of sampling values of the microphone signal is converted into a spectral representation. The spectral representation or spectrum for the block with time index n = 1, ie block 151, is shown in the middle representation of FIG. 1d , and the spectral representation of the second block 2 is denoted by the reference numerals shown in the lower figure of FIG. 1d . Corresponds to 152. Also, for illustrative reasons, each spectrum is shown as having 10 frequency bins, i.e., the frequency index k is between 1 and 10.

Thus, the time-frequency tile (k, n) is the time-frequency tile (10, 1) at 153, and another example shows another time-frequency tile (5,2) at 154. The further processing performed by the apparatus for generating the sound field description is illustrated in FIG. 1d , exemplarily illustrated using these time-frequency tiles, denoted by reference numerals 153 and 154 .

Further, it is assumed that the direction determiner 102 determines the sound direction or "DOA" (direction of arrival) exemplarily indicated by the unit reference vector n. Alternative direction indications include azimuth, elevation, or both angles together. To this end, the direction determiner 102 uses all microphone signals of a plurality of microphone signals, where each microphone signal is represented by a subsequent block of frequency bins as shown in FIG. 1d , and the direction determiner of FIG. 1c ( 102) determines the sound direction or DOA, for example. Thus, illustratively, the time-frequency tile (10, 1) has a sound direction n(10, 1), and the time-frequency tile (5, 2) has a sound direction n( 5, 2). In the three-dimensional case, the sound direction is a three-dimensional vector with x, y, or z components. Of course, other coordinate systems can also be used, such as spherical coordinates, which depend on two angles and a radius. Alternatively, the angle may be, for example, azimuth and elevation. Then the radius is not needed. Similarly, Cartesian coordinates, i.e. in the case of two dimensions, such as x and y directions, there are two components of the sound direction, but alternatively circular coordinates with radius and angle or azimuth and elevation angle can also be used.

This procedure is performed not only for time-frequency tiles (10, 1) and (5, 2), but for all time-frequency tiles in which the microphone signal is represented.

Then, one or more spatially based functions required are determined. In particular, it is determined what number of sound field components or, in general, representations of the sound field components are to be generated. The number of spatially based functions currently used by the spatial based function evaluator 103 of FIG. 1C finally determines the number of sound field components for each time-frequency tile in the spectral representation or the number of sound field components in the time domain. do.

For another embodiment, it is assumed that four sound field components are determined, wherein, illustratively, these four sound field components are an omnidirectional sound component (corresponding to the same order as 0) and 3 directional in the corresponding coordinate directions of the Cartesian coordinate system. It may be a directional sound field component of dogs.

The lower figure of FIG. 1E shows the evaluated spatial based function G _i for different time-frequency tiles. Thus, in this example, it becomes clear that for each time-frequency tile four evaluated spatially based functions are determined. For example, assuming that each block has 10 frequency bins, 40 evaluated spatially based functions G for each block, such as block n = 1 and block n = 2, as shown in FIG. 1E . _i is determined. So, to sum up, if only 2 blocks are considered and each block has 10 bins, then there are 20 time-frequency tiles in 2 blocks and each time-frequency tile has 4 evaluated spatial based functions. , the procedure yields 80 evaluated spatial-based functions.

Fig. 1f shows a preferred embodiment of the sound field component calculator 201 of Fig. 1c. FIG. 1f shows in the top two figures a block of two frequency bins for the determined reference signal input to block 201 of FIG. 1c via line 134 . In particular, a reference signal, which may be a specific microphone signal or a combination of different microphone signals, is processed in the same manner as discussed in relation to FIG. 1D . Thus, illustratively, the reference signal is represented by a reference spectrum for block n=1 and a reference signal spectrum for block n=2. Accordingly, the reference signal is decomposed into the same time-frequency pattern used in the calculation of the evaluated spatial based function for the time-frequency tile output via line 133 from block 103 to block 201 .

Then, as indicated at 155 , the actual calculation of the sound field component is performed via a functional combination between the corresponding time-frequency tile for the reference signal P and the spatially based function G evaluated in relation to it. Preferably, the functional combination expressed by f(...) is the multiplication shown by 115 in FIGS. 3A and 3B, which will be described later. However, other functional combinations may be used as discussed above. By functional combination at block 155, one or more sound field components Bi are frequency domain (spectral) representations of sound field components Bi, as shown at 156 for block n = 1 and 157 for block n = 2 is calculated for each time-frequency tile to obtain

Thus, illustratively, _{the frequency domain representation of the sound field component B i} is, on the one hand, for the time-frequency tile (10, 1) and on the other hand for the time-frequency tile (5, 2) for the second block is shown However, at 156 and 157 it becomes clear once again that the number of sound field components Bi shown in Fig. 1F is equal to the number of evaluated spatially based functions shown in the lower part of Fig. 1E.

If only sound field components in the frequency domain are required, the calculation is completed with the outputs of blocks 156 and 157 . However, the time-domain representation of the sound field component is required to obtain in another embodiment, the first sound component time domain representation of the B _1, the second field components, such as additional time domain expression for B _2.

To this end, the sound field component B1 from the frequency bin 1 to the frequency bin 10 of the first block 156 is a frequency-time transmission block 159 to obtain the first block and a time domain representation for the first component. ) is inserted into

Similarly, to determine and calculate the first component in the time domain, ie, b _{1 (t), the spectral sound field component B 1} for the second block going from frequency bin 1 to frequency bin 10 is a different frequency -converted to a time domain representation by a time transform 160 .

Because an overlap window is used as shown in the upper part of FIG. 1D , the cross-fade or overlap-add operation 161 shown at the bottom of FIG. 1F is the block 1 and block 2 shown at 162 in FIG. 1G. ) can be used to compute the output time domain samples of the first spectral representation b _{1 (d) within the overlapping range between .}

The same procedure is performed to calculate _{the second time domain sound field component b 2} (t) within the overlap range 163 between the first block and the second block. In addition, for calculating the third sound field component b ₃ (t) in the time domain, and in particular for calculating the samples within the overlap range 164 , the component D ₃ from the first block and the component D 3 from the second block Component D ₃ is correspondingly converted to a time domain representation by procedures 159 , 160 , and the resulting value is cross-fade/overlapping-added at block 161 .

Finally, to obtain a final sample of the fourth time domain representation sound field component b4(t) in the overlapping range 165 as shown in Fig. 1g, the fourth component B4 and the second block for the first block The same procedure is performed for B4.

In order to obtain a time-frequency tile, when processing is not performed with overlapping blocks and processing is performed with non-overlapping blocks, any cross-fade/overlapping-addition as shown in block 161 is not required. Note that it is not

Also, in the case of higher overlap, where two or more blocks overlap each other, a correspondingly larger number of blocks 159, 160 is required and the cross-fade/overlapping-addition of block 161 is shown in Fig. 1g. It is computed with 3 inputs as well as 2 inputs to finally obtain a sample of the time domain representation.

Also note that, for example, _{samples for the time domain representation for the overlapping range OL 23} are obtained by applying the procedure of blocks 159 and 160 to the second and third blocks. Correspondingly, samples for the overlap range OL _0,1 are calculated by performing procedures 159, 160 _{on the corresponding spectral sound field components B i} for the specific number i for block 0 and block 1 .

Also, as already summarized, the representation of the sound field component may be a frequency domain representation as shown in FIG. 1f for 156 and 157 . Alternatively, the representation of the sound field components may be a time domain representation as shown in FIG. 1G , wherein the four sound field components represent a direct sound signal having a sample sequence associated with a particular sampling rate. In addition, either a frequency domain representation or a time domain representation of the sound field component may be encoded. This encoding can be performed separately so that each sound field component is encoded as a mono signal or the encoding can be performed jointly, for example, four sound field components B ₁ to B ₄ are regarded as a multi-channel signal having 4 channels. do. Thus, a frequency domain encoded representation or a time domain representation that is encoded with any useful encoding algorithm is also a representation of sound field components.

Also, even the representation in the time domain prior to the cross-fade/overlapping-add performed by block 161 may be a useful representation of the sound field component for a particular implementation. Also, one kind of vector quantization may be performed on block n for a particular component, such as component 1, to compress the frequency domain representation of the sound field component for transmission or storage or other processing operation.

preferred embodiment

Figure 2a shows the current novel approach given by block 10 which allows synthesizing the ambisonics component of the desired order (level) and mode from the signals of multiple (two or more) microphones. Unlike some of the more recent approaches, there are no restrictions on microphone settings. This means that multiple microphones can be arranged in any shape, for example in a coincident setup, a linear array, a planar array, or a three-dimensional error. Also, each microphone may be omni-directional or any directional. The directivity of different microphones may be different.

To obtain the desired ambisonics component, the multiple microphone signals are first transformed into a time-frequency representation using block 101 . For this purpose, it is possible to use, for example, a filter bank or a short-time Fourier transform (STFT). The output of block 101 is a number of microphone signals in the time-frequency domain. Note that the following processing is performed separately for time-frequency tiles.

After transforming the multiple microphone signals in the time-frequency domain, one or more sound directions (relative to the time-frequency tile) are determined in block 102 from the two or more microphone signals. Sound direction describes the direction in which the salient sound for the time-frequency tile arrives at the microphone array. This direction is commonly referred to as the direction-of-arrival (DOA) of the sound. Instead of DOA, one can consider the direction of propagation of sound, which is the opposite of DOA, or another measure that describes the direction of sound. One or multiple sound directions or DOAs are estimated in block 102 using, for example, a state-of-the-art narrowband DOA estimator and can be used for almost any microphone setup. A suitable exemplary DOA estimator is listed in Example 1. The number of sound directions or one or more DOAs computed in block 102 depends on, for example, acceptable computational complexity as well as the performance of the DOA estimator or microphone geometry used as well. The sound direction can be estimated, for example, in 2D space (eg, expressed in the form of azimuth) or in 3D space (eg, expressed in the form of azimuth and elevation). In the following, most of the explanations are based on the general 3D case, although it is simple to apply all processing steps even to the 2D case. In many cases, the user specifies the number of estimated sound directions or DOAs (eg, 1, 2, or 3) per time-frequency tile. Alternatively, the number of salient sounds can be estimated using a state-of-the-art method, for example the method described in [SourceNum].

The one or more sound directions estimated for the time-frequency tile in block 102 are used in block 103 to compute one or more responses of the spatially based function of the desired order (level) and mode for the time-frequency tile. One response is computed for each estimated sound direction. As described in the previous section, spatially based functions can represent, for example, spherical harmonics (eg, if processing is performed in 3D space) or cylindrical harmonics (eg, if processing is performed in 2D space). . The response of the spatially based function is a spatially based function evaluated in the corresponding estimated sound direction, as described in more detail in the first embodiment.

The one or more sound directions estimated for the time-frequency tile are further used in block 201 , ie to compute one or more ambisonics components of the desired order (level) and mode over the time-frequency tile. These ambisonics components synthesize ambisonics components for directional sounds arriving from the estimated sound direction. Additional inputs to block 201 are one or more microphone signals for a given time-frequency tile, as well as one or more responses of a spatially based function computed for the time-frequency tile of block 103 . At block 201, one ambisonics component of the desired order (level) and mode is computed for each estimated sound direction and corresponding response of the spatial based function. The processing steps of block 201 are further discussed in the examples below.

The invention 10 includes a selection block 301 capable of computing the diffuse sound ambisonics component of a desired order (level) and mode over a time-frequency tile. This component synthesizes an ambisonics component for, for example, a purely diffused sound field or ambient sound. Inputs to block 301 are one or more sound directions and one or more microphone signals estimated in block 102 . The processing steps of block 301 are further discussed in later embodiments.

The diffuse sound ambisonics component computed in optional block 301 may be further decorrelated in optional block 107 . For this purpose, a state-of-the-art decorrelator may be used. Some examples are listed in Example 4. Typically we will apply different decorrelators or different realizations of decorrelators for different orders (levels) and modes. By doing this, the decorrelated diffuse sound ambisonics components of different orders (levels) and modes will be uncorrelated with each other. This mimics the expected physical behavior, i.e., ambisonics components of various orders (levels) and modes are not correlated with diffuse or ambient sound, as explained for example in [SpCoherence].

The desired order (level) and mode computed for the time-frequency tile of block 201 and one or more (direct sound) ambisonics components of the corresponding diffuse sound ambisonics component computed at block 301 are block 401 . is combined in As discussed in the examples below, the combination may be realized as, for example, a (weighted) sum. The output of block 401 is the final synthesized ambisonics component of the desired order (level) and mode for a given time-frequency tile. Obviously, if a single (direct sound) ambisonics component of the desired order (level) and mode is computed in block 201 for a time-frequency tile (and no diffuse sound ambisonics component), then the combiner 401 is unnecessary.

After computing the final ambisonics component of the desired order (level) and mode for all time-frequency tiles, the ambisonics component is transformed into an inverse time-frequency transform (20) which can be realized, for example, as an inverse filter bank or an inverse STFT can be converted back to the time domain. Note that inverse time-frequency transformation is not required in all applications, so this is not part of the present invention. In practice, we will compute the ambisonics component for every desired order and mode to obtain the desired ambisonics signal of the maximum desired order (level).

Figure 2b shows a slightly modified implementation of the same invention. In this figure, an inverse time-frequency transform 20 is applied before the combiner 401 . This is possible because the inverse time-frequency transform is usually a linear transform. By applying the inverse time-frequency transform before the combiner 401, it is possible to perform decorrelation in the time domain, for example (instead of the time-frequency domain as in Fig. 2a). This can have substantial advantages for some applications when implementing the present invention.

It should be noted that the inverse filter bank may be somewhere else. In general, combiners and decorrelators (usually the latter) should be applied in the time domain. However, in the frequency domain, only two blocks or one block can be applied.

Accordingly, the preferred embodiment comprises a spread component calculator 301 for calculating, for each time-frequency tile of the plurality of time-frequency tiles, one or more diffuse sound components. Also, this embodiment includes a combiner 401 for combining the diffuse sound information and the direct sound field information to obtain a frequency domain representation or a time domain representation of a sound field component. Also, according to an implementation, the diffuse component calculator further comprises a decorrelator 107 for decorrelating the diffuse sound information, wherein the decorrelator is a frequency It can be implemented within a domain. Alternatively, the decorrelator is configured to operate in the time domain as shown in FIG. 2b , such that a correlation in the time domain of the time-representation of a particular diffuse sound component of a particular order is performed.

Another embodiment of the present invention includes a time-frequency converter, such as time-frequency converter 101, for converting each of a plurality of time domain microphone signals into a frequency representation having a plurality of time-frequency tiles. Another embodiment is a frequency-time such as block 20 of FIG. 2A or 2B for converting one or more sound field components or a combination of one or more sound field components, i.e., direct sound field components and sound field components of diffuse sound components, into time domain representations. includes a converter.

In particular, the frequency-time converter 20 is configured to process one or more sound field components to obtain a plurality of time domain sound field components in which these time domain sound field components are direct sound field components. Further, the frequency-time converter 20 is configured to process the diffuse sound (field) component to obtain a plurality of time domain diffuse (sound field) components, and the combiner is configured to obtain a plurality of time domain diffuse (sound field) components, for example, in the time domain as shown in Fig. 2b. It is configured to perform a combination of a domain (direct) sound field component and a time domain diffusion (sound field component). Alternatively, the combiner 401 is configured to combine one or more (direct) sound field components for a time-frequency tile and a diffuse sound (field) component for a corresponding time-frequency tile in the frequency domain, the frequency-to-time converter 20 is then configured to process the result of the combiner 401 to obtain a sound field component in the time domain, ie a representation of the sound field component in the time domain, for example as shown in FIG. 2A .

The following examples illustrate some implementations of the present invention in more detail. Examples 1-7 consider one sound direction per time-frequency tile (thus considering only one response of level, mode, and spatial based function and one die P sound ambisonics component per time and frequency) take note of Embodiment 8 describes an example in which more than one sound direction is considered per time-frequency tile. The concept of this embodiment is directly applicable to all other embodiments.

Example 1

Figure 3a shows an embodiment of the present invention which allows to synthesize ambisonics components of desired order (level) l and mode m from signals of multiple (two or more) microphones.

The input to the present invention is the signal of multiple (two or more) microphones. The microphones may be arranged in any shape, such as, for example, a coincident setup, a linear array, a planar array, or a three-dimensional error. Also, each microphone may be omni-directional or any directional. The directivity of different microphones may be different.

으로 표시되는 시간-주파수 도메인에서의 다수의 마이크로폰 신호이며, 여기서 k는 주파수 인덱스이고, n은 시간 인덱스이고, M은 마이크로폰의 수이다. 이하의 처리가 시간-주파수 타일 (k, n)에 대해 개별적으로 수행됨에 유의한다.The multiple microphone signals are transformed into the time-frequency domain in block 101 using, for example, a filter bank or a short-time Fourier transform (STFT). The output of the time-frequency transform 101 is

A plurality of microphone signals in the time-frequency domain denoted by , where k is the frequency index, n is the time index, and M is the number of microphones. Note that the following processing is performed separately for time-frequency tiles (k, n).

를 사용하여 시간 및 주파수마다 블록(102)에서 사운드 방향 추정이 수행된다. 이 실시예에서, 단일 사운드 방향은 시간 및 주파수마다 결정된다. (102)에서의 사운드 방향 추정에 있어서, 다양한 마이크로폰 어레이 구조에 대한 문헌에서 이용 가능한 최신 협대역 도착 방향(DOA) 추정기가 사용될 수 있다. 예를 들어, 임의의 마이크로폰 설정에 적용할 수 있는 MUSIC 알고리즘[MUSIC]이 사용될 수 있다. 균일한 선형 어레이, 등거리 격자점을 갖는 비균일 선형 어레이, 또는 무지향성 마이크로폰의 원형 어레이의 경우, MUSIC보다 계산상 효율적인 루트 MUSIC 알고리즘[RootMUSIC1, RootMUSIC2, RootMUSIC3]이 적용될 수 있다. 선형 불변 서브 어레이 구조를 갖는 선형 어레이 또는 평면 어레이에 적용될 수 있는 또 다른 잘 알려진 협대역 DOA 추정기는 ESPRIT[ESPRIT]이다.After converting the microphone signal into the time-frequency domain, two or more microphone signals

Sound direction estimation is performed at block 102 per time and frequency using In this embodiment, a single sound direction is determined per time and frequency. For the sound direction estimation in (102), the latest narrowband direction of arrival (DOA) estimator available in the literature for various microphone array structures can be used. For example, the MUSIC algorithm [MUSIC] applicable to any microphone setting may be used. In the case of a uniform linear array, a non-uniform linear array with equidistant grid points, or a circular array of omnidirectional microphones, the root MUSIC algorithm [RootMUSIC1, RootMUSIC2, RootMUSIC3], which is computationally more efficient than MUSIC, can be applied. Another well-known narrowband DOA estimator that can be applied to a linear array or a planar array with a linear invariant sub-array structure is ESPRIT [ESPRIT].

및/또는 앙각

의 관점에서 표현될 수 있으며, 이는 예를 들어In this embodiment, the output of the sound direction estimator 102 is the sound direction for time instance n and frequency index k. The sound direction can be, for example, a unit norm vector n(k, n) or an azimuth

and/or elevation

can be expressed in terms of

is related to

이 추정되지 않으면(2D 경우), 다음 단계에서 0 고도, 즉

으로 가정할 수 있다. 이 경우, 단위 놈 벡터 n(k, n)은elevation

is not estimated (in the 2D case), then in the next step 0 altitude, i.e.

can be assumed as In this case, the unit norm vector n(k, n) is

can be written as

로 표시되고After estimating the sound direction in block 102 , the estimated sound direction information is used to determine the response of a spatially based function of desired order (level) l and mode m in block 103 individually for each time and frequency. The response of a spatially based function of order (level) l and mode m is

is displayed as

is calculated as

는 벡터 n(k, n) 또는 방위각

및/또는 앙각

에 의해 지시되는 방향 및/또는 방위각에 의존하는 차수(레벨) l 및 모드 m의 공간 기반 함수이다. 따라서, 응답

은 벡터 n(k, n) 또는 방위각

및/또는 앙각

에 의해 지시된 방향으로부터 도착하는 사운드에 대한 공간 기반 함수

의 응답을 기술한다. 예를 들어, 공간 기반 함수로서 N3D 정규화를 갖는 실수 구형 고조파를 고려하는 경우,

는 [SphHarm,Ambix,FourierAcoust]에서와 같이 계산될 수 있으며,here,

is the vector n(k, n) or azimuth

and/or elevation

It is a spatially based function of order (level) l and mode m dependent on the direction and/or azimuth indicated by . Therefore, the response

is vector n(k, n) or azimuth

and/or elevation

A space-based function for a sound arriving from the direction indicated by

describe the response of For example, if we consider real square harmonics with N3D regularization as a spatial based function,

can be calculated as in [SphHarm,Ambix,FourierAcoust],

here

은 예를 들어 [FourierAcoust]에서 정의된 앙각에 따른 차수(레벨) l 및 모드 m의 연관된 르장드르(Legendre) 다항식이다. 원하는 차수(레벨) l 및 모드 m의 공간 기반 함수

의 응답은 각각의 방위각 및/또는 앙각에 대해 미리 계산되어 룩업 테이블에 저장되고 그 다음에 추정된 사운드 방향에 따라 선택될 수 있음에 유의한다.is the N3D normalization constant,

is, for example, an associated Legendre polynomial of order (level) l and mode m according to the elevation angle defined in [FourierAcoust]. Spatial based function of desired order (level) l and mode m

Note that for each azimuth and/or elevation, the response of may be pre-computed and stored in a lookup table and then selected according to the estimated sound direction.

로 지칭된다, 즉In this embodiment, without loss of generality, the first microphone signal is the reference microphone signal.

is referred to as

to be.

는 블록(103)에서 결정된 공간 기반 함수의 응답

을 갖는 시간 주파수 타일 (k,n)에 곱해져(115) 결합되며, 즉In this embodiment, the reference microphone signal

is the response of the spatially based function determined in block 103 .

It is multiplied (115) by the time frequency tile (k,n) with

을 초래한다. 최종 앰비소닉스 성분

은 결국 역 필터 뱅크 또는 역 STFT를 사용하여 시간 도메인으로 다시 변환되거나, 예를 들어 공간 사운드 재생 응용을 위해 저장되고, 송신되거나, 또는 사용될 수 있다. 실제로, 원하는 최대 차수(레벨)의 원하는 앰비소닉스 신호를 획득하기 위해 원하는 모든 차수 및 모드에 대해 앰비소닉스 성분을 컴퓨팅할 것이다., which is the desired ambisonics component of order (level) l and mode m for the time-frequency tile (k, n)

causes Final Ambisonics Component

may in turn be transformed back to the time domain using an inverse filter bank or inverse STFT, or stored, transmitted, or used, for example, for spatial sound reproduction applications. In practice, we will compute the ambisonics component for every desired order and mode to obtain the desired ambisonics signal of the maximum desired order (level).

Example 2

Figure 3b shows an embodiment of the present invention which allows to synthesize ambisonics components of desired order (level) l and mode m from signals of multiple (two or more) microphones. The embodiment is similar to embodiment 1 but additionally includes a block 104 for determining a reference microphone signal from the plurality of microphone signals.

As in embodiment 1, the input to the present invention is the signal of multiple (two or more) microphones. The microphones may be arranged in any shape, such as, for example, a coincident setup, a linear array, a planar array, or a three-dimensional error. Also, each microphone may be omni-directional or any directional. The directivity of different microphones may be different.

으로 표시되는 시간-주파수 도메인의 마이크로폰 신호이다. 이하의 처리는 시간-주파수 타일 (k, n)에 대해 개별적으로 수행된다.As in embodiment 1, multiple microphone signals are transformed into the time-frequency domain in block 101 using, for example, a filter bank or short-time Fourier transform (STFT). The output of the time-frequency transform 101 is

It is a microphone signal in the time-frequency domain denoted by . The following processing is performed separately for time-frequency tiles (k, n).

를 사용하여 시간 및 주파수마다 블록(102)에서 사운드 방향 추정이 수행된다. 대응하는 추정은 실시예 1에서 논의되었다. 사운드 방향 추정기(102)의 출력은 시간 인스턴스 n 및 주파수 인덱스 k 당 사운드 방향이다. 사운드 방향은 예를 들어 단위 놈벡터 n(k, n) 또는 방위각

및/또는 앙각

의 관점에서 표현될 수 있으며, 이는 실시예 1에서 설명한 바와 같다.As in embodiment 1, two or more microphone signals

Sound direction estimation is performed at block 102 per time and frequency using Corresponding estimates are discussed in Example 1. The output of the sound direction estimator 102 is the sound direction per time instance n and frequency index k. The sound direction can be, for example, a unit norm vector n(k, n) or an azimuth

and/or elevation

It can be expressed in terms of , which is the same as described in Example 1.

로 표시된다. 예를 들어, 공간 기반 함수로서 N3D 정규화를 갖는 실수 값의 구면 고조파를 고려할 수 있고,

은 실시예 1에서 설명한 바와 같이 결정될 수 있다.As in Example 1, the response of the spatially based function of the desired order (level) l and mode m is determined in block 103 for each time and frequency using the estimated sound direction information. The response of a space-based function is

is displayed as For example, we can consider real-valued spherical harmonics with N3D regularization as a space-based function,

may be determined as described in Example 1.

로부터 기준 마이크로폰 신호

가 결정된다. 이 목적을 위해, 블록(104)은 블록(102)에서 추정된 사운드 방향 정보를 사용한다. 상이한 기준 마이크로폰 신호가 상이한 시간-주파수 타일에 대해 결정될 수 있다. 사운드 방향 정보에 기초하여 다수의 마이크로폰 신호

로부터 기준 마이크로폰 신호

를 결정하는 다른 가능성이 존재한다. 예를 들어, 추정된 사운드 방향에 가장 가까운 다수의 마이크로폰으로부터 마이크로폰을 시간 및 주파수별로 선택할 수 있다. 이 접근법은 도 1b에서 볼 수 있다. 예를 들어, 마이크로폰 포지션이 포지션 벡터

에 의해 주어진다고 가정하면, 가장 가까운 마이크로폰의 인덱스 i(k, n)는 문제In this embodiment, at block 104 multiple microphone signals

Reference microphone signal from

is decided For this purpose, block 104 uses the sound direction information estimated in block 102 . Different reference microphone signals may be determined for different time-frequency tiles. Multiple microphone signals based on sound direction information

Reference microphone signal from

There are other possibilities for determining For example, a microphone may be selected by time and frequency from a plurality of microphones closest to the estimated sound direction. This approach can be seen in Figure 1b. For example, the microphone position is the position vector

Assuming given by , the index i(k, n) of the nearest microphone is

can be found by solving

The reference microphone signal for the considered time and frequency is

is given as

를 결정하기 위한 대안적인 접근법은 마이크로폰 신호에 멀티 채널 필터를 적용하는 것이며, 즉In the example of Figure 1b, as d ₃ is closed for n(k, n), the reference microphone for the time-frequency tile (k, n) will be microphone number 3, i.e. i(k, n) = 3 will be. Reference microphone signal

An alternative approach to determine , is to apply a multi-channel filter to the microphone signal, i.e.

는 다수의 마이크로폰 신호를 포함한다. [OptArrayPr]에서 예를 들어 파생된 지연 및 합 필터 또는 LCMV 필터와 같은

을 계산하는 데 사용할 수 있는 많은 다른 최적의 멀티 채널 필터 w(n)가 있다. 다중 채널 필터를 사용하면, [OptArrayPr]에서 설명한 여러 장단점을 얻을 수 있는데, 예를 들어 마이크로폰 자체 노이즈를 감소시킬 수 있다., where w(n) is a multi-channel filter dependent on the estimated sound direction, and

contains multiple microphone signals. From [OptArrayPr], e.g. delay and sum filters or LCMV filters

There are many other optimal multi-channel filters w(n) that can be used to compute Using a multi-channel filter, several advantages and disadvantages described in [OptArrayPr] can be achieved, for example, reducing the noise of the microphone itself.

는 블록(103)에서 결정된 공간 기반 함수의 응답

과 시간 주파수 타일 (k,n)을 곱하여(115) 결합되며, 이는 시간-주파수 타일 (k, n)에 대한 차수(레벨) l 및 모드 m의 원하는 앰비소닉스 성분

을 초래한다. 결과적인 앰비소닉스 성분

은 결국 역 필터 뱅크 또는 역 STFT를 사용하여 시간 도메인으로 다시 변환되거나, 저장되거나, 송신되거나 또는 예를 들어 공간 사운드 재생을 위해 사용된다. 실제로, 원하는 최대 차수(레벨)의 원하는 앰비소닉스 신호를 획득하기 위해 원하는 모든 차수 및 모드에 대해 앰비소닉스 성분을 컴퓨팅할 것이다.As in embodiment 1, the reference microphone signal

is the response of the spatially based function determined in block 103 .

is combined by multiplying (115) the time frequency tile (k, n), which is the desired ambisonics component of the order (level) l and mode m for the time-frequency tile (k, n).

causes The resulting ambisonics component

is in turn transformed back to the time domain using an inverse filter bank or inverse STFT, stored, transmitted or used, for example, for spatial sound reproduction. In practice, we will compute the ambisonics component for every desired order and mode to obtain the desired ambisonics signal of the maximum desired order (level).

Example 3

Figure 4 shows another embodiment of the present invention which allows synthesizing ambisonics components of desired order (level) l and mode m from the signals of multiple (two or more) microphones. The embodiment is similar to embodiment 1, but calculates ambi sonic components for a direct sound signal and a diffuse sound signal.

As in embodiment 1, the input to the present invention is the signal of multiple (two or more) microphones. The microphones may be arranged in any shape, such as, for example, a coincident setup, a linear array, a planar array, or a three-dimensional error. Also, each microphone may be omni-directional or any directional. The directivity of different microphones may be different.

으로 표시되는 시간-주파수 도메인의 마이크로폰 신호이다. 이하의 처리는 시간-주파수 타일 (k, n)에 대해 개별적으로 수행된다.As in embodiment 1, multiple microphone signals are transformed into the time-frequency domain in block 101 using, for example, a filter bank or short-time Fourier transform (STFT). The output of the time-frequency transform 101 is

It is a microphone signal in the time-frequency domain denoted by . The following processing is performed separately for time-frequency tiles (k, n).

를 사용하여 시간 및 주파수마다 블록(102)에서 사운드 방향 추정이 수행된다. 대응하는 추정은 실시예 1에서 논의되었다. 사운드 방향 추정기(102)의 출력은 시간 인스턴스 n 및 주파수 인덱스 k 당 사운드 방향이다. 사운드 방향은 예를 들어 단위 놈벡터 n(k, n) 또는 방위각

및/또는 앙각

의 관점에서 표현될 수 있으며, 이는 실시예 1에서 설명한 바와 같다.As in embodiment 1, two or more microphone signals

Sound direction estimation is performed at block 102 per time and frequency using Corresponding estimates are discussed in Example 1. The output of the sound direction estimator 102 is the sound direction per time instance n and frequency index k. The sound direction can be, for example, a unit norm vector n(k, n) or an azimuth

and/or elevation

It can be expressed in terms of , which is the same as described in Example 1.

로 표시된다. 예를 들어, 공간 기반 함수로서 N3D 정규화를 갖는 실수 값의 구면 고조파를 고려할 수 있고, 실시예 1에서 설명한 바와 같이

이 결정될 수 있다.As in Example 1, the response of the spatially based function of the desired order (level) l and mode m is determined in block 103 for each time and frequency using the estimated sound direction information. The response of a space-based function is

is displayed as For example, we can consider real-valued spherical harmonics with N3D normalization as a spatial-based function, as described in Example 1

This can be determined.

로 표시되며 가능한 모든 방향(예를 들어, 확산 사운드 또는 주변 사운드)에서 도착하는 사운드에 대한 공간 기반 함수의 응답을 나타낸다. 평균 응답

을 정의하는 한 가지 예는 가능한 모든 각도

및/또는

에 대한 공간 기반 함수

의 제곱 크기의 적분을 고려하는 것이다. 예를 들어, 구의 모든 각도에 대해 통합하는 경우, In this embodiment, the average response of the spatially based function of the desired order (level) l and mode m independent of the temporal index n is obtained from block 106 . This average response is

denotes the response of a spatially-based function to a sound arriving from all possible directions (e.g., diffuse or ambient sound). average response

One example that defines all possible angles is

and/or

A space-based function for

to consider the integral of the squared magnitude of For example, if you integrate for all angles of a sphere,

get

의 이러한 정의는 다음과 같이 해석될 수 있다: 제1 실시예에서 설명한 바와 같이, 공간 기반 함수

는 차수 l의 마이크로폰의 지향성으로 해석될 수 있다. 증가하는 차수에 대해, 그러한 마이크로폰은 점점 더 지향적이 될 것이고, 따라서 무지향성 마이크로폰(차수 l = 0의 마이크로폰)에 비해 실용적인 음장에서 덜 확산된 사운드 에너지 또는 주변 사운드 에너지가 캡쳐될 것이다. 위에서 주어진

의 정의에 따라, 평균 응답

는 무지향성 마이크로폰과 비교하여 차수 l의 마이크로폰 신호에서 확산 사운드 에너지 또는 주변 사운드 에너지가 얼마나 감쇠되는지를 설명하는 실수 값 인자가 된다. 명백하게, 구의 방향에 대해 공간 기반 함수

의 제곱 크기를 통합하는 것 외에도 평균 응답

를 정의하는 다양한 대안이 존재한다, 예를 들어: 원의 방향에 대한

의 제곱 크기를 적분, 원하는 방향

의 세트에 대해

의 제곱 크기를 적분, 원하는 방향

의 임의의 세트에 대해

의 제곱 크기를 평균화, 제곱된 크기 대신

의 크기를 적분하거나 평균화, 임의의 방향

의 세트에 대한

의 가중 합을 고려, 또는 확산 사운드 또는 주변 사운드에 대한 차수 1의 예상되는 마이크로폰의 원하는 감도에 대응하는

에 대한 임의의 실수 값을 지정.average response

This definition of can be interpreted as follows: As described in the first embodiment, a spatially based function

can be interpreted as the directivity of a microphone of order l. For increasing orders, such microphones will become increasingly directional and thus less diffuse or ambient sound energy will be captured in practical sound fields compared to omni-directional microphones (microphones of order l=0). given above

As per the definition of the mean response

is a real-valued factor that describes how much the diffuse sound energy or ambient sound energy is attenuated in a microphone signal of order l compared to an omni-directional microphone. Obviously, a space-based function for the direction of the sphere

In addition to incorporating the squared magnitude of the mean response

Various alternatives exist for defining

Integrate the squared magnitude of , in the desired direction

about the set of

Integrate the squared magnitude of , in the desired direction

for any set of

average the squared magnitude of , instead of the squared magnitude

Integrate or average the magnitude of , in any direction

for a set of

Consider a weighted sum of, or corresponding to the desired sensitivity of the microphone of order 1 to diffuse or ambient sound.

Specifies a random real value for .

The average spatial based function response may be precomputed and stored in a lookup table and determination of the response value is performed by accessing the lookup table and retrieving the corresponding value.

이다.As in embodiment 1, without loss of generality, the first microphone signal is referred to as a reference microphone signal, i.e.

to be.

으로 표시되는 다이렉트 사운드 신호 및

로 표시되는 확산 사운드 신호를 계산하기 위해 기준 마이크로폰 신호

가 사용된다. 블록(105)에서, 다이렉트 사운드 신호

는 예를 들어 단일 채널 필터

을 기준 마이크로폰 신호에 적용함으로써 계산될 수 있다, 즉In this embodiment, at block 105

Direct sound signal indicated by and

A reference microphone signal to calculate a diffuse sound signal denoted by

is used At block 105, the direct sound signal

is for example a single channel filter

can be calculated by applying to the reference microphone signal, i.e.

to be.

을 계산하는 문헌에는 여러 가지 가능성이 있다. 예를 들어 [Victaulic]에서Optimal single channel filter

There are several possibilities in the literature for calculating . For example in [Victaulic]

A well-known square root Wiener filter defined by

중 임의의 2개의 마이크로폰을 사용하여 추정될 수 있다. 블록(105)에서, 다이렉트 사운드 신호

는 예를 들어 단일 채널 필터

을 기준 마이크로폰 신호에 적용함으로써 계산될 수 있다, 즉where SDR(k, n) is the signal-to-diffuse ratio (SDR) at time instance n and the frequency index k representing the power ratio between direct and diffuse sounds discussed in [VirtualMic]. SDR is a state-of-the-art SDR estimator available in the literature, for example, an estimator proposed in [SDRestim] based on spatial coherence between two arbitrary microphone signals.

can be estimated using any two microphones. At block 105, the direct sound signal

is for example a single channel filter

can be calculated by applying to the reference microphone signal, i.e.

to be.

을 계산하는 문헌에는 여러 가지 가능성이 있다. 예를 들어 [VirtualMic]에서Optimal single channel filter

There are several possibilities in the literature for calculating . For example in [VirtualMic]

A well-known square-root Wiener filter defined by , where SDR (k, n) is the SDR that can be estimated as previously discussed.

는 블록(103)에서 결정된 공간 기반 함수의 응답

을 시간 및 주파수마다 곱하여(115a) 결합된다, 즉In this embodiment, the direct sound signal determined at block 105 .

is the response of the spatially based function determined in block 103 .

is multiplied by time and frequency (115a) and combined, i.e.

을 초래한다. 또한, 블록(105)에서 결정된 확산 사운드 신호

는 블록(106)에서 결정된 공간 기반 함수의 평균 응답

와 시간 및 주파수 당 곱해져(115b) 결합된다, 즉, which is the direct sound ambisonics component of order (level) l and mode m for the time-frequency tile (k, n)

causes Also, the diffuse sound signal determined in block 105 .

is the mean response of the spatially based function determined in block 106 .

is multiplied per time and frequency (115b) and combined, i.e.

와 모드 m을 초래한다., which is the order sound level ambisonics component for the time-frequency tile (k, n)

and mode m.

및 확산 사운드 앰비소닉스 성분

은 예를 들어 합산 연산(109)을 통해 결합되어, 시간-주파수 타일 (k, n)에 대한 원하는 차수(레벨) l 및 모드 m의 최종 앰비소닉스 성분

을 획득한다, 즉Finally, the direct sound ambisonics component

and diffuse sound ambisonics components

are combined, e.g. via a summation operation 109, so that the final ambisonics component of the desired order (level) l and mode m for the time-frequency tile (k, n)

to obtain, i.e.

to be.

은 결국 역 필터 뱅크 또는 역 STFT를 사용하여 시간 도메인으로 다시 변환되거나, 저장되거나, 송신되거나 또는 예를 들어 공간 사운드 재생을 위해 사용된다. 실제로, 원하는 최대 차수(레벨)의 원하는 앰비소닉스 신호를 획득하기 위해 원하는 모든 차수 및 모드에 대해 앰비소닉스 성분을 컴퓨팅할 것이다.The resulting ambisonics component

is in turn transformed back to the time domain using an inverse filter bank or inverse STFT, stored, transmitted or used, for example, for spatial sound reproduction. In practice, we will compute the ambisonics component for every desired order and mode to obtain the desired ambisonics signal of the maximum desired order (level).

을 계산하기 전에, 즉 연산(109) 전에 수행될 수 있음을 강조하는 것이 중요하다. 이는 먼저 시간 도메인으로

및

)을 다시 변환할 수 있고, 그 다음에 성분 양자 모두를 연산(109)으로 합산하여 최종 앰비소닉스 성분

을 획득할 수 있음을 의미한다. 이것은 역 필터 뱅크 또는 역 STFT가 일반적으로 선형 연산이기 때문에 가능하다.For example, a transform to the time domain using an inverse filter bank or inverse STFT would be

It is important to emphasize that it can be performed before calculating , ie before operation 109 . This is first in the time domain

and

) can be transformed back, and then both components are summed in operation 109 to produce the final ambisonics component.

means that it can be obtained. This is possible because the inverse filter bank or inverse STFT is usually a linear operation.

및 확산 사운드 앰비소닉스 성분

이 서로 다른 모드(차수) l에 대해 컴퓨팅되도록 구성될 수 있음에 유의한다. 예를 들어,

은 차수 l = 4까지 컴퓨팅될 수 있고, 한편

는 단지 차수 l=1까지만 컴퓨팅될 수 있다 (이 경우에, 큰 차수 l = 1의 경우

은 0이 될 것이다). 이것은 실시예 4에서 설명한 바와 같은 구체적인 이점을 갖는다. 특정 차수 (레벨) l 또는 모드 m에 대하여

만을 계산하고

은 계산하지 않기를 원한다면, 예를 들어 블록(105)은 확산 사운드 신호

가 0이 되도록 구성될 수 있다. 이것은 예를 들어 이전의 방정식에서 필터

를 0으로 설정하고 필터

을 1로 설정함으로써 달성될 수 있다. 대안적으로, 이전 방정식의 SDR을 수동으로 매우 높은 값으로 설정할 수 있다.The algorithm of this embodiment is a direct sound ambisonics component

and diffuse sound ambisonics components

Note that this can be configured to compute for different modes (orders) l. For example,

can be computed up to order l = 4, while

can be computed only up to order l=1 (in this case, for large orders l = 1)

will be 0). This has specific advantages as described in Example 4. For a specific order (level) l or mode m

count only

If we do not wish to calculate, for example, block 105

may be configured to be 0. This is for example a filter in the previous equation

set to 0 and filter

can be achieved by setting Alternatively, the SDR of the previous equation can be manually set to a very high value.

Example 4

Figure 5 shows another embodiment of the present invention which allows synthesizing ambisonics components of desired order (level) l and mode m from signals of multiple (two or more) microphones. The embodiment is similar to embodiment 3 but additionally includes a decorrelator for the diffuse ambisonics component.

As in embodiment 3, the input to the present invention is the signal of multiple (two or more) microphones. The microphones may be arranged in any shape, such as, for example, a coincident setup, a linear array, a planar array, or a three-dimensional error. Also, each microphone may be omni-directional or any directional. The directivity of different microphones may be different.

으로 표시되는 시간-주파수 도메인의 마이크로폰 신호이다. 이하의 처리는 시간-주파수 타일 (k, n)에 대해 개별적으로 수행된다.As in embodiment 3, multiple microphone signals are transformed into the time-frequency domain in block 101 using, for example, a filter bank or a short-time Fourier transform (STFT). The output of the time-frequency transform 101 is

It is a microphone signal in the time-frequency domain denoted by . The following processing is performed separately for time-frequency tiles (k, n).

를 사용하여 시간 및 주파수마다 블록(102)에서 사운드 방향 추정이 수행된다. 대응하는 추정은 실시예 1에서 논의되었다. 사운드 방향 추정기(102)의 출력은 시간 인스턴스 n 및 주파수 인덱스 k 당 사운드 방향이다. 사운드 방향은 예를 들어 단위 놈벡터 n(k, n) 또는 방위각

및/또는 앙각

의 관점에서 표현될 수 있으며, 이는 실시예 1에서 설명한 바와 같다.As in embodiment 3, two or more microphone signals

Sound direction estimation is performed at block 102 per time and frequency using Corresponding estimates are discussed in Example 1. The output of the sound direction estimator 102 is the sound direction per time instance n and frequency index k. The sound direction can be, for example, a unit norm vector n(k, n) or an azimuth

and/or elevation

It can be expressed in terms of , which is the same as described in Example 1.

로 표시된다. 예를 들어, 공간 기반 함수로서 N3D 정규화를 갖는 실수 값의 구면 고조파를 고려할 수 있고, 실시예 1에서 설명한 바와 같이

이 결정될 수 있다.As in embodiment 3, the response of the spatially based function of the desired order (level) l and mode m is determined in block 103 for each time and frequency using the estimated sound direction information. The response of a space-based function is

is displayed as For example, we can consider real-valued spherical harmonics with N3D normalization as a spatial-based function, as described in Example 1

This can be determined.

로 표시되며 가능한 모든 방향(예를 들어, 확산 사운드 또는 주변 사운드)에서 도착하는 사운드에 대한 공간 기반 함수의 응답을 나타낸다. 평균 응답

는 실시예 3에 기술된 바와 같이 획득될 수 있다.As in embodiment 3, an average response of the spatially based function of the desired order (level) l and mode m independent of the temporal index n is obtained from block 106 . This average response is

denotes the response of a spatially-based function to a sound arriving from all possible directions (e.g., diffuse or ambient sound). average response

can be obtained as described in Example 3.

이다.As in embodiment 3, without loss of generality, the first microphone signal is referred to as the reference microphone signal P_ref (k, n), i.e.

to be.

으로 표시되는 다이렉트 사운드 신호 및

로 표시되는 확산 사운드 신호를 계산하기 위해 기준 마이크로폰 신호

가 사용된다.

및

의 계산은 실시예 3에서 설명된다.As in Example 3, at block 105

Direct sound signal indicated by and

A reference microphone signal to calculate a diffuse sound signal denoted by

is used

and

Calculation of is described in Example 3.

는 블록(103)에서 결정된 공간 기반 함수의 응답

을 시간 및 주파수 타일마다 곱하여(115a) 결합되며, 이는 시간-주파수 타일 (k, n)에 대한 차수(레벨) l 및 모드 m의 다이렉트 사운드 앰비소닉스 성분

을 초래한다. 또한, 블록(105)에서 결정된 확산 사운드 신호

는 블록(106)에서 결정된 공간 기반 함수의 평균 응답

을 시간 및 주파수 타일마다 곱하여(115b) 결합되며, 이는 시간-주파수 타일 (k, n)에 대한 차수(레벨) l 및 모드 m의 확산 사운드 앰비소닉스 성분

을 초래한다.As in embodiment 3, the direct sound signal determined in block 105

is the response of the spatially based function determined in block 103 .

are combined by multiplying (115a) per time and frequency tile, which is the direct sound ambisonics component of order (level) l and mode m for the time-frequency tile (k, n).

causes Also, the diffuse sound signal determined in block 105 .

is the mean response of the spatially based function determined in block 106 .

is combined by multiplying (115b) per time and frequency tile, which is the diffuse sound ambisonics component of order (level) l and mode m for the time-frequency tile (k, n).

causes

은 상관 해제기를 사용하여 블록(107)에서 상관 해제되며, 이는

으로 표시되는 상관 해제된 확산 사운드 앰비소닉스 성분을 초래한다. 상관 해제를 위해 최신 상관 해지 기술이 사용될 수 있다. 상이한 상관 해제기 또는 상관 해제기의 실현은 일반적으로 상이한 차수 (레벨) 및 모드 m의 확산 사운드 앰비소닉스 성분

에 적용되어, 서로 다른 레벨 및 모드의 결과적인 상관 해제된 확산 사운드 앰비소닉스 성분

은 상호 관련이 없다. 이렇게 함으로써, 확산 사운드 앰비소닉스 성분

은 예상된 물리적 거동을 가진다, 즉 음장가 주변 또는 확산이면 서로 다른 차수와 모드의 앰비소닉스 성분은 상호 관련이 없다 [SpCoherence]. 상관 해제기(107)를 적용하기 전에 예를 들어 역 필터 뱅크 또는 역 STFT를 사용하여 확산 사운드 앰비소닉스 성분

을 시간 도메인으로 다시 변환될 수 있음에 유의한다.In this embodiment, the calculated diffuse sound ambisonics component

is decorrelated at block 107 using the decorrelator, which

resulting in a decorrelated diffuse sound ambisonics component denoted by The latest de-correlation techniques can be used for de-correlation. The realization of different decorrelators or decorrelators is usually achieved with diffuse sound ambisonics components of different orders (levels) and modes m

applied to the resulting uncorrelated diffuse sound ambisonics components of different levels and modes

are not related to each other. By doing so, the diffuse sound ambisonics component

has the expected physical behavior, i.e., ambisonics components of different orders and modes are not correlated if the sound field is ambient or diffuse [SpCoherence]. Before applying the decorrelator 107, for example, using an inverse filter bank or an inverse STFT, diffuse sound ambisonics components

Note that can be converted back to the time domain.

및 상관 해제된 확산 사운드 앰비소닉스 성분

은 예를 들어 합산 연산(109)을 통해 결합되어, 시간-주파수 타일 (k, n)에 대한 원하는 차수(레벨) l 및 모드 m의 최종 앰비소닉스 성분

을 획득한다, 즉Finally, the direct sound ambisonics component

and uncorrelated diffuse sound ambisonics components

are combined, e.g. via a summation operation 109, so that the final ambisonics component of the desired order (level) l and mode m for the time-frequency tile (k, n)

to obtain, i.e.

to be.

은 결국 역 필터 뱅크 또는 역 STFT를 사용하여 시간 도메인으로 다시 변환되거나, 저장되거나, 송신되거나 또는 예를 들어 공간 사운드 재생을 위해 사용된다. 실제로, 원하는 최대 차수(레벨)의 원하는 앰비소닉스 신호를 획득하기 위해 원하는 모든 차수 및 모드에 대해 앰비소닉스 성분을 컴퓨팅할 것이다.The resulting ambisonics component

is in turn transformed back to the time domain using an inverse filter bank or inverse STFT, stored, transmitted or used, for example, for spatial sound reproduction. In practice, we will compute the ambisonics component for every desired order and mode to obtain the desired ambisonics signal of the maximum desired order (level).

을 계산하기 전에, 즉 연산(109) 전에 수행될 수 있음을 강조하는 것이 중요하다. 이는 먼저 시간 도메인으로

및

)을 다시 변환할 수 있고, 그 다음에 성분 양자 모두를 연산(109)으로 합산하여 최종 앰비소닉스 성분

을 획득할 수 있음을 의미한다. 이것은 역 필터 뱅크 또는 역 STFT가 일반적으로 선형 연산이기 때문에 가능하다. 동일한 방식으로, 상관 해제기(107)는

을 시간 도메인으로 다시 변환 한 후에 확산 사운드 앰비소닉스 성분

에 적용될 수 있다. 이것은 몇몇 상관 해제기가 시간 도메인 신호 상에서 동작하기 때문에 실제로 유리할 수 있다.For example, a transform to the time domain using an inverse filter bank or inverse STFT would be

It is important to emphasize that it can be performed before calculating , ie before operation 109 . This is first in the time domain

and

) can be transformed back, and then both components are summed in operation 109 to produce the final ambisonics component.

means that it can be obtained. This is possible because the inverse filter bank or inverse STFT is usually a linear operation. In the same way, the decorrelator 107

Diffuse sound ambisonics component after converting back to time domain

can be applied to This can be advantageous in practice because some decorrelators operate on time domain signals.

It should also be noted that a block such as the inverse filter bank before the decorrelator can be added to FIG. 5 and the inverse filter bank can be added anywhere in the system.

및 확산 사운드 앰비소닉스 성분

이 서로 다른 모드(차수) l에 대해 컴퓨팅되도록 구성될 수 있음에 유의한다. 예를 들어,

은 차수 l=4까지 컴퓨팅 될 수 있고, 한편

은 단지 l=1까지만 컴퓨팅될 수 있다. 이는 계산상의 복잡성을 감소시킬 것이다.As described in Example 3, the algorithm of this example uses direct sound ambisonics components.

and diffuse sound ambisonics components

Note that this can be configured to compute for different modes (orders) l. For example,

can be computed up to order l=4, while

can be computed only up to l=1. This will reduce computational complexity.

Example 5

Figure 6 shows another embodiment of the present invention which allows synthesizing ambisonics components of desired order (level) l and mode m from the signals of multiple (two or more) microphones. The embodiment is similar to the fourth embodiment, but a direct sound signal and a diffuse sound signal are determined from a plurality of microphone signals and using arrival direction information.

As in embodiment 4, the input to the present invention is the signal of multiple (two or more) microphones. The microphones may be arranged in any shape, such as, for example, a coincident setup, a linear array, a planar array, or a three-dimensional error. Also, each microphone may be omni-directional or any directional. The directivity of different microphones may be different.

으로 표시되는 시간-주파수 도메인의 마이크로폰 신호이다. 이하의 처리는 시간-주파수 타일 (k, n)에 대해 개별적으로 수행된다.As in embodiment 4, multiple microphone signals are transformed into the time-frequency domain in block 101 using, for example, a filter bank or a short-time Fourier transform (STFT). The output of the time-frequency transform 101 is

It is a microphone signal in the time-frequency domain denoted by . The following processing is performed separately for time-frequency tiles (k, n).

를 사용하여 시간 및 주파수마다 블록(102)에서 사운드 방향 추정이 수행된다. 대응하는 추정은 실시예 1에서 논의되었다. 사운드 방향 추정기(102)의 출력은 시간 인스턴스 n 및 주파수 인덱스 k 당 사운드 방향이다. 사운드 방향은 예를 들어 단위 놈벡터 n(k, n) 또는 방위각

및/또는 앙각

의 관점에서 표현될 수 있으며, 이는 실시예 1에서 설명한 바와 같다.As in embodiment 4, two or more microphone signals

Sound direction estimation is performed at block 102 per time and frequency using Corresponding estimates are discussed in Example 1. The output of the sound direction estimator 102 is the sound direction per time instance n and frequency index k. The sound direction can be, for example, a unit norm vector n(k, n) or an azimuth

and/or elevation

It can be expressed in terms of , which is the same as described in Example 1.

로 표시된다. 예를 들어, 공간 기반 함수로서 N3D 정규화를 갖는 실수 값의 구면 고조파를 고려할 수 있고, 실시예 1에서 설명한 바와 같이

이 결정될 수 있다.As in embodiment 4, the response of the spatially based function of the desired order (level) l and mode m is determined in block 103 for each time and frequency using the estimated sound direction information. The response of a space-based function is

is displayed as For example, we can consider real-valued spherical harmonics with N3D normalization as a spatial-based function, as described in Example 1

This can be determined.

로 표시되며 가능한 모든 방향(예를 들어, 확산 사운드 또는 주변 사운드)에서 도착하는 사운드에 대한 공간 기반 함수의 응답을 나타낸다. 평균 응답

는 실시예 3에 기술된 바와 같이 획득될 수 있다.As in embodiment 4, the average response of the spatially based function of the desired order (level) l and mode m independent of the temporal index n is obtained from block 106 . This average response is

denotes the response of a spatially-based function to a sound arriving from all possible directions (e.g., diffuse or ambient sound). average response

can be obtained as described in Example 3.

및 확산 사운드 신호

은 2개 이상의 이용 가능한 마이크로폰 신호

로부터 시간 인덱스 n 및 주파수 인덱스 k마다 블록(110)에서 결정된다. 이 목적을 위해, 블록(110)은 일반적으로 블록(102)에서 결정된 사운드 방향 정보를 이용한다. 이하,

및

을 결정하는 방법을 설명하는 블록(110)의 다른 예가 설명된다.In this embodiment, the direct sound signal

and diffuse sound signals

is two or more available microphone signals

is determined in block 110 for each time index n and frequency index k from For this purpose, block 110 generally uses the sound direction information determined in block 102 . Below,

and

Another example of block 110 is described that describes a method for determining

로 표시되는 기준 마이크로폰 신호는 블록(102)에 의해 제공된 사운드 방향 정보에 기초하여 다수의 마이크로폰 신호

로부터 결정된다. 기준 마이크로폰 신호

는 고려된 시간 및 주파수에 대해 추정된 사운드 방향에 가장 가까운 마이크로폰 신호를 선택함으로써 결정될 수 있다. 기준 마이크로폰 신호

를 결정하는 선택 처리는 실시예 2에서 설명되었다.

을 결정한 후, 다이렉트 사운드 신호

및 확산 사운드 신호

은 예를 들어 기준 마이크로폰 신호

에 각각 단일 채널 필터

및

를 적용함으로써 계산될 수 있다. 이 접근법 및 대응하는 단일 채널 필터의 계산은 실시예 3에서 설명되었다.In the first example of block 110,

The reference microphone signal, denoted by , is a plurality of microphone signals based on the sound direction information provided by block 102 .

is determined from Reference microphone signal

can be determined by selecting the microphone signal closest to the estimated sound direction for the considered time and frequency. Reference microphone signal

The selection process for determining ? was described in Example 2.

After determining the direct sound signal

and diffuse sound signals

is for example the reference microphone signal

on each single channel filter

and

It can be calculated by applying This approach and the calculation of the corresponding single channel filter are described in Example 3.

를 결정하고, 단일 채널 필터

을

에 적용함으로써

을 컴퓨팅한다. 그러나, 확산 신호를 결정하기 위해, 제2 기준 신호

를 선택하고 단일 채널 필터

에 제2 기준 신호

를 적용한다, 즉In a second example of block 110 , as in the previous example, the reference microphone signal

to determine the single channel filter

of

by applying to

to compute However, to determine the spread signal, the second reference signal

and select the single channel filter

2nd reference signal to

apply, i.e.

to be.

는 실시예 3에서 예를 들어 설명된 바와 같이 컴퓨팅될 수 있다. 제2 기준 신호

는 이용 가능한 마이크로폰 신호

중 하나에 대응한다. 그러나, 상이한 차수 l 및 모드 m에 대해, 제2 기준 신호로서 상이한 마이크로폰 신호를 사용할 수 있다. 예를 들어, 레벨 l = 1 및 모드 m = -1에 대해, 제1 마이크로폰 신호를 제2 기준 신호로 사용할 수 있다, 즉

이다. 레벨 l = 1 및 모드 m = 0에 대해, 제2 마이크로폰 신호를 사용할 수 있다, 즉

이다. 레벨 l = 1 및 모드 m = 1에 대해, 제3 마이크로폰 신호를 사용할 수 있다, 즉

이다. 이용 가능한 마이크로폰 신호

은 상이한 차수 및 모드에 대한 제2 기준 신호

에 예를 들어 무작위로 할당될 수 있다. 확산 또는 주변 레코딩 상황의 경우 모든 마이크로폰 신호에는 일반적으로 유사한 사운드 출력이 포함되어 있기 때문에 실제로는 합리적인 접근법이다. 상이한 차수 및 모드에 대해 상이한 제2 기준 마이크로폰 신호를 선택하는 것은 결과적인 확산 사운드 신호가 종종 상이한 차수 및 모드에 대해 (적어도 부분적으로) 상호 상관되지 않는다는 이점을 갖는다.filter

may be computed as described for example in embodiment 3. second reference signal

is the available microphone signal

corresponds to one of However, for different orders l and mode m, it is possible to use a different microphone signal as the second reference signal. For example, for level l = 1 and mode m = -1, the first microphone signal can be used as the second reference signal, i.e.

to be. For level l = 1 and mode m = 0, a second microphone signal can be used, i.e.

to be. For level l = 1 and mode m = 1, a third microphone signal can be used, i.e.

to be. available microphone signal

is the second reference signal for a different order and mode

may be randomly assigned to, for example. For diffuse or ambient recording situations, this is actually a sensible approach since all microphone signals usually contain similar sound outputs. Choosing a different second reference microphone signal for different orders and modes has the advantage that the resulting diffuse sound signal is often (at least partially) uncorrelated for different orders and modes.

는

로 표시되는 멀티 채널 필터를 다수의 마이크로폰 신호

에 적용함으로써 결정된다, 즉In a third example of block 110, a direct sound signal

is

Multi-channel filter indicated by multiple microphone signals

is determined by applying to

은 추정된 사운드 방향에 의존한다, 벡터

는 다수의 마이크로폰 신호를 포함한다. 사운드 방향 정보로부터

을 계산하는 데 사용될 수 있는 많은 문헌에서 다른 최적의 다중 채널 필터

, 예를 들어 [InformedSF]에서 도출되는 필터가 존재한다. 유사하게, 확산 사운드 신호

는

로 표시되는 멀티 채널 필터를 다수의 마이크로폰 신호

에 적용함으로써 결정된다, 즉, where the multi-channel filter

depends on the estimated sound direction, vector

contains multiple microphone signals. from sound direction information

There are many other optimal multi-channel filters in the literature that can be used to compute

, for example, a filter derived from [InformedSF] exists. Similarly, diffuse sound signals

is

Multi-channel filter indicated by multiple microphone signals

is determined by applying to

는 추정된 사운드 방향에 의존한다.

을 계산하는 데 사용될 수 있는 문헌에서 많은 다른 최적의 다중 채널 필터

, 예를 들어 [DiffuseBF]에서 도출되는 필터가 존재한다., where the multi-channel filter

depends on the estimated sound direction.

There are many other optimal multi-channel filters in the literature that can be used to compute

, for example, a filter derived from [DiffuseBF] exists.

및

을 각각 적용함으로써 이전 예에서와 같이

및

을 결정한다. 그러나, 상이한 차수 l 및 모드 m에 대해 결과적인 확산 사운드 신호

가 상호 상관되지 않도록, 상이한 차수 l 및 모드 m에 대해 상이한 필터

를 사용한다. 출력 신호들 사이의 상관을 최소화하는 이들 상이한 필터

은 예를 들어 [CovRender]에서 설명된 바와 같이 컴퓨팅될 수 있다.In a fourth example of block 110, a multi-channel filter on the microphone signal p(k, n)

and

As in the previous example, by applying each

and

to decide However, for different orders l and mode m the resulting diffuse sound signal

different filters for different orders l and mode m, so that

use These different filters to minimize the correlation between the output signals

may be computed, for example, as described in [CovRender].

는 블록(103)에서 결정된 공간 기반 함수의 응답

을 시간 및 주파수 타일마다 곱하여(115a) 결합되며, 이는 시간-주파수 타일 (k, n)에 대한 차수(레벨) l 및 모드 m의 다이렉트 사운드 앰비소닉스 성분

을 초래한다. 또한, 블록(105)에서 결정된 확산 사운드 신호

는 블록(106)에서 결정된 공간 기반 함수의 평균 응답

을 시간 및 주파수 타일마다 곱하여(115b) 결합되며, 이는 시간-주파수 타일 (k, n)에 대한 차수(레벨) l 및 모드 m의 확산 사운드 앰비소닉스 성분

을 초래한다.As in embodiment 4, the direct sound signal determined in block 105

is the response of the spatially based function determined in block 103 .

are combined by multiplying (115a) per time and frequency tile, which is the direct sound ambisonics component of order (level) l and mode m for the time-frequency tile (k, n).

causes Also, the diffuse sound signal determined in block 105 .

is the mean response of the spatially based function determined in block 106 .

is combined by multiplying (115b) per time and frequency tile, which is the diffuse sound ambisonics component of order (level) l and mode m for the time-frequency tile (k, n).

causes

및 확산 사운드 앰비소닉스 성분

은 예를 들어 합산 연산(109)을 통해 결합되어, 시간-주파수 타일 (k, n)에 대한 원하는 차수(레벨) l 및 모드 m의 최종 앰비소닉스 성분

을 획득한다. 결과적인 앰비소닉스 성분

은 결국 역 필터 뱅크 또는 역 STFT를 사용하여 시간 도메인으로 다시 변환되거나, 저장되거나, 송신되거나 또는 예를 들어 공간 사운드 재생을 위해 사용된다. 실제로, 원하는 최대 차수(레벨)의 원하는 앰비소닉스 신호를 획득하기 위해 원하는 모든 차수 및 모드에 대해 앰비소닉스 성분을 컴퓨팅할 것이다. 실시예 3에서 설명한 바와 같이, 시간 도메인으로의 변환은

을 컴퓨팅하기 전에, 즉 연산(109) 전에 수행될 수 있다.As in Example 3, computed direct sound ambisonics component

and diffuse sound ambisonics components

are combined, e.g. via a summation operation 109, so that the final ambisonics component of the desired order (level) l and mode m for the time-frequency tile (k, n)

to acquire The resulting ambisonics component

is in turn transformed back to the time domain using an inverse filter bank or inverse STFT, stored, transmitted or used, for example, for spatial sound reproduction. In practice, we will compute the ambisonics component for every desired order and mode to obtain the desired ambisonics signal of the maximum desired order (level). As described in Example 3, the transformation to the time domain is

may be performed before computing , that is, before operation 109 .

및 확산 사운드 앰비소닉스 성분

이 서로 다른 모드(차수) l에 대해 컴퓨팅되도록 구성될 수 있음에 유의한다. 예를 들어,

은 차수 l = 4까지 컴퓨팅될 수 있고, 한편

는 단지 차수 l=1까지만 컴퓨팅될 수 있다 (이 경우에, 큰 차수 l = 1의 경우

은 0이 될 것이다). 특정 차수 (레벨) l 또는 모드 m에 대하여

만을 계산하고

은 계산하지 않기를 원한다면, 예를 들어 블록(110)은 확산 사운드 신호

가 0이 되도록 구성될 수 있다. 이것은 예를 들어 이전의 방정식에서 필터

를 0으로 설정하고 필터

을 1로 설정함으로써 달성될 수 있다. 유사하게, 필터

은 0으로 설정될 수 있다.The algorithm of this embodiment is a direct sound ambisonics component

and diffuse sound ambisonics components

Note that this can be configured to compute for different modes (orders) l. For example,

can be computed up to order l = 4, while

can be computed only up to order l=1 (in this case, for large orders l = 1)

will be 0). For a specific order (level) l or mode m

count only

If we do not wish to calculate, for example, block 110 is a diffuse sound signal

may be configured to be 0. This is for example a filter in the previous equation

set to 0 and filter

can be achieved by setting Similarly, filter

may be set to 0.

Example 6

Figure 7 shows another embodiment of the present invention which allows to synthesize ambisonics components of the desired order (level) l and mode m from the signals of multiple (two or more) microphones. The embodiment is similar to embodiment 5, but additionally includes a decorrelator for the diffuse ambisonics component.

As in embodiment 5, the input to the present invention is the signal of multiple (two or more) microphones. The microphones may be arranged in any shape, such as, for example, a coincident setup, a linear array, a planar array, or a three-dimensional error. Also, each microphone may be omni-directional or any directional. The directivity of different microphones may be different.

으로 표시되는 시간-주파수 도메인의 마이크로폰 신호이다. 이하의 처리는 시간-주파수 타일 (k, n)에 대해 개별적으로 수행된다.As in embodiment 5, multiple microphone signals are transformed into the time-frequency domain in block 101 using, for example, a filter bank or short-time Fourier transform (STFT). The output of the time-frequency transform 101 is

It is a microphone signal in the time-frequency domain denoted by . The following processing is performed separately for time-frequency tiles (k, n).

. 를 사용하여 시간 및 주파수마다 블록(102)에서 사운드 방향 추정이 수행된다. 대응하는 추정은 실시예 1에서 논의되었다. 사운드 방향 추정기(102)의 출력은 시간 인스턴스 n 및 주파수 인덱스 k 당 사운드 방향이다. 사운드 방향은 예를 들어 단위 놈벡터 n(k, n) 또는 방위각

및/또는 앙각

의 관점에서 표현될 수 있으며, 이는 실시예 1에서 설명한 바와 같다.As in embodiment 5, two or more microphone signals

. Sound direction estimation is performed at block 102 per time and frequency using Corresponding estimates are discussed in Example 1. The output of the sound direction estimator 102 is the sound direction per time instance n and frequency index k. The sound direction can be, for example, a unit norm vector n(k, n) or an azimuth

and/or elevation

It can be expressed in terms of , which is the same as described in Example 1.

로 표시된다. 예를 들어, 공간 기반 함수로서 N3D 정규화를 갖는 실수 값의 구면 고조파를 고려할 수 있고, 실시예 1에서 설명한 바와 같이

이 결정될 수 있다.As in embodiment 5, the response of the spatially based function of the desired order (level) l and mode m is determined in block 103 for each time and frequency using the estimated sound direction information. The response of a space-based function is

is displayed as For example, we can consider real-valued spherical harmonics with N3D normalization as a spatial-based function, as described in Example 1

This can be determined.

로 표시되며 가능한 모든 방향(예를 들어, 확산 사운드 또는 주변 사운드)에서 도착하는 사운드에 대한 공간 기반 함수의 응답을 나타낸다. 평균 응답

는 실시예 3에 기술된 바와 같이 획득될 수 있다.As in embodiment 5, the average response of the spatially based function of the desired order (level) l and mode m independent of the temporal index n is obtained from block 106 . This average response is

denotes the response of a spatially-based function to a sound arriving from all possible directions (e.g., diffuse or ambient sound). average response

can be obtained as described in Example 3.

및 확산 사운드 신호

은 2개 이상의 이용 가능한 마이크로폰 신호

로부터 시간 인덱스 n 및 주파수 인덱스 k마다 블록(110)에서 결정된다. 이 목적을 위해, 블록(110)은 일반적으로 블록(102)에서 결정된 사운드 방향 정보를 이용한다. 블록 (110)의 다른 예가 실시예 5에서 설명된다.As in embodiment 5, direct sound signal

and diffuse sound signals

is two or more available microphone signals

is determined in block 110 for each time index n and frequency index k from For this purpose, block 110 generally uses the sound direction information determined in block 102 . Another example of block 110 is described in Example 5.

는 블록(103)에서 결정된 공간 기반 함수의 응답

을 시간 및 주파수 타일마다 곱하여(115a) 결합되며, 이는 시간-주파수 타일 (k, n)에 대한 차수(레벨) l 및 모드 m의 다이렉트 사운드 앰비소닉스 성분

을 초래한다. 또한, 블록(105)에서 결정된 확산 사운드 신호

는 블록(106)에서 결정된 공간 기반 함수의 평균 응답

을 시간 및 주파수 타일마다 곱하여(115b) 결합되며, 이는 시간-주파수 타일 (k, n)에 대한 차수(레벨) l 및 모드 m의 확산 사운드 앰비소닉스 성분

을 초래한다.As in embodiment 5, the direct sound signal determined in block 105

is the response of the spatially based function determined in block 103 .

are combined by multiplying (115a) per time and frequency tile, which is the direct sound ambisonics component of order (level) l and mode m for the time-frequency tile (k, n).

causes Also, the diffuse sound signal determined in block 105 .

is the mean response of the spatially based function determined in block 106 .

is combined by multiplying (115b) per time and frequency tile, which is the diffuse sound ambisonics component of order (level) l and mode m for the time-frequency tile (k, n).

causes

은 상관 해제기를 사용하여 블록(107)에서 상관 해제되며, 이는

으로 표시되는 상관 해제된 확산 사운드 앰비소닉스 성분을 초래한다. 상관 해제 뒤에 있는 추론 및 방법은 실시예 4에서 논의된다. 실시예 4에서와 같이, 상관 해제기(107)를 적용하기 전에 예를 들어 역 필터 뱅크 또는 역 STFT를 사용하여 확산 사운드 앰비소닉스 성분

이 시간 도메인으로 다시 변환될 수 있다.As in Example 4, the calculated diffuse sound ambisonics component

is decorrelated at block 107 using the decorrelator, which

resulting in a decorrelated diffuse sound ambisonics component denoted by The reasoning and methods behind decorrelation are discussed in Example 4. As in embodiment 4, before applying the decorrelator 107, using for example an inverse filter bank or an inverse STFT, a diffuse sound ambisonics component

This can be transformed back into the time domain.

및 상관 해제된 확산 사운드 앰비소닉스 성분

은 예를 들어 합산 연산(109)을 통해 결합되어, 시간-주파수 타일 (k, n)에 대한 원하는 차수(레벨) l 및 모드 m의 최종 앰비소닉스 성분

을 획득한다. 결과적인 앰비소닉스 성분

은 결국 역 필터 뱅크 또는 역 STFT를 사용하여 시간 도메인으로 다시 변환되거나, 저장되거나, 송신되거나 또는 예를 들어 공간 사운드 재생을 위해 사용된다. 실제로, 원하는 최대 차수(레벨)의 원하는 앰비소닉스 신호를 획득하기 위해 원하는 모든 차수 및 모드에 대해 앰비소닉스 성분을 컴퓨팅할 것이다. 실시예 4에서 설명한 바와 같이, 시간 도메인으로의 변환은

을 컴퓨팅하기 전에, 즉 연산(109) 전에 수행될 수 있다.As in Example 4, direct sound ambisonics component

and uncorrelated diffuse sound ambisonics components

are combined, e.g. via a summation operation 109, so that the final ambisonics component of the desired order (level) l and mode m for the time-frequency tile (k, n)

to acquire The resulting ambisonics component

is in turn transformed back to the time domain using an inverse filter bank or inverse STFT, stored, transmitted or used, for example, for spatial sound reproduction. In practice, we will compute the ambisonics component for every desired order and mode to obtain the desired ambisonics signal of the maximum desired order (level). As described in Example 4, the transformation to the time domain is

may be performed before computing , that is, before operation 109 .

및 확산 사운드 앰비소닉스 성분

이 서로 다른 모드(차수) l에 대해 컴퓨팅되도록 구성될 수 있음에 유의한다. 예를 들어,

은 차수 l=4까지 컴퓨팅 될 수 있고, 한편

은 단지 l=1까지만 컴퓨팅될 수 있다.As described in Example 4, the algorithm of this example uses direct sound ambisonics components.

and diffuse sound ambisonics components

Note that this can be configured to compute for different modes (orders) l. For example,

can be computed up to order l=4, while

can be computed only up to l=1.

Example 7

Figure 8 shows another embodiment of the present invention which allows synthesizing ambisonics components of desired order (level) l and mode m from signals of multiple (two or more) microphones. The embodiment is similar to embodiment 1, but further includes a block 111 for applying a smoothing operation to the computed response of the spatial based function.

As in embodiment 1, the input to the present invention is the signal of multiple (two or more) microphones. The microphones may be arranged in any shape, such as, for example, a coincident setup, a linear array, a planar array, or a three-dimensional error. Also, each microphone may be omni-directional or any directional. The directivity of different microphones may be different.

으로 표시되는 시간-주파수 도메인의 마이크로폰 신호이다. 이하의 처리는 시간-주파수 타일 (k, n)에 대해 개별적으로 수행된다.As in embodiment 1, multiple microphone signals are transformed into the time-frequency domain in block 101 using, for example, a filter bank or a short-time Fourier transform (STFT). The output of the time-frequency transform 101 is

It is a microphone signal in the time-frequency domain denoted by . The following processing is performed separately for time-frequency tiles (k, n).

이다.As in embodiment 1, without loss of generality, the first microphone signal is referred to as a reference microphone signal, i.e.

to be.

를 사용하여 시간 및 주파수마다 블록(102)에서 사운드 방향 추정이 수행된다. 대응하는 추정은 실시예 1에서 논의되었다. 사운드 방향 추정기(102)의 출력은 시간 인스턴스 n 및 주파수 인덱스 k 당 사운드 방향이다. 사운드 방향은 예를 들어 단위 놈벡터 n(k, n) 또는 방위각

및/또는 앙각

의 관점에서 표현될 수 있으며, 이는 실시예 1에서 설명한 바와 같다.As in embodiment 1, two or more microphone signals

Sound direction estimation is performed at block 102 per time and frequency using Corresponding estimates are discussed in Example 1. The output of the sound direction estimator 102 is the sound direction per time instance n and frequency index k. The sound direction can be, for example, a unit norm vector n(k, n) or an azimuth

and/or elevation

It can be expressed in terms of , which is the same as described in Example 1.

로 표시된다. 예를 들어, 공간 기반 함수로서 N3D 정규화를 갖는 실수 값의 구면 고조파를 고려할 수 있고, 실시예 1에서 설명한 바와 같이

이 결정될 수 있다.As in Example 1, the response of the spatially based function of the desired order (level) l and mode m is determined in block 103 for each time and frequency using the estimated sound direction information. The response of a space-based function is

is displayed as For example, we can consider real-valued spherical harmonics with N3D normalization as a spatial-based function, as described in Example 1

This can be determined.

에 평활화 연산을 적용하는 블록(111)에 대한 응답으로서 응답

이 사용된다. 블록(111)의 출력은

으로 표시되는 평활화된 응답 함수이다. 평활화 연산의 목적은 예를 들어 블록(102)에서 추정된 사운드 방향

및/또는

에 노이즈가 있으면 실제로 일어날 수 있는, 값

의 원치 않는 추정 분산을 감소시키는 것이다.

에 적용되는 평활화는 예를 들어 시간 및/또는 주파수에 걸쳐 수행될 수 있다. 예를 들어, 시간 평활화는 잘 알려진 재귀 평균화 필터In contrast to Example 1,

In response to block 111 applying a smoothing operation to

this is used The output of block 111 is

It is a smoothed response function denoted by . The purpose of the smoothing operation is, for example, in block 102 the estimated sound direction.

and/or

A value that can actually happen if there is noise in

is to reduce the unwanted estimated variance of .

The smoothing applied to may be performed over time and/or frequency, for example. For example, time smoothing is a well-known recursive averaging filter

은 이전 시간 프레임에서 컴퓨팅된 응답 함수이다. 또한, α는 시간 평활화의 강도를 제어하는 0과 1 사이의 실수 값이다. 0에 가까운 값의 경우, 강한 시간 평균이 수행되는 반면, 1에 가까운 α의 값에 대해서는 짧은 시간 평균이 수행된다. 실제 응용에서 α의 값은 응용에 따라 다르며 예를 들어 α = 0.5와 같이 일정하게 설정될 수 있다. 대안적으로, 스펙트럼 평활화가 블록(111)에서도 수행될 수 있으며, 이것은 응답

이 다수의 주파수 대역에 걸쳐 평균된다는 것을 의미한다. 소위 ERB 대역 내에서의 그러한 스펙트럼 평활화는 예를 들어 [ERBsmooth]에 설명되어 있다.can be achieved using

is the response function computed in the previous time frame. Also, α is a real value between 0 and 1 that controls the strength of temporal smoothing. For values close to zero, strong time averaging is performed, while for values of α close to 1, short time averaging is performed. In actual applications, the value of α depends on the application and can be set constant, for example, α = 0.5. Alternatively, spectral smoothing may also be performed at block 111, which is

This means that it is averaged over multiple frequency bands. Such spectral smoothing within the so-called ERB band is described for example in [ERBsmooth].

는 최종적으로 블록(111)에서 결정된 공간 기반 함수의 평활화된 응답

)을 시간 주파수 타일마다 곱하여(115) 결합되며, 이는 시간-주파수 타일 (k, n)에 대한 차수(레벨) l 및 모드 m의 원하는 앰비소닉스 성분

을 초래한다. 결과적인 앰비소닉스 성분 B_lm (k, n)은 결국 역 필터 뱅크 또는 역 STFT를 사용하여 시간 도메인으로 다시 변환되거나, 저장되거나, 송신되거나 또는 예를 들어 공간 사운드 재생을 위해 사용된다. 실제로, 원하는 최대 차수(레벨)의 원하는 앰비소닉스 신호를 획득하기 위해 원하는 모든 차수 및 모드에 대해 앰비소닉스 성분을 컴퓨팅할 것이다.In this embodiment, the reference microphone signal

is the smoothed response of the spatial-based function finally determined in block 111 .

) is combined by multiplying (115) per time frequency tile, which is the desired ambisonics component of order (level) l and mode m for the time-frequency tile (k, n).

causes The resulting ambisonics component B_lm (k, n) is in turn transformed back to the time domain using an inverse filter bank or inverse STFT, stored, transmitted or used, for example, for spatial sound reproduction. In practice, we will compute the ambisonics component for every desired order and mode to obtain the desired ambisonics signal of the maximum desired order (level).

Obviously, the gain smoothing at block 111 can also be applied to all other embodiments of the present invention.

Example 8

및/또는 앙각

의 면에서 표시되는 다수의 사운드 방향이다.The present invention can also be applied in the case of so-called multi-wave, in which more than one sound direction is considered per time-frequency tile. For example, Embodiment 2 shown in Fig. 3B can be realized in the case of multi-wave. In this case, block 102 estimates J sound direction per time and frequency, where J is an integer value greater than one, eg J=2. Modern estimators such as ESPRIT or Root MUSIC described in [ESPRIT, RootMUSIC1] can be used to estimate multiple sound directions. In this case, the output of block 102 is multiple azimuths, such as multiple azimuths.

and/or elevation

Multiple sound directions displayed in terms of

을 컴퓨팅한다. 또한, 블록(102)에서 계산된 다수의 사운드 방향은 블록(104)에서 사용되어 다수의 사운드 방향 각각에 대해 하나인 다수의 기준 신호

를 계산한다. 다수의 기준 신호 각각은 예를 들어 실시예 2에서 설명한 바와 같이 다수의 마이크로폰 신호에 멀티 채널 필터

을 적용함으로써 계산될 수 있다. 예를 들어, 제1 기준 신호

는 최신의 멀티 채널 필터 w_1 (n)을 적용함으로써 획득될 수 있으며, 멀티 채널 필터

는 방향

및/또는

에서 사운드를 추출하면서 다른 모든 사운드 방향의 사운드를 감쇠시킬 것이다. 이러한 필터는 예를 들어 [InformedSF]에 설명된 정보가 주어진 LCMV 필터로 컴퓨팅될 수 있다. 다수의 기준 신호

는 그 다음에 대응하는 다수의 응답

이 곱해져 다수의 앰비소닉스 성분

을 획득한다. 예를 들어, j번째 사운드 방향 및 기준 신호에 대응하는 j번째 앰비소닉스 성분은Multiple sound directions are used in block 103, multiple responses being one response to each estimated sound direction as discussed in Example 1

to compute In addition, the number of sound directions calculated in block 102 is used in block 104 for a plurality of reference signals, one for each of the plurality of sound directions.

to calculate Each of the plurality of reference signals may be subjected to a multi-channel filter on the plurality of microphone signals, for example as described in Embodiment 2

can be calculated by applying For example, the first reference signal

can be obtained by applying the latest multi-channel filter w_1 (n),

direction

and/or

will attenuate the sound in all other sound directions while extracting the sound from Such a filter may be computed, for example, with an LCMV filter given the information described in [InformedSF]. multiple reference signals

is followed by a number of corresponding responses

multiplied by multiple ambisonics components

to acquire For example, the jth sound direction and the jth ambisonics component corresponding to the reference signal are

is calculated as

을 획득한다, 즉Finally, the J ambisonics components are summed so that the final desired ambisonics component of order (level) l and mode m for the frequency-time tile (k,n)

to obtain, i.e.

to be.

을 계산할 수 있다. 그 다음에, 다수의 다이렉트 사운드는 다수의 다이렉트 사운드 앰비소닉스 성분

에 이르는 대응하는 다수의 응답

을 합산하여 최종적인 원하는 다이렉트 사운드 앰비소닉스 성분

을 획득할 수 있다.Obviously, other above-described embodiments can also be extended to the case of multi-wave. For example, in Examples 5 and 6, a plurality of direct sounds, one for each of a plurality of sound directions, using the same multi-channel filter as mentioned in this embodiment

can be calculated. Then, the multiple direct sound consists of multiple direct sound ambisonics components.

corresponding multiple responses leading to

The final desired direct sound ambisonics component by summing

can be obtained.

It should be noted that the present invention is applicable not only to two-dimensional (cylindrical) or three-dimensional (spherical) ambisonics techniques, but also to any other technique that relies on a spatially based function for calculating arbitrary sound field components.

Examples of the invention as a list

1. Convert multiple microphone signals into the time frequency domain.

2. Calculate one or more directions by time and frequency from multiple microphone signals.

3. Compute one or more response functions for each time and frequency according to one or more sound directions.

4. Acquire one or more reference microphone signals for each time and frequency.

5. Multiply for each time and frequency one or more reference microphone signals by one or more response functions to obtain one or more ambisonics components of the desired order and mode.

6. When several ambisonics components are secured for a desired order and mode, the ambisonics components are summed to obtain a final desired ambisonics component.

4. In some embodiments, in step 4, compute one or more direct sounds and diffuse sounds from multiple microphone signals instead of one or more reference microphone signals.

5. Multiply the one or more direct and diffuse sounds by one or more corresponding direct and diffuse sound responses to obtain one or more direct sound ambisonics components and diffuse sound ambisonics components in a desired order and mode.

6. Diffuse sound ambisonics components may further be uncorrelated for different orders and modes.

7. Sum the direct loud ambisonics components and spread the desired ambisonics components to obtain a final desired ambisonics component of the desired order and mode.

references

[Ambsonics] R. K. Furness, "Ambsonics - An overview," in AES 8th International Conference, April 1990, pp. 181-189.

[Ambix] C. Nachbar, F. Zotter, E. Deleflie, and A. Sontacchi, "AMBIX - A Suggested Ambisonics Format", Proceedings of the Ambisonics Symposium 2011.

[ArrayDesign] M. Williams and G. Le Du, “Multichannel Microphone Array Design,” in Audio Engineering Society Convention 108, 2008.

[CovRender] J. Vilkamo and V. Pulkki, “Minimization of Decorrelator Artifacts in Directional Audio Coding by Covariance Domain Rendering”, J. Audio Eng. Soc, vol. 61, no. 9, 2013.

[DiffuseBF] O. Thiergart and E. A. P. Habets, “Extracting Reverberant Sound Using a Linearly Constrained Minimum Variance Spatial Filter,” IEEE Signal Processing Letters, vol. 21, no. 5, May 2014.

[DirAC] V. Pulkki, ''Directional audio coding in spatial sound reproduction and stereo upmixing,'' in Proceedings of The AES 28th International Conference, pp. 251-258, June, 2006.

[EigenMike] J. Meyer and T. Agnello, "Spherical microphone array for spatial sound recording," in Audio Engineering Society Convention 115, October 2003

[ERBsmooth] A. Favrot and C. Faller, “Perceptually Motivated Gain Filter Smoothing for Noise Suppression”, Audio Engineering Society Convention 123, 2007.

[ESPRIT] R. Roy, A. Paulraj, and T. Kailath, "Direction-of-arrival estimation by subspace rotation methods - ESPRIT," in IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Stanford, CA , USA, April, 1986.

[FourierAcoust] E. G. Williams, "Fourier Acoustics: Sound Radiation and Nearfield Acoustical Holography," Academic Press, 1999.

[HARPEX] S. Berge and N. Barrett, "High Angular Resolution Planewave Expansion,'' in 2nd International Symposium on Ambisonics and Spherical Acoustics, May, 2010.

[InformedSF] O. Thiergart, M. Taseska, and E. A. P. Habets, "An Informed Parametric Spatial Filter Based on Instantaneous Direction-of-Arrival Estimates," IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 22, no. 12, December 2014.

[MicSetup3D] H. Lee and C. Gribben, “On the optimum microphone array configuration for height channels,” in 134 AES Convention, Rome, 2013.

[MUSIC] R. Schmidt, "Multiple emitter location and signal parameter estimation," IEEE Transactions on Antennas and Propagation, vol. 34, no. 3, pp. 276-280, 1986.

[OptArrayPr] B. D. Van Veen and K. M. Buckley, “Beamforming: A versatile approach to spatial filtering”, IEEE ASSP Magazine, vol. 5, no. 2, 1988.

[RootMUSIC1] B. Raoand and K.Hari, "Performance analysis of root-MUSIC," in Signals, Systems and Computers, 1988. Twenty-Second Asilomar Conference on, vol. 2, 1988, pp. 578-582.

[RootMUSIC2] A. Mhamdi and A. Samet, "Direction of arrival estimation for nonuniform linear antenna," in Communications, Computing and Control Applications (CCCA), 2011 International Conference on, March 2011, pp. 1-5.

[RootMUSIC3] M. Zoltowski and CP Mathews, "Direction finding with uniform circular arrays via phase mode excitation and beamspace root-MUSIC," in Acoustics, Speech, and Signal Processing, 1992. ICASSP-92., 1992 IEEE International Conference on, vol. 5, 1992, pp. 245-248.

[SDRestim] O. Thiergart, G. Del Galdo, and E A. P. Habets, "On the spatial coherence in mixed sound fields and its application to signal-to-diffuse ratio estimation", The Journal of the Acoustical Society of America, vol. 132, no. 4, 2012.

[SourceNum] J.-S. Jiang and M.-A. Ingram, “Robust detection of number of sources using the transformed rotational matrix,” in Wireless Communications and Networking Conference, 2004. WCNC. 2004 IEEE, vol. 1, March, 2004.

[SpCoherence] D. P. Jarrett, O. Thiergart, E. A. P. Habets, and P. A. Naylor, “Coherence-Based Diffuseness Estimation in the Spherical Harmonic Domain,” IEEE 27th Convention of Electrical and Electronics Engineers in Israel (IEEEI), 2012.

[SphHarm] F. Zotter, "Analysis and Synthesis of Sound-Radiation with Spherical Arrays", PhD thesis, University of Music and Performing Arts Graz, 2009.

[VirtualMic] O. Thiergart, G. Del Galdo, M. Taseska, and E. A. P. Habets, "Geometry-based Spatial Sound Acquisition Using Distributed Microphone Arrays," IEEE Transactions on in Audio, Speech, and Language Processing, vol. 21, no. 12, De

Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of a corresponding method, where blocks and devices correspond to method steps or features of method steps. Similarly, an aspect described in the context of a method step also represents a description of a feature of a corresponding block or item or corresponding device.

A signal of the present invention may be stored in a digital storage medium or transmitted over a transmission medium such as a wired transmission medium such as the Internet or a wireless transmission medium.

Depending on specific implementation requirements, embodiments of the present invention may be implemented in hardware or software. An implementation may implement a digital storage medium, for example a floppy disk, DVD, CD, ROM, PROM, EPROM, having stored thereon electronically readable control signals that cooperate with (or can cooperate with) a programmable computer system to cause the respective method to be performed. , using EEPROM or flash memory.

Some embodiments in accordance with the present invention comprise a non-transitory data carrier having an electronically readable control signal capable of cooperating with a programmable computer system to perform one of the methods described herein.

In general, embodiments of the present invention may be implemented as a computer program product having program code operative to perform one of the methods when the computer program product is run on a computer. The program code may be stored on a machine readable carrier, for example.

Another embodiment comprises a computer program for performing one of the methods described herein, stored on a machine readable carrier.

In other words, an embodiment of the method of the present invention is, therefore, a computer program having program code for performing one of the methods described herein when the computer program is run on a computer.

Thus, another embodiment of the method of the present invention is a data carrier (or digital storage medium or computer readable medium) having recorded thereon a computer program for performing one of the methods described herein.

Accordingly, another embodiment of the method of the present invention is a data stream or sequence of signals representing a computer program for performing one of the methods described herein. A data stream or sequence of signals may be configured to be transmitted over a data communication connection, for example over the Internet.

Another embodiment comprises processing means, for example a computer or programmable logic device, configured or adapted to perform one of the methods described herein.

Another embodiment comprises a computer installed with a computer program for performing one of the methods described herein.

In some embodiments, a programmable logic device (eg, a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, the field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. In general, the method is preferably performed by any hardware device.

The embodiments described above are only for illustrating the principles of the present invention. It is understood that modifications and variations of the constructions and details described herein will be apparent to those skilled in the art. Accordingly, it is to be limited only by the scope of the appended claims and not by the specific details provided by the description and description of the embodiments herein.

Claims (17)

An apparatus for generating a sound field description having a representation of a sound field component, comprising:
a direction determiner 102 for determining one or more sound directions for each time-frequency tile of the plurality of time-frequency tiles of the plurality of microphone signals;
a space-based function evaluator (103) for evaluating, for each time-frequency tile of the plurality of time-frequency tiles, one or more spatial-based functions using the one or more sound directions; and
For each time-frequency tile of the plurality of time-frequency tiles, using the one or more spatial-based functions evaluated using the one or more sound directions and using a reference signal for a corresponding time-frequency tile a sound field component calculator (201) for calculating one or more sound field components corresponding to the one or more spatially based functions, the reference signal being derived from one or more microphone signals of the plurality of microphone signals;

a direct or diffuse sound determiner (105) for determining a direct portion or a diffuse portion of the plurality of microphone signals as the reference signal;
The sound field component calculator 201 is configured to use the direct part only when calculating one or more direct sound field components,

The diffuse component calculator 301 is configured to calculate the diffuse sound component up to a first predetermined number or order,
The sound field component calculator 201 is configured to calculate a direct sound field component up to a second predetermined number or order,
the second predetermined number or order is greater than the first predetermined number or order;
and the first predetermined number or order is greater than or equal to one. According to claim 1,
the diffuse component calculator 301 is configured to calculate, for each time-frequency tile of the plurality of time-frequency tiles, one or more diffuse sound components,
An apparatus for generating a sound field description with a representation of a sound field component, further comprising a combiner (401) for combining the diffuse sound information and the direct sound field information to obtain a frequency domain representation or a time domain representation of the sound field component . 3. The method of claim 2,
and the diffuse component calculator (301) further comprises a decorrelator (107) for decorrelating the diffuse sound information. According to claim 1,
for generating a sound field description with a representation of a sound field component, further comprising a time-frequency converter (101) for converting each of a plurality of time domain microphone signals into a frequency representation having said plurality of time-frequency tiles Device. According to claim 1,
and a frequency-time converter (20) for converting the one or more sound field components or a combination of the one or more sound field components and diffuse sound components into a time domain representation of the sound field components. A device for creating sound field technology. 6. The method of claim 5,
The frequency-time converter 20 is configured to process the one or more sound field components to obtain a plurality of time domain sound field components, and the frequency-time converter processes the diffuse sound components to generate a plurality of time domain spread components. and the combiner 401 is configured to perform combining of the time domain sound field component and the time domain diffusion component in the time domain;
The combiner 401 is configured to combine one or more sound field components for a time-frequency tile in the frequency domain and a diffuse sound component for the corresponding time-frequency tile, wherein the frequency-time converter 20 comprises a sound field component for the corresponding time-frequency tile in the time domain. and process the result of the combiner (401) to obtain a component. According to claim 1,
using one or more of the sound directions, or
using selecting a particular microphone signal from the plurality of microphone signals based on the one or more sound directions;
using a multi-channel filter applied to two or more microphone signals, wherein the multi-channel filter depends on the respective position of the microphone and the one or more sound directions from which the plurality of microphone signals are obtained;
and a reference signal calculator (104) for calculating a reference signal from the plurality of microphone signals. According to claim 1,
The spatially based function evaluator 103 uses a parameterized expression for the spatially based function, wherein a parameter of the parameterized representation is a sound direction, to obtain an evaluation result for each spatially based function. configured to insert a parameter corresponding to a sound direction into the parameterized representation;
The spatially based function evaluator (103) is configured to use a lookup table for each spatially based function having a spatial based function identification and sound direction as input and an evaluation result as an output, the spatial based function evaluator (103) ) determines a corresponding sound direction of the lookup table input for the one or more sound directions determined by the direction determiner, or is a weighted value between two lookup table inputs adjacent to the one or more sound directions determined by the direction determiner. or to calculate an unweighted average;
The spatially based function evaluator 103 uses a parameterized representation for the spatially based function, wherein the parameter of the parameterized representation is a sound direction, wherein the sound direction is one-dimensional or three-dimensional, such as azimuth in a two-dimensional situation. two-dimensional, such as azimuth and elevation in a dimensional situation - a sound field component, configured to insert a parameter corresponding to the sound direction into the parameterized representation to obtain an evaluation result for each spatial-based function A device for generating a sound field description having a representation of According to claim 1,
The direct or diffuse sound determiner 105 is
calculate the direct part and the diffuse part from a single microphone signal, or - the diffuse component calculator 301 is configured to calculate the one or more diffuse sound components using the diffuse part as the reference signal, the sound field component calculator 301 201) is configured to calculate the one or more direct sound field components using the direct portion as the reference signal;
or calculate a diffusion portion from a microphone signal different from the microphone signal for which the direct portion is calculated, wherein the diffusion component calculator 301 is configured to calculate the one or more diffusion sound components using the diffusion portion as the reference signal, the sound field component calculator 201 is configured to calculate the one or more direct sound field components using the direct portion as the reference signal;
calculate the spreading part for different spatially based functions using different microphone signals, or - the spreading component calculator 301 uses the first spreading part as a reference signal for the average spatial based function response corresponding to the first number, and use a different second diffusion portion as a reference signal corresponding to a second number-averaged spatial-based function response, wherein the first number is different from the second number, and the first number and the second number are the one represents any order or level and mode of the above spatial-based function;
calculating the direct portion using a first multi-channel filter applied to the plurality of microphone signals, and calculating the spreading portion using a second multi-channel filter applied to the plurality of microphone signals, or - the second multi-channel filter is different from the first multi-channel filter, wherein the diffusion component calculator 301 is configured to calculate the one or more diffuse sound components using the diffusion portion as the reference signal, and the sound field component calculator 201 includes the and calculate the one or more direct sound field components using a direct portion as the reference signal;
to calculate a spreading part for different spatially based functions using different multi-channel filters for the different spatially based functions - the spreading component calculator 301 uses the spreading part as the reference signal to determine the one or more diffuse sound components and wherein the sound field component calculator (201) is configured to calculate the one or more direct sound field components using the direct portion as the reference signal. device to create. According to claim 1,
In order to obtain a smoothed evaluation result, the spatial-based function evaluator 103 includes a gain smoother 111 operating in a time direction or a frequency direction to smooth the evaluation result,
and the sound field component calculator (201) is configured to use the smoothed evaluation result when calculating the one or more sound field components. According to claim 1,
and the spatial based function evaluator (103) is configured to use one or more spatially based functions for ambisonics in a two-dimensional or three-dimensional situation. 12. The method of claim 11,
The apparatus for generating a sound field description with a representation of a sound field component, characterized in that the spatial based function calculator (103) is configured to use a spatial based function of at least two levels or orders or at least two modes. 13. The method of claim 12,
the sound field component calculator 201 is configured to calculate a sound field component for at least two levels of a level group including level 0, level 1, level 2, level 3, and level 4;
The sound field component calculator 201 is configured for at least two modes of a mode group including mode-4, mode-3, mode-2, mode-1, mode0, mode1, mode2, mode3, and mode4. An apparatus for generating a sound field description having a representation of a sound field component, characterized in that the apparatus is configured to calculate a sound field component. A method of generating a sound field description having a representation of a sound field component, the method comprising:
determining (102) one or more sound directions for each time-frequency tile of a plurality of time-frequency tiles of the plurality of microphone signals;
for each time-frequency tile of the plurality of time-frequency tiles, evaluating (103) one or more spatially based functions using the one or more sound directions;
for each time-frequency tile of the plurality of time-frequency tiles, using one or more spatial-based functions evaluated using the one or more sound directions and using a reference signal for a corresponding time-frequency tile. calculating (201) one or more sound field components corresponding to one or more spatially based functions, wherein the reference signal is derived from one or more microphone signals of the plurality of microphone signals;

determining a direct portion or a diffusion portion of the plurality of microphone signals as the reference signal;
The calculating step 201 includes using the direct portion only when calculating one or more direct sound field components,

calculating a diffuse sound component up to a first predetermined number or order;
The calculating step 201 includes calculating a direct sound field component up to a second predetermined number or order,
the second predetermined number or order is greater than the first predetermined number or order;
and the first predetermined number or order is greater than or equal to one. A storage medium storing a computer program for performing the method for generating the sound field technology of claim 14 when driven on a computer or processor. delete delete

Images