EP3449643B1 - Method and system of broadcasting a 360° audio signal - Google Patents

Description

Field of the invention

The present invention relates to the field of sound signal processing.

State of the art

Methods and systems are known in the state of the art for broadcasting 360 ° video signals. There is a need in the prior art to be able to associate audio signals with these 360 ° video signals.

Until now, 3D audio has been reserved for sound professionals and researchers. The purpose of this technology is to capture as much spatial information as possible during recording, then restore it to the listener and give them a feeling of immersion in the sound scene. In the field of video, the interest is growing for videos filmed at 360 ° and reproduced via a virtual reality headset for a complete immersion in the image: the user can turn his head and explore the visual scene all around from him. To obtain the same fidelity in the field of sound, the most compact solution is the use of a network of microphones, such as for example the Eigenmike from mh acoustics, the Soundfield from TSL Products, and the TetraMic from Core Sound. The document WO 2005/015954 shows a method for converting the signals of an array of microphones placed in a spherical configuration into an ambisonic format. Equipped with four to thirty-two microphones, these products are expensive and therefore reserved for professional use. Recent research has reduced the number of microphones ( Palacino, JD, & Nicol, R. (2013). "Spatial sound pick-up with a low number of microphones." ICA 2013. Montreal, Canada .), and microphones of reduced size and cost can be used such as those available in mobile phones. The shape of the microphone networks, a polyhedron, remains standardized, however, from the dodecahedron for the EigenMike to the tetrahedron for the Soundfield and the TetraMic. This geometric shape allows the use of simple formulas to convert microphone signals into a format ambisonic, and were developed by Gerzon in 1975 ( Gerzon, M. (1975). "The design of precisely coincident microphone arrays for stereo and surround sound. »50th Audio Engineering Society Conference . ). The ambisonic format is a set of audio channels which contains all the information necessary for the spatial reconstruction of the sound field. A novelty brought by the present patent is the possibility of using any form of array of microphones. Thus, it is quite possible to use an already existing form, such as a 360 ° camera or a mobile phone, to include a certain number of microphones. The result is a complete and compact 360 ° sound and image recording system.

Statement of the invention

The present invention intends to remedy the drawbacks of the prior art by proposing a method for processing the sound signal making it possible to capture the sound signal in all directions, then to restore said sound signal.

To this end, the present invention relates, in its most general sense, to a method for processing the sound signal, according to claim 1.

Thus, thanks to the method according to the present invention, it is possible to capture the sound signal in all directions, then to restore said sound signal.

Advantageously, the matrix calculation involves a matrix H calculated by the method of least squares from the measured directivities of the N microphones and the ideal directivities of the ambisonic components.

According to one embodiment, said microphones are arranged in a circle on a plane, spaced at an angle equal to 360 ° / N or at each corner of a portable telephone.

According to one embodiment, said method uses four microphones spaced at an angle of 90 ° to the horizontal.

According to one embodiment, said method implements a filter bandpass filter from 100 Hz to 6 kHz.

According to one embodiment, the order R of the ambisonic type format is equal to one.

Advantageously, during said restitution step, information relating to the orientation of the head of a user listening to the sound signal is used.

Preferably, said information relating to the orientation of the head of a user listening to the sound signal is captured by a sensor in a mobile phone or by a sensor located in a headset or a virtual reality headset.

According to one embodiment, during said restitution step, the data in ambisonic format are transformed into data in binaural format.

The present invention also relates to a sound signal processing system according to claim 9.

Brief description of the drawings

• Les Figures 1 et 3 représentent les différentes étapes du procédé selon la présente invention ;
• La Figure 2 illustre les traitements appliqués dans le cadre de la seconde étape du procédé selon la présente invention ;
• Les Figures 4a, 4b et 4c représentent les composantes W, Y et X idéales d'un format ambisonique d'ordre 1 (sur plan horizontal) ;
• Les Figures 5a, 5b et 5c illustrent les composantes W, Y et X approximées d'un format ambisonique d'ordre 1 ; et
• La Figure 6 représente le placement de huit haut-parleurs virtuels, chacun placé à 45° autour d'un utilisateur.

The invention will be better understood with the aid of the description, given below for purely explanatory purposes, of an embodiment of the invention, with reference to the Figures in which:

• The Figures 1 and 3 represent the different steps of the method according to the present invention;
• The Figure 2 illustrates the treatments applied in the context of the second step of the method according to the present invention;
• The Figures 4a, 4b and 4c represent the ideal components W, Y and X of a first order ambisonic format (on a horizontal plane);
• The Figures 5a, 5b and 5c illustrate the approximate W, Y and X components of a first order ambisonic format; and
• The Figure 6 represents the placement of eight virtual speakers, each placed at 45 ° around a user.

Detailed description of the embodiments of the invention

The present invention relates to a method for processing the sound signal, according to claim 1.

The Figures 1 and 3 illustrate the various steps of the method according to the present invention.

In one embodiment, said microphones are arranged in a circle on a plane, spaced at an angle equal to 360 ° / N or at each corner of a portable telephone.

In one embodiment, the method according to the present invention uses four microphones spaced at an angle of 90 ° to the horizontal.

In one embodiment, the order R of the ambisonic type format is equal to one.

The first step of the method according to the present invention consists in recording the sound signal. For this recording, N microphones are used, N being a natural integer greater than or equal to three, said microphones being arranged in a circle on a plane, spaced at an angle equal to 360 ° / N or at each corner of a mobile telephone. In the embodiment described below, N is equal to four and the microphones are spaced 90 ° apart. These microphones are arranged in a circle on a plane. In a particular example of implementation, the radius of said circle is two centimeters, and the microphones are omnidirectional.

The sound signal is picked up by said microphones and digitized. It is a synchronous capture.

At the end of this first step, four sampled digital signals are obtained.

The second step of the method according to the present invention consists in encoding said four sampled digital signals, in an ambisonic type format of order R, R being a natural integer greater than or equal to one.

It is recalled here that the ambisonic format is a standardized format for audio coding in several dimensions.

In the embodiment described below, the order R is equal to one. This order 1 makes it possible to represent the sound with the following concepts: Front - Rear and Left - Right.

The Figures 4a, 4b and 4c represent the ideal components W, Y and X of an order 1 ambisonic format (on a horizontal plane).

The Figures 5a, 5b and 5c illustrate the approximate W, Y and X components of a first order ambisonic format.

The Figure 2 illustrates the treatments applied in the context of the second step of the method according to the present invention.

We observe on the Figure 2 that the input data are in the time domain, pass into the frequency domain following a fast Fourier transform operation (FFT or " Fast Fourier Transform" in English terminology), then the output data are in the time domain following an inverse fast Fourier transform operation (IFFT or " Inverse Fast Fourier Transform " in English terminology).

Preferably, Hanning windows are used with an overlap by implementing a “ overlap-add” type function .

We also observe on the Figure 2 that the input frequency data is modified using a matrix multiplication. This matrix includes weighting coefficients for each microphone signal and each frequency.

We also observe on the Figure 2 that filtering using a bandpass filter is performed on the data before output.

In one embodiment, the method according to the present invention implements a filter bandpass filter from 100 Hz to 6 kHz. This eliminates the lower part and the acute part.

In order to calculate the coefficients of the weighting matrix, the impulse responses of the N microphones are measured, in the present case of the four microphones, with a source positioned every 5 ° or every 10 ° around the microphone array.

Using a fast Fourier transform, the frequency responses of the N microphones are obtained as a function of the angles measured, or in other words the directivities of the N microphones as a function of the frequency.

It is possible to use at this stage the principles of the process described in the international application published under the number WO 2015/128160 " Automated acoustic equalization process and system" to equalize the frequency responses on the axis of each of the microphones. The same equalization filters are applied to all microphones and for all source angular positions.

The microphones' responses are then placed in a matrix C.

où N est le nombre de microphones (quatre dans le présent exemple de réalisation), D est le nombre de positions angulaires de source mesurées (108 dans le présent exemple de réalisation) et V est le nombre de canaux ambisoniques (trois dans le présent exemple de réalisation), C_DxN désigne les directivités des microphones, H_NxV désigne la matrice qui transforme les directivités des microphones en les directivités désirées, et P_DxV désigne les directivités prescrites par le format ambisonique (W, X et Y dans le présent exemple de réalisation).In the frequency domain, for each frequency index k, we have $${\mathrm{VS}}_{\mathrm{D}\times \mathrm{NOT}}.{\mathrm{H}}_{\mathrm{NOT}\times \mathrm{V}}={\mathrm{P}}_{\mathrm{D}\times \mathrm{V}}$$

where N is the number of microphones (four in the present embodiment), D is the number of angular source positions measured (108 in the present embodiment) and V is the number of ambisonic channels (three in the present example) C _DxN denotes the directivities of the microphones, H _NxV denotes the matrix which transforms the directivities of the microphones into the desired directivities, and P _DxV denotes the directivities prescribed by the ambisonic format (W, X and Y in the present example of production).

We thus have H _NxV = P _DxV / C _DxN for each frequency index k if C _DxN is invertible.

In practice, C _DxN is not invertible. In one embodiment, a least squares method is implemented to solve for C _108x4 . H _4x3 = P _108x3

The matrix H is defined once for the future uses of the network of microphones considered. Then, with each use, a matrix multiplication is carried out in the frequency domain.

Said matrix H has as many lines as there are microphones, therefore four in the present embodiment, and as many columns as required by the order of the ambisonic format used, therefore three columns in the present embodiment, in which the order 1 is implemented horizontally.

We have Out = In x H, where H denotes the previously calculated matrix, In denotes the input (audio channels coming from the network of microphones, passed in the frequency domain) and Out denotes the output (Out being reconverted in the temporal domain for get ambisonic format).

The method according to the present invention implements, during this second step, an algorithm called least squares algorithm for each frequency, with for example 512 frequency points.

At the end of this second step, data is obtained in the ambisonic format (in the present exemplary embodiment the signals W, X and Y).

The third step of the method according to the present invention consists in restoring the sound signal, by means of a transformation of the data in ambisonic format into two binaural channels.

During this third step, the information relating to the orientation of the head of the user listening to the sound signal is collected and used. This can be realized by a sensor in a mobile phone, a headset or a virtual reality headset.

This orientation information consists of a vector comprising three angle values, in Anglo-Saxon terminology "pitch","yaw" and "roll".

In the present embodiment, on a plane, the angle value "yaw" is used.

The ambisonic format is transformed into eight audio channels corresponding to a virtual placement of eight speakers, each placed at 45 ° around the user.

The Figure 6 represents the placement of eight virtual speakers, each placed at 45 ° around a user.

où W, X et Y sont les données relatives au format ambisonique, et où θ_n représente l'angle horizontal du nième haut-parleur. Par exemple, dans le présent exemple de réalisation θ₀ = 0°, θ₁ = 45°, θ₂ = 90°, etc.Each virtual speaker reproduces an audio signal from the ambisonic components according to the formula: $${\mathrm{P}}_{\mathrm{not}}=\mathrm{W}+{\mathrm{Xcos\theta }}_{\mathrm{not}}+{\mathrm{Ysin\theta }}_{\mathrm{not}}$$

where W, X and Y are the data relating to the ambisonic format, and where θ _n represents the horizontal angle of the n th loudspeaker. For example, in the present embodiment réalisation ₀ = 0 °, θ ₁ = 45 °, θ ₂ = 90 °, etc.

Then, a filtering step is carried out with a pair of HRTF per loudspeaker, HRTF meaning “ Head-related transfer function ” in English terminology. We pair a pair of HRTF filters (left ear and right ear) with each virtual speaker, then add (all channels "left ear" and all channels "right ear" together), to form two output channels .

IIR (“ Infinite Impulse Response ”) coefficients are implemented at this stage, said HRTF filters being modeled as IIR filters.

When the user turns his head, the position of the virtual speakers is changed. For example for a rotation of the head by an angle a, the angle of the virtual loudspeakers becomes β _n = θ _n -α. We then replace θ _n by (θ _n -α) in formula (1) to calculate the signal reproduced by the n ^th virtual speaker.

Thus, thanks to the method according to the present invention, it is possible to capture the sound signal in all directions, then to restore said sound signal.

The Figure 3 represents the different stages of the process according to the present invention.

The present invention also relates to a sound signal processing system according to claim 9.

This sound signal processing system includes at least one computing unit and one memory unit.

The invention is described in the foregoing by way of example. It is understood that a person skilled in the art is able to produce different variants of the invention without going beyond the scope of the patent.

Claims (9)

1. Sound signal processing method, comprising the following steps:

   - synchronously acquiring an input sound signal (S_entrée) by means of N microphones, N being a natural number greater than or equal to three;

   - encoding said input sound signal (S_entrée) in a sound data format (D), said encoding comprising a sub-step of transforming said input signal into an ambisonic-type format of order R, R being a natural number greater than or equal to one, said sub-step of transforming into an ambisonic-type format being carried out by means of a Fast Fourier Transform, a matrix multiplication, an Inverse Fast Fourier Transform and by means of a band-pass filter; and

   - delivering an output sound signal (S_sortie) by means of digitally processing said sound data (D);

   and characterized in that the matrix multiplication uses a matrix H calculated by the method of least squares from measured directivities of the N microphones and ideal directivities of the ambisonic components.
2. Sound signal processing method according to claim 1, characterised in that said microphones are positioned in a circle on a plane, spaced apart by an angle equal to 360°/N or at each corner of a mobile phone.
3. Sound signal processing method according to claim 2, characterised in that it implements four microphones spaced apart by an angle of 90° to the horizontal.
4. Sound signal processing method according to any of the previous claims, characterised in that it implements a band-pass filter filtering frequencies from 100 Hz to 6 kHz.
5. Sound signal processing method according to any of the previous claims, characterised in that the order R of the ambisonic-type format is equal to one.
6. Sound signal processing method according to any of the previous claims, characterised in that, during said delivering step, an information item relative to the orientation of the head of a user listening to the sound signal, is exploited.
7. Sound signal processing method according to claim 6, characterised in that acquisition of said information item relative to the orientation of the head of a user listening to the sound signal, is carried out by a sensor in a telephone, an audio headset or a virtual reality headset.
8. Sound signal processing method according to any of the previous claims, characterised in that, during said delivering step, the data in ambisonic format is transformed into data in binaural format.
9. Sound signal processing system, comprising means for:

   - synchronously acquiring an input sound signal (S_entrée) by means of N microphones, N being a natural number greater than or equal to three;

   - encoding said input sound signal (S_entrée) in a sound data format (D), and means for transforming said input signal into an ambisonic-type format of order R, R being a natural number greater than or equal to one, said means for transformation into an ambisonic-type format being carried out by means of a Fast Fourier Transform, a matrix multiplication, an Inverse Fast Fourier Transform and by means of a band-pass filter; and

   - delivering an output sound signal (S_sortie) by means of digitally processing the said sound data (D);

   and characterized in that the matrix multiplication uses a matrix H calculated by the method of least squares from measured directivities of the N microphones and ideal directivities of the ambisonic components.

Images