EP1695335A1

EP1695335A1 - Method for synthesizing acoustic spatialization

Info

Publication number: EP1695335A1
Application number: EP03819273A
Authority: EP
Inventors: Rozenn Nicol; David Virette; Marc Emerit
Original assignee: France Telecom SA
Current assignee: Orange SA
Priority date: 2003-12-15
Filing date: 2003-12-15
Publication date: 2006-08-30
Also published as: AU2003301502A1; US20070160216A1; WO2005069272A1; CN1886780A

Abstract

The invention relates to the synthesis and the joint spatialization of sounds emitted by virtual sources. According to the invention, a step (ETA) is provided that consists of determining parameters including at least one gain (gi) for defining, at the same time, a loudness characterizing the nature of the virtual source and the position of the source relative to a predetermined origin.

Description

Sound synthesis and spatialization

The present invention relates to the synthesis of audio signals, in particular in music editing applications, video games, or even ringtones for mobile phones.

More particularly, the invention relates to both sound synthesis techniques and three-dimensional (or "3D") sound techniques.

To offer innovative services, based on sound synthesis (to create ringtones, or even in the context of games on mobile phones), we are currently seeking to enrich sound synthesis methods. However, as the terminals are limited in terms of memory and computing power, it is preferable to develop methods which are both efficient and economical in complexity.

* Sound synthesis techniques

Many sound synthesis techniques have been developed in the past decades. It is pointed out that, in reality, there is no universal technique capable of generating any sound. Indeed, the production models that have been established so far each have their restrictions. A taxonomy established by Julius Smith in:

"Viewpoints on the History of Digi tal Synthesis", Smith JO; Keynote paper, Proc. Int. Comp. usic Conf. 1991, Montreal, is shown below. The techniques are categorized into four groups: - computational techniques (FM for "freguency modulation", "waveshaping" for working with waveforms, summer), "sampling" and other processing of recordings (for example a synthesis by wave table, ete), techniques based on spectral models (such as additive synthesis, or even the "source-filter", ete), techniques based on physical models (modal synthesis, by 'wave,...) .

Certain techniques, depending on their use, may fall into more than one category.

The choice of synthesis technique suitable for a terminal or a rendering system can be made according to three families of criteria, in particular criteria of the type proposed by the acoustics and signal processing laboratory. University of Helsinki in the context of an evaluation of the different synthesis methods: ".Evaluation of Modem Sound Synthesis Methods", Tolonen T., Vâlimâki V., Karjalainen M; Report 48, Espoo 1998.

A first family of criteria concerns the use of the following parameters: 1 intuitiveness, - perceptibility, physical sense and behavior.

The quality and diversity of the sounds that are produced determine the second family of criteria, according to the following parameters: robustness of the identity of the sound, extent of the sound palette, and with a preliminary analysis phase, if necessary.

Finally, the third family of criteria deals with implementation solutions, with parameters such as: the cost of calculations, the memory required, - control, latency and multitasking.

It has recently appeared that techniques based on spectral modeling (with reproduction of the spectral image perceived by a listener) or physical modeling (with simulation of the physical origin of sound) are the most satisfactory and have great potential for future systems.

However, currently, the methods based on the synthesis by wave table are the most widespread. The principle of this technique is as follows. First of all, all natural audio signals can be broken down into four phases: attack, decline, support and release, generally grouped under the term "ADSR envelope" (English: Attack, Decay, Sustain, Release) which ^'will be described later.

The principle of wave table synthesis consists of taking one or more signal periods (corresponding to a recording or a synthetic signal), then applying treatments to it (with looping, modification of the fundamental frequency, ete ) and finally to apply the above-mentioned ADSR envelope to it. This very simple synthesis method makes it possible to obtain satisfactory results. A technique similar to the synthesis by wave table is that called "sampling" which differs from it however by the fact that it uses natural signal recordings instead of synthetic signals.

Another example of simple synthesis is synthesis by frequency modulation, better known under the name of "FM synthesis". Here, a frequency modulation is carried out for which the frequency of the modulant and the modulated (f _m and f _c ) is in the audible range (20 to 20,000 Hz). It is also indicated that the respective amplitudes of the harmonics with respect to the fundamental mode can be chosen to define a timbre of the sound.

There are different formats for transmitting information to sound synthesizers. First of all, it is possible to transmit a musical score in MIDI formats or according to the MPEG4-Structured 'Audio standard so that it can then be synthesized by the chosen sound synthesis technique. In some systems, it is also possible to transmit information on the instruments to be used by the synthesizer, for example by using the DLS format which makes it possible to transmit the information necessary for the synthesis of sounds by wave table. Similarly, algorithmic languages of the "CSound" or "MPEG-4 SAOL" type make it possible to represent sounds as a sound synthesis technique.

The present invention relates to the combination of sound synthesis with the spatialization of the sounds resulting from this synthesis. We recall below some known sound spatialization techniques.

* Sound spatialization techniques

These are audio signal processing methods applied to the simulation of acoustic and psycho-acoustic phenomena. These techniques aim at the generation of signals to be broadcast on loudspeakers or on headphones, in order to give the listener the hearing illusion of sound sources placed at a predetermined position around him. They find an advantageous application in the creation of virtual sound sources and images.

Among the sound spatialization techniques, there are mainly two categories.

The methods based on a physical approach generally consist in reproducing the sound field 1 identical to the original sound field within an area of finite dimensions. These processes do not take into account a priori the perceptual properties of the auditory system, in particular in terms of auditory localization. With such systems, the listener is thus immersed in a field identical in every way to that which he would have perceived in the presence of real sources and he is therefore able to locate the sound sources as in a real listening situation.

The methods based on a psycho-acoustic approach rather seek to take advantage of the 3D sound perception mechanisms in order to simplify the process of sound reproduction. For example, instead of reproducing the sound field over an entire area, one can be content to reproduce it only at the level of the two ears of the listener. Similarly, one can impose a faithful reproduction of the sound field on only a fraction of the spectrum, in order to relax the constraint on the rest of the spectrum. The objective is to take into account the perception mechanisms of the auditory system in order to identify the minimum amount of information to reproduce, to obtain a psycho-acoustically identical field to the original field, ie such as the ear, due to the limitation of its performance, is unable to distinguish them from each other.

In the first category, different techniques have been identified: holophony, which is typically a technique of physical reconstruction of a sound field, since it constitutes the acoustic equivalent of holography. It consists of reproducing a sound field from a recording on a surface (hollow sphere, or other). Further details are given in: "spatial sound removal over a large area: Application to telepresence", R. Nicol; Thesis from the University of Maine, 1999; the surround technique (from English "ambisonic"), which is another example of physical reconstruction of the acoustic field, using a decomposition of the sound field on the basis of proper functions, called "spherical harmonixju.es".

In the second category, we identify for example: stereophony, which exploits differences in time or intensity to position the sound sources between two speakers, based on the interaural differences in time and intensity which define the criteria auditory of hearing localization in a horizontal plane; - binaural techniques which aim to reconstruct the sound field only at the level of the listener's ears, so that their eardrums perceive a sound field identical to that which the real sources would have induced.

Each technique is characterized by a specific method of encoding and decoding spatialization information in an adequate format of audio signals.

The different sound spatialization techniques are also distinguished by the extent of spatialization that they provide. Typically, 3D spatialization such as surround encoding, holophony, binaural or transaural synthesis (which is a transposition of the binaural technique on two distant speakers) includes all directions of space. Furthermore, two-dimensional ("2D") spatialization, such as stereophony, or a 2D restriction of holophony or ambisonic technique, is limited to the horizontal plane.

Finally, the different techniques are distinguished by their possible broadcasting systems, for example: - broadcasting on headphones for binaural techniques, or stereophony, - broadcasting on two speakers, in particular for stereophony or for a transaural system, - or a broadcast on a network with more than two speakers, for an extended listening area (in particular for multi-listener applications), in holophony, or in surround sound reproduction.

A wide range of current devices offers possibilities for sound synthesis. These devices range from musical instruments (such as a keyboard, a rhythm machine, or the like), mobile terminals, for example of the PDA type (for "Personal Digital Assistant"), or even computers on which are installed music editing software, or effects pedals with a MIDI interface. Sound reproduction systems (headphones, stereo speakers or multi-speaker systems) and the quality of sound synthesis systems are very varied, in particular according to the more or less limited computing capacities and according to the environments of use of such systems.

Systems are currently known capable of spatializing previously synthesized sounds, in particular by cascading a sound synthesis engine and a spatialization engine. Spatialization is then applied to the synthesizer output signal (on a mono channel or two stereo channels) after mixing the different sources. We thus know of implementations of this solution so as to spatialize the sounds coming from a synthesizer.

More generally known are implementations consisting of 3D rendering engines, which can be applied to any type of digital audio signals, whether synthetic or not. For example, the different musical instruments of a MIDI score (classic sound synthesis format) can then be positioned in the sound space. However, to obtain such spatialization, it is first necessary to convert the midi signals into digital audio signals and then to apply spatialization processing to the latter.

This implementation is particularly costly in terms of processing time and processing complexity. One of the aims of the present invention is a sound synthesis method offering the possibility of directly spatializing synthetic sounds.

More particularly, an object of the present invention is to associate with sound synthesis spatialization tools of satisfactory quality. However, this association combines the complexity due to sound synthesis with that of spatialization, which makes it difficult to implement spatialized sound synthesis on terminals with high constraints (that is to say computing power and with relatively limited memory size).

Another object of the present invention aims to optimize the complexity of the spatialization of synthetic sounds according to the capabilities of the terminal.

To this end, the present invention firstly proposes a method of sound synthesis and spatialization, in which a synthetic sound to be generated is characterized by the nature of a virtual sound source and by its position relative to a chosen origin.

The method within the meaning of the invention comprises a joint step consisting in determining parameters including at least one gain, in order to define at the same time:

- a sound intensity characterizing the nature of the source, and

- the position of the source with respect to a predetermined origin. It will thus be understood that the present invention makes it possible to integrate a sound spatialization technique with a sound synthesis technique, so as to obtain a global processing using common parameters for the implementation of the two techniques.

In one embodiment, the spatialization of the virtual source takes place in a surround context. The method then includes a step of calculating gains associated with surround components in a base of spherical harmonics.

In a variant, the synthetic sound is intended to be reproduced in a holophonic, or binaural, or transaural context, on a plurality of reproduction channels. It will be understood in particular that this "plurality of restitution channels" can as well relate to two restitution tracks, in binaural or transaural context, or even more than two restitution tracks, for example in holophonic context. During said joint step, a delay between restitution channels is also determined, to define at the same time:

- an instant of triggering of the sound characterizing the nature of the source, and - the position of the source with respect to a predetermined origin.

In this embodiment, the nature of the virtual source is configured at least by a temporal variation of sound intensity, over a chosen duration and including an instant of triggering of the sound. In practice, this time variation can advantageously be represented by an ADSR envelope as described above.

Preferably, this variation comprises at least: - an instrumental attack phase,

- a phase of decline,

- a support phase, and

- a relaxation phase.

Of course, more complex variations of the envelope can be envisaged.

The spatialization of the virtual source is preferably carried out by a binaural synthesis based on a linear decomposition of transfer functions, these transfer functions being expressed by a linear combination of terms depending on the frequency of the sound and weighted by terms depending on sound direction. This measurement proves to be advantageous in particular when the position of the virtual source is liable to change over time and / or when several virtual sources are to be spatialized.

Preferably, the direction is defined by at least one azimuthal angle (for spatialization in a single plane) and, preferably, by an azimuthal angle and an elevation angle (for three-dimensional spatialization).

In the context of a binaural synthesis based on a linear decomposition of transfer functions, the position of the virtual source is advantageously configured at least by: several filterings, functions of the sound frequency, several weighting gains each associated with a filtering, and - a delay by "left" and "right" channel.

Preferably, the nature of the virtual source is parameterized at least by a sound timbre, by associating selected relative sound intensities with harmonics of a frequency corresponding to a pitch of the sound. In practice, this modeling is advantageously carried out by an FM synthesis, described above.

In an advantageous embodiment, there is provided a sound synthesis engine capable of generating spatialized sounds, with respect to a predetermined origin.

Preferably, the synthesis engine is implemented in the context of musical editing, and a man / machine interface is also provided for placing the virtual source at a chosen position relative to the predetermined origin.

To synthesize and spatialize a plurality of virtual sources, each source is assigned to a respective position, preferably using a linear decomposition of the transfer functions in binaural context, as indicated above.

The present invention also relates to a module for generating synthetic sounds, comprising in particular a processor, and comprising in particular a working memory capable of storing instructions for the implementation of the above method, so as to simultaneously process a synthesis and a spatialization of the sound, according to one of the advantages which the present invention provides.

As such, the present invention also relates to a computer program product, stored in a memory of a central unit or of a terminal, in particular mobile, or on a removable medium suitable for cooperating with a reader of said central unit, and comprising instructions for implementing the above process.

Other characteristics and advantages of the invention will appear on examining the detailed description below, and the attached drawings in which:

FIG. 1 schematically illustrates positions of sound sources i and positions of microphones j in three-dimensional space, FIG. 2 schematically represents a simultaneous spatialization and sound synthesis processing, within the meaning of the invention,

- Figure 3 schematically represents the application of HRTFs transfer functions to Si signals for spatialization in binaural or transaural synthesis, Figure 4 schematically represents the application of a pair of delays (a delay by left or right channel) and several gains (one gain per directional filter) in binaural or transaural synthesis, using the linear decomposition of HRTFs, - Figure 5 schematically represents the integration of spatialization processing, within a plurality of synthetic sound generators, for spatialization and sound synthesis in a single step, - Figure 6 represents an ADSR envelope model in sound synthesis,

- And Figure 7 shows schematically a sound generator in FM synthesis.

It will be recalled that the present invention proposes to integrate a technique of spatialization of sound with a technique of sound synthesis so as to obtain a global, optimized processing, of spatialized sound synthesis. In the context of highly constrained terminals, the pooling of certain sound synthesis operations, on the one hand, and sound spatialization, on the other hand, is particularly interesting.

Generally, a sound synthesis engine (typically a "synthesizer") has the role of generating one or more synthetic signals, on the basis of a sound synthesis model, a model which is controlled from a set of parameters, hereinafter called "synthesis parameters". The synthetic signals generated by the synthesis engine can correspond to distinct sound sources (which are, for example, the different instruments of a score) or can be associated with the same source, for example in the case of different notes of the same instrument. Thereafter, the terms "tone generator" designate a module for producing a musical note. Thus, it will be understood that a synthesizer is composed of a set of tone generators.

Still generally, a sound spatialization tool is a tool which admits a given number of audio signals as input, these signals being representative of sound sources and, in principle, free of spatialization processing. It is in fact indicated that, if these signals have already undergone spatial processing, this prior processing is not taken into account here. The role of the spatialization tool is to process the input signals, according to a diagram which is specific to the spatialization technique chosen, to generate a given number of output signals which define the spatialized signals representative of the sound scene in format of spatialization chosen. The nature and the complexity of the spatialization processing obviously depend on the technique chosen, depending on whether one considers a rendering in stereophonic, binaural, holophonic or ambiophonic format.

More particularly, for many spatialization techniques, it appears that the processing is reduced to an encoding phase and a decoding phase, as will be seen below.

The encoding corresponds to the sound recording of the sound field generated by the different sources at a given time. This "virtual" sound recording system can be more or less complex depending on the sound spatialization technique used. So, we simulate a sound recording by a number more or less important microphones with different positions and directivities. In all cases, the encoding is reduced, to calculate the contribution of a sound source, at least to the application of gains and, more often than not, delays (typically in holophony or in binaural or transaural synthesis) to different copies of the signal from the source. There is one gain (and, if necessary, a delay) per source for each virtual microphone. This gain (and delay) depends on the position of the source relative to the microphone. If a virtual sound pickup system provided with K microphones is provided, there are K signals output from the encoding system.

Referring to FIG. 1, the signal Ej represents the sum of the contributions of all the sound sources on the microphone j. In addition, we call: If the sound emitted by the source i, - Ej the signal encoded at the output of the microphone j, Gji the attenuation of the sound Si due to the distance between the source i and the microphone j, to the directivity of the source, at the obstacles between the source i and the microphone j, and finally at the very directivity of the microphone j, tji the delay of the sound Si due to the propagation from the source i towards the microphone j, and - x, y, z the Cartesian coordinates of the position of the source, assumed to be variable over time.

The encoded signal Ej is given by the expression: L Ej (t) = ∑S (t- tji (x, y, z)) * Gβ (x, y, z) Si (t) ι = l In this expression, we assume that we must process L sources (i = 1, 2, ..., L), while the encoding format provides for K signals (j = 1, 2, ..., K ). The gains and the delays depend on the position of the source i relative to the microphone j at the instant t. The encoding is therefore a representation of the sound field generated by the sound sources at this instant t. It is simply recalled here that in a surround context (consisting of a decomposition of the field in a base of spherical harmonics), the delay does not really intervene in the spatialization processing.

In the case where the sound sources are in a room, the image sources must be added. These are the images of the sound sources reflected by the walls of the room. Image sources, in turn reflecting on the walls, generate image sources of higher order.

In the above expression, L therefore no longer represents the number of sources, but the number of sources to which the number of image sources is added. The number of image sources is infinite, which is why, in practice, we keep only audible image sources and whose direction we perceive. Image sources that are audible but whose direction is no longer perceived are grouped together and their contribution is synthesized using an artificial reverberator.

The decoding step aims to restore the signals E _j encoded on a given device, comprising a predetermined number T of sound transducers (headphones, loudspeaker, 5/069272

19

speaker). This step consists in applying a TxK matrix of filters to the encoded signals. This matrix depends only on the rendering device, and not on the sound sources. Depending on the encoding and decoding technique chosen, this matrix can be very simple (for example identity) or very complex.

There is schematically shown in Figure 2 a flowchart showing the different steps above. A first step ST constitutes a start-up step during which a user defines sound commands Ci, C ₂ , ..., C _N to be synthesized and spatialized (for example by providing a man / machine interface to define a musical note , an instrument to play this note and a position of this instrument playing this note in space). As a variant, for example for the spatialization of sound with a mobile terminal, the spatialization information can be transmitted in a stream parallel to the synthetic audio stream, or even directly in the synthetic audio stream.

Then, it is indicated that the invention advantageously provides for a single ETA step of synthesis and spatialization of the sound, jointly. In general, a sound can be defined at least by: the frequency of its fundamental mode, characterizing the pitch, its duration, and its intensity. So, in the example of a sensitive keyboard synthesizer, if the user plays a loud note,

The intensity associated with the Ci command will be greater than the intensity associated with a piano note. More particularly, it is indicated that the intensity parameter can, in general, take into account the spatialization gain gi in a context of spatialization processing, as will be seen below, according to one of the major advantages which it provides. the present invention.

In addition, a sound is, of course, also defined by its triggering instant. Typically, if the spatialization technique chosen is not a surround treatment, but rather binaural or transaural synthesis, holophony, or the like, the spatialization delay τ _± (which will be described in detail below) can make it possible to control in addition the instant of triggering of the sound.

Referring again to FIG. 2, a sound synthesis and spatialization device Dl comprises: - a synthesis module proper Ml, capable of defining, as a function of a command Ci, at least the frequency fi and the duration Di of the sound i associated with this command Ci, and a spatialization module M2, capable of defining at least the gain gi (in surround context in particular) and, moreover, the spatialization delay Ti, in holophony or binaural or transaural synthesis. As indicated above, these last two parameters g _± and Xi can be used jointly for the spatialization, but also for the synthesis of the sound itself, when a sound intensity (or a pan in stereophony) and a triggering moment are defined. sound.

More generally, it is indicated that in a preferred embodiment, the two modules M1 and M2 are grouped together in the same module making it possible to define in a single step all the parameters of the signal if to be synthesized and spatialized: its frequency, its duration, its gain in spatialization, its delay in spatialization, in particular.

These parameters are then applied to an encoding module M3 of the sound synthesis and spatialization device D1. Typically, for example in binaural or transaural synthesis, this module M3 performs a linear combination on the signals si which in particular involves the spatialization gains, as will be seen below. This encoding module M3 can also apply compression encoding to the signals Si to prepare a transmission of the encoded data to a restitution device D2.

However, it is indicated that this encoding module M3 is, in a preferred embodiment, directly integrated into the modules Ml and M2 above, so as to create directly, within a single module Dl which would simply consist of a motor sound synthesis and spatialization, the signals Ej as if they were delivered by microphones j, as explained above. Thus, the sound synthesis and spatialization engine Dl produces, at the output, K sound signals Ej representing the encoding of the virtual sound field that the different synthetic sources would have created if they had been real. At this stage, we have a description of a sound scene in a given encoding format.

Of course, provision may also be made to add (or "mix") to this sound scene other scenes coming from an actual sound recording or from the output of other sound processing modules, provided that they are in the same spatialization format. The mixing of these different scenes then passes through a single and unique decoding system M '3, provided at the input of a reproduction device D2. In the example shown in Figure 2, this rendering device

D2 includes two channels, here for a binaural reproduction (reproduction on stereo headphones) or transaural (reproduction on two speakers) on two channels L and R.

A preferred embodiment of the invention is described below, here applied to a mobile terminal and in the context of sound spatialization by binaural synthesis.

On telecommunication terminals, especially mobile, it naturally provides a sound rendering with a stereo headset. The preferred sound source positioning technique is then synthesis binaural. It consists, for each sound source, in filtering the monophonic signal by acoustic transfer functions called HRTFs (for "Head Related Transfer Functions"), which model the transformations generated by the torso, the head and the flag of the listener to the signal from a sound source. For each position in space, we can measure a pair of these functions (a function for the right ear, a function for the left ear). The HRTFs are therefore functions of the position [θ, φ] (where θ represents the azimuth and φ the elevation) and the sound frequency f. We then obtain, for a given subject, a database of 2M acoustic transfer functions representing each position of space for each ear (M being the number of directions measured). Conventionally, the implementation of this technique is done in a so-called "two-channel" form.

Another binaural synthesis, based on a linear decomposition of HRTFs, corresponds to an implementation which proves to be more effective in particular when several sound sources are spatialized, or in the case where the sound sources change position over time. In this case, we speak of "dynamic binaural synthesis".

These two embodiments of binaural synthesis are described below. * Binaural "Jbicanale" synthesis

Referring to FIG. 3, the two-channel binaural synthesis consists in filtering the signal of each source S (i = 1, 2, ..., N) which one wishes to position in space at a position [θi, φi], by the acoustic transfer functions left (HRTF_1) and right (HRTF__r) corresponding to the appropriate directions [θi, φi] (step

31). Two signals are obtained which are then added to the left and right signals resulting from the spatialization of the other sources (step 32), to give the L and R signals broadcast to the subject's left and right ears with a stereo headset.

It is indicated that, in this implementation, the positions of the sound sources are not expected to change over time. However, if one wishes to vary the positions of the sound sources in space over time, it is preferable to modify the filters used to model the left and right HRTFs. On the other hand, these filters being in the form of filters, either with finite impulse response (FIR), or with infinite impulse response (IIR), problems of discontinuities of the left and right output signals appear, causing audible "clicks" . The technical solution used to overcome this problem is to make, rotate two sets of binaural filters in parallel. The first game simulates the first position [θl, φl] at an instant tl, the second the second position [Θ2, φ2] at an instant t2. The signal giving the illusion of a displacement between the first and second positions is then obtained by a crossfade of the left and right resulting from the first and second filtering process. Thus, the complexity of the sound source positioning system is then multiplied by two compared to the static case. In addition, the number of filters to be implemented is proportional to the numbers of sources to be spatialized.

If N sound sources are considered, the number of filters required is then 2. N for a static binaural synthesis and 4 .N for a dynamic binaural synthesis.

An advantageous variant is described below. * Binaural synthesis based on a linear breakdown of HRTFs We first indicate that such an implementation has a

"complexity which no longer depends on the total number of sources to be positioned in space. Indeed, these techniques make it possible to decompose HRTFs on the basis of orthogonal functions, common to all the positions of space, and therefore no longer depend frequency f. This reduces the number of filters required. More specifically, the number of filters is fixed and no longer depends on the number of sources to be positioned, so that adding an additional sound source only requires the application of a delay, with then an operation of multiplication by several gains depending only on the position [θ, φ] and an operation of addition, as we will see with reference to Figure 4. These techniques of linear decomposition also have a interest in the case of dynamic binaural synthesis (position of sound sources variable over time). Indeed, in this case, the coefficients of the filters are no longer varied, but only the values of the gains which are a function of the position.

The linear decomposition of HRTFs aims to separate the spatial and frequency dependencies of the transfer functions. Beforehand, the excess phase of the HRTFs is extracted, then modeled in the form of a pure delay τ. The linear decomposition then applies to the minimum phase component of the HRTFs. Each HRTF is written as a sum of P spatial functions Cj (θ, φ) and reconstruction filters Lj (f): HRTF (Θ, φ, f) = ex _V (j2πfτ (θ, φ)) ∑C, (θ, φ) L _} (f) (1)

The implementation scheme of binaural synthesis based on a linear decomposition of HRTFs is illustrated in Figure 4. The interaural delays τi (step 41) associated with the different sources are first applied to the signal of each source to be spatialized Si ( with i = l, ..., N). The signal from each source is then decomposed into P channels corresponding to the P basic vectors of the linear decomposition. To each of these channels are then applied the directional coefficients Cj (θ _± , φi) (denoted C) resulting from the linear decomposition of HRTFs

(step 42). These spatialization parameters τi and Ci have the particularity of depending only on the position

[θi, φi] where we want to place the source. They do not depend on the sound frequency. For each source, the number of these coefficients corresponds to the number P of the basic vectors that we used for the linear decomposition of HRTFs.

For each channel, the signals from the N sources are then added (step 43) then filtered (step 44) by the filter Lj (f) corresponding to the j ^x th ^base vector.

The same scheme is applied separately for the right and left channels. The figure distinguishes the delays applied on the left (τ _L i) and right (τ _R i) channels, as well as the directional coefficients applied on the left (Ci, j) and right (-Di, ₇ ) channels. Finally, the signals summed and filtered in steps 44 and 45 are summed again (step 45 in FIG. 4), as in step 32 in FIG. 3, for reproduction on stereophonic headphones. It is indicated that steps 41, 42 and 43 may correspond to the spatial encoding proper, for binaural synthesis, while steps 44 and 45 may correspond to a spatial decoding before restitution, which the module M '3 of Figure 2, as described above. In particular, the signals coming from the summers after step 43 of FIG. 4 can be conveyed via a communication network, for spatial decoding and restitution with a mobile terminal, in steps 44 and 45 described above.

The advantage of this implementation is that, unlike the binaural "Jicanale" synthesis, the addition of an additional source does not require the addition of two additional filters (FIR or IIR type). In other words, P basic filters are shared by all the sources present. In addition, in the case of dynamic binaural synthesis, it is possible to vary the coefficients Cj (θi, φi) without causing audible clicks at the output of the device. In this case, only 2. P filters are necessary, whereas 4.N filters were necessary for the dynamic two-channel implementation described above.

In other words, the delays τ and the gains C and D, which constitute the spatialization parameters and are specific to each sound source as a function of its position, can therefore be dissociated from the directional filters L (f) in setting work of binaural synthesis based on a linear decomposition of HRTFs. Consequently, the directional filters are common to the N sources, regardless of their position, their number or their possible displacement. The application of the spatialization parameters then represents the spatial encoding, properly speaking, of the signals relating to the sources themselves, while the directional filters carry out the effective processing of spatial decoding, with a view to restitution, which no longer depends on the position of the sources, but of the sound frequency.

Referring to FIG. 5, this dissociation between the spatialization parameters and the directional filters is advantageously exploited by integrating the application of the spatialization delay and gain in the sound synthesizer. Sound synthesis and spatial encoding (delays and gains) controlled by the azimuth and the elevation are thus carried out simultaneously within the same module such as a sound generator, for each sound signal (or note, in musical edition) to be generated (step 51). The spatial decoding is then taken care of by the directional filters Li (f), as indicated above (step 52).

We will now describe, with reference to FIGS. 6 and 7, stages of the generation of signals in sound synthesis. In particular, FIG. 6 represents the main parameters of an ADSR envelope of the aforementioned type, commonly used in different sound synthesis techniques. In particular, FIG. 6 represents the temporal variation of the envelope of a synthesized sound signal, for example a note played on a piano, with: an attack parameter, modeled by an ascending ramp 61, corresponding for example to the duration of a hammer hammering against a piano string, - a decline parameter, modeled by a descending ramp 62, with strong decay, corresponding for example to the duration of a hammer release from a string piano, - a support parameter (free vibration), modeled by a slightly descending ramp 63, due to natural acoustic damping, corresponding for example to the duration of a sound of a pressed piano key, and a parameter release, modeled by a descending ramp 64, corresponding for example to the rapid acoustic damping produced by a felt on a piano string. Of course, more complex variations of the envelope can be envisaged, comprising for example more than four phases.

It is indicated however that most of the synthesized sounds can be modeled by a variation of envelope as described above. Preferably, the parameters of the ADSR envelope are defined before performing the filters provided for the spatialization processing, due to the time variables involved.

It will thus be understood that the maximum of the sound amplitude (in arbitrary units in FIG. 6) can be defined by the spatialization processing, in correspondence then to the gains dj and Dij mentioned above, for each left and right channel. Similarly, the instant of triggering of the sound (start of the ramp 61) can be defined through the delays τ _L i and τ _R i.

We now refer to FIG. 7 in which a simple operator of sound synthesis by frequency modulation ("FM synthesis") has been shown. We initially define a carrier frequency f _c (typically the frequency of the fundamental mode), which defines for example the tone of a musical note. One then uses one or more oscillators OSCl to define one or more harmonics f _m (corresponding in principle to frequencies multiple of the carrier frequency f _c ), with which are associated relative intensities I _m . Through for example, the intensities I _m , compared to the intensity of the fundamental mode, are higher for a metallic sound (such as that of a new guitar string). Generally speaking, FM synthesis makes it possible to define the timbre of a synthesized sound. The signals (sinusoids) coming from the oscillator (s) OSCl are added to the signal drawn from the carrier frequency f _c by the module AD, which delivers a signal to an output oscillator OSC2 which receives the amplitude A _c of the sound with reference. at the carrier frequency f _c . Here again, it is indicated that this setpoint A _c can be directly defined by the spatialization processing, through the gains C and D (in binaural synthesis), as we have seen above. Finally, the oscillator OSC2 delivers a signal S'i, to which an ADSR envelope of the type shown in FIG. 6 is then applied, as well as a pair of delays τ _L i and τ _R i and several gains Cij and Dij, respectively for each left and right channel, as shown in FIG. 4, and to finally obtain a signal such as one of the signals delivered by the sound generators of FIG. 5.

It will thus be understood that such a measure makes it possible to avoid, in a particularly advantageous manner, from generating, from a score in MIDI format, the sounds in a standard format of audio reproduction (for example in "wave" format) and to encode them again for a spatialization of sound, as in known implementations.

The present invention makes it possible to directly implement both the spatialization steps and sound synthesis. It will be understood in particular that any sound synthesis processing, requiring the definition of an intensity (and, where appropriate, an instant of triggering of the sound), can be carried out in conjunction with a spatialization processing, proposing a gain (and, the delay, if applicable) by return.

Generally speaking, a sound synthesizer works by reading a score which gathers information on the instruments to be synthesized, the moments when the sounds should be played, the pitch of these sounds, their strength, etc. When reading this partition, a sound generator is associated with each sound, as indicated above with reference to FIG. 5.

We first consider the case where the same source plays several notes simultaneously. These notes, which come from the same source, are spatialized at the same position and therefore with the same parameters. It is therefore preferred to group the spatialization processing for the sound generators associated with the same source. Under these conditions, the signals associated with the notes from the same source are preferably summed beforehand so as to apply the spatialization processing globally to the resulting signal, which, on the one hand, advantageously reduces the cost of implementation. and, on the other hand, advantageously guarantees the coherence of the sound scene.

Additionally, gains and delays can be applied by taking advantage of the synthesizer structure. On the one On the other hand, the delays (left channel and right channel) of spatialization are implemented in the form of delay lines. On the other hand, within the framework of the synthesizer, the delays are managed by the instants of triggering of the sound generators in agreement with the partition. In the context of a spatialized sound synthesis, the two previous approaches (delay line and control of the triggering instant) are combined in order to optimize the processing. One saves therefore one delay line per source, by playing on the instants of triggering of the sound generators. For this purpose, we extract the difference between the delays of the left channel and the right channel for spatialization. It is then planned to add the lower of the two delays at the time of triggering of the generator. It then remains to apply the time difference between the left and right channels to only one of the two channels, in the form of a delay line, it being understood that this difference in delay can take both positive and negative values.

As far as gains are concerned, the balance (or "pan") parameter, which is typically associated with the stereophonic system, no longer needs to be. It is therefore possible to delete the gains associated with the balance. In addition, the sound generator volume parameter can be applied at the level of the different gains corresponding to the spatial encoding, as described above.

It is further indicated that the present invention makes it possible to apply sound spatialization, source by source, the fact that the spatialization tool is integrated into the heart of the sound synthesis engine. This is not the case if we proceed on the contrary by simply cascading the synthesis engine and the spatialization tool. In this case, in fact, it is recalled that the spatialization can only be applied globally to the entire sound scene.

According to another advantage which the present invention provides, the sound synthesis and spatialization tools can be judiciously combined, in order to achieve an optimized implementation of a spatialized sound synthesis engine, with, in particular, an optimization of the combination of synthesis and spatialization operations, taking into account in particular at least one gain and / or a spatialization delay, or even a spatialization filter.

In the case where the synthesis parameters already apply one or more of these parameters (gain, delay, filter), the spatialization parameters are advantageously taken into account by simple modification of the synthesis parameters, without modification of the synthesis model itself. -even.

Furthermore, by simply adding a gain and a delay to the sound synthesis engine, possibly supplemented by a filter, a spatialized sound synthesis, based on different possible spatialization techniques, can be obtained. These spatialization techniques (binaural / transaural synthesis, holophony, surround sound, summer) can be of variable complexity and performance but overall offer a much richer and more complete spatialization than stereophony, with in particular a natural and particularly immersive rendering of the sound scene. Indeed, the sound spatialization within the meaning of the invention retains the full potential of three-dimensional sound rendering, in particular in terms of immersion, with true 3D spatialization.

Of course, provision can also be made for integration of the spatialization and room effect processing, in the simplified form of at least one gain and / or of a delay (possibly supplemented with filters), and of a reverberator. artificial for late reverberation.

Claims

claims

1. A method of sound synthesis and spatialization, in which a synthetic sound to be generated is characterized by the nature of a virtual sound source and by its position relative to a chosen origin, characterized in that it comprises a joint step consisting to determine parameters including at least one gain, to define at the same time: - a sound intensity characterizing the nature of the source, and - the position of the source with respect to a predetermined origin.

2. Method according to claim 1, in which the spatialization of the virtual source takes place in a surround context, characterized in that it comprises a step of calculating gains associated with surround components in a base of spherical harmonics.

3. Method according to claim 1, in which the synthetic sound is intended to be reproduced in a holophonic, or binaural, or transaural context, on a plurality of reproduction channels, characterized in that, during said joint step, it is further determined a delay between restitution channels, to define at the same time:

4. Method according to claim 3, characterized in that the nature of the virtual source is configured at least by a temporal variation of sound intensity, over a chosen duration and including an instant of triggering of the sound.

5. Method according to claim 4, characterized in that said variation comprises at least: - an instrumental attack phase,

- a phase of decline,

- a support phase, and

- a relaxation phase.

6. Method according to one of claims 3 to 5, characterized in that the spatialization of the virtual source is carried out by a binaural synthesis based on a linear decomposition of transfer functions, these transfer functions being expressed by a combination linear of terms depending on the frequency of sound (L (f)) and weighted by terms depending on the direction of sound (τ _{R /} τ _L , C, D).

7. Method according to claim 6, characterized in that the direction is defined by at least one azimuth angle

(θ) and preferably by an azimuth angle (θ) and an elevation angle (φ).

8. Method according to one of claims 6 and 7, characterized in that the position of the virtual source is parameterized at least by: several filterings, functions of the sound frequency (Li (f)), several weighting gains each associated with a filtering, and a delay by "left" and "right" channels.

9. Method according to one of the preceding claims, characterized in that the nature of the virtual source is configured at least by a sound timbre, by associating chosen relative sound intensities with harmonics of a frequency corresponding to a pitch of the his.

10. Method according to one of the preceding claims, characterized in that it provides a sound synthesis engine capable of generating spatialized sounds, with respect to said predetermined origin.

11. The method of claim 10, wherein the synthesis engine is implemented in the context of music editing, characterized in that the method further provides a man / machine interface for placing the virtual source at a chosen position relative to the 'predetermined origin.

12. The method of claim 11, taken in combination with claim 6, wherein there is provided a plurality of virtual sources to synthesize and spatialize, characterized in that each source is assigned to a respective position.

13. Synthetic sound generation module, comprising in particular a processor, characterized in that it further comprises a working memory suitable for storing instructions for the implementation of the method according to one of the preceding claims.

14. Computer program product, stored in a memory of a central unit or of a terminal, in particular mobile, or on a removable medium suitable for cooperating with a reader of said central unit, characterized in that it comprises instructions for implementing the method according to one of claims 1 to 12.