EP1869949A1

EP1869949A1 - Method and system for spatializing an audio signal based on its intrinsic qualities

Info

Publication number: EP1869949A1
Application number: EP06743580A
Authority: EP
Inventors: Jean-Philippe Thomas
Original assignee: Charlet Delphine; Collet Mikael; France Telecom SA
Current assignee: Charlet Delphine; Collet Mikael; Orange SA
Priority date: 2005-03-15
Filing date: 2006-03-15
Publication date: 2007-12-26
Also published as: WO2006097633A1

Abstract

The invention concerns a method and a system for spatializing an audio signal (SS), which consists in: subdividing (A) the audio signal into successive frames (TSSi), analyzing (B) the frames by frame blocks to determine at least one spectral and/or physiological parameter (Pspj) in each frame, allocating (C) to each frame an index (j,k) representing said parameter to generate a sequence of classified frames and subjecting (D) each frame (F) attributed a common index (j,k), to a spatializing processing (Sj,k). The invention is useful for spatializing monophonic, stereophonic, speech, telephone signals and the like.

Description

METHOD AND SYSTEM FOR SPATIALIZING A SOUND SIGNATURE BASED ON THE INTRINSIC QUALITY OF THE SAME

At present, almost all sound exchanges are monophonic. The sound exchanges essentially concern the emission and reception of sound signals, directly perceptible to the human ear. These include sound signals provided by radio, telephone, messaging, television and others.

Among the qualities recognized by a sound signal, that relating to the relief or the spatial rendering of the latter is the most sought after. To spatialize a signal or sound content, the current techniques consist of either making a spatialized sound recording in the studio, or performing a non-real-time and manual studio treatment, in particular for the spatialization of movie soundtracks.

As far as spatialized sound is concerned, these methods, based on a physical approach, consist, in general, in reproducing the identical sound field of the original sound field within a zone of finite dimensions. Thanks to these processes, the listener is immersed in a field identical in every way to that which he would have perceived in the presence of real sources and the latter is thus able to locate the sound sources as in real listening situation. For a more detailed description of this type of technique, in particular of holophony, reference may be made to the work entitled "Spatial sound reproduction over a wide area: Application to telepresence" R. Nicol; University of Maine thesis, 1999.

Another example of physical reconstruction of the acoustic field is known as ambisonic technique, which uses a decomposition of the sound field on the basis of eigenfunctions called "spherical harmonics".

The known technique of stereophony, in turn, exploits the differences in propagation time or loudness to position the sound sources between two or more speakers, from interaural differences in time and of intensity that define the perceptual criteria of auditory localization in a substantially horizontal plane.

The binaural techniques, finally, aim to reconstruct the acoustic field only in the vicinity of the listener's ears, so that the eardrums of the latter perceive an acoustic field substantially identical to that which would have been generated by real sources.

The object of the present invention is to overcome the drawbacks of the techniques of the prior art, in order to allow the widest application of spatialization techniques to any monophonic and / or stereophonic sound signal, or even to a more complex sound signal.

In particular, the subject of the present invention is a method for spatialising a sound signal that is remarkable in that it consists at least in subdividing this sound signal into successive frames, analyzing the frames of the sound signal in blocks of frames to determine at least a spectral and / or physiological parameter of this sound signal in each frame, assigning each frame an index representative of at least this spectral and / or physiological parameter, to generate a series of classified frames and to submit each frame or group of classified frames , assigned the same index, to a spatialization processing, depending on the value of this index assigned to each frame.

The subject of the present invention is furthermore a system for spatializing a remarkable sound signal in that it comprises at least, in combination, a module for analyzing this sound signal in successive frames, to determine and assign to each frame successively an index representative of at least one spectral and / or physiological parameter of said sound signal in each frame, to generate a sequence of indexes of classified frames and a processing module of each frame or group of frames of the sound signal, classified and assigned the same index, according to a spatialization processing, depending on the value of the assigned index. The invention is applicable to the electronics industry for processing sound signals, in particular stereo or mono-phonic signals, recording phonograms and / or videograms, to the fixed telephony industry. or mobile, to voice communication and / or transmission of speech signals over the IP network. It will be better understood from reading the description and from the following drawings in which:

FIG. 1a represents, purely by way of illustration, a flowchart of the essential steps for implementing the method that is the subject of the present invention; FIG. 1b represents, for purely illustrative purposes, a method of classifying acoustic frames according to acoustic classes, according to a remarkable aspect of the method, object of the present invention;

FIGS. 2a and 2b represent, by way of illustration, a nonlimiting variant of implementation of the method which is the subject of the present invention applied in the deferred mode, either to the recording and restitution, or to the partial offline diffusion, or still the real-time online communication of phonograms and / or the sound portion of videograms, according to the method object of the present invention;

FIG. 3 a represents, by way of illustration, a spatialization system of a sound signal, in accordance with the subject of the present invention, operating in partial offline diffusion mode;

FIG. 3b represents, by way of illustration, a system for spatializing a sound signal in online communication mode, more particularly intended for devices of the telephone or other type.

A more detailed description of the method of spatialization of a sound signal according to the intrinsic qualities of this signal, in accordance with the subject of the present invention, will now be given in connection with FIG. 1a and FIG.

With reference to FIG. 1a, it is indicated that the method which is the subject of the invention is implemented on a sound signal SS, this sound signal corresponding for example to a digital signal, digital audio, speech or telephony, for example. The method which is the subject of the invention consists, in the first place, in a step A in subdividing the sound signal SS into successive frames. The subdivision operation is noted

SS → [TSSiJ.

The aforementioned subdivision step can be executed in a conventional manner by subdividing the signal into successive frames with a duration of between 10 and 20 milliseconds, for example, typically 16 milliseconds. A sampling of the sound signal is thus performed to obtain a succession of contiguous frames for example, i denoting the rank of the frame. Typical values are frames with a duration of 32 milliseconds, calculated every 16 milliseconds, thus with an overlap of 16 milliseconds. Step A is followed by a step B of analyzing a block of frames of the sound signal TSS, to determine at least one spectral and / or physiological parameter of the sound signal in each frame.

The operation of step B of analyzing each frame is represented by the relation

where [TSSi + denotes the block of frames and

[PspjJo denotes a set of spectral and / or physiological parameters resulting from the analysis.

By spectral and / or physiological parameter of the sound signal, it is indicated that this or these parameters may correspond for example to a frequency parameter for a musical signal, a formant for a speech signal or the like, as will be described later in the description. It is understood in particular that several significant parameters can be associated with each frame in particular when the sound signal

SS is a signal that includes both music and song lyrics, for example, or in other situations. The number of spectral and / or physiological parameters assigned to each frame is not limiting, as will be described later in the description.

Step B is then followed by a step C of assigning to each frame TSSj an index representative of at least one spectral or physiological parameter to generate a sequence of classified frames.

In FIG. 1a, the step C of assigning an index is represented by the symbolic relation

[PspJo -> TSSio.k). The operation B of frame analysis and C of index assignment work on one. block of frames i-K_before, i + K_after. This block definition is generic and the online mode is K_after = 0.

In general, and in accordance with a remarkable aspect of the method, object of the present invention, the notion of index assigned to each frame corresponds for example to an index j assigned a sub-index k, the subindex k allowing, for example, to specify classification variants for the main classification j assigned to each frame.

Step C thus makes it possible to obtain a fine classification of each frame as a function of the spectral and / or physiological parameter or parameters of the sound signal SS in the frame considered TSSi.

Step C is then followed by a spatialization step D itself, this step consisting in submitting each classified frame or group of frames of the same index and noted to the same spatialization processing function of the value of the aforementioned index assigned to the frame in question.

In step B of figure la the spatialization operation is noted by the symbolic relation

Sj, k (TSSio, k)). I

In the previous relation, we indicate that S _j , _k (.) Indicates the specific spatialization treatment applied to any rank i-frame to which the index j, k has been assigned or to any corresponding frame group. It is understood in particular that for a frame duration typically equal to 16 milliseconds, for example, a succession of frames can of course be assigned the same index j, k.

Of course, step D of FIG. 1a is followed by a step E consisting of comparing the value of rank i of the frame considered TSSj with a maximum value I of the decomposition or subdivision into frames.

On a negative response to the test of step E, the rank of the frame considered i is incremented to the value i + 1 in step F to return to step A and continue the process as long as there is a frame not subject to the subdivision process of step A to ensure the complete processing of the SS sound signal. More specifically, it is indicated that the method that is the subject of the invention with reference to FIG. 1b consists in classifying each frame or group of frames according to a plurality of acoustic classes denoted by C _j .

With reference to the above-mentioned figure, each acoustic class C _j such as Music, Speech / Talker Type, Brouhaha, or Silence, for example, can be associated with an index value j taking the corresponding value 0, 1, 2, 3. for the aforementioned acoustic classes in a non-limiting manner.

In addition, a sub-index value k can be associated with the index value j as represented in FIG. 1b for the music, k being able to take, with the value j = 0, the values 1, 2 or 3 for example , for classical music, background music or other, for example.

the acoustic class Word / speaker type can be associated with the index value j = 1 and the subindex values 2, 3, 4, the aforementioned subindex values being able to advantageously make it possible, for example, to discriminate the speaker ; the acoustic class Brouhaha can be associated with the index value j = 2 of subindex values 2, 4 and 5 for example, which can correspond to different environments such as urban hubbub, industrial or other, for example;

The acoustic class of Silence can be associated with the index value j = 3, sub-index values 4, 5 and 6 for example can advantageously be provided to correspond either to absolute silence or to specific noises such as crunches, squeaks or others, which can testify to different ambiances of silence for example.

In general, it is recalled that the value of the index j, 0, 1, 2 or 3 is a value obtained by analysis from the spectral and / or physiological parameters of the sound signal SS in each frame and that the value subindex k associated with each index value may be an arbitrary value or corresponding to a particular quality of the sound signal.

As regards, more specifically, the acoustic class relating to the

Speech / Type of speaker, for Cj = I, it is indicated that the execution of the classification of each frame or group of frames by type of speaker, that is to say choice of the speaker 2, 3 or 4 when the speech signal corresponding to Cj = I is detected, is to discriminate the speaker change from among any number of speakers in the sound signal for example. The technique used to process documents that are a priori unknown is the following: - a change of speaker is detected (break in the signal of a particular index);

- the speaker after change is compared to all the speakers already identified in the document and he is either recognized as one of them, or considered as a new speaker and therefore increases the size of the "reference dictionary" of speakers for this document or sound signal.

For a more detailed description of a procedure for discriminating the type of speaker, in a speech signal or an audio signal comprising a speech signal, reference may be made to the publication entitled "Segmentation and indexing by speakers". an audio document "published by Perrine Delacourt, EURECOM Institute, 2229 Crest Road, 0.6904 Sophia Antipolis, France and published by RJC in 1999.

With regard to the spatialization operation performed in step D for each of the classified successive frames, it is indicated that the spatialization process applied to each classified frame may be chosen from among a plurality of spatialization processes such as reverberation, attenuation, the fundamental frequency change, the harmonic filtering coloration, the delay for example, or the holophonie, stereophonic, binaural or other techniques. It will be understood that for any index value j and subindex k assigned to each frame TSSj or group of frames can thus be chosen, as a function of the aforementioned index and subindex value, a specific spatialization treatment. and in particular the most appropriate treatment according to the desired effect.

By way of nonlimiting example, during an operation of spatialization of a speech signal and in particular for a type of speaker discrimination, it is indicated that the processing may consist in applying a so-called "fun" effect by changing the speech signal. voice tone, for example, for one or more speakers of the SS sound signal. It is thus understood that the method which is the subject of the invention makes it possible to carry out differentiated sound renderings according to the aforementioned index values and in particular of the acoustic classes C _j defined previously in the description.

The method which is the subject of the invention thus makes it possible to automatically apply sound renderings or differentiated sound positions to any sound documents which result in SS sound signals for which no additional information is available or for which there is no control. not the sound.

According to a particularly remarkable aspect of the method which is the subject of the invention, the latter also makes it possible to reduce the load or the bit rate of the networks or the spectrum of the radio waves, since while the transmission of the sound signal SS can be carried out on a signal monophonic for example, he. then it is quite possible to execute the spatialization process upon reception of this signal and therefore after transmission at least from the point of view of the load or the speed of the transmission network.

Different variants of implementation of the method which is the subject of the present invention will now be described with reference to FIGS. 2a and 2b.

With reference to FIG. 2a, it is indicated that the method which is the subject of the invention can advantageously be executed in off-line mode in deferred time.

In this situation, it is at least to perform the frame analysis B and frame assignment B frames subdivision operations to generate a sequence of classified frames and then record on a recording medium. at least the index sequence assigned to the sound signal subdivided into frames and the sound signal SS or successive frames representative of the latter.

In FIG. 2a, steps A, B represent the same steps as those bearing the same reference in FIG. The index assignment step can then be subdivided into a step Co corresponding to choosing the corresponding index j and subindex k, assigned to each frame TSSj, the step Co being followed by a step C ₁ consisting of compare the rank of the frame i to the final value I representative of the number of frames. On negative response to the test C ₁ , the rank of the frame is incremented by the relation i = i + 1 at step C ₂ for a return to the analysis of the frames in step B and continuation of the process as long as the rank of the frame did not reach the value I. On the contrary, on a positive response to the test of step Ci, a record C ₃ of all the frames noted [TSSiJ, and a series of indexes noted

The concept of recording executed in step C ₃ refers to a recording on any storage medium such as, for example, a non-volatile memory, a permanent memory, a CD or DVD type optical recording disk. example.

The offline mode implementation in deferred time of the method which is the subject of the invention then consists, as represented in FIG. 2b, from the record available, obtained in step C ₃ , of reading the support of corresponding record, comprising at least the index sequence assigned to the. sound signal subdivided into frames and the sound signal or at least the sequence of frames representative of the latter in a step C shown in Figure 2b.

The reading step D ₀ is then followed by a step D ₁ of applying to the sound signal and to each current frame of this sound signal, a spatialization processing function of the index assigned to the current frame of the sound signal according to the symbolic relationship of step D of figure la.

The spatialized sound signal is thus restored in accordance with the method that is the subject of the present invention. The spatialization and rendering step Di is then followed by a test step D ₂ consisting of comparing the value of the rank of the frame i with the maximum value I. On a negative response to the test D ₂ , the procedure is followed. incrementing the frame rank i = i + 1 and returning to the reading in step C 0 of the next frame until the value I of the frame rank is reached. The method that is the subject of the present invention can also be implemented in offline broadcast mode with a limited time offset, not exceeding the duration of a few frames, by analysis and classification of each successive frame and successive spatialization processing. each frame according to the assigned index.

In this situation, it is understood, for example, that the recording operation as described in connection with Figure 2a in step C3 can be started. Memorization intervening for example on an addressable electronic memory and the reading and spatialization operation for restitution of the spatialized sound signal represented in FIG. 2b that can take place after obtaining a few frames by reading off the aforementioned electronic memory, the reading intervening in a synchronous manner but offset by a duration of a few frames for example.

Typically, regardless of the sound rendering system, the system adapts to the recording.

In addition, the method that is the subject of the present invention, as illustrated according to the implementation of FIGS. 2a and 2b, can be executed in line communication mode with minimum time shift, not exceeding the duration of a frame, by analyzing and classifying and spatially processing each frame of the delayed sound signal to a maximum of one frame duration.

In this situation, it is also understood that the operations of FIGS. 2a and 2b can be performed with storage in a memory such as a random access memory for example, the time offset being able to be reduced to the calculation time of the analysis operation of frame and index assignment for a current frame, that is to say the B and Co steps of Figure 2a, this calculation time can of course be made much smaller than the duration of a frame. The abovementioned procedure for execution in online communication mode of the method that is the subject of the present invention can advantageously be used for a spatialization processing of a telephone communication, for example, on a speech signal transmitted over a fixed or mobile telephone network. for example. It can also be implemented in online communication mode for the transmission of an IP network speech signal - for example.

A more detailed description of a spatialization system of a sound signal according to the subject of the present invention will now be given in connection with FIGS. 3a and 3b.

In general, with reference to the above figures, the system which is the subject of the invention comprises at least in combination a module 1 for analyzing the sound signal in successive frames in order to determine and assign to each successive frame an index representative of minus one spectral and / or physiological parameter of the sound signal in each frame to thus generate a sequence of indexes of classified frames. In FIGS. 3 a and 3b, the index sequence of classified frames is represented by the relation

With reference to FIGS. 3a and 3b, the system according to the invention furthermore comprises a module 2 for processing each frame or group of frames of the sound signal SS and in particular frames classified and assigned the same index according to a same spatialization processing, a function of the value of the assigned index and, as previously described in the description, of the sub-index k associated with any index value j representative of an acoustic class. With reference to FIGS. 3a and 3b, it is indicated that the spatialization processing module 2 is able, from the sequence of indexes of the aforementioned classified frames and, of course, from the sequence of frames which it has at its disposal. apply the spatialization processing and reproduce the spatialized sound signal on a set of loudspeakers denoted HP in FIGS. 3 a and 3b. More specifically, with reference to FIG. 3a, it is indicated that for processing of the sound signal SS in diffusion mode, that is to say off-line mode, the analysis module 1 and the processing module 2 by spatialisation are connected in cascade. In this situation, the analysis module 1 delivers to the spatialization processing module 2 a signal representative of the index sequence as indicated above, as well as either the sound signal SS or the sequence of frames [TSSi] J shifted (e) temporeinlement of a duration substantially equal to a plurality of frame duration to the spatialization processing module 2.

On the contrary, as represented in FIG. 3b, for a processing of the sound signal SS in online communication mode, the analysis module 1 and the processing module 2 are connected substantially in parallel as shown in the above-mentioned figure, to execute each subdivided into frames in parallel with the sound signal.

In this situation, the analysis module 1 and the spatialization processing model 2 may advantageously each comprise a frame subdivision module, bearing the reference Io respectively ₂₀ , which are synchronized by a signal synchronization S _y exchanged for example between the module 1 and module 2 analysis processing by spatialization.

In the case of FIG. 3b, as in the case of FIG. 3a, the analysis module 1 furthermore comprises a module 1 ₁ executing the operations B and C of FIG. 1a, that is to say allowing the analysis of frames and the assignment of indexes to generate the sequence of classified frames and, in particular, the index sequence previously described.

The spatialization processing module 2 comprises a processing module ensuring the actual spatialization operation on the series of classified frames.

In the case of FIG. 3b, it is indicated that the processing of analysis module 1 and, in particular, the module I ₁ of the latter can be much lower than that of a frame duration, which makes it possible to ensure a mode of operation of the system shown in Figure 3b with a processing time much lower than a frame time. The corresponding system then appears particularly well suited to processing the sound signal SS in online communication mode under the conditions indicated above in the description. Of course, the method which is the subject of the present invention and the corresponding system as described in FIGS. 3a and 3b can be implemented from corresponding software modules and in particular from a program recorded on a storage medium and executed by a computer.

This is in particular with regard to the modules Io of the analysis module 1 represented in FIG. 3a or 3b, this module making it possible to execute the subdivision into frames, as well as software modules allowing the analysis of each successive frame of the signal. sound to determine at least one spectral and / or physiological parameter of the sound signal in each frame, and then assign to each frame the representative index of the spectral and / or physiological parameter to generate a sequence of classified frames, that is to say say the signal corresponding to the index sequence previously described in the description. These operations are executed by a software module I ₁ implanted on the analysis module 1 represented in FIG. 3a or 3b.

The same applies to the spatialization module 2 _\ , which makes it possible to execute and submit each frame to the group of classified frames affected. from the same index to a spatialization process according to the value of the index assigned to each frame.

With regard to the mode of implementation of the method which is the subject of the invention, represented in FIG. 2a or 2b, it is indicated, in particular, that the method which is the subject of the invention can be implemented in completely software form, in particular for the spatialization processing of sound signals transmitted in connection with html pages.

In this case, the operating mode is that shown in FIG. 1a. The sound signal SS is then successively subjected to the steps A, B, C, D and E of the above-mentioned figure on a terminal from a transmitted SS sound signal. via IP packets without any prior processing performed at the server transmitting the corresponding html pages. In this case, the operating mode corresponds to an on-line processing with a processing delay corresponding to at most one frame duration.

Claims

1. A method of spatialization of a sound signal, characterized in that it consists at least of: subdividing said sound signal into successive frames; analyzing the frames of said frame block sound signal to determine at least one spectral and / or physiological parameter of said sound signal in each frame; assigning each frame an index representative of said at least one spectral and / or physiological parameter, to generate a sequence of classified frames; subject each frame, or group of frames, classified, assigned the same index, to a spatialization processing, a function of the value of said index assigned to each frame.

2. Method according to claim 1, characterized in that it consists in classifying each frame or group of frames according to a plurality of acoustic classes, each associated with an index representative of at least one sound signal of music, speech, type. of speaker, hubbub, silence.

3. Method according to claim 2, characterized in that, for a classification of each frame or group of frames by type of speaker, it consists in discriminating the change of speaker among any number of speakers in said sound signal.

4. Method according to one of claims 1 to 3, characterized in that said same distinct spatialization processing is selected from a plurality of spatialization processing such as reverberation, attenuation, fundamental frequency change, coloring by harmonic filtering, delay, or holophony techniques, stereophony, binaural techniques.

5. Method according to one of the preceding claims, characterized in that it is executed offline off-line mode, said method consisting at least, following the steps of analysis and index assignment to generate a Following classified frames, to record on a recording medium at least the sequence of indexes assigned to the sound signal subdivided into frames and said sound signal.

6. Method according to claim 5, characterized in that it is executed in off-line mode in deferred time, said method consisting at least of: - reading a recording medium comprising at least said index sequence assigned to the signal sound divided into frames and said sound signal; applying to said sound signal and to each current frame of this sound signal a spatialization processing, a function of the index assigned to the current frame of said sound signal.

7. Method according to one of claims 1 to 4, characterized in that it is executed in partial offline diffusion mode in restricted time offset, not exceeding the duration of a number of frames by analysis and classification. of each successive frame and successive spatialization processing of each frame according to the assigned index.

8. Method according to one of claims 1 to 4, characterized in that it is executed in online communication mode, minimum time offset not exceeding the duration of a frame, by analysis, classification and processing of spatialisation of each frame of the delayed sound signal to a maximum of one frame duration.

9. Spatialization system of a sound signal, characterized in that it comprises at least, in combination: means for analyzing said sound signal in successive frames, to determine and assign to each successive frame an index representative of minus one spectral and / or physiological parameter of said sound signal in each frame, to generate a sequence of indexes of classified frames; means for processing each frame, or group of frames, of the sound signal, classified and assigned to the same index according to a spatialization processing, a function of the value of said assigned index.

10. System according to claim 9, characterized in that, for a processing of said sound signal in diffusion mode, off line partial, said analysis means and said processing means are connected in cascade, said analysis means delivering said processing means a signal representative of an index sequence and a sound signal temporally offset by a duration substantially equal to a plurality of frame durations to said spatialization processing means.

11. System according to claim 9, characterized in that, for a treatment of said sound signal in online communication mode, said analysis means and said processing means are connected in parallel to each perform a subdivision into frames in parallel with said signal. sound, said analysis means delivering to said processing means a sequence of indexes of classified frames, said processing means performing a spatialization processing of each frame, a function of the value of said index assigned to each frame with a time shift on the sound signal not exceeding a frame duration.

12. Recording on a recording medium of a sound signal, characterized in that, in addition to a sound signal recorded in the form of a set of frames, it comprises a series of indexes, each index being representative of at least one spectral and / or physiological parameter of said sound signal in a frame.

13. Program product recorded on a storage medium for execution by a computer, characterized in that, when executed by a computer, it allows at least: subdividing a sound signal into successive frames; analyzing each successive frame of said sound signal to determine at least one spectral and / or physiological parameter of said sound signal, in each frame; assigning each frame an index representative of said at least one spectral and / or physiological parameter, to generate a sequence of classified frames; - Subject each frame, or group of frames, classified, assigned the same index, to the same spatialization processing, a function of the value of said index assigned to each frame.