EP2552130A1

EP2552130A1 - Method for sound signal processing, and computer program for implementing the method

Info

Publication number: EP2552130A1
Application number: EP11305977A
Authority: EP
Inventors: Benjamin Bernard; François BECKER
Original assignee: Longcat
Current assignee: Auro Technologies NV
Priority date: 2011-07-27
Filing date: 2011-07-27
Publication date: 2013-01-30
Anticipated expiration: 2031-07-27
Also published as: DK2552130T3; EP2552130B1

Abstract

A method for sound signal processing a sound originating from a source (S) situated inside a first room (R1) and propagating to a listener (L) situated inside a second room (R2), and the first and second rooms (R1, R2) having an opening (O) between them. The method comprises the steps of determining a position of a single relay point (V), and determining a listener sound signal (s_L ) based on a source sound signal (s_S ) of the source (S), and the position of the relay point (V).

Description

The present invention concerns a method for sound signal processing.
Acoustic sound simulation methods, that determines a time sound signal at any point inside a three dimensional environment, are known. Ray tracing methods or image source methods are well known.
Such methods are applied for predicting room acoustic response, such as concert halls.
Unfortunately, these methods require lots of calculi during many minutes, even if it is processed on a powerful computer. Such method is therefore very expensive.
One object of the present invention is to provide a method for sound signal processing that may be done in real time.
To this effect, the invention is a method for sound signal processing, said sound originating from a source situated inside a first room and propagating to a listener situated inside a second room, said second room being adjacent to said first room, and said first and second rooms having an opening between them, the method comprising the steps of:

determining a position of a single relay point belonging to a plane comprising the opening, said relay point being substantially a point from which the listener perceives the sound originating from the source, and
determining a listener sound signal for the listener, said listener sound signal being based on a source sound signal of the source inside the first room, and the position of the relay point.

Thanks to these features, the sound signal processing needs few calculi and it can be processed in real time on a standard small computer.
The positions and properties of the sources and listeners can be changed and tuned. The effects of the modifications on the listener sound signal can be heard in real time by a sound engineer. Additionally, a predetermined displacement of any listener inside the three dimensional environment can be defined, and the listener sound signal can be computed accordingly. Additionally, the change of properties of any source or element in the environment can be defined over time and the listener sound signal can be also computed accordingly.
Such method may reproduce many acoustical room effects (reflection, and eventually reverberation) and sound propagation effects inside the room (attenuation, doppler).
In various embodiments of the method, one and/or other of the following features may optionally be incorporated:

the source is a punctual source propagating a substantially spherical sound wave inside the first room, around the source position, and having an attenuation law that decreases the sound wave amplitude in relation to a distance from the position of the source;
the position of the relay point is determined so that a distance being the length of a first segment between the source and the relay point added to the length of a second segment between the relay point to the listener (L) is minimal, the relay point being a point belonging to the opening;
the listener sound signal is determined by the source sound signal attenuated by the attenuation law applied to the distance, and delayed by a propagation delay corresponding to the distance;
the source is an ambient source propagating a substantially plane sound wave inside the first room in a direction of a source direction vector;
the position of the relay point is determined by the intersection between the plane of the opening and a line passing through the listener and having a direction of a perceived direction vector corresponding to the direction from which the source is perceived by the listener;
the position of the relay point is determined by the direction of the perceived direction vector from which the source is perceived by the listener and a distance which is the length of the shortest path between the listener and a point of the opening.
the perceived direction vector is a vector forming an angle (β') in relation to an opening direction vector, said perceived direction vector being in a transverse plane, and said angle (β') being given by a function f: $βʹ = f (α β)$

wherein
- the opening direction vector is defined as a unity vector in the direction of a vector L.C between the listener and a centre point, wherein the centre point is a point being the mass centre of a surface being the projection of the second room without the opening on a unity sphere centred around the listener,
- the transverse plane is defined by the listener, the opening direction vector and the source direction vector,
- α is a first angle between the opening direction vector and a deviation vector (L.O1) between the listener and a first point, said first point being the intersection of the periphery of the opening with the transverse plane in the direction of the source direction vector, and
- β is a second angle between the opening direction vector and the source direction vector;
- the angle (β') between the opening direction vector and the perceived direction vector is given by: $βʹ = f (β α) = β . (π - α) / π;$
- the listener sound signal is determined by the source sound signal attenuated by the attenuation law applied to a distance between the relay point and the listener, delayed by a propagation delay corresponding to said distance, and multiplied by a gain factor g;
- the gain factor g is determined as follow:
  - $if 0 < = β < α, g (β) = \frac{1}{\sin^{2} (α - β)},$
    
    and
  - $if α < β < π, g (β) = 1.$

The invention also relates to a computer program comprising machine code for operating the above method steps, wherein the computer program is operated on at least one computer or on at least one embedded processing board.
Other features and advantages of the invention will be apparent from the following detailed description of one embodiment given by way of non-limiting example, with reference to the accompanying drawings.
In the drawings:

Figure 1 is a schematic top view of an environment comprising a first and second room, for applying the method for sound signal processing,
Figure 2 is a perspective view of the environment of figure 1,
Figure 3 is a top view of the environment of figure 1 showing the processing of three cases of punctual sources,
Figure 4 is a top view of the environment of figure 1 showing the processing of an ambient source,
Figure 5 is an example of a mixing console implementing the method for sound signal processing,
Figure 6 is an example of a computer software's user interface, said computer software contains an implementation of the method of the present invention.

1/ Introduction and definitions

The Figure 1 shows a top view of a three-dimensional (3D) environment wherein the method is applied and Figure 2 shows the same 3D environment in a perspective view.
The 3D environment comprises for example a first room R1 and a second room R2, said rooms having an opening O between them.
A room is substantially a convex 3D-space volume having none, one or a plurality of surfaces, said surfaces being globally substantially closed and convex, and enclosing the 3D-space volume.
The room may comprise for example four walls, a floor, and a ceiling. In the figures, the first room R1 comprises four lateral walls 11, 12, 13 and 14, a floor 10 and a ceiling 16. The second room R2 comprises four lateral walls 21, 22, 23 and 24, a floor 20 and a ceiling 26. The second room R2 is adjacent to the first room R1, and the walls 11 and 21 are in that case a common wall of the first and second rooms.
The surfaces of the rooms have formed with a material having a sound reflecting property.
An opening O is a convex portion of the surfaces belonging to a room. The surfaces are often plane surfaces. The opening is more often a plane portion. The opening O is a passage belonging to a room through which sound wave may go through, passing from the first room R1 to the second room R2, or reciprocally.
The opening O is between the first and second rooms R1, R2 through said common wall 11, 21.
The opening O comprises a periphery, the periphery being a convex curve belonging to the common wall 11, 21 separating the rooms.
Such 3D space environment is for example a model of a real scene, defined with sets of data stored inside a memory of a computer. The method is operated by such computer to simulate propagation of sound inside said environment between sources S and listeners L, located inside said environment.
In the figures, a source S is located inside the first room R1. The source S is emitting a sound inside the first room R1. The emitted sound from the source S is a time signal named the source sound signal s_s.
A listener L is located inside the second room R2. The listener L is receiving a sound inside the second room R2. The received sound at listener L is a time signal named the listener sound signal s_L .
A plane PO is a plane comprising the opening O, and being adjacent to the wall 21 belonging to the second room R2.
The opening O for example represents a door or a window between the first and second rooms R1, R2.
Each surface of a room behaves as a real surface. It may produce reflection of an incident sound wave, according to material properties of said surface.
Such room is a closed 3D space (reflection on all the surfaces), or semi-open 3D space (one surface has no reflection), or an open 3D space (a single surface has reflection), or any other type of space.
For example, the surfaces have no reflection and only direct path from the source S to the listener are taken into account during the method. In that case, the method is less expensive for the computer calculi.
By defining the material properties, the sizes and the positions of each surfaces belonging to each room, that is to say the three-dimensional model of the environment, any type of space may be defined.
A reflection is the change of direction of a sound propagating inside a space when the wave arrives at an interface between two mediums (a surface belonging to a room), i.e. interface between air inside a room and the material of a surface. The sound wave returns into the medium from which it is originated. The sound wave arrives to the surface with an angle of incidence in relation to a normal vector to the surface. The reflected sound wave continues to propagate from said surface with an angle of reflection. The angle of incidence and angle of reflection are usually equal in value.
Here, a source S is considered to be either a punctual source type or an ambient source type. We will describe below how each type of source is represented and simulated inside the above defined environment.
For a source S of punctual source type, the source S comprises a determined position inside the 3D space of the room. Such source is represented on the figures with a cross. The sound propagates from said determined position substantially as a spherical wave, i.e. in all the directions. The sound wave energy going through a determined area and propagating from a punctual source decreases with the distance between the position of the sound source and a point of space. The punctual source therefore comprises a predetermined attenuation law.
An example of attenuation law for a punctual source propagating a sound in free field is given by the following expression that gives a sound intensity I at a receiver point at a distance r from the source S: $I (r) = \frac{P}{4 π r^{2}},$

where

r is a distance between the source position and receiver position,
P is the total source emitted power.

For convenience and continuity reasons, the intensity can be limited to a pre-defined maximum. Hence, the attenuation law can be given by the following expression: $I (r) = \min (I_{\max} \frac{P}{4 π r^{2}})$

where I_max is the pre-defined maximum intensity.
For a source S of ambient source type, the source S is a sound source for which the perceived sound for a listener is always the same for any position of said listener inside said room. Such ambient source is defined by a direction of incidence, but no position inside the room, and no attenuation law. Therefore, such ambient source may be positioned anywhere inside the room. Such source is represented on the figures with an arrow (a source direction vector).

2/ Method of the invention

The method of present invention can be applied for a punctual source type or an ambient source type.
The method comprises the following steps:

determining a position of a single relay point V belonging to the plane PO comprising the opening O, said relay point V being substantially a point from which the listener L perceives the sound originating from the source S, and
determining a listener sound signal s_L for the listener L on the bases of the source sound signal s_s of the source S inside the first room R1, and the determined position of the relay point V.

If the source S is a punctual source, (see figure 2 and 3 ), the position of the relay point V is determined by the shortest path P for going from the source S to the listener L through the opening O.
The figures 2 and 3 illustrate three cases wherein the source S has three different positions inside the first room R1: these are identified by the references S, S' and S".
For the first case, the source S is visible from the listener L inside the second room R2. In other words, a straight line between the source S and the listener L intersects the plane PO with a point belonging to the opening O area or periphery. Said point is the relay point V, defining a direction from the relay point V to the listener L, said direction being the direction from which the listener L perceives the sound originating from the source S.
For the second and third cases, the source S' and S" are not visible from the listener L inside the second room R2. The positions of the relay point V', V" are determined by the following conditions:

the relay point V', V" is positioned at the periphery of opening O; and
a path comprising a first line segment S'V', S"V" and a second line segment V'L, V"L is determined to have a minimal length.

There is only one single point following these properties.
In all cases, the relay point V is a point defined so that:

the relay point V belongs to the opening O; and
the position of the relay point V is determined by: length (SV) +length (VL) is minimal.

Then, the listener sound signal s_L is determined by applying to the source sound signal s_s the effect of the propagation of sound from the source S to the listener L through the defined path passing via the relay point V. The sound signal is attenuated by the attenuation law by taking into account the length l of the path, and delayed by Δt taking into account the wave speed c in the medium. $l = \overline{SV} + \overline{VL}$
$I = I (l)$
$Δ t = l / c$
If the source S is an ambient source inside first room R1 having a direction of incidence defined by a source direction vector a (see figure 4 ), the method also comprises the following steps:

determining a perceived direction vector a' representing the direction from which the ambient source S is perceived by the listener L through the opening O, and
determining the relay point V representing the ambient source S inside the second room R2, said relay point V being a source of punctual type that propagates sound wave inside the second room R2.

In order to determine the perceived direction vector a, the following steps are performed:

a unit size sphere SP is defined around the listener L,
a projection SP2 is the central projection on the sphere SP of the surfaces belonging to the second room R2 without the opening O,
a point C is a centre of mass of said projection SP2,
an opening direction vector f is the vector L.C, from the listener L to the point C, normalized, i.e. $\vec{f} = \frac{\vec{L . C}}{‖ \vec{L . C} ‖} .$

The ambient source S is perceived by the listener L with a perceived direction vector á' modified by the opening O, said perception direction vector á' is determined by the following process:

If f and á' are collinear, á' = á.

If f and á are not collinear, a transverse plane PT is defined, said transverse plane PT being defined by the listener L, the opening direction vector f, and the source direction vector a. The transverse plane PT intersects the periphery of the opening with a first point 01 being in the half-plane of the transverse plane PT in the direction of the source direction vector a, and a second point 02 being in the other half-plane of the transverse plane PT in the opposite direction of the source direction vector a.
A first angle α is the angle between the opening direction vector f and the vector L.O1 between the listener L and the first point 01,
A second angle β is the angle between the opening direction vector f and the source direction vector a.
The perceived direction vector a' is defined to be in the transverse plane PT and such as a third angle β' is an angle between the opening direction vector f and said perceived direction vector a', the third angle β' being determined for example by the following formula: $βʹ = β . (π - α) / π$
If β = 0, the ambient source S inside the first room R1 has a source direction vector aligned with the opening O and the listener L. In such case, the opening O does not modify the perceived direction of the source S, and β' = 0.
The opening O modifies the perceived direction of the source S, by a proportional coefficient of (π-α)/π. This means that the opening modifies more the perceived direction of the source S if the perceived angle of the opening viewed from the listener L is greater.
Other formulas for modifying the source S perceived by the listener may be defined.
As illustrated in figure 4, the perceived direction vector a' is a vector having an angle β' with the opening direction vector f.
The relay point V representing the ambient source S inside the second room R2 is defined by the line passing through the listener L and collinear to the perceived direction vector a', and a distance defined as the shortest distance between the listener L and the opening O (see figure 4), and chosen such as L.V is in the opposite direction of the perceived direction vector a'.
The position of the relay point V depends on the position of the listener L.
The relay point V is not necessarily positioned onto the plane PO.
The relay point V is considered for the listener L to be a punctual source generating a sound inside the second room R2, said sound being equal to the source signal s_s of the ambient source S belonging to the first room R1.
According to a first variant, the listener sound signal s_L is determined by applying to the source sound signal s_s the effect of the propagation of sound from the relay point V to the listener L. The sound signal is attenuated by the attenuation law by taking into account the length of the path VL, and delayed by taking into account the wave speed c in the medium.
According to a second variant, the listener sound signal s_L is determined by applying to the source sound signal s_S the same attenuation and delay, and by multiplying it by a gain factor g.
Thanks to this gain factor g, the amplitude of the sound from the ambient source S from the first room R1 may be decreased, notably if the source direction vector is not directed in the first room R1 in direction to the opening O.
The gain factor g may be a function of β, wherein the gain factor g is an increasing function between zero and α, and is equal to one between α and π.
Additionally, the gain factor g may be determined by: $if 0 < = β < α, g (β) = \frac{1}{\sin^{2} (α - β)},$

and $if α < β < π, g (β) = 1.$
The relay point V is considered to behave as a punctual source that propagates the sound of the ambient source S located inside the first room R1 to the listener L inside the second room R2.
Then, the listener sound signal _SL is determined by applying to the source sound signal s_S the attenuation, the delay and the gain factor.
The previous processes for a punctual source or an ambient source may be improved by taking into account a plurality of paths between the source S and the relay point V, and/or between the relay point V and the listener L.
In that case, the listener sound signal s_L is determined by a sum of sound components, each component corresponding to a path of sound between a first point to a second point inside a room, each component corresponding to a direct path between the first point and the second point, or corresponding to a predetermined number of reflections on the surfaces of the room.
If only direct path is used to determine the listener sound signal s_L, the method is less expensive in calculations.
If multiple paths are computed, the determined listener sound signal s_L may produce reverberation effects, due to the room model parameters (for example: size, room surface materials).

3/ Generalisation to a plurality of openings and a plurality of rooms

In case of a plurality of openings O_i between the first room R1 and the second room R2, i being an index number of the opening belonging to the plurality, the method is explained below.
If the source S is a punctual source, each opening O_i comprises a relay point V_i corresponding to the opening O_i, said relay point representing the source S perceived by the listener L inside the second room R2. Such relay point V_i is determined by the following rules:

length (SV_i )+length (V_iL) is minimal,
V_i belongs to the opening O_i.

Then, all the relay points V_i are a passage of sound from the first room R1 to the second room R2. The listener sound signal s_L is therefore the sum of all the independent listener signals s_Li for each relay point V_i.
If the source S is an ambient source, a global perceived direction vector a)' representing the direction of the ambient source S perceived by the listener L through all the openings O_i is determined by a mean vector value: $\vec{aʹ} = \sum_{i} \vec{a} ʹ_{i},$

each perceived direction vector $\vec{a'_{i}}$
being determined for each opening O_i between the first and second rooms R1, R2, as it was already explained.
Then, a single relay point V representing the ambient source S inside the second room R2 through all the openings O_i can be determined by the line passing through the listener and collinear to the global perceived direction vector a' and a distance defined as the shortest distance between the listener L and the surface SCH, said SCH is defined by the area of the opening plane PO bordered by the convex hull of all the openings O_i, and chosen such as L.V is in the opposite direction of a'.
The listener sound signal s_L can be determined as previously explained.
Thanks to this method, only one relay point V is used to determine the listener sound signal s_L through a plurality of openings O_i between the first room R1 and the second room R2. The method is therefore not expensive in computer power.
In case of a plurality of successive rooms R_j, j being an index number of the room, each room being in communication with the following by an opening O_j, a source S being inside the first room R1 and a listener L being inside the last room Rr, the method to determine a relay point V is expendable.
If the source S is a punctual source, a relay point V_j is determined at each opening O_j, so that the length of all the segments SV₁, V₁V₂, to V_r-2V_r-1,, V_r-1L is minimal. Each relay point V_j belongs to the corresponding opening O_j (or is at the periphery of it). The perceived direction vector is determined by the last relay point (V_r-1) alone.
If the source S is an ambient source, a relay point V, is determined regarding the opening O₁ by the described method that transforms the ambient source S into a relay point V₁ behaving has a punctual source; the projection is done on the sphere SP centered on a V' ₂, the point of the opening O₂ on the shortest path between the center of the opening O₁ and the listener L through the successive openings. Then, the others relay points V₂ to V_r-1 are determined by the minimal length through the corresponding openings O₁ to O_r-1.
Thanks to the previous generalisations, all configurations of rooms may be built, and any complex environment may be established.

4/ Industrial applications of the invention

The above described method is simpler than ray tracing acoustic calculation methods. The method may be implemented in real time.
The method may be used for sound mixing.
The method may be used for sound simulation in computer games.
The method may be used for sound mixing for a cinema movie. In the film industry, it is usual to record each source and to mix the different recordings later, during a post production editing step. The sound mixing may be done for a listener being at the 3D position of the camera, so that to reproduce the 3D environment acoustic effects. However, the sound mixing may be done for a listener being at a different position than the position of the camera, so that to modify the acoustic effects, to be more pleasant or to be amplified or attenuated. Such choice is done by the sound engineer doing the film mixing. In fact the sound engineer may modify all the environment properties: for example, the wall and surfaces materials, the attenuation laws, the positions, and any other property of the environment model.
The method may be implemented as software in one personal computer or a plurality of personal computers connected together by any known network. The method may be computed using the main Central Processing Unit CPU, or the Graphics Processing Unit GPU, or both of them, or any signal processing unit.
The method may be implemented inside a standalone mixing console schematically represented on figure 5 , having:

signal inputs I₁ to I_m, m being a number of inputs,
signal outputs O₁ to On, n being a number of outputs, and
a control interface adapted to modify the mixing.

The control interface may be a keyboard, a screen or a touch screen, haptic or motion tracking systems, or any device that automatically change the parameters of mixing. For example, a camera may record a scene comprising an actor who moves inside the three dimensional environment. The control interface may analyse the images of the camera to track the actor and determine its 3D position inside the environment, and change the position of a listener corresponding to said actor inside the predetermined environment stored inside the control interface.
Moreover, the inputs I₁ to I_m and the outputs O₁ to On may be analog and/or digital ports.
The outputs may deliver data in any known audio format, such as stereo, Dolby digital, 5.1 audio format, 6.1 audio format or 7.1 audio format, or any audio format, for diffusion on multiple device types, such as but not limited to headphones, home cinema speaker sets, cinema speaker sets. All these outputs may be delivered simultaneously. The audio mixing is therefore similar for all of them, and only one mixing is done by the sound engineer.

Claims

A method for sound signal processing, said sound originating from a source (S) situated inside a first room (R1) and propagating to a listener (L) situated inside a second room (R2), said second room (R2) being adjacent to said first room (R1), and said first and second rooms (R1, R2) having an opening (O) between them, the method comprising the steps of:
- determining a position of a single relay point (V), said relay point (V) being substantially a point from which the listener (L) perceives the sound originating from the source (S), and

- determining a listener sound signal (s_L ) for the listener (L), said listener sound signal (s_L ) being based on a source sound signal (s_S ) of the source (S) inside the first room (R1), and the position of the relay point (V).
The method according to claim 1, wherein the source (S) is a punctual source propagating a substantially spherical sound wave inside the first room (R1), around the source position, and having an attenuation law that decreases the sound wave amplitude in relation to a distance from the position of the source (S), and the relay point (V) belongs to the plane (PO) comprising the opening (O) .
The method according to claim 2, wherein the position of the relay point (V) is determined so that a distance being the length of a first segment between the source (S) and the relay point (V) added to the length of a second segment between the relay point (V) to the listener (L) is minimal, the relay point (V) being a point belonging to the opening (O) .
The method according to claim 3, wherein the listener sound signal is determined by the source sound signal (s_S ) attenuated by the attenuation law applied to said distance, delayed by a propagation delay corresponding to said distance.
The method according to claims 1, wherein the source (S) is an ambient source propagating a substantially plane sound wave inside the first room (R1) in a direction of a source direction vector (a).
The method according to claim 5, wherein the position of the relay point (V) is determined a line passing through the listener (L) and having a direction of a perceived direction vector (a') corresponding to the direction from which the source (S) is perceived by the listener (L), a distance equal to the shortest distance between the listener (L) and the opening (O), and such as L.V is in the opposite direction of the perceived direction vector (a').
The method according to claim 6, wherein the perceived direction vector (a') is a vector forming an angle (β') in relation to an opening direction vector (f), said perceived direction vector (a') being in a transverse plane (PT) , and said angle (β' ) being given by a function f: $βʹ = f (α β)$

wherein
- the opening direction vector (f) is defined as a unity vector in the direction of a vector L.C between the listener (L) and centre point (C), wherein the centre point (C) is a point being the mass centre of a surface being the projection of the second room (R2) without the opening (O) on a unity sphere centred around the listener (L),

- the transverse plane (PT) is defined by the listener (L), the opening direction vector (f) and the source direction vector (a),

- α is a first angle between the opening direction vector (f) and a deviation vector (L.O1) between the listener (L) and a first point (01), said first point (01) being the intersection of the periphery of the opening (O) with the transverse plane (PT) in the direction of the source direction vector (a), and

- β is a second angle between the opening direction vector (f) and the source direction vector (a).
The method according to claim 7, wherein the angle (β') between the opening direction vector (f) and the perceived direction vector (a') is given by: $βʹ = f (β α) = β . (π - α) / π .$
The method according to claim 8, wherein the listener sound signal (s_L) is determined by the source sound signal (s_S) attenuated by the attenuation law applied to a distance between the relay point (V) and the listener (L), delayed by a propagation delay corresponding to said distance, and multiplied by a gain factor g.
The method according to claim 9, wherein the gain factor g is determined as follow:
- $if 0 < = β < α, g (β) = \frac{1}{\sin^{2} (α - β)},$
and

- $if α < β < π, g (β) = 1.$
A computer program comprising machine code for operating the method steps according to anyone of the claims 1 to 10, wherein said computer program is operated on at least one computer or on at least one embedded processing board.