RU2780536C1 - Equipment and method for reproducing a spatially extended sound source or equipment and method for forming a bitstream from a spatially extended sound source - Google Patents

Abstract

FIELD: acoustics.

SUBSTANCE: invention relates to tools for reproducing a spatially extended sound source. The projection of a two or three-dimensional shell associated with a spatially extended sound source on the projection plane is calculated using the position of the listener, information relating to the geometry of the spatially extended sound source, and information relating to the position of the spatially extended sound source. The positions of at least two sound sources for the spatially extended sound source are calculated using the projection plane. At least two sound sources are rendered in the positions to obtain the reproduction of the spatially extended sound source with two or more output signals. Said rendering includes a stage of using different sound signals for different positions, wherein different sound signals are associated with the spatially extended sound source.

EFFECT: increase in the effectiveness of reproducing a spatially extended sound source. The position of a listener is assumed.

41 cl, 11 dwg

Description

Specification

The present invention relates to the processing of audio signals and, in particular, to the encoding or decoding or reproduction of a spatially extended sound source.

For a long time, the reproduction of sound sources through several loudspeakers or headphones has been studied. The simplest way to reproduce sound sources in such arrangements is to make them point sources, i.e. very (ideally: infinitely) small sound sources. However, this theoretical concept makes it almost impossible to model existing physical sound sources in a realistic way. For example, a piano has a large vibrating wooden top with many spatially distributed strings inside and therefore appears much larger to the ear than a point source (particularly when the listener (and microphones) are close to the piano). Many real-world sound sources are of significant size ("spatial extent"), such as musical instruments, cars, an orchestra or choir, or ambient sounds (the sound of a waterfall).

Correct/realistic reproduction of such sound sources becomes the goal of many sound reproduction techniques, whether they are binaural (i.e. using so-called human hearing aid transfer functions (HRTF) or binaural room impulse responses (BRIR)) using headphones, or traditionally using speaker setups ranging from 2 speakers ("stereo") to multiple speakers arranged in a horizontal plane ("surround sound") and multiple speakers surrounding the listener in all three dimensions ("3D audio").

The purpose of the present invention is to provide a concept for encoding or reproducing spatially extended sound sources with possibly complex geometric shapes.

2D Source Width

This section describes techniques that relate to rendering () extended sound sources on a 2D surface facing the listener's perspective, for example, over a range of azimuth changes at zero degrees elevation (as is the case in traditional stereo/surround sound), or in some ranges of azimuth and elevation (as is the case in 3D audio or virtual reality with 3 degrees of freedom ("3DoF") of user movement, i.e. head rotation in the lateral/vertical/longitudinal axes).

Increasing the apparent width of an audio object that is being panned between two or more speakers (forming a so-called phantom image or phantom source) can be achieved by reducing the correlation of the channel signals involved. As the correlation decreases, the dispersion of phantom sources increases until, for correlation values close to zero (and not too wide opening angles), it covers the entire range between loudspeakers.

Decorrelated versions of the source signal are obtained by extracting and applying the appropriate decorrelation filters. In the work of the author Lauridsen, it is proposed to add/subtract the scaled version of the source signal with a time delay relative to itself in order to obtain two decorrelated versions of the signal. More complex approaches, for example, are proposed in the work of the author Kendall. It iteratively extracts paired decorrelation all-pass filters based on combinations of random number sequences. The work of Faller et al. suggests suitable decorrelation filters ("scatterers") c. Also in the work of Zotter et al., filter pairs are extracted in which frequency dependent phase or amplitude differences are used to achieve phantom source expansion. In addition, decorrelation filters based on velvet noise are proposed, which are further optimized.

In addition to reducing the correlation of the respective phantom source channel signals, the source width can also be increased by increasing the number of phantom sources assigned to the audio object. B, the source width is controlled by panning the same source signal in (slightly) different directions. Initially, a method was proposed for stabilizing the perceived phantom spread of VBAP-panned sources as they move around in the soundstage. This is advantageous because, depending on the direction of the source, the rendered source is played back through two or more speakers, which can lead to undesirable changes in the perceived width of the source.

DirAC in the Virtual World is an extension of the traditional Directional Audio Coding (DirAC) approach for audio synthesis in virtual worlds. To render spatial extent, the directional audio components of a source are randomly panned within a certain range around the source's original direction, with the panning directions varying in time and frequency.

A similar approach is employed in which spatial extent is achieved by randomly distributing source signal bandwidths for different spatial directions. It is a method for creating a spatially distributed and enveloped sound that comes in the same way from all directions, instead of controlling the exact degree of extension.

In the work of Verron et al., spatial extent of the source is achieved not by using panned correlated signals, but by synthesizing several incoherent versions of the source signal, distributing them evenly around the listener, and mixing them together. The number and amplification of simultaneously active sources determine the intensity of the expansion effect. This method is implemented as a spatial sweep for the synthesizer for ambient sounds.

3D Source Width

This section describes techniques that relate to rendering extended sound sources in 3D space, i.e. in a three-dimensional way, which is required for virtual reality with 6 degrees of freedom ("6DoF"). This means 6 degrees of freedom for user movement, i.e. rotation of the head along the transverse, vertical and longitudinal axes) plus 3 directions x / y / z of translational movement in space.

The work of Potard et al. extends the concept of source extent as a one-dimensional source parameter (ie, its width between two speakers) by examining the perception of source shapes. It generates multiple non-coherent point sources by applying (time-varying) decorrelation techniques to the original source signal and then placing the non-coherent sources at different spatial locations and thereby giving them a three-dimensional extent.

In the advanced MPEG-4 AudioBIFS standard, 3D objects/shapes (shell, box, ellipsoid and cylinder) can be filled with multiple equally spaced and decorrelated sound sources in order to evoke a 3D source extension.

In order to increase and control the extent of the source using ambiophony, Schmele et al. proposed a blending decrease in the order of ambiphony of the input signal, which internally increases the apparent width of the source and the distribution of decorrelated copies of the source signal around the listening space.

Another approach is introduced by Zotter et al., which adapts the principle proposed in (i.e., extracting filter pairs that introduce frequency dependent phase and magnitude differences to achieve source extent in stereo playback arrangements) for ambiophony.

A common drawback of pan-based approaches (for example) is their dependence on the listener's position. Even a small deviation from the sweet spot causes the spatial image to collapse to the loudspeaker closest to the listener. This drastically limits their application in the context of virtual reality and augmented reality with 6 degrees of freedom (6DoF), when the listener is supposed to move freely. Additionally, the distribution of time-frequency bins in DirAC-based approaches (for example) does not always guarantee proper rendering of the spatial extent of phantom sources. In addition, it typically significantly degrades the timbre of the source signal.

Decorrelation of source signals is usually achieved by one of the following methods: i) extracting filter pairs with complementary absolute value (for example), ii) using all-pass filters with constant absolute value but (randomly) scrambled phase (for example), or iii) spatially random distribution time-frequency bins of the source signal (for example).

All approaches have their own consequences: Complementary filtering of the source signal according to i) typically results in an altered perceived timbre of the decorrelated signals. Although all-pass filtering as noted in ii) preserves the timbre of the source signal, the scrambled phase violates the original phase relationships and, in particular for transient signals, causes severe temporal dispersion and blurring artifacts. Spatial distribution of time-frequency bins is effective for some signals, but also changes the perceived timbre of the signal. In addition, it exhibits strong signal dependence and introduces severe artifacts for pulsed signals.

Filling volumetric shapes with multiple decorrelated versions of a source signal proposed in the enhanced AudioBIFS standard implies the availability of a large number of filters that produce mutually decorrelated outputs (typically more than ten point sources per volumetric shape are used). However, finding such filters is not a trivial task and becomes more difficult the more such filters are required. Also, if the source signals are not fully decorrelated and the listener is moving such a shape, for example in a (virtual reality) scenario, the individual source-to-listener distances correspond to different source signal delays, and their overlap in the listener's ears results in position-dependent comb filtering. , potentially introducing annoying transient coloration of the source signal.

Controlling the source width with ambiphony-based technology by lowering the ambiphony order exhibits an audible effect only for transitions from second to first or zero order. In addition, these transitions are perceived not only as an extension of the source, but also, and often, as a movement of the phantom source. While summing decorrelated versions of a source signal can help stabilize the perception of the source's apparent width, it also introduces comb filtering effects that change the timbre of the phantom source.

The purpose of the present invention is to provide an improved concept for reproducing a spatially extended sound source or generating a bit stream from a spatially extended sound source.

This object is achieved by the equipment for reproducing a spatially extended sound source according to claim 1, the equipment for generating a bit stream according to claim 27, the method for reproducing a spatially extended sound source according to claim 35, the method for generating a bit stream according to claim 36, the bit stream according to paragraph 41 or a computer program according to paragraph 47.

The present invention is based on such findings that the reproduction of a spatially extended sound source can be achieved and, in particular, even becomes possible by calculating the projection of a two-dimensional or three-dimensional shell associated with a spatially extended sound source onto the projection plane using the listening position. This projection is used to calculate the positions of at least two sound sources for a spatially extended sound source, and the at least two sound sources are prepared by rendering positions to obtain a reproduction of a spatially extended sound source, in which the rendering results in two or more output signals, and in which different sound signals for different positions are used, but different sound signals are associated with the same spatially extended sound source.

A high-quality two-dimensional or three-dimensional audio reproduction is obtained because, on the one hand, the time-varying relative position between the spatially extended sound source and the (virtual) listener position is taken into account. On the other hand, a spatially extended sound source is effectively represented by geometric information regarding the perceived extent of the sound source and by a number of at least two sound sources, such as peripheral point sources, which can be easily processed by renderers known in the art. . In particular, simple renderers in the art are always in position to render sound sources at some position relative to some output format or speaker layout. For example, two sound sources calculated by the sound position calculation module at some positions may be prepared by rendering at those positions, such as by amplitude panning.

When, for example, sound positions are between left and left surround in 5.1 output format, and when other sound sources are between right and right surround in output format, the amplitude pan procedure performed by the renderer should result in sufficiently similar signals. for the left and left surround channels for one sound source and, respectively, to sufficiently similar signals for the right and right surround sound for another sound source in such a way that the user perceives the sound sources as coming from the positions calculated by the sound position calculation module. However, due to the fact that all four signals are ultimately associated and associated with a spatially extended sound source, the user simply does not perceive two phantom sources associated with the positions calculated by the sound position calculation module, and the listener perceives one spatially extended sound source.

The equipment for reproducing a spatially extended sound source having a given position in the geometry in space contains an interface, a projector, a sound position calculation module and a rendering module. The present invention makes it possible to take into account the improved sound situation that occurs, for example, in a piano. The piano is a large device, and by now, the piano sound can be rendered as coming from a single point source. However, this does not fully represent the true characteristics of the piano's sound. In accordance with the present invention, an exemplary piano for a spatially extended sound source is reflected by at least two audio signals, where one audio signal can be recorded by a microphone positioned close to the left side of the piano, i. close to the bass strings, while another sound source can be recorded with another second microphone positioned close to the right side of the piano, i.e. near the upper sound frequency strings that form high tones. Naturally, both microphones must record sounds that differ from each other due to the reflective situation in the piano and of course also due to the fact that the bass string is closer to the left microphone than to the right microphone and vice versa. However, on the other hand, both microphone signals must have a significant amount of similar sonic components that ultimately make up the unique sound of a piano.

In accordance with the present invention, a bit stream representing a spatially extended sound source such as a piano is generated by recording signals by recording also the geometric information of the spatially extended sound source, and optionally also by either recording location information associated with various microphone positions (or , in general, with two different positions associated with two different sound sources), or providing a description of the perceived geometric shape of the sound (piano). To reflect the position of the listener in relation to sound sources, i.e. that a listener can "walk around" in virtual reality or augmented reality, or any other soundstage, the shell projection associated with a spatially extended sound source, such as a piano, is computed using the position of the listener, and, the positions of at least two sources sound are computed using the projection plane, with particular preferred embodiments referring to the positioning of sound sources at peripheral points of the projection plane.

It is possible, with reduced computational overhead and reduced rendering overhead, to actually represent an exemplary piano sound in a 2D or 3D situation such that when the listener is, for example, closer to the left side of a sound source such as a piano, the sound which the listener perceives is different from the sound produced when the user is close to the right side of a sound source such as a piano, or even behind a sound source such as a piano.

In view of the foregoing, the idea of the invention is unique in that, on the encoder side, a method for characterizing a spatially extended sound source is provided that allows the use of a spatially extended sound source in a sound reproduction situation for a true two-dimensional or three-dimensional layout. In addition, the use of a listener's position within a very flexible description of a spatially extended sound source is made possible in an efficient manner by calculating the projection of a two-dimensional or three-dimensional shell onto the projection plane using the listener's position. The sound positions of at least two sound sources for a spatially extended sound source are calculated using the projection plane, and at least two sound sources are prepared by rendering at the positions calculated by the sound position calculation module in order to obtain a reproduction of the spatially extended source audio having two or more headphone outputs or multi-channel outputs for two or more channels in a stereo playback layout or a playback layout having more than two channels, such as five, seven or more channels.

Compared to the prior art method for filling three-dimensional volume with sound by placing many different point sources in all parts of the volume to be filled, the projection eliminates the need to model a large number of sound sources and drastically reduces the number of point sources used due to the need to fill only the shell projection, those. two-dimensional space. In addition, the number of point sources required is further reduced by modeling preferably only the sources on the projection shell, which, in extreme cases, can be simply one sound source at the left boundary of the spatially extended sound source and one sound source at the right boundary of the spatially extended sound source. . Both stages of reduction are based on two psychoacoustic observations:

Unlike the azimuth (and elevation) of a sound source, its distance cannot be perceived very reliably. Thus, projecting the original loudness onto a plane perpendicular to the listener does not significantly change the perception (but may help to reduce the number of point sources needed for rendering).

Two decorrelated sounds that are distributed as point sources to the left and right, respectively, tend to perceptually fill the space between them with sound.

In addition, the encoder side not only allows characterization of one spatially extended sound source, but is also flexible in that the bitstream generated as a representation can include all data for two or more spatially extended sound sources, which is preferably associated, with respect to their geometric information and location, to the same coordinate system. On the decoder side, playback cannot be performed for only one spatially extended sound source, but may be performed for several spatially extended sound sources, with the projector calculating the projection for each sound source using the (virtual) listening position. Additionally, the sound position calculation module calculates the positions of at least two sound sources for each spatially extended sound source, and the rendering module renders all calculated sound sources for each spatially extended sound source, for example, by summing two or more outputs from each spatially extended sound source signal-by-signal or channel-by-channel and by providing the summed channels to appropriate headphones for binaural playback, or to appropriate loudspeakers in a speaker-related arrangement for playback, or alternatively to a storage device to store (combined) two or more output signals for later use, or transmission.

On the generator or encoder side, a bitstream is generated using equipment for generating a bitstream representing a compressed description for a spatially extended sound source, the equipment comprising an audio provider for providing one or more different audio signals for a spatially extended sound source, and an output data generator generates a bitstream representing a compressed soundstage, the bitstream containing one or more different audio signals, preferably in a compressed manner, e.g. compressed by a bitrate compression encoder, e.g. MP3-, AAC-, USAC- or MPEG-H -encoder. In addition, the output data generator is configured to introduce into the bit stream, in the case of two or more different audio signals, optional separate location information for each audio signal from two or more different audio signals, indicating the location of the corresponding audio signal, preferably with respect to spatial geometry information. extended sound source, i.e. that the first signal is the signal recorded on the left side of the piano in the above example and the signal recorded on the right side of the piano.

However, alternatively, the location information need not necessarily be associated with the geometry of the spatially extended sound source, but may also be associated with a common origin, although a relationship with the geometry of the spatially extended sound source is preferred.

In addition, the equipment for generating the compressed bitstream also includes a geometry provider for calculating information about the geometry of the spatially extended sound source, and the output data generator is configured to introduce, into the bitstream, information about the geometry, information about the individual location information for each audio signal, in addition to at least two audio signals, such as audio signals recorded by microphones. However, the audio provider need not necessarily actually pick up the microphone signals, but the audio signals can also be generated, at the encoder side, using decorrelation processing as appropriate. At the same time, only a small number of audio signals or even one audio signal may be transmitted for a spatially extended audio signal, and the remaining audio signals are generated on the playback side using decorrelation processing. This is preferably signaled by a bitstream element in the bitstream in such a way that the audio playback module always knows how many audio signals are included per spatially extended sound source, so that the playback module can determine, in particular in the calculation module audio positions, how many audio signals are available, and how many audio signals are to be extracted on the decoder side, for example, through correlation processing or signal synthesis.

In this embodiment, the regenerator module writes a bitstream element to a bitstream indicating the number of audio signals included for a spatially extended audio source, and, on the decoder side, the audio playback module decodes the bitstream element from the bitstream, reads the bitstream element, and determines, based on the bitstream element, how many signals for preferably peripheral point sources or auxiliary sources placed between peripheral audio sources should be calculated based on at least one received audio signal in the bitstream.

The preferred embodiments of the present invention will now be explained with reference to the accompanying drawings, in which:

Fig. 1 is a general block diagram representation of a preferred embodiment of the playback side;

Fig. 2 illustrates a spherical spatially extended sound source with a different number of peripheral point sources;

Fig. 3 illustrates an ellipsoid spatially extended sound source with multiple peripheral point sources;

Fig. 4 illustrates a linear spatially extended sound source with various methods for distributing the location of peripheral point sources;

Fig. 5 illustrates a cuboid spatially extended sound source with various procedures for distributing peripheral point sources;

Fig. 6 illustrates a spherical spatially extended sound source at various distances;

Fig. 7 illustrates a spatially extended piano-shaped sound source in approximately parametric ellipsoidal form;

Fig. 8 illustrates a spatially extended piano-shaped sound source with three peripheral point sources distributed over the extreme points of the projected convex hull;

Fig. 9 illustrates a preferred implementation of equipment or method for reproducing a spatially extended sound source;

Fig. 10 illustrates a preferred implementation of equipment or method for generating a bit stream representing a compressed description for a spatially extended sound source; and

Fig. 11 illustrates a preferred implementation of a bitstream generated by the equipment or method illustrated in FIG. ten.

Fig. 9 illustrates a preferred implementation of equipment for reproducing a spatially extended sound source having a given position and geometry in space. The equipment includes an interface 100, a projector 120, a sound position calculation module 140, and a rendering module 160. The interface is configured to receive the position of the listener. In addition, the projector 120 is configured to calculate a projection of a two-dimensional or three-dimensional shell associated with a spatially extended sound source onto the projection plane using the listener position received through the interface 100 and use, additionally, information regarding the geometry of the spatially extended sound source, and additionally, using information regarding the position of a spatially extended sound source in space. Preferably, a given position of the spatially extended sound source in space, and additionally, the geometry of the spatially extended sound source in space, is received to reproduce the spatially extended sound source via the bit stream input to the bit stream demultiplexer 180 or scene parser. The bitstream demultiplexer 180 extracts, from the bitstream, geometry information of the spatially extended sound source and provides this information to the projector. In addition, the bitstream demultiplexer also extracts the position of the spatially extended sound source from the bitstream and forwards this information to the projector. Preferably, the bitstream also contains location information for at least two different audio sources, and preferably, the bitstream demultiplexer also extracts, from the bitstream, a compressed representation of the at least two audio sources, and the at least two audio sources are decompressed. /decoded by the decoder as an audio decoder 190. The decoded at least two sound sources are finally redirected to the renderer 160, and the renderer prepares by rendering at least two sound sources at the positions provided by the sound position calculation module 140 to the renderer 160 .

Although FIG. 9 illustrates bitstream related playback equipment having a bitstream demultiplexer 180 and an audio decoder 190, playback can also be performed in a situation different from the encoder/decoder scenario. For example, the given position and geometry in space may already exist in the playback hardware, such as a virtual reality or augmented reality style scene in which data is generated at a site and consumed at that site. The bitstream demultiplexer 180 and audio decoder 190 are not actually required, and spatially extended sound source geometry and spatially extended sound source position information is available without being extracted from the bitstream at all. In addition, location information relating the location of at least two sound sources to the geometric information of the spatially extended sound source may also be fixed in advance and thus does not need to be transmitted from the encoder to the decoder, or, alternatively, this data is generated, again, on site.

Therefore, it should be noted that location information is only provided in the embodiments, and it is not necessary to transmit this information even in the case of two or more sound source signals. The decoder or playback module, for example, may always receive the first sound source signal in the bitstream as the sound source on the projection placed more to the left. Likewise, the second sound source signal in the bit stream may be received as the sound source in the projection placed more to the right.

In addition, although the sound position calculation module calculates the positions of at least two sound sources for a spatially extended sound source using a projection plane, the at least two sound sources need not be received from the bitstream. Instead, only one audio source from at least two audio sources can be received via the bit stream and the other audio source, and because of this, also other position or location information can actually be generated only on the reproduction side without the need to transmit this information from the generator. bit streams to the playback module. However, in other embodiments, all of this information may be transmitted, and additionally, a higher number than one or two audio signals may be transmitted in the bitstream when the bitrate requirements are not stringent and the audio decoder 190 must decode two, three or more sound signals representing at least two sound sources, the positions of which are calculated by the sound position calculation module 140 .

Fig. 10 illustrates the encoder side of this scenario when playback is applied in an encoder/decoder application. Fig. 10 illustrates equipment for generating a bit stream representing a compressed description for a spatially extended sound source. In particular, a sound provider 200 and an output generator 240 are provided. In this implementation, a spatially extended sound source is represented by a compressed description having one or more different audio signals, and the output generator generates a bitstream representing the compressed sound stage, the bitstream containing at least one or more different audio signals. and geometric information associated with the spatially extended sound source. This represents the situation illustrated with respect to FIG. 9, in which all other information, such as the position of the spatially extended sound source (see the dotted arrow in block 120 of FIG. 9), can be freely selected by the user on the playback side. Thus, a unique description of a spatially extended sound source is provided with at least one or more different audio signals for that spatially extended sound source, these audio signals being simply point source signals.

The shaping hardware further comprises a geometry provider 220 for providing, for example, computation of geometry information for a spatially extended sound source. Other methods of providing geometric information other than computation include receiving user input, such as a user-drawn drawing, or any other information provided by the user, such as through speech, tones, gestures, or any other user action. In addition to one or more different audio signals, also geometry information is input to the bit stream.

Optionally, information regarding individual location information for each audio signal from one or more different audio signals is also input to the bitstream, and/or position information for a spatially extended audio source is also input to the bitstream. The position information for the sound source may be separate from the geometry information, or may be included in the geometry information. In the first case, the geometry information may be output regarding the position information. In the second case, the geometric information may contain, for example, for a sphere, a center point in coordinates and a radius or diameter. For a box-shaped spatially extended sound source, eight or at least one of the corner points may be output in absolute coordinates.

The location information for each of the one or more different audio signals is preferably associated with geometric information of the spatially extended sound source. However, alternatively, the absolute location information associated with the same coordinate system as the position information or geometric information of the spatially extended sound source is also useful, and alternatively, the geometric information can also be output in the absolute coordinate system with absolute coordinates, and not in a relative way. However, providing this data in a relative manner not related to a common coordinate system allows the user to position the spatially extended sound source in the playback arrangement as indicated by the dashed line directed to the projector 120 of FIG. 9.

In a further embodiment, the audio provider 200 of FIG. 10 is configured to provide at least two different audio signals for a spatially extended audio source, and the output data generator is configured to generate a bit stream such that the bit stream contains at least two different audio signals, preferably in an encoded format and optionally, information of a separate locations for each sound signal from at least two different sound signals, either in absolute coordinates or relative to the geometry of a spatially extended sound source.

In an embodiment, the audio provider is configured to record a natural sound source at multiple distinct microphone positions or orientations, either: to extract an audio signal from a single base signal or multiple base signals via one or more decorrelation filters, such as as explained with respect to FIG. 1, paragraphs 164 and 166. The basic signals used in the generator may be the same or different from the basic signals provided at the playback site or transmitted from the generator to the playback module.

In a further embodiment, the geometry provider 220 is configured to extract, from the geometry of the spatially extended sound source, a parametric description or polygonal description, and the output generator is configured to inject, into the bitstream, this parametric description or polygonal description.

In addition, the output data generator is configured to introduce, into the bitstream, a bitstream element, in the preferred embodiment, this bitstream element indicating the number of at least one other audio signal for a spatially extended audio source included in the stream. bitstream or included in the encoded audio signal associated with the bitstream, wherein the number is 1 or greater than 1. The bitstream generated by the output data generator need not necessarily be a complete bitstream with audio waveform data on the one hand and metadata on the other hand. the other side. Instead, the bitstream may also only be a single metadata bitstream containing, for example, a bitstream field for the number of audio signals for each spatially extended audio source, geometric information for the spatially extended audio source, and, in an embodiment, also position information for a spatially extended sound source, and optionally location information for each sound signal and for each spatially extended sound source, geometric information for a spatially extended sound source, and, in an embodiment, also position information for a spatially extended sound source. The audio waveforms, typically available in compressed form, are transmitted via a separate data stream or a separate transmission channel to the playback module such that the playback module receives, from one source, the encoded metadata and from another source (encoded) waveforms.

In addition, an embodiment of the bit stream generator includes a controller 250. The controller 250 is configured to control the audio provider 200 regarding the number of audio signals to be provided by the audio provider. In accordance with this procedure, the controller 250 also provides bitstream element information to the output generator 240, indicated by a dashed line indicating an optional feature. The output data generator inputs, in the bitstream element, specific information regarding the number of audio signals controlled by the controller 250 and provided by the audio provider 200 . Preferably, the number of audio signals is controlled such that the output bit stream containing the encoded audio audio signals satisfies the external bit rate requirements. When the allowed bit rate is high, the audio provider must provide more audio signals than when the allowed bit rate is low. As a last resort, the audio provider must provide only one audio signal for a spatially extended audio source when bit rate requirements are stringent.

The playback module must read the corresponding given element of the bitstream and must continue, in the rendering module 160, to synthesize, on the decoder side and using the transmitted sounds signal, the appropriate number of additional audio signals, so that, ultimately, the required number of peripheral point sources and , optionally, auxiliary sources.

However, when the bitrate requirements are not very stringent, the controller 250 should control the audio provider to provide a high number of different audio signals, such as those recorded by an appropriate number of microphone or microphone orientations. Thereafter, on the playback side, decorrelation processing is not required at all or only to a small extent, so that ultimately a better playback quality is obtained by the playback module due to reduced or optional decorrelation processing on the playback side. The trade-off between bit rate on the one hand and quality on the other hand is preferably obtained through the functionality of the bitstream element indicating the number of sound signals per spatially extended sound source.

Fig. 11 illustrates a preferred embodiment of a bit stream generated by the bit stream generating equipment illustrated in FIG. 10. The bit stream contains, for example, the second spatially extended sound source 401, indicated as SESS ₂ with the corresponding data.

In addition, FIG. 11 illustrates the details for each spatially extended sound source with respect to spatially extended sound source number 1. In the example of FIG. 11, two audio signals serve for a spatially extended sound source, which are generated in a bitstream generator, for example, from the output of microphones taken from microphones placed at two different locations in a spatially extended sound source. The first audio signal is audio signal 1, indicated as 301, and the second audio signal is audio signal 2, indicated as 302, and both audio signals are preferably encoded via an audio encoder for bit rate compression. In addition, item 311 represents a bitstream element indicating the number of audio signals for a spatially extended sound source 1, such as controlled by the controller 250 of FIG. ten.

Geometric information for a spatially extended sound source is entered as shown in block 331. Item 301 indicates optional location information for audio signals, preferably relative to geometric information, for example, relative to a piano example, indicating "close to bass strings" for audio signal 1 and "close to the upper audio frequency strings" for audio signal 2, indicated as 302. Geometric information, for example, can be a parametric representation or a polygonal representation of a piano-based model, and this piano-based model should be different, for example, for a grand piano or (small) piano. Item 341 further illustrates optional data regarding position information for a spatially extended sound source in space. As indicated, this position information 341 is optional when the user provides position information as indicated by the dotted line in FIG. 9 towards the projector. However, even when the position information 341 is included in the bitstream, the user can still replace or modify the position information through user interaction.

The following explains the preferred embodiments of the present invention. Embodiments relate to rendering spatially extended sound sources in 6DoF VR/AR (virtual reality/augmented reality).

Preferred embodiments of the invention are directed to a method, equipment or computer program designed to improve the reproduction of spatially extended sound sources (SESS). In particular, embodiments of the inventive method or equipment consider a time-varying relative position between a spatially extended sound source and the listener's virtual position. In other words, embodiments of the inventive method or equipment allow the auditory width of the source to match the spatial extent of the presented sound object at any relative position to the listener. In this regard, an embodiment of the inventive method or equipment applies in particular to 6 DoF (6 DoF) virtual, mixed and augmented reality style applications in which a spatially extended sound source complements traditionally used point sources.

An embodiment of the inventive method or equipment renders a spatially extended sound source by using a plurality of peripheral point sources fed with (preferably significantly) decorrelated signals. Unlike other methods, the locations of these peripheral point sources depend on the listener's position relative to the spatially extended sound source. Fig. 1 illustrates a block diagram of a general representation of a spatially extended sound source rendering module according to an embodiment of the inventive method or equipment.

The key components of the flowchart are as follows:

Listener Position: This block provides the instantaneous position of the listener, such as measured by a virtual reality-style tracking system. The block may be implemented as a detector 100 for discovery or an interface 100 for receiving a listener position.

Spatial Sound Source Position and Geometry: This block provides position and geometry data for a spatially extended sound source to be rendered, for example, as part of a VR-style scene representation.

Projection and Convex Hull Computation: This block 120 computes the convex hull of the geometry of spatially extended sound sources and then projects it towards the listener's position (eg, the "image plane", see below). Alternatively, the same function can be achieved by first projecting the geometry to the listener's position and then computing its convex hull.

Location of peripheral point sources: This block 140 calculates the locations of used peripheral point sources from the convex hull projection data computed by the previous block. In this calculation, it can also consider the position of the listener and therefore the proximity/distance of the listener (see information below). The output is n locations of peripheral point sources.

Renderer core: The renderer core 162 auralizes n peripheral point sources by positioning them at specified target locations. It may, for example, be binaural renderers using human hearing aid transfer functions, or renderers for playback through loudspeakers (eg, for vector amplitude panning). The renderer core generates l speaker or headphone outputs from k input basic audio signals (eg, decorrelated instrument recording signals) and m ≥ (n-k) additional decorrelated audio signals.

Basic Source Signals: This block 164 is an input for k basic audio signals that are (sufficiently) decorrelated from each other and represent the audio source to be rendered (e.g., mono (k=1) or stereo (k=2 ) recording of a musical instrument). The K basic audio signals are, for example, extracted from a bitstream (see, for example, elements 301, 302 of FIG. 11) received from a generator at the decoder side, or may be provided at the site for playback from an external source.

Decorrelators: This optional block 166 generates the additional decorrelated audio signals required to render the n peripheral point sources.

Signal output: The renderer provides output signals l for loudspeaker rendering (eg n=5.1) or binaural rendering (typically n=2).

Fig. 1 illustrates a general block diagram representation of an embodiment of an inventive method or equipment. Dashed lines indicate the transfer of metadata such as geometry and positions. Solid lines indicate audio transmission, where k, l and m indicate the set of audio channels. The renderer core 162 receives possibly k+m audio signals and n (<=k+m) positional data. Blocks 162, 164, 166 together form an embodiment of the overall module 160 renderer.

The locations of the peripheral point sources depend on the geometry, in particular the spatial extent, the spatial extent of the sound source, and the relative position of the listener relative to the spatial extent of the sound source. In particular, peripheral point sources can be located on the projection of the convex shell of a spatially extended sound source onto the projection plane. The projection plane can be either the picture plane, i.e. a plane perpendicular to the line of sight from the listener to a spatially extended sound source, or a spherical surface around the listener's head. The projection plane is located at an arbitrary small distance from the center of the listener's head. Alternatively, the convex hull of a projection of a spatially extended sound source may be calculated from azimuth and elevation angles, which are a subset of spherical coordinates relative to the perspective of the listener's head. In the illustrative examples below, the projection plane is preferred due to its more intuitive nature. In a projected convex hull implementation, the angular representation is preferred due to simpler formalization and lower computational complexity. It should be noted that the projection of the convex hull of a spatially extended sound source is the same convex hull of the geometry of the projected spatially extended sound sources, i.e. the calculation of the convex hull and the projection onto the plane of the picture can be used in any order.

Locations of peripheral point sources can be distributed over the projection of the convex hull of a spatially extended sound source in a variety of ways, which include the following:

- They can perturb uniformly around the shell projection

- They can be distributed at the extremum points of the shell projection

- They can be located at the horizontal and/or vertical extremum points of the shell projection (see Fig. in the "Practical examples" section).

In addition to peripheral point sources, other auxiliary point sources can also be used to generate an improved sense of acoustic filling at the expense of additional computational complexity. Additionally, the projected convex hull can be modified to position peripheral point sources. For example, the projected convex hull may contract towards the center of gravity of the projected convex hull. Such a compressed projected convex hull can take into account the additional spatial dispersion of individual peripheral point sources introduced by the rendering method. The modification of the convex hull can further differ between scaling the horizontal and vertical directions.

When the position of the listener relative to the spatially extended sound source changes, then the projection of the spatially extended sound source onto the projection plane changes accordingly. In turn, the locations of peripheral point sources change accordingly. The locations of the peripheral point sources should preferably be chosen such that they change smoothly to continuously move the spatially extended sound source and listener. Additionally, the projected convex hull changes as the geometry of the spatially extended sound source changes. This involves rotating the geometry of spatially extended sound sources in 3D space, which changes the projected convex hull. The rotation of the geometry is equal to the angular displacement of the position of the listener relative to the spatially extended sound source and is, for example, referred to in an inclusive manner as "the relative position of the listener and the spatially extended sound source". For example, the circular motion of the listener around a spherical spatially extended sound source is represented by the rotation of peripheral point sources around the center of gravity. Similarly, rotation of a spatially extended sound source with a stationary listener results in the same change in the locations of peripheral point sources.

The spatial extent that is generated by an embodiment of the inventive method or equipment is internally reproduced correctly for any distance between the spatially extended sound source and the listener. Naturally, as the user approaches the spatially extended sound source, the opening angle between the peripheral point source increases in a way that is appropriate for simulating physical reality.

Since the angular placement of peripheral point sources is uniquely determined by location on the projected convex hull onto the projection plane, the distances of peripheral point sources can further be selected in a variety of ways, including:

All peripheral point sources have the same distance, equal to the distance of the entire spatially extended sound source, for example, a spatially extended sound source specified through the center of gravity relative to the listener's head.

The distance of each peripheral point source is determined by rear projection of the locations on the projected convex hull onto the geometry of the spatially extended sound source such that the projection of the peripheral point sources onto the projection plane results in the same point. The rear projection of peripheral point sources from a projected convex hull onto a spatially extended sound source may not always be uniquely determined, so additional projection rules must apply (see the Practical Examples section).

Distance of peripheral point sources may not be specified at all if the rendering of peripheral point sources does not require a distance property, but only requires relative angular placement in azimuth and elevation.

To indicate the geometric shape/convex hull of a spatially extended sound source, an approximation is used (and possibly passed to a renderer or renderer core) that includes a simplified one-dimensional, eg, linear, curve; two-dimensional, for example, an ellipse, a rectangle, polygons; or a three-dimensional shape such as ellipsoid, cuboid, and polyhedrons. The geometry of a spatially extended sound source, or the corresponding approximate shape, respectively, can be described in a variety of ways, including:

Parametric description, i.e. the formalization of the geometry through a mathematical expression that allows additional parameters. For example, an ellipsoid shape in 3D can be described by an implicit function on a Cartesian coordinate system, and the additional parameters are the extent of the principal axes in all three directions. Additional parameters may include functions of three-dimensional rotation, deformation of the ellipsoidal surface.

Polygonal description, i.e. a collection of primitive geometric shapes such as lines, triangles, squares, tetrahedrons, and cuboids. Primate polygons and polyhedron can be concatenated into larger more complex geometries.

The signals of peripheral point sources are extracted from the basic signals of a spatially extended sound source. Basic signals can be obtained in various ways, such as: 1) Recording a natural sound source in one or more microphone positions and orientations (example: recording a piano sound, as shown in practical examples); 2) Synthesis of an artificial sound source (example: sound synthesis with varying parameters); 3) A combination of any audio signals (example: various mechanical sounds of a car, for example, engine, tires, doors, etc.). Additionally, additional peripheral point source signals can be generated artificially from the basic signals by means of several decorrelation filters (see the previous section).

In some application scenarios, attention is focused on compact and interoperable storage/transmission of 6DoF VR/AR content. In this case, the whole chain consists of three stages:

Authoring/encoding the required spatially extended sound sources into a bitstream

Transmission/storage of the generated bit stream. In accordance with the present invention, the bit stream contains, among other elements, a description of the geometries of spatially extended sound sources (parametric or polygonal) and associated basic source signal(s), such as a monophonic or stereophonic piano recording. The waveforms may be compressed (see item 260 in FIG. 10) using perceptual audio coding algorithms such as MP3 or MPEG-2/4 Advanced Audio Coding (AAC).

Decoding/rendering of spatially extended audio sources based on the transmitted bitstream as described above.

In addition to the basic method described earlier, there are several options for further processing:

Option 1. Dynamic selection of the number and location of peripheral point sources

Depending on the distance of the listener to the spatially extended sound source, the number of peripheral point sources may vary. As an example, when the spatially extended sound source and the listener are at a large distance from each other, the opening angle (aperture) of the projected convex hull becomes small, and thus a smaller number of peripheral point sources can be advantageously selected, thereby reducing computational complexity and storage complexity. In the extreme case, all peripheral point sources are reduced to one remaining point source. Appropriate downmix techniques can be applied to ensure that interference between the base and extracted signals does not degrade the sound quality of the resulting peripheral point source signals. Similar techniques can also be applied at a short distance of the spatially extended sound source from the position of the listener, if the geometry of the spatially extended sound source is very irregular depending on the relative viewpoint of the listener. For example, the geometry of spatially extended sound sources, which is a line of finite lengths, may deteriorate on the projection plane to one point. In general, if the angular extent of the peripheral point sources on the projected convex hull is low, the spatially extended sound source can be represented by fewer peripheral point sources. In the extreme case, all peripheral point sources are reduced to one remaining point source.

Option 2: Spread compensation

Because each peripheral point source also exhibits spatial spread beyond the projection of the convex hull, the perceived auditory image width of the rendered spatially extended sound source is somewhat larger than the convex hull used for rendering. To match it with the desired target geometry, there are two options:

Compensation during authoring: Additional variation in the rendering procedure is considered during content authoring. In particular, to a certain extent, the smaller geometry of the spatially extended sound sources is chosen during authoring of the content, so that the actual rendered size is desirable. This can be checked by monitoring the effect of a renderer or renderer core in an authoring environment (eg, in a production studio). In this case, the transmitted bitstream and the renderer or renderer core use a reduced target geometry compared to the target size.

Compensation during rendering: The renderer or renderer core of the spatially extended sound source renderer may be notified of the additional perceptual spread by the rendering procedure and thus may be able to compensate for this effect. As a simple example, the geometry used for rendering might be:

- decrease by a constant factor < 1.0 (e.g. a=0.9), or

- decrease by a constant opening angle alpha=5 degrees

- before it is used to locate peripheral point sources. In this case, the transmitted bit stream contains a finite target size of the geometry of the spatially extended sound sources.

In addition, a combination of these approaches is useful.

Option 3. Formation of signal forms of peripheral point sources

Additionally, the actual signals to be fed to the peripheral point sources can be generated from the recorded audio signals by considering the user's position relative to the spatially extended sound source in order to model spatially extended sound sources with geometry-dependent sound fractions, such as a piano with low note sounds on the left side. and vice versa.

Example: The sound of a piano is distinguished by its acoustic behavior. It is modeled by (at least) two basic audio signals, one near the bottom end of the piano-like keyboard ("low notes") and one near the top end of the keyboard ("high notes"). These basic signals can be obtained by appropriate use of a microphone when recording piano sound and passed to the 6DoF renderer or 6DoF renderer core, ensuring that there is sufficient decorrelation between them.

Peripheral point source signals are then extracted from these basic signals by considering the user's position relative to the spatially extended sound source:

When the user faces the piano from the front side (keyboard), the two peripheral spot sources have a sufficiently large width separation from each other near the left and right ends of the piano keyboard, respectively. In this case, the base signal for low keys can be directly supplied to the left peripheral point source, and the base signal for high keys can be directly used to drive the right peripheral point source.

As the listener walks around the piano approximately 90 degrees to the right, the two peripheral spot sources are panned very close to each other because the projection of the piano loudness model (eg, ellipse) is small when viewed from the side. If the basic signals continue to be used to directly drive peripheral point source signals, one of the peripheral point sources should contain predominantly high notes, while the other should carry mainly low notes. Since this is undesirable from a physical point of view, rendering can be improved by rotating the two basic signals so as to form the peripheral point source signals by rotating Givens by the same angle as the user moves relative to the center of gravity of the piano. Thus, both signals contain signals of similar spectral content while being decorrelated (assuming the underlying signals are decorrelated).

Option 4. Post-processing of a spatially extended sound source prepared by rendering

The actual signals may be pre- or post-processed to account for position and direction dependent effects, such as the radiation pattern of a spatially extended sound source. In other words, the whole sound emitted from a spatially extended sound source as described above can be modified such that it exhibits, for example, a direction-dependent sound emission pattern. In the case of a piano signal, this may mean that the radiation towards the rear of the piano has less high frequency content than towards the front. Additionally, pre- and post-processing of peripheral point source signals can be adjusted separately for each of the peripheral point sources. For example, the radiation pattern may be chosen differently for each of the peripheral point sources. In this example of a spatially extended sound source representing a piano, the low and high key range patterns may be similar to those described above; however, additional cues, such as pedaling noise, have a more omnidirectional pattern.

Several advantages of the preferred embodiments are summarized below:

- Lower computational complexity compared to completely filling the inside of a spatially extended sound source with point sources (for example, used in the advanced AudioBIFS standard)

- Less potential for destructive interference between point source signals

- Compact size of bitstream information (approximations of geometry, one or more waveforms)

- Provides the ability to use legacy recordings (such as stereo piano recordings) that are shaped for music consumption for VR/AR rendering purposes

The following are various practical implementation examples:

- Spherical spatially extended sound source

- Ellipsoidal spatially extended sound source

- Linear spatially extended sound source

- Cuboid spatially extended sound source

- Distance dependent peripheral point sources

- Spatially extended sound source in the form of a piano

As described in the embodiments of the inventive method or equipment above, various methods can be used to determine the location of peripheral point sources. The following practical examples demonstrate some isolated methods in specific cases. In a complete implementation of an embodiment of an inventive method or equipment, the various methods can be combined appropriately in view of computational complexity, intended use, audio quality, and ease of implementation.

The geometry of spatially extended sound sources is indicated as a green surface mesh. It should be noted that the mesh visualization does not imply that the geometry of spatially extended sound sources is described in a polygonal manner, since in fact the geometry of spatially extended sound sources may be generated from a parametric specification. The listener position is indicated by a blue triangle. In the examples below, the picture plane is chosen as the projection plane and illustrated as a transparent gray plane that indicates the final subset of the projection plane. The projected geometry of a spatially extended sound source onto the projection plane is illustrated with the same surface mesh in green. Peripheral point sources on the projected convex hull are illustrated as red crosses on the projection plane. Rear projected peripheral point sources onto the geometry of spatially extended sound sources are illustrated as red dots. Corresponding peripheral point sources on the projected convex hull and rear projected peripheral point sources on the spatially extended sound source geometry are connected by red lines to help identify the visual match. The positions of all envisaged objects are illustrated in the Cartesian coordinate system with units in meters. The choice of the illustrated coordinate system does not imply that the calculations provided are performed with Cartesian coordinates.

The first example in Fig. 2 considers a spherical spatially extended sound source. A spherical spatially extended sound source has a fixed size and a fixed position relative to the listener. Three other sets of three, five, and eight peripheral point sources are selected on the projected convex hull. All three sets of peripheral point sources are chosen with uniform spacing on the convex hull curve. The offset positions of the peripheral point sources on the curve of the convex hull are deliberately chosen such that the horizontal extent of the geometry of the spatially extended sound sources is well represented.

Fig. 2 illustrates a spherical spatially extended sound source with various numbers (ie 3 (upper), 5 (middle) and 8 (lower)) of peripheral point sources uniformly distributed on a convex hull.

The next example in Fig. 3 considers an ellipsoidal spatially extended sound source. An ellipsoidal spatially extended sound source has a fixed shape, position and rotation in three-dimensional space. Four peripheral point sources are selected in this example. The three different ways of locating peripheral point sources are roughly illustrated:

a) two peripheral point sources are placed at two horizontal extreme points and two peripheral point sources are placed at two vertical extreme points. At the same time, the positioning of extremum points is simple and often appropriate. This example shows what this method can produce as a result of the location of peripheral point sources that are relatively close to each other.

b) All four peripheral point sources are distributed evenly on the projected convex hull. The position offset of the peripheral point sources is chosen such that the position of the uppermost peripheral point source is the same as the position of the uppermost peripheral point source in a). It can be seen that the selection of the offset location of the peripheral point source has a significant impact on the representation of the geometry through the peripheral point sources.

c) All four peripheral point sources are distributed uniformly on the compressed projected convex hull. The offset location of the peripheral point source locations is equal to the offset location selected in b). The operation of shrinking the projected convex hull is performed towards the center of gravity of the projected convex hull with a direction-independent stretch factor.

Fig. 3 illustrates an ellipsoid spatially extended sound source with four peripheral point sources according to three different ways of locating peripheral point sources: a/upper) horizontal and vertical extremum points, b/middle) evenly spaced points on a convex hull, c/lower) evenly spaced points on a compressed convex hull.

The next example in Fig. 4 considers a linear spatially extended sound source. Based on the fact that the previous examples consider the volumetric geometry of spatially extended sound sources, this example demonstrates that the geometry of spatially extended sound sources can be selected as a one-dimensional object in three-dimensional space. Attached drawing a) illustrates two peripheral point sources placed at the extremum points of the geometry of finite linear spatially extended sound sources. b) Two peripheral point sources are placed at the extreme points of the geometry of the final linear spatially extended sound sources, and one additional point source is placed in the middle of the line. As described in embodiments of the inventive method or equipment, placing additional point sources within the geometry of spatially extended sound sources can help fill large gaps in large geometries of spatially extended sound sources. c) The same geometry of linear spatially extended sound sources as in a) and b) is considered; however, the relative angle to the listener has been changed in such a way that the projected length of the linear geometry is much shorter. As described in the inventive method or equipment embodiments above, the reduced size of the projected convex hull can be represented by a reduced number of peripheral point sources, in this particular example, by a single peripheral point source located at the center of the line geometry.

Fig. 4 illustrates a linear spatially extended sound source with three different ways to distribute the location of peripheral point sources: a/top) two extremum points on the projected convex hull; b/middle) two extremum points on a projected convex hull with an additional point source at the center of the line; c/lower) over one peripheral point source at the center of the bulge, because the projected convex hull of the rotating line is too small to resolve more than one peripheral point source.

The next example in Fig. 5 considers a cuboid spatially extended sound source. A cuboid spatially extended sound source has a fixed size and a fixed location; however, the listener's relative position changes. Attached drawings a) and b) illustrate different ways of placing four peripheral point sources on a projected convex hull. Rear projected locations of peripheral point sources are uniquely determined by selection on the projected convex hull. c) illustrates four peripheral point sources that do not have well-separated rear projection locations. Instead, the distances of the locations of the peripheral point sources are chosen to be equal to the distance of the center of gravity of the geometry of the spatially extended sound sources.

Fig. 5 illustrates a cuboid spatially extended sound source with three different ways to distribute peripheral point sources: a/upper) two peripheral point sources on the horizontal axis and two peripheral point sources on the vertical axis; b/middle) two peripheral point sources at the horizontal extremum points of the projected convex hull and two peripheral point sources at the vertical extremum points of the projected convex hull; c/bottom) the rear projection distances of the peripheral point source are chosen such that they are equal to the distance of the center of gravity of the geometry of the spatially extended sound sources.

The next example in Fig. 6 considers a spherical spatially extended sound source of a fixed size and shape, but at three different distances from the listener's position. Peripheral point sources are distributed evenly on the curve of the convex hull. The number of peripheral point sources is dynamically determined from the length of the curve of the convex hull and the minimum distance between possible locations of peripheral point sources. a) A spherical spatially extended sound source is located at a small distance such that four peripheral point sources are selected on the projected convex hull. b) A spherical spatially extended sound source is at an average distance such that three peripheral point sources are selected on the projected convex hull. a) A spherical spatially extended sound source is at a great distance such that only two peripheral point sources are selected on the projected convex hull. As described in the embodiments of the inventive method or equipment above, the number of peripheral point sources can also be determined from the extent represented in spherical angular coordinates.

Fig. 6 illustrates a spherical spatially extended sound source of equal size but at different distances: a/upper) a small distance with four peripheral point sources distributed evenly on the projected convex hull; b/medium) average distance with three peripheral point sources distributed evenly on the projected convex hull; c/bottom) long distance with two peripheral point sources distributed evenly on the projected convex hull.

The last example in Fig. 7 and 8 considers a spatially extended piano-shaped sound source placed in a virtual world. The user wears a head-mounted display (HMD) and headphones. The user is presented with a virtual reality-style scene consisting of an open world canvas and a piano-based 3D model standing on the floor in a free-roaming zone (see FIG. 7). An open world canvas is a spherical static image projected onto a sphere surrounding the user. In this case, the open world canvas illustrates a blue sky with white clouds. The user is able to walk around and watch and listen to the piano from different angles. In this scene, the piano is rendered as either a single point source placed at the center of gravity or as a spatially extended sound source with three peripheral point sources on a projected convex hull (see Fig. 8). Rendering experiments show a far superior realism to the way a peripheral point source is rendered compared to rendering as a single point source.

To simplify computing the locations of peripheral point sources, the piano geometry is abstracted into an ellipsoidal shape with similar dimensions, see FIG. 7. Additionally, two interchangeable point sources are placed at the left and right extremum points on the equatorial line, while the third interchangeable point remains at the north pole, see FIG. 8. This arrangement guarantees a consistent horizontal source width from all angles at a greatly reduced computational cost.

Fig. 7 illustrates a spatially extended piano-shaped sound source (illustrated in green) with an approximate parametric ellipsoidal shape (indicated as a red mesh).

Fig. 8 illustrates a spatially extended piano-shaped sound source with three peripheral point sources distributed over the vertical extremum points of the projected convex hull and the vertical top position of the projected convex hull. It should be noted that for better visualization, peripheral point sources are placed on a stretched projected convex hull.

The following provides specific features of the embodiments of the invention. The characteristics of the presented embodiments are as follows:

In order to fill the perceived acoustic space of a spatially extended sound source, preferably not all of its interior is filled with decorrelated point sources (peripheral point sources), but only its periphery, which faces the listener (for example, "projection of the convex shell of a spatially extended sound source towards the listener") . In particular, this means that the locations of peripheral point sources are not attached to the geometry of the spatially extended sound sources, but are calculated dynamically, taking into account the relative position of the spatially extended sound source relative to the position of the listener.

Dynamic calculation of peripheral point sources (number and location)

An approximation of the shape of a spatially extended sound source is used (for a scenario using a compressed representation: transmitted as part of a bitstream).

The application of the described technology may be specified as part of the 6DoF VR/AR audio standard. In this context, a classic scenario based on encoding/bitstream/decoder (+renderer) is provided:

At the encoder, the shape of the spatially extended sound source shall be encoded as ancillary information along with the "basic" waveforms of the spatially extended sound source, which may be:

- mono signal, or

- a stereo signal (preferably sufficiently decorrelated), or

- even more recorded signals (also preferably sufficiently decorrelated),

- characterizing a spatially extended sound source. These waveforms may be encoded at a low bit rate.

At the decoder/renderer, the shape of the spatially extended sound source and the corresponding waveforms are extracted from the bitstream and used to render the spatially extended sound source, as described above.

Depending on the embodiments used, and as alternatives to the embodiments described, it should be noted that the interface may be implemented as an actual tracker or listener position detector. However, the listening position would typically be received from an external tracker device and provided to the playback equipment via an interface. However, the interface may simply represent input to output from an external tracker, or it may also represent the tracker itself.

In addition, as indicated, additional auxiliary audio sources between the peripheral sound source may be required.

In addition, it has been found that left/right peripheral sources and optionally horizontally (relative to the listener) spaced auxiliary sources are more important to the perceptual experience than vertically spaced peripheral sound sources, ie. a peripheral sound source above and below a spatially extended sound source. When, for example, resources are scarce, it is preferable to use at least horizontally spaced peripheral (and optionally auxiliary) sound sources, while vertically spaced peripheral sound sources may be omitted in the interest of saving processing resources.

In addition, as indicated, the bitstream generator can be implemented to generate a bitstream with only one audio signal for a spatially extended audio source, and the remaining audio signals are generated on the decoder side or on the reproduction side by decorrelation. When only one signal exists, and when all space is to be filled equally by that one signal, no location information is needed at all. However, it may be useful to have, in such a situation, at least additional information regarding the geometry of the spatially extended sound source calculated by a geometric information calculation module, such as the module illustrated as 220 in FIG. ten.

It should be noted here that all alternatives or aspects explained above and all aspects defined by independent claims in the following claims may be used alone, i. without alternatives or purposes other than the intended alternative, purpose or independent claim. However, in other embodiments, two or more of the alternatives or aspects or independent claims may be combined with each other, and in other embodiments, all aspects or alternatives and all independent claims may be combined with each other.

The inventive encoded sound field description may be stored on a digital storage medium or a non-volatile storage medium, or may be transmitted over a transmission medium such as a wireless transmission medium or a wired transmission medium such as the Internet.

Although some aspects are described in the context of equipment, it is obvious that these aspects also represent a description of the corresponding method, with the block or device corresponding to a method step or a feature of a method step. Likewise, aspects described in the context of a method step also provide a description of the associated block or element, or feature of the associated equipment.

Depending on some implementation requirements, embodiments of the invention may be implemented in hardware or in software. The implementation may be performed using a digital storage medium, such as a floppy disk, DVD, CD, ROM, PROM, EPROM, EEPROM, or flash memory, having electronically readable control signals stored that interact (or are capable of interacting) with a programmable computer system in such a manner that the corresponding method is carried out.

Some embodiments of the invention comprise a storage medium having electronically readable control signals that are capable of interacting with a programmable computer system such that one of the methods described herein is implemented.

In general, embodiments of the present invention may be implemented as a computer program product with program code, wherein the program code is configured to perform one of the methods when the computer program product is running on the computer. The program code may, for example, be stored on a computer-readable medium.

Other embodiments comprise a computer program for carrying out one of the methods described herein stored on a computer-readable medium or non-volatile storage medium.

In other words, an embodiment of the inventive method is therefore a computer program having program code for carrying out one of the methods described herein when the computer program is running on a computer.

Therefore, a further embodiment of the inventive methods is a storage medium (digital storage medium or computer-readable medium) containing a recorded computer program for carrying out one of the methods described herein.

Therefore, an additional embodiment of the inventive method is a stream of data or a sequence of signals representing a computer program for implementing one of the methods described herein. The data stream or signal sequence, for example, may be configured to be transmitted over a data connection, such as the Internet.

An additional embodiment comprises processing means, such as a computer or programmable logic device, configured to perform one of the methods described herein.

An additional embodiment comprises a computer having a computer program installed to implement one of the methods described herein.

In some embodiments, a programmable logic device (eg, a field programmable gate array) may be used to perform some or all of the functionality of the methods described herein. In some embodiments, a user-programmable gate array may interface with a microprocessor to perform one of the methods described herein. In general, the methods are preferably carried out by any hardware.

The above described embodiments are merely illustrative of the principles of the present invention. It should be understood that modifications and changes to the settings and details described herein should be apparent to those skilled in the art. Therefore, they are intended to be limited only by the scope of the following claims, and not by the specific details provided through the description and explanation of the embodiments herein.

Bibliography

Alary, B., Politis, A. and Välimäki, V., 2017, "Velvet Noise Decorrelator".

Baumgarte, F. and Faller, C., 2003, "Binaural Cue Coding-Part I: Psychoacoustic Fundamentals and Design Principles", Speech and Audio Processing, IEEE Transactions on, 11(6), pp. 509-519.

Blauert, J., 2001, Spatial hearing (3 Ausg.). Cambridge; Mass: MIT Press.

Faller, C. and Baumgarte, F., 2003, "Binaural Cue Coding-Part II: Schemes and Applications", Speech and Audio Processing, IEEE Transactions on, 11(6), pp. 520-531.

Kendall, G. S., 1995, "The Decorrelation of Audio Signals and Its Impact on Spatial Imagery", Computer Music Journal, 19(4), pp. 71-87.

Lauridsen, H., 1954, "Experiments Concerning Different Kinds of Room-Acoustics Recording", Ingenioren, 47.

Pihlajamäki, T., Santala, O. and Pulkki, V., 2014, "Synthesis of Spatially Extended Virtual Source with Time-Frequency Decomposition of Mono Signals", Journal of the Audio Engineering Society, 62(7/8), p. 467-484.

Potard, G., 2003, "The study on sound source apparent shape and wideness".

Potard, G. and Burnett, I., 2004, "Decorrelation Techniques for the Rendering of Apparent Sound Source Width in 3D Audio Displays".

Pulkki, V., 1997, "Virtual Sound Source Positioning Using Vector Base Amplitude Panning", Journal of the Audio Engineering Society, 45(6), pp. 456-466.

Pulkki, V., 1999, "Uniform spreading of amplitude panned virtual sources".

Pulkki, V., 2007, "Spatial Sound Reproduction with Directional Audio Coding", J. Audio Eng. Soc, 55(6), pp. 503-516.

Pulkki, V., Laitinen, M.-V. and Erkut, C., 2009, "Efficient Spatial Sound Synthesis for Virtual Worlds".

Schlecht, S. J., Alary, B., Välimäki, V. and Habets, E. A., 2018, "Optimized Velvet-Noise Decorrelator".

Schmele, T. and Sayin, U., 2018, "Controlling the Apparent Source Size in Ambisonics Unisng Decorrelation Filters".

Schmidt, J. and Schröder, E. F., 2004, "New and Advanced Features for Audio Presentation in the MPEG-4 Standard".

Verron, C., Aramaki, M., Kronland-Martinet, R. and Pallone, G., 2010, "The 3-D Immersive Synthesizer for Environmental Sounds", Audio, Speech and Language Processing, IEEE Transactions on, title= "A Backward-Compatible Multichannel Audio Codec", 18(6), pp. 1550-1561.

Zotter, F. and Frank, M., 2013, "Efficient Phantom Source Widening", Archives of Acoustics, 38(1), pp. 27-37.

Zotter, F., Frank, M., Kronlachner, M. and Choi, J.-W., 2014, "Efficient Phantom Source Widening and Diffuseness in Ambisonics".

Claims (109)

1. Equipment for reproducing a spatially extended sound source having a given position and geometry in space, the equipment comprising: - interface (100) for receiving the position of the listener; a projector for calculating a projection of a two-dimensional or three-dimensional shell associated with a spatially extended sound source onto a projection plane using the position of the listener, information regarding the geometry of the spatially extended sound source, and information regarding the position of the spatially extended sound source; a sound position calculation module for calculating the positions of at least two sound sources for a spatially extended sound source using a projection plane; and - a rendering module for rendering at least two sound sources at said positions in order to obtain a reproduction of a spatially extended sound source having two or more output signals, while the rendering module is configured to use different audio signals for different positions, while different audio signals associated with a spatially extended sound source. 2. Equipment according to item 1, - in which the detector is configured to detect the instantaneous position of the listener in space using a tracking system, or the interface (100) is configured to use positional data entered through the interface. 3. Equipment according to claim 1, configured to receive a scene description, wherein the scene description contains information regarding a given position and information regarding a given geometry of a spatially extended sound source and at least one sound signal associated with a spatially extended sound source, - wherein the equipment further comprises a scene description parser (180) for parsing the scene description in order to extract information regarding a given position (341), information regarding a given geometry (331) and at least one signal (301, 302) of the sound source, or - while the description of the scene contains for a spatially extended sound source at least two basic sound signals (301, 302) and location information (321) for each basic sound signal regarding information regarding the geometry (331) of a spatially extended sound source, and at the same time the module (140) calculation of sound positions is configured to use the location information for at least two basic signals when calculating the positions of at least two sound sources using the projection plane. 4. Equipment according to item 1, - in which the projector (120) is configured to calculate the shell of the spatially extended sound source using information regarding the geometry (331) of the spatially extended sound source and project the shell towards the listener using the listener's position to obtain a projection of a two-dimensional or three-dimensional shell onto the projection plane , or - at the same time, the projector (120) is configured to project the geometry of the spatially extended sound source, given by information regarding the geometry (331) of the spatially extended sound source, towards the position of the listener and calculate the shell of the projected geometry in order to obtain a projection of a two-dimensional or three-dimensional shell onto a plane projections. 5. Equipment according to item 1, - in which the sound position calculation module (140) is configured to calculate the positions of sound sources in space from shell projection data and listener position. 6. Equipment according to item 1, - in which the sound position calculation module (140) is configured to calculate the position in such a way that at least two sound sources are peripheral sound sources and are located on the projection plane, or - at the same time, the sound position calculation module (140) is configured to calculate in such a way that the position of the peripheral sound source from the peripheral sound sources is located to the right of the projection plane relative to the listener, and / or to the left of the projection plane relative to the listener, and / or above the plane projection relative to the listener, and/or below the projection plane relative to the listener. 7. Equipment according to item 1, - in which the rendering module (160) is configured to render at least two sound sources using: - panning operations depending on the positions of the sound sources to receive speaker signals for a preset speaker setup, or - binaural rendering operations using the transfer functions of the human hearing aid, depending on the positions of the sources, in order to receive headphone signals. 8. Equipment according to item 1, - in which the first number of associated source signals is associated with a spatially extended sound source, and the first number is one or more than one, while the associated source signals are associated with the same spatially extended sound source, - at the same time, the module (140) for calculating sound positions determines the second number of sound sources used for rendering a spatially extended sound source, and the second number is greater than one, and - at the same time, the rendering module (160) contains one or more decorrelators (166) for generating a decorrelated signal from one or more signals (164) of the sources of the first number when the second number exceeds the first number. 9. Equipment according to item 1, - in which the interface (100) is configured to take a time-varying position of the listener in space, - while the projector (120) is configured to calculate a time-varying projection in space, - at the same time, the sound position calculation module (140) is configured to calculate the time-varying number of sound sources or the time-varying positions of sound sources in space, and - at the same time, the rendering module (160) is configured to render a time-varying number of sound sources or at least two sound sources at time-varying positions in space. 10. Equipment according to item 1, - in which the interface (100) is configured to take the position of the listener with six degrees of freedom, and - while the projector (120) is configured to calculate the projection depending on the six degrees of freedom. 11. Equipment according to claim 1, in which the projector (120) is configured to: - calculate the projection as the plane of the picture, for example, a plane perpendicular to the listener's field of view, or - calculate the projection as a spherical surface around the listener's head, or - calculate the projection as a projection plane located at a predetermined distance from the center of the listener's head, or - calculate the projection of the convex hull of a spatially extended sound source from the angle of azimuth and angle of elevation, extracted from spherical coordinates relative to the perspective of the listener's head. 12. Equipment according to item 1, - in which the sound position calculation module (140) is configured to calculate the positions in such a way that the positions are evenly distributed around the shell projection, either in such a way that the positions are placed at the extreme or peripheral points of the shell projection, or in such a way that the positions are located in horizontal or vertical extreme or peripheral points of the shell projection. 13. Equipment according to item 1, - in which the sound position calculation module (140) is configured to determine, in addition to the positions for peripheral sound sources, the positions for auxiliary sound sources located on or in front of, or behind, or within the projection of the shell relative to the listener. 14. Equipment according to item 1, - in which the projector (120) is configured to additionally compress the projection of the shell, for example, to the center of gravity of the shell or projection by a variable or predetermined amount, or by various variables or predetermined values in different directions, for example, in a horizontal direction and in vertical direction. 15. Equipment according to claim. 1, in which the module (140) calculation of the positions of the sound is made with the ability to calculate in such a way that at least one additional auxiliary sound source is located on the projection plane between the left peripheral sound source and the right peripheral sound source relative to the position of the listener , or - at the same time, the module (140) for calculating sound positions is configured to calculate in such a way that at least one additional auxiliary sound source is located on the projection plane between the left peripheral sound source and the right peripheral sound source relative to the listener's position, while one additional auxiliary source placed in the middle between the left peripheral sound source and the right peripheral sound source, or two or more additional auxiliary sources are placed at the same distance between the left peripheral sound source and the right peripheral sound source. 16. Equipment according to item 1, - in which the sound position calculation module (140) is configured to rotate the sound positions of the spatially extended sound source, preferably around the center of gravity of the projection, in case of receiving a circular motion of the listener around the spatially extended sound source through the interface, or in the case of receiving rotation of the spatially extended sound source relative to the stationary listener through the interface. 17. Equipment according to item 1, - in which the rendering module (160) is configured to receive, for each sound source, the opening angle depending on the distance between the listener and the sound source, and to render the sound source depending on the opening angle. 18. Equipment according to item 1, - in which the rendering module (160) is configured to receive distance information for each sound source, and - at the same time, the rendering module (160) is configured to render the sound source depending on the distance in such a way that the sound source placed closer to the listener is rendered with a higher volume compared to the sound source placed less close to the listener and having the same volume. 19. Equipment according to claim 1, wherein the sound position calculation module (140) is configured to: - determine, for each sound source, a distance equal to the distance of the spatially extended sound source relative to the listener, or - determine the distance of each sound source through the rear projection of the location of the sound source on the projection onto the geometry of the spatially extended sound source, and - at the same time, the rendering module (160) is configured to generate sound sources using information regarding the distance. 20. Equipment according to item 1, - in which the geometry information (331) is specified as a one-dimensional line or curve, a two-dimensional region such as an ellipse, rectangle or polygon or a group of polygons, or a three-dimensional body such as an ellipsoid, cuboid or polyhedron, and/or - in this case, the information is specified as a parametric description or a polygonal description or a parametric representation of a polygonal description. 21. Equipment according to item 1, - in which the sound position calculation module (140) is configured to determine the number of sound sources depending on the distance of the listener to the spatially extended sound source, while the number of sound sources is higher for a smaller distance compared to a lower number for a larger distance between the listener and spatially extended sound source. 22. Equipment according to claim. 1, configured to receive information regarding the spread introduced by a spatially extended sound source, and wherein the projector (120) is configured to apply a compression operation to the shell or projection using the spread information to at least partially compensate for the spread. 23. Equipment according to item 1, - in which the rendering module (160) is configured to render, in the case of positions of sound sources that are the same with each other within a given tolerance range, sound sources by combining basic signals associated with a spatially extended sound source, for example, using Givens rotation for in order to receive rotated basic signals and render rotated basic signals at positions. 24. Equipment according to item 1, - in which the rendering module (160) is configured to perform pre-processing or post-processing when generating at least two sound sources in accordance with a position- or direction-dependent characteristic. 25. Equipment according to item 1, - in which the spatially extended sound source has, as information regarding the geometry (331), the information that the spatially extended sound source is a spherical and ellipsoidal, linear, cuboid or piano-shaped spatially extended sound source. 26. Equipment according to claim 1, made with the ability to: - receiving a bitstream representing a compressed description for a spatially extended sound source, wherein the bitstream contains a bitstream element (311) indicating a first number of different audio signals for a spatially extended sound source included in the bitstream or encoded audio signal received by said equipment , where the number is one or more than one, - reading an element (311) of the bitstream and extracting a first number of different audio signals for a spatially extended audio source included in the bitstream or encoded audio signal, and - at the same time, the module (140) for calculating sound positions determines the second number of sound sources used for rendering a spatially extended sound source, and the second number is greater than one, and - at the same time, the rendering module (160) is configured to generate (164, 166), depending on the first number extracted from the bit stream, the third number of one or more decorrelated signals, the third number being extracted from the difference between the second number and the third number. 27. Equipment for generating a bit stream representing a compressed description for a spatially extended sound source, and the equipment contains: - a sound provider (200) for providing at least two different sound signals (301, 302) for a spatially extended sound source; - a geometry provider (220) for computing information (331, 341) regarding geometry for a spatially extended sound source; and - an output generator (240) for generating a bit stream representing a compressed description, the bit stream containing at least two different audio signals (301, 302), geometry information (331, 341) and individual location information (321) for each an audio signal from at least two different audio signals, the individual location information (321) indicating the location of the corresponding audio signal relative to the information regarding the geometry (331) of the spatially extended sound source. 28. The equipment of claim 27, wherein the geometry information (331, 341) comprises position information (341) indicating the position of the spatially extended sound source in space. 29. The equipment of claim 27, wherein the sound provider (200) is configured to: - record a natural sound source in one microphone position or orientation, or in multiple microphone positions or orientations, or - extract an audio signal from one basic signal or from several basic signals by means of one or more decorrelation filters. 30. Equipment according to clause 27, - in which the audio provider (200) is configured to compress at least two audio signals in terms of bit rate using an encoder (260) of audio signals, and - in this case, the output data generator (240) is configured to use bit rate compressed at least two audio signals (301, 302) for a spatially extended sound source. 31. The equipment of claim 27, wherein the geometry provider (220) is configured to extract, from the geometry of a spatially extended sound source, a parametric description or a polygonal description or a parametric representation of a polygonal description, and wherein the output generator (240) is configured to input, in the bitstream, a parametric description or a polygonal description or a parametric representation of a polygonal description as geometry information (331, 341). 32. Equipment according to claim 27, wherein the output data generator (240) is configured to insert, into the bitstream, a bitstream element (311) indicating the number of at least two different audio signals (301, 302) for a spatially extended source audio included in the bitstream or included in the encoded audio signal associated with the bitstream, where the number is two or more than two. 33. A method for reproducing a spatially extended sound source having a given position and geometry in space, the method comprising the steps of: - take the position of the listener; - calculate the projection of a two-dimensional or three-dimensional shell associated with a spatially extended sound source onto the projection plane using the position of the listener, information regarding the geometry (331) of the spatially extended sound source and information regarding the position (341) of the spatially extended sound source; - calculate the positions of at least two sound sources for a spatially extended sound source using the projection plane; and - rendering at least two sound sources in positions to obtain a reproduction of a spatially extended sound source having two or more output signals, while the rendering contains a step in which different sound signals are used for different positions, while different sound signals are associated with spatially extended sound source. 34. A method for generating a bitstream representing a compressed description for a spatially extended sound source, the method comprising the steps of: - provide at least two different audio signals for a spatially extended sound source; - provide information (331, 341) regarding the geometry for a spatially extended sound source; and - generating a bit stream representing a compressed description, the bit stream containing at least two different audio signals (301, 302), geometry information (331, 341) for a spatially extended sound source, and individual location information (321) for each audio signal of at least two different audio signals, the individual location information (321) indicating the location of the corresponding audio signal with respect to information regarding the geometry (331) of the spatially extended sound source. 35. The method of claim 34, wherein the geometry information (331, 341) for the spatially extended sound source comprises position information (341) of the spatially extended sound source in space. 36. The method of claim 34, wherein generating the bitstream comprises introducing, into the bitstream, a bitstream element (311) indicating the number of at least two different audio signals (301, 302) for a spatially extended sound source included in the bitstream or included in the encoded audio signal associated with the bitstream, and the number is two or more than two. 37. A physical storage medium having a compressed description stored on it for a spatially extended sound source, containing: - at least two different sound signals (301, 302) for a spatially extended sound source; - information (331, 341) regarding geometry for a spatially extended sound source, first location information (321) for a first sound signal (301) of at least two different sound signals (301, 302) indicating a first location of the first sound signal (301) with respect to information regarding geometry (331, 341) of the spatially extended sound source; and information (321) of the second location for the second sound signal (302) from at least two different sound signals (301, 302), indicating the second location of the second sound signal (302) with respect to information regarding the geometry (331 341) of the spatially extended sound source, and the second location information is different from the first location information. 38. The physical storage medium of claim 37, wherein the spatially extended sound source geometry information (331, 341) comprises position information (341) of the spatially extended sound source. 39. The physical storage medium of claim 37, further comprising a bitstream element (311) indicating the number of at least one other audio signal for a spatially extended audio source included in the bitstream or included in an encoded audio signal associated with the bitstream, wherein the number is one or more than one. 40. A physical storage medium having a computer program stored thereon for carrying out, when executed on a computer or processor, the method of claim 35. 41. A physical storage medium having a computer program stored thereon for carrying out, when executed on a computer or processor, the method of claim 36.

Images