RU2810920C2 - Audio processing in audio services with effect of presence - Google Patents

Abstract

FIELD: acoustics.

SUBSTANCE: immersive audio processing apparatus comprising or connected to a microphone system (302) comprising one or more microphones for capturing audio, the apparatus comprising: a receiving unit (304) configured to: receive (S13) directional audio (320) captured by the microphone system; receiving (S14) metadata (322) associated with the microphone system, wherein the metadata comprises spatial data of the microphone system, wherein the spatial data characterizes the spatial orientation and/or spatial position of the microphone system and comprises at least one of a list including: angle (angles) of azimuth, pitch, roll and spatial coordinates of the microphone system; a computing unit (306) configured to: modify at least some portion of the directional sound to produce a modified directional sound, wherein the directionality of the sound is modified based on the spatial orientation and/or spatial position of the microphone system; downmixing the modified directional audio based on the spatial orientation of the microphone system using a downmix matrix; encoding the modified directional audio and downmix matrix into digital audio data (328); a transmission unit (308) configured to transmit digital audio data.

EFFECT: obtaining audio signals with the effect of presence.

28 cl, 7 dwg

Description

P16835370RU

AUDIO PROCESSING IN INVENTIVE AUDIO SERVICES

Technical field

The invention disclosed herein generally relates to the capture, audio pre-processing, encoding, decoding and reproduction of directional sound from a sound stage. In particular, it relates to a device configured to modify the directivity of captured directional sound based on spatial data of the microphone system capturing the directional sound. The invention also relates to a reproducing device configured to modify the directionality of received directional sound based on received spatial data.

BACKGROUND OF THE INVENTION

The introduction of high-speed 4G/5G wireless access into telecommunications networks, coupled with the availability of increasingly powerful hardware platforms, has provided the foundation for faster and easier adoption of advanced communications and multimedia services than ever before.

The 3rd Generation Partnership Project (3GPP) Enhanced Voice Services (EVS) codec has provided highly significant improvements in user experience with the introduction of ultra-wideband (SWB) and full-band (FB) speech and audio coding, along with improved packet loss resiliency. However, increased audio bandwidth is just one of the characteristics required for true presence. Resource-efficient immersion of the user into a believable virtual world theoretically requires support beyond the single- and multi-channel recording currently provided by EVS.

In addition, the currently approved audio codecs in 3GPP provide suitable quality and compression for stereo content, but do not have the speech elements (eg low enough latency) required for voice and teleconferencing. These codecs also lack the multi-channel functionality that is required for immersive services such as real-time or user-initiated content streaming, virtual reality (VR), and immersive teleconferencing.

The development of an extension to the EVS codec has been proposed for immersive voice and audio services (IVAS) to fill this technology gap and meet the increasing demand for feature-rich multimedia services. In addition, the IVAS codec, used as an enhanced speech codec that supports multi-stream coding (eg, channel-based, object-based and scene-based audio), will be preferred for teleconferencing applications over 4G/5G. Use cases for this next-generation codec include, but are not limited to, voice-to-conversation, multi-stream teleconferencing, content streaming to VR conversations, and user-initiated real-time and non-real-time.

Thus, IVAS is expected to provide users with immersive VR, AR and/or XR experiences. In many of these applications, the device (eg, a mobile phone) capturing directional (immersive) sound may, in many cases, move during the session relative to the acoustic stage, causing rotation and/or translational movement in the space of the captured sound stage. This behavior may or may not be required depending on the type of experience being provided, such as immersive, VR, AR or XR, and depending on the specific use case. For example, the listener may feel uncomfortable if the scene being played rotates whenever the gripper is rotated. In the worst case, it can cause motion sickness.

Therefore, there is a need for improvement within this context.

Brief description of graphic materials

The following is a description of the exemplary embodiments with reference to the accompanying drawings, in which:

in fig. 1 illustrates a method for encoding directional audio according to embodiments;

in fig. 2 illustrates a method for reproducing directional sound according to embodiments;

in fig. 3 shows an encoding device configured to implement the method of FIG. 1, according to embodiments;

in fig. 4 shows a reproducing device configured to implement the method of FIG. 2, according to embodiments;

in fig. 5 shows a system containing the devices of FIG. 3 and fig. 4, according to embodiments;

in fig. 6 illustrates a VR physical conference scenario according to embodiments;

in fig. 7 illustrates a virtual conference space according to embodiments.

All figures are schematic and generally show only those parts necessary to explain the invention, while other parts may be omitted or simply implied. Unless otherwise indicated, in different figures, like parts are designated by like reference numerals.

Detailed description

Thus, with the foregoing in mind, it is an object to provide devices and associated capture, audio pre-processing and/or encoding techniques for compensating for unwanted surround sound stage movements that may be caused by unintentional movements of a microphone system capturing directional sound. It is yet another object to provide a corresponding decoder and/or playback device and associated methods for decoding and reproducing directional audio. Systems are also provided, comprising, for example, an encoder and a reproducing device.

I. Review – sending side

According to a first aspect, there is provided a device comprising or connected to a microphone system comprising one or more microphones for capturing sound. The device (also referred to herein as a sending side or a capture device) includes a receiving unit configured to:

receiving directional sound captured by the microphone system;

receiving metadata associated with the microphone system, wherein the metadata comprises spatial data of the microphone system, wherein the spatial data characterizes the spatial orientation and/or spatial position of the microphone system and contains at least one of a list including: azimuth, pitch, roll and spatial coordinates of the microphone system.

As used herein, the term "directional audio" generally refers to immersive audio, that is, audio captured by directional microphone systems that can capture sounds, including the directions from which they originate. Playing directional sound provides a natural three-dimensional sound experience (binaural playback). Audio, which may contain audio objects and/or channels (eg representing scene-based audio in the "Ambisonics B" format or channel-based audio), is thus associated with the directions from which it is received. In other words, directional sound originates from directional sources and is incident from a direction of arrival (DOA), represented by azimuth and elevation angles, for example. At the same time, it is assumed that the diffuse ambient sound is omnidirectional, that is, spatially constant or spatially uniform. Other expressions that may be used in relation to the feature "directional audio" include "spatial audio", "spatial audio", "immersive audio", "immersive audio", "stereo" and "surround audio" "

As used herein, the term “spatial coordinates” generally refers to the spatial position of a microphone system or pickup device in space. Cartesian coordinates are one implementation of spatial coordinates. Other examples include cylindrical or spherical coordinates. It should be noted that position in space can be relative (for example, coordinates in a room or relative to another device/unit, etc.) or absolute (for example, GPS coordinates or the like).

As used herein, the term “spatial data” generally describes either the current angular orientation and/or spatial position of the microphone system, or a change in angular orientation and/or spatial position from a previous orientation/position of the microphone system.

Thus, the device receives metadata containing spatial data characterizing the spatial orientation and/or spatial position of the microphone system capturing directional sound.

The device further comprises a computing unit configured to modify at least some portion of the directional sound to produce a modified directional sound, wherein the directionality of the sound is modified based on the spatial orientation and/or spatial position of the microphone system.

The modification may be accomplished by any suitable means, for example, by determining a rotation/transition matrix based on the spatial data and multiplying the directional audio by this matrix to obtain the modified directional audio. Matrix multiplication is suitable for non-parametric spatial audio. Parametric spatial audio can be modified by adjusting spatial metadata, such as the directional parameters of the audio object(s).

The modified directional audio is then encoded into digital audio data, and this data is transmitted via the device's transmission unit.

The authors concluded that the rotational/translational movements of the sound-capturing device (microphone system) are best compensated at the sending end, that is, at the sound-capturing end. This can be expected to provide the best possible stabilization of the captured soundstage with respect to, for example, random movements. Such compensation may be part of the capture process, i.e. occurring during audio pre-processing, or part of the IVAS encoding stage. Moreover, by performing compensation at the sending end, the need to transfer spatial data from the sending end to the receiving end is reduced. In the event that compensation for the rotational/translational movements of the sound-capturing device had to be performed at the sound receiver, then all spatial data had to be transmitted to the receiving end. Assuming that all rotation coordinates on all three axes are represented by 8 bits and estimated and transmitted at 50 Hz, the resulting bit rate is 1.2 kbps. A similar assumption can be made for the spatial coordinates of the microphone system.

In some embodiments, the spatial orientation of the microphone system is represented by parameters describing the rotational motion/orientation using one degree of freedom, DoF, in the spatial data. For example, it may be sufficient to consider only the azimuth angle for a conference call.

In some embodiments, the spatial orientation of the microphone system is represented by parameters describing the angular orientation/motion using three degrees of freedom, DoF, in the spatial data.

In some embodiments, the spatial data of the microphone system is represented in six DoFs. In this embodiment, the spatial data of the microphone system captures the changed position (herein referred to as spatial coordinates) of the microphone system as translational movement along three perpendicular axes forward/backward (longitudinal), up/down (vertical), left/right (transverse), along with changes in the orientation (or current angular orientation) of a microphone system due to rotation about three perpendicular axes, often called yaw or azimuth (normal/vertical axis), pitch (transverse axis), and roll (longitudinal axis).

In some embodiments, the received directional audio comprises audio containing the directional metadata. For example, such sound may contain audio objects, that is, object-based audio (OBA). OBA is a parametric form of spatial/directional audio with spatial metadata. A special form of parametric spatial audio is metadata-assisted spatial audio (MASA).

In some embodiments, the computing unit is further configured to encode at least portions of the metadata comprising spatial data of the microphone system into said digital audio data. Preferably, this allows compensation for directional corrections made to the captured audio at the receiving end. In accordance with the definition of a suitable rotational coordinate system, for example, where the z-axis corresponds to the vertical direction, in many cases it may be necessary to transmit only the azimuth angle (at a speed of, for example, 400 bps). The pitch and roll angles of the gripper in a rotary coordinate system may only be necessary in certain VR applications. By compensating for the spatial data of the microphone system on the sending side and including, under certain conditions, at least portions of the spatial data in the encoded digital audio data, the case in which the reproduced sound stage should not change due to the position of the capture device, and other cases in which the reproduced The sound stage should rotate with corresponding movements of the gripping device.

According to some embodiments, the receiving unit is further configured to receive first commands instructing the computing unit whether to include at least portions of metadata comprising spatial data of the microphone system in said digital audio data, causing the computing unit to act accordingly. Therefore, the sending side under certain conditions includes parts of the spatial data in the digital audio data so that the bit rate is preserved as much as possible. The command may be received more than once during a session, so whether spatial data (parts thereof) should be included in the digital audio data or not changes over time. In other words, adaptations may be provided during the session, where the first commands may be received by the device either continuously or periodically. Constantly means, for example, once every frame. Periodically there may be only one time when a new command needs to be issued. It is also possible to receive the first command only once when establishing a session.

In some embodiments, the receiving unit is further configured to receive second commands instructing the computing unit which spatial data parameter or parameters of the microphone system to include in the digital audio data, causing the computing unit to act accordingly. As in the example above, the sending side can be commanded to include only the azimuth or to include all data defining the spatial orientation of the microphone system. A command may be received more than once during a session, so the number of parameters included in the digital audio data changes over time. In other words, adaptations may be provided during a session where second commands may be received by the device either continuously or periodically. Constantly means, for example, once every frame. Periodically there may be only one time when a new command needs to be issued. It is also possible to receive the second command only once when establishing a session.

In some embodiments, the transmission unit is configured to transmit digital audio data to an additional device, wherein instructions regarding first and/or second commands are received from said additional device. In other words, the receiving side (containing a playback device for reproducing the received decoded audio), depending on the context, can send commands to the sending side about whether or not to include a portion of the spatial data in the digital audio data and/or what parameters to include. In other embodiments, instructions regarding the first and/or second commands may be received, for example, from a coordinating unit (call server) for an immersive multi-user audio/video conference or any other unit that is not directly involved in the reproduction of directed audio.

According to some embodiments, the receiving unit is further configured to receive metadata comprising a timestamp indicating the time of capture of the directional audio, wherein the computing unit is configured to encode said time stamp into said digital audio data. Preferably, this timestamp may be used for synchronization at the receiving end, for example to synchronize an audio playback device with a video playback device or to synchronize multiple digital audio data received from different capture devices.

In some embodiments, encoding the modified audio signals includes downmixing the modified directional audio, wherein the downmixing is performed taking into account the spatial orientation of the microphone system, and encoding the result of the downmix and the downmix matrix used in the downmix into said digital audio data. For example, shaping a sound beam towards a particular directional directional sound source is preferably applied based on the directional modification made to the directional sound.

In some embodiments, the device is implemented in virtual reality, VR, or augmented reality, AR hardware, comprising a microphone system and a head tracking device configured to detect 3-6 DoF spatial data of the device. In other embodiments, the device is implemented in a mobile phone comprising a microphone system.

II. Overview - receiving side

According to a second aspect, a device for reproducing audio signals is provided. The device (also referred to herein as a receiving end or reproducing device) includes a receiving unit configured to receive digital audio data. The device further comprises a decoding unit configured to decode the received digital audio data into directional audio and into metadata, wherein the metadata comprises spatial data comprising at least one of a list including: azimuth, pitch, roll angle(s), and spatial coordinates. The spatial data may be received, for example, in the form of parameters such as 3 DoF angles. In other embodiments, the spatial data may be received in the form of a rotation/transition matrix.

The device additionally contains a playback unit configured to:

modifying the directionality of directional sound using spatial rotation data; And

reproduction of modified directional sound.

Preferably, the device according to this aspect can modify the directional audio as indicated in the metadata. For example, playback may take into account the movements of the device capturing the sound.

In some embodiments, the spatial data indicates the spatial orientation and/or spatial position of a microphone system comprising one or more directional sound pickup microphones, wherein the rendering unit modifies the directionality of the directional sound to at least partially reproduce the audio environment of the microphone system. In this embodiment, the device applies sound stage rotation by reapplying at least portions of the sound stage rotation (relative, that is, the stage rotation occurs relative to the movement of the microphone system) that was compensated in the gripping device.

In some embodiments, the spatial data includes parameters describing rotational motion/orientation using one degree of freedom, DoF.

In some embodiments, the spatial data includes parameters describing rotational motion/orientation using three degrees of freedom, DoF.

In some embodiments, the decoded directional audio comprises audio containing directional metadata. For example, the decoded directional audio may contain audio objects, that is, object-based audio (OBA). In other embodiments, the decoded directional audio may be channel-based and be, for example, Ambisonics B-format stage-based audio or channel-based audio.

In some embodiments, the device includes a transmission unit configured to transmit commands to an additional device from which the digitized audio is received, wherein the commands indicate to the additional device what (if any) parameter or parameters the rotation data should contain. Therefore, the rendering device may send commands to the capture device to transmit, for example, only the rotation parameters, only the azimuth parameter, or all 6 DoF parameters, depending on the application and/or available bandwidth. Moreover, the playback device may make such a decision based on the available computing resources in the playback device to apply the sound stage rotation or complexity level of the playback block. Commands can be issued more than once during a session, and thus they change over time, that is, based on the above. In other words, adaptations may be made during a session where the device may transmit commands either continuously or periodically. Constantly means, for example, once every frame. Periodically there may be only one time when a new command needs to be issued. It is also possible to send the command only once when establishing a session.

According to some embodiments, the decoding unit is further configured to extract a timestamp indicating the time of directional audio capture from the digital audio data. This timestamp can be used for synchronization purposes, as discussed above.

According to some embodiments, decoding the received digital audio data into directional audio by the decoding unit includes:

decoding the received digital audio data into downmixed audio;

upmixing by a unit for decoding the downmixed audio into directional audio using a downmix matrix included in the received digital audio data.

In some embodiments, the spatial data comprises spatial coordinates, and wherein the playback unit is further configured to adjust the volume of the reproduced audio based on the spatial coordinates. In this embodiment, the volume of sound received "from afar" may be reduced compared to the volume of sound received from a location closer. It should be noted that the relative proximity of the received sound can be determined based on virtual space, with the position of the capturing device in that space relative to the receiving device determined based on the spatial coordinates of the devices using a suitable distance metric, such as the Euclidean metric. Another step may include using some arbitrary mapping scheme to determine audio reproduction parameters, such as audio level, based on the distance metric. Preferably, in this embodiment, the presence effect of the reproduced sound can be improved.

In some embodiments, the device is implemented in virtual reality, VR, or augmented reality, AR hardware, comprising a head tracking device configured to measure the spatial orientation and spatial position of the device in six DoF. In this embodiment, also the spatial data of the playback device can be used when modifying the directivity of the directional sound. For example, the received rotation/transition matrix can be multiplied by a similar matrix defining, for example, the rotation state of the playback device, and the resulting matrix can then be used to modify the directivity of the directional sound. Preferably, in this embodiment, the presence effect of the reproduced sound can be improved. In other embodiments, the device is implemented in a device for telephone conferencing or the like, which is intended to be stationary, and any rotation state of the device is neglected.

In some embodiments, the playback unit is configured for binaural audio playback.

III. Overview – system

According to the third aspect, a system is proposed comprising:

a first device according to a first aspect, configured to transmit digital audio data to a second device according to a second aspect, wherein the system is configured to conduct audio and/or video conferencing.

In some embodiments, the first device further comprises a video recording unit, wherein the first device is configured to encode the recorded video into digital video data and transmit the digital video data to the second device, wherein the second device further includes a display for displaying the decoded digital video data.

According to the fourth aspect, a system is proposed comprising:

a first device according to the first aspect, configured to transmit digital audio data to a second device, wherein the second device comprises:

a receiving unit configured to receive digital audio data,

a decoding unit configured to:

decoding the received digital audio data into directional audio and metadata, the metadata comprising spatial data comprising at least one of a list including: azimuth, pitch, roll angle(s), and spatial coordinates;

a playback unit for playing sound;

wherein the playback unit is configured to, upon subsequent receipt by the second device of encoded video data from the first device:

modifying the directivity of directional sound using spatial data, and

reproduction of modified directional sound;

wherein the playback unit is configured to: if the second device does not receive encoded video data from the first device:

directional sound reproduction.

Preferably, the decision whether to reproduce the sound environment of the microphone system by compensating for the spatial orientation and/or spatial position of the microphone system is made based on whether video is transmitted or not. In this embodiment, the sending device may not always know when compensation for its movement is needed or required. Consider, for example, a situation where audio is played along with video. In this case, at least when video capture is performed by the same device that captures audio, it may preferably be possible to either rotate the sound stage along with the moving visual scene, or keep the sound stage stationary. Keeping the sound stage static by compensating for the movement of the gripper may be a preferable choice if video is not used.

According to a fifth aspect, there is provided a non-transitory computer-readable medium on which instructions are stored that, when executed by one or more processors, cause the one or more processors to perform operations according to any of the first to fourth aspects.

IV. Overall overview

Aspects two through five may have generally the same or similar features and benefits as the first aspect.

Other objects, features and advantages of the present invention will become apparent from the following detailed description, the dependent claims of the appended claims, and the drawings.

The steps of any method or device implementing a number of steps that are disclosed herein do not necessarily have to be performed in the exact order given, unless explicitly stated otherwise.

V. Exemplary Embodiments

Immersive voice and audio services are expected to offer immersive virtual reality (VR) experiences to users. Augmented reality (AR) and extended reality (XR) experiences may also be offered. This invention relates to the fact that mobile devices, such as a portable UE, that capture an immersive scene or an AR/VR/XR scene can, in many cases, move during the session relative to the sound stage. Here, cases are highlighted where it is necessary to avoid reproducing the rotational movements of the gripping device as a corresponding rotation of the reproduced scene by the receiving device. This invention is directed to how the above problems can be effectively solved while meeting user requirements for context-sensitive immersive audio.

It should be noted that while some examples herein will be described in the context of an IVAS encoder, decoder and/or playback device, it should be understood that this is only one type of encoder/decoder/playback device in which the general principles of the invention, and that there may be many other types of encoders, decoders and playback devices that may be used in combination with the various embodiments described herein.

It should also be noted that although the terms “upmixing” and “downmixing” appear throughout this document, they may not necessarily imply increasing and decreasing the number of channels, respectively. While this may often be true, it should be understood that either term can refer to either a decrease or an increase in the number of channels. Thus, both terms are included in the more general concept of "mixing".

Next, with reference to FIG. 1 describes a method 1 for encoding and transmitting a directional audio representation according to one embodiment. An apparatus 300 configured to implement method 1 is shown in FIG. 3.

The device 300 may generally be a mobile phone (smartphone), however, the device may also be a piece of VR/AR/XR equipment or any other type of device comprising a microphone system 302 containing one or more microphones for capturing directional audio, or connected with her. Thus, device 300 may include a microphone system 302 or be connected (via a wired or wireless connection) to a microphone system 302 located remotely therefrom. In some embodiments, device 300 is implemented in VR hardware or AR hardware comprising a microphone system 302 and a head tracker configured to detect 1-6 DoF spatial data of the device.

In some audio capture scenarios, the position and/or spatial orientation of the microphone system 302 may change during directional audio capture.

Next, two scenarios will be described as examples.

Changing the position and/or spatial orientation of the microphone system 302 during audio capture may cause a rotation/transition in the space of the rendered scene at the receiving device. This behavior may or may not be required depending on the type of experience being provided, such as immersive, VR, AR or XR, and depending on the specific use case. One example of where this may be required is a situation in which the service additionally provides a visual component and in which the capture camera (eg, 360-degree video capture, not shown in FIG. 1) and microphones 302 are integrated into the same device. . In this case, it is expected that rotation of the gripping device should result in a corresponding rotation of the reproduced audiovisual scene.

On the other hand, if audio and video are not captured by the same physical device, or if there is no video component, the listener may feel uncomfortable if the playing scene rotates every time the capture device is rotated. In the worst case, it can cause motion sickness. Thus, it is necessary to compensate for changes in position (transition and/or rotation) of the gripping device. Examples include applications for immersive telephony and immersive conferencing using a smartphone as a capture device (ie, containing a set of microphones 302). In these applications, it can often happen that the microphone array is unintentionally moved, either because it is portable or because the user touches it during operation. The user of the gripping device may not be aware that moving the device may cause instability in the reproduced spatial audio in the receiving devices. In general, the user cannot be expected to hold the phone still during a call.

The methods and apparatus described below are defined by some or all of the above scenarios.

Thus, device 300 contains or is connected to a microphone system 302 containing one or more microphones for capturing audio. Thus, a microphone system can contain 1, 2, 3, 5, 10, etc. microphones. In some embodiments, the microphone system includes a plurality of microphones. The device 300 contains a plurality of functional blocks. Blocks may be implemented in hardware and/or software and may contain one or more processors to perform the functions of the blocks.

The device 300 includes a receiving unit 304 that is configured to receive S13 directional audio 320 captured by the microphone system 302. The directional audio 320 is preferably a representation of audio that readily allows for rotation and/or transition of the sound stage. Directional audio 320 may include, for example, audio objects and/or channels that provide the ability to rotate and/or transition the sound stage. Directional sound may contain:

- channel-based audio (CBA), such as stereo, multi-channel/surround, 5.1, 7.1, etc.;

- scene-based sound (SBA), such as first-order and higher-order ambiophony;

- object based audio (OBA).

CBA and SBA are non-parametric forms of spatial/directional audio, while OBA is parametric with spatial metadata. A special form of parametric spatial audio is metadata-assisted spatial audio (MASA).

The receiving unit 304 is further configured to receive S14 metadata 322 associated with the microphone system 302. The metadata 322 includes spatial data of the microphone system 302. The spatial data characterizes the spatial orientation and/or spatial position of the microphone system 302. The spatial data of the microphone system includes at least one from a list including: azimuth, pitch, roll angle(s) and spatial coordinates of the microphone system. Spatial data can be represented by one degree of freedom, DoF (for example, only the azimuth angle of the microphone system), three DoF (for example, the spatial orientation of the microphone system in 3 DoF), or six DoF (both the spatial orientation in 3 DoF and the spatial position in 3 DoF). DoF). Of course, spatial data can be represented by any number of DoFs from one to six.

The device 300 further includes a computing unit 306 that receives the directional audio 320 and metadata 322 from the receiving unit 304 and modifies S15 at least some of the directional audio 320 (eg, at least some of the directional audio objects) to produce the modified directional audio. Such modification causes the directionality of sound to be modified based on the spatial orientation and/or spatial position of the microphone system.

The computing unit 306 then encodes the digital data S16 by encoding the modified directional audio S17 into the digital audio data 328. The device 300 further includes a transmission unit 310 configured to transmit (via a wired or wireless connection) the digital audio data 328, such as a bitstream.

By compensating for rotational and/or translational movements of the microphone system 302 as early as the encoder 300 (can also be referred to as the sending device, the capturing device, the transmitting device, the sending side), the requirements for transmitting spatial data of the microphone system 302 become less stringent. If such compensation were to be performed by a device receiving encoded directional audio (eg, an immersive audio device), then all required metadata would need to be included in the digital audio data 328 at all times. Assuming that all rotation coordinates of microphone system 302 on all three axes are represented by 8 bits and estimated and transmitted at 50 Hz, the resulting increase in the bit rate of signal 332 is 1.2 kbps. In addition, it is likely that variations in the auditory scene in the absence of motion compensation on the acquisition side may make spatial audio encoding more difficult and possibly less efficient.

Moreover, since the information on which the modification decision is based is readily available in the device 300, it is already suitable for compensating for rotational/translational movements of the microphone system 302, which can thus be performed efficiently. In this way, the maximum algorithmic latency for this operation can be reduced.

Another advantage is that by constantly compensating (rather than conditionally on demand) for rotational/translational movements in the gripper 300 and providing the receiving ends with knowledge of the spatial orientation of the gripper system, possible conflicts are eliminated when servicing multiple endpoints with different playback requirements, such as in multi-party conference applications.

The above covers all cases in which the reproduced sound stage should not change depending on the position and rotation of the microphone system 302 capturing directional sound. For other cases in which the reproduced sound stage must be rotated by corresponding movements of the microphone system 302, the computing unit 306 may optionally be configured to encode S18 at least portions of the metadata 322 containing the spatial data of the microphone system into said digital audio data 328. For example, in accordance with the determination of a suitable rotational coordinate system, for example one in which the z-axis corresponds to the vertical direction, in many cases it may be necessary to transmit only the azimuth angle (at a speed of, for example, 400 bps). The pitch and roll angles of the microphone system 302 in a rotary coordinate system may only be necessary in certain VR applications.

Provided under certain conditions, turn/transition parameters can typically be conveyed as a single conditional element of the IVAS RTP payload format. Thus, these parameters will require a small portion of the allocated bandwidth.

To accommodate different scenarios, the receiving unit 304 may optionally be configured to receive commands S10 regarding how to operate on the metadata 322 when the computing unit 306 encodes the digital audio data 328. The commands may be received by S10 from a playback device (such as the other party in an audio conference) or from a coordination device such as a call server or the like.

In some embodiments, the receiving unit 304 is further configured to receive S11 first commands instructing the computing unit 306 whether to include at least portions of the metadata 322 containing microphone system spatial data in said digital audio data. In other words, the first instructions tell the device 300 whether to include any metadata in the digital audio data 328 or not at all. For example, if device 300 transmits digital audio data 328 as part of an audio conference, then the first commands may determine that no portion of the metadata 322 needs to be included.

Alternatively or in addition, in some embodiments, the receiving unit 304 is further configured to receive second instructions instructing the computing unit which microphone system spatial data parameter or parameters to include in the digital audio data, causing the computing unit to act accordingly. For example, due to bandwidth or other reasons, the second instructions may direct the computing unit 306 to include only the azimuth angle in the digital audio data 328.

The first and/or second commands may typically be negotiated upon session establishment. Thus, none of these commands require transmissions during the session and will not require any dedicated bandwidth, for example, for immersive audio/video conferencing.

As noted above, device 300 may be part of a video conference. Therefore, the receiving unit 304 may also be configured to receive metadata (not shown in FIG. 1) containing a timestamp indicating the time the directional audio was captured, wherein the computing unit 306 is configured to encode said time stamp into said digital audio data. Preferably, the modified directional audio can then be synchronized with the captured video on the playback side.

In some embodiments, encoding S17 of the modified directional audio includes downmixing the modified directional audio, wherein the downmixing is performed taking into account the spatial orientation of the microphone system 302, and encoding the result of the downmix and the downmix matrix used in the downmix into said digital audio data 328. The downmixing may, for example, include adjusting the directional sound beamforming operation 320 based on the spatial data of the microphone system 302.

Thus, digital audio data is transmitted to S19 from device 300 as part of a transmission of, for example, an immersive audio/video conference scenario. The digital audio data is then received by the device to reproduce audio signals, such as the reception portion of an immersive audio/video conference scenario. The playback device 400 will now be described with reference to FIG. 2 and 4.

The device 400 reproducing audio signals includes a receiving unit 402 configured to receive S21 (via a wired or wireless connection) digital audio data 328.

Device 400 further includes a decoding unit 404 configured to decode S22 the received digital audio data 328 into directional audio 420 and metadata 422, wherein the metadata 422 includes spatial data comprising at least one of a list including: azimuth angle(s), pitch, roll and spatial coordinates.

In some embodiments, decoding block 404 performs upmixing. In these embodiments, decoding the received digital audio data 328 into directional audio 420 by decoding unit 404 includes: decoding the received digital audio data 328 into downmixed audio and upmixing by decoding unit 404 of the downmixed audio into directional audio 420 using the downmix matrix included in received digital audio data 328.

The device further includes a playback unit 406 configured to modify S23 the directivity of the directional sound using spatial data; and reproducing S24 the modified directional sound 424 through speakers or headphones.

Thus, the device 400 (its playback unit 406) is configured to apply sound stage rotation/transition based on the received spatial data.

In some embodiments, the spatial data indicates the spatial orientation and/or spatial position of a microphone system comprising one or more directional sound pickup microphones, wherein the rendering unit modifies S23 the directionality of the directional sound to at least partially reproduce the audio environment of the microphone system. In this embodiment, device 400 reapplies at least part of the soundstage rotation that was compensated at the capture end by device 300 of FIG. 3.

The spatial data may comprise spatial data containing rotation data representing three degrees of freedom, DoF, motion. Alternatively or in addition, the spatial data may include spatial coordinates.

The decoded directional audio may, in some embodiments, comprise audio objects or, more generally, audio associated with spatial metadata, as described above.

Decoding S22 of received digital audio data into directional audio by decoding unit 404 may, in some embodiments, include decoding the received digital audio data into downmixed audio and upmixing by decoding unit 404 of downmixed audio into directional audio using a downmix matrix included in the received digital audio data. 328.

To provide increased flexibility and/or meet bandwidth requirements, device 400 may include a transmitter 306 configured to send commands S20 to an additional device from which digital audio data 328 is received, wherein the commands indicate to the additional device which (if any) in general) the parameter or parameters must contain rotation or transition data. Thus, this feature may help satisfy possible user preferences or preferences related to playback and/or type of service used.

In some embodiments, device 400 may also be configured to transmit commands instructing the additional device to include metadata containing spatial data in digital audio data 328 or not. In these embodiments, if the digital audio data 328 received by S21 does not contain any such metadata, then the rendering unit will reproduce the decoded directional audio as received (possibly upmixed as described above) without any modification of the directionality of the directional audio due to compensations made in the gripping device 300. However, in some embodiments, the received directional audio is modified based on head tracking information from the playback device (as further described below).

The device 400, in some embodiments, may be implemented in VR hardware or AR hardware comprising a head tracking device configured to measure the spatial orientation of the device in six DoF. The playback unit 406 may be configured for binaural audio playback.

In some embodiments, the playback unit 406 is configured to adjust S25 the volume of the reproduced audio based on the spatial coordinates received in the metadata. This feature will be further described below with reference to FIG. 6–7.

In fig. 5 shows a system comprising a capturing device 300 (described with reference to FIG. 3) and a rendering device 400 (described with reference to FIG. 4). Capture device 300 may, in some embodiments, receive S10 commands 334 transmitted to S20 from playback device 400 indicating whether and to what extent capture device 300 should include spatial data from the capture device's microphone system in digital audio data 328.

In some embodiments, the capture device 300 further includes a video recording unit and is configured to encode the recorded video into digital video data 502 and transmit the digital video data to the playback device 400, wherein the playback device 400 further includes a display for displaying the decoded digital video data.

As described above, changing the position and/or spatial orientation of the microphone system of the capturing device 300 during audio capture may cause a rotation/transition in the space of the rendered scene in the rendering device 400. Such behavior may be required or undesirable depending on the type of experience being provided, e.g. immersive, VR, AR or XR, and depending on the specific application. One example of where this may be required is a situation in which the service additionally provides a visual component 502 and in which the capture camera and one or more microphones 302 are integrated into the same device. In this case, it is expected that rotation of the gripping device 300 should result in a corresponding rotation of the rendered audiovisual scene in the playback device 400.

On the other hand, if audio and video are not captured by the same physical device, or if there is no video component, the listener may feel uncomfortable if the playback scene rotates whenever the capture device 300 is rotated. In the worst case, motion sickness may be caused. .

Therefore, in some embodiments, the playback unit of the playback device 400 may be configured, when the playback device 400 subsequently receives encoded video data 502 from the capture device 300, to modify the directionality of the directional audio (received in the digital audio data 328) using the spatial data and playback the modified directional audio. .

However, if the playback device 400 does not receive encoded video data from the capture device 300, the playback unit of the playback device 400 can be configured to play directional audio without any directional modification.

In other embodiments, the playback device 400 receives information before the conference that a video component will not be included in the data received from the capture device 300. In this case, the playback device 400 may indicate in commands 334 that the digital audio data 328 should not include any spatial data of the microphone system of the capture device 300 at all, and the playback unit of the playback device 400 is configured to reproduce the directional audio received in digital audio data 328, without any directional modification.

Downmixing and/or encoding of directional audio in a capture device has been briefly described above. This will be further discussed in detail below.

In many cases, the capture device 300 does not contain information about whether the decoded representation (at the playback device) will be output to a single mono speaker, stereo speakers, or headphones. The actual playback scenario may also change during a service session, for example, as the connected playback equipment may change, such as when headphones are connected to or disconnected from a mobile phone. Another scenario in which the capabilities of the rendering device are not known is when a single capture device 300 must support multiple endpoints (playing devices 400). For example, in an IVAS conference or VR content distribution application, one endpoint could use a headset and the other could play through stereo speakers, but it would be preferable if the same encoding result could be sent to both endpoints as this would reduce would be more complex on the encoding side and could also reduce the total network throughput required.

The simplest, but less desirable, way to provide support in these cases is to always assume the lowest capability of the receiving device, which is mono, and select the appropriate audio mode. However, it is more practical that the codec used (eg the IVAS codec), even when used in a presentation mode supporting spatial, binaural or stereo audio, can always generate a decoded audio signal that can be presented on the device 400 with relatively more low sound reproduction capability. In some embodiments, the signal encoded as a spatial audio signal may also be decodable for binaural, stereo, and/or monaural playback. Likewise, a signal encoded as binaural may be decoded as stereo and/or monaural, and a signal encoded as stereo may be decoded as mono. By way of illustration, capture device 300 should be able to provide only one encoding result (digital audio data 328) and send that same encoding result to multiple end devices 400, some of which may support binaural presentation and some of which may only support stereo presentation.

It should be noted that the codec discussed above may be implemented in a capture device or a call server. In the case of a call server, the call server will receive digital audio data 328 from the capturing device and transcode the digital audio data to conform to the above requirements before sending the transcoded digital audio data to one or more playback devices 400. Such a scenario will be discussed as an example below with with reference to FIG. 6.

A physical VR conference scenario 600 is shown in FIG. 6. Five VR/AR conference users 602a–e from different locations virtually hold a meeting. VR/AR conference users 602a-e may be provided with the ability to use IVAS. Each of them uses VR/AR equipment, which provides, for example, binaural playback and video playback via a head-mounted display (HMD). For all users, the equipment supports movements in 6DOF with corresponding head position tracking. The user equipment, UE, 602 exchanges upstream and downstream encoded audio with the conference call server 604. Visually, users can be represented by corresponding avatars, which can be rendered based on information associated with relative position parameters and their angular orientation.

To further enhance the user's presence, the rotation and/or translation of the listener's head is also taken into account when playing audio received from the other participant(s) in a conference scenario. Therefore, head position tracking provides the rendering unit of the user's playback device (reference number 400 in FIGS. 4-5) with information about the current spatial data (6DOF) of the user's VR/AR equipment. This spatial data is combined (eg, by matrix multiplication or modification of metadata associated with the directional audio) with the spatial data received in the digital audio data received from the other user 602, wherein the rendering unit modifies the directionality of the directional audio received from said other user 602 to based on a combination of spatial data. The modified directional sound is then played back to the user.

In addition, the volume of reproduced audio received from a particular user can be adjusted based on spatial coordinates received in the digital audio data. The volume may be increased or decreased based on the virtual (or real) distance between two users (calculated by the playback device or call server 604) to further enhance the user's presence.

In fig. 7 illustrates as an example a virtual conference space 700 created by a conference call server. First, the server places users Ui, i=1...5 of the conference (also denoted as 702a–e) based on the coordinates Ki = (xi, yi, zi) of the virtual position. The virtual conference space is shared between users. Accordingly, in such a space, audio and video data are reproduced for each user. For example, with respect to user U5 (corresponding to user 602d in FIG. 6), during playback, other conference participants will be virtually placed at relative positions Ki - K5, i≠5. For example, user U5 will perceive user U2 at a distance |Ki – K5| and in the direction of the vector (Ki – K5)/|Ki – K5|, with directional playback performed relative to the angular position of the user U5. In fig. 2 also shows user U5 moving towards user U4. This movement will affect the position of user U5 relative to other users, which will be taken into account during playback. At the same time, U5's UE sends its position change data to the conference server 604, which updates the virtual conference space with U5's new coordinates. Since the virtual conference space is shared by users U1 to U4, they become aware of the movement of user U5 and can adjust their respective playback devices accordingly. The simultaneous movement of the user U2 is based on the corresponding principles. The call server 604 is configured to store data about the position of participants 702a-e in the shared meeting space.

In the scenario of FIG. 6-7, with respect to audio, one or more of the following 6DOF requirements may apply to the encoding medium.

providing a metadata environment for representing and transmitting upstream position information of a receiving terminal device, including spatial coordinates and/or rotation coordinates (as described above with reference to FIGS. 1-4);

the ability to associate input audio elements (such as objects) with 6DOF attributes, including spatial coordinates, rotation coordinates, directionality;

the ability to simultaneously spatially reproduce multiple received audio elements in accordance with their associated 6DOF attributes;

appropriate adjustments to the reproduced scene during rotations and/or translational movements of the listener’s head.

It should be noted that the above also applies to XR meetings, which are a combination of a physical and virtual meeting. Physical participants see and hear avatars representing remote participants through their AR glasses and headphones. They interact with avatars in discussions as if they were physically present participants. For them, interactions with other physical and virtual participants take place in mixed reality. The positions of real and virtual participants are merged into a combined, shared virtual meeting space (eg, via call server 604) that corresponds to the positions of real participants in the physical meeting space and is mapped to the virtual meeting space using absolute and relative data. physical/real positions.

In a VR/AR/XR scenario, virtual conference subgroups can be created. These subgroups can be used to inform the call server 604 about which users, for example, the quality of service, QoS, should be high, and for which users the QoS can be lower. In some embodiments, only participants in the same subgroup are included in the virtual environment provided to those subgroups through the VR/AR/XR equipment. An example scenario in which subgroups can be created is a poster session that invites remote virtual participation. Remote participants are equipped with an HMD and headphones. They are virtually present and can walk from booth to booth. They can listen to ongoing booth presentations and move closer to the presentation if they find the topic or discussion going on interesting. To improve the possibility of immersive interactions between virtual and physical participants, subgroups can be created based on, for example, the booth of the set of booths in which participants are currently interested.

Options for this scenario include:

receiving topics from virtual conference participants via the teleconference system;

grouping participants into virtual conference subgroups based on topics through the teleconferencing system;

receiving, by the teleconferencing system, a request from a new participant's device to join the virtual conference, the request being associated with an indicator indicating a preferred topic;

selecting, through the teleconferencing system, a subgroup of the subgroups based on the preferred topic and topics of the subgroups;

providing, through the teleconferencing system, to the new participant's device a virtual environment for the virtual conference, wherein the virtual environment indicates at least one of a visual virtual proximity or an audio virtual proximity of the new participant relative to one or more participants of the selected subgroup.

In some embodiments, the virtual environment indicates a visual virtual proximity or an audio virtual proximity by at least providing a visual display of a virtual reality or an audio virtual reality field where the new member's avatar and one or more avatars of the selected subgroup members are adjacent to each other.

In some embodiments, each participant is connected via open-back headphones and augmented reality glasses.

VI. Equivalents, additions, alternatives, etc.

Additional embodiments of the present invention will become apparent to one skilled in the art upon examination of the description above. Although the present description and drawings disclose embodiments and examples, the invention is not limited to these specific examples. Numerous modifications and changes are possible within the scope of the present invention as defined by the appended claims. Any reference numerals appearing in the claims should not be construed as limiting its scope.

In addition, upon examination of the drawings, description, and accompanying claims, modifications to the disclosed embodiments may be apparent to those skilled in the art that may be useful to those skilled in the art in the practice of the invention. In the claims, the word “comprising” does not exclude other elements or steps, and the singular does not exclude the plural. The mere fact that certain features are mentioned in mutually distinct dependent claims does not mean that a combination of these features cannot be used to advantage.

The systems and methods disclosed above may be implemented in software, firmware, hardware, or combinations thereof. When implemented in hardware, the division of tasks between functional blocks discussed in the above description does not necessarily correspond to the division into physical blocks; on the contrary, one physical component can perform multiple functions and one task can be performed by multiple physical components in cooperation. Some or all of the components may be implemented as software, executed by a digital signal processor or microprocessor, or implemented as hardware or an application specific integrated circuit. Such software may be distributed on machine-readable media, which may contain computer storage media (or permanent media) and communications media (or transient media). As is well known to one skilled in the art, the term "computer storage media" includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVDs) or other optical storage disks, magnetic cassettes, magnetic tape, magnetic disk information storage devices or other magnetic information storage devices, or any other medium that can be used to store desired information and that can be accessed by a computer. As is also well known to those skilled in the art, communications media typically include machine-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transmission mechanism, and include any means of delivering information.

All figures are schematic and generally show only those parts necessary to explain the invention, while other parts may be omitted or simply implied. Unless otherwise indicated, in different figures, like parts are designated by like reference numerals.

Claims (50)

1. An immersive audio processing device, comprising or coupled to a microphone system (302) comprising one or more microphones for capturing audio, the device comprising: receiving unit (304), configured to: receiving (S13) directional sound (320) captured by the microphone system; receiving (S14) metadata (322) associated with the microphone system, wherein the metadata comprises spatial data of the microphone system, wherein the spatial data characterizes the spatial orientation and/or spatial position of the microphone system and comprises at least one of a list including: angle ( angles) of azimuth, pitch, roll and spatial coordinates of the microphone system; computing unit (306) configured to: modifying at least some portion of the directional sound to produce a modified directional sound, wherein the directionality of the sound is modified based on the spatial orientation and/or spatial position of the microphone system; downmixing the modified directional audio based on the spatial orientation of the microphone system using a downmix matrix; encoding the modified directional audio and downmix matrix into digital audio data (328); a transmission unit (308) configured to transmit digital audio data. 2. The device according to claim 1, characterized in that the spatial orientation of the microphone system is represented by parameters describing the rotational movement/orientation using one degree of freedom, DoF, in spatial data. 3. The device according to claim 1, characterized in that the spatial orientation of the microphone system is represented by parameters describing the rotational movement/orientation using three DoFs in spatial data. 4. The device according to any of claim 1 or claim 3, characterized in that the spatial data of the microphone system is presented in six DoF. 5. Device according to any one of paragraphs. 1-4, characterized in that the received directional audio contains audio containing directional metadata. 6. Device according to any one of paragraphs. 1-5, characterized in that the computing unit is further configured to encode at least portions of metadata containing spatial data of the microphone system into said digital audio data. 7. The apparatus of claim 6, wherein the receiving unit is further configured to receive (S11) first commands (334) instructing the computing unit whether to include at least portions of metadata containing spatial data of the microphone system in said digital audio data. , due to which the computing unit acts accordingly. 8. Device according to any one of paragraphs. 6-7, wherein the receiving unit is further configured to receive (S12) second instructions (334) instructing the computing unit which spatial data parameter or parameters of the microphone system to include in the digital audio data, causing the computing unit to act accordingly. 9. Device according to any one of paragraphs. 7-8, characterized in that the transmission unit is configured to transmit digital audio data to an additional device (400), while instructions regarding the first and/or second commands are received from the specified additional device. 10. Device according to any one of paragraphs. 1-9, characterized in that the receiving unit is further configured to receive metadata containing a time stamp indicating the time of capture of the directional audio, wherein the computing unit is configured to encode said time stamp into said digital audio data. 11. The device according to claim 1, characterized in that the downmixing includes beamforming. 12. Device according to any one of paragraphs. 1-11, characterized in that it is implemented in equipment (602a-e) of virtual reality, VR, or equipment (602a-e) of augmented reality, AR, containing a microphone system and a head position tracking device, configured to determine the spatial data of the device at 3-6 DoF. 13. A device (400) for reproducing sound signals, the device comprising: a receiving unit (402) configured to receive (S21) digital audio data (328); decoding block (404), configured to: decoding (S22) the received digital audio data into downmixed audio, a downmix matrix, and metadata (422) associated with the microphone system, wherein the metadata comprises spatial data of the microphone system, wherein the spatial data characterizes the spatial orientation and/or spatial position of the microphone system , and contain at least one of the list, including: the angle (angles) of azimuth, pitch, roll and spatial coordinates of the microphone system; And upmixing the downmixed audio into directional audio (420) using a downmix matrix; playback unit (406) configured to: modifying (S23) the directivity of the directional sound using spatial data; And playback (S24) of the modified directional sound (424). 14. The device according to claim 13, characterized in that the spatial data indicates the spatial orientation and/or spatial position of a microphone system (302) containing one or more microphones capturing directional sound, wherein the playback unit modifies the directivity of the directional sound with at least partial reproduction of the sound environment of the microphone system. 15. Device according to any one of paragraphs. 13-14, characterized in that the spatial data contains parameters describing the rotational motion/orientation using one degree of freedom, DoF. 16. Device according to any one of paragraphs. 13-14, characterized in that the spatial data contains parameters describing the rotational movement/orientation using three DoFs. 17. Device according to any one of paragraphs. 13-16, characterized in that the decoded directional audio contains audio containing directional metadata. 18. Device according to any one of paragraphs. 13-17, characterized in that it additionally contains a transmission block (306) configured to transmit commands (334) to an additional device (300) from which the digitized sound is received, wherein the commands indicate to the additional device which parameter or parameters the rotation data must contain. 19. The device according to any one of paragraphs. 13-18, characterized in that the decoding unit is further configured to extract a time stamp indicating the time of directional audio capture from the digital audio data. 20. The device according to any one of paragraphs. 13-19, wherein the spatial data comprises spatial coordinates, and wherein the playback unit is further configured to adjust the volume of the reproduced sound based on the spatial coordinates. 21. The device according to any one of paragraphs. 13-20, characterized in that it is implemented in virtual reality, VR equipment (602a-e), or augmented reality, AR equipment (602a-e), containing a head position tracking device, configured to measure the spatial orientation and spatial position of the device in six DoF. 22. The device according to any one of paragraphs. 13-21, characterized in that the playback unit is designed for binaural sound playback. 23. A system for processing immersive audio, comprising: the first device (300) according to any one of claims. 1-12, configured to transmit digital audio data to the second device (400) according to any one of claims. 13-22, wherein the system is configured to conduct audio and/or video conferences. 24. The system of claim 23, wherein the first device further comprises a video recording unit and is configured to encode the recorded video into digital video data and transmit the digital video data to the second device, wherein the second device further comprises a display for displaying the decoded digital video data. 25. A system for processing immersive audio, comprising a first device (300) according to any one of claims. 1-12, configured to transmit digital audio data to a second device, the second device comprising: a receiving unit configured to receive digital audio data; a decoding unit configured to: decoding the received digital audio data into directional audio and metadata, the metadata comprising spatial data comprising at least one of a list including: azimuth, pitch, roll angle(s), and spatial coordinates; a playback unit for playing sound; wherein the playback unit is configured to, upon subsequent receipt by the second device of encoded video data from the first device: modifying the directivity of directional sound using spatial data, and reproduction of modified directional sound; wherein the playback unit is configured to: if the second device does not receive encoded video data from the first device: directional sound reproduction. 26. A non-transitory computer-readable medium on which instructions are stored that, when executed by one or more processors, cause one or more processors to perform the operations of any of the preceding paragraphs.