BR112021004495A2 - apparatus for processing audiovisual data for a scene and method of processing audiovisual data - Google Patents

Abstract

APPARATUS FOR PROCESSING AUDIOVISUAL DATA FOR AN AUDIOVISUAL DATA PROCESSING SCENE AND METHOD. The present invention relates to an apparatus for processing audiovisual data for a scene which comprises a receiver (201) for receiving audiovisual data for the scene. The audiovisual data comprises audio data for the scene which comprises a plurality of audio elements and image data for at least a first image of the scene the first image having a first aspect ratio. An image remapper (203) performs a content-dependent non-uniform mapping of the first image to a second image that has a different aspect ratio. The image remapper (207) is arranged to generate mapping data describing the content-dependent non-uniform mapping. An audio remapper (207) replaces a first audio element of the plurality of audio elements with a second audio element generated by modifying a spatial property for the first audio element in response to the mapping data. The spatial property being modified can be a position and/or spatial distribution of the first audio element.

Description

APPARATUS FOR PROCESSING AUDIOVISUAL DATA FOR A SCENE AND METHOD OF PROCESSING AUDIOVISUAL DATA FIELD OF INVENTION

[001] The invention relates to an apparatus and a method of processing audiovisual data and, in particular, but not exclusively, the processing of audiovisual data, for example, to provide a virtual reality experience.

BACKGROUND OF THE INVENTION

[002] The variety and reach of imaging and video applications has increased substantially in recent years with new services and ways of using and consuming video continually being developed and introduced. In particular, many spatial and interactive services, applications and experiences are being developed to give users a more immersive and immersive experience.

[003] Examples of such applications are Virtual Reality (VR, Virtual Reality) and Augmented Reality (AR) applications, which are quickly becoming popular with various solutions aimed at the consumer market. Various standards are also being developed by various standards bodies. Such standardization activities are actively developing standards for the various aspects of VR/AR systems, including, for example, streaming, broadcasting, rendering, etc.

[004] VR apps tend to provide user experiences that correspond to the user being in a different world/environment/scene, whereas AR apps tend to provide user experiences that correspond to the user being in the current environment, but with additional information or virtual objects or information being added. In this way, VR applications tend to provide a fully inclusive synthetically generated world/scene, while RA applications tend to provide a partially synthetic world/scene that is superimposed on the real scene in which the user is physically present. However, the terms are often used interchangeably and have a high degree of overlap. Next, the term Virtual Reality/VR will be used to denote both Virtual Reality and Augmented Reality.

[005] As an example, an increasingly popular service is providing images and audio in such a way that a user is able to actively and dynamically interact with the system to change rendering parameters so that it adapts to movement and changes in the user's position and orientation. A very interesting feature in many applications is the ability to change the viewer's effective viewing position and viewing direction, for example, enabling the viewer to move and “look around” in the scene being presented.

[006] This feature can specifically enable a virtual reality experience to be provided to a user. This can enable the user to move (relatively) freely in a virtual environment and dynamically change their position and the direction in which they are looking. Generally, such virtual reality applications are based on a three-dimensional model of the scene, with the model being dynamically calculated to provide the specific visualization requested. This approach is also well known,

for example, in gaming applications, as in the first-person shooter category, for computers and consoles.

[007] It is also desirable, in particular, for virtual reality applications, that the image that is presented is a three-dimensional image. In fact, to optimize viewer immersion, it is generally preferred that the user experience the scene presented as a three-dimensional scene. In fact, a virtual reality experience should preferably allow a user to select their own position, camera viewpoint and moment in time in relation to a virtual world.

[008] In addition to visual rendering, most VR/RA applications additionally provide a corresponding audio experience. In many applications, the audio preferably provides a spatial audio experience where audio sources are perceived to arrive from positions that correspond to the positions of the corresponding objects in the visual scene. In this way, audio and video scenes are preferably perceived as being consistent, both of which provide a complete spatial experience.

[009] Video content, such as that used in virtual reality applications, for example), is increasingly being captured with a wide variety of different types of devices, from professional camera systems to smartphones, tablets, video cameras. action, camera add-ons (360 degrees) for smartphones, etc. Furthermore, all of this video content is being consumed on an equally wide range of display devices, for example,

smartphones, TVs, tablets, head mounted displays (HMD) for RV, etc.

[010] All of these devices have specific properties regarding the native and/or supported aspect ratios of the video content. For capturing devices, this is primarily defined by the size and shape of the video sensor chip, while for displaying devices, the size and shape of the monitor determines the optimal aspect ratio of the video content to be displayed on the device.

[011] The result is that often the aspect ratios of the video content and the device on which the content is displayed do not match, especially with more and more User Generated Content (“UGC”) being captured by the consumers on a myriad of devices.

[012] To prevent a large part of the monitor from being unused or otherwise “cropping” part of the video image, an aspect ratio conversion (“upscaling”) can be applied to the video image. However, when done most directly by linearly stretching or compressing one (or more) dimensions, this often results in undesirable image distortion.

[013] To solve this image distortion problem, more advanced "smart" video upscaling algorithms have been developed in recent years, which apply a non-uniform remapping to the image that takes into account the characteristics of different parts of the image. video, for example, making a distinction between foreground and background, between “centers” and “peripheral” parts, and/or between “objects” and “textures”. These distinctions can be made based on pure visual parameters such as local contrast, but also based on more advanced analysis algorithms based on Artificial Intelligence (AI)/machine learning, eg object recognition such as " person” or “car”.

[014] The image analysis required to perform this “smart” video upscaling is computationally intensive, and for AI-based analyses, it also requires a fairly large learning database.

[015] For this reason, the AI-based algorithm is preferably not run directly on the end-user device, but on a distributed (remote) computer network. This network can be “the cloud” or some defined computer network. This also has the advantage that the AI algorithm can benefit from feedback from many users to, over time, improve the subjective quality of intelligent upscaling.

[016] MPEG has recently started a new standardization activity in “Network-Based Media Processing” (“NBMP”) which aims to enable distributed analysis and processing of media content such as the intelligent upscaling described above. This activity is part of the new MPEG-I standard for immersive media (VR, AR, MR).

[017] However, although this intelligent aspect ratio adaptation can often provide a better user experience by adapting the presented video to the specific aspect ratio, the approach alone may not provide an optimal usage experience.

[018] Thus, an improved approach for processing audiovisual data for a scene would be advantageous. In particular, an approach that enables improved operation, greater flexibility, reduced complexity, easier implementation, an improved user experience, a more consistent scene perception, an improved virtual reality experience, and/or performance and/or a operation better.

SUMMARY OF THE INVENTION

[019] Consequently, the invention preferably seeks to mitigate, alleviate or eliminate one or more of the above mentioned disadvantages, individually or in any combination.

[020] According to an aspect of the invention, there is an apparatus for processing audiovisual data for a scene, the apparatus comprising: a receiver for receiving audiovisual data for the scene, the audiovisual data comprising audio data for the scene comprising a plurality of audio elements and image data for at least a first image of the scene, the first image having a first aspect ratio; an image remapper to perform content-dependent non-uniform mapping of the first image to a second image with a second aspect ratio, the second aspect ratio being different from the first aspect ratio, and the image remapping is arranged in order to generate mapping data describing content-dependent non-uniform mapping; and an audio remapper arranged to replace a first audio element of the plurality of audio elements with a second audio element generated by modifying a spatial property for the first audio element in response to the mapping data.

[021] This can provide an improved user experience in many modalities and can, in many scenarios, provide an improved and more consistent perception of audio and video, which typically can lead to a more realistic and immersive experience. This can still provide a practical and low-complexity implementation.

[022] An improved and more natural perception of the scene can typically be achieved and, in many scenarios, interference and inconsistency resulting from the audiovisual scene representation can be mitigated or reduced. The approach can be particularly advantageous for Virtual Reality, VR (including Augmented Reality, RA) applications.

[023] The approach can, in many modalities, provide improved performance while maintaining low complexity and low resource usage.

[024] The inventors realized that while adapting the adaptive aspect ratio to the content can lead to better adaptation of the rendered video to the specific rendering display media, it can also result, in some situations, in an overall experience degraded due to adaptation, resulting in a potential discrepancy between the spatial perceptions of audio and video. Furthermore, the inventors have realized that an approach of pre-determined tailoring the audio to match the specific display media and aspect ratio will tend to result in sub-optimal results. The inventors realized that improved overall user perception, and specifically a more consistent approach,

it can be achieved by applying a remapping to the audio, where an image remapper generates mapping data that describes the specific remapping performed, and the audio remapping is adapted based on that mapping data.

[025] The content-dependent non-uniform mapping can be a non-linear mapping that is adapted in response to the content of the first image. Mapping data can indicate how different image segments have moved from the first to the second image. The offset can be, for example, an absolute offset or a relative offset between different image segments/positions/pixels. Content-dependent non-uniform mapping can be a two-dimensional mapping from positions in the first image to positions in the second image (for at least some positions). Two-dimensional mapping can be on the display plane, and positions can be display plane positions. Content-dependent non-uniform mapping can be an image aspect ratio that alters content-dependent non-uniform mapping.

[026] The mapping data can describe the content-dependent non-uniform mapping by describing a relationship between positions in the first image and positions in the second image (for at least some positions). The positions in the first image and/or in the second image can be two-dimensional positions in the display plane. The positions in the first image and/or in the second image can be absolute and/or relative positions.

[027] The second audio element can represent the same audio content as the first audio element, but with a different spatial property, such as a different position and/or spatial distribution property.

[028] In many embodiments, the apparatus can comprise a functionality for rendering the scene based on the audio data and visual data (after modification by the image remapper and by the audio remapper). However, in other modalities, this rendering may be external to the device.

[029] An aspect ratio can be the ratio between the width and height of an image or screen.

[030] According to an optional feature of the invention, the audio remapper is arranged to generate the second audio element by modifying a spatial position property of the first audio element in response to the mapping data.

[031] This can provide a particularly beneficial user experience and/or performance in many modalities. This can typically provide better consistency between the audiovisual representation of the scene.

[032] According to an optional feature of the invention, the audio remapper is arranged to generate the second audio element by changing a spatial position for the first audio element from a position corresponding to an image position in the first aspect ratio for an image position in the second image.

[033] This can provide a particularly beneficial user experience and/or performance in many modalities.

[034] According to an optional feature of the invention, the audio remapper is arranged to generate the second audio element by modifying a spatial distribution property of the first audio element in response to the mapping data.

[035] This can provide an improved user experience in many modalities and can provide additional flexibility in adapting the sound stage to provide an experience compatible with the visual output provided. The spatial distribution property can be indicative of a degree of physical or perceived size, indicating the amount of area that an audiovisual object covers from the user's viewing, display, and/or range of incoming sound pressure wave arrival directions. An indication of how much larger the sound element is, or should be when rendered, than a point source, its diffusion. In different modalities, a suitable measure for the spatial distribution can be, for example, absolute dimension(s), angle(s), radius(s).

[036] According to an optional feature of the invention, the audio remapper is further arranged to generate the second audio element by modifying the spatial property in response to an audio property of the first audio element.

[037] This can provide improved performance and/or user experience. The audio property can be an acoustic property.

[038] According to an optional feature of the invention, the acoustic property comprises a degree of spatial distribution for the first audio element.

[039] This can enable a more accurate adaptation of the determination of the modified spatial property.

[040] According to an optional feature of the invention, the audio remapper is arranged to reduce a change of position between the second audio element and the first audio element to an increasing degree of spatial distribution for the first audio element. audio.

[041] This can provide improved performance and/or user experience in many modalities.

[042] According to an optional feature of the invention, the audio remapper is arranged to select whether it is desired to apply the modification to the spatial property or not, depending on the degree of spatial distribution.

[043] This can provide improved performance and/or user experience in many modalities. In some embodiments, the audio remapper can be arranged to select whether or not it is desired to replace the first audio element with the second audio element, depending on the degree of spatial distribution.

[044] According to an optional feature of the invention, the audio remapper is arranged to modify the spatial property in response to an audio type of the first audio element.

[045] This can provide improved performance and/or user experience in many modalities. Audio types can include one or more from an audio channel group, a higher-order ambisonic audio element, and an audio object.

[046] According to an optional feature of the invention, the audio remapper is arranged to modify the spatial property in response to whether or not the first audio element is linked to an image object in the first image.

[047] This can provide improved performance and/or user experience in many modalities.

[048] According to an optional feature of the invention, the audio remapper is arranged to modify the spatial property depending on whether or not the first audio element is designated as background audio.

[049] This can provide improved performance and/or user experience in many modalities.

[050] In some embodiments, the audio remapper can be arranged to apply a non-uniform spatial amplification to the audio elements among the plurality of audio elements that are audio channels.

[051] According to an optional feature of the invention, the audio remapper is arranged to modify the spatial property in response to a property of a monitor for rendering the second image.

[052] This can provide improved performance and/or user experience in many modalities. The monitor property can be specifically a monitor size.

[053] According to an optional feature of the invention, the audio remapper is arranged to modify the spatial property in response to a viewer's position in relation to a monitor to render the second image.

[054] This can provide improved performance and/or user experience in many modalities. The spectator position can be a nominal or assumed spectator position or it can, for example, be a measured and identified spectator position.

[055] According to an aspect of the invention, a method of processing audiovisual data for a scene is provided, the method comprising: receiving audiovisual data for the scene, the audiovisual data comprising audio data for the scene that comprise a plurality of audio and image data elements for at least a first image of the scene, the first image having a first aspect ratio; performing a content-dependent non-uniform mapping of the first image to a second image with a second aspect ratio, where the second aspect ratio is different from the first aspect ratio, and the content-dependent non-uniform mapping comprises data generation of mappings that describe content-dependent non-uniform mapping; and replacing a first audio element of the plurality of audio elements with a second audio element generated by modifying a spatial property for the first audio element in response to the mapping data.

[056] These and other aspects, features and advantages of the invention will become evident and will be elucidated with reference to one or more modalities described below.

BRIEF DESCRIPTION OF THE DRAWINGS

[057] The embodiments of the invention will be described, by way of example only, with reference to the drawings, in which: Figure 1 illustrates an example of a client-server arrangement to provide a virtual reality experience;

Figure 2 illustrates an example of elements of an audio device according to some embodiments of the invention; Figure 3 illustrates an example of a rendering of an audio apparatus according to some embodiments of the invention; and Figure 4 illustrates an example of a rendering of an audio apparatus in accordance with some embodiments of the invention.

DETAILED DESCRIPTION OF SOME MODALITIES OF THE INVENTION

[058] The following description focuses on the embodiments of the invention applicable to a virtual reality system. However, it should be considered that the invention is not limited to this application, but can be applied to many other systems for distribution, processing and/or rendering of audiovisual data.

[059] The description will additionally focus on the handling and processing of audiovisual data in the context of client-server approaches supported by a network that can be specifically a public network, such as the Internet. However, it should be recognized again that this is only an exemplary application of the principles and approaches described and that these can be used in many other systems and applications. including, for example, systems where the audiovisual data is provided in the form of a dedicated data stream transmitted directly to a rendering/rendering device or modalities where the rendering device itself retrieves or generates the audiovisual data.

[060] Virtual (including augmented) experiences that enable a user to move in a virtual world are becoming increasingly popular and services are being developed to satisfy such demand. In many of these approaches, visual and audio data can be dynamically generated to reflect the current pose of a user (or viewer).

[061] In the field, the terms "positioning" and "pose" are used as a common term for position and/or direction/orientation. The combination of position and direction/orientation, for example of an object, a camera, a head or a view, can be called a pose or placement. Thus, an indication of positioning or pose can comprise six values/components/degrees of freedom with each value/component typically describing an individual property of the corresponding object's position/location or orientation/direction. Of course, in many situations a placement or pose can be considered or represented with fewer components, for example if one or more components are considered fixed or irrelevant (for example if all objects are considered to be at the same height and have an orientation horizontally, four components can provide a complete representation of an object's pose). In the following text, the term “pose” is used to refer to a position and/or orientation that can be represented by one to six values (corresponding to the maximum number of possible degrees of freedom).

[062] Typically, the virtual reality application generates three-dimensional output in the form of separate visualization images for the left and right eyes. These can then be presented to the user by appropriate means, such as individual left and right eye indications from a VR headset. In other modalities, one or more preview images can, for example, be presented in an autostereoscopic display, or indeed, in some modalities, only a single two-dimensional image can be generated (for example, using a conventional two-dimensional display ).

[063] Similarly, for a pose of a given user/viewer/listener, an audio representation of the scene can be provided. The audio scene is typically rendered to provide a spatial experience where audio sources are understood to originate from desired positions. As audio sources can be static in the scene, changes in the user's pose will result in a change in the relative position of the audio source in relation to the user's pose. Consequently, the spatial perception of the audio source must change to reflect the new position in relation to the user. The audio rendering can therefore be adapted according to the user's pose.

[064] In many embodiments, audio rendering is a binaural rendering that uses Head Related Transfer Functions (HRTFs) or Binaural Room Impulse Responses (BRIRs) (or similar) to provide the desired spatial effect for a user wearing a headset. However, it should be noted that, in some systems, audio can be output using a speaker system and that the signals for each speaker can be transmitted in such a way that the overall effect on the user matches the intended spatial experience.

[065] The input of the pose of the observer or user can be determined in different ways in different applications. In many modalities, a user's physical movement can be tracked directly. For example, a camera inspecting a user's area can detect and track the user's head (or even their eyes). In many modalities, the user can wear a VR headset that can be tracked by external and/or internal means. For example, the headset may include accelerometers and gyroscopes that provide information about the movement and rotation of the headset and therefore the head. In some instances, the VR headset may transmit signals or include identifiers (eg, visuals) that allow an external sensor to determine the position of the VR headset.

[066] In some systems, the observer pose can be provided by manual means, for example, when the user manually controls a joystick or similar manual input. For example, the user can manually move the virtual viewer around the virtual scene by controlling a first analog joystick with one hand and manually controlling the direction in which the virtual viewer is looking by manually moving a second analog stick with the other hand.

[067] On some systems, the VR application can be deployed and executed remotely from the viewer. For example, a device local to the user can detect/receive motion/present data that is transmitted to a remote device that processes the data to generate the pose of the viewer. The remote device can then generate visualization images suitable for the viewer's pose based on scene data describing the scene. The preview images are then transmitted to the local device to the viewer where they are presented. For example, the remote device can directly generate a video stream (typically a stereo/3D video stream) that is directly presented by the local device. Similarly, the remote device can generate an audio scene reflecting the virtual audio environment. This can, in many modalities, be done by generating audio signals that correspond to the relative position of different audio sources in the virtual audio environment, for example by applying binaural processing to the individual audio components that correspond to the current position. of these in relation to the pose of the head. So, in this example, the local device cannot perform any VR processing except to transmit motion data and present received video and audio data.

[068] On many systems, functionality can be distributed across a local device and a remote device. For example, the local device can process incoming sensor and input data to generate observer poses that are continuously transmitted to the remote VR device. The remote VR device can then generate the corresponding view images and transmit them to the local device for presentation. In other systems, the remote VR device may not directly generate the view images, but may select relevant scene data and transmit it to the local device which can then generate the view images that will be presented. For example, the remote VR device can identify the closest capture point and extract the corresponding scene data (eg spherical and depth image data from the capture point) and transmit it to the local device. The local device can then process the received scene data to generate the images for the current specific viewing pose.

[069] Likewise, the remote VR device can generate audio data representing an audio scene by various audio elements, which can be different audio components, audio objects, audio signals, audio channel signals , etc., which correspond to different audio sources in the audio scene. For some or all of these audio elements, audiovisual data can, in some modalities, comprise positional information indicative of the position of audio sources (which can, for example, dynamically change to moving objects) as well as potentially other spatial data or acoustic metadata (for example, how is the broadcast or reverberation of a particular audio source). The local VR device can then process these signals properly, for example, by applying suitable binaural processing that reflects the relative position of the audio sources to the audio components.

[070] Figure 1 illustrates the example of a VR system in which a remote server 101 communicates with a client device 103 through, for example, a network 105, such as the Internet. Remote server 101 can be arranged to simultaneously support a potentially large number of client devices 103.

[071] Such an approach can, in many scenarios, provide a better compromise, for example, between complexity and resource requirements for different devices, communication requirements, etc. for example, observer pose and corresponding scene data can be transmitted at longer intervals with the local device that processes locally received observer pose and scene data to provide a real-time experience with little delay. This can, for example, substantially reduce the required communication bandwidth, while providing a low-delay experience and at the same time enabling scene data to be centrally stored, generated and maintained. It may, for example, be suitable for applications where a VR experience is provided to a plurality of remote devices.

[072] In some embodiments, the server 101 can generate a continuous stream of audiovisual data that comprises a real-time video stream and an associated real-time audio stream that corresponds to the viewers' current posture (as determined from the data received from the customer 103). This real-time audiovisual stream can be generated so that it can be rendered directly by the client 103 and specifically no posing or point of view or virtual reality environment displacement processing can, in some modalities, be performed or required by the client 103 This can be suitable for many low-end processing devices, such as many low-end mobile devices or headset devices. The processing required at the client can be kept to a minimum.

[073] Such an approach may also be particularly suitable for scenarios where the experience is a shared experience, in which a plurality of people are interested in having the same perception of a scene.

This scenario is typical in broadcast, including, for example, 3D broadcast of a sporting event. This approach is not limited to traditional audiovisual broadcasts, but can also be used in systems where the viewing pose is dynamically determined and variable. Shared experience may also be applicable to some VR experiences and services.

[074] For example, for an electronic sport (or eSport) broadcast, the player can control movement in a virtual reality environment with the resulting audiovisual data for the specific poses being also distributed by server 101 to a large number of clients passive 103. A user on the individual passive client can, in such a modality, see essentially the electronic sport through the player's eyes.

[075] For example, for an electronic sport where players are playing a first-person shooter, the single player is controlling their avatar and moving into a scene in which the battle takes place. Other players will be able to see the avatar, but the player himself doesn't see his own avatar (or at least doesn't see the entire avatar), but sees the view of the scene that his avatar would experience (that is, what a person in the virtual scene would see from that point by sight). Electronic sports have recently become a very popular sport, with very large events attracting tens of thousands of spectators who can directly see the game in large exhibitions where it is being played. Views can, for example, present an overview of the scene with an indication of the position of individual players and with views of individual players.

[076] In addition, these e-sports events can also often be broadcast to viewers who are not on site, and can often be broadcast to a large number of viewers, which, in some cases, may be hundreds of thousands or even millions of viewers watching in real time. In such cases, the preview generated for one of the players that depends on that player's input and control can result in preview images (video footage) that are not only presented to the player, but are also streamed live to a large number of viewers separated in different places and circumstances, and with the use of different equipment for visualization.

[077] One problem with proving content to many, and especially to a large number of clients, is that the characteristics of the clients, and specifically the means of rendering, can vary substantially for different devices and may not be known on the server. Consequently, in many approaches, the server 101 will generate the audiovisual data stream for a particular nominal or standardized device. Specifically, the data stream can be generated for a given nominal render monitor. Such considerations can include resolution, color depth, aspect ratio, size, etc. If the local rendering method does not match the assumed characteristics, an adaptation can be performed locally on the device 103.

[078] In particular, adapting the aspect ratio to match the specific display used on the individual client can have a significant impact on the usage experience as this very directly affects the visual presentation. For example, using the wrong aspect ratio can result in black bars around the image, and many low-complexity aspect adjustments result in significant geometric distortions.

[079] Figure 2 illustrates an example of a client device 103 that is arranged to adapt a received audiovisual data stream to match the specific aspect ratio of a monitor on which the visual data will be presented.

[080] The client device 103 comprises a receiver 201, which is arranged to receive audiovisual data for a scene with the audiovisual data comprising data describing the visual and sonic aspects of the scene. The audiovisual data specifically comprises image data for at least a first image of the scene, and specifically in the described embodiments comprises a video stream which provides images in the form of frames of a video sequence. The first image has a certain aspect ratio (that is, a certain aspect ratio between the width and height of the image). In many modalities, a 3D video stream can be provided with frames for the left and right eye images corresponding to a specific viewing pose for the scene.

[081] In addition, audiovisual data comprises audio data for the scene, with the audio data describing various audio elements for the scene. For example, audio data comprising different audio elements in the form of audio channels, audio objects, audio channel, higher order ambisonics, etc. In some modalities the audio elements of the audio data can all be the same type, but in many modalities the audio scene can represent the audio scene by the audio data comprising different types of audio elements for different sources of audio.

[082] Thus, in many embodiments, the received audio data describes an audio scene that corresponds to the scene visually represented by the video data by several different audio elements, which can be different types of audio elements.

[083] For example, some audio elements may correspond to a single audio source with a specific and well-defined spatial position in the scene (ie potentially a virtual spatial position in a virtual scene). This audio source can, for example, be represented by a specific and individual audio object with associated position data. For example, an audio object can be provided for each instrument in an orchestra and/or for each person in a given scene. Other audio elements may correspond to a combination of different audio sources and typically may have less defined spatial positions. For example, a background audio ambient signal can be provided that represents the overall background or ambient sound in the scene. This audio signal can be more diffused and distributed than an audio object that directly represents a single audio source.

[084] The receiver 201 is coupled to an image remapper 203 that receives the image data, i.e., receives image data for at least one image, but typically for a large number of frames of a video sequence. The image remapper 203 is arranged to perform a content-dependent non-uniform mapping of the first image to a second image with an aspect ratio different from the original aspect ratio (a second aspect ratio). For example, received audiovisual data may include images or frames in a certain aspect ratio, such as the generalized aspect ratio of 16:9. However, the apparatus of Figure 2 can be arranged to drive a monitor with a different aspect ratio. For example, the monitor can be a 4:3 aspect ratio, a 21:9 aspect ratio, or an 18:9 aspect ratio (which is often used for VR headsets).

[085] The image remapper 203 is arranged to generate a modified image having a different aspect ratio. For example, a first image received in the received audiovisual data can be modified to provide a second image with a different aspect ratio. Aspect ratio conversion is not just a predetermined default extension/compression of the input aspect ratio image to display ratio, but it is a non-uniform and content-dependent mapping of one aspect ratio to another. In mapping, some parts of the image are stretched/repositioned according to a different function than other parts. The image remapper 203 not only applies a predetermined mapping, but adapts the applied mapping depending on the content of the images.

[086] For example, in many embodiments, the image remapper 203 can be arranged to divide the image into segments and can apply different mappings to at least two different segments of the image. For example, the image remapper 203 can divide the segments into various categories or types and can apply a specific remapping algorithm to each type of segment. The resulting remapped segments can then be combined in the second image, for example, by selecting the foreground segment for areas of the segments that, after remapping, overlap and fill holes that occur in the mapping (for example, from a segment of the background).

[087] As a specific example of low complexity, the image remapper 203 can perform the segmentation of a received image. Targeting can be based on image content (eg colors, brightness, etc.) and possibly a depth map associated with an incoming image, as will be well known to the skilled person. Based on segmentation, segments can be divided into a group comprising segments corresponding to protruding objects, respectively (eg foreground objects, objects corresponding to people, detected faces, etc.) and non-protruding objects (including , for example, the background). An intermediate image can be made up of all non-salient objects/segments and a predetermined aspect ratio mapping (eg a predetermined stretch) can be applied to that image. For example, a simple predetermined uniform stretch or a predetermined anamorphic stretch can be applied to generate a second intermediate image with an aspect ratio corresponding to the monitor. The holes in the image can then be filled in, for example,

by filling using extrapolation from neighboring areas, as known in the art, or by stretching the background image between them.

[088] New positions can be determined for the salient objects and they will typically be determined with less distortion compared to the original image than for the predetermined non-salient image mapping. For example, the protruding objects/images can be determined to have positions that do not change relative to each other, ie the distances between the protruding objects can be kept to remain the same. The absolute position can, for example, be determined to have the same offset from the center of the image. Finally, protruding objects can be superimposed on the second intermediate image to generate an output image corresponding to the original image, but with the desired aspect ratio, such as typically that of the monitor on which the image/video will be rendered.

[089] Such an example can, for example, result in protruding objects that remain in the same position relative to each other and in the center of the image, but with the background being stretched to match the new aspect ratio. For example, in the case of two people standing on a beach, the generated image might have the two people standing in the same position in the displayed image and with the same proximity to each other, but with the beach and sea in the background being extended to fill the entire image.

[090] It should be noted that this is just an example of low complexity and that many different, and often more complex, algorithms can be used instead. Examples can be found, for example, in ““Use cases and draft requirements for Network Based Media Processing (v1)”, document generated at the 119th MPEG meeting, in Turin (Italy), July 2017.

[091] The image remapper 203 is coupled to a video renderer 205 which receives the remapped images/frames from the image remapper 203 and generates a unit signal suitable for a monitor (not shown) on which the images/frames are displayed. It should be recognized that the exact function of the video renderer depends on the properties/characteristics of the specific monitor and the specific properties of the image. For example, the video renderer 205 can be arranged to generate an output signal in accordance with a specific screen/video trigger pattern.

[092] For example, the image remapper 203 can, in some embodiments, operate on uncoded data, that is, the images can be represented by uncoded pixel values (eg direct RGB pixel values) and the output of the image remapper 203 can therefore be composed of uncoded pixel values. The apparatus can be arranged to drive a monitor using a display standard that includes video compression (eg Display Stream Compression VESA), and the video renderer can be arranged to compress the remapper data of 203 images to generate a video stream compressed according to the specific standard.

[093] As another example, if the device powers a dedicated, proprietary VR headset and the images provided by the image remapper 203 are encoded, the video renderer 205 can decode the video images to render, for example, RGB values to each pixel and they can be converted into trigger signals suitable for each pixel of the VR headset monitor. The unit's signals can then be sent to an external display panel trigger to directly trigger the headset display(s).

[094] As another example, the video renderer 205 can be arranged to generate a stream of data suitable for a monitor by adding and structuring the image data from the image remapper 203. For example, to communicate with a monitor using, for example, HDMI or DisplayPort interfaces, the 205 video renderer can add a clock, an error correction, control data, etc., to generate a bitstream corresponding to the specific standard.

[095] The video renderer 205 can therefore perform the necessary conversion, processing and translation between the image data of the images of the image remapper 203 and the signal/data flow required by a subsequent monitor.

[096] The device in Figure 2 can, therefore, adapt the visual data received to the specific monitor used and provide a better visual and image experience. However, the inventors further realized that a problem with such flexible, content-based aspect ratio mapping is that it can degrade the audio side of the user experience. The inventors realized that image remapping can result in a greater perceptual experience of the audio being inaccurate or artificial and that the overall experience can be considered less natural. To address, for example, such issues, the device in Figure 2 comprises a functionality to modify the audio side before rendering.

[097] The apparatus comprises an audio remapper 207 which receives the audio data from the receiver 201. Accordingly, the audio remapper 207 receives audio data describing various audio elements of the audio scene. Audio remapper 207 is arranged to modify a spatial property of one or more received audio elements. Specifically, the audio remapper 207 can replace a first audio element in the audio scene/audio data with a second audio element that has a modified spatial property, such as a different position or a different position/distribution .

[098] The audio remapper 207 is coupled to an audio renderer 209 which receives the audio data including the modified audio elements (ie the second audio element is received instead of the first audio element once that it has been replaced). The audio renderer 209 is arranged to render the received audio data using any suitable rendering approach. The audio renderer 209 can generate audio signals for eg spatial audio rendering using speakers or eg headsets (binaural rendering).

[099] It should be recognized that many algorithms and approaches for rendering spatial audio that use speakers, specifically in surround sound systems, are known to the person skilled in the art and that any suitable approach can be used without deviating from the invention .

[0100] For example, the audio renderer 209 can output audio signals for five speakers in a surround sound configuration with a center speaker, left front speaker, right front speaker, surround speaker left and right surround speaker. The audio renderer 209 can generate a set of audio signals comprising an audio signal for each speaker. The signals can then be amplified to generate trigger signals for the individual speaker.

[0101] In some embodiments, an audio element being rendered using speakers can be received, such as a stereo downmix, and the audio renderer 209 can perform an upmix to generate the surround signals that , in some cases, can be rendered directly. This approach can, for example, be useful for audio elements that represent a diffuse sound that is not directly related to the user's pose. For example, an audio element that represents generic diffuse ambient audio can be provided as a stereo downmix that can be directly upmixed to provide the appropriate surround sound audio channels. Each of the resulting upmixed signals can be combined with corresponding speaker signals generated from other audio elements to generate the set of output signals.

[0102] Some audio elements that are rendered via the speaker configuration can, for example, be provided in the form of audio objects. This audio object can be represented by audio data that describes the specific audio and the associated position data that describes the position of the audio source. Based on the position data and the positions of the speakers (whether they are actual or nominal positions for the surround sound speaker setup), the audio renderer 209 can determine coefficients for a matrix or vector that maps the signal. audio for the different surround sound channels.

[0103] In many embodiments, audio rendering by audio renderer 209 is a binaural rendering process that utilizes suitable binaural transfer functions to provide the desired spatial effect for a user wearing headphones. For example, the audio renderer 209 can be arranged to generate an audio component that is perceived when arriving from a specific position using binaural processing.

[0104] Binaural processing is known to be used to provide a spatial experience through virtual positioning of sound sources that use individual signals to the listener's ears. With proper binaural rendering processing, the signals needed in the eardrums for the listener to perceive sound coming from any desired direction can be calculated and the signals can be rendered so that they provide the desired effect. These signals are then recreated in the eardrum using headsets or a crosstalk cancellation method (suitable for rendering on speakers that are very close together). Binaural rendering can be considered an approach to generating signals to a listener's ears, resulting in tricking the human auditory system into thinking that a sound is coming from the desired positions.

[0105] Binaural rendering relies on binaural transfer functions that vary from person to person due to the acoustic properties of the head, ears, and reflective surfaces such as the shoulders. For example, binaural filters can be used to create a binaural recording that simulates multiple sources at multiple locations. This can be done by convoluting each sound source signal with the pair of, for example, head-related impulse responses (HRIRs) that correspond to the position of the sound source.

[0106] A well-known method of determining binaural transfer functions is binaural recording. It is a method of sound recording that uses a dedicated microphone arrangement and is intended for playback using headphones. Recording is done by placing microphones in an individual's ear canal or using a doll's head with built-in microphones, a bust that includes pinnas (external ears). The use of such a puppet's head, including the pinna, provides a spatial impression very similar to as if the person listening to the recordings were present during the recording.

[0107] By measuring, for example, the responses from a sound source at a specific location in 2D or 3D space to microphones placed in or near the human ears, the appropriate binaural filters can be determined. Based on these measurements, binaural filters can be generated that reflect the acoustic transfer functions to the user's ears. Binaural filters can be used to create a binaural recording that simulates multiple sources at multiple locations. This can be done, for example, by convoluting each sound source with the measured impulse response pair to a desired sound source position. To create the illusion that a sound source is moved around the listener, a large number of binaural filters with adequate spatial resolution is typically needed, eg 10 degrees.

[0108] The head-related binaural transfer functions can be represented, for example, as Head Related Impulse Responses (HRIR), or equivalently, as Head Related Transfer Functions (HRTFs) or, Room Impulse Responses (BRIRs), or Binaural Room Transfer Functions (BRTFs). The transfer function (eg, estimated or assumed) from a given position to the listener's ears (or eardrums) can, for example, be provided in the frequency domain, in which case it is typically called HRTF or BRTF, or in the time domain, in this case, it is typically called the HRIR or BRIR. In some scenarios, head-related binaural transfer functions are determined to include aspects or properties of the acoustic environment and specifically the environment in which measurements are taken, while in other examples only user characteristics are considered. Examples of the first type of functions are BRIRs and BRTFs.

[0109] The audio renderer 209 can therefore comprise a storage device with binaural transfer functions for a typically high number of different positions, with each binaural transfer function providing information on how an audio signal should be processed/filtered so that it is perceived as originating from that position. Individually applying binaural processing to a plurality of audio signals/sources and combining the result can be used to generate an audio scene with multiple audio sources positioned at the appropriate positions on the sound stage.

[0110] The audio renderer 209 can, for a given audio element that is to be perceived as originating from a given position in relation to the user's head, select and retrieve the stored binaural transfer function that most closely matches the desired position ( or, in some cases, it can generate this by interpolating between a plurality of closely related binaural transfer functions). It can then apply the selected binaural transfer function to the audio signal from the audio element, thereby generating an audio signal for the left ear and an audio signal for the right ear.

[0111] The output stereo signal generated as the left and right ear signal is suitable for headphone rendering and can be amplified to generate trigger signals that are supplied to a user's headphone. Next, the user will notice that the audio element originates from the desired position.

[0112] In the apparatus of Figure 2, the image remapper 203 is arranged to generate mapping data describing at least part of the image remapping that was performed by the image remapper 203 to change the aspect ratio. For example, the image remapper 203 may provide information indicating the modified or new positions of specific image objects in the second image and/or may, for example, include data indicating the mapping that was used for the background image or for a particular segment of the image.

[0113] Mapping data can be any data that describes a property of the mapping from the first image to the second image performed by content-dependent non-uniform mapping. The mapping data can specifically include data describing a relationship between positions in the first image and positions in the second image for at least some positions. The mapping data can, for example, describe how image positions in the first image are mapped to image positions in the second image by content-dependent non-uniform mapping. The positions can be two-dimensional and can be absolute and/or relative positions typically with respect to a coordinate system for the corresponding first or second image.

[0114] Mapping data is provided from image remapping 203 to audio remapping 207, and audio remapping 207 is arranged to modify the spatial property of the audio elements depending on/in response to/as a function from/based on the mapping data (and therefore based on the content-dependent non-uniform mapping of the image's aspect ratio). For example, a position mapping can be applied to the first audio element to generate the second audio element, and the position mapping can be adapted based on the received mapping data.

[0115] For example, the mapping data can describe a relationship between the positions in the first image and the second image, and the audio remapper 207 can be arranged to modify the positions of the audio elements based on the position change between the first image and the second image as described by the mapping data.

[0116] As a specific example, if the mapping data indicates that the position of an image object corresponding to a certain object in the scene is moved to a different position in the image, the audio remapper 207 can be arranged to modify the spatial position of the audio element that represents the sound generated by that object so that it matches the new position. For example, in some embodiments, an audio element may comprise associated position metadata that is used by the audio renderer 209 to render the audio element such that it is perceived as originating from the given position. In such modalities, the audio remapper 207 can directly change the position metadata before it is provided to the audio renderer 209. Specifically, the audio remapper 207 can retrieve the position of the audio element and then can shift it by an amount that corresponds to the offset of the corresponding image object in the changed aspect ratio. The modified position can then be given to audio renderer 209 and used by it when rendering the audio element. Consequently, the audio element will be rendered to appear as originating from the modified position. This approach can often be particularly suitable for audio objects that often receive explicit position data.

[0117] In other embodiments, the exact position may not be known, but it may be derived from the received audio data. For example, if the audio element is represented by, say, five surround sound channels (or a downmix thereof), the relative signal level and time in each of the audio channels will indicate the spatial position of the audio source. corresponding. That position can then be estimated and modified, and the signals from the audio channels can be compensated to reflect the new position. In many embodiments, the modification of the spatial position can be performed directly by processing the audio signals and without any explicit derivation from the original position. For example, if a surround channel audio element is received that is linked to an image object indicated by the mapping data to be shifted, for example, 10 degrees to the left, the audio remapper 207 can apply an operation of matrix to the audio channel signals that rotate these 10 degrees in the correct direction.

[0118] In fact, in the case of audio content that is defined in a channel-based format, the spatial distribution of audio sources in the audiovisual scene is typically defined implicitly by relative level and time relationships between channels, for example, due to panning algorithms used in the generation of channel-based audio elements. Also, sound from multiple sources can be mixed into channels, so signals from individual sources may not be readily available separately, as in the case of audio objects.

[0119] In the channel-based case, the remapping of the audio elements can, for example, be obtained with the use of a spatial amplification algorithm, of which there is a great variety. However, unlike how such spatial magnification algorithms are commonly used (ie, stretching or compressing the spatial audio scene evenly across the spatial dimension), the spatial magnification algorithm can be applied non-uniformly, corresponding to the remapping non-uniform video image.

[0120] Channel level differences between channels can be calculated in time-frequency blocks and used to combine these time-frequency blocks with visual objects or image segments represented in the mapping data. Furthermore, the phase differences in the time-frequency blocks and their stability over time can indicate whether a time-frequency block represents an isolated point source (no phase difference or stable phase difference over time) or a diffuse sound or a mix of multiple point sources (typically non-zero, unstable phase difference over time).

[0121] Channel level differences and phase differences can guide the adaptation of audio elements. For example, channel level differences for a time-frequency block between several pairs of channels can determine a correspondence with a segment of the first image, by combining the positions of the channel in relation to the dimensions of the image. This can be known, measured or assumed. Specifically, for example, a left channel can correspond to the left edge of the image, a right channel to the right edge of the image, and a center channel to the middle of the image. This correspondence may be different for the first and second images.

[0122] In this example, the max(𝑃 ,10−6) channel level difference between left and center (𝐶𝐿𝐷𝐿𝐶 = 10 ∗ log10 (max(𝑃𝐿 ,10−6)) with PL and PC

𝐶 being the signal strength in the left and center channels respectively) can correspond to a position in the left half of the image. For example, a CLDLC of −60 corresponds to the middle of the image, and a CLDLC of 60 corresponds to the left edge. CLDLC values in the middle can be interpolated linearly or non-linearly. With additional height channels, a two-dimensional position relative to the image can be derived.

[0123] The estimated position for each time-frequency block can be combined with an object or segment in the image and the corresponding mapping data provided by the image remapper. Adapting the audio can follow the same mapping as, for example, changing the position of the object or segment. For example, ̅̅̅ 𝑏2 = 𝑏̅1 ∙ 𝑀, where 𝑏̅1 and ̅̅̅ 𝑏2 represent the channel base vectors of a time-frequency block of the first and second audio elements, respectively, and M is a modification matrix that modifies the signal strength in the time-frequency blocks of the channels to achieve a desired CLD value for the second audio element.

[0124] In an example where CLDLC1 corresponds to position 𝑝 ̅̅̅1 in the image, which, according to the mapping data, is moved to position ̅̅̅ 𝑝2 in the image, which corresponds to CLDLC2, according to the used interpolation scheme above, the modifications 𝑃𝐿2 = 𝛼𝐿 ∙ 𝑃𝐿1 and 𝑃𝐶2 = 𝛼𝐶 ∙ 𝑃𝐶1 can be derived under the general requirement that the power remains equal power: 𝑃𝐿1 + 𝑃𝐶1 = 𝑃𝐿2 + 𝑃𝐶2 .

(𝑃𝐿1 + 𝑃𝐶1 ) ∙ 10𝐶𝐿𝐷𝐿𝐶2⁄10 𝛼𝐿 = 𝑃𝐿1 ∙ 10𝐶𝐿𝐷𝐿𝐶2⁄10 + 𝑃𝐶1 ∙ 10𝐶𝐿𝐷𝐿𝐶1⁄10 (𝑃𝐿1 + 𝑃𝐶1 ) ∙ 10𝐶𝐿𝐷𝐿𝐶1 ⁄10 𝛼𝐶 = 𝑃𝐿1 ∙ 10𝐶𝐿𝐷𝐿𝐶2⁄10 + 𝑃𝐶1 ∙ 10𝐶𝐿𝐷𝐿𝐶1 ⁄10

[0125] If this is the only adaptation for this time-frequency block, the M adaptation matrix for 5 channels (L, R, C, LS, RS) can be: √𝛼𝐿 0 0 0 0 0 1 0 0 0 𝑀= 0 0 √𝛼𝐶 0 0 0 0 0 1 0 ( 0 0 0 0 1)

[0126] If there are more adaptations, the individual adaptation matrices can be multiplied, for example: 𝑀 = 𝑀𝐿𝐶 𝑀𝑅𝐶 𝑀𝐿𝐿𝑠 𝑀𝑅𝑅𝑠 .

[0127] In the described system, the content-dependent non-uniform mapping is inherently a dynamic mapping, since it depends on the content and, therefore, varies depending on the image content. The mapping is not a fixed or predetermined mapping, but the mapping that is applied to the first image to generate the second image depends on the first image and is typically therefore not known in advance. The actual mapping that is applied in the specific case and for this specific image can thus be described, and in the approach, the image remapper 203 generates mapping data that describes the specific content-dependent non-uniform mapping that is applied to the first image . In this way, the data can describe the specific mapping applied to that image, thus enabling the audio remapper 207 to perform an audio remapping that corresponds to the image remapping. In this way, the mapping data describing the image mapping makes it possible to remap audio by dynamic and adapt to the aspect ratios of specific images.

[0128] It will be recognized that there are many ways that mapping data can describe content-dependent non-uniform mapping and that the specific approach used will depend on the preferences and requirements of the individual implementation and application. For example, the mapping data can describe how the image/pixel positions in the first image are mapped to the image/pixel positions in the second image. Mapping data can, for example, be provided as absolute positions in the two images or as relative images.

[0129] When the audio remapper 207 receives this mapping data, it can remap the audio elements accordingly. For example, for an audio source that matches an image object at a position that hasn't changed much, audio remapper 207 can proceed to make no spatial changes to the corresponding audio element. However, if the mapping data instead describes this for that specific image, and the position of the image corresponding to the audio source has changed drastically, the audio remapper 207 can proceed to make a corresponding change to the position of the audio element. corresponding. For example, if the content-dependent non-uniform mapping has shifted the corresponding position substantially in the most opposite direction from the middle towards the right edge of the second image, the audio remapper 207 may change the position of the corresponding audio element so that it is more to the right.

[0130] Thus, in addition to performing a content-dependent non-uniform mapping for an image, the image remapper 203 also generates mapping data that describes the specific mapping that is applied to the image. This mapping data is provided to the audio remapper 207, which can modify the spatial property of the audio elements based on the description provided by the mapping, that is, the spatial property modifications can be adapted to fit the non-uniform mapping dependent on specific content that was executed in order to provide the aspect ratio change between the first and second images. Obviously, the exact mapping data (eg what is described and how it is described) and the specific adaptation of spatial properties are application and implementation specific, and will depend on the results and properties that are desired in the specific modality .

[0131] In some embodiments, the amount of adaptation can be further controlled by the corresponding phase difference and/or phase difference variation in the corresponding time-frequency block or frequency range. For example, when the corresponding phase difference is zero, full adaptation is done and the adaptation is reduced as the phase difference increases.

[0132] As an alternative or in addition to applying a spatial amplification algorithm, a source separation algorithm (of which there is also a wide variety) can be used to extract individual sources along with directional information from the channel-based content. These individual extracted sources and their corresponding directional information can then be remapped in a similar manner as described above for audio objects.

[0133] Thus, in many embodiments, the audio remapper 207 can be arranged to change a spatial position for the first audio element from a corresponding position to an image position in the first image to an image position in the second Image.

[0134] In many embodiments, the received data can include metadata that link audio elements to image objects. Audio Remapper 207 can use this connection information to determine the mapping that should be applied to the individual audio element. In other embodiments, no explicit connection data can be provided, and the apparatus can be arranged to analyze the received audiovisual data to link audio elements and image objects. For example, after identifying protruding objects, the audio mapper 207 can, for a given image object, analyze the audio elements in order to determine an audio element that matches the expected properties for the image object. Specifically, the audio element that has an indicated position closest to the protruding object in the first image can be selected. In some embodiments, the characteristics of the image object and the audio elements can be additionally taken into account. For example, if the image object is identified as corresponding to a face (for example, the image object may have been determined by face detection), the audio element can be selected as an audio element that has properties that are indicative of speech.

[0135] In the device in Figure 2, the change in the audio/video aspect ratio is associated with a change in the spatial properties of the associated spatial audio content. Audio content can be modified to maintain a better quality of overall experience. Linked modification of the audio content aspect ratio has been found to provide a substantially improved user experience, especially for immersive media content such as VR and 360 degree movie content.

[0136] The most direct approach would be to apply a linear remapping of the spatial positions of the audio elements according to the relationship between the original and modified aspect ratio. However, in the device in Figure 2, a more advanced approach is used, with the content, dependent on the aspect ratio remapping, performed on the visual side, and with this dynamic mapping the remapping of the audio side is further controlled. . This can provide an enhanced effect and, in particular, can provide a more consistent user experience.

[0137] The approach can, therefore, link the remapping of the audio elements directly to the non-linear remapping of the video image. For example, if the horizontal and vertical x and y positions of a certain pixel or visual object in the video image are remapped to the ax and by positions, the audio corresponding to the position (x, y) can also be remapped to the modified position (ax , by).

[0138] In the approach, the spatial remapping of the audio elements of an audiovisual content item can be performed depending on a non-uniform content dependent video/image remapping operation that is performed to adapt the video image aspect ratio to the aspect ratio of a monitor.

[0139] In many embodiments, the audio remapper 207 can be combined additionally or alternatively to modify a spatial distribution property of the first audio element in response to the mapping data. The spatial distribution property can reflect the degree of location of the audio element and therefore the degree to which the rendered audio element is considered to originate from a single point. Dispersion can typically be indicated by size parameters or by one or more radii, indicating the size or physical or acoustically perceived extent of an object. Alternatively, it can be an immersion rate or an angle range for which the user is receiving or sensing received sound waves. As another example, it might be a value that indicates the object's diffusion as a value, for example, between 0 and 1, that the renderer converts to a render between a pure point source rendering and a fully immersive fuzzy rendering. Typically this means rendering to multiple positions or to multiple speakers. Larger distributions will require smaller correlations between various rendered positions to avoid artifacts and improve naturalness.

[0140] The audio remapper 207 can therefore be arranged to change the perceived width or spread for a given audio element.

[0141] For example, the distribution of a certain audio element can be obtained by rendering the object in several positions. For example, by convoluting the object's signal with multiple HRTFs for binaural processing. Gains can be applied to renderings at various positions to vary the energy distribution spatially. Different implementations can make different tradeoffs between rendering the position range (maximum distance between render positions) and locating the spatial energy distribution by gains. Typically, this approach works well for small distribution and rendering values in several positions relatively close to each other, for example within a 30 degree angle as seen from the listener.

[0142] Other modalities, typically for high quality rendering and large distribution values, can generate multiple mutually decorrelated (or partially decorrelated) versions of the object and render them in a variety of locations dependent on the distribution data.

[0143] Alternatively, this can be achieved by applying a spatial amplification algorithm to the audio channels of the audio element. For an audio object, a de-correlated signal can, for example, be generated, and the output signal (eg, a binaural signal) can be generated as a combination of the de-correlated signal with the original signal.

[0144] The algorithm for modifying the distribution of an audio element is adapted based on the mapping data provided by the image remapping 203. For example, if the corresponding object or visual segment of the image is stretched or compressed, the element distribution of audio can also be respectively increased. This scenario typically occurs for background features that do not protrude from the scene.

[0145] As an example of low complexity, as illustrated in Figures 3 and 4, the device can render an audio object as having an average distribution, by rendering it from four different positions and rendering an audio object as having a large average distribution by rendering it from four different positions that are more distributed and can further add decorrelation between the different positions. The audio object in the second example will be perceived as having a much larger distribution than in the first example.

[0146] In many modalities, the spatial modification of the audio element can also depend on the acoustic/audio properties of the audio elements, for example, the audio properties can be evaluated to decide if the spatial property should be changed and how. must be done. Thus, in many modalities, the spatial modification of audio elements may depend on an audio characteristic represented by the audio element.

[0147] In particular, in many embodiments, the spatial modification of a particular audio element performed by the audio remapper 207 may depend on the degree of spatial distribution of the audio element. Thus, the spatial modification may depend on the degree of location or the spatial extent of the audio element. This can, for example, be represented or estimated by the degree of diffusion of the audio element. The more diffuse the audio rendered by the audio renderer 209, the less the audio source corresponding to that audio element will be perceived as being in a specific location and the more it will be perceived as being distributed over a larger area.

[0148] In some embodiments, the audio remapper 207 can be arranged to apply, for example, a larger shift in position to a greater degree of spatial distribution of the audio element. For example, an audio element with a high degree of distribution can be moved further to the side after an aspect ratio change to provide a wider spatial impression that can match the wider viewing angle. For example, a wider monitor can benefit from generated audio that extends further to the side and with more audio sources positioned to the side of the listener rather than in front. However, greater consistency between audio and video is maintained if this greater sound stage width is achieved by audio sources that are not associated with specific, narrow, well-defined positions, but rather with audio sources that are not directly linked to specific positions in the audio scene.

[0149] In many embodiments, the audio remapper 207 is arranged to reduce the level of change in the position of an audio element in response to an increasing degree of spatial distribution for the audio element. This can, for example, be particularly suitable for audio elements that are directly linked to image objects that are moved by remapping images 203. For example, a specific, well-defined and localized audio object can be moved to directly match the new position. However, if the image object is linked to an audio element that is more diffuse and spread over a larger domain, the audio remapper 207 can reduce the position shift. For example, the new position can be determined to be somewhere between the original position and the modified position. This can, in many scenarios, provide improved perception as it can provide a better compromise between the original sound stage and the modified sound stage. This may, for example, reflect that there is a reduced perceptual link between a specific image object and the sound linked to that image object for greater audio distribution.

[0150] In fact, in many embodiments, the audio remapper 207 can be arranged to select whether to apply a modification to the spatial property or not depending on the degree of spatial distribution. For example, for audio elements that have a distribution measure that is below a certain threshold, it can be considered sufficiently tightly linked to a specific scene object so that when the image object corresponding to the scene object is moved by the image remapper 203, the audio element is correspondingly moved to maintain the strict correspondence between the image object and the audio object, that is, between the visual and audio experience. However, for an audio element for which the distribution measure is above the threshold, no position modification can be introduced by the audio remapper 207. This can result in the audio element being rendered to be perceived as being of the original position rather than being from the image object's new position, but as the perception of the position will be fuzzy and imprecise, any inconsistencies between the visual side and the audio side may be insignificant and may be outweighed by the probability of providing greater consistency between elements such as specifically between ambient audio sources. Thus, this can provide improved consistency of the rendered audio stage.

[0151] In some embodiments, the audio remapper 207 can be arranged to modify the spatial property in response to whether the first audio element is designated as background audio or not. For example, audio elements can be divided into two categories that correspond to foreground objects and background objects, and audio elements from the two categories can be processed differently. As an example of low complexity, all audio elements categorized as foreground objects can be repositioned to directly match the repositioning of the corresponding image elements, while no position changes can be applied to audio elements categorized as background elements . This can, in many scenarios, provide an enhanced sensing experience.

[0152] In this way, a distinction can be made between “foreground” and “background” audio elements. "Foreground" audio elements can be determined as distinct and localized audio elements, while "background" audio elements can be determined as nonlocalized audio elements, such as "ambient" sounds or background music . For example, in an object-based audio use case, a distinction can be made between highly localized (point) sources that can be considered to correspond to “foreground” elements and non-localized (i.e., fuzzy) objects that can be considered as corresponding to “background” elements.

[0153] In other embodiments, categorization as "foreground" or "background" audio elements may be based on explicit indications in the received bit stream, eg metadata may be provided for each audio element indicating whether this is a foreground or background audio object. In fact, several standardized audio formats already provide metadata elements to signal that an audio object is a punctual or diffuse sound source, which can be used as an indication of whether the audio object is a foreground audio object or background.

[0154] As mentioned, the audio remapper 207 can simply consider that only the foreground audio elements should be subjected to a possible position remapping, while the spatial distribution of the background sounds/audio elements can be maintained without modification. It should be recognized that, in other embodiments, other adaptations can be used, for example, a magnification or widening effect can be added to the background audio elements but not to the foreground audio elements.

[0155] In some embodiments, the audio remapper 207 can be arranged to modify the spatial property in response to an audio type of the first audio element. The audio type can be specifically determined as a type from a group comprising at least one audio object, one audio channel, and one higher-order ambisonic audio element.

[0156] For example, audio objects are often used to represent individual and specific audio sources, while audio and ambisonic channels are used to represent more ambient audio, including a plurality (and typically many) audio sources many different. Therefore, audio objects can, for example, in some modalities, be subject to a change in position corresponding to the change in aspect ratio, while audio channels and/or ambisonic elements can be left unmodified.

[0157] As another example, in some embodiments, the audio remapper 207 can be arranged to apply non-uniform spatial magnification to audio elements that are audio channels, while no spatial magnification is applied to audio objects.

[0158] In a hybrid use case where the audio element may comprise audio objects and audio elements based on channels and/or scene (eg higher order ambisonics), a distinction can be made between, by a on the one hand, objects (“foreground”) and, on the other hand, channel/scene-based elements (“background”). As a refinement of this, for audio objects, a distinction can also be made between point and diffuse sources, as described above. Likewise, scene-based elements, such as higher-order ambisonic elements, can be separated into “primary” and “environmental” components. These different categories can then be processed differently by the audio remapper 207.

[0159] For example, audio objects can be rendered according to the mapping data of their corresponding first position in relation to the image, while audio channels, such as the front left and right channels, are rendered at larger angles or smaller than their nominal positions, depending on an average or maximum position change resulting from the aspect ratio change. Higher Order Ambisonics (“HOA”) elements can be rendered unmodified or with minimal spatial stretch or compression to avoid artifacts. HOA elements can be converted to directional components for rendering with HRTFs or loudspeaker rendering algorithms such as VBAP. Front stage directional component signals can be rendered at wider or narrower angles to stretch or compress the front stage according to changing the aspect ratio. If the HOA directional components have sufficient spatial resolution, the render position updates can be varied to match the spatial remapping variation of the image.

[0160] The rendered positions of the directional components outside the front stage can be left unchanged or can be rendered at smaller or narrower wider angles, similar to what happens with the front stage, to obtain a spatial fading of the stretch effect / compression towards positions further away from the front stage.

[0161] The front stage can be defined as the part of 3D space that is visible to the user. For example, the part that is displayed on the 2D or HMD monitor.

[0162] In some embodiments, the audio remapper 207 can be arranged to modify the spatial property in response to whether the first audio element is linked to an image object in the first image or not. For example, if the data stream is composed of metadata that indicates that a certain audio element is linked to a specific image object (or even that it only has a corresponding image object in the image without any further indication of what that is. image), the audio remapper 207 can modify the spatial position of the audio element so that it follows the image segment that has a corresponding position in the original input image. However, for audio elements that are not linked to objects in the image, the audio remapper 207 can proceed to not change any position but possibly increase the spatial distribution/spread. This can be beneficial as a wider distribution diffuses the object's location and therefore reduces the likelihood of a mismatch with a potential visual counterpart that is modified by the image remapper.

[0163] In some embodiments, the modification of the 207 audio remapper may be based on an explicit indication in the bitstream that a specific audio element (for example, an audio object) is linked to a specific visual element (by example, a visual object) in the visual image. In this way, it is possible to identify audio elements that must be modified according to a possible remapping of the linked visual object. So, in this case, an audio element can be modified and remapped only if the linked visual is remapped by the intelligent video upscaling algorithm performed by the 203 image remapper. audio elements that are linked to visual elements that are not remapped by the image remapper 203 intelligent video upscaling algorithm might not be remapped in this example.

[0164] In a variation of this example, explicit indication in the bitstream may, less specifically, simply indicate that an audio element is "screen-related" (which implies that an upscaling of the visual image should result in a similar remapping of that audio element).

[0165] In some embodiments, the audio remapper 207 may be arranged to modify the spatial property in response to a monitor property used to render the second image. In addition to indirectly adapting the monitor's aspect-ratio-dependent processing through the mapping data of the image remapper 203, the audio remapper 207 can also consider other monitor properties, such as specifically the monitor size.

[0166] For example, in many embodiments, the audio remapper 207 can receive data indicating the size of the monitor. If the monitor is a small monitor intended for viewing from a distance, the audio remapper 207 can be arranged so as not to increase the spatial distribution (or may even reduce it) even for background and ambient audio. However, if the monitor is large, the 207 audio remapper can substantially increase the spatial distribution (eg,

increase diffusion or perform spatial expansion). In this way, for a small monitor, an audio experience can be provided which is perceived as less expansive than for a large monitor, thus corresponding with the visual experience.

[0167] In some embodiments, the audio remapper 207 can be arranged to modify the spatial property in response to a viewer's position in relation to the monitor. The audio remapper 207 can, for example, adapt the processing to reflect the user's orientation and position relative to the monitor so that rendered audio is perceived as coming from the proper position. For example, if the user is located on the side of the monitor and views the monitor at an angle, the placement of audio sources can be corrected to reflect this.

[0168] As another example, the distance from the viewer's position to the monitor can be considered. For example, for a user sitting close to the monitor, the position of most audio elements, including many more fuzzy audio elements, can be modified to follow corresponding changes in image position. This can allow for a closer match between the visual experience and the audio experience as even more diffuse sounds are perceived as coming from the “right” part of the monitor. However, for a user positioned further away from the monitor, it may not be possible to differentiate the audio coming from different parts of the monitor and it may be more attractive to generate a user experience that provides a larger audio scene than one that directly matches the size of the monitor.

Therefore, spatial magnification and other algorithms to increase the spatial distribution of more ambient audio can be applied by Audio Remapper 207, resulting in a more immersive experience.

[0169] The size of the monitor and/or the position of the viewer can, for example, be used to estimate or determine a viewing window or opening for the viewer. Audio Remapper 207 can adapt processing based on this viewport. The fit may additionally or alternatively depend on the position of the audio element in relation to the viewport of the first image.

[0170] For example, audio elements that are positioned outside the original viewport may not be modified, while the audio element within the viewport is modified to reflect the aspect ratio remapping changes. For example, audio objects that are placed behind or above the user may not be modified. Another example is an ambient audio component that is present around the user, for example an ambient HOA element. This does not require the environment component to be the same in all directions. It could be, for example, an HOA element that contains a complete 3D recording of a restaurant's background ambiance).

[0171] It should be considered that, for clarity, the above description described the embodiments of the invention with reference to different circuits, units and functional processors. However, it will be evident that any suitable distribution of functionality between different functional circuits, units or processors can be used without departing from the invention. For example, illustrated functionality being performed by separate processors or controllers may be performed by the same processor or by the same controllers. Therefore, references to specific functional units or circuits should be considered only as references to suitable means of providing the described functionality and not as indicative of a strict logical or physical structure or organization.

[0172] The invention may be implemented in any suitable form, including hardware, software, firmware or any combination thereof. The invention may optionally be implemented at least partially as computer software that runs on one or more data processors and/or digital signal processors. The elements and components of an embodiment of the invention may be physically, functionally and logically implemented in any suitable way. In fact, functionality can be implemented in a single unit, in a plurality of units, or as part of other functional units. Thus, the invention can be implemented in a single unit or it can be physically and functionally distributed among different units, circuits and processors.

[0173] Although the present invention has been described in connection with some embodiments, it is not intended to limit it to the specific form presented here. Rather, the scope of the present invention is limited only by the appended claims. Additionally, although it may appear that a feature is described in connection with specific embodiments, one skilled in the art will recognize that various features of the described embodiments may be combined in accordance with the invention. In the claims, the term “comprising” does not exclude the presence of other elements or other steps.

[0174] Furthermore, although individually mentioned, a plurality of means, elements, circuits or method steps can be implemented, for example, by means of a single circuit, a single unit or a single processor.

Additionally, although individual features may be included in different claims, they may be advantageously combined, and their inclusion in different claims does not imply that a combination of features is not feasible and/or advantageous.

In addition, the inclusion of an appeal in a category of claims does not imply limitation to that category, but rather indicates that the appeal is equally applicable to other categories of claims, as appropriate.

Furthermore, the order of features in claims does not imply any specific order in which the features need to be worked and, in particular, the order of individual steps in a method claim does not imply that the steps need to be performed in that order.

The steps can actually be performed in any suitable order.

Furthermore, singular references do not exclude a plurality.

Thus, references to “a”, “some”, “first”, “second”, etc., do not exclude a plurality.

Reference signs in the claims are provided merely as illustrative examples and are not to be construed as limiting the scope of the claims in any way.

Claims (15)

1. APPARATUS FOR PROCESSING AUDIOVISUAL DATA FOR A SCENE, characterized in that it comprises: a receiver (201) for receiving audiovisual data for the scene, the audiovisual data comprising audio data for the scene comprising a plurality of audio and data elements image for at least one first image of the scene, the first image having a first aspect ratio; an image remapper (203) for performing content-dependent non-uniform mapping of the first image to a second image with a second aspect ratio, the second aspect ratio being different from the first aspect ratio, and the image remapping ( 203) is arranged to generate mapping data describing the content-dependent non-uniform mapping; and an audio remapper (207) arranged to replace a first audio element of the plurality of audio elements with a second audio element generated by modifying a spatial property for the first audio element based on the mapping data. Apparatus according to claim 1, characterized in that the audio remapper (207) is arranged to generate the second audio element by modifying a spatial position property of the first audio element in response to the mapping data. 3. APPARATUS according to any one of claims 1 or 2, characterized in that the audio remapper (207) is arranged to generate the second audio element by changing a spatial position to the first audio element of a corresponding position to an image position in the first aspect ratio to an image position in the second image. Apparatus according to any one of claims 1 to 3, characterized in that the audio remapper (207) is arranged to generate the second audio element by modifying a spatial distribution property of the first audio element in response to the mapping data. APPARATUS according to any one of claims 1 to 4, characterized in that the audio remapper (207) is arranged to generate the second audio element by modifying the spatial property in response to an audio property of the first element of audio. Apparatus according to claim 5, characterized in that the acoustic property comprises a degree of spatial distribution for the first audio element. Apparatus according to claim 6, characterized in that the audio remapper (207) is arranged to reduce a change in position between the second audio element and the first audio element to an increasing degree of spatial distribution to the first audio element. 8. APPARATUS according to any one of claims 6 or 7, characterized in that the audio remapper (207) is arranged to select whether or not the modification should be applied to the spatial property depending on the degree of spatial distribution. 9. APPARATUS according to any one of claims 1 to 8, characterized by the audio remapper (207) be arranged to modify the spatial property in response to an audio type of the first audio element. 10. APPARATUS according to any one of claims 1 to 9, characterized in that the audio remapper (207) is arranged to modify the spatial property in response to whether or not the first audio element is linked to an image object On the first image. Apparatus according to any one of claims 1 to 10, characterized in that the audio remapper (207) is arranged to modify the spatial property depending on whether the first audio element is designated as background audio or not . Apparatus according to any one of claims 1 to 11, characterized in that the audio remapper (207) is arranged to modify the spatial property in response to a property of a monitor to render the second image. Apparatus according to any one of claims 1 to 12, characterized in that the audio remapper (207) is arranged to modify the spatial property in response to a position of the viewer in relation to a monitor to render the second image. 14. METHOD OF PROCESSING AUDIOVISUAL DATA FOR A SCENE, characterized in that it comprises: receiving audiovisual data for the scene, the audiovisual data comprising audio data for the scene comprising a plurality of audio elements and image data for at least a first image of the scene, the first image having a first aspect ratio; performing a content-dependent non-uniform mapping of the first image to a second image with a second aspect ratio, where the second aspect ratio is different from the first aspect ratio, and the content-dependent non-uniform mapping comprises data generation of mappings that describe content-dependent non-uniform mapping; and replacing a first audio element of the plurality of audio elements with a second audio element generated by modifying a spatial property for the first audio element based on the mapping data. 15. COMPUTER PROGRAM PRODUCT, characterized in that it comprises computer program code means adapted to perform all the steps as defined in claim 14 when said program is executed on a computer.