RU2807582C2 - Method, device and stream for 3d video format - Google Patents

Abstract

FIELD: volumetric video content.

SUBSTANCE: invention is related in particular to encoding and/or formatting of data representing volumetric content. According to the proposed solution, the first data representing the texture of the 3D scene is encoded into at least the first track, the second data representing depth associated with points of the 3D scene is encoded into at least a second track, the instructions for retrieving at least a portion of said first data and second data from at least a portion of said first track and second track is encoded into at least a third track. The first data is located in a plurality of first segments into which a first frame is divided, with a portion of the 3D scene associated with each first segment of the plurality of first segments. The second data is located in a plurality of second segments into which the second frame is divided, wherein the total number of second segments of the second frame is greater than the total number of first segments of the first frame. The resulting rendered content may be presented on an end user's viewing device, such as an HMD, according to a viewpoint orientation consistent with the instructions.

EFFECT: efficient encoding/decoding of data representing volumetric content.

20 cl, 21 dwg

Description

1. Field of technology

The present invention relates to the field of volumetric video content. The present invention should also be understood in the context of encoding and/or formatting data representing rich content, for example, for display on end user devices such as mobile devices or Head Mounted Displays.

2. State of the art

This section is intended to familiarize the reader with various aspects of the art that may be associated with various aspects of the present invention that are described and/or claimed below. It is intended that this discussion will be useful in providing the reader with background information to facilitate a better understanding of the various aspects of the present invention. Accordingly, these statements should be read in that light and not as statements of prior art.

Recently, there has been an increase in the availability of content with a large field of view (up to 360°). Such content is potentially not fully visible to the user viewing the content on immersive display devices such as Head-Mounted Displays, smart glasses, PC screens, tablets, smart phones and the like. This means that a user can only view part of the content at a given time. Typically, however, the user can navigate the content through various means, such as moving the head, moving the mouse, touch screen, voice, and the like. It is usually desirable to encode and decode this content.

Immersive video, also called 360° flat video, allows the user to view everything around them by rotating their head around a stationary point of view. Rotations provide only 3 Degrees of Freedom (3DoF) perception. Even if 3DoF video is sufficient for the first omnidirectional video perception, for example using a Head Mounted Display (HMD) device, 3DoF video can quickly become a disappointment for a viewer who would have expected more freedom, for example through parallax perception. In addition, 3DoF can also cause dizziness because the user never only rotates their head, but also moves their head in three directions, and these movements are not reproduced in the perception of 3DoF video.

The wide field of view content can be, among other things, a 3D computer graphics video scene (3D CGI scene), a point cloud, or an immersive video. Many concepts can be used to create such immersive videos: Virtual Reality (VR), 360, panoramic, 4π steradian, immersive, omnidirectional or wide field of view, for example.

Volumetric video (also known as 6 Degrees of Freedom (6DoF) video) is an alternative to 3DoF video. When watching 6DoF video, in addition to rotation, the user can also move their head, and even their body, within the content being viewed and perceive parallax and even volumes. Such videos significantly increase the sense of immersion and depth perception of a scene, and prevent vertigo by providing consistent visual feedback during head movements. The content is created through special sensors that simultaneously record the color and depth of the scene of interest. The use of a color camera setup combined with photogrammetry techniques is a common way of performing such recording.

Whereas 3DoF videos contain a sequence of images resulting from the demapping of texture images (e.g., spherical images encoded according to a latitude/longitude projection mapping or an equirectangular projection mapping), 6DoF video frames contain information from multiple viewpoints. They can be viewed as time series of point clouds obtained from 3D capture. Two types of three-dimensional videos can be considered depending on viewing conditions. The former (i.e., full 6DoF) allows for complete free navigation of video content, while the latter (also known as 3DoF+) limits the user's viewing space to a limited volume, allowing limited head movement and parallax perception. This second context is a valuable compromise between free navigation and the passive viewing environment of a seated audience member.

3DoF videos may be encoded into the stream as a sequence of rectangular color images formed according to a selected projection mapping (eg, cubic projection mapping, pyramidal projection mapping, or equirectangular projection mapping). The advantage of this encoding is the use of generally accepted image and video processing standards. 3DoF+ and 6DoF videos require additional data to encode the depth of the color pixels of the point clouds. The rendering type (i.e., 3DoF or volumetric rendering) for a volumetric scene is not known in advance when the scene is encoded into the stream. Today, streams are coded for one type of visualization or another. There is no stream, and associated methods and devices, that can carry data representing a volumetric scene that can be encoded once and decoded as either 3DoF video or volumetric video (3DoF+ or 6DoF).

Moreover, the amount of data that must be transported for, for example, visualization on end-user devices can be very important, significantly increasing the bandwidth requirements across networks.

3. Essence of the invention

References in the technical description to “one embodiment,” “an embodiment,” “an exemplary embodiment,” or “a specific embodiment” indicate that the described embodiment may include a particular feature, structure, or characteristic, but that each embodiment does not may necessarily include a specific feature, structure or characteristic. Moreover, such phrases do not necessarily refer to the same embodiment. Moreover, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is contemplated that one skilled in the relevant art would be able to influence such feature, structure, or characteristic in connection with other embodiments, whether or not it is explicitly described.

The present invention relates to a method for encoding data representing a 3D scene, the method comprising the steps of:

- encoding, into at least a first track, first data representing a texture of the 3D scene that is visible in accordance with the first viewpoint, the first data being located in a plurality of first tiles of the first frame, wherein a portion of the 3D scene is associated with each first segment of the plurality of first segments;

- encoding, into at least a second track, second data representing depth associated with points of the 3D scene, wherein the second data is located in a plurality of second segments of the second frame, wherein the total number of second segments of the second frame is greater than the total number of first segments of the first frame, wherein for each first segment at least a portion of the plurality of first segments:

a set of second segments comprising at least one second segment of the plurality of second segments is allocated to each first segment of at least a portion of the plurality of first segments;

a set of at least a patch is located within a set of second segments, each patch corresponding to a two-dimensional parameterization of a group of 3D points that are contained in a portion of the 3D scene associated with each first segment, and comprising second data representing depth associated with the 3D group points;

- encoding, into at least the third track, at least an instruction for retrieving at least a portion of the first data and the second data from at least a portion of the at least the first track and at least the second track.

The present invention relates to a device configured to encode data representing a 3D scene, the device comprising a memory coupled to at least one processor configured to:

- encoding, into at least a first track, first data representing a texture of a 3D scene that is visible in accordance with the first viewpoint, the first data being located in a plurality of first segments of the first frame, a portion of the 3D scene being associated with each first segment from a plurality of first segments;

- encoding, into at least a second track, second data representing depth associated with points in the 3D scene, the second data being located in a plurality of second segments of the second frame, wherein the total number of second segments of the second frame is greater than the total number of first segments of the first frame, wherein for each first segment at least a portion of the plurality of first segments:

a set of second segments comprising at least one second segment of the plurality of second segments is allocated to each first segment of at least a portion of the plurality of first segments;

the set of at least a fragment is located in a set of second segments, each fragment corresponding to a two-dimensional parameterization of a group of 3D points that are contained in a portion of the 3D scene associated with each first segment, and contains second data representing a depth associated with the 3D points of the group;

- encoding, into at least a third track, at least instructions for retrieving at least a portion of the first data and second data from at least a portion of the at least the first track and at least the second track.

The present invention relates to a device configured to encode data representing a 3D scene, the device comprising:

- an encoder configured to encode, into at least a first track, first data representing the texture of a 3D scene that is visible in accordance with the first viewpoint, the first data being located in a plurality of first segments of the first frame, wherein a portion of the 3D scene associated with each first segment of the plurality of first segments;

- an encoder configured to encode, into at least a second track, second data representing depth associated with points in the 3D scene, the second data being located in a plurality of second segments of the second frame, wherein the total number of second segments of the second frame is greater the total number of first segments of the first frame, wherein for each first segment at least part of the plurality of first segments:

a set of second segments comprising at least one second segment of the plurality of second segments is allocated to each first segment of at least a portion of the plurality of first segments;

the set of at least a fragment is located in a set of second segments, each fragment corresponding to a two-dimensional parameterization of a group of 3D points that are contained in a portion of the 3D scene associated with each first segment, and contains second data representing a depth associated with the 3D points of the group;

- an encoder configured to encode, into at least a third track, at least instructions for extracting at least a portion of the first data and second data from at least a portion of the at least the first track and at least the second track.

The present invention relates to a device configured to encode data representing a 3D scene, the device comprising:

- means for encoding, into at least a first track, first data representing the texture of a 3D scene that is visible in accordance with a first viewpoint, the first data being located in a plurality of first segments of the first frame, a portion of the 3D scene being associated with each a first segment of the plurality of first segments;

means for encoding, into at least a second track, second data representing depth associated with points in the 3D scene, the second data being located in a plurality of second segments of the second frame, wherein the total number of second segments of the second frame is greater than the total number of the first segments of the first frame, wherein for each first segment at least part of the plurality of first segments:

a set of second segments comprising at least one second segment of the plurality of second segments is allocated to each first segment of at least a portion of the plurality of first segments;

the set of at least a fragment is located in a set of second segments, each fragment corresponding to a two-dimensional parameterization of a group of 3D points that are contained in a portion of the 3D scene associated with each first segment, and contains second data representing a depth associated with the 3D points of the group;

- means for encoding, into at least a third track, at least instructions for retrieving at least a portion of the first data and second data from at least a portion of the at least the first track and at least the second track.

In accordance with the particular distinctive property, each fragment further includes third data representing a texture associated with the 3D points of the group, the third data being encoded into at least a second track.

In accordance with a particular feature, a set of third frame third segments comprising at least one third segment is allocated to each first segment, and a set of at least a fragment containing third data representing a texture associated with the 3D points of the group is allocated to the set third segments, wherein the at least fragment corresponds to a two-dimensional parameterization of a group of 3D points, wherein the third data is encoded into at least a third track.

According to another particular feature, when the size of a set fragment is larger than the size of the second segment into which the fragment is to be placed, then the fragment is split into a plurality of sub-fragments, each of a size smaller than the size of the second segment.

In accordance with a further special feature, the tiles of the set are prioritized based on the visual importance of the tiles, wherein the visual importance depends on second data associated with the tiles.

In accordance with a further special feature, the second segments have the same size, which is fixed for a plurality of successive second frames.

The present invention relates to a method for decoding data representing a 3D scene, the method comprising the steps of:

- decoding, from at least the third track, at least an instruction for extracting first data and second data from at least the first track and at least the second track;

- decoding first data from at least a first track, wherein the first data represents a 3D scene texture that is visible in accordance with the first viewpoint, wherein the first data is located in a plurality of first segments of the first frame, wherein a portion of the 3D scene is associated with each first segment of a set of first segments;

- decoding second data from at least a second track, wherein the second data represents depth associated with points in the 3D scene, wherein the second data representing depth is contained in at least a fragment that is located in at least a set of second segments of the second frame wherein a set of second segments is allocated to each first segment, wherein the at least fragment corresponds to a two-dimensional parameterization of a group of 3D points that are contained in at least a portion of the 3D scene associated with each first segment, wherein the second data represents a depth associated with the 3D points groups.

The present invention relates to a device configured to decode data representing a 3D scene, the device comprising a memory associated with at least one processor configured to:

- decoding, from at least the third track, at least instructions for extracting first data and second data from at least the first track and at least the second track;

- decoding first data from at least a first track, wherein the first data represents a texture of a 3D scene that is visible in accordance with the first viewpoint, wherein the first data is located in a plurality of first segments of the first frame, wherein a portion of the 3D scene is associated with each first segment of a set of first segments;

- decoding second data from at least a second track, wherein the second data represents depth associated with points in the 3D scene, wherein the second data representing depth is contained in at least a fragment that is located in at least a set of second segments of the second frame wherein a set of second segments is allocated to each first segment, wherein the at least fragment corresponds to a two-dimensional parameterization of a group of 3D points that are contained in at least a portion of the 3D scene associated with each first segment, wherein the second data represents a depth associated with the 3D points groups.

The present invention relates to a device configured to decode data representing a 3D scene, the device comprising:

- a decoder configured to decode, from at least the third track, at least instructions for extracting first data and second data from at least the first track and at least the second track;

- a decoder configured to decode first data from at least a first track, wherein the first data is a texture of a 3D scene that is visible in accordance with the first viewpoint, wherein the first data is located in a plurality of first segments of the first frame, wherein a portion of the 3D scene is associated with each first segment of the plurality of first segments;

- a decoder configured to decode second data from at least a second track, wherein the second data represents depth associated with points in the 3D scene, wherein the second data representing depth is contained in at least a fragment that is located in at least a set of second segments of a second frame, wherein the set of second segments is allocated to each first segment, wherein the at least fragment corresponds to a two-dimensional parameterization of a group of 3D points that are contained in at least a portion of the 3D scene associated with each first segment, wherein the second data represents depth , associated with the 3D points of the group.

The present invention relates to a device configured to decode data representing a 3D scene, the device comprising:

- means for decoding, from at least the third track, at least instructions for extracting first data and second data from at least the first track and at least the second track;

- means for decoding first data from at least a first track, wherein the first data is a texture of a 3D scene that is visible in accordance with the first viewpoint, wherein the first data is located in a plurality of first segments of the first frame, and a portion of the 3D scene is associated with each first a segment of a plurality of first segments;

- means for decoding second data from at least a second track, wherein the second data represents depth associated with points in the 3D scene, wherein the second data representing depth is contained in at least a fragment that is located in at least a set of second segments of a second frame, wherein a set of second segments is allocated to each first segment, wherein the at least segment corresponds to a two-dimensional parameterization of a group of 3D points that are contained in at least a portion of the 3D scene associated with each first segment, wherein the second data represents a depth associated with 3D points group.

In accordance with a particular feature, third data representing a texture associated with 3D group points that are contained in each fragment is further decoded from at least a second track.

In accordance with a particular feature, third data representing a texture is further decoded from at least a third track, wherein the third data is allocated to a plurality of third segments of a third frame, wherein a set of third segments comprising at least one third segment is allocated to each first segment wherein the set of at least a fragment containing third data associated with the 3D points of the group is located in the set of third segments, wherein the at least fragment corresponds to a two-dimensional parameterization of the group of 3D points.

In accordance with another distinctive property, rendering of at least a portion of the 3D scene is carried out in accordance with the first and second data.

The present invention also relates to a bitstream carrying data representing a 3D scene, the data comprising, at least in a first syntax element, first data representing a texture of the 3D scene that is visible in accordance with a first viewpoint, wherein the first data is associated with a plurality of first segments of a first frame, wherein a portion of the 3D scene is associated with each first segment of the plurality of first segments; at least in a second syntax element, second data representing depth associated with points in the 3D scene, wherein the second data is associated with a plurality of second segments of a second frame, wherein the total number of second segments of the second frame is greater than the total number of first segments of the first frame, wherein the set of second segments comprises at least one second segment allocated to said each first segment, wherein the set of at least a fragment is located within the set of second segments, each fragment corresponding to a two-dimensional parameterization of a group of 3D points that are contained in a portion of the 3D scene associated with each first segment , and contains second data representing the depth associated with the 3D points of the group; wherein the bitstream further carries, in at least a third syntax element, at least an instruction for retrieving at least a portion of the first data and the second data.

The present invention also relates to a computer program product containing program code instructions for executing steps of a method for encoding or decoding data representing a 3D scene when the program is executed on a computer.

The present invention also relates to a (non-transitory) processor-readable storage medium having instructions stored thereon for causing the processor to perform at least the above-mentioned method of encoding or decoding data representing a 3D scene.

4. List of figures

The present invention will be better understood, and other special features and advantages will be apparent, after reading the following description, the description being referred to the accompanying drawings, in which:

- Figure 1 shows an image representing a three-dimensional scene containing a surface representation of several objects, in accordance with a non-limiting embodiment of the present principles;

- Figure 2 illustrates an exemplary arrangement of viewpoints in the scene of Figure 1 and visible points of a given scene from different viewpoints of a given arrangement, in accordance with a non-limiting embodiment of the present principles;

- Figure 3 illustrates parallax perception by showing different views of the scene of Figure 1 in accordance with the point of view of Figure 2, in accordance with a non-limiting embodiment of the present principles;

- Figure 4 shows a texture image of the scene points of Figure 1 as seen from the perspective of Figure 2 in accordance with an equirectangular projection mapping, in accordance with a non-limiting embodiment of the present principles;

- Figure 5 shows an image of the same scene points as in Figure 4, presented in a cubic projection display, in accordance with a non-limiting embodiment of the present principles;

- Figure 6 shows a depth image (also called a depth map) of a 3D scene of Figure 1 in accordance with the view of Figure 2, in accordance with a non-limiting embodiment of the present principles;

- Figure 7 shows a three-dimensional (3D) object model of the 3D scene of Figure 1 and point cloud points corresponding to the 3D model, in accordance with a non-limiting embodiment of the present principles;

- Figure 8 shows an image of Figure 4 divided into multiple segments, in accordance with a non-limiting embodiment of the present principles;

- Figure 9 shows a 2D parameterization of a portion of the 3D scene of Figure 1 associated with one image segment of Figure 4, in accordance with a non-limiting embodiment of the present principles;

- Figure 10 shows a two-dimensional angular sector view of a portion of the scene illustrated in Figure 9, in accordance with a non-limiting embodiment of the present principles;

- Figure 11 shows an example of groups of 3D scene fragments obtained by 2D parameterization of portions of the scene associated with segments of Figure 8, in accordance with a non-limiting embodiment of the present principles;

- Figure 12 shows an example of encoding, transmitting and decoding a 3D scene sequence in a format that is both 3DoF rendering compatible and 3DoF+ rendering compatible, in accordance with a non-limiting embodiment of the present principles;

- Figure 13 shows an example of placing fragments of Figure 11 into frame segments, in accordance with a non-limiting embodiment of the present principles;

- Figure 14 shows an example process for placing fragments of Figure 11 in the segmented frame of Figure 13, in accordance with a non-limiting embodiment of the present principles;

- Figure 15 shows examples of fourth frames obtained from segments of the segmented frame of Figure 13, in accordance with a non-limiting embodiment of the present principles;

- Figure 16 shows an example of the syntax of a bitstream carrying information and data representing the 3D scene of Figure 1, in accordance with a non-limiting embodiment of the present principles;

- Figure 17 shows an example of a process to encode the 3D scene of Figure 1, in accordance with a non-limiting embodiment of the present principles;

- Figure 18 shows an example of a process to decode a 3D scene of Figure 1, in accordance with a non-limiting embodiment of the present principles;

- Figure 19 shows an exemplary architecture of a device that may be configured to implement the method or process described with respect to Figures 14, 17, 18, 20 and/or 21, in accordance with a non-limiting embodiment of the present principles;

- Figure 20 illustrates an example of a method for encoding data representing a 3D scene of Figure 1, implemented, for example, in the device of Figure 19, in accordance with a non-limiting embodiment of the present principles;

- Figure 21 illustrates a method for decoding data representing a 3D scene of Figure 1, implemented, for example, in the device of Figure 19, in accordance with a non-limiting embodiment of the present principles.

5. Detailed Description of Embodiments

The subject matter of the invention is now described with reference to the drawings, in which like reference numerals are used to refer to like elements throughout. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the subject matter of the invention. However, it may be apparent that embodiments of the subject matter may be practiced without these specific details.

The present description illustrates the principles of the present invention. Thus, it will be appreciated that those skilled in the art may develop various arrangements which, although not expressly described or shown herein, embody the principles of the invention.

In accordance with non-limiting embodiments of the present invention, methods and apparatuses are disclosed for encoding volumetric video images (also referred to as 3DoF+ or 6DoF video), for example, into one or more frames, a container, and/or a bitstream. Methods and apparatuses for decoding volumetric video images from a stream and/or frames are also disclosed. Examples of bitstream syntax for encoding one or more volumetric video images/frames are also disclosed.

In accordance with a non-limiting aspect, the present principles will be described with reference to the first specific embodiment of a method (and apparatus configured to) encode data representing a 3D scene (represented by volumetric content, also called immersive video) into one or multiple container and/or bitstream frames.

To achieve this goal, first data representing texture (e.g., color information associated with elements, e.g., points, of a 3D scene) of a 3D scene that is visible according to a first viewpoint is encoded in one or more first tracks associated by with at least part of the first segments of the first frame. The first data corresponds to the 3DoF texture information of the scene view/image, i.e. view of the scene according to the central viewpoint with only texture and no geometry (i.e. depth or 3D information), i.e. “flat” view/image of a 3D scene. The first frame is divided into a plurality of first segments (a segment corresponding to a sub-zone of the frame, and is, for example, defined in the HEVC (High Efficiency Video Coding) standard or Google's VP9 with a different abbreviation), with a portion of the 3D scene associated with each first segment of the first frame.

The second frame is divided into a plurality of second segments, the total number of second segments being greater than the total number of first segments of the first frame. For at least a portion of the first segments, a set of second segments is allocated to each first segment. With respect to each first segment, portions of the 3D scene corresponding to each first segment are allocated to the distributed second segments. The fragment corresponds to a two-dimensional (2D) parameterization of the 3D portion of the scene and contains at least second data representing the depth of at least some points that are contained in the 3D portion.

The second data is encoded in one or more second tracks associated with the second segments.

Instructions adapted to retrieve at least a portion of the first data (from at least a portion of the first tracks) and at least a portion of the second data (from at least a portion of the second tracks) are encoded into one or more third tracks.

The 2D parameterization of the 3D part of a 3D scene corresponds to the 2D representation of the 3D part (for example, the points of the 3D scene that are contained in the 3D part). The 2D parameterization can be obtained in various ways, as will be explained in more detail in the description.

A frame corresponds to an image (eg, a sequence of temporally sequential still images forming a video) and corresponds to an array of pixels, with an attribute (eg, depth information and/or texture information) associated with the pixels of the frame.

A corresponding method (and apparatus capable of) decoding data representing a 3D scene is also described with respect to a non-limiting aspect of the present principles.

Figure 1 shows an image representing a three-dimensional (3D) scene 10 containing a surface representation of several objects. The scene could have been produced using any suitable technology. For example, it may have been created using computer graphics interface (CGI) tools. It could be obtained using color and depth imaging devices. In such a case, it is possible that one or more portions of objects that are not visible from acquisition devices (eg, cameras) may not be represented in the scene as described with respect to Figure 2. The example scene illustrated in Figure 1 contains houses, two characters and a well. Cube 11 illustrates a view space from which a user can view a 3D scene. The view space 11, for example, is centered at the first view point. The 3D scene (or part thereof) may, for example, be represented by a planar image (or planar video, also called omnidirectional image/video) representing the 3D scene according to the first viewpoint, and allowing the user (for example, while wearing an HMD ) view the scene in accordance with 3 degrees of freedom (3Dof), i.e. with rotations around the longitudinal, vertical and transverse axis. The 3D scene (or additional portions of the 3D scene) may be represented by additional data (in addition to the flat image texture/color data) associated with portions of the 3D scene viewed from viewpoints of cube 11 other than the first viewpoint for texture data. The additional data, for example, may contain one or any combination of the following information:

- depth information associated with points in the 3D scene viewed from the first viewpoint;

- depth information associated with parts of the 3D scene viewed from the viewpoints of the cube 11, excluding the first viewpoint;

- texture information (also called color information) associated with portions of the 3D scene viewed from viewpoints of cube 11 excluding the first viewpoint.

The additional data combined with flat video provides 3DoF+ and/or 6DoF viewing in the 3D scene view.

Figure 2 shows an exemplary arrangement of viewpoints in a scene, for example, the 3D scene 10 of Figure 1. Figure 2 also shows the points of a given 3D scene 10 that are visible from/according to different viewpoints of a given arrangement. To enable visualization and presentation via an immersive rendering device (eg, a CAVE or a Head Mounted Monitor (HMD) device), the 3D scene is viewed from a first viewpoint (also referred to as a first viewpoint), such as a first viewpoint 20. The scene point 21 corresponding to the first character's right elbow is visible from the first view point 20 since there is no opaque object lying between the first view point 20 and the scene point 21. In contrast, a point 22 of the 3D scene 10, which corresponds to, for example, the second character's left elbow, is not visible from the first view point 20 because it is obscured by the first character's points.

For 3DoF visualization, only one viewpoint is considered, for example, the first viewpoint 20. The user can rotate their head in three degrees of freedom around the first viewpoint to view different parts of the 3D scene, but the user cannot move the first viewpoint. The points in the scene to be encoded in the stream are the points that are visible from a given first viewpoint. There is no need to encode points in the scene that are not visible from this first viewpoint, since the user cannot access them by moving the first viewpoint.

For 6DoF rendering, the user can move the viewpoint to any location in the scene. In this case, it is important to encode each point in the scene in the content bitstream, since each point is potentially accessible to the user, who can move his/her point of view. At the encoding stage, there is no means of knowing in advance from what point of view the user will view the 3D scene 10.

With respect to 3DoF+ rendering, the user can move the viewpoint within a limited space around the viewpoint, for example, around the first viewpoint 20. For example, the user may move his or her viewpoint within the cube 11 centered on the first viewpoint 20. This allows parallax to be perceived, as illustrated with respect to Figure 3. Data representing the portion of the scene that is visible from any point of view space, such as cube 11, must be encoded into a stream, including data representing the 3D scene that is visible in in accordance with the first review point 20. The size and shape of the view space can, for example, be decided and determined at the encoding stage and can be encoded into a bitstream. The decoder can obtain given information from the bitstream and the renderer limits the view space to the space defined by the received information. According to another example, the renderer determines the view space in accordance with hardware limitations, for example, in relation to the capabilities of the sensor(s) that detect the user's movements. In such a case, if, during the encoding phase, a point that is visible from a point within the view space of the renderer has not been encoded in the bitstream, then the rendering of that point will not occur. According to a further example, data (eg, texture and/or geometry) representing each point in a 3D scene is encoded into the stream without regard to the view space of the rendering. To optimize stream size, only a subset of the scene points may be encoded, for example, the subset of points that can be seen according to the view space of the rendering.

Figure 3 illustrates the perception of parallax that is provided by volumetric (i.e., 3DoF+ and 6DoF) imaging. Figure 3B illustrates the part of the scene that the user can see from the first view point 20 of figure 2. From this first view point 20, two characters are in a given spatial configuration, for example, the left elbow of the second character (in a white shirt) is hidden by the body of the first character, while that his head is visible. When the user rotates his/her head according to three degrees of freedom around the first view point 20, this configuration does not change. If the viewpoint is fixed, the second character's left elbow is not visible. Figure 3A illustrates the same portion of the scene as seen from a viewpoint located on the left side of the figure 2's view space 11. From such a viewpoint, point 22 of the figure 2 is visible due to a parallax effect. As a consequence, in relation to volumetric rendering, point 22 must be encoded into a stream. If not encoded, this point 22 will not be rendered. Figure 3C illustrates the same portion of the scene as viewed from a viewpoint located on the right side of the view space 11 of Figure 3. From this viewpoint, the second character is almost completely hidden by the first character.

By moving the viewpoint in a 3D scene, the user can perceive a parallax effect.

Figure 4 shows a texture image 40 (also referred to as a color image) containing texture information (e.g., RGB data or YUV data) of points in the 3D scene 10 that are visible from the first view point 20 of Figure 2, which texture information is obtained in accordance with equirectangular projection mapping. The equirectangular projection mapping is an example of a spherical projection mapping.

Figure 5 shows an image of 50 points of a 3D scene obtained or encoded according to a cubic projection mapping. There are different cubic projection mappings. For example, the faces of the cube may be placed differently in the figure 5 image 50 and/or the faces may be oriented in a different manner.

The projection mapping used to obtain/encode the scene points that are visible from a certain viewpoint is selected, for example, according to compression criteria or, for example, according to a standard option. One skilled in the art will know that it is possible to transform an image obtained by projecting, for example, a point cloud according to a projection mapping into an equivalent image of the same point cloud according to another projection mapping. Such a conversion may still incur some loss in projection resolution.

Figures 4 and 5 are shown in grayscale. It is naturally clear that these are examples of (color) texture images (encoding the texture (color) of points in the scene), for example in RGB or in YUV. Each of the images 40 and 50 contains data necessary for 3DoF rendering of a 3D scene. A decoder receiving a bit stream or data stream containing, in a first syntax element, an image, such as the exemplary images 40 and/or 50 of Figure 4 and/or Figure 5, decodes the image using a method that correlates with the method used to encode the image. The stream may be encoded according to standard image and video compression techniques and a standard image and video transport format, such as MPEG-2, H.264 or HEVC. The decoder may pass the decoded 3DoF image (or sequence of images) to a renderer or module for reformatting, for example. The 3DoF renderer will first deproject the decoded image from the projection surface (e.g., a sphere in relation to ERP image 40 of Figure 4 or a cube in relation to ERP image 50 of Figure 5) and then generate a rectangular viewport requested by the end user's viewing device. In an embodiment, the renderer transforms the image according to another different projection mapping before projecting it.

The image is compatible with 3DoF rendering, where the image encodes points in the 3D scene according to the projection mapping. The scene can contain 360° points. Projection mappings that are commonly used to encode images compatible with 3DoF rendering correspond, for example, to spherical mappings: equirectangular projection; longitude/latitude projections; or different layouts of cubic projection maps or pyramidal projection maps.

Figure 6 shows a depth image (also called a depth map) of a 3D scene 10 according to a first view point 20. Depth information is required for volume rendering. In the Figure 6 image encoding example, the darker the pixel, the closer the point that is projected to that pixel is from the viewpoint. For example, the depth may be encoded in twelve bits, i.e. depth is represented as an integer in the range from 0 to 2 ¹² -1 (= 4095). If, for example, the closest point is 1 meter from a certain viewpoint and the farthest point is 25 meters from a certain viewpoint, then the linear depth encoding will be done in steps of 0.586 centimeters (=(2500-100)/4096). Depth can also be encoded according to a logarithmic scale, since the inaccuracy in the depth value of a point that is farther away from the viewpoint is less important than the inaccuracy in the depth value for a point that is closer to the viewpoint. In the exemplary embodiment of Figure 6, the depth of the scene points that are visible from the viewpoint is encoded in the depth map in accordance with the same projection mapping as the projection map used to encode the color map of Figure 5. In another embodiment, the depth may be encoded in consistent with another projection mapping. The renderer converts the depth map and/or color image into de-projection points of the scene encoded in this data.

Figure 7 shows a three-dimensional (3D) model of an object 70 and point cloud 71 points corresponding to the 3D model 70. Model 70 may be a 3D representation of a mesh, and point cloud 71 points may be mesh vertices. The points of the point cloud 71 may also be points scattered across the surface of the mesh faces. Model 70 may also be represented as a splatting version of point cloud 71, wherein the surface of model 70 is created by splatting points of point cloud 71. The model 70 can be represented through a variety of different representations, such as voxels or splines. Figure 7 illustrates the fact that a point cloud can be defined using a surface representation of a 3D object and that a surface representation of a 3D object can be generated from a cloud point. As used herein, projecting 3D object points (by extending 3D scene points) onto an image is equivalent to projecting any image representation of a given 3D object to create the object.

A point cloud can be viewed as a vector structure, with each point having its own coordinates (for example, 3D XYZ coordinates, or depth/distance from a given viewpoint) and one or more attributes, also called a component. An example of a component is a color component, which can be expressed in different color spaces, such as RGB (Red, Green and Blue) or YUV (Y is the luminance component and UV is the two chrominance components). A point cloud is a representation of an object from a given viewpoint or range of viewpoints. The point cloud can be obtained through a variety of methods, for example:

from the capture of a real object, filmed by a camera setup, optionally supplemented with an active depth recording device;

from the capture of a virtual/artificial object captured by a virtual camera setup in a modeling tool;

from a mixture of both real and virtual objects.

Volumetric portions of a 3D scene may, for example, be represented using one or more point clouds, such as point cloud 71.

Figure 8 shows a segmentation of an image 40 (also referred to as the first frame 40 below), in accordance with a non-limiting embodiment of the present principles. The first frame 40 corresponds to a flat view of a large field of view (up to 360°) of the 3D scene, i.e. A 2D representation with only texture (color) information and no depth. Wide field of view content (also called omnidirectional content) is content with a field of view that is typically larger than the viewing window of the end user's device, i.e. only a sub-portion of the first frame 40 is displayed at a time on the end user's device. The first frame 40 is divided into a plurality of sub-parts or sub-zones 81 to 88, which may correspond to segments in the HEVC, VP9 or AV1 sense (developed by the Alliance for Open Media), for example. The segments can, for example, be obtained by using HEVC motion-constrained segmentation (MCTS).

According to a non-limiting example, the first frame (after equirectangular projection (EPR) has been performed) is divided into 8 HEVC segments 81 to 88. HEVC segments correspond to non-overlapping rectangular regions containing an integer number of CTUs (Coding Tree Unit, CTU corresponding to the largest block encoding, e.g. size 64x64) and organized in a grid pattern with rows and columns of fixed or variable size (e.g. Core Profile 10 Level 5.1 in HEVC sets the maximum number of rows and columns of segments to 11x10). The content of the first frame may, for example, be encoded at one or more resolutions using Motion Constrained Segment Sets (MCTS), i.e. in the absence of another prediction from other segments. Each MCTS encoded sequence can be stored encapsulated in a separate segment track within, for example, an ISOBMFF (ISO Basic Multimedia File Format) container. The number of segments is not limited to 8 and can be any integer, such as greater than or equal to 2 and less than 110 (11x10) if based on the Level 5.1 HEVC Core Profile 10.

A different 3D portion or 3D section of the 3D scene is associated with each segment T1 81, T2 82, T3 83, T4 84, T5 85, T6 86, T7 87 and T8 88 of the first frame 40 (as explained in more detail in relation to Figure 10), wherein the video content of a specific segment of the first frame corresponds to the result of projecting a 3D portion of a 3D scene associated with that specific segment.

Figure 9 shows a 2D parameterization of a 3D portion of a 3D scene associated with one first frame segment T6 86, in accordance with a non-limiting embodiment of the present principle. The exact same process can be applied to each 3D portion of the 3D scene associated with segments T1, T2, T3, T4, T5, T7, and T8 of the first frame 40.

The 3D scene points that are contained in the 3D portion of the 3D scene associated with the segment T6 86 may be grouped to form one or more groups of points, wherein the points belonging to the same group have, for example, depth information that contained within a specific depth range (i.e. points in the same group are consistent in depth). The range of viewpoints that are contained in the view space 11 (eg, cube), for example, is obtained by sampling into multiple viewpoints (eg, 5, 10, or 20 viewpoints). According to another example, a plurality of viewpoints is obtained from view space 11 by defining a tetrahedron centered on a first viewpoint 20, with each of the four vertices of the tetrahedron defining a viewpoint from a plurality of viewpoints in addition to the first viewpoint 20. The points of the 3D portion of the 3D scene associated with the segment T6 86 are grouped taking into account the 3D portion from each viewpoint of the set. To avoid that the same point belongs to multiple groups, grouping may be first performed on the first viewpoint, and points that have been grouped according to the first viewpoint are not considered for the next grouping according to another viewpoint. Examples of groups of points are illustrated in Figure 9, for example groups 91 to 95, with each group containing, for example, points with a consistent depth, for example group 94 containing points with a depth that is between 2m and 2.5m (distance from the first viewpoint ), group 95 contains points with depths that are between 3.2m and 3.9m, group 93 contains points with depths that are between 3.9m and 10m.

For each group of points, a fragment is formed, and the fragment corresponds to the 2D parameterization of the group of points. For example, fragment 904 is formed from a group of 94 points. One or more fragments of each group of points of the 3D part are formed to represent each group of points in two dimensions, i.e. according to 2D parameterization. One or more 2D parameterizations can be obtained for each group of 91, 92, 93, 94 and 95 points. The 2D parameterization may vary from group of points to another. According to an embodiment, all 2D parameterizations associated with all groups of points are of the same type, for example, linear perspective projection or orthogonal projection. According to an option, different 2D parameterizations can be used for the same group of points.

A 2D parameterization associated with one given group of points, for example group 94, corresponds to a 2-dimensional view of a given 3D group of points of a point cloud, which allows the selection of a given 3D group of points, i.e. A 2D representation of the content (i.e., point(s)) of a given 3D group of points, containing a number of samples (which may correspond to the pixels of the fragment image), the number of which depends on the sampling step applied. The 2D parameterization can be represented by a set of parameters and can be obtained through a variety of methods, for example by applying any of the following methods:

- equirectangular projection (EPR) of points from a group of points of the point cloud onto a sphere associated with the viewpoint, and the parameters representing the ERP projection contain the location of the sphere, its orientation and the spatial sampling step;

- linear perspective projection of points from a group of points of the point cloud onto a plane associated with the viewpoint, wherein the parameters representing the linear perspective projection include the location of the virtual camera, focal length, spatial sampling step and field of view in 2 dimensions;

- orthogonal projection of points from a group of points of the point cloud onto the surface, and the parameters representing the orthogonal projection contain the geometry (shape, size and orientation) of the projecting surface and the spatial sampling step;

- LLE (Local Linear Embedding), which corresponds to the mathematical operation of dimensionality reduction, is used here to convert/convert from 3D to 2D, with the parameters representing the LLE containing the conversion coefficients.

Each piece is predominantly rectangular in shape to facilitate the packaging process, which will be explained below. Each fragment can contain geometric information obtained by projecting/2D parameterizing points from a related group of points. Geometric information may correspond to depth information or information about the position of mesh element vertices. The fragment may additionally contain texture information associated with the points of the associated group of points, except when the fragment is derived from points that are visible from the first viewpoint. For points that are visible from the first viewpoint, texture information may be obtained from the first frame 40. According to an embodiment, geometric fragments containing only geometric information are generated, and further texture fragments containing only texture information are generated, if necessary.

Display information can be generated that associates each 2D parameterization with its associated fragment. Display information may be generated to provide a link between the 2D parameterization and the associated geometric fragment and texture fragment in the frames into which the fragments are placed. The display information may, for example, be in the form:

{2D parameterization parameters; Geometric fragment ID; Texture fragment ID}

wherein the ID of the geometric fragment may be an integer value or a pair of values containing the column index U and row index V to which the geometric fragment belongs in the frame in which it is placed; The texture fragment ID may be an integer value or a pair of values containing the column index U' and the row index V' to which the texture fragment belongs in the matrix of frame(s) fragments (or frame(s) segments) to which the fragments belong.

Exactly the same mapping information is generated for each 2D parameterization of both the associated geometric fragment and the texture fragment. This mapping information allows the corresponding parts of the 3D scene to be reconstructed by linking the 2D parameterization with the corresponding geometric and texture fragments. If the 2D parameterization is a projection, then the corresponding portion of the 3D scene can be reconstructed by de-projecting (backprojecting) the geometric information contained in the associated geometry fragment and the texture information in the associated texture fragment. Then the display information corresponds to the list of display information:

{2D parameterization parameters; Geometric and texture fragment ID} _i ,

For i=1 to n, where n is the number of 2D parameterizations.

Figure 10 shows a two-dimensional view of an angular viewing sector corresponding to a 3D portion of a 3D scene associated with segment T6 80 of the first frame 40, in accordance with a non-limiting embodiment of the present principle.

As explained with respect to Figure 8, each first segment T1 81 to T8 88 is associated with a 3D portion of the 3D scene. When the first frame 40 is obtained by equirectangular or cube map projection mapping, the 3D portion associated with a given first segment T (T∈[1,8] in the non-limiting example of Figure 8) corresponds to the angular range 101 of the view from the first viewpoint, which may be defined using . The number of 3D tiles can be obtained from the 3D portion to form tiles, such as tile 904, by 2D parameterizing the 3D tiles. A 3D fragment corresponds to a subset of a 3D part, i.e. group of points obtained as explained with respect to Figure 9. The 3D fragment 102 may, for example, be defined by a viewing angle and a range of distances from a particular view point (either the first view point 20 or any of the view points obtained from the space discretization 11 vision). The 3D fragment 102 is, for example, defined using , wherein correspond to the angular range corresponding to the viewing angle, and corresponds to a range of distances. One or more constraints can be applied when defining 3D fragments (or groups of points). For example, spherical range associated with the 3D fragment must be contained or must match the viewing angle range the first segment T6 86 defined in the first frame to avoid the same 3D fragment covering two different first segments of the first frame (or equivalently two different viewing angle ranges associated with two different segments). A fragment 904 containing depth information and/or texture information is obtained by 2D parameterizing the points of a 3D fragment 102, as explained in relation to Figure 9. As explained in relation to Figure 9, a geometric fragment 904 associated with a 3D fragment 102, and further a geometric texture fragment may be generated associated with the 3D fragment 102. When the 3D fragment 102 is viewed/determined from the first viewpoint 20, the fragment 904 contains only geometric information, and the texture information can be extracted from the first frame 40.

Figure 11 shows examples of groups of fragments obtained by 2D parameterization of 3D fragments of 3D portions of the scene associated with the first segments of the first frame, in accordance with a non-limiting embodiment of the present principle.

Figure 11 illustrates some of the groups of fragments obtained by 2D parameterization of groups of points that are contained in parts of the scene associated with some of the first segments of the first frame 40. Figure 11 shows, for example, a group of fragments S1 associated with the first segment T1 81 of the first frame 40 , a group of fragments S3 associated with the first segment T3 83 of the first frame 40, a group of fragments S5 associated with the first segment T5 85 of the first frame 40, a group of fragments S6 associated with the first segment T6 86 of the first frame 40, and a group of fragments S8 associated with the first segment T8 88 of the first frame 40. Even if not illustrated, groups of fragments S2, S4 and S7 can be obtained from the content of the 3D portions of the scene associated with the corresponding first segments T2 82, T4 84 and T7 87.

A group of fragments may contain fragments containing depth information and texture information. In an embodiment, the group of fragments may contain fragments containing only depth information or only texture information. When a tile group contains tiles that contain only depth information (respectively, texture information), a corresponding tile group can be formed containing tiles that contain only texture information (respectively, depth information). According to the latter embodiment, the number of depth fragments that are contained in the corresponding group (for example, the group S1 associated with the first segment T1) may be different from (for example, be greater than) the number of texture fragments that are contained in the corresponding group (for example, in group S1', not illustrated, associated with the same first T1 segment as group S1).

The number of fragments that are contained in a group may vary from group to other group. For example, the number of fragments that is contained in group S1 is greater than the number of fragments that is contained in groups S5 and S8, but less than the number of fragments that is contained in groups S3 and S6. The number of fragments that are contained in a group (eg, group S1) may depend on the content of the 3D portion of the scene associated with the first segment associated with said group (eg, first segment T1). The number of fragments may, for example, depend on the number of depth-matched groups of points that are contained in the 3D portion of the scene corresponding to the first segment (eg, the first segment T1) associated with the group of fragments (eg, S1).

According to an embodiment, the number of fragments is the same in each of the fragment groups. The number of fragments may be a user-defined value or a default value that is stored in a device configured to generate fragments and groups of fragments.

Figure 12 shows a non-limiting example of encoding, transmitting and decoding data representing a 3D scene in a format that is simultaneously compatible with 3DoF and 3DoF+ rendering.

A 3D scene 120 (or a sequence of 3D scenes) is encoded into a stream 122 by an encoder 121. The stream 122 includes a first syntax element carrying data representing a 3D scene for 3DoF rendering (data from the first image 40 of Figure 8), and at least a second syntax element carrying data representing a 3D scene for 3DoF rendering (data from the first image 40 of Figure 8). carrying data representing a 3D scene for 3DoF+ rendering (eg, data of one or more second/third images 130 of Figure 13 and/or data of one or more fourth images 151, 152 of Figure 15).

Encoder 121 is, for example, compatible with an encoder such as:

HEVC (its specification is on the ITU website, Recommendation T, Issue H, h265, http://www.itu.int/rec/T-REC-H.265-201612-I/en);

3D-HEVC (an extension of HEVC whose specification is on the ITU website, Recommendation T, Issue H, h265, http://www.itu.int/rec/T-REC-H.265-201612-I/en appendix G and I);

VP9, developed by Google; or

AV1 (AOMedia Video 1) developed by the Alliance for Open Media.

Decoder 123 receives stream 122 from the source. For example, the source belongs to the set containing:

- local memory, for example, video memory or RAM (or Random Access Memory), flash memory, ROM (or Read Only Memory), hard drive;

- a storage interface, for example, an interface with a mass storage device, RAM, flash memory, ROM, optical disk or magnetic media;

- a communication interface, for example a wired interface (such as a bus interface, a WAN interface, a LAN interface) or a wireless interface (such as an IEEE 802.11 interface or a Bluetooth® interface); And

- a user interface, such as a Graphical User Interface, allowing the user to enter data.

Decoder 123 decodes the first syntax element of stream 122 for 3DoF rendering 124. For 3DoF+ rendering 125, the decoder decodes both the first syntax element and the second syntax element of stream 122.

Decoder 123 is compatible with encoder 121, for example, compatible with a decoder such as:

HEVC;

3D-HEVC (HEVC extension);

VP9; or

AV1.

Figure 13 shows an example of the arrangement of tiles that are contained in the tile groups of Figure 11 in second segments 130 ₁ to 130 ₃₂ of second frame 130, in accordance with a non-limiting embodiment of the present principles.

The second frame 130 is divided into a plurality of second segments 130 ₁ to 130 ₃₂ , for example, 32 second segments in the non-limiting example of Figure 13. The number of second segments is not limited to 32 and can be any number, provided that the number of second segments is greater than the number of first segments of the first image 40. As with the first segments, the size (eg, height and width) of every second segment from 130 ₁ to 130 ₃₂ is such that the second segment can contain an integer number (greater than or equal to 1) CTU (Coding Tree Unit, CTU corresponding to the largest encoding block, for example, of size 64x64), with the second segments 130 ₁ to 130 ₃₂ organized in a grid pattern, with rows and columns of fixed or variable size (for example, the Main Profile 10 level 5.1 HEVC sets the maximum number of rows and columns of segments in 11x10).

Second segments 130 ₁ to 130 ₃₂ are associated with first segments T1 through T8, with one or more second segments associated with each first segment. For example, 2 second segments (eg, second segments 130 ₁ to 130 ₂ ) may be assigned to the first segment T1, 4 second segments (eg, second segments 130 ₃ to 130 ₆ ) may be assigned to the first segment T2, 5 second segments ( e.g. second segments 130 ₇ to 130 ₁₁ ) can be assigned to the first segment T3, 5 second segments (e.g. second segments 130 ₁₂ to 130 ₁₆ ) can be assigned to the first segment T4, 3 second segments (e.g. second segments 130 ₁₇ to 130 ₁₉ ) can be assigned to the first segment T5, 5 second segments (for example, second segments 130 ₂₀ to 130 ₂₄ ) can be assigned to the first segment T6, 5 second segments (for example, second segments 130 ₂₅ to 130 ₂₉ ) can be assigned to the first segment T7 and 3 second segments (for example, second segments 130 ₃₀ to 130 ₃₂ ) can be assigned to the first segment T8. The second segments are used to contain and transport fragments obtained by 2D parameterization of the 3D portions of the scene associated with the first segments T1 to T8. The fragments that are contained in the fragment group S ₁ obtained by 2D parameterization of the 3D part associated with the first T1 segment are placed in the second segments 130 ₁ to 130 ₂ assigned to the first T1 segment; fragments of the group S ₂ are placed in the second segments from 130 ₃ to 130 ₆ assigned to the first segment T2; fragments of group S ₃ are placed in the second segments from 130 ₇ to 130 ₁₁ assigned to the first segment T3, etc.

The second segments are obtained, for example, by subdividing the first segments. The first segment can, for example, be subdivided into 2, 4 or 8 second segments.

The number of second segments assigned to the first segment may, for example, depend on the number of fragments that are contained in a group of fragments associated with the first segment. This number may be further limited to a maximum value such that each first segment may be assigned to at least one second segment.

The second image 130 may contain slices containing depth and texture information. In an embodiment, the second image 130 may contain fragments containing only depth information, and a third image is generated to contain fragments containing texture information associated with the first segments, excluding the fragments obtained from the first viewpoint. The third image contains a plurality of third segments, the number of third segments being greater than the number of first segments.

Parameters such as frame size and number of segments may be the same for the first and second images. In an embodiment, one or more parameters are different for the first and second images.

Figure 14 shows a non-limiting example of a process for placing tiles of a group of tiles S ₁ through S ₈ in the second segments of the second frame 130 (and/or in the third frame).

The initial budget for the second segments is 32 in accordance with the example of Figure 14.

The input for the process consists of a list of groups S _t of fragments ( And integers, and corresponds to the number of fragments in the group, and = 8 in accordance with the example of Figure 14), and , each group S ₁ to S ₈ corresponds to the first segment T1 to T8 in the first frame 40.

The first operation checks whether the size of each input fragment is no wider or higher than the size of the second segment. For example, given that the size of the second frame 130 is 2048×1024 pixels, and that the 32 second segments are the same size, the size of the second segment is 256×256 pixels. When a fragment does not fit into the second segment's target size, the fragment is split into smaller fragments, each smaller than the second segment's target size.

In the second operation, the input fragments sorted in descending order of visual importance. The visual importance of a fragment may depend on its distance from the viewing point (the farther away, the less important), and/or its angular location relative to the main viewing direction (the far left, right, top or bottom fragments are less important than the central ones), and /or obstructions (fragments that are obscured from the central viewing point are less important). Sorting according to visual importance is performed on all input fragments, i.e. by processing all input fragments of all fragment groups S ₁ to S ₈ at the same time and without considering the fragment groups S ₁ to S ₈ in parallel or one after the other. However, information related to the membership of input fragments in fragment groups is retained for subsequent operations.

packing algorithms can, for example, be executed in parallel, with one packing algorithm for each group of fragments S ₁ to S ₈ . As examples, the Shelf, Guillotine, Maximum Rectangle and Horizon methods, with all their variants, described in the document “A thousand ways to pack the bin - A practical approach to two-dimensional rectangle bin packing” can be used as a packing algorithm. by J. Jylänki.

In the third operation, the input fragments can be processed one by one, starting with the most visually important. The fragment group to which the input fragment belongs is determined (eg, extracted from the metadata associated with the input fragments) and the input fragments are directed to the packaging branch (t=1 to 8) corresponding to the fragment group to which each belongs. One or more second segments 130 ₁ to 130 ₃₂ are assigned to each branch of the package.

When one of the packing branches no longer has room in the current second segment for the current incoming fragment, a new empty second segment is created and packaging resumes until the next round of packaging. For subsequent fragments in that packaging branch, all second segments created during the previous packaging round remain as potential target second segments. Of course, the new incoming input fragment may be smaller than the previous one and may be placed in the previously created second segment.

The process stops when all input fragments have been processed or the budget for the second segments has been used and all second segments are full. If the second frame 130 is not of sufficient size and all input fragments cannot be packed, the remaining unpacked fragments are not packed and are discarded. Since the input tiles have been processed according to their visual importance, the discarded input tiles correspond to the least visually important input tiles, limiting problems when rendering a 3D scene from the tiles of the second frame 130.

Figure 15 shows non-limiting examples of two fourth frames 151, 152 derived from second segments of second frame 130 (and/or from third segments of third frame).

Each of the fourth frames 151, 152 contains fragments that are contained in a portion of the second segments of the second frame 130. The second segments in the fourth frame correspond to a selection of some of the first segments in the first frame, and there is a mapping between the first segments and the second segments. On the rendering side, only a portion of the first frame may be needed because the viewport of the end user device that is used to render the content may be smaller than the content viewport of the first frame 40. According to the view direction, which may be defined at the end user device level, may Only a sub-portion of frame 40 will be shown, and only the first segments corresponding to that sub-portion must be decoded and rendered.

The fourth frame 151 contains, for example, 20 fourth segments arranged in 5 columns and 4 rows (with, for example, a resolution of 1280×1024 pixels). The fourth frame 151 contains data corresponding to a viewport that covers the 3D portions of the 3D scene associated with the first segments T1, T2, T5, and T6, and contains fragments obtained by 3D parameterizing the 3D portions of the 3D scene associated with these first segments T1, T2, T5 and T6, i.e. fragments contained in the second segments 130 ₁ , 130 ₂ , 103 ₃ , 103 ₄ , 130 ₅ , 130 ₆ , 130 ₁₇ , 130 ₁₈ , 130 ₁₉ , 130 ₂₀ , 130 ₂₁ , 130 ₂₂ , 130 ₂₃ and 130 ₂₄ . A segment of a given fourth frame may correspond to a second segment and contain the content (fragments and data contained in the fragments) of the corresponding second segment, the second segment being identified by its index, i.e. 1 to 32. The remaining segments of this fourth frame, i.e. segments filled with diagonal stripes are empty and contain no data or contain dummy data.

The fourth segments can be obtained from the second segments or directly from the 2D parameterization process, without generating the second segments of the second frame of Figure 13.

Each of the first, second, third and fourth frames has a time-fixed size and a time-fixed number of segments. The size and/or number of segments may be either the same for the first, second, third and fourth frames, or may be different between the first, second, third and/or fourth frames.

The fourth frame 152 contains 20 fourth segments arranged in 5 columns and 4 rows as for the fourth frame 151. The fourth frame 152 contains data corresponding to a viewport that covers the 3D portions of the 3D scene associated with the first segments T2, T3, T6 and T7 , and contains fragments obtained by 3D parameterization of 3D parts of the 3D scene, connected in these first segments T2, T3, T6 and T7, i.e. fragments contained in the second segments 103 ₃ , 103 ₄ , 130 ₅ , 130 ₆ , 103 ₇ , 103 ₈ , 130 ₉ , 130 ₁₀ , 130 ₁₁ , 130 ₂₀ , 130 ₂₁ , 130 ₂₂ , 130 ₂₃ , 130 ₂₄ , 130 ₂₅ , 130 ₂₆ , 130 ₂₇ , 130 ₂₈ and 130 ₂₉ . The second segments allocated to the fourth frame are identified by their index 3, 4, 5, 6, 7, 8, 9, 10, 11,20, 21, 22, 23, 24, 25, 26, 27, 28 and 29. The remaining segment of the fourth frame, i.e. a segment filled with diagonal stripes is empty and contains no data or contains dummy data.

Generally speaking, there is combinations of first segments in the first frame 40 corresponding to an omnidirectional planar image/video scene, each combination of the first segments correspond to different orientations of the viewing window ( is an integer, for example equal to 4). Only requested the first segments from the complete set of first segments ( wherein W is 4 and H is 2 in the example of Figure 8) and the extractor track aggregates them into a segmented HEVC (or VP9 or AV1) compatible bitstream before decoding.

A similar aggregation process is performed for the subset second segments (from the full set of second segments) corresponding to one of combinations of the first segments. The second segments are all the same size, and aggregation into segments into a rectangular frame (for example, fourth frame 151, 152) is always possible from segments . There are 3 possibilities:

(i) the full segmentation grid on the decoder side (ie the fourth frame) is used for aggregation;

(ii) only part of the decoder-side segmentation grid is used, and dummy values fill in the unused segments (shown with diagonal bars);

(iii) the full decoder-side segmentation grid is used (i.e. the fourth frame), but some second segments are discarded; since each fragment is assigned a visual importance value (in accordance with the packing algorithm described in relation to Figure 14), the visual importance of segments is defined as the maximum visual importance of the fragments packed within it; Segments with the least visual importance are reset with priority.

Figure 16 shows a non-limiting example of an embodiment of a stream syntax carrying information and data representing a 3D scene when the data is transmitted through a packet-based transfer protocol. Figure 16 shows an exemplary volumetric video stream structure 16. The structure consists of a container that organizes flow into independent syntax elements. The structure may contain a header portion 161, which is a set of data common to each of the flow syntax elements. For example, the header portion contains metadata about the syntax elements, describing the nature and role of each one. The header portion may also contain the coordinates of the viewpoint used to encode the first frame for 3DoF rendering, and information about the size and resolution of the frames. The structure contains a payload containing a first syntax element 162 and at least one second syntax element 163. The first syntax element 162 contains data representing a first frame prepared for 3DoF rendering, corresponding to, for example, a first video track associated with the texture data of the first frame encoded in the first syntax element.

One or more second syntax elements 163 comprise geometric and texture information associated with, for example, one or more second video tracks. One or more second syntax elements 163 contain, for example, data representing one or more second frames and/or third frames described with respect to Figure 13.

In an embodiment, one or more second syntax elements 163 comprise data representing one or more fourth frames described with respect to FIG. 15.

According to a further embodiment, the stream further comprises at least instructions for retrieving at least a portion of said first data and second data in one or more third syntax elements.

For purposes of illustration, in the context of the ISOBMFF file format standard, texture chunks, geometry chunks, and metadata will typically be accessed in ISOBMFF tracks in a block of type moov, with the texture data and geometry data themselves embedded in a media data block of type mdat.

Figure 17 shows a non-limiting example of a process to encode a 3D scene. The process of Figure 17 corresponds to segment-based encoding (eg, HEVC segment-based or AV1 segment-based) and volumetric video file encapsulation.

The sequence 171 of the first frames 40 can be encoded to obtain:

A plurality of segment tracks 1710 that carry backwards compatible omnidirectional video, each segment track containing data that is contained in one distinct first segment of the first segments T1 to T8 (for example, one segment track to carry the content of the first segment T1, one segment track to carry the content of the first segment T2, one segment track to carry the content of the first segment T3, ... and one segment track to carry the content of the first segment T8 ); wherein the number of segment tracks may be between, for example, 2 and N, with N corresponding to the number of first segments in the first frame 40;

Metadata 1712 corresponding to the projection that is used to obtain the first frame, stitching and packing into regions to render omnidirectional video;

One or more extractor tracks 1711, up to a maximum number of extractor tracks corresponding to the number C of possible viewport orientations; the extractor contains instructions for reconstructing a bit stream that can be decoded by a single decoder, the reconstructed bit stream being syntactically correct for decoding by the decoder.

The output signal from the encoding process may contain, for example, 8 segment tracks 1710 (8 corresponds to the number of first segments in the example of Figure 8), 4 extractor tracks 1711 (4 corresponds to the number of possible combinations of first segments in accordance with the orientation of the viewport on the end device user, namely a first combination containing the first segments T1, T2, T5, T6; a second combination containing the first segments T2, T3, T6, T7; a third combination containing the first segments T3, T4, T7, T8; and a fourth combination, containing the first segments T4, T1, T8, T5), and metadata 1712.

In an embodiment, the signal contains only a portion of the 8 segment tracks 1710, only one extractor track 1711, and metadata 1712. For example, the decoder may transmit a request that requests a given viewport orientation corresponding to one combination of the first segments, for example, a second combination containing the first segments T2, T3, T6, T7. In response to a request, the encoder can encode only the extractor track corresponding to the request with segment tracks 2, 3, 6, and 7 containing the associated first segments T2, T3, T6, and T7, and transmit the requested extractor, the 4 segment tracks needed to reconstruct the bitstream to render the requested viewport, with associated metadata 1712. This option reduces encoding costs and reduces the bandwidth required to transport the data. Such viewport-dependent content delivery can be implemented based on the DASH streaming mechanism (ISO/IEC 23009-1, Dynamic adaptive streaming over HTTP).

The encoding of the sequence 171 provides visualization of omnidirectional video after decoding, and the tracks obtained from the encoding of this sequence 171 are compatible, for example, with the OMAF standard (ISO/IEC 23090-2 Information Technology - Coded representation of immersive media (MPEG-I) - Part 2: Omnidirectional media format"

The sequence 172 of second frames 130 may be encoded to obtain:

A plurality of segment tracks 1720 that carry second segments and associated fragments, each segment track containing data that is contained in one distinct second segment of second segments from 130 ₁ to 130 ₃₂ ;

Metadata 1722 containing, for example, fragment deprojection parameters, information representing the location within the second segments of the second frame (respectively, the location within the third segments of the third frame, if applicable); And

One or more extractor tracks 1721, up to a maximum number of extractor tracks corresponding to the number C of possible viewport orientations; the extractor contains one or more instructions for reconstructing a bitstream that can be decoded by a single decoder, wherein the reconstructed bitstream is syntactically correct for decoding by the decoder; The extractor contains an instruction to extract data from another track that is associated with the track in which the extractor is located, as described in the document "Information Technology - Coding of audiovisual objects - Part 15: carriage of NAL unit structured video in the ISO Base Media File Format." AMENDMENT 1: Carriage of Layered HEVC", ISO/IEC 14496-15:2014/PDAM 1, dated 11 July 2014.

The output signal from the encoding process may contain, for example, 32 segment tracks 1720 (32 corresponds to the number of second segments in the example of Figure 13), 4 extractor tracks 1721 (4 corresponds to the number of possible combinations of the first segments according to the orientation of the end user device's viewing window, namely, a first combination containing second segments allocated to the first segments T1, T2, T5, T6; a second combination containing second segments allocated to the first segments T2, T3, T6, T7; a third combination containing second segments allocated to the first segments T3, T4, T7, T8; and a fourth combination containing the second segments allocated to the first segments T4, T1, T8, T5), and metadata 1722.

In an embodiment, the signal contains only a portion of the 32 segment tracks 1720, only one extractor track 1721, and metadata 1722. For example, the decoder may transmit a request that requests a given viewport orientation corresponding to one combination of the first segments, for example, a second combination containing the second segments allocated for the first segments T2, T3, T6, T7. In response to the request, the encoder may generate only the extractor track corresponding to the request, and encode the segment tracks containing the second segments allocated to the first segments T2, T3, T6 and T7 containing data associated with the second segments allocated to the first segments T2, T3, T6 and T7, and transmit the requested extractor, wherein the associated segment tracks are needed to recreate the bitstream for rendering the requested viewport with associated metadata 1722. This option reduces encoding costs and reduces the bandwidth required to transport the data. Such viewport-dependent content delivery can be implemented based on the DASH streaming mechanism (document “ISO/IEC 23009-1, Dynamic adaptive streaming over HTTP”).

Encoding sequence 172 with sequence 171 provides volumetric video rendering.

If sequence 172 corresponds to a sequence of second images containing only geometric information, then an additional sequence of third frames containing texture information may be encoded in exactly the same manner as sequence 172.

Figure 18 shows a non-limiting example of a process to decode a bitstream obtained using the encoding process of Figure 17.

The decoding process contains two main processes, a first process to decode data representing omnidirectional content (texture information only) for 3DoF rendering, and a second process to decode data representing volumetric content rendering for 3DoF+ rendering.

The first process parses the segment tracks 1810 associated with the requested viewport orientation (tracks containing data associated with the first segments T2, T3, T6, and T7 in accordance with the non-limiting example of Figure 17) by a parser 181 in accordance with the instructions adopted in the extractor track. Parser 181 allows you to recreate a bitstream 1811 that is compatible with decoder 183 (eg, a HEVC-compatible bitstream if decoder 183 is HEVC-compatible, or an AV1-compatible bitstream if decoder 183 is AV1-compatible). The data that is contained in the bitstream 1811 is decoded by the decoder 183 and the decoded data is transmitted to the renderer 185, which is configured to render omnidirectional content using the metadata 1712 that is contained in the bitstream obtained through the encoding process of Figure 17.

The second process parses the segment tracks 1820 associated with the requested viewport orientation (tracks containing fragments contained in the second segments (or third segments for texture data) associated with the first segments T2, T3, T6, and T7, according to with the non-limiting example of Figure 17) through the parser 182 in accordance with the instructions received in the extractor track. Parser 182 allows you to recreate a bitstream 1821 that is compatible with decoder 184 (eg, a HEVC-compatible bitstream if decoder 184 is HEVC-compatible, or an AV1-compatible bitstream if decoder 184 is AV1-compatible). The reconstructed bit stream contains, for example, the data of one or more fourth frames described with respect to figure 15. The data that is contained in the bit stream 1821 is decoded by the decoder 184 and the decoded data is transmitted to the renderer 186, which is configured to render a volumetric portion of the volumetric content with using metadata 1722 that is contained in the bitstream obtained by the encoding process of Figure 17. The second process may be performed for segment tracks containing depth information and for segment tracks containing texture information.

Figure 19 shows an exemplary architecture of a device 19 that may be configured to implement the method described with respect to Figures 17, 18, 20 and/or 21. The device 19 may be implemented as an encoder 121 or a decoder 123 of Figure 12.

The device 19 contains the following elements, which are connected together by an address and data bus 191:

- microprocessor 192 (or CPU), which is, for example, a DSP (or Digital Signal Processor);

- ROM 193 (or Read Only Memory);

- RAM 194 (or Random Access Memory);

- storage interface 195;

- interface 196 I/O for receiving data for transmission from the application; And

- power source, for example a battery.

According to the example, the power supply is external to the device. In each memory mentioned, the word "register" used in the specification may correspond to an area of small capacity (a few bits) or a very large area (for example, an entire program or a large amount of received or decoded data). ROM 193 contains at least a program and parameters. ROM 193 can store algorithms and instructions to perform techniques in accordance with these principles. When turned on, the CPU 192 downloads the program into RAM and executes the corresponding instructions.

RAM 194 contains, in a register, a program executed by the CPU 192 and unloaded after turning on the device 19, input data in the register, intermediate data in different states of the method in the register, and other variables that are used to execute the method in the register.

The implementations described herein may be implemented in, for example, a method or process, device, computer program product, data stream, or signal. Even when discussed only in the context of a single form of implementation (for example, when discussed only as a method or device), the implementation of the features discussed may also be implemented in other forms (for example, in a program). The device may be implemented in, for example, appropriate hardware, software or firmware. The methods may be implemented in, for example, a device such as, for example, a processor, which refers to processing devices in general, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communications devices, such as, for example, computers, cellular phones, portable/personal digital assistants (“PDAs”), and other devices that provide communications for the transfer of information between end users.

According to the example encoding or encoder 121 of Figure 12, a three-dimensional scene is obtained from the source. For example, the source belongs to the set containing:

- local memory (193 or 194), for example, video memory or RAM (or Random Access Memory), flash memory, ROM (or Read Only Memory), hard drive;

- a storage interface (195), for example, an interface with a mass storage device, RAM, flash memory, ROM, optical disk or magnetic media;

- a communication interface (196), for example a wired interface (such as a bus interface, a WAN interface, a LAN interface) or a wireless interface (such as an IEEE 802.11 interface or a Bluetoth® interface); And

- a user interface, such as a Graphical User Interface, allowing the user to enter data.

According to the examples of decoding or decoder(s) 123 of Figure 12, the stream is sent to the recipient; specifically, the recipient belongs to the set containing:

- local memory (193 or 194), for example, video memory or RAM, flash memory, hard drive;

- a storage interface (195), for example, an interface with a mass storage device, RAM, flash memory, ROM, optical disk or magnetic media;

- a communication interface (196), for example, a wired interface (for example, a bus interface (for example, USB (or Universal Serial Bus)), a WAN interface, a LAN interface, an HDMI (High Definition Multimedia Interface) interface) or a wireless interface (such such as IEEE 802.11 interface or WiFi® and Bluettoth® interface).

According to encoding or encoder examples, a bitstream containing data representing a volumetric scene is sent to a recipient. As an example, a bitstream is stored in local or remote memory, such as video memory or RAM, on a hard disk. In an embodiment, the bitstream is sent to a storage interface, such as an interface with a mass storage device, flash memory, ROM, optical disk, or magnetic media, and/or transmitted through a communications interface, such as an interface with a point-to-point link, a communications bus , point-to-multipoint link or broadcast network.

According to the decoding examples, either the decoder or the renderer 123 of Figure 12, a bit stream is obtained from the source. As an example, a bitstream is read from local memory, such as video memory, RAM, ROM, flash memory, or hard disk. In an embodiment, the bitstream is received from a storage interface, such as an interface with a mass storage device, RAM, ROM, flash memory, optical disk, or magnetic media, and/or is received from a communication interface, such as an interface with a point-to-point link , bus, point-to-multipoint link, or broadcast network.

According to the examples, the device 19 is configured to implement the method described in relation to figures 17, 18, 20 and/or 21, and belongs to a set containing:

- mobile device;

- communication device;

- gaming device;

- tablet (or tablet computer);

- laptop;

- camera for still images;

- video camera;

- coding chip;

- server (for example, broadcast server, video-on-demand server or web server).

Figure 20 illustrates a method for encoding data representing a 3D scene, eg, 3D scene 10, in accordance with a non-limiting embodiment of the present principles. The method may, for example, be implemented in the encoder 121 and/or in the device 19. Various parameters of the device 19 can be updated. The 3D scene may, for example, be obtained from a source, one or more viewpoints in the 3D scene space may be defined, and parameters associated with the projection display(s) may be initialized.

In a first operation 201, first data representing a 3D scene texture is encoded or formatted into one or more first tracks of a container or file in accordance with a specific format, for example, in accordance with the HEVC/H265 document: “ITU-T H.265 TELECOMMUNICATION STANDARDIZATION SECTOR OF ITU (10/2014), SERIES H: AUDIOVISUAL AND MULTIMEDIA SYSTEMS, Infrastructure of audiovisual services - Coding of moving video, High efficiency video coding, Recommendation ITU-T H.265" or in accordance with AV1. The container corresponds, for example, to an ISOBMFF file (ISO Basic Multimedia File Format, document ISO/IEC 14496-12-MPEG-4 Part 12). The first data refers to the parts (eg, points or mesh elements) of the 3D scene that are visible according to one first viewpoint. The first data includes, for example, metadata and signaling information indicative of a first bitstream syntax element that contains texture information encoded in the pixels of the first frame, the first frame being divided into a plurality of segments. A 3D portion of the 3D scene 10 is associated with each first segment of the first frame 40. Each first track, for example, is associated with one distinct first segment and contains the first data of the associated first segment. The first data, once decoded or interpreted, provides a 3DoF representation of the 3D scene according to the first viewpoint, i.e. parallax-free view.

In a second step 202, second data representing depth associated with points in the 3D scene is encoded into one or more second tracks of the container or file. The second data is located in second segments of a second frame, which may correspond to the fourth frame 151, 152 of Figure 15. A different set of second segments is associated with each first segment, and the number of second segments that are contained in the second frame is greater than the number of first segments of the first frame. Portions of the second data (the second data corresponding to, for example, geometric information (depth)) are placed in the second segments and encoded into a corresponding second track, for example, in accordance with the HEVC or AV1 format. Each fragment corresponds to a 2D parameterization of a part (i.e., a point group) of a 3D scene. The fragments obtained by 2D parameterization of the 3D part of the 3D scene associated with the given first segments are placed in the second segments allocated for the given first segment. The second data, once decoded or interpreted, allows a 3DoF+ representation of the 3D scene to be obtained according to multiple viewpoints, including the first viewpoint, i.e. parallax view.

In an embodiment, the fragments contain texture information in addition to the geometric information, except for the fragments derived from the first viewpoint. In other words, the second data contains geometry data and texture data.

According to a further embodiment, the fragments contain only geometric information. In this embodiment, one or more additional second frames may be obtained, each additional second frame being divided into a plurality of second segments, the number of second segments being greater than the total number of first segments. One or more second segments are associated with each first segment, and pieces of texture data are placed in the second segments and encoded into additional tracks called fourth tracks. The fragments obtained by 2D parameterization of the 3D part of the 3D scene associated with the given first segments are placed in the second segments allocated to the given first segment.

In a third operation 204, one or more instructions are encoded into one or more third tracks of the container or file. The instructions are configured to extract first and second data from the first and second tracks and place the extracted first and second data into a bit stream formatted for decoding by a single decoder. The third track corresponds, for example, to the extractor track according to HEVC. The second data is encoded into the second frame(s). The number of third tracks, for example, is equal to the number of C values of possible viewport orientations described with respect to Figure 17.

In accordance with a further optional embodiment, the data and instructions of the first, second and third tracks form a bitstream that is transmitted to a decoder or renderer including the decoder.

Figure 21 illustrates a method for decoding data representing a 3D scene, eg, 3D scene 10, in accordance with a non-limiting embodiment of the present principles. The method may, for example, be implemented in the decoder 123 and/or in the device 19.

In a first operation 211, one or more instructions are decoded from one or more third tracks of the container or file. The instructions are configured to extract first and second data from the first and second tracks and place the extracted first and second data into a bit stream formatted for decoding by a single decoder. The third track corresponds, for example, to the extractor track according to HEVC.

At a second step 212, first data representing the texture of the portion of the 3D scene that is visible in accordance with the first viewpoint is decoded or interpreted from the first segments that are contained in one or more first tracks, as prescribed by the instructions received at step 211. First data allow you to get a 3DoF representation of the 3D scene in accordance with the first viewpoint, i.e. parallax-free view.

In a third operation 213, second data representing the geometry of the 3D scene that is visible in accordance with the set of viewpoints containing the first viewpoint is decoded or interpreted from second segments that are contained in one or more second tracks, as prescribed by the instructions received at the operation 211. The second data is contained in fragments obtained by 2D parameterization of a group of points in a 3D scene. In an embodiment, third data representing a 3D scene texture that is visible according to said set of viewpoints other than the first viewpoint is decoded or interpreted from one or more third segments of additional tracks. The second data with the third data provides a 3DoF+ representation of the 3D scene in accordance with said viewpoints, i.e. parallax view.

In an embodiment, the decoded first and second data are placed in a bit stream as prescribed by the instructions received at operation 211 to form a bit stream to be decoded by the decoder. In another embodiment, the resulting bitstream is decoded by a decoder for rendering by a renderer. The resulting rendered content may be presented on an end user's viewing device, such as an HMD, according to a viewpoint orientation consistent with the instructions.

Naturally, the present invention is not limited to the previously described embodiments.

In particular, the present invention is not limited to a method and apparatus for encoding/decoding data representing a 3D scene, but also extends to a method for generating a bit stream containing encoded data and an apparatus implementing this method, and in particular to any devices containing at least one CPU and/or at least one GPU.

The present invention also relates to a method (and apparatus configured) for displaying images resulting from rendering from decoded bitstream data.

The present invention also relates to a method (and apparatus configured) for transmitting and/or receiving a bit stream.

The implementations described herein may be implemented in, for example, a method or process, device, computer program product, data stream, or signal. Even when discussed only in the context of one form of implementation (for example, when discussed only as a method or device), the implementation of the features that have been discussed may also be implemented in other forms (for example, a program). The device may be implemented in, for example, appropriate hardware, software and firmware. The methods may be implemented in, for example, a device such as, for example, a processor, which refers to processing devices in general, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communications devices, such as, for example, smart phones, tablets, computers, mobile phones, portable/personal digital assistants (“PDAs”), and other devices that provide communications for the transfer of information between end users.

Implementations of the various processes and features described herein may be embodied in a variety of different equipment or applications, particularly, for example, equipment or applications associated with data encoding, data decoding, view generation, texture processing and other image processing, and related texture information and/or depth information. Examples of such equipment include an encoder, decoder, post processor, processing output from the decoder, a preprocessor providing input to the encoder, video encoder, video decoder, video codec, web server, set-top box, laptop, personal computer, cell phone, PDA, and others. communication devices. As will be clear, the equipment may be mobile or even installed on a mobile vehicle.

Additionally, the methods may be implemented by instructions executed by a processor, and such instructions (and/or data values created by the implementation) may be stored on a processor-readable storage medium, such as, for example, an integrated circuit, software storage medium, or other storage device , such as, for example, a hard disk, compact diskette (“CD”), optical disk (such as, for example, a DVD, often referred to as a digital versatile disk or digital video disk), random access memory (“RAM”), or read only memory (“ROM”). The instructions may form an application program that is physically embodied on a processor-readable storage medium. The instructions may be, for example, in hardware, firmware, software, or a combination. Instructions can be found, for example, in the operating system, a separate application, or a combination of the two. As a result, a processor may be characterized as, for example, both a device configured to carry out a process and a device that includes a processor-readable storage medium (such as a storage device) with instructions for carrying out the process. In addition, the processor-readable storage medium may store, in addition to or instead of instructions, data values created by the implementation.

As will be apparent to one skilled in the art, implementations may create a variety of signals formatted to carry information that may, for example, be stored or transmitted. The information may include, for example, instructions for performing a method, or data generated by one of the described implementations. For example, the signal may be formatted to carry as data the rules for writing or reading the syntax of the described embodiment, or to carry as data the actual syntax values written by the described embodiment. Such a signal may be formatted, for example, as an electromagnetic wave (eg, using the radio frequency portion of the spectrum) or as a baseband signal. Formatting may include, for example, encoding the data stream and modulating the carrier with the encoded data stream. The information carried by the signal can be, for example, analog or digital information. The signal can be transmitted through a variety of different wired or wireless communication lines, as is known. The signal may be stored on a processor-readable storage medium.

A number of implementations have been described. However, it will be appreciated that various modifications may be made. For example, elements of different implementations can be combined, extended, modified, or removed to create different implementations. Additionally, one skilled in the art will appreciate that the disclosed structures and processes may be substituted for other structures and processes, and the resulting implementations will perform at least substantially the same function(s) in at least substantially the same way. thereby achieving at least substantially the same result(s) as disclosing the implementation. Accordingly, this application contemplates these and other implementations.

Claims (36)

1. A method for encoding data representing a 3D scene (10), the method comprising the steps of: - encoding (201) into at least a first track first data representing the texture of the 3D scene that is visible in accordance with the first view point (20), the first data being located in a plurality of first segments into which the first frame (40) is divided wherein a portion of the 3D scene is associated with each first segment of the plurality of first segments; - encoding into at least a second track second data representing depth associated with points in the 3D scene, wherein the second data is located in a plurality of second segments of the second frame (151, 152), wherein the total number of second segments of the second frame is greater than the total number of the first segments of the first frame, wherein for each first segment of at least a portion of the plurality of first segments: a set of second segments comprising at least one second segment of the plurality of second segments is allocated to each first segment of at least a portion of the plurality of first segments, a set containing at least a fragment (904) that is placed within a set of second segments, each fragment corresponding to a two-dimensional parameterization of a group of 3D points (94) that are contained in a portion of the 3D scene associated with said each first segment, and contains second data representing the depth associated with the 3D points of the group; - encoding into at least a third track at least an instruction for extracting at least a portion of said first data and a second data from at least a portion of said at least a first track and at least a second track. 2. A device (19) configured to encode data representing a 3D scene (10), the device comprising a memory (194) coupled to at least one processor (192) configured to: - encoding into at least a first track first data representing the texture of the 3D scene that is visible in accordance with the first view point (20), the first data being located in a plurality of first segments into which the first frame (40) is divided, wherein a portion of the 3D scene is associated with each first segment of the plurality of first segments; - encode into at least a second track second data representing depth associated with points in the 3D scene, the second data being located in a plurality of second segments of the second frame (151, 152), wherein the total number of second segments of the second frame is greater than the total number of the first segments of the first frame, wherein for each first segment of at least a portion of the plurality of first segments: a set of second segments comprising at least one second segment of the plurality of second segments is allocated to each first segment of at least a portion of the plurality of first segments, the set contains at least a fragment (904) that is located in a set of second segments, each fragment corresponding to a two-dimensional parameterization of a group of 3D points (94) that are contained in a portion of the 3D scene associated with each first segment, and contains second data representing the depth associated with the 3D points of the group; - encode into at least a third track at least an instruction for retrieving at least a portion of said first data and second data from at least a portion of said at least a first track and at least a second track. 3. The method of claim 1, wherein said each fragment further comprises third data representing a texture associated with the 3D points of the group, the third data being encoded into said at least second track. 4. The method of claim 1, wherein a set of third frame segments comprising at least one third segment is allocated to said each first segment, and a set of at least a fragment comprising third data representing a texture associated with the 3D points group is located in said set of third segments, wherein at least the fragment corresponds to a two-dimensional parameterization of a group of 3D points, and the third data is encoded into at least a third track. 5. The method according to one of claims 1, 3 and 4, wherein when the size of the fragment of the set is larger than the size of the second segment in which the fragment is to be placed, then the fragment is divided into a plurality of sub-fragments, each of a size less than the size of the second segment. 6. The method according to one of claims 1, 3 and 4, wherein the fragments of the set are placed in priority order depending on the visual importance of the fragments, the visual importance depending on the second data associated with the fragments. 7. The method according to one of claims 1, 3 and 4, in which the second segments have the same size, which is fixed for a plurality of consecutive second frames. 8. A method for decoding data representing a 3D scene (10), the method comprising the steps of: - decoding from at least the third track, at least an instruction for extracting first data and second data from at least the first track and at least the second track; - decoding said first data from said at least first track, wherein said first data represents the texture of a 3D scene that is visible in accordance with the first view point (20), wherein the first data is located in a plurality of first segments into which the first frame (40) is divided ), wherein a portion of the 3D scene is associated with each first segment of the plurality of first segments; - decoding said second data from said at least second track, wherein said second data represents depth associated with points in the 3D scene, wherein said second data representing depth is contained in at least a fragment that is located in at least , a set of second segments of a second frame, wherein the set of second segments is allocated to said each first segment, wherein at least the fragment corresponds to a two-dimensional parameterization of a group of 3D points that are contained in at least a portion of the 3D scene associated with said each first segment, wherein the second data represents the depth associated with the 3D points of the group. 9. A device (19) configured to decode data representing a 3D scene, the device comprising a memory (194) coupled to at least one processor (192) configured to: - decode from at least the third track, at least an instruction for extracting first data and second data from at least the first track and at least the second track; - decode said first data from said at least first track, wherein said first data represents the texture of the 3D scene that is visible in accordance with the first view point (20), wherein the first data is located in a plurality of first segments into which the first frame is divided ( 40), wherein a portion of the 3D scene is associated with each first segment of the plurality of first segments; - decode said second data from said at least a second track, wherein said second data represents depth associated with points in the 3D scene, wherein said second data representing depth is contained in at least a fragment that is located in at least , a set of second segments of a second frame, wherein the set of second segments is allocated to said each first segment, wherein at least the fragment corresponds to a two-dimensional parameterization of a group of 3D points that are contained in at least a portion of the 3D scene associated with said each first segment, with the second data representing the depth associated with the 3D points of the group. 10. The method of claim 8, wherein third data representing a texture associated with the 3D group points that are contained in said each fragment is further decoded from at least the second track. 11. The method of claim 8, wherein third data representing the texture is further decoded from at least a third track, the third data being allocated to a plurality of third segments of the third frame, the set of third segments comprising at least one third segment is allocated to said each first segment, wherein a set of at least a fragment containing third data associated with the 3D points of the group is allocated to the set of third segments, wherein the at least fragment corresponds to a two-dimensional parameterization of the group of 3D points. 12. The method according to one of claims 8, 10 and 11, wherein the rendering of at least a portion of the 3D scene is carried out in accordance with the first and second data. 13. The apparatus of claim 2, wherein said each fragment further comprises third data representing a texture associated with the 3D points of the group, the third data being encoded into said at least second track. 14. The apparatus of claim 2, wherein a set of third frame segments comprising at least one third segment is allocated to said each first segment, and a set of at least a fragment comprising third data representing a texture associated with the 3D points group is located in said set of third segments, wherein at least the fragment corresponds to a two-dimensional parameterization of a group of 3D points, and the third data is encoded into at least a third track. 15. The device according to one of claims 2, 13 and 14, wherein when the size of the fragment of the set is larger than the size of the second segment in which the fragment is to be placed, then the fragment is divided into a plurality of subfragments, each of which is smaller than the size of the second segment. 16. The apparatus according to one of claims 2, 13 and 14, wherein the tiles of the set are placed in priority order based on the visual importance of the tiles, wherein the visual importance depends on second data associated with the tiles. 17. The device according to one of claims 2, 13 and 14, wherein the second segments have the same size, which is fixed for a plurality of consecutive second frames. 18. The apparatus of claim 9, wherein third data representing a texture associated with the 3D group points contained in said each fragment is further decoded from at least the second track. 19. The apparatus of claim 9, wherein third data representing the texture is further decoded from at least a third track, the third data being allocated to a plurality of third segments of the third frame, the set of third segments comprising at least one third segment , is allocated to said each first segment, wherein a set of at least a fragment containing third data associated with the 3D points of the group is allocated to the set of third segments, wherein the at least fragment corresponds to a two-dimensional parameterization of the group of 3D points. 20. The device according to one of claims 9, 18 and 19, wherein the rendering of at least a portion of the 3D scene is carried out in accordance with the first and second data.

Images