RU2788439C2 - Method and device for generation of points of three-dimensional (3d) scene - Google Patents

Abstract

FIELD: information technology; image processing.

SUBSTANCE: method for generation of points of a three-dimensional scene from a depth image contains stages, at which, for the current pixel of the mentioned depth image, information about depth, associated with the mentioned current pixel, is compared with information about depth, associated with pixels spatially adjacent to the mentioned current pixel on the mentioned depth image, when the difference of depths between the mentioned current pixel and the adjacent pixel is more than the first value or equal to it and less than the second value, or equal to it, wherein the second value indicates that the current pixel and the adjacent pixel relate to the same object of the tree-dimensional scene or to the same part of the object of the three-dimensional scene.

EFFECT: increase in the accuracy of building of a three-dimensional image scene.

15 cl, 17 dwg

Description

1. FIELD OF TECHNOLOGY

This document relates to the field of three-dimensional (3D) scene processing. In particular, but not exclusively, the field of technology of the present invention relates to the processing of points in a point cloud, which is the geometry and/or structure of a three-dimensional scene or part thereof, for example, the geometry and/or structure of a three-dimensional object. The technical field of the present invention may also relate to the encoding/decoding of 3D image data using a structure and depth projection scheme.

2. STATE OF THE ART

This section is intended to familiarize the reader with various aspects of the art that may relate to various aspects of this document, which are described and/or claimed below. It is intended that the benefit of this discussion is to provide the reader with background information to provide a more complete understanding of the various aspects of the present invention. Accordingly, these statements are to be understood in that light and not as an admission of prior art.

It is well known to represent a 3D object using a point cloud or grid.

A mesh contains mesh elements (like triangles), each of which is represented by vertices and edges connecting the vertices. A mesh is usually intended to represent the outer surface of a 3D object.

A point cloud is a set of points that are usually intended to represent the outer surface of a 3D object, but also for more complex geometries such as hair and fur that cannot be represented efficiently with other data formats such as meshes. Each point in a point cloud is often defined by a 3D spatial location (X, Y, and Z coordinates in 3D space) and possibly other relevant attributes such as, for example, color represented in RGB or YUV color space, transparency, reflectance, two-component vector normals, etc. One can consider a cloud of colored points, that is, a set of 6-component points (X, Y, Z, R, G, B) or, equivalently, (X, Y, Z, Y, U, V), where (X, Y, Z) specifies the spatial location of a point in 3D space, and (R, G, B) or (Y, U, V) specifies the color or texture of that point.

A point cloud can be represented using texture and depth projections corresponding to an image-based point cloud representation.

After depth compression, some points in the 3D scene may be reconstructed erroneously or simply missing, resulting in holes in the 3D scene that can be seen from the viewpoint.

3. SUMMARY OF THE INVENTION

References in the description to "one embodiment", "an embodiment", "an exemplary embodiment", "a specific embodiment" indicate that the described embodiment may include a particular feature, structure, or characteristic, but each embodiment may not necessarily include a specific feature, structure, or characteristic. Moreover, such phrases do not necessarily refer to the same embodiment. In addition, when a particular feature, structure, or characteristic is described in relation to one embodiment, it is assumed that one skilled in the art is aware that it affects such feature, structure, or characteristic in relation to other embodiments, whether or not described. obviously.

The present document relates to a method for generating 3D scene points from a depth image, the method comprising the steps of:

- for the current depth image pixel, comparing the depth information associated with the current pixel with the depth information associated with pixels spatially adjacent to the current pixel in the depth image;

- when the depth difference between the current pixel and the adjacent pixel is greater than the first value and less than the second value, at least one additional point of the three-dimensional scene is generated in addition to the current point associated with the current pixel of the depth image, while the number of additional points depends on this difference.

The present document also relates to a device capable of generating 3D scene points from a depth image, the device comprising a memory associated with at least one processor configured to:

- for the current pixel of the depth image, comparing the depth information associated with the current pixel with the depth information associated with pixels spatially adjacent to the current pixel in the depth image;

- when the depth difference between the current pixel and the adjacent pixel is greater than the first value and less than the second value, generating at least one additional point of the three-dimensional scene in addition to the current point associated with the current pixel of the depth image, while the number of additional points depends on the difference.

This document also relates to a device capable of generating 3D scene points from a depth image, the device comprising:

means for comparing, for the current pixel of the depth image, the depth information associated with the current pixel with the depth information associated with pixels spatially adjacent to the current pixel in the depth image;

- means for generating at least one additional point of the three-dimensional scene in addition to the current point associated with the current pixel of the depth image, when the depth difference between the current pixel and the adjacent pixel is greater than the first value and less than the second value, while the number of additional points depends on this differences.

In accordance with one characteristic, an adjacent pixel is selected having the largest depth difference with the current pixel, among adjacent pixels having a depth difference with the current pixel lying between the first value and the second value, while the number of additional points generated depends on the largest depth difference.

In accordance with one particular characteristic, a volume unit is associated with a point in a 3D scene, with a depth difference corresponding to a certain number of volume units, and the number of generated points corresponding to a depth difference minus 1.

In accordance with another characteristic, attributes are determined to be associated with at least one additional point, the attributes being determined from the attributes associated with the current point and adjacent pixel.

In accordance with one additional characteristic, the attributes to be associated with at least one additional point contain depth information and/or texture information.

According to an additional characteristic, the depth image is decoded from the received bitstream.

According to another characteristic, the points of a 3D scene are part of a point cloud.

The present document also relates to a computer program product containing program code instructions for at least one processor to execute the above method for generating 3D scene points from a depth image when the program is executed on a computer.

The present document also relates to a processor-readable medium (non-volatile storage) storing instructions causing a processor to execute at least the above-mentioned method for generating 3D scene points from a depth image.

4. LIST OF DRAWINGS

This document will be better understood, and other specific features and advantages will become clear after reading the following description, and the description refers to the accompanying drawings, in which:

- fig. 1 shows an example of a 3D scene object according to one example of the present invention;

- fig. 2A and 2B show display examples of attributes associated with the 3D object in FIG. 1, in accordance with one non-limiting embodiment of the present invention;

- fig. 3 shows a first example of image acquisition of the attributes of FIG. 2A and 2B in accordance with one non-limiting embodiment of the present invention;

- fig. 4 shows a second example of image acquisition of the attributes of FIG. 2A and 2B in accordance with one non-limiting embodiment of the present invention;

- fig. 5A and 5B show points of the 3D object in FIG. 1 in accordance with a certain viewpoint in accordance with one non-limiting embodiment of the present invention;

- fig. 6A and 6B show a 3D representation of some points of the 3D object in FIG. 1 and a corresponding depth image in accordance with a non-limiting embodiment of the present invention;

- each of FIG. 7A and 7B show a 2D representation of the holes present in the 3D representation of FIG. 6A, in accordance with one non-limiting embodiment of the present invention;

- fig. 8 shows a method for generating one or more points of the 3D scene of FIG. 1 in accordance with one non-limiting embodiment of the present invention;

- fig. 9 shows an example of a device architecture for implementing the method shown in FIG. 8, 12 and/or 13, in accordance with one non-limiting embodiment of the present invention;

- fig. 10 shows an example of an encoding/decoding scheme for a point cloud representing the 3D object in FIG. 1, in accordance with one example of the present invention;

- fig. 11 shows an example of the syntax of a signal transmitting a bitstream obtained by the circuit in FIG. 10, in accordance with one non-limiting embodiment of the present invention;

- fig. 12 shows an example of a process implemented in encoding a point cloud representing the 3D object in FIG. 1, in accordance with one non-limiting embodiment of the present invention; and

- fig. 13 shows an example of a bitstream decoding process to obtain a decoded point cloud representing the 3D object in FIG. 1, in accordance with one non-limiting embodiment of the present invention.

5. DETAILED DESCRIPTION OF EMBODIMENTS

The subject matter will now be described with reference to the drawings, in which like reference numerals are used throughout to refer to like elements. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the subject matter of the invention. However, it may be apparent that embodiments of the invention may be practiced without these specific details.

In accordance with non-limiting embodiments, a method and apparatus for generating points in a three-dimensional (3D) scene are disclosed.

In the description below, an image contains one or more arrays of discrete values (pixel values) in a specific image/video format that define all information regarding the pixel values of the image (or video) and all information that can be used, for example, by a display and/or any other a device for rendering and/or decoding an image (or video). The image contains at least one component in the form of a first array of discrete values, typically a luminance (or brightness) component, and possibly at least one other component in the form of at least one other array of discrete values, typically a color component. Or, equivalently, the same information can also be represented as a set of arrays of discrete color values, such as the traditional three-color RGB representation.

In the future, the frame can be considered as an image, that is, an array of discrete values, or as a set of images.

A pixel value is represented by a vector of nv values, where nv is the number of components. Each vector value is represented by some number of bits, which defines the maximum dynamic range of the pixel values.

The present principles will be described with reference to a specific embodiment of a method (and apparatus capable of) generating one or more points of a 3D scene, such as one or more points of a 3D scene object in addition to the existing points. Attributes corresponding to texture information (texture attributes) and/or depth information (depth attributes) can be associated with existing points in a 3D scene. Depth (or distance) attributes are stored in the depth image, i.e. associated with depth image pixels, and texture (or color) attributes are stored in the texture image, i.e. associated with texture image pixels. The depth information associated with the current depth image pixel is compared with the depth information associated with depth image pixels that are adjacent to the current pixel. The current pixel with its adjacent pixels forms a block of 9 pixels with 8 adjacent pixels surrounding the current pixel. This block of 9 pixels is associated with a set of 9 points in the 3D scene, with 8 points associated with 8 adjacent pixels, forming a neighborhood of the current point associated with the current pixel. For at least one of the adjacent pixels having a depth difference relative to the current pixel, which is less than the first value and greater than the second value, one or more additional points of the three-dimensional scene are generated in addition to the current point associated with the current depth map pixel. The number of additional points generated depends on the depth difference: the greater the depth difference, the greater the number of additional points generated.

The neighborhood of a 3D scene point in the depth image (respectively, the texture image) corresponds to the pixels of the depth image (respectively, the texture image) located in the vicinity of the pixel of the depth image (respectively, the texture image), the latter corresponding to the specified point of the 3D object. The neighborhood of a pixel may, for example, cover all pixels adjacent to said pixel, without being limited to such an example. For example, the neighborhood may include pixels adjacent to said pixel and all pixels adjacent to each pixel adjacent to said pixel, or even more.

Generating additional points in the 3D scene allows you to fill holes in the 3D scene that may be visible when changing the viewpoint. It can also allow you to increase the density of points in some areas of the 3D scene. Using a depth image (instead of the 3D space of a 3D object) allows you to quickly identify holes that need to be filled.

Even if the description is given with reference to a single depth image of a 3D scene, the present principles apply equally to a plurality of depth images.

Fig. 1 shows two different representations of a three-dimensional object or part thereof, in accordance with a non-limiting embodiment. The 3D object may, for example, be included in a 3D scene containing one or more 3D objects. According to the example in FIG. 1, the object is a person, for example, moving inside the scene, and the part of the object corresponding to the head is shown in FIG. 1. FIG. 1 may also depict a three-dimensional (3D) model of the object 10 and points of the point cloud 11 corresponding to the three-dimensional model 10. The model 10 may be a three-dimensional grid representation and the points of the point cloud 11 may be grid vertices. The points 11 may also be points distributed over the surface of the mesh faces. The model 10 can also be represented as a splatted (i.e., snowball-throwing) version of the point cloud 11, whereby the surface of the model 10 is created by splatting ("snowball-throwing") a point from the point cloud 11. The model 10 can be represented using many different representations such as voxels or splines. Fig. 1 illustrates the fact that it is always possible to define a point cloud from a representation of the surface of a 3D object, and vice versa, it is always possible to create a representation of the surface of a 3D object from points in the cloud. In the present description, projecting points of a 3D object (by propagating points of a 3D scene) onto an image is equivalent to projecting any representation of that 3D object onto the object.

The first representation of the 10th part of the object is a point cloud. A point cloud corresponds to a large set of points representing an object, such as the outer surface or outer shape of an object. A point cloud can be thought of as a vector-based structure in which each point has its own coordinates (for example, 3D XYZ coordinates or depth/distance from a given viewpoint) and one or more attributes, also called a component. An example of a component is the color component, which can be expressed in different color spaces, such as RGB (red, green and blue) or YUV (Y is the luminance component and UV are the two chrominance components). A point cloud is a representation of an object as seen from a given viewpoint or range of viewpoints. A point cloud can be obtained in many ways, for example:

by capturing a real object captured by a camera stand, optionally supplemented with an active depth sensor;

by capturing a virtual/synthetic object captured by the virtual camera stand in the modeling tool;

from a combination of both real and virtual objects.

In the first case (by capturing a real object), a set of cameras generates a set of images or sequences of images (video) corresponding to different angles (different viewpoints). Depth information, i.e. the distance from the center of each camera to the surface of the object, is obtained either using an active depth sensor, for example, in the infrared range and based on the analysis of structured light or propagation time, or based on disparity algorithms. In both cases, all cameras must be calibrated, internally and externally. Disparity algorithms consist of searching for similar visual features in a pair of corrected camera images, which is usually performed along a one-dimensional line: the greater the difference in pixel columns, the closer the surface of this feature. In the case of a camera array, global depth information can be obtained by combining a plurality of pairwise disparity information, taking advantage of multiple camera pairs, thereby improving the signal-to-noise ratio.

In the second case (synthetic object), the modeling tool provides the depth information directly.

The point cloud 10 may be a dynamic point cloud that evolves over time, ie. the number of points may vary over time and/or depending on the location (eg, at least one of the X, Y, and Z coordinates) of one or more points. The evolution of a point cloud may correspond to the movement of the object represented by the point cloud and/or any change in the shape of the object or part(s) of the object.

The point cloud 10 may be represented on a frame or in one or more groups of successive frames, with each frame containing a representation of the point cloud at a specific time "t". One or more groups of consecutive frames may form a video representing at least a portion of the cloud 10 points.

The second representation 11 of the object part can be obtained from the point cloud representation 10, the second representation corresponding to the surface representation. A point cloud can be processed to compute its surface. To this end, for a given point in the point cloud, the neighboring points of that given point are used to calculate the normal to the local surface at that given point, with the surface element associated with that given point derived from that normal. The process is repeated for all points to obtain a surface. Methods for restoring a surface from a point cloud are described, for example, by Matthew Berger et al. in “ State of the Art in Surface Reconstruction from Point Clouds ”, State of the Art Report, 2014. In one embodiment, the surface element associated with a given cloud point points, is obtained by applying the splat rendering method ("throwing snowballs") to this given point. The surface of an object (also called the estimated surface or outer surface of the object) is obtained by blending all the "snowball tracks" (eg, ellipsoids) associated with points in the point cloud.

In one particular embodiment, the point cloud only represents partial views of the object, not the object as a whole, and this corresponds to how the object should look from the rendered side, such as in a cinematic scene. For example, when shooting a character facing a flat camera array, a point cloud is generated only on the side of the booth. The back of the character does not exist at all, the object is not closed, and therefore the geometric characteristics of this object are the totality of all surfaces oriented in the direction of the stand (the angle between the normal to each local surface and the beam back to the gripping device is, for example, less than 90°).

Each of the FIGS. 2A and 2B show a 10 point cloud frame in accordance with specific non-limiting embodiments of the present principles.

Fig. 2A shows a first example of a point cloud frame 20, such as a GOP (Group of Pictures) frame of a point cloud. Frame 20 consists of a set of n images 201, 202, 203, 20n, where n is an integer greater than or equal to 2. Each of images 201 to 20n corresponds to an array of pixels, the size and/or definition of which may vary from one image to another. another image. For example, the definitions of images 201 and 20n are the same, while the definitions of images 202, 203 differ from each other and from the definitions of images 201 and 20n. In the example in FIG. 2A, images 201 to 20n are spatially arranged to cover the entire frame 20 with no overlap between images. According to one embodiment, images 201 to 20n do not cover the entire frame, there is space between images 201 to 202, or between at least some of them, i.e. the edges of two adjacent images may not touch. Each pixel of images 201 to 20n may have data associated with it, such as texture information and/or depth information. Texture information, for example, may be stored as gray levels associated with each channel of a color space (for example, RGB color space or YUV color space), with the gray level for each channel, for example, expressed using the first certain number of bits, for example, 8, 10 or 12 bits. Depth information, for example, can be stored as a value, for example, in the α-channel using a second certain number of bits, such as 8, 10 or 12 bits. Thus, four RGBα or YUVα components (eg, four 10-bit channels) may be associated with, for example, each pixel in Figure 20 to represent a point cloud at a particular time "t". According to one embodiment, the first frame 20 is used to store texture information (e.g. 3 RGB or YUV components) and the second frame with the same image arrangement is used to store depth information, both images representing a point cloud at the time time "t". The set of images forming frame 20 can be obtained, for example, by projecting the points of the point cloud in accordance with the projection(s), for example, a different projection for each first image, as shown in Figure 3.

Fig. 3 shows a cube 31 delimiting at least a portion of a point cloud 10 in accordance with a non-limiting example of the present principles.

Cube 31 is partitioned, for example, into 8 sub-cubes (only one sub-cube 32 out of 8 sub-cubes is shown for clarity) at a first level of partitioning. Subcube 32 is also split into 8 subcubes (only one subcube 33 out of 8 subcubes is shown for clarity) at the second level of partitioning. At each mesh level, a portion of the points in the point cloud may be projected (eg, in accordance with an orthographic projection) onto one or more faces of the cube(s) (eg, gray-filled faces). For example, point cloud points are projected onto face 301 of cube 31, onto face 302 of cube 32, and onto face 303 of cube 33. For example, the faces are sampled into pixel arrays with definitions/sizes that depend on the level of cube subdivision. For example, for a cube face pixel, the point in the point cloud that is projected onto said pixel corresponds to the point in the point cloud that is closest to the specified pixel when raytraced from the specified pixel and orthogonal to the face containing the specified pixel. The attributes associated with said pixel correspond to the attributes (texture and/or depth) of a point projected onto said pixel.

Face 301 is used for imaging 201, face 302 for imaging 302, and face 303 for imaging 303, for example.

Fig. 2B illustrates a second example of a point cloud frame 21, such as a point cloud GOP frame. Frame 21 consists of a set of m images 211, 212, 213, 214, 21m, where m is an integer greater than or equal to 2. The arrangement of images 211 to 21m may differ from the arrangement of frame 20, for example, with free space between images with 211 to 21m. Images 211 to 21m may have different sizes and/or definitions. Each image may receive attributes from point cloud points, said attributes being associated with at least some of each image's pixels 211 to 21m. For example, the part of each image that receives attributes from the point cloud is shown as a gray area, and the part of the image that does not receive attributes from the point cloud is shown as a white area, and the specified white area can be filled with a default value, as free space between images. As for frame 20 in FIG. 2A, data associated with image pixels 211 to 21n may correspond to texture information and/or depth information. In one embodiment, the first frame 21 is used to store texture information (e.g. 3 RGB or YUV components), and the second frame 21 with the same arrangement of images 211 to 21m is used to store depth information, both images representing a cloud point in time point "t".

The set of images forming frame 21 may include, for example, one or more first images and potentially one or more second images. The first image(s) (at least the gray area of each first image) can be obtained, for example, by projecting the points of the point cloud in accordance with the first projection(s), for example, a different first projection on each first image, as shown in FIG. four.

Fig. 4 depicts the acquisition of the first images of the set of images constituting frame 21 in accordance with a non-limiting example of the present principles. The point cloud representing the 3D object 4 is partitioned into a plurality of 3D parts, such as 50, 100, 1000 or more 3D parts, 3 of which are shown in FIG. 4, that is, three-dimensional portions 42, 43 and 44, wherein the three-dimensional portion 44 contains points of a point cloud representing a part of a human head (corresponding, for example, to point cloud 10 in FIG. 1), the three-dimensional portion 42 contains points of a point cloud representing armpit of a person, and the three-dimensional part 43 contains the points of a point cloud representing a human arm. One or more images of each three-dimensional part or part of three-dimensional parts are generated in order to represent each three-dimensional part in two dimensions, that is, according to a two-dimensional parameterization. For example, a 2D parameterization 401 is obtained for the 3D part 44, a 2D parameterization 402 is obtained for the 3D part 42, and 2 different 2D parameterizations 403 and 404 are obtained for the 3D part 43. The 2D parameterization may vary from one 3D part to another. For example, the 2D parametrization 401 associated with the 3D part 41 is a linear perspective projection, while the 2D parametrization 402 associated with the 3D part 42 is LLE, and the 2D parametrization 403 and 404 associated with the 3D part 43 are both orthogonal. projections corresponding to different viewpoints. According to one embodiment, all 2D parameterizations associated with all 3D parts are of the same type, such as linear perspective projection or orthographic projection. According to one embodiment, different 2D parameterizations may be used for the same 3D part.

The 2D parameterization associated with one given 3D part of the point cloud corresponds to viewing in 2 dimensions the given 3D part of the point cloud allowing discretization of the given 3D part, i.e. the 2D representation of the content (i.e. the point(s)) of this given 3D part contains the set discrete values (which may correspond to the pixels of the first image), the number of which depends on the applied sampling step. The two-dimensional parametrization can be obtained in various ways, for example, by implementing any of the following methods:

- a linear perspective projection of the points of the three-dimensional part of the point cloud onto the plane associated with the viewpoint, while the parameters characterizing the linear perspective projection include the location of the virtual camera, the spatial sampling step and the field of view in two dimensions;

- orthographic projection of points of the three-dimensional part of the point cloud onto the surface, while the parameters characterizing the orthographic projection include the geometry (shape, size and orientation) of the projecting surface and the spatial discretization step;

- LLE (local-linear embedding, Locally-Linear Embedding), which corresponds to the mathematical operation of dimensionality reduction, is used here for converting / converting from 3D to 2D, while the parameters characterizing LLE include conversion coefficients.

Each image is preferably rectangular to facilitate the packaging process on frame 21.

Fig. 5A and 5B show a 2D representation of a portion of the 3D scene of FIG. 1 in accordance with a non-limiting example of the present principles.

In FIG. 5A shows an example of a 2D representation 5A of a portion of a 3D object 10, such as a cross section of a 3D object 10. Cross section points 5A are represented by a square that corresponds to a cube or voxel in 3D representation 10. For example, a square (or cube) 510 is represented by a point 51. Cubes or voxels are associated with points in a 3D scene (or points in a point cloud when the 3D scene is represented by a point cloud) to fill the space of the 3D scene. Some of the squares shown in Fig. 5A may be empty, i.e. there is no point in the 3D scene associated with them, i.e. there is no texture and depth associated with them. The surface of the three-dimensional object 10 is represented by a straight line 50, and the points 501 to 506 correspond to the points seen from the viewpoint 52. The squares associated with points 501 to 506 are shown in grayscale. By projecting these points 501 to 506 onto the respective depth image corresponding to the viewpoint 52, a column of pixels associated with points 501 to 506 is obtained, with one pixel associated with one point.

Naturally, volume elements other than a square, such as spheres, can be associated with points in a 3D scene. The remainder of the description will use the term "volume unit" to refer to the volume element associated with a point, with the volume unit corresponding to, for example, a 1 by 1 by 1 voxel, such as 1 mm by 1 mm by 1 mm (with a volume of 1 mm ³ ) or 1 cm by 1 cm by 1 cm (with a volume of 1 cm ³ ) or any other size.

Fig. 5B shows a 2D representation 5B of the same part of the 3D object 10 as the part shown in FIG. 5A, wherein the 2D representation 5B is obtained by backprojecting points 501 to 506 from the depth image obtained from the 2D representation 5A corresponding to the viewpoint 52. Points (and corresponding units of volume) 501 to 506 are shown using a shade of gray. Points (and associated volume units) 511 to 518 are shown with a diagonal stripe pattern and correspond to holes in the three-dimensional object 10 as viewed from viewpoint 53. These holes may be the result of a reconstruction of the 3D object using a depth image obtained by projecting points according to a viewpoint 52 that is different from the viewpoint 53, such holes leading to visible artifacts when the reconstructed 3D object is viewed from the viewpoint 53. Hole points 511 to 518 do not have any texture or depth information associated with them.

Fig. 6A shows a subset 6A of points of the 3D object in FIG. 1 in three dimensions, and FIG. 6B shows a portion 6B of a depth image associated with subset 6A, in accordance with a non-limiting example of the present principles.

Fig. 6A shows a subset 6A of 3D object points and their associated volume units 601, 602, 603, 604, 605, 606, 607, 608, and 609, shown with cubes or voxels in FIG. 6A. Cubes 601 to 609 extend in the three dimensions X, Y, and Z, where the Z-axis is the depth, X is the abscissa, and Y is the y-axis. Arrows 61, 62 and 63 represent different viewpoints from which subset 6A can be viewed. Viewpoint 61 allows you to look at the subset 6A from the side of the Z axis; viewpoint 62 allows you to look at the subset 6A from the side of the Y axis; and viewpoint 63 allows the subset 6A to be viewed from the x-axis side. Cubes 602 to 609 (or equivalently points 602 to 609) may be considered to form a neighborhood of cube 601 (or equivalently points 601).

Fig. 6B shows a portion 6B of the depth image associated with subset 6A. The depth image portion 6B can be obtained by projecting cubes (dots) 601 to 609 in accordance with the viewpoint 61 and the corresponding Z-axis. As seen in FIG. 6B, part 6B forms a block of 9 pixels, 8 adjacent pixels 612 to 619 (each of which corresponds to/refers to respective cubes/dots 602 to 609, respectively, from subset 6A) surrounding pixel 611 (which corresponds to/refers to the associated cube /point 601 from subset 6A). Cubes/dots 602 through 609 correspond to the neighborhood of cube/dot 601, as do their respective corresponding depth image pixels 612-619 correspond to neighboring depth image pixels surrounding pixel 611 corresponding to cube/point 601.

When viewed from the viewpoint 61, there are no holes in the subset 6A (the view according to the viewpoint 61 corresponds to the depth image part 6B). But when viewed from other viewpoints, hole(s) may appear. For example, when viewed from the viewpoint 62, a hole appears between the cubes/points 601 and 602, the space occupied by such a hole corresponds to the space of the cube 6001 shown with stripes. Such a hole 6001 is shown in FIG. 7A. Fig. 7A shows part of the view of subset 6A from viewpoint 62, in particular the part containing cubes 601 and 602 and the free space between them, i.e. the space not occupied by the cube/point from subset 6A. When viewed from the viewpoint 63, a hole appears between the cubes/points 601 and 604, the space occupied by such a hole corresponds to the space of the two cubes shown by the stripes, i.e. cubes 6001 and 6002. Such a hole 6001, 6002 is shown in FIG. 7B, while FIG. 7B shows part of the view of subset 6A from viewpoint 63, in particular the part containing cubes 601 and 604 and the free space between them, ie the space not occupied by the cube/point from subset 6A.

The depth image associated with the 3D scene may be used to determine where holes may be located in the region(s) in the 3D scene. For example, to obtain the location of the hole(s), the depth image portion 6B associated with (and derived from) the point portion 6A of the three-dimensional object 10 is processed and analyzed as explained below. depth information associated with pixels 611 to 619. Block 6C in FIG. 6B shows depth information associated with pixels 611 to 619. Such a process is, for example, applied to the current pixel, namely the current pixel 611 (which is associated with the current point 601) in the example of FIG. 6A and 6B. In the example in FIG. 6B, the current pixel 611 has a reference depth associated with it, for example, the reference depth is 0. The depth associated with pixels 612 through 619 adjacent to the current pixel 611 is expressed in units of volume (volume units corresponding to the cube/voxel in the example of FIG. 6A, 6B). As for the current pixel, the depth associated with the neighboring pixel 612 is +2 volume units; the depth associated with the adjacent pixel 613 is +1 volume unit; the depth associated with neighboring pixel 614 is +3 volume units; the depth associated with neighboring pixel 615 is 0 volume units; the depth associated with neighboring pixel 616 is -1 volume unit; the depth associated with neighboring pixel 617 is 0 volume units; the depth associated with neighboring pixel 618 is 0 volume units; and the depth associated with the adjacent pixel 619 is +1 volume unit. Naturally, the depth associated with pixels can be expressed in terms of any other metric, such as the number of units of volume separating points (i.e., associated cubes) from viewpoints associated with the depth image, i.e., the distance between the viewpoint and the points 3D scene (for example, the depth associated with the current pixel 611 could be 1254 volume units, the depth associated with neighboring pixels 612 through 619 would then be 1256, 1255, 1257, 1254, 1253, 1254, 1254, and 1255 respectively ). In accordance with another example, the depth may be expressed in centimeters or meters, or using any other metric corresponding to the distance, expressed in centimeters or meters, from the viewpoint associated with the depth image.

The depth information associated with each adjacent pixel 612 through 619 is compared with the depth information associated with the current pixel 611. For example, the depth difference between each adjacent pixel 612 through 619 and the current pixel 611 is calculated. When the difference is greater than or equal to the first to a certain value (for example, 2 volume units, when depth is expressed in terms of the number of volume units), this means that the cubes associated with the considered adjacent pixel and with the current pixel are not adjacent in 3D scene space, which means that they are not touching with each other in a 3D scene, which means there is a hole between them. For example, the depth difference between pixels 612 and 611 is +2, which means that there is a hole between the respective cubes 601 and 602 in the 3D scene, as clearly seen in FIG. 6A (the hole corresponding to the empty space is shown with the cube 6001). The depth difference between pixels 614 and 611 is +3, which means that there is a hole between the respective cubes 601 and 604 in the 3D scene, as can be clearly seen in FIG. 6A (hole corresponding to empty space shown with cubes 6001 and 6002). According to another example, the depth difference between pixels 617 and 611 is +0, which means that there is no hole between the respective cubes 607 and 601 in the 3D scene, as clearly seen in FIG. 6A (cubes 607 and 601 are in contact with each other because they both belong to the plane defined by the X and Y axes; they share a face). According to another example, the depth difference between pixels 619 and 611 is +1, which means that there is no hole between the respective cubes 609 and 601 in the 3D scene, as clearly seen in FIG. 6A (cubes 609 and 601 are in contact with each other because they share a common angle).

The same processing may be performed for each point of the 3D scene (or at least each point of a portion of the points of the 3D scene, for example, for each point every two points or each point every 8 points), i. e. each point can be treated as a current point and its depth compared to the depth of its surroundings (ie, in the space of the corresponding depth image).

According to one embodiment, only positive depth differences are considered and compared to the first determined value. A negative depth difference can be considered by taking an adjacent pixel having a negative depth difference as a new or different current pixel. Considering only positive depth differences speeds up the processing of the 3D scene to detect the hole(s).

According to another embodiment, the depth difference between two adjacent pixels is also compared to the second determined value, and only depth differences that are less than or equal to this second determined value can be considered to determine if a hole exists between the cubes associated with adjacent pixels. Comparison with the second specific value ensures that the two cubes belong to the same 3D scene object or the same part of the 3D object. Indeed, if the difference in depth between two neighboring pixels (that is, between two corresponding cubes) is too large, then this means that these two cubes do not belong to the same surface. The second determined value may be, for example, 10, 50 or 100 volume units. A hole between two cubes is defined when the following conditions are met:

Th1≤d≤Th2 Equation 1

where d corresponds to the depth difference between the two considered pixels (and corresponding cubes); Th1 corresponds to the first defined value; and Th2 corresponds to the second determined value.

To fill a certain hole(s), additional cubes/points can be generated between two cubes/points having a depth difference d (in the depth image) satisfying equation 1. Additional cubes/points can be generated by computing their respective depth and texture from depth and a texture associated with the cubes used to determine the hole (eg, by interpolating the points/cubes used to determine the presence of a hole). The number of additional points generated may be a function of the depth difference, such as d minus 1 (d-1) when the depth difference is expressed in terms of the number of volume units. For example, the number of dots/cubes generated between dots/cubes 601 and 602 is 1, and the depth difference between the respective depth image pixels 611 and 612 is 2. According to another example, the number of dots/cubes generated between dots/cubes 601 and 604 is 2, wherein the depth difference between the respective pixels 611 and 614 of the depth image is 3.

An additional point/cube 6001 generated between existing points/cubes 601 and 602 may take as texture information the average value of the texture values associated with points/cubes 601 and 602, and as depth information the depth value associated with point/ cube 601 plus 1 (d ₆₀₁ +1).

Additional points/cubes 6001 and 6002 generated between the existing points/cubes 601 and 604 may receive as texture information the average value of texture values associated with points/cubes 601 and 604, and as depth information the depth value of point 601 plus 1 for cube 6001 (d ₆₀₁ +1) and point depth value 601 plus 2 (d ₆₀₁ +2) for cube 6002.

In accordance with one embodiment, a weight is associated with the texture of the existing dots/cubes to determine the texture of the additional dot/cube. For example, more weight may be associated with dot texture 601 than with dot texture 604 when calculating additional dot texture 6001 because additional dot 6001 is closer to point 601 than to point 604. Conversely, more weight may be associated with dot texture 604 than with the texture of point 601 when calculating the texture of the additional point 6002, since the additional point 6002 is closer to point 604 than to point 601. For example, the weight associated with the texture value when interpolating the texture value that should be associated with the generated additional point , can be inversely proportional to the distance (depth) separating the generated additional point from the point used to generate it. For example, when calculating the texture of additional point 6001, a weight of 2 may be associated with the texture of point 601, and a weight of 1 may be associated with the texture of point 604, while the distance (depth difference) between additional point 6001 and point 601 is 1 volume unit, while the distance (depth difference) between the additional point 6001 and point 604 is 2 volume units.

In accordance with another embodiment, the texture information to be associated with the generated additional point corresponds to the texture information of one of the points used to generate it. For example, dot texture information 601 may be applied to the generated additional point 6001, and point texture information 604 may be applied to the generated additional point 6002.

According to a further optional embodiment, instead of generating additional point(s) each time it is determined that the depth difference d (between the current pixel and the pixel adjacent to that current pixel) is greater than or equal to the first determined value Th1 and less than or equal to the second a certain value of Th2, additional points are generated only between the current pixel and the pixel adjacent to the pixel having the largest depth difference d _max within the interval [Th1; Th2]. For example, instead of generating point/cube 6001 twice (once using points/cubes 601 and 602 and once using point/cubes 601 and 604), point/cube 6001 is generated only once using points/cubes 601 and 604, since the depth difference (i.e. +3) between points 601 and 604 is greater than the depth difference (i.e. +2) between points 601 and 602. To achieve this, the largest depth difference d _max (which is less than or equal to Th2) is chosen among all depth differences d ₆₁₂ to d ₆₁₉ of the block of pixels 6B, and only the neighboring pixel 614 corresponding to the largest depth difference d _max among all neighboring pixels 612 to 619 is considered with the current pixel 611 to generate additional points/cubes (from the corresponding points /cubes 601 and 604).

Fig. 8 shows a method for generating one or more points of a 3D scene, implemented in, for example, device 9 (described with reference to FIG. 9), in accordance with a non-limiting embodiment of the present principles.

In the first operation 81, the depth information associated with the current pixel is compared with the depth information associated with pixels spatially adjacent to the current pixel in the depth image. The depth image is associated with existing points of the 3D scene or part of the 3D scene (eg, existing points of the 3D object). Each pixel of the depth image is associated with a corresponding point of the 3D scene, wherein the depth image is obtained by projecting the points of the 3D scene onto the corresponding pixels, and the depth attributes of the points are stored in the depth image and associated with the corresponding pixels.

In the second operation 82, one or more additional points of the 3D scene are generated in addition to the current point that is associated with the current pixel. The current point, for example, is generated by backprojecting the current pixel using the projection parameters used to obtain the depth image from existing points in the 3D scene. One or more points are generated, for example, by using the current point and a 3D scene point obtained by backprojecting a depth image pixel adjacent to the current pixel when the depth difference between the current pixel and the adjacent pixel is greater than the first value Th1 and less than the second Th2 value. Additional points may be generated, for example, for each pair of points (each pair includes a current point and a point from a neighborhood of the current point corresponding to a depth image pixel adjacent to the current pixel) having a depth difference lying between the first value and the second value. The number of additional points generated depends on the depth difference: the greater the depth difference, the greater the number of additional points generated.

In accordance with one embodiment, the adjacent pixel having the largest depth difference with the current pixel is selected among adjacent pixels having a depth difference with the current pixel lying between the first value and the second value, and additional points are generated using only the current point and the neighboring point. corresponding to said adjacent pixel having the largest depth difference.

The attributes to be associated with the extra points, in particular the texture attributes, are defined using the current point and neighbor point attributes (corresponding to the current pixel and adjacent pixel, respectively) used to generate the extra points. The texture attributes are obtained, for example, by interpolating the texture attributes of the current point and the neighboring point.

The first and second operations may be repeated for a plurality of current pixels (or, equivalently, corresponding current points in the 3D scene), for example, for each depth image pixel or for each pixel of a portion of the depth image pixels.

Fig. 9 shows an example architecture of a device 9 capable of implementing at least one of the methods described with reference to FIG. 6 and 8, in accordance with a non-limiting embodiment of the present principles.

Device 9 contains the following elements, which are connected by a data bus and an address bus 91:

a microprocessor 92 (or CPU), which is, for example, a DSP (Digital Signal Processor);

- ROM (or read only memory) 93;

- RAM (or random access memory) 94;

- storage interface 95;

- an input/output interface 96 for receiving data for transmission from an application; and

- a power source, such as a battery.

According to one example, the power supply is external to the device. For each memory mentioned above, the word "register" used in the description may correspond to a small area (a few bits) or a very large area (eg, the entire program or a large amount of received or decoded data). ROM 93 contains at least the program and parameters. ROM 93 may store algorithms and instructions for performing methods in accordance with these principles. When turned on, the CPU 92 loads the program into RAM and executes the corresponding instructions.

RAM 94 contains in a register a program executed by the CPU 92 and loaded after the device 9 is turned on, input data in the register, method intermediate data in various states in the register, and other variables used to execute the method in the register.

The implementations described herein may be implemented in, for example, a method or process, device, computer program product, data stream, or signal. Even if an implementation has only been discussed in the context of one form of implementation (eg, discussed only as a method or apparatus), the implementation of the features discussed may also be implemented in other forms (eg, program form). The device may be implemented, for example, in suitable hardware, software and firmware. The methods may be implemented, for example, in a device such as, for example, a processor, which refers to processing devices, including, in general, for example, a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices such as, for example, computers, cellular phones, portable/personal digital assistants ("PDAs"), and other devices that enable the transfer of information between end users.

3D scene points (eg, point cloud points) and associated data/attributes (eg, point depth and texture) are obtained from a source. For example, the source belongs to a set containing:

- local memory (93 or 94), for example, video memory or RAM (random access memory, Random-Access Memory), flash memory, ROM (read only memory), hard disk;

- interface (95) storage, for example, an interface with support for mass storage, RAM, flash memory, ROM, optical or magnetic disks;

- a communication interface (96), such as a wired interface (eg a bus interface, a WAN interface, a LAN interface) or a wireless interface (such as an IEEE 802.11 interface or a Bluetooth® interface); and

- a user interface, such as a graphical user interface that allows the user to enter data.

According to examples of decoding/rendering or decoder(s)/render(s), the decoded points or the reconstructed 3D scene are sent to the destination; namely, the destination belongs to the set containing:

- local memory (93 or 94), such as video memory or RAM, flash memory, hard disk;

- interface (95) storage, for example, an interface with support for mass storage, RAM, flash memory, ROM, optical or magnetic disks; and

- communication interface (96), e.g. wired interface (e.g. bus interface (e.g. USB (Universal Serial Bus)), WAN interface, LAN interface, HDMI (High Definition Multimedia Interface )) or a wireless interface (such as an IEEE 802.11, WiFi ® or Bluetooth ® interface).

According to the examples, the device 9 is configured to implement the method described with reference to at least one of FIGS. 6 to 8, and belongs to the set containing:

- mobile device;

- communication device;

- gaming device;

- tablet (or tablet computer);

- laptop;

- camera;

- video camera;

- coding chip;

- server (eg broadcast server, video-on-demand server or web server);

- set-top box;

- TV set; and

- display (e.g. HMD).

Fig. 10 shows an encoding/decoding scheme for a 3D scene, such as a point cloud 103, in accordance with a specific and non-limiting embodiment of the present principles.

The point cloud 103 is encoded into encoded data as a bitstream 104 by an encoding process 101 implemented in module M101. The bit stream is transmitted to module M102, which implements a decoding process 102 for decoding encoded data to obtain a decoded point cloud 105 . Modules M101 and M102 may be hardware, software, or a combination of hardware and software.

The point cloud 103 corresponds to a large set of points representing an object, such as the outer surface or outer shape of the object. A point cloud can be thought of as a vector-based structure in which each point has its own coordinates (eg, 3D XYZ coordinates or depth/distance from a given viewpoint) and one or more components. An example of a component is the color component, which can be expressed in different color spaces, such as RGB (red, green and blue) or YUV (Y is the luminance component and UV are the two chrominance components). A point cloud may be a representation of an object as seen from one or more viewpoints. A point cloud can be obtained in many ways, for example:

by capturing a real object captured by one or more cameras, optionally supplemented with an active depth sensor;

by capturing a virtual/synthetic object captured by one or more virtual cameras in the modeling tool;

from a combination of both real and virtual objects.

The point cloud 103 may be a dynamic point cloud that evolves over time, ie. the number of points may vary over time and/or depending on the location (eg, at least one of the X, Y, and Z coordinates) of one or more points. The evolution of a point cloud may correspond to the movement of the object represented by the point cloud and/or any change in the shape of the object or part(s) of the object.

The point cloud 103 may be represented in a frame or in one or more groups of consecutive frames, with each frame containing a representation of the point cloud at a specific time "t". One or more groups of consecutive frames may form a video representing at least a portion of the point cloud 103 .

The encoding process 101 may, for example, implement intra-frame coding and/or inter-frame coding. Intra-frame coding is based on intra-frame prediction which exploits spatial redundancy, ie correlation between pixels within the same image, by calculating prediction values by extrapolation from already encoded pixels for efficient delta coding. Interframe coding is based on interframe prediction which uses temporal redundancy. Time-independently encoded so-called intra-coded "I" frames use only intra-coding. Time-coded "P" (or "B") predicted frames may use intra-frame and inter-frame prediction.

The decoding process 102 may correspond, for example, to the inverse operations of the encoding process 101 for decoding data encoded with the encoding process.

Fig. 11 shows an example of the syntax of a signal transmitted between two remote devices, for example, when data is transmitted over a packet-based transmission protocol. Each transmitted packet P contains a header H and payload data PAYLOAD.

According to embodiments, the PAYLOAD payload may comprise at least one of the following elements:

- bits that represent at least one image representing a cloud of points at a certain time t, for example, the bits may represent texture information and/or depth information associated with pixels of at least one image;

- bits, which are projection and correspondence information data between projections and images of at least one frame.

Fig. 12 shows operations for encoding a 3D scene, such as a point cloud 103, in accordance with a specific and non-limiting embodiment of the present principles. The operations may be part of the encoding process 101 and may be implemented by the device 9 in FIG. nine.

In operation 120, point cloud frame 20 data is encoded with encoder ENC1. Frame 20 is, for example, part of a group of frames (GOP) and contains data representing a cloud of points at a certain time "t". Frame 20 may contain a set of images, wherein at least one of the images in the set contains attributes that correspond to at least a portion of the frame 20 data. In this case, the attributes correspond to the attributes of the points of the portion of the point cloud projected onto said each image. The attributes may correspond to texture (or color) information and/or depth information (or viewpoint distance). The ENC1 encoder, for example, is compatible with a traditional encoder such as:

JPEG specification ISO/CEI 10918-1 Recommendation UIT-T T.81, https://www.itu.int/rec/T-REC-T.81/en,

AVC, also called MPEG-4 AVC or h264. Described in UIT-T H.264 and ISO/CEI MPEG-4 Part 10 (ISO/CEI 14496-10), http://www.itu.int/rec/T-REC-H.264/en,

HEVC (description can be found on the ITU website, recommendation T, series H, h265, <http://www.itu.int/rec/T-REC-H.265-201612-I/en>), or

3D-HEVC (extension of HEVC, described on the ITU website, recommendation T, series H, h265, http://www.itu.int/rec/T-REC-H.265-201612-I/en Appendix G and I).

The encoded data of frame 20 may be stored and/or transmitted in bitstream 104.

In operation 121, the encoded data of frame 20 is decoded by the decoder DEC1. DEC1 decoder is compatible with ENC1 encoder, for example, compatible with traditional decoder such as:

JPEG,

AVC, also called MPEG-4 AVC or h264,

HEVC or

3D-HEVC (HEVC extension).

The attributes encoded in operation 120 are decoded and retrieved in operation 121, e.g. stored in buffer memory for use in generating the reference frame 125 associated with frame 20.

In operation 122 implemented by module M122, each image is processed as described with reference to FIG. 6 and/or 8 to generate point cloud points and to fill holes, which may be the result of encoding/decoding processes 120, 121, for example.

Reference frame 125 may be obtained from the corrected image(s) obtained from module M122.

The reference frame 125 may be used in operation 124 implemented using the module M124. Operation 124 comprises, for example, generating inter prediction predictors for encoding one or more point cloud frames other than frame 20 (eg, point cloud frames at specific times other than time t of frame 20). The point cloud 103 or frames representing the point cloud can then be encoded by referring to the reference frame 125. In accordance with one embodiment, the M124 module is part of the ENC1 encoder.

Naturally, in the same manner as reference frame 125, a plurality of reference frames may be obtained, with each reference frame in the set being derived from a particular point cloud frame, with point cloud 103 encoding referring to one or more reference frames. .

Fig. 13 shows operations for decoding an encoded version of a point cloud 103 from a bit stream 104 in accordance with a specific and non-limiting embodiment of the present principles. The operations may be part of the decoding process 102 and may be implemented by device 9 in FIG. nine.

In operation 131, the encoded data of one or more frames (eg, frames of one or more GOPs or intra period) of the point cloud is decoded by the decoder DEC2 from the received bitstream 104. Bitstream 104 contains the encoded data of one or more frames. Each frame contains a set of images, at least one of the images from this set contains attributes that correspond to at least a portion of the image data that has been encoded. The attributes can be obtained by projecting, according to the first projection, a portion of the point cloud in each first image, the attributes corresponding to the attributes of the points of the portion of the point cloud projected onto said each first image. The attributes may correspond to texture (or color) information and/or depth information (or viewpoint distance). Decoder DEC2 may correspond to decoder DEC1 in FIG. 12, and it is compatible with, for example, a traditional decoder such as:

JPEG,

AVC, also called MPEG-4 AVC or H264,

HEVC or

3D-HEVC (HEVC extension).

The attributes decoded in operation 131 are retrieved, eg stored in buffer memory, for use in generating one or more reference frames 135, with each reference frame associated with one frame. In the following, for clarity and brevity, only one reference frame associated with one frame will be considered.

In operation 132, implemented by module M132 (which may be identical to module M122 in FIG. 12), the decoded attributes are processed as described with reference to FIG. 6 and/or 8 to generate point cloud points and to fill in holes that may result, for example, from encoding/decoding processes 120, 131.

Reference frame 135 (which may be identical to reference frame 125 of FIG. 12) may be obtained from the frame by merging the decoded first attributes obtained in operation 121 with the second attributes obtained in operation 123. The reference frame may contain the same structure as and a frame, i.e. the same spatial arrangement of the plurality of images, but with different data, i.e. decoded first attributes and obtained second attributes. A detailed description of an exemplary process for obtaining a reference frame has been provided above in connection with FIG. nine.

The reference frame 135 may be used in operation 134 implemented with module M134. Operation 134 includes, for example, generating predictors for inter prediction when decoding the encoded data contained in the bitstream. This data associated with the generation of the predictor may contain

- type of prediction, for example a flag indicating whether the prediction mode is internal or external,

- motion vector and/or

- index for specifying the key frame in the list of key frames.

Naturally, in the same manner as reference frame 135, a plurality of reference frames may be obtained, with each reference frame of this set derived from the decoded data of a particular point cloud frame, while the decoding of bitstream data 104 may be based on one or several reference frames to obtain a decoded cloud of 105 points.

Naturally, this document is not limited to the embodiments described above.

The present document is not limited to a method for generating point(s) of a 3D scene, but also covers a method for encoding and/or decoding a 3D scene/point cloud and/or a method and apparatus for transmitting a bitstream obtained by encoding a 3D scene/point cloud, and /or a method and apparatus for receiving a bitstream obtained by encoding a three-dimensional scene/point cloud. The present document also covers a method and apparatus for rendering and/or displaying a decoded 3D scene/point cloud, i.e., images of a 3D object represented by a decoded point cloud, with each image having an associated viewpoint.

The implementations described herein may be implemented in, for example, a method or process, device, software program, data stream, or signal. Even if an implementation has only been discussed in the context of one form of implementation (eg, discussed only as a method or apparatus), the implementation of the features discussed may also be implemented in other forms (eg, program form). The device may be implemented, for example, in suitable hardware, software and firmware. The methods may be implemented, for example, in a device such as, for example, a processor, which refers to processing devices, including, in general, for example, a computer, microprocessor, integrated circuit, or programmable logic device. Processors also include communication devices such as, for example, smartphones, tablets, computers, mobile phones, portable/personal digital assistants ("PDAs"), and other devices that enable the transfer of information between end users.

Implementations of the various processes and features described herein may be embodied in a variety of different applications or equipment, in particular, for example, applications or equipment related to data encoding, data decoding, rendering generation, texture processing, and other processing of images and related information. about textures and/or depth information. Examples of such equipment include an encoder, a decoder, a post-processor that processes the output of the decoder, a preprocessor that provides input to the encoder, a video encoder, a video decoder, a video codec, a web server, a set-top box, a laptop, a personal computer, a cellular phone, a PDA, an HMD ( helmet display, Head-Mounting Display), smart glasses and other communication devices. As should be clear, the equipment may be mobile and even installed in a mobile vehicle.

Additionally, the methods may be implemented using instructions executable by a processor, and such instructions (and/or data values generated by the implementation) may be stored on a processor-readable medium, such as, for example, an integrated circuit, a program medium, or other storage device, such as , for example, a hard disk, a compact diskette (“CD”), an optical disk (such as, for example, a DVD, often referred to as a digital versatile disk or digital video disc), random access memory (“RAM”), or read only memory (“ROM”). The instructions may form an application program tangibly embodied on a processor-readable medium. The instructions may be, for example, in hardware, firmware, software, or a combination thereof. Instructions can be found, for example, in the operating system, a single application, or a combination of both. Therefore, a processor can be characterized as, for example, both a device capable of executing a process and a device that includes a processor-readable medium (such as a storage device) having instructions for executing a process. In addition, the processor-readable medium may store, in addition to or instead of instructions, data values generated by the implementation.

As will be appreciated by one of skill in the art, implementations may create a variety of signals formatted to carry information that may, for example, be stored or transmitted. The information may include, for example, instructions for performing a method or data generated by one of the described implementations. For example, the signal may be formatted to be transported as rules data for writing or reading the syntax of the described embodiment, or transported as data of actual syntax values written using the described embodiment. Such a signal may be formatted, for example, as an electromagnetic wave (eg, using the radio frequency portion of the spectrum) or as a baseband signal. The formatting may include, for example, encoding the data stream and carrier modulation with the encoded data stream. The information carried by the signal may be, for example, analog or digital data. As is known, a signal can be transmitted over a variety of different wired or wireless links. The signal may be stored on a processor-readable medium.

A number of implementations have been described above. However, it should be borne in mind that various modifications can be made. For example, elements from different implementations may be combined, added, modified, or removed to create other implementations. In addition, one skilled in the art will appreciate that other structures and processes may replace the disclosed ones, and the resulting implementations will perform at least substantially the same function(s) in at least substantially the same manner to achieve at least at least essentially the same result(s) as the disclosed implementations. Accordingly, these and other implementations are contemplated by this application.

Claims (19)

1. A method for generating points of a three-dimensional scene from a depth image, the method comprising the steps of: for a current pixel of said depth image, comparing depth information associated with said current pixel with depth information associated with pixels spatially adjacent to said current pixel in said depth image; when the depth difference between said current pixel and an adjacent pixel is greater than or equal to the first value and less than or equal to the second value, the second value indicating that the current pixel and the adjacent pixel refer to the same 3D scene object or to the same part of a 3D scene object, at least one additional point of said 3D scene is generated in addition to the current point associated with said current depth image pixel, the number of additional points depending on said difference. 2. The method according to claim 1, further comprising selecting an adjacent pixel having the largest depth difference with said current pixel among adjacent pixels having a depth difference with said current pixel lying between said first value and said second value, when the number of additional points generated depends on the largest depth difference mentioned. 3. The method according to claim 1, in which a volume unit is associated with a point of said 3D scene, while said depth difference corresponds to a certain number of volume units, while the number of generated points corresponds to said depth difference minus 1. 4. The method of claim 1, further comprising determining attributes to be associated with at least one additional point, said attributes being determined from attributes associated with the current point and said adjacent pixel. 5. The method of claim 4, wherein said attributes to be associated with at least one additional point contain depth information and/or texture information. 6. The method of claim 1, further comprising decoding said depth image from the received bitstream. 7. The method according to claim 1, wherein said points of the 3D scene are part of a point cloud. 8. A device for generating points of a three-dimensional scene from a depth image, and the device contains a memory associated with at least one processor configured to: for a current pixel of said depth image, comparing depth information associated with said current pixel with depth information associated with pixels spatially adjacent to said current pixel in said depth image; when the depth difference between said current pixel and an adjacent pixel is greater than or equal to the first value and less than or equal to the second value, the second value indicating that the current pixel and the adjacent pixel refer to the same 3D scene object or to the same part of a 3D scene object, generating at least one additional point of said 3D scene in addition to the current point associated with said current depth image pixel, wherein the number of additional points depends on said difference. 9. The apparatus of claim 8, wherein the at least one processor is further configured to select an adjacent pixel having the largest depth difference with said current pixel among adjacent pixels having a depth difference with said current pixel lying between said first value and said second value, with the number of additional points generated depending on said largest depth difference. 10. The device according to claim 8, in which a volume unit is associated with a point of said 3D scene, while said depth difference corresponds to a certain number of volume units, while the number of generated points corresponds to said depth difference minus 1. 11. The apparatus of claim 8, wherein the at least one processor is further configured to determine attributes to be associated with at least one additional point, said attributes being determined from attributes associated with the current point and said adjacent pixel. 12. The apparatus of claim 11, wherein said attributes to be associated with at least one additional point contain depth information and/or texture information. 13. The apparatus of claim 8, wherein the at least one processor is further configured to decode said depth image from the received bitstream. 14. The apparatus of claim 8, wherein said points of the 3D scene are part of a point cloud. 15. A processor-readable non-volatile storage medium that stores instructions that cause the processor to execute the method according to one of paragraphs. 1-7.

Images