JP2022150216A - Point group decryption device, point group decryption method, and program - Google Patents

Abstract

To provide a point group decryption device, a point group decryption method, and a program capable of improving an inter-prediction accuracy of a point group, and improving coding efficiency.SOLUTION: A point group decryption device 200 includes an up-sampling unit 25 which is configured to up-sample points of reference geometric information, and an occupation prediction unit 26 which is configured to predict occupancy information by using the up-sampled reference geometric information.SELECTED DRAWING: Figure 2

Description

The present invention relates to a point cloud decoding device, a point cloud decoding method, and a program.

Non-Patent Document 1 discloses a technique for encoding and decoding point cloud geometric information using inter-prediction. Such inter-prediction is composed of a plurality of elemental technologies.

Non-Patent Document 2 describes a technique for realizing motion compensation in units of nodes by associating small regions (nodes) divided by the Octree method between two time-series continuous point groups. is disclosed.

In Non-Patent Document 3, when dividing the associated node into eight child nodes, it is determined whether or not a point exists in each child node (occupancy information), and the previous point group in chronological order A technique is disclosed in which the occupancy information of a node (reference frame) is used to predict the occupancy information of a node in a current point group (current frame) in time series.

Prediction of such occupancy information is expressed by classification of no pred indicating unpredictable, pred0 indicating non-occupancy, pred1 indicating occupancy, and predL indicating strong occupancy.

For pred1 and predL, if the number of points present in the child node is less than a fixed threshold, it is classified as pred1, otherwise it is classified as predL.

Here, when the occupation information of the node of the current frame is arithmetically coded, the prediction is used to select the context of each bit, and the coding performance can be improved by selecting different occurrence probabilities for each context. can.

In Non-Patent Document 4, by estimating and aligning motion parameters (rotation, translation) of the entire point group between two time-series continuous point groups, encoding based on Non-Patent Documents 2 and 3 Techniques for improving the performance of are disclosed.

Exploratory model for inter-prediction in G-PCC, ISO/IEC JTC1/SC29/WG11 N18096 [PCC] On motion compensation for geometry coding in TM3, ISO/IEC JTC1/SC29/WG11 MPEG2018/m42521 [PCC] How to use a predictive set of points for geometry coding in TMC3, ISO/IEC JTC1/SC29/WG11 MPEG2018/m42520 [PCC] Global motion compensation for point cloud compensation in TM3, ISO/IEC JTC1/SC29/WG11 MPEG2018/m44751

However, the inter prediction in Non-Patent Documents 1 to 4 has a problem that the prediction accuracy is low and the coding efficiency is low when the density of the point cloud in the spatial direction is low and the correlation in the time direction is low.

Therefore, the present invention has been made in view of the above problems, and provides a point cloud decoding device, a point cloud decoding method, and a program that can improve the inter prediction accuracy of the point cloud and improve the encoding efficiency. intended to

A first feature of the present invention is a point cloud decoding device configured to decode a point cloud from a bitstream, comprising an upsampling unit configured to upsample points of reference geometric information; and an occupancy prediction unit configured to predict occupancy information using the upsampled reference geometric information.

A second aspect of the present invention is a point cloud decoding method adapted to decode a point cloud from a bitstream, comprising the steps of upsampling points of reference geometry information; and predicting occupancy information using the information.

A third feature of the present invention is a program that causes a computer to function as a point cloud decoding device configured to decode a point cloud from a bitstream, the point cloud decoding device comprising points of reference geometric information and an occupancy prediction unit configured to predict occupancy information using the upsampled reference geometric information.

Advantageous Effects of Invention According to the present invention, it is possible to provide a point cloud decoding device, a point cloud decoding method, and a program capable of improving the inter-prediction accuracy of the point cloud and improving the encoding efficiency.

FIG. 1 is a diagram showing an example of the configuration of a point cloud processing system 10 according to one embodiment. FIG. 2 is a diagram showing an example of functional blocks of the point cloud decoding device 200 according to one embodiment. FIG. 3A is a diagram showing an example of an outline of nodes for each hierarchy in one embodiment. FIG. 3B is a diagram illustrating an example of an outline of nodes for each hierarchy in one embodiment. FIG. 4 is a schematic diagram of upsampling to a location where the L1 distance is 1. In FIG. FIG. 5 is a flow chart showing an example of processing for generating a model in the point cloud decoding device 200 according to one embodiment. FIG. 6 is a flow chart showing an example of processing for generating a model in the point cloud decoding device 200 according to one embodiment. FIG. 7 is a schematic diagram of upsampling using a set of vectors obtained by maximum likelihood estimation. FIG. 8 is a diagram showing an example of using azimuth angles as attribute information.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings. It should be noted that constituent elements in the following embodiments can be appropriately replaced with existing constituent elements and the like, and various variations including combinations with other existing constituent elements are possible. Therefore, the following description of the embodiments is not intended to limit the scope of the invention described in the claims.

(First embodiment)
A point cloud processing system 10 according to a first embodiment of the present invention will be described below with reference to FIGS. 1 to 8. FIG. FIG. 1 is a diagram showing a point cloud processing system 10 according to an embodiment according to this embodiment.

As shown in FIG. 1 , the point cloud processing system 10 has a point cloud encoding device 100 and a point cloud decoding device 200 .

The point cloud encoding device 100 is configured to generate encoded data (bitstream) by encoding an input point cloud signal. The point cloud decoding device 200 is configured to generate an output point cloud signal by decoding the bitstream.

Note that the input point cloud signal and the output point cloud signal are composed of position information and attribute information of each point in the point cloud. The attribute information is, for example, color information and reflectance of each point.

Here, such a bitstream may be transmitted from the point cloud encoding device 100 to the point cloud decoding device 200 via a transmission line. Also, the bitstream may be stored in a storage medium and then provided from the point cloud encoding device 100 to the point cloud decoding device 200 .

(Point group decoding device 200)
The point group decoding device 200 according to this embodiment will be described below with reference to FIG. FIG. 2 is a diagram showing an example of functional blocks of the point group decoding device 200 according to this embodiment.

As shown in FIG. 2, the point group decoding device 200 includes a motion decoding unit 21, an arithmetic decoding unit 22, an octree decoding unit 23, a motion compensation unit 24, an upsampling unit 25, and an occupation prediction unit 26. , a probability acquisition unit 27 , and a model generation unit 28 .

Note that, in the present embodiment, the processing in the model generation unit 28 may be performed in advance before the decoding processing is performed.

Each functional block of the point cloud decoding device 200 according to this embodiment will be described below. Specific processing in the motion decoding unit 21, the arithmetic decoding unit 22, the octree decoding unit 23, the motion compensation unit 24, the occupancy prediction unit 26, and the probability acquisition unit 27 is described, for example, in Non-Patent Documents 1 to 4 above. The described method can be used.
The motion decoding unit 21 is arranged to decode encoded motion parameters contained in the input bitstream. Here, the motion parameters are expressed as rotation matrices and translation vectors that change the attitude of the entire point group or only part of the point group.

The arithmetic decoding unit 22 is configured to decode the octree data (bit string) using the input bitstream and the given data occurrence probabilities. Such data may be binary, 0 or 1, or otherwise. The probability of occurrence of such data is given from the probability acquisition unit 24 .

Since the octree decoding unit 23, which will be described later, performs processing on a node-by-node basis, the arithmetic decoding unit 22 performs processing once when eight bits representing occupation information of eight child nodes included in one node are decoded. is configured to stop

The arithmetic decoding unit 22 is configured to resume processing after the octree decoding unit 23 completes the processing for these eight bits.

In this manner, the arithmetic decoding unit 22 and the octree decoding unit 23 are configured to alternately perform processing until the entire input bitstream is processed.

In this embodiment, the octatree data may be replaced with known quadtree data or binary tree data in all or some of the nodes.

The octree decoding unit 23 is configured to decode geometric information from a bit string using a known decoding method based on the octree method.

That is, the octree decoding unit 23 is configured to generate a child node at a position where the bit is 1 when 8 bits representing occupation information of 8 child nodes included in a certain node are given. there is

The octree decoding unit 23 is configured to hierarchically perform such processing from the upper layer to the lower layer, and output the coordinates of the nodes in the lowest layer as geometric information representing a point group.

Since the octree method is a method of performing processing on a node-by-node basis, the octree decoding unit 23 once stops processing when the generation processing of eight child nodes included in the current node is completed. is configured to

The octree decoding unit 23 is configured to restart processing after decoding eight bits representing occupation information of eight child nodes of the node to be processed next by the arithmetic decoding unit 22 .

In this manner, the arithmetic decoding unit 22 and the octree decoding unit 23 are configured to alternately perform processing until the entire input bitstream is processed.

Here, since nodes are expressed as voxels that divide the three-dimensional space, it is possible to select reference geometric information corresponding to the region of the node currently being processed or the child node of the node. Such selection is a process of extracting only a portion included inside the region from the reference geometric information expressed as a point group.
In the present embodiment, the octree decoding unit 23 is configured to output to the upsampling unit 25 a region of voxels represented by eight child nodes of the node currently being processed.

In this embodiment, the octatree method may be replaced with a known quadtree method or binary tree method for all or some of the nodes.
The motion compensator 24 is configured to transform the orientation of reference geometric information expressed as a point group using motion parameters, and output the reference geometric information. Such pose transformation may be the entire point cloud or only a portion of the point cloud.

That is, the motion compensation unit 24 performs posture transformation using a common motion parameter for all points in the case of the entire point group, and performs posture transformation on some points in the case of only a part of the point group. In contrast, it is configured to perform posture transformation using individual motion parameters. The motion parameters are given from the motion decoder 21 .

The upsampling unit 25 is configured to upsample the reference geometric information expressed as a point group. Such reference geometric information may be one whose posture has been transformed by the motion compensator 24, or may be one which has not been posture-transformed by the motion compensator 24. FIG.

The upsampling unit 25 is configured to generate new points around each point of the reference geometric information using a given model. Such a model is given from the model generator 28 .

The upsampling unit 25 generates points by using or calculating such attribute information when the model requires attribute information of the points (for example, the distance from the LiDAR sensor), which is not shown.

In other words, the upsampling unit 25 is configured to perform upsampling using different models for each piece of attribute information. That is, the upsampling unit 25 is configured to change the point upsampling method according to the point attribute information.

Here, if the attribute information is assigned to each point of the reference geometry information in advance, it is used as it is, and if it is not assigned to each point of the reference geometry information in advance, it is newly calculated.

In this embodiment, the upsampling unit 25 is configured to select reference geometric information using voxel regions represented by eight child nodes of the node output from the octree decoding unit 23 .

The upsampling unit 25 is configured to upsample the selected reference geometric information.

However, such upsampling may be performed on the input geometric reference information or the geometric reference information output from the motion compensation unit 24 .

Also, the upsampling unit 25 is configured to receive syntax from the model generation unit 28 and determine information on whether or not to perform upsampling and the upsampling method based on the syntax. good too.

Further, the upsampling section 25 may be configured to distinguish between sparse points and dense points in a given point group and perform upsampling only on the sparse points.

Here, as a method of discriminating sparse points, for example, whether or not the distance from each point to the nearest point is equal to or greater than a threshold value, or whether or not the number of points located within the radius r[m] of each point is determined. Whether or not the number is equal to or less than a threshold, or whether or not the number of points located inside the voxels to which each point belongs may be used, or other methods may be used.

The occupation prediction unit 26 is configured to predict occupation information using the point cloud representing the reference geometric information upsampled by the upsampling unit 25 .

That is, the occupancy prediction unit 26 determines, for example, no pred (unpredictable), The classifications pred0 (unoccupied), pred1 (occupied) and predL (strongly occupied) are obtained. Such a voxel region is given from the octree decoding unit 23 .

The occupation prediction unit 26 is configured to output the prediction of occupation information of the eight child nodes to the probability acquisition unit 27 .

In this embodiment, the occupancy information is predicted based on the result of upsampling the reference geometric information included in the voxel region represented by the eight child nodes of the node output from the octree decoding unit by the upsampling unit. do.
However, the occupancy prediction unit 26 upsamples the input reference geometry information or the reference geometry information output from the motion compensation unit 24, and then the eight child nodes of the node output from the octree decoding unit 23 represent The occupation information may be predicted based on the result of selecting the reference geometric information included in the voxel region.

Also, in the octree method, it is expected that the number of points included inside a node increases in higher layers and decreases in lower layers. Therefore, the threshold th used for classifying pred1 and predL may be th=λ×n instead of a fixed value. Here, n is a hierarchical number in the octree method (the lowest layer is 1 and the number increases toward the higher layers), and λ is an arbitrary coefficient.

3A and 3B show an outline of nodes for each hierarchy. Although the nodes are actually represented by a three-dimensional grid, they are represented here by a two-dimensional grid to simplify the drawing.

FIG. 3A shows a schematic diagram of four nodes in the upper layer (n=4). A rectangle inside each node represents the minimum unit that can express a point, and a black rectangle means that a point exists. FIG. 3B shows a schematic diagram of four nodes in the lower layer (n=2).

Higher nodes may contain more points due to their larger voxel volume. On the other hand, since the nodes in the lower layer are obtained by dividing the nodes in the upper layer, the number of points is relatively small.

Therefore, when a fixed threshold value th is used, the higher the layer, the higher the possibility of being classified into predL, and the lower the layer, the higher the possibility of being classified into pred1. For example, when th=3, all upper-layer nodes shown in FIG. 3A are classified into predL, and all lower-layer nodes shown in FIG. 3B are classified into pred1.

On the other hand, for example, when λ = 1 and th = λ × n, among the upper and lower nodes, the upper left node shown in FIGS. 3A and 3B is classified as predL, and the other nodes are It is classified into pred1.

In this way, by varying the threshold according to the hierarchy, it is possible to obtain the effect of classifying the nodes in the same hierarchy according to the density of points.

Alternatively, any statistic may be used instead of the hierarchy number. For example, the average value m of the number of points in the occupied node in a certain layer may be counted, and the threshold used in the layer one level below may be set to th=λ×m. Alternatively, the statistic may be the median value of the number of points in the occupied node in a certain layer, or the cumulative moving average value from the top layer. Thereby, the threshold can be adaptively changed with respect to the density of the input point group.

The probability acquisition unit 27 is configured to acquire the probability of occurrence of data corresponding to the node decoded by the arithmetic decoding unit 22 using prediction of occupation information provided from the occupation prediction unit 26 and the like. Such probability of occurrence may be a fixed value or an adaptive variable value.

When acquiring the occurrence probability, the probability acquisition unit 27 obtains information such as occupation information of neighboring nodes of the parent node of the node decoded by the arithmetic decoding unit 22 and occupation information of eight child nodes having a common parent node. It may be configured to use other information.

The probability acquisition unit 27 is configured to output the acquired occurrence probability to the arithmetic decoding unit 22 .

Model generator 28 is configured to generate a model for upsampling the points. Model generator 28 may be configured to generate such models based on a systematic method.

For example, the model generation unit 28 may be configured to generate a model up-sampled to six locations where the L1 distance is 1 in the three-dimensional space with the position of the point as a reference, or It may be configured to generate an up-sampled model at the vertex position of the Geodesic dome when arranged, or may be configured to generate an up-sampled model by another method.

FIG. 4 shows a schematic diagram of upsampling to a location where the L1 distance is 1. In FIG. The points are actually represented in three dimensions, but are represented here in two dimensions to simplify the drawing.

In FIG. 4, the black squares represent one point in the reference frame and the black circles represent points upsampled from this point. In FIG. 4, the points are upsampled to four locations where the L1 distance is 1 on the two-dimensional grid.

The model generation unit 28 may also be configured to generate such a model based on maximum likelihood estimation using a point cloud for training. FIG. 5 shows a flow chart of processing in such a case.

As shown in FIG. 5, in step S101, when a training point group is input, in step S102, the motion compensation unit 24 converts a plurality of time-series continuous point groups measured by LiDAR into a training point group. , that is, a pair of the reference frame and the current frame, and pose transformation of the reference frame is performed.

In step S103, the model generating unit 28 associates each point of the reference frame with the point of the current frame by nearest neighbor search, and calculates a vector representing the positional error of the corresponding points.

In step S104, the model generator 28 may quantize such vectors in order to increase the appearance frequency.

For example, the model generation unit 28 may replace the directions and magnitudes of the vectors with representative values that are discretized so that they are spaced equally, or a group of vectors classified into k types by the k-means method. may be replaced with a representative value of , or another method may be used.

In step S105, the model generation unit 28 counts the number of times the same vector appears, and selects the top N vectors with the highest appearance frequency. The model generating unit 28 may generate a model for up-sampling points to points indicated by the vectors, using the positions of the points as references.

In step S106, the model generator 28 outputs the selected vector set.

FIG. 6 shows a flowchart of another case of the process of generating the model described above.

As shown in FIG. 6, in step S201, when a training point cloud and vector set are input, in step S202, the model generation unit 28 encodes the current frame using a reference frame without upsampling, Get the code amount.

In step S203, the model generating unit 28 repeats the processing of steps S204 to S209 until the number of elements in the vector set becomes zero.

In step S204, the model generator 28 repeats the operations of steps S205 to S209 for each of the remaining vectors in the vector set.

In step S205, the model generation unit 28 upsamples the reference frame using the selected vector set and one vector selected in step S204.

In step S206, the model generation unit 28 encodes the current frame using the upsampled reference frame and acquires the code amount.

In step S207, the model generation unit 28 determines whether or not the code amount obtained in step S206 is the lowest ever. If Yes, the process proceeds to step S208, and if No, the process proceeds to step S209.

In step S208, the model generation unit 28 adds one vector selected in step S204 to the selected vector set.

In step S209, the model generator 28 excludes one vector selected in step S204 from the vector set.

In step S210, the model generator 28 outputs the selected vector set.

When determining the number N of vectors, the model generation unit 28 may be configured to select N that maximizes the encoding efficiency when the training point group is actually encoded.

Alternatively, the model generation unit 28 may be configured to select the N vectors in order of increasing efficiency of encoding the training point group, instead of selecting the N vectors in descending order of appearance frequency.

That is, the model generating unit 28 is configured to examine the coding efficiency when all the vectors to be selected are included in the model, and then include one vector with the greatest improvement effect in the model. may be Then, the model generation unit 28 may be configured to exclude such vectors from the candidates and repeat the same process until the encoding efficiency is no longer improved. In this manner, model generator 28 may select a set of vectors to include in the model.

Alternatively, the model generation unit 28 first selects a set of N vectors with high frequency of appearance by the processing shown in FIG. The process (steps S204-S209) may be configured to select the set of vectors to include in the model in only N iterations.

FIG. 7 shows a schematic diagram of upsampling using a set of vectors obtained by maximum likelihood estimation. The points are actually represented in three dimensions, but here they are represented in two dimensions to simplify the drawing.
In FIG. 7, a black square represents a point in the reference frame and a black circle represents a point upsampled from that point.

Here, as an example of the obtained vector set, v1 = (1, 1), v2 = (2, -2), v3 = (-1, 2), v4 = (-2, 0), v5 = (- 1, -2). In FIG. 7, the points are upsampled relative to the location of the point in the reference frame to the locations pointed to by these vectors.

In addition, the model generation unit 28 may be configured to generate such a model individually for each point attribute information.

Such attribute information is arbitrary features of a point, such as the distance from the center, the elevation angle, and the azimuth angle when the coordinates of the point are expressed in a three-dimensional polar coordinate system. Alternatively, such attribute information may be an angle formed by a vector from a point to its two neighboring points, or may be the magnitude or direction of a vector from a point to the center of its two neighboring points. Alternatively, such attribute information is described in Non-Patent Document 5 (Stoyanov, Todor et al., "Fast and Accurate Scan
registration through minimization of the distance between compact 3D NDT representations. , The International Journal of
Robotics Research 31.12 (2012): 1377-1393), the variance-covariance matrix calculated using the points around the point is singular value decomposed, and the sphere or It may be classified as a plane or a line, or may be another feature.

The maximum likelihood estimation described above may be performed for each point group classified by such attribute information. That is, the model generator 28 may be configured to generate different models for each piece of attribute information.

At this time, the value of attribute information may be quantized in order to increase the frequency of matching attribute information between points in the reference frame and points in the current frame. For example, the distance from the center may be represented by bins spaced 10 meters apart, and points belonging to the same bin may be considered to have the same distance.

Moreover, when classifying according to attribute information, a plurality of pieces of attribute information may be combined to increase classification patterns.

FIG. 8 shows an example of using azimuth angles as attribute information. In the example of FIG. 8, the black circle represents the origin, and the triangles and diamonds each represent points in the reference frame.

Since the position of a point in the reference frame is generally represented by (x, y, z) coordinates relative to the origin in a three-dimensional Cartesian coordinate system, it can be converted to a three-dimensional polar coordinate system representation.

The example of FIG. 8 shows a case where the triangular points have an azimuth angle of 45° and the rhombic points have an azimuth angle of 150°. In this way, attribute information can be assigned to each point, and the points can be classified according to the attribute information.

Further, the model generation unit 28 may be configured to output information as to whether or not the upsampling unit 25 performs upsampling and the upsampling method as syntax.

Such a syntax could, for example, represent a rule where 0 represents no upsampling, 1 represents upsampling based on a regular method, and 2 upsampling based on maximum likelihood estimation. This is information representing a rule that

Also, the point cloud encoding device 100 and the point cloud decoding device 200 described above may be implemented as a program that causes a computer to execute each function (each process).

In the above-described embodiments, the present invention is applied to the point group encoding device 100 and the point group decoding device 200 as examples, but the present invention is not limited to such examples. A point cloud encoding/decoding system having the functions of the point cloud encoding device 100 and the point cloud decoding device 200 can be similarly applied.

In addition, according to this embodiment, for example, since it is possible to improve the overall service quality in video communication, the United Nations-led Sustainable Development Goals (SDGs) Goal 9 "Develop resilient infrastructure, It will be possible to contribute to the promotion of sustainable industrialization and the expansion of innovation.

10 Point cloud processing system 100 Point cloud encoding device 200 Point cloud decoding device 21 Motion decoding unit 22 Arithmetic decoding unit 23 Octtree decoding unit 24 Motion compensation unit 25 Upsampling unit 26 Occupancy prediction Part 27... Probability acquisition part 28... Model generation part

Claims (9)

A point cloud decoding device configured to decode a point cloud from a bitstream, comprising:
an upsampling unit configured to upsample points of the reference geometric information;
and an occupancy prediction unit configured to predict occupancy information using the upsampled reference geometric information. further comprising a model generator configured to generate a model representing the method of upsampling the points;
2. The point cloud decoding device according to claim 1, wherein the upsampling unit is configured to upsample the points using the model generated by the model generating unit. 3. The point cloud decoding device according to claim 1, wherein the upsampling unit is configured to upsample the points using a model generated based on a regular method. . 4. The method of any one of claims 1-3, wherein the upsampling unit is configured to upsample the points using a model generated based on maximum likelihood estimation. Point cloud decoder. 5. The point cloud decoding apparatus according to claim 4, wherein the maximum likelihood estimation estimates the positions and the number of points to be up-sampled so as to maximize coding efficiency in the training point cloud. The point group decoding device according to any one of claims 1 to 5, wherein the upsampling unit is configured to change an upsampling method of the points according to the attribute information of the points. . A point cloud decoding device configured to decode a point cloud from a bitstream, comprising:
A point cloud decoding device, comprising: an occupancy prediction unit configured to predict the occupancy information using a threshold expressed as a function of a layer number in the octree method. A point cloud decoding method configured to decode a point cloud from a bitstream, comprising:
Upsampling points of reference geometry information;
and predicting occupancy information using the upsampled reference geometric information. A program that causes a computer to function as a point cloud decoding device configured to decode a point cloud from a bitstream,
The point group decoding device is
an upsampling unit configured to upsample points of the reference geometric information;
an occupancy prediction unit configured to predict occupancy information using the upsampled reference geometric information.

Images