GB2575514A

GB2575514A - Method and system for compressing and decompressing digital three-dimensional point cloud data

Info

Publication number: GB2575514A
Application number: GB1811558.4A
Authority: GB
Inventors: Kaminski Marcin
Original assignee: Vividq Ltd
Current assignee: Vividq Ltd
Priority date: 2018-07-13
Filing date: 2018-07-13
Publication date: 2020-01-15
Anticipated expiration: 2038-07-13
Also published as: WO2020012187A1; GB2575514B; GB201811558D0

Abstract

Compressing (and/or decompressing) 3D image data (e.g. real world scene) comprising a point cloud (600), each point having one or more attributes, comprises encoding (620) the position data using an octree decomposition (610). Encoding the attribute data involves constructing (630) a relative neighbourhood graph (an undirected graph) for the points in each sub-octree partitioned from the octree, wherein the attributes of each point are signals on the graph. A combinatorial Laplacian matrix of each graph is calculated, as is an eigenvector decomposition for each matrix. A graph Fourier Transform of each attribute for each graph based on the corresponding eigenvector decomposition is then calculated (640), with a portion (e.g. at least 50%, 70% or 90%) of each graph Fourier Transform then being discarded. Each reduced graph Fourier Transform is compressed (650) (e.g. entropy encoded) to provide compressed point cloud attribute data, and a bit stream is generated corresponding to octree voxels occupancy. The bit stream is compressed (e.g. entropy encoded), providing (660) compressed point cloud position and attribute data. The octree may have a depth corresponding to a point cloud resolution level such that each voxel in the point cloud contains no more than one point.

Description

METHOD AND SYSTEM FOR COMPRESSING AND DECOMPRESSING DIGITAL THREE-DIMENSIONAL POINT CLOUD DATA

Field of the Invention

The invention relates to the field of digital image processing and in particular to the compression of three-dimensional image data. The invention is particularly suited to the compression of three-dimensional image data representing real-world objects or scenes. The invention has particular advantages for systems that render moving three-dimensional images, such as holograms, in real-time.

Background to the Invention

The capture and display of digital image data continues to become more sophisticated. Realtime video data capture, transmission and display is now commonplace, with services such as Skype and Facetime available on over mobile data networks. The rendering of highly realistic images in online computer games is also now widespread.

The generation or capture of three-dimensional images is also now becoming more widespread. Three-dimensional modelling of manufactured products, three-dimensional printing, holography, medical scanning, mapping, augmented reality and virtual reality are all areas in which three dimensional image data needs to be handled on digital computing systems.

The mathematics required to display three-dimensional images based on digital image data is well understood. But the amount of data that needs to be stored and transmitted can be massive, and the computational requirements for rendering three-dimensional images, such as holograms can be very high. This is particularly true if real-time rendering of moving threedimensional images is required. For that reason systems for displaying three dimensional images are still relatively expensive and specialist. And systems that can render good quality moving three-dimensional images in real time are not available.

Three-dimensional image data can be represented by three dimensional point clouds. A point cloud comprises a set of co-ordinates in three dimensional space, with one or more attributes associated with each point.

Various strategies have been used to compress point cloud data. One method that shows promise is the use of graph transforms to compress the attributes. It would be desirable to find a graph transform based approach that it is computationally inexpensive and well suited to real-world images.

Summary of the Invention

In a first aspect of the invention, there is provided a computer implemented method of compressing three dimensional image data, wherein the three dimensional image data comprises a point cloud, each point in the point cloud having one or more attributes, the method comprising:

a) creating an octree comprising a plurality of voxels, each of the points in the point cloud occupying a voxel in the octree;

b) partitioning the octree into a plurality of sub-octrees;

c) creating a relative neighbourhood graph for the points in each sub-octree, wherein the attributes of each point are signals on the relative neighbourhood graph;

d) calculating a combinatorial Laplacian matrix of each relative neighbourhood graph;

e) calculating an eigenvector decomposition for each of the combinatorial Laplacian matrices;

f) calculating a graph Fourier transform of each attribute for each relative neighbourhood graph based on the corresponding eigenvector decomposition;

g) discarding a portion of each graph Fourier transform to provide a plurality of reduced graph Fourier transforms;

h) compressing each reduced graph Fourier transform to provide compressed point cloud attribute data; and

i) generating a bit stream corresponding to the occupancy of the voxels in the octree and compressing the bit stream to provide compressed point cloud position data.

As used herein, a relative neighbourhood graph is an undirected graph defined on a set of points by connecting two points p and q by an edge whenever there does not exist a third point r that is closer to both p and q than they are to each other. Advantageously, the relative neighbourhood graph is unweighted. In other words, preferably there are no weighting factors associated with the edges of the graph.

The three dimensional image data may be of a real-world scene. The three dimensional image data may be representative of a real world scene or natural object but artificially generated, or it may be generated by capturing an image of a real world scene or object. A relative neighbourhood graph has been found to be particularly advantageous for real-world scenes as compared to point clouds representative of other data.

As used herein (graph) Fourier transform is intended to cover any equivalent spacefrequency transforms.

The step of compressing the bit stream preferably comprises using a lossless compression algorithm, such as an entropy encoding algorithm. For example, the LZMA algorithm may be used.

The step of compressing each reduced graph Fourier transform preferably comprises using a lossless compression algorithm, such as an entropy encoding algorithm. For example, the LZMA algorithm may be used.

Step (g) preferably comprises discarding at least 50% of each graph Fourier transform. Step (g) may comprise discarding between 50% and 95%, and preferably between 70% and 90% of each graph Fourier transform.

The octree may have a depth corresponding to a level of resolution of the point cloud such that each voxel in the point cloud contains no more than one point. Alternatively, the octree may have a depth such that some of the voxels contain more than one point. In that case, an average value of each of the attributes associated with the points in each voxel may be used as signals on the relative neighbourhood graph.

Each of the sub-octrees is preferably of equal size. However, it may be possible to use suboctrees of different size depending on the number of points within each sub-octree.

The method may further comprise the step of storing or transmitting the compressed point cloud attribute data and compressed point cloud position data together.

The method may comprise a step of capturing image data and a step of generating the point cloud from the image data. The step of capturing image data may be performed using one or more sensors, such as cameras or scanners.

In a second aspect of the invention, there is provided a computer implemented method of decompressing compressed three dimensional image data, the compressed three dimensional image data comprising compressed point cloud attribute data and compressed point cloud position data, comprising:

a) decompressing the point cloud position data to generate a point cloud;

b) creating an octree comprising a plurality of voxels, each of the points in the point cloud occupying a voxel in the octree;

c) partitioning the octree into a plurality of sub-octrees;

d) creating a relative neighbourhood graph for the points in each sub-octree

e) calculating a combinatorial Laplacian matrix of each relative neighbourhood graph;

f) calculating an eigenvector decomposition for each of the combinatorial Laplacian matrices;

g) decompressing the point cloud attribute data to provide a plurality of graph Fourier transforms;

h) applying an inverse transform to each of the graph Fourier transforms and a corresponding eigenvector decomposition to provide a plurality of point cloud attribute vectors; and

i) reconstructing the three dimensional image data from the point cloud and the plurality of point cloud attribute vectors.

The method may further include the step of displaying the three dimensional image data. The step of displaying may comprise generating a hologram.

In a third aspect of the invention, there is provided a method of displaying three dimensional image data comprising:

receiving three dimensional image data comprising a point cloud, each point in the point cloud having one or more attributes, compressing the three dimensional image data using the method of the first aspect of the invention, transmitting the compressed point cloud attribute data and compressed point cloud position data together to an image display device, decompressing the compressed point cloud attribute data and compressed point cloud position data to reconstruct the three dimensional image data using the method of the second aspect of the invention, and displaying a three dimensional image using the image display device based on the three dimensional image data.

The method may comprise a step of capturing image data and a step of generating the point cloud from the image data. The step of capturing image data may be performed using one or more sensors, such as cameras.

The image display system may be a holographic image display system and the step of displaying may comprise generating a hologram from the three dimensional image data.

In a fourth aspect of the invention, there is provided a computer system comprising:

an input buffer configured to receive three dimensional image data comprising a point cloud, each point in the point cloud having one or more attributes;

an image encoder configured to compress the three dimensional image data by performing the steps of the first aspect of the invention, to produce compressed three dimensional image data comprising compressed point cloud attribute data and compressed point cloud position data; and an output buffer configured to store the compressed three dimensional image data.

The computer system may further comprise an image decoder configured to decompress the compressed three dimensional image data by performing the steps of the second aspect of the invention.

The computer system may further comprise an image display device configured to display an image based on the decompressed three dimensional image data. The image display device may be a holographic image display device.

The computer system may be configured to display a moving three dimensional image. The computer system may be configured to display a succession of static three-dimensional images, each static three dimensional image corresponding to a point cloud, to generate a moving three dimensional image.

In a fifth aspect of the invention, there is provided one or more computer readable storage media storing computer executable instructions for causing a computer, when programmed thereby, to perform the method of the first aspect of the invention or the second aspect of the invention.

Brief Description of the Drawings

Embodiments of the invention will now be described, by way of example only, with reference to the accompanying drawings, in which:

Figure 1 is a schematic illustration of a system handling three-dimensional image data;

Figure 2 is a schematic illustration of the components of a computer system configured to carry out image compression in accordance with the invention;

Figure 3 is an illustration of a point cloud and an associated octree;

Figure 4 illustrates how a relative neighbourhood graph is constructed;

Figure 5 illustrates a two-dimensional relative neighbourhood graph and its associated Laplacian matrix;

Figure 6 is a schematic illustration of the functional components of an encoder according to the invention; and

Figure 7 is a schematic illustration of the functional components of an decoder according to the invention.

Detailed Description

Figure 1 is a schematic illustration of a computer network system that handles three dimensional image data. The system comprises an image capture system 10, which typically comprises one or more cameras or a scanner. Instead of, or in addition to, an image capture system, there may be an image generating system, such as three-dimensional modelling software running on a computer. The image capture system generates three dimensional image data in the form of a point cloud.

The system includes an compression system 12 configured to compress the point cloud. The compression system receives the point cloud from the image capture system. A data store 14 is connected to the image capture system and to the compression system and can store the point cloud in a compressed or an uncompressed format. A transmitter 16 receives compressed point cloud data from the compression system or from the data store and transmits the compressed point cloud data over a data communication network to a receiver 18. Connected to the receiver 18 is a decompression system 20, which is configured to decompress the point cloud data. The decompression system sends the decompressed point cloud data to an image rendering system 22, such as holographic projector or a 3D printer. A receiver side data store 24 may be used to store the point cloud data in compressed or uncompressed form.

A system of the type shown in Figure 1 may be used, for example, to send moving three dimensional image data to a holographic projector in real time, to allow for a holographic video call. In such a system, the ability to effectively compress and decompress the point cloud data quickly is paramount.

Figure 2 illustrates the basic components of the compression system of Figure 1. The compression system comprises an input buffer that receives the point cloud data. Connected to the input buffer is the encoder 122 that performs the compression process of the point cloud data. An output buffer 124 receives the compressed point cloud data from the encoder 122 before passing it to a transmitter or to a data store as a compressed data bit stream.

The point cloud comprises a set of co-ordinates in a three dimensional space, with one or more attributes associated with each co-ordinate. For example, a point may have x, y and z co-ordinates together with attributes such as colour, transparency and surface normal vector. The amount of data contained in a single point cloud representing an image of a real world scene or object may be massive. However, there may also be large regions of the three dimensional space that are empty of data points.

In order to compress the point cloud, the three dimensional space is divided using an octree decomposition. This is illustrated in Figure 3. In an octree decomposition of a cube shaped space 30, the space is first divided into eight identical cubes 31. Each of the identical cubes is further divided in eight smaller cubes 32. Each smaller cube is divided into eight, and so on until the space is divided into cubes of a desired size or resolution. The smallest divisions of the octree can be referred to as voxels. The number of divisions of the space is referred to the number of levels, or the depth, of the octree. Each point in the point cloud is contained in one of the cubes of the octree decomposition. The occupancy of octree by the points of the point cloud is defined by a sequence of bits corresponding to the occupancy in each level of the octree. This is also illustrated in Figure 3. There are three points shown in the point cloud of Figure 3. The encoder chooses a particular order for the cubes (or octants), which is the same for each node of the octree. The encoder writes a node signature for each level of the octree, which is a sequence of eight bits, indicating the presence or absence of a point within that level. In Figure 3, all three points are in the same block (block 64) in the first level of the octree. In the second level of the octree, there are two occupied blocks (blocks 2 and 3). In the third level of the octree there are three occupied blocks. The resulting bit stream for the position data of the point cloud in the example of Figure 3 is 00010000 011000000 01000000 11000000. This bit stream sequence is subsequently compressed with a lossless data compression algorithm, which in this example is an entropy encoding algorithm, LZMA.

In this embodiment, the depth of the octree is chosen to correspond to the resolution of the image data so that only one point is present in any one voxel of the octree. However, it is possible, if lower resolution is required for any reason, to have more than one point in a voxel. In that case average values for the attributes of the points in each voxel are used.

The position data for the point cloud is represented using an octree decomposition as described. But the attribute data cannot readily be compressed in this format. Instead, the attribute data is encoded as values on the nodes in a relative neighbourhood graph. Figure 4 illustrates how a relative neighbour graph is constructed from a set of points. For ease of illustration, a two dimensional graph is illustrated but the principles can be extended to three or more dimensions. A relative neighbourhood graph is an undirected graph defined on a set of points by connecting two points p and q by an edge whenever there does not exist a third point r that is closer to both p and q than they are to each other. In the example shown in Figure 4, there are four points p, q, s and t. Points s and t are adjacent in the graph because the there is no point closer to both of them than they are to each other (the lune area is empty). So points s and t are connected by an edge. But points s and p are not adjacent in the graph because there is another point q in the lune area. So points s and p are not connected by an edge.

Figure 5 illustrates a relative neighbourhood graph of 8 points and its associated Laplacian matrix. Again a two dimensional graph is shown for ease of illustration. The Laplacian L is related to the adjacency matrix A of a graph and the degree matrix D, by L=D-A. The adjacency matrix A is simply a symmetric matrix of the connections or edges between points. The edges are not given weights. So when there is an edge it is always represent by a value of 1 in the adjacency matrix. The degree matrix D is a diagonal matrix where the element on position j is the number of edges connected to the j^th point. In the example of Figure 5, point 1 has two connections, to point 2 and to point 7. Similarly points 3 to 6 each have two connections. Point 7 has three connections. Point 8 has only a single connection. The

Laplacian representation of the graph is a symmetrical matrix and is shown in Figure 5. The Laplacian is a real symmetric matrix and so has a set of orthogonal eigenvectors and real non-negative eigenvalues.

To compress the attributes, each attribute is considered to be a signal on the relative neighbourhood graph of the point cloud. The attributes are encoded by graph signal processing, and in particular by using a graph Fourier transform.

In order to create relative neighbourhood graphs of manageable size, the encoder divides the point cloud P’ into s smaller point clouds. It is preferable that this number s equals (k/t)^A3, where k is the depth of the octree and t is an integer that divides k. This way the octree structure can be used to efficiently partition P’ into smaller point clouds P’i, P’₂. ..., P’s. A graph is created for each of the smaller point clouds. The same method applies to all of P’i, P’2. ..., P’s and the graph creation can be done in parallel.

Each relative neighbourhood graph can be constructed using an algorithm described in Agarwal, Pankaj K.; Matausek, Jiri (1992), Relative neighborhood graphs in three dimensions, Proc. 3rd ACM-SIAM Symp. Discrete Algorithms, pp. 58-65. It is desirable that the number of points in each graph is relatively small (around 40-60) and this can be achieved by choosing an appropriate value for t.

Given a graph G with n vertices, the encoder computes its combinatorial Laplacian L and its eigenvector decomposition together with the corresponding eigenvalues. That is, it finds a matrix F such that L=FAF^T, where F is a square matrix of size η x n containing the eigenvectors of L as columns and A is another square η x n matrix which contains the eigenvalues of L on its diagonal sorted in non-decreasing order. There a number of standard algorithms that can be implemented in the encoder to compute this decomposition.

A signal on the graph G is an n-dimensional vector. If x is a signal on G, then the graph Fourier transform of x is defined as x’=F^Tx. The inverse graph Fourier transform is defined as x=Fx’. If one of the attributes of the points in the point cloud is denoted as a, then the encoder computes the graph Fourier transform a’ of a. The encoder computes the graph Fourier transform for each of the attributes on each of the graphs. The encoder processes each graph G’ from a list Gi, G₂. ..., G_s separately and in parallel.

If a’jj denotes the graph Fourier transform of attribute i on graph Gj, to encode the attributes, the encoder writes each of a’i,jS (in some fixed order) but keeps only a percentage of elements from each vector a’jj. The elements retained are those corresponding to the smallest eigenvalues. In this example, only 20% of the elements from each vector a’jj are retained. This massively reduces the amount of data that is compressed and provides an effective way of cutting out the information that has the lowest impact on perceived image quality.

The retained sequence of attributes will be subsequently compressed with a lossless data compression algorithm. An entropy encoding algorithm, such as LZMA, may be used.

Figure 6 illustrates the functional elements in the encoder described. The point cloud data is received at the input to the encoder. Block 610 is the octree builder, which constructs the octree decomposition for the point cloud. The depth of the octree is provided as an input to the octree builder 610. The position data of the point cloud is output from the octree builder 610 to entropy encoding block 620. Entropy encoding block 620 compressed the position data bit stream and provides it to output 660.

The attribute data is passed from octree builder 610 to the graph builder 630. The graph builder generates a plurality of graphs corresponding to portions of the point cloud, taking into account the depth of the octree as described. Each of the generated graphs undergoes the graph Fourier transform process described to produce the sequence of transformed attributes a’jj in transform block 640. As described 80% of the transformed attributes, corresponding to the largest eigenvalues, are discarded. The remaining bit stream is entropy encoded by encoding block 650 before being passed to the output 660. So the position data and the attribute data are encoded separately and in parallel before being output together.

In order to decompress the compressed data from the encoder 12, the decoder 20 carries out a corresponding sequence of operations. The compressed data is received at input 700. The position data and attribute data are then separated. The position data is entropy decoded in block 710 and the octree constructed in octree builder 720. The position data is then passed to the point cloud reconstruction block 760.

The attribute data is entropy decoded in block 730 to provide the transformed attribute vectors. In order to inverse transform the attribute vectors the relative neighbourhood graphs G1, G2. ..., G_s are reconstructed from the position data using graph builder 740, corresponding to graph builder 630 in Figure 6. The Laplacian matrix for each graph is constructed and its eigenvectors and eigenvalues, and then the matrix F calculated for each graph. In block 750, the attribute vectors a for each graph are calculated using the matrix F and the transformed attribute vectors.

The attributes are then combined with the position data in point cloud reconstruction block 760 to produce a reconstructed point cloud. The reconstructed point cloud is then passed to the output 780. The reconstructed point cloud will have a point at the mid-point of every occupied voxel. There is some loss of attribute data as a result of the compression method and some loss of position data because of octree method.

Both the encoder and decoder can be implemented as software running on a microprocessor. The encoder and decode can be implemented on multiple processing cores, running in 10 parallel. For example, the encode and decoder may be implemented as software running on multiple GPUs in parallel.

Claims

1. A computer implemented method of compressing three dimensional image data, wherein the three dimensional image data comprises a point cloud, each point in the point cloud having one or more attributes, the method comprising:

b) partitioning the octree into a plurality of sub-octrees;

c) creating a relative neighbourhood graph for the points in each suboctree, wherein the attributes of each point are signals on the relative neighbourhood graph;

2. A computer implemented method according to claim 1, wherein the step of compressing the bit stream comprises using a lossless compression algorithm, such as an entropy encoding algorithm.

3. A computer implemented method according to claim 1 or 2, wherein the step of compressing each reduced graph Fourier transform comprises using a lossless compression algorithm, such as an entropy encoding algorithm.

4. A computer implemented method according to any one of the preceding claims, wherein step (g) comprises discarding at least 50% and more preferably between 70% and 90% of each graph Fourier transform.

5. A computer implemented method according to any one of the preceding claims, wherein the octree has a depth corresponding to a level of resolution of the point cloud such that each voxel in the point cloud contains no more than one point.

6. A computer implemented method according to any one of the preceding claims, wherein the three dimensional image data is of a real-world scene.

7. A computer implemented method according to any one of the preceding claims, further comprising the step of storing or transmitting the compressed point cloud attribute data and compressed point cloud position data together.

8. A computer implemented method of decompressing compressed three dimensional image data, the compressed three dimensional image data comprising compressed point cloud attribute data and compressed point cloud position data, comprising:

a) decompressing the point cloud position data to generate a point cloud;

c) partitioning the octree into a plurality of sub-octrees;

d) creating a relative neighbourhood graph for the points in each sub-octree

9. A method of displaying three dimensional image data comprising:

receiving three dimensional image data comprising a point cloud, each point in the point cloud having one or more attributes, compressing the three dimensional image data using the method of any one of claims 1 to 7, transmitting the compressed point cloud attribute data and compressed point cloud position data together to an image display device, decompressing the compressed point cloud attribute data and compressed point cloud position data to reconstruct the three dimensional image data using the method of claim 8, and displaying a three dimensional image using the image display device based on the three dimensional image data.

10. A method according to claim 9, wherein the image display system is a holographic image display system and the step of displaying comprises generating a hologram from the three dimensional image data.

11. A computer system comprising:

an image encoder configured to compress the three dimensional image data by performing the steps of any one of claims 1 to 7, to produce compressed three dimensional image data comprising compressed point cloud attribute data and compressed point cloud position data; and an output buffer configured to store the compressed three dimensional image data.

12. A computer system according to claim 11, further comprising an image display device configured to display an image based on the three dimension image data.

13. A computer system according to claim 12, wherein the image display device is a holographic image display device.

14. One or more computer readable storage media storing computer executable instructions for causing a computer, when programmed thereby, to perform the method of any one of claims 1 to 9.