CN111899162A - Point cloud data processing method and system based on segmentation - Google Patents
Point cloud data processing method and system based on segmentation Download PDFInfo
- Publication number
- CN111899162A CN111899162A CN201910372764.6A CN201910372764A CN111899162A CN 111899162 A CN111899162 A CN 111899162A CN 201910372764 A CN201910372764 A CN 201910372764A CN 111899162 A CN111899162 A CN 111899162A
- Authority
- CN
- China
- Prior art keywords
- point cloud
- cloud data
- segmentation
- projection
- data processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000011218 segmentation Effects 0.000 title claims abstract description 123
- 238000003672 processing method Methods 0.000 title claims abstract description 25
- 238000000034 method Methods 0.000 claims abstract description 56
- 230000006835 compression Effects 0.000 claims abstract description 47
- 238000007906 compression Methods 0.000 claims abstract description 47
- 239000012634 fragment Substances 0.000 claims abstract description 43
- 230000009467 reduction Effects 0.000 claims abstract description 8
- 238000012545 processing Methods 0.000 claims description 25
- 239000013598 vector Substances 0.000 claims description 24
- 238000007405 data analysis Methods 0.000 claims description 13
- 238000013135 deep learning Methods 0.000 claims description 8
- 238000013178 mathematical model Methods 0.000 claims description 7
- 239000011159 matrix material Substances 0.000 claims description 7
- 238000000926 separation method Methods 0.000 claims description 7
- 101000969688 Homo sapiens Macrophage-expressed gene 1 protein Proteins 0.000 claims description 4
- 102100021285 Macrophage-expressed gene 1 protein Human genes 0.000 claims description 4
- 238000009826 distribution Methods 0.000 claims description 4
- 238000005457 optimization Methods 0.000 claims description 4
- 230000009466 transformation Effects 0.000 claims description 4
- 239000000126 substance Substances 0.000 claims description 3
- 238000000354 decomposition reaction Methods 0.000 claims description 2
- 238000000513 principal component analysis Methods 0.000 claims 1
- 230000008569 process Effects 0.000 abstract description 3
- 238000007781 pre-processing Methods 0.000 abstract description 2
- 238000010586 diagram Methods 0.000 description 8
- 238000012847 principal component analysis method Methods 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 239000006185 dispersion Substances 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000003860 storage Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/12—Edge-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T9/00—Image coding
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
Abstract
The invention provides a point cloud data processing method and system based on segmentation, which comprises the following steps: dividing the point cloud data according to the geometric and attribute characteristics of the point cloud data to obtain a plurality of sub-point clouds; respectively projecting the sub-point clouds; and splicing the projected two-dimensional fragments to obtain a two-dimensional image or a video frame. According to the method, the original overall point cloud data is firstly segmented according to geometric and attribute characteristics to obtain different sub-point clouds, different projection schemes are designed aiming at the different sub-point clouds, and the projection efficiency of each part of points is maximized. In addition, a splicing process of the image after the projection of the sub-point cloud is designed, and the compression efficiency of the whole point cloud is further fully improved. The whole scheme carries out preprocessing in modes of point cloud data segmentation, dimension reduction, splicing and the like, and compression coding is carried out by utilizing the existing coding tool, so that not only is the compression performance improved, but also the efficiency can be greatly improved.
Description
Technical Field
The invention relates to the field of point cloud data processing, in particular to a point cloud data processing method and system based on segmentation.
Background
In recent decades, three-dimensional scanning technology and systems have become mature, the manufacturing cost of 3D scanners is reduced, the precision is higher and higher, the applications are wider and wider, and three-dimensional coordinate information of the surfaces of actual objects can be rapidly and accurately acquired and stored, so that point cloud data can be widely applied to the related fields of image processing in scientific research and industry.
The point cloud data is data information of three-dimensional coordinates of the object after three-dimensional scanning, and information such as RGB (red, green, blue), depth and the like can be recorded. With the improvement of the precision and the speed of a three-dimensional scanning system, the amount of scanned point cloud data reaches the order of magnitude of millions or even more, and at present, the massive point cloud data increases heavy burden for computer storage, processing and transmission.
Point cloud compression algorithms have been studied more systematically. Most of the static point cloud compression methods are realized based on octree space decomposition. And decomposing the three-dimensional space in which the point clouds are positioned by using an octree structure, wherein each node represents a cube of a specific area in the space. The approximated point space coordinates can be calculated from the octree structure and the corresponding node information. Therefore, static point cloud compression can be realized by carrying out serialization coding on the octree structure. While a typical compression scheme for dynamic point clouds is to convert 3D point cloud data into 2D pictures for processing by projection mapping. This arrangement allows the point cloud to be compressed and transmitted by conventional processing means. However, when performing projection, due to the irregularity and density of the point cloud, there usually occurs a problem that projected points are overlapped on a two-dimensional plane to cause data loss, that is, a so-called occlusion problem. In addition, most of the existing point cloud projection schemes perform unified projection based on the original point cloud, and do not consider the characteristic differences of the color, the reflection coefficient and the like of each part, and also do not consider that different parts of the point cloud are suitable for different projection modes. How to design reasonable projection and post-processing modes aiming at the problem of the shielding of the point cloud and the characteristic difference of each part, and improve the coding efficiency of each part of the original point cloud as pertinently as possible so as to realize the overall high-efficiency compression of the point cloud is a key problem to be solved urgently.
Disclosure of Invention
Aiming at the defects in the prior art, the invention aims to provide a point cloud data processing method and system based on segmentation.
The invention provides a point cloud data processing method based on segmentation, which comprises the following steps:
dividing the point cloud data according to the geometric and attribute characteristics of the point cloud data to obtain a plurality of sub-point clouds;
respectively projecting the sub-point clouds; and
and splicing the projected two-dimensional fragments to obtain a two-dimensional image or a video frame.
Preferably, still further comprising:
before the point cloud data is segmented, data analysis of the point cloud data is included.
Preferably, the data analysis of the point cloud data adopts a principal component analysis method.
Preferably, still further comprising:
the resulting two-bit image or video frame is further compression encoded.
Preferably, the compression encoding comprises:
JPEG/PNG is adopted for image compression;
the video compression adopts HEVC or MPEG1/MPEG2/MPEG 4/H264.
Preferably, the segmentation method for segmenting the point cloud data includes any one or more of the following:
an edge-based segmentation algorithm, a region-growth-based segmentation algorithm, an attribute-based segmentation algorithm, a model-based segmentation algorithm, a graph-based segmentation algorithm, a deep learning-based segmentation algorithm.
Preferably, the model-based segmentation algorithm:
and segmenting by using a mathematical model of an original geometric form as prior knowledge, and classifying point cloud data with the same mathematical expression into the same region, wherein the original geometric form comprises a plane, a cylinder, a cone or a sphere.
Preferably, the point cloud data is segmented according to geometric and attribute features of the point cloud data:
the geometric and attribute features include any one or more of:
geometric features, textural features, normal vector features, or reflectance features.
Preferably, the method further comprises the following steps:
and correspondingly setting projection modes for projection according to the geometric and attribute characteristics of the point cloud data, and respectively projecting to obtain different two-dimensional fragments.
Preferably, the projection mode is to project a projection line to a selected projection plane by passing the projection line through a point or other object, and obtain a pattern on the plane, and includes any one of the following:
a hexahedral projection scheme, an N-hedron projection scheme, a cylindrical projection scheme, and a conical projection scheme.
Preferably, the hexahedral projection scheme includes:
analyzing the sub-point cloud data, and judging a projection axis which furthest retains the original distribution characteristics; assume a sample point of xiAnd assuming that the new coordinate system after projective transformation is W ═ W1,w2,w3In which w1,w2,w3For the basis vector of the new coordinate system, x can be obtainediThe projection in the hyperplane is WTxiThe optimization target is as follows:
s.tWTW＝I
using the lagrange multiplier method for the above equation, one can obtain:
XXTW＝λW
wherein the content of the first and second substances,
i represents a unit array;
lambda represents a matrix of undetermined eigenvalues;
X＝(x1,x2,…,xn) The method comprises the steps of (1) a point cloud data point set, wherein n is the total number of points contained in the point cloud data, and i is 1, 2, …, n;
for covariance matrix XXTDecomposing the eigenvalues, wherein the obtained maximum eigenvectors of the first eigenvalues are the standard orthogonal basis vectors of the space where the sample is located after dimensionality reduction;
with the corresponding feature vector w1,w2,w3And establishing a new space rectangular coordinate system for the X, Y, Z axes to determine the optimal projection plane.
Preferably, the method further comprises the following steps:
splicing according to the spatial correlation between the two-dimensional fragments;
the spatial correlation includes the separation distance in the original point cloud data and the similarity degree between the attribute features.
Preferably, the method further comprises the following steps:
when the two-dimensional fragments are spliced, the splicing adjusting factors comprise any one or more of the following factors:
the compactness factor of the spliced image and the connection degree factor among the two-dimensional fragments.
Preferably, the method further comprises the following steps:
when the splicing mode is selected, the reference for selecting the splicing mode comprises any one or more of the following: the size of the whole image/video frame, the number and proportion of effective pixels, the spacing distance of each two-dimensional fragment and the like.
The invention provides a point cloud data processing system based on segmentation, which comprises:
the segmentation module is used for segmenting the point cloud data according to the geometric and attribute characteristics of the point cloud data to obtain a plurality of sub-point clouds;
the projection module is used for projecting the sub-point clouds respectively; and
and the splicing module is used for splicing the projected two-dimensional fragments to obtain a two-dimensional image or a video frame.
Preferably, still further comprising:
and the data analysis module is used for carrying out data analysis on the point cloud data before the point cloud data is segmented.
Preferably, still further comprising:
and the compression coding module is used for further performing compression coding on the obtained two-bit image or video frame.
Preferably, the segmentation module:
the segmentation method for segmenting the point cloud data comprises any one or more of the following methods:
an edge-based segmentation algorithm, a region-growth-based segmentation algorithm, an attribute-based segmentation algorithm, a model-based segmentation algorithm, a graph-based segmentation algorithm, a deep learning-based segmentation algorithm.
Preferably, the model-based segmentation algorithm:
and segmenting by using a mathematical model of an original geometric form as prior knowledge, and classifying point cloud data with the same mathematical expression into the same region, wherein the original geometric form comprises a plane, a cylinder, a cone or a sphere.
Preferably, the point cloud data is segmented according to geometric and attribute features of the point cloud data:
the geometric and attribute features include any one or more of:
geometric features, textural features, normal vector features, or reflectance features.
Preferably, the method further comprises the following steps:
and correspondingly setting projection modes for projection according to the geometric and attribute characteristics of the point cloud data, and respectively projecting to obtain different two-dimensional fragments.
Preferably, the projection mode is to project a projection line to a selected projection plane by passing the projection line through a point or other object, and obtain a pattern on the plane, and includes any one of the following:
a hexahedral projection scheme; an N-face projection scheme; a cylindrical projection scheme; and a cone projection scheme.
Preferably, the method further comprises the following steps:
splicing according to the spatial correlation between the two-dimensional fragments;
the spatial correlation includes the separation distance in the original point cloud data and the similarity degree between the attribute features.
Preferably, the method further comprises the following steps:
when the two-dimensional fragments are spliced, the splicing adjusting factors comprise any one or more of the following factors:
compactness of the spliced image and a connection degree factor among the two-dimensional fragments.
Preferably, the method further comprises the following steps:
when the splicing mode is selected, the reference for selecting the splicing mode comprises any one or more of the following: the size of the whole image/video frame, the number of effective pixels, the proportion of the effective pixels, the spacing distance of each two-dimensional fragment and the like.
Compared with the prior art, the invention has the following beneficial effects:
according to the method, the original overall point cloud data is firstly segmented according to geometric and attribute characteristics to obtain different sub-point clouds, different projection schemes are designed aiming at the different sub-point clouds, and the projection efficiency of each part of points is maximized. In addition, a splicing process of the image after the projection of the sub-point cloud is designed, and the compression efficiency of the whole point cloud is further fully improved. The whole scheme carries out preprocessing in modes of point cloud data segmentation, dimension reduction, splicing and the like, and compression coding is carried out by utilizing the existing coding tool, so that not only is the compression performance improved, but also the efficiency can be greatly improved. In other words, the point cloud data of the three-dimensional space is firstly segmented, the original complete point cloud is decomposed into different parts, corresponding projection is carried out based on the segmented result, and a corresponding splicing scheme is designed, so that the projection compression method for converting the three-dimensional point cloud into the two-dimensional picture/video suitable for coding is realized.
Drawings
Other features, objects and advantages of the invention will become more apparent upon reading of the detailed description of non-limiting embodiments with reference to the following drawings:
FIG. 1 is a schematic diagram of an undivided raw point cloud in an embodiment of the invention;
FIG. 2 is a schematic diagram of point cloud segmentation according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a segmented sub-point cloud according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a projection flow under a hexahedral projection based scheme in an embodiment of the present invention;
FIG. 5 is a schematic diagram of original point cloud data and its cylindrical projection surface under a cylindrical projection-based scheme according to an embodiment of the present invention;
FIG. 6 is a schematic spatial diagram of a point cloud projected onto a cylindrical surface according to an embodiment of the present invention;
FIG. 7 is a schematic diagram of a two-dimensional image after being expanded for a projection plane of a cylinder under a scheme based on projection of the cylinder in the embodiment of the present invention;
FIG. 8 is a flow chart of a segmentation-based point cloud data processing method according to an embodiment of the present invention;
FIG. 9 is a block diagram of a segmentation-based point cloud data processing system according to an embodiment of the present invention.
Detailed Description
The present invention will be described in detail with reference to specific examples. The following examples will assist those skilled in the art in further understanding the invention, but are not intended to limit the invention in any way. It should be noted that it would be obvious to those skilled in the art that various changes and modifications can be made without departing from the spirit of the invention. All falling within the scope of the present invention.
The invention provides a point cloud data processing method based on segmentation, which comprises the following steps:
dividing the point cloud data according to the geometric and attribute characteristics of the point cloud data to obtain a plurality of sub-point clouds;
respectively projecting the sub-point clouds; and
and splicing the projected two-dimensional fragments to obtain a two-dimensional image or a video frame.
Specifically, the method further comprises the following steps:
before the point cloud data is segmented, data analysis of the point cloud data is included.
Specifically, the data analysis of the point cloud data adopts a principal component analysis method.
Specifically, the method further comprises the following steps:
the resulting two-bit image or video frame is further compression encoded.
Specifically, the compression encoding includes:
JPEG/PNG is adopted for image compression;
the video compression adopts HEVC or MPEG1/MPEG2/MPEG 4/H264.
Specifically, the segmentation method for segmenting the point cloud data includes any one or more of the following:
an edge-based segmentation algorithm, a region-growth-based segmentation algorithm, an attribute-based segmentation algorithm, a model-based segmentation algorithm, a graph-based segmentation algorithm, a deep learning-based segmentation algorithm.
Specifically, the model-based segmentation algorithm:
and segmenting by using a mathematical model of an original geometric form as prior knowledge, and classifying point cloud data with the same mathematical expression into the same region, wherein the original geometric form comprises a plane, a cylinder, a cone or a sphere.
Specifically, the point cloud data is segmented according to the geometric and attribute features of the point cloud data:
the geometric and attribute features include any one or more of:
geometric features, textural features, normal vector features, or reflectance features.
Specifically, the method further comprises the following steps:
and correspondingly setting projection modes for projection according to the geometric and attribute characteristics of the point cloud data, and respectively projecting to obtain different two-dimensional fragments.
Specifically, the projection mode is to project a projection line to a selected projection plane through a point or other object, and obtain a pattern on the plane, and includes any one of the following:
a hexahedral projection scheme, an N-hedron projection scheme, a cylindrical projection scheme, and a conical projection scheme.
In particular, the hexahedral projection scheme includes:
analyzing the sub-point cloud data, and judging a projection axis which furthest retains the original distribution characteristics; assume a sample point of xiAnd assuming that the new coordinate system after projective transformation is W ═ W1,w2,w3In which w1,w2,w3For the basis vector of the new coordinate system, x can be obtainediThe projection in the hyperplane is WTxiThe optimization target is as follows:
s.tWTW＝I
using the lagrange multiplier method for the above equation, one can obtain:
XXTW＝λW
wherein the content of the first and second substances,
i represents a unit array;
lambda represents a matrix of undetermined eigenvalues;
X＝(x1,x2,…,xn) The method comprises the steps of (1) a point cloud data point set, wherein n is the total number of points contained in the point cloud data, and i is 1, 2, …, n;
for covariance matrix XXTDecomposing the eigenvalues, wherein the obtained maximum eigenvectors of the first eigenvalues are the standard orthogonal basis vectors of the space where the sample is located after dimensionality reduction;
with the corresponding feature vector w1,w2,w3And establishing a new space rectangular coordinate system for the X, Y, Z axes to determine the optimal projection plane.
Specifically, the method further comprises the following steps:
splicing according to the spatial correlation between the two-dimensional fragments;
the spatial correlation includes the separation distance in the original point cloud data and the similarity degree between the attribute features.
Specifically, the method further comprises the following steps:
when the two-dimensional fragments are spliced, the splicing adjusting factors comprise any one or more of the following factors:
the compactness factor of the spliced image and the connection degree factor among the two-dimensional fragments.
Specifically, the method further comprises the following steps:
when the splicing mode is selected, the reference for selecting the splicing mode comprises any one or more of the following: the size of the whole image/video frame, the number and proportion of effective pixels, the spacing distance of each two-dimensional fragment and the like.
The invention provides a point cloud data processing system based on segmentation, which comprises:
the segmentation module is used for segmenting the point cloud data according to the geometric and attribute characteristics of the point cloud data to obtain a plurality of sub-point clouds;
the projection module is used for projecting the sub-point clouds respectively; and
and the splicing module is used for splicing the projected two-dimensional fragments to obtain a two-dimensional image or a video frame.
Specifically, the method further comprises the following steps:
and the data analysis module is used for carrying out data analysis on the point cloud data before the point cloud data is segmented.
Specifically, the method further comprises the following steps:
and the compression coding module is used for further performing compression coding on the obtained two-bit image or video frame.
Specifically, the segmentation module:
the segmentation method for segmenting the point cloud data comprises any one or more of the following methods:
an edge-based segmentation algorithm, a region-growth-based segmentation algorithm, an attribute-based segmentation algorithm, a model-based segmentation algorithm, a graph-based segmentation algorithm, a deep learning-based segmentation algorithm.
Specifically, the model-based segmentation algorithm:
and segmenting by using a mathematical model of an original geometric form as prior knowledge, and classifying point cloud data with the same mathematical expression into the same region, wherein the original geometric form comprises a plane, a cylinder, a cone or a sphere.
Specifically, the point cloud data is segmented according to the geometric and attribute features of the point cloud data:
the geometric and attribute features include any one or more of:
geometric features, textural features, normal vector features, or reflectance features.
Specifically, the method further comprises the following steps:
and correspondingly setting projection modes for projection according to the geometric and attribute characteristics of the point cloud data, and respectively projecting to obtain different two-dimensional fragments.
Specifically, the projection mode is to project a projection line to a selected projection plane through a point or other object, and obtain a pattern on the plane, and includes any one of the following:
a hexahedral projection scheme; an N-face projection scheme; a cylindrical projection scheme; and a cone projection scheme.
Specifically, the method further comprises the following steps:
splicing according to the spatial correlation between the two-dimensional fragments;
the spatial correlation includes the separation distance in the original point cloud data and the similarity degree between the attribute features.
Specifically, the method further comprises the following steps:
when the two-dimensional fragments are spliced, the splicing adjusting factors comprise any one or more of the following factors:
compactness of the spliced image and a connection degree factor among the two-dimensional fragments.
Specifically, the method further comprises the following steps:
when the splicing mode is selected, the reference for selecting the splicing mode comprises any one or more of the following: the size of the whole image/video frame, the number of effective pixels, the proportion of the effective pixels, the spacing distance of each two-dimensional fragment and the like.
The present invention will be described more specifically below with reference to preferred examples.
Preferred example 1:
the mainstream processing method of the point cloud is to convert the point cloud into a two-dimensional plane for processing, but most of the schemes are to convert the point cloud into a whole for projection or mapping, so that the characteristics of geometry, texture, reflection coefficient, normal vector and the like of each part of the point cloud data cannot be sufficiently found, and the compression efficiency is limited. In addition, the accuracy, irregularity and density of the point cloud acquisition cause the problem of point coincidence during projection, namely the problem of occlusion. The common processing mode is multiple projections at different angles, but a large number of black dispersion points and even stripes appear on a two-dimensional picture obtained after multiple projections, which is extremely not beneficial to compression coding. People are further upgrading the consumption of media nowadays, and point clouds are indispensable as an important representation form of immersive media. In the fields of automatic driving, security monitoring and the like, efficient point cloud compression is also needed to optimize a solution. In order to realize higher performance and efficiency of point cloud coding based on two-dimensional plane compression, the invention provides a point cloud projection compression method based on segmentation.
First, a point cloud is partitioned.
In the invention, the point cloud data is segmented according to the geometric and attribute characteristics of the point cloud data, the segmented attribute basis can be various, and preferably, the segmentation can be realized by taking the following attributes as an example:
geometric characteristics: the most basic data form is (X, Y, Z), which is the distance between each point and each coordinate axis direction in the space rectangular coordinate system, and represents the space position of each point;
texture characteristics: the most basic data form is (R, G, B), which is the numerical values of three color channels of red, green and blue of each point and reflects the color attribute of each point;
normal vector: namely, the normal vector characteristics of the point cloud data;
reflection coefficient: namely the reflection coefficient characteristics of the point cloud data, and the brightness change of each point when the point is viewed in different directions is reflected.
In this step, the segmentation can be performed in a number of different ways, each with a focus. Each individual piece of the point cloud after segmentation is referred to as a sub-point cloud.
Preferably, reference is made to several methods:
an edge-based segmentation method for obtaining a segmented region by detecting an edge region, i.e., a region where the intensity of a point cloud changes rapidly or a surface normal vector changes rapidly, and delineating edge information hidden in point cloud data.
Region growing-based segmentation algorithm, which combines points with the same attributes into isolated regions in the neighborhood region while ensuring maximum difference between the remaining surrounding regions.
An attribute-based segmentation algorithm, which is a segmentation algorithm that uses the characteristic attributes of the point cloud for clustering.
Model-based segmentation algorithms that segment point cloud data with the same mathematical expression into the same region using a mathematical model of the original geometry (e.g., plane, cylinder, cone, sphere, etc.) as a priori knowledge.
Graph-based segmentation algorithms, which construct graph structures using point cloud data, each point cloud corresponding to a vertex in the graph, and an edge between two vertices connecting two adjacent point cloud data. Each edge is assigned a weight that represents the similarity of a pair of points in the point cloud data.
And a segmentation algorithm based on deep learning, wherein the method realizes semantic segmentation of point cloud data by designing different point cloud segmentation deep learning frameworks. The point cloud has the characteristic of irregular spatial relationship during segmentation, so that the existing image classification segmentation frame cannot be directly sleeved on the point cloud.
Secondly, projecting each sub-point cloud. The method of projecting a projection line through a point or other object onto a selected projection surface and obtaining a pattern on that surface is called projection.
In the present invention, the projection mode may be various, and preferably, the projection mode may be implemented by the following projection schemes:
hexahedral projection scheme: and setting the size of the bounding box according to the size of the original point cloud, and regularizing the point cloud with different resolutions. Projecting by taking six planes of the bounding box as a reference;
n-face projection scheme: and setting the size of the bounding box according to the size of the original point cloud, and regularizing the point cloud with different resolutions. Projecting by taking N planes of the bounding box as a reference;
the cylindrical projection scheme is as follows: setting the radius and the projection axis of a projected cylinder according to the size of the original point cloud, projecting each point onto a cylindrical surface, expanding the points, and then regularizing to obtain a projected image;
cone projection scheme: and setting the radius and the projection axis of the bottom of the cone of the projection according to the size of the original point cloud, projecting each point onto the conical surface, expanding the points, and regularizing to obtain a projection image.
Including but not limited to hexahedral projection, N-hedral projection, cylindrical projection, conical projection, etc. Next, referring to fig. 4, to illustrate the hexahedral projection method as an example, first, each sub-point cloud data is analyzed, and a projection axis that maximally retains the original distribution characteristics can be determined by using a principal component analysis method; assume a sample point of xiAnd assuming that the new coordinate system after projective transformation is W ═ W1,w2,w3In which w1,w2,w3For the basis vector of the new coordinate system, x can be obtainediThe projection in the hyperplane is WTxi. The optimization objective is
s.tWTW＝I
Using Lagrange multiplier method for the above equation
XXTW＝λW
Then, for the covariance matrix XXTCarry out characteristic valueAnd decomposing, wherein the obtained feature vector of the largest first 3 feature values is the standard orthogonal basis vector of the space where the sample is located after dimensionality reduction. With the corresponding feature vector w1,w2,w3And establishing a new space rectangular coordinate system for the X, Y, Z axes to determine the optimal projection plane.
And thirdly, splicing images obtained after projection of the sub point clouds into two-dimensional images/video frames. In the process, factors such as the compactness of the spliced image and the connection degree among sub-blocks are considered, the splicing mode can be selected by taking the values such as the size of the whole image/video frame, the number and proportion of effective pixels, the spacing distance of two-dimensional fragments and the like as references, and the overall coding compression efficiency is improved. And after the two-dimensional image/video frame splicing is finished, coding and compressing by adopting a traditional two-dimensional image/video compression mode. The image compression can adopt JPEG/PNG and other modes. The main compression tool for HEVC coding has high video compression efficiency. The modes of MPEG1/MPEG2/MPEG4/H264 and the like are also optional.
Fig. 8 shows a detailed flowchart of a segmentation-based point cloud data processing method, which is briefly described as follows: after the original point cloud is obtained, the geometric and attribute characteristics are analyzed to determine the segmentation mode. The original point cloud is divided into a plurality of sub-point clouds, and then each sub-point cloud is respectively projected to obtain two-dimensional fragments, and two-dimensional images/video frames are formed by splicing. And finally, compressing by using a traditional image/video compression tool.
The segmentation-based point cloud data processing system comprises four modules, as shown in fig. 8.
The data analysis module and the segmentation module in the invention can be set independently or collectively, and in this embodiment, the two modules are integrated into one point cloud analysis and segmentation module, which is not limited by the invention.
The point cloud analysis and segmentation module comprises: the input is original point cloud, the geometric and attribute characteristics of the point cloud are analyzed, the segmentation mode is determined and the segmentation is finished, and the point cloud is output as sub-point cloud.
The projection module is a point cloud projection dimension reduction module in the embodiment: the input is a plurality of sub-point clouds, projection dimensionality reduction is carried out on each sub-point cloud respectively, and two-dimensional fragments obtained by projection of each sub-point cloud are output.
The stitching module is a two-dimensional image/video frame stitching module in this embodiment: the input is a two-dimensional fragment and it is stitched and output as a complete two-dimensional image/video frame.
The compression and encoding module, i.e. the two-dimensional image/video frame compression module in this implementation: the input is a two-dimensional image/video frame, the module uses a traditional image/video compression tool to carry out compression coding, and the output is a compressed code stream.
It should be noted that the segmentation, projection, stitching and compression schemes are only described in the present invention by taking the above method as an example, and are not limited to the above method.
In summary, the method and system for processing point cloud data based on segmentation provided by the invention firstly analyze the geometric and attribute characteristics of the original point cloud data, and perform segmentation processing according to the geometric and attribute characteristic point cloud, thereby obtaining a plurality of independent sub-point clouds; aiming at the sub-point clouds after segmentation, corresponding projection modes are set according to geometric and attribute characteristics, and different two-dimensional fragments are obtained through respective projection; finally, reasonable splicing is carried out according to the spatial correlation (such as the separation distance in the original point cloud and the similarity degree among attribute features) among the two-dimensional fragments, so that the sub-blocks are naturally transited, compactly arranged and suitable for coding, and further, the traditional image/video coding mode is adopted for compression coding.
In the present embodiment, each functional module in the segmentation-based point cloud data processing system corresponds to the segmentation-based point cloud data processing method in the above embodiment, so that the structure and technical elements in the device can be formed by corresponding conversion of the generation method, and the description is omitted here and will not be repeated.
Those skilled in the art will appreciate that, in addition to implementing the systems, apparatus, and various modules thereof provided by the present invention in purely computer readable program code, the same procedures can be implemented entirely by logically programming method steps such that the systems, apparatus, and various modules thereof are provided in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers and the like. Therefore, the system, the device and the modules thereof provided by the present invention can be considered as a hardware component, and the modules included in the system, the device and the modules thereof for implementing various programs can also be considered as structures in the hardware component; modules for performing various functions may also be considered to be both software programs for performing the methods and structures within hardware components.
The foregoing description of specific embodiments of the present invention has been presented. It is to be understood that the present invention is not limited to the specific embodiments described above, and that various changes or modifications may be made by one skilled in the art within the scope of the appended claims without departing from the spirit of the invention. The embodiments and features of the embodiments of the present application may be combined with each other arbitrarily without conflict.
Claims (25)
1. A point cloud data processing method based on segmentation is characterized by comprising the following steps:
dividing the point cloud data according to the geometric and attribute characteristics of the point cloud data to obtain a plurality of sub-point clouds;
respectively projecting the sub-point clouds; and
and splicing the projected two-dimensional fragments to obtain a two-dimensional image or a video frame.
2. The segmentation-based point cloud data processing method of claim 1, further comprising:
before the point cloud data is segmented, data analysis of the point cloud data is included.
3. The segmentation-based point cloud data processing method of claim 2, wherein the data analysis of the point cloud data employs principal component analysis.
4. The segmentation-based point cloud data processing method of claim 1, further comprising:
the resulting two-bit image or video frame is further compression encoded.
5. The segmentation-based point cloud data processing method of claim 4, wherein the compression encoding comprises:
JPEG/PNG is adopted for image compression;
the video compression adopts HEVC or MPEG1/MPEG2/MPEG 4/H264.
6. The segmentation-based point cloud data processing method of claim 1, wherein the segmentation manner for segmenting the point cloud data comprises any one or more of the following:
an edge-based segmentation algorithm, a region-growth-based segmentation algorithm, an attribute-based segmentation algorithm, a model-based segmentation algorithm, a graph-based segmentation algorithm, a deep learning-based segmentation algorithm.
7. The segmentation-based point cloud data processing method of claim 6, wherein the model-based segmentation algorithm:
and segmenting by using a mathematical model of an original geometric form as prior knowledge, and classifying point cloud data with the same mathematical expression into the same region, wherein the original geometric form comprises a plane, a cylinder, a cone or a sphere.
8. The segmentation-based point cloud data processing method of claim 1, wherein the point cloud data is segmented according to geometric and attribute features of the point cloud data:
the geometric and attribute features include any one or more of:
geometric features, textural features, normal vector features, or reflectance features.
9. The segmentation-based point cloud data processing method of claim 1, further comprising:
and correspondingly setting projection modes for projection according to the geometric and attribute characteristics of the point cloud data, and respectively projecting to obtain different two-dimensional fragments.
10. The segmentation-based point cloud data processing method of claim 9, wherein the projection method comprises projecting a projection line through a point or other object onto a selected projection plane to obtain a pattern on the selected projection plane, and comprises any one of the following:
a hexahedral projection scheme, an N-hedron projection scheme, a cylindrical projection scheme, and a conical projection scheme.
11. The segmentation-based point cloud data processing method of claim 10, wherein the hexahedral projection scheme comprises:
analyzing the sub-point cloud data, and judging a projection axis which furthest retains the original distribution characteristics; assume a sample point of xiAnd assuming that the new coordinate system after projective transformation is W ═ W1,w2,w3In which w1,w2,w3For the basis vector of the new coordinate system, x can be obtainediThe projection in the hyperplane is WTxiThe optimization target is as follows:
s.tWTW＝I
using the lagrange multiplier method for the above equation, one can obtain:
XXTW＝λW
wherein the content of the first and second substances,
i represents a unit array;
lambda represents a matrix of undetermined eigenvalues;
X＝(x1,x2,…,xn) The method comprises the steps of (1) a point cloud data point set, wherein n is the total number of points contained in the point cloud data, and i is 1, 2, …, n;
for covariance matrix XXTBy decomposition of characteristic values, the largest first several characteristic values being obtainedThe characteristic vector is a standard orthogonal basis vector of a space where the sample is located after dimensionality reduction;
with the corresponding feature vector w1,w2,w3And establishing a new space rectangular coordinate system for the X, Y, Z axes to determine the optimal projection plane.
12. The segmentation-based point cloud data processing method of claim 1, further comprising:
splicing according to the spatial correlation between the two-dimensional fragments;
the spatial correlation includes the separation distance in the original point cloud data and the similarity degree between the attribute features.
13. The segmentation-based point cloud data processing method of claim 1, further comprising:
when the two-dimensional fragments are spliced, the splicing adjusting factors comprise any one or more of the following factors:
the compactness factor of the spliced image and the connection degree factor among the two-dimensional fragments.
14. The segmentation-based point cloud data processing method of claim 1, further comprising:
when the splicing mode is selected, the reference for selecting the splicing mode comprises any one or more of the following: the size of the whole image/video frame, the number and proportion of effective pixels, the spacing distance of each two-dimensional fragment and the like.
15. A segmentation-based point cloud data processing system, comprising:
the segmentation module is used for segmenting the point cloud data according to the geometric and attribute characteristics of the point cloud data to obtain a plurality of sub-point clouds;
the projection module is used for projecting the sub-point clouds respectively; and
and the splicing module is used for splicing the projected two-dimensional fragments to obtain a two-dimensional image or a video frame.
16. The segmentation-based point cloud data processing system of claim 15, further comprising:
and the data analysis module is used for carrying out data analysis on the point cloud data before the point cloud data is segmented.
17. The segmentation-based point cloud data processing system of claim 15, further comprising:
and the compression coding module is used for further performing compression coding on the obtained two-bit image or video frame.
18. The segmentation-based point cloud data processing system of claim 15, wherein the segmentation module:
the segmentation method for segmenting the point cloud data comprises any one or more of the following methods:
an edge-based segmentation algorithm, a region-growth-based segmentation algorithm, an attribute-based segmentation algorithm, a model-based segmentation algorithm, a graph-based segmentation algorithm, a deep learning-based segmentation algorithm.
19. The segmentation-based point cloud data processing system of claim 18, wherein the model-based segmentation algorithm:
and segmenting by using a mathematical model of an original geometric form as prior knowledge, and classifying point cloud data with the same mathematical expression into the same region, wherein the original geometric form comprises a plane, a cylinder, a cone or a sphere.
20. The segmentation-based point cloud data processing system of claim 15, wherein the segmentation of the point cloud data is based on geometric and attribute features of the point cloud data:
the geometric and attribute features include any one or more of:
geometric features, textural features, normal vector features, or reflectance features.
21. The segmentation-based point cloud data processing system of claim 15, further comprising:
and correspondingly setting projection modes for projection according to the geometric and attribute characteristics of the point cloud data, and respectively projecting to obtain different two-dimensional fragments.
22. The segmentation-based point cloud data processing system of claim 21, wherein the projection means projects a projection line through a point or other object onto a selected projection surface and results in a pattern on the surface, comprising any one of:
a hexahedral projection scheme; an N-face projection scheme; a cylindrical projection scheme; and a cone projection scheme.
23. The segmentation-based point cloud data processing system of claim 15, further comprising:
splicing according to the spatial correlation between the two-dimensional fragments;
the spatial correlation includes the separation distance in the original point cloud data and the similarity degree between the attribute features.
24. The segmentation-based point cloud data processing system of claim 15, further comprising:
when the two-dimensional fragments are spliced, the splicing adjusting factors comprise any one or more of the following factors:
compactness of the spliced image and a connection degree factor among the two-dimensional fragments.
25. The segmentation-based point cloud data processing system of claim 15, further comprising:
when the splicing mode is selected, the reference for selecting the splicing mode comprises any one or more of the following: the size of the whole image/video frame, the number of effective pixels, the proportion of the effective pixels, the spacing distance of each two-dimensional fragment and the like.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910372764.6A CN111899162A (en) | 2019-05-06 | 2019-05-06 | Point cloud data processing method and system based on segmentation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910372764.6A CN111899162A (en) | 2019-05-06 | 2019-05-06 | Point cloud data processing method and system based on segmentation |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111899162A true CN111899162A (en) | 2020-11-06 |
Family
ID=73169470
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910372764.6A Pending CN111899162A (en) | 2019-05-06 | 2019-05-06 | Point cloud data processing method and system based on segmentation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111899162A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113012177A (en) * | 2021-04-02 | 2021-06-22 | 上海交通大学 | Three-dimensional point cloud segmentation method based on geometric feature extraction and edge perception coding |
| CN115468576A (en) * | 2022-09-29 | 2022-12-13 | 东风汽车股份有限公司 | Automatic driving positioning method and system based on multi-mode data fusion |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102997871A (en) * | 2012-11-23 | 2013-03-27 | 南京大学 | Method for inverting effective leaf area index by utilizing geometric projection and laser radar |
| CN109544456A (en) * | 2018-11-26 | 2019-03-29 | 湖南科技大学 | The panorama environment perception method merged based on two dimensional image and three dimensional point cloud |
-
2019
- 2019-05-06 CN CN201910372764.6A patent/CN111899162A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102997871A (en) * | 2012-11-23 | 2013-03-27 | 南京大学 | Method for inverting effective leaf area index by utilizing geometric projection and laser radar |
| CN109544456A (en) * | 2018-11-26 | 2019-03-29 | 湖南科技大学 | The panorama environment perception method merged based on two dimensional image and three dimensional point cloud |
Non-Patent Citations (4)
| Title |
|---|
| S SCHWARZ: "Emerging MPEG Standards for Point Cloud Compression", 《IEEE JOURNAL ON EMERGING AND SELECTED TOPICS IN CIRCUITS AND SYSTEMS》 * |
| 万怡平等: "地面点云数据快速重建建筑模型的分层方法", 《测绘工程》 * |
| 师域轩等: "三维点云数据分割原理及应用", 《科技资讯》 * |
| 李刚: "基于非线性降维方法的高维数据可视化平台研究与实现", 《信息科技》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113012177A (en) * | 2021-04-02 | 2021-06-22 | 上海交通大学 | Three-dimensional point cloud segmentation method based on geometric feature extraction and edge perception coding |
| CN115468576A (en) * | 2022-09-29 | 2022-12-13 | 东风汽车股份有限公司 | Automatic driving positioning method and system based on multi-mode data fusion |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Bouwmans et al. | On the role and the importance of features for background modeling and foreground detection | |
| US10547812B2 (en) | Video capture device and method | |
| Rusu et al. | 3d is here: Point cloud library (pcl) | |
| US8761509B1 (en) | Method and apparatus for fast computational stereo | |
| CA2851498C (en) | System and method for digital image signal compression using intrinsic images | |
| Wang et al. | Sparse tensor-based point cloud attribute compression | |
| CN111899162A (en) | Point cloud data processing method and system based on segmentation | |
| US20150326878A1 (en) | Selective perceptual masking via scale separation in the spatial and temporal domains using intrinsic images for use in data compression | |
| Ainala et al. | An improved enhancement layer for octree based point cloud compression with plane projection approximation | |
| Nousias et al. | A saliency aware CNN-based 3D model simplification and compression framework for remote inspection of heritage sites | |
| Figueiredo et al. | Saliency maps for point clouds | |
| Karakottas et al. | 360D: a dataset and baseline for dense depth estimation from 360 images | |
| Abri et al. | Multi-thread frame tiling model in concurrent real-time object detection for resources optimization in yolov3 | |
| Huang et al. | Generalized rank conditions in multiple view geometry with applications to dynamical scenes | |
| CN111899152A (en) | Point cloud data compression method and system based on projection and video stitching | |
| US20150350647A1 (en) | Selective perceptual masking via downsampling in the spatial and temporal domains using intrinsic images for use in data compression | |
| Schiopu et al. | Subaperture image segmentation for lossless compression | |
| Jin et al. | Meshreduce: Scalable and bandwidth efficient 3d scene capture | |
| CN111862098A (en) | Individual matching method, device, equipment and medium based on light field semantics | |
| Koschan | Improving robot vision by color information | |
| WO2024066123A1 (en) | Lossless hierarchical coding of image feature attributes | |
| Li et al. | 3D point cloud attribute compression based on cylindrical projection | |
| US11676334B2 (en) | Method and apparatus for plenoptic point clouds generation | |
| Bhende et al. | A novel feature extraction scheme for medical X-ray images | |
| Ryde et al. | Generating Edge Voxel Maps with Structure Tensor Analysis |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20201106 |
|
| RJ01 | Rejection of invention patent application after publication |