CN117218143B

CN117218143B - Point cloud segmentation method and device

Info

Publication number: CN117218143B
Application number: CN202311467571.1A
Authority: CN
Inventors: 宋昱; 束健; 杜东晖; 郭胜男
Original assignee: Faoyiwei Suzhou Robot System Co ltd
Current assignee: Faoyiwei Suzhou Robot System Co ltd
Priority date: 2023-11-07
Filing date: 2023-11-07
Publication date: 2024-01-23
Anticipated expiration: 2043-11-07
Also published as: CN117218143A

Abstract

The application provides a point cloud segmentation method and device, wherein a local coordinate system is built based on points aiming at each point in a point cloud of a target object, and projection information of adjacent points adjacent to the points in the local coordinate system is obtained. And fitting according to projection information of adjacent points of the points to obtain the principal curvature of the points, determining the surface type of the points according to the principal curvature, selecting any one of the points of the surface type as an initial seed point aiming at various surface types existing in the point cloud, and starting from the initial seed point and carrying out region growth according to the points of the surface type to determine the corresponding surface so as to realize the segmentation of the point cloud. In the scheme, the area of various surface types can be segmented by determining the surface type and selecting the seed point from the points of the surface type to perform area growth, and in the area growth process, only the points of the same surface type are added and cannot grow into the areas of other surface types, so that the point cloud comprising the various surface types can be segmented well.

Description

Point cloud segmentation method and device

Technical Field

The invention relates to the technical field of three-dimensional point cloud data processing, in particular to a point cloud segmentation method and device.

Background

The point cloud segmentation divides the point cloud according to the characteristics of space, geometry, texture and the like, so that the point clouds in the same division have similar characteristics. Region growing is a point cloud segmentation algorithm that utilizes the normal consistency of seed points and neighboring points to grow seed point regions.

In the conventional implementation manner of performing point cloud segmentation by using region growth, a point with the smallest surface curvature in the point cloud is generally selected as a seed point, the seed point is added in a current region, and the seed point is added in a current seed point list. Then searching adjacent points near the seed point, and judging each adjacent point as follows: if the normal direction of the adjacent point is consistent with the normal direction at the seed point, if the adjacent point is near the local plane at the seed point, the adjacent point is added into the current area if both conditions are satisfied. And then judging whether the surface curvature of the seed point is smaller than a certain threshold value, and if so, adding the adjacent point into the seed point list. If the seed point list is newly added, repeating the operation on the newly added seed points until no new adjacent points are added in the current area and no new seed points are added. This results in a region in the point cloud. And (5) continuing to select seed points from the rest points in the point cloud, and performing region growing operation until all points finish the region growing process.

Since the existing method uses the point with the minimum surface curvature as the initial seed point to perform growth, and if a new seed point needs to be added into the current seed point list during the area growth process, the surface curvature of the point is required to be smaller than a threshold value. Therefore, existing conventional segmentation algorithms can only segment flat areas in the point cloud, or called planar areas. If a curved surface area exists in the point cloud, the existing area growth segmentation algorithm cannot be well processed.

Disclosure of Invention

The invention aims at providing a point cloud segmentation method and a point cloud segmentation device, which can segment point clouds containing multiple surface types.

Embodiments of the invention may be implemented as follows:

in a first aspect, the present invention provides a point cloud segmentation method, the method comprising:

constructing a local coordinate system based on each point in a point cloud of a target object, and obtaining projection information of adjacent points adjacent to the point in the local coordinate system;

fitting according to projection information of adjacent points of the points to obtain the principal curvature of the points;

determining the surface type of the point according to the principal curvature;

And selecting any one point of points belonging to the surface types as an initial seed point aiming at various surface types in the point cloud, and starting from the initial seed point and carrying out region growth according to the points belonging to the surface types to determine the corresponding surface so as to realize the segmentation of the point cloud.

In an optional embodiment, the method further includes filtering the point cloud in advance, and the step includes:

constructing a local plane of each point in the point cloud;

calculating the distance between each adjacent point of the points and the local plane, and taking the adjacent point with the distance smaller than a preset distance value as a target adjacent point;

if the number of the target adjacent points is greater than or equal to a preset threshold value, reserving the points;

and if the number of the target adjacent points is smaller than the preset threshold value, filtering the points.

In an alternative embodiment, the step of constructing a local coordinate system based on the points includes:

constructing a local plane of the point, and obtaining a normal vector of the point;

and constructing and obtaining a local coordinate system based on the normal vector and the two mutually perpendicular direction vectors on the local plane.

In an alternative embodiment, the step of obtaining projection information of neighboring points neighboring the point in the local coordinate system includes:

calculating, for each neighboring point adjacent to the point, a difference vector between the neighboring point and the point;

and projecting the difference vector to each coordinate axis of the local coordinate system to obtain projection information on each coordinate axis.

In an alternative embodiment, the step of fitting to obtain the principal curvature of the point according to the projection information of the points adjacent to the point includes:

fitting the constructed quadratic polynomial according to projection information of all adjacent points of the points, and determining coefficients in the quadratic polynomial;

based on the coefficient of the quadratic polynomial, obtaining a hessian matrix corresponding to the quadratic polynomial;

and decomposing the characteristic value of the hessian matrix to obtain the principal curvature of the point.

In an alternative embodiment, the principal curvatures comprise a first principal curvature and a second principal curvature;

the step of determining the surface type to which the point belongs according to the principal curvature comprises the following steps:

and determining the surface type of the point according to the values of the first principal curvature and the second principal curvature, wherein the surface type is a plane type, a cylindrical-like surface type, a spherical-like surface type or a saddle-like surface type.

In an optional embodiment, the step of selecting, for each surface type existing in the point cloud, any one of the points belonging to the surface type as an initial seed point includes:

for each point in various surface types existing in the point cloud, adding the point into a surface type point area corresponding to the surface type when more than a preset number of adjacent points with the same surface type as the point exist;

and selecting any one point from the surface type point area as an initial seed point.

In an alternative embodiment, the step of determining the corresponding surface by performing region growing from the initial seed point and according to the point of the surface type includes:

searching adjacent points of the initial seed points in sequence, and taking the adjacent points as new seed points if the adjacent points have normal consistency and surface consistency with the initial seed points;

and sequentially searching new adjacent points of the new seed points aiming at the new seed points, and taking the new adjacent points as the new seed points if the new adjacent points have normal consistency and surface consistency with the new seed points until all the new seed points are searched out so as to determine the corresponding surfaces.

In an alternative embodiment, if the included angle between the normal vector of the neighboring point and the normal vector of the initial seed point is smaller than a preset included angle, determining that the neighboring point has normal consistency with the initial seed point;

and if the included angle between the difference vector between the adjacent point and the initial seed point and the normal vector of the initial seed point is in a preset range, determining that the adjacent point and the initial seed point have surface consistency.

In a second aspect, the present invention provides a point cloud segmentation apparatus, the apparatus comprising:

the construction module is used for constructing a local coordinate system based on each point in the point cloud of the target object and obtaining projection information of adjacent points adjacent to the point in the local coordinate system;

the fitting module is used for fitting according to projection information of adjacent points of the points to obtain the principal curvature of the points;

a determining module, configured to determine, according to the principal curvature, a surface type to which the point belongs;

and the growth module is used for selecting any one of the points belonging to the surface types as an initial seed point aiming at various surface types existing in the point cloud, starting from the initial seed point, and carrying out region growth according to the points belonging to the surface types to determine the corresponding surface so as to realize the segmentation of the point cloud.

The beneficial effects of the embodiment of the invention include, for example:

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the embodiments will be briefly described below, it being understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and other related drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

Fig. 1 is a flowchart of a point cloud segmentation method provided in an embodiment of the present application;

fig. 2 is a flowchart of a filtering processing method in the point cloud segmentation method provided in the embodiment of the present application;

FIG. 3 is a flowchart of the sub-steps involved in S11 of FIG. 1;

FIG. 4 is another flow chart of sub-steps involved in S11 of FIG. 1;

FIG. 5 is a flowchart of the substeps involved in S12 of FIG. 1;

FIG. 6 is a schematic plan view of a point cloud;

FIG. 7 is a schematic view of a cylinder-like surface in a point cloud;

FIG. 8 is a schematic diagram of a sphere-like surface in a point cloud;

FIG. 9 is a saddle-like surface schematic view of a point cloud;

FIG. 10 is a flowchart of the substeps involved in S14 of FIG. 1;

FIG. 11 is another flow chart of sub-steps involved in S14 of FIG. 1;

fig. 12 is a logic schematic diagram of an overall implementation of the point cloud segmentation method provided in the embodiment of the present application;

fig. 13 is a logic schematic diagram of an overall implementation of filtering processing provided in an embodiment of the present application;

FIG. 14 is a logical schematic of the overall implementation of principal curvature and principal direction determination provided by embodiments of the present application;

FIG. 15 is a logic diagram of an overall implementation of the area determination of the face type points in an embodiment of the present application;

fig. 16 is a functional block diagram of a point cloud segmentation apparatus according to an embodiment of the present application;

fig. 17 is a block diagram of an electronic device according to an embodiment of the present application.

Icon: 110-a point cloud segmentation device; 111-building a module; 112-a fitting module; 113-a determination module; 114-a growth module; a 120-processor; 130-memory; 140-communication module.

Detailed Description

For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. The components of the embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations.

Thus, the following detailed description of the embodiments of the invention, as presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

It should be noted that: like reference numerals and letters denote like items in the following figures, and thus once an item is defined in one figure, no further definition or explanation thereof is necessary in the following figures.

In the description of the present invention, it should be noted that, if the terms "first," "second," and the like are used merely to distinguish the descriptions, they are not to be construed as indicating or implying relative importance.

It should be noted that the features of the embodiments of the present invention may be combined with each other without conflict.

The existing traditional point cloud segmentation method can only segment a flat area in the point cloud, and if a curved surface area exists in the point cloud, the existing area growth segmentation method cannot be well processed.

First, from the process of selecting seed points, in the conventional manner, each time a new region needs to be grown, a point with the minimum surface curvature is selected from the point cloud as an initial seed point. The surface curvature describes the degree of flatness of points in the point cloud, with the point of least surface curvature being the flattest point, i.e., the point closest to the plane. During the region growing process, if a new seed point needs to be added to the current seed point list, the surface curvature of the point needs to be smaller than a threshold value, that is, only the point of the flat region can be added to the seed point list as the new seed point. I.e. the seed points all belong to a flat area.

Secondly, from the process of region growing, a criterion for judging whether a point can be added to the current region is that the normal direction of the current point is consistent with the normal direction of the seed point, and the condition also determines that only points on the same plane can be added to the current region.

From the two aspects of the seed point selection process and the region growing process, the conventional region growing algorithm is determined that only a flat region in the point cloud can be segmented, but a curved surface region cannot be segmented. When there is a curved surface region in the point cloud, such as a curved surface like a cylindrical surface, a conical surface, a spherical surface, the existing region growing algorithm will not be able to handle.

Based on the research, the application provides a point cloud segmentation scheme, and the areas of various surface types can be segmented by determining the surface types in the point cloud and selecting seed points from the points of the surface types to perform area growth. In the region growing process, only the points with the same surface type are added, the points can not grow into the regions with other surface types, and the point cloud containing multiple surface types can be well segmented.

Fig. 1 is a schematic flow chart of a point cloud segmentation method provided in an embodiment of the present application, where the point cloud segmentation method may be performed by a point cloud segmentation apparatus, and the point cloud segmentation apparatus may be implemented by software and/or hardware, and may be configured in an electronic device, and the electronic device may be a computer device with related software installed. The detailed steps of the point cloud segmentation method are described below.

S11, constructing a local coordinate system based on each point in the point cloud of the target object, and obtaining projection information of adjacent points adjacent to the point in the local coordinate system.

And S12, fitting according to projection information of points adjacent to the point to obtain the principal curvature of the point.

S13, determining the surface type of the point according to the principal curvature.

S14, selecting any one of points belonging to the surface types as an initial seed point aiming at various surface types in the point cloud, and starting from the initial seed point and carrying out region growth according to the points belonging to the surface types to determine the corresponding surface so as to realize the segmentation of the point cloud.

In this embodiment, the target object may be any workpiece model, and the image capturing device, for example, a 3D camera, may be used to capture the workpiece model to obtain corresponding image information, and obtain a three-dimensional point cloud corresponding to the workpiece model based on the image information.

The three-dimensional point cloud is composed of a plurality of points, which are surface points of the target object. There may be multiple faces in a point cloud of a target object, for example, the surface of which is composed of multiple planes, curved surfaces. Therefore, it is necessary to determine each face existing in the point cloud of the target object so as to divide each face.

In the case of dividing a plane existing in a point cloud, it is necessary to divide the plane based on information of each point. In this embodiment, the corresponding surface may be determined by determining the surface type to which each point belongs, and further performing region growing for each different surface type based on the point to which each surface type belongs.

From the differential geometry point of view, the plane differs from the curved surface by its principal curvature, whereas the curved surface is mainly divided into three basic curved surfaces, namely a cylindrical-like surface, a spherical-like surface and a saddle-like surface, the three curved surfaces differ from each other by their principal curvature values and signs. Therefore, the principal curvature and the principal direction of each point in the point cloud can be obtained to determine whether the point belongs to a plane or a curved surface, and if the point belongs to the curved surface, the point can be determined to which curved surface.

Therefore, in this embodiment, the principal curvatures of the respective points in the point cloud may be determined first. In particular, for each point, a local coordinate system may be constructed based on the point, which may be understood as a coordinate system centered on the point in which the point and the neighboring points of the point are located. Wherein, the adjacent point of a certain point is the point which is close to the point in position.

Each adjacent point in the local coordinate system has projection information on each coordinate axis of the local coordinate system. The projection information of the neighboring point in the local coordinate system describes the position of the neighboring point in the local coordinate system. In this embodiment, the above-described operations are performed for all neighboring points of a certain point.

Based on the obtained projection information of all adjacent points, a fitting mode, such as a least square fitting mode, is adopted to obtain the principal curvature of the points. The type of face to which the point belongs can be determined based on the principal curvature of the point.

By the method, the surface type of each point in the point cloud can be determined, and the same surface type is generally provided with a plurality of points. A group of point clouds has one or more face types, and even if there is only one face type, for example, there is only a plane type, the number of faces is plural, that is, there are plural planes, and therefore, it is also necessary to divide the plural planes.

In this embodiment, a point cloud segmentation algorithm for region growing is adopted, and for various surface types, region growing may be performed starting with a seed point under the surface type. In the region growing process, the growth is performed based on the points belonging to the face type, and therefore, it is possible to determine the face region extending from the seed point and containing only the points belonging to the same face type. The determined surface may be only one surface included in the surface type, or may be one of multiple surfaces included in the surface type, and the determination of other surfaces included in the surface type is also determined in the same manner, so that multiple surfaces included in the surface type may be determined.

In this way, the faces under each face type and each face under a single face type can be determined, so that the point cloud segmentation is realized.

According to the point cloud segmentation method provided by the embodiment, the surface types existing in the point cloud and the surface types to which each point belongs are determined, and the seed points are selected from the points of the surface types to perform region growth, so that the regions of various surface types can be segmented. In the region growing process, only the points with the same surface type are added, the points can not grow into the regions with other surface types, and the point cloud containing multiple surface types can be well segmented.

When a 3D camera is used for acquiring a point cloud, noise inevitably exists in the point cloud, the point on the surface of an object is fluctuated to a certain extent, some points exist near the surface of the object, and some miscellaneous points and isolated points exist. The presence of these points greatly affects the execution of the principal curvature calculation, and therefore, a filtering process is required for the point cloud to remove noise points therein.

Based on this, in this embodiment, after the point cloud is obtained and before the point cloud segmentation is performed, filtering processing may be performed on the point cloud in advance, and referring to fig. 2, this step may be implemented as follows:

S21, constructing a local plane of each point in the point cloud.

S22, calculating the distance between each adjacent point of the points and the local plane, and taking the adjacent point with the distance smaller than a preset distance value as a target adjacent point.

S23, if the number of the target adjacent points is greater than or equal to a preset threshold value, reserving the points.

S24, if the number of the target adjacent points is smaller than the preset threshold, filtering the points.

In this embodiment, the points to be filtered in the point cloud include points near the object surface, miscellaneous points, isolated points, and the like. It has been found that the relationship between the neighboring points of these points and the point is relatively loose, i.e. the point and its neighboring points cannot form a point cloud resembling the object plane.

Based on this, in this embodiment, for each point in the point cloud, a local plane of the point may be first constructed, and a normal vector of the point may be obtained, with a plane perpendicular to the normal vector as a local plane.

For each neighboring point of the point, the distance between the neighboring point and the local plane may be calculated, and when the distance between the neighboring point and the local plane is smaller than a preset distance value, for example, 1.5, the neighboring point may be considered to be on the local plane. When the point has enough neighboring points on the local plane of the point, for example, the number of neighboring points meeting the condition is greater than or equal to a preset threshold, the point and the neighboring points can form a point cloud of the object surface, and the point can be reserved.

If the number of the adjacent points meeting the above condition is smaller than the preset threshold, it indicates that the point and the adjacent points cannot form a point cloud similar to the object surface, and the point can be filtered.

According to the embodiment, the local plane is constructed in the mode, and the points on the surface of the object are determined and noise points are filtered through the distance between the adjacent points and the local plane, so that the accuracy of the subsequent processing flow can be ensured.

After noise points in the point cloud are filtered, main curvature and main direction calculation are carried out, and then region growth is carried out on various surface types to realize point cloud segmentation.

From the above, it is known that the type of face to which a point belongs can be determined based on the principal curvature of the point, which describes the degree of curvature of the face in the principal direction. The main direction includes a direction of maximum curvature and a direction of minimum curvature of the corresponding faces, respectively. The principal curvature and principal direction are eigenvalues and eigenvectors of the Weingarten transform, and coefficients of the first and second basic forms of the curved surface are used for representing the Weingarten transform, and then a coefficient matrix of the Weingarten transform is as follows:

wherein,E、F、Gis a coefficient of a first basic form of the curved surface,L、M、Nis a coefficient of the second basic form of the curved surface. When the curved surface is represented as a parameterized way r=r(u,v) When this is the case, these coefficients are calculated as follows:

E=<r _u ,r _u >

F=<r _u ,r _v >

G=<r _v ,r _v >

L=<r _uu ,n>

M=<r _uv ,n>

N=<r _vv ,n>

wherein,r _u representation of parametersuIs used for the deflection of the tube,r _v representation of parametersvIs used for the deflection of the tube,r _uu representation of parametersuIs used for the second-order deflection of the (a),r _vv representation of parametersvIs used for the second-order deflection of the (a),r _uv representation of parametersuvIs used for the mixing deflection of the (a),nrepresenting curved surface at%u,v) A unit normal vector at which the position of the sample is determined,<,>representing the inner product.

It can be seen from the above calculation process that if the principal curvature and principal direction of the point cloud at a certain point need to be calculated, at least a quadratic polynomial fit to the curved surface is required. If the formula is adoptedz=ax ² +bxy+cy ² +dx+ey+fThe quadric surface fitting is performed on the local point cloud, so that the fitted curved surface is not accurate enough and the point position is sensitive, and therefore, the quadric surface fitting mode based on the formula is abandoned, and the quadric surface fitting is performed on the basis of the primary curved surface (plane) fitting in the embodiment.

In this embodiment, the principal curvature of the point is obtained by constructing a local coordinate system of the point and fitting based on projection information of neighboring points in the local coordinate system.

Referring to fig. 3, in this embodiment, when the local coordinate system is constructed for each point, the following manner may be implemented:

s111, constructing a local plane of the point, and obtaining a normal vector of the point.

And S112, constructing a local coordinate system based on the normal vector and the two direction vectors perpendicular to each other on the local plane.

In this embodiment, for any point pi in the point cloud, a normal vector and a local plane of the point pi are obtained, where the local plane is a plane perpendicular to the normal vector. The normal vector of the point pi is taken as the z-axis and two mutually perpendicular direction vectors are chosen on the local plane as the x-axis and the y-axis, respectively. The point pi is used as the origin of coordinates of the local coordinate system to construct the local coordinate system. The local coordinate system is a coordinate system centered on the point pi in which the neighboring points of the point pi are located.

Referring to fig. 4, when obtaining projection information of a neighboring point of the point pi in a local coordinate system, the following steps may be implemented:

s113, for each neighboring point neighboring the point, a difference vector between the neighboring point and the point is calculated.

And S114, projecting the difference vector to each coordinate axis of the local coordinate system to obtain projection information on each coordinate axis.

In this embodiment, for any one of the neighboring points pj of the point pi, first, a difference vector pj-pi between the neighboring point pj and the point pi can be calculated. And respectively projecting the difference vector to the x-axis, the y-axis and the z-axis of the local coordinate system to obtain projection information of the difference vector on each coordinate axis. The projection information of the difference vector on each coordinate axis can be the length value of the projection on the coordinate axis, and the length value is recorded as a projection coefficient. These projection information describe the position of the neighboring point pj in the local coordinate system.

On this basis, referring to fig. 5, when the principal curvature of a point is obtained by fitting projection information of neighboring points of the point, it can be achieved by:

and S121, fitting the constructed quadratic polynomial according to projection information of all adjacent points of the point, and determining coefficients in the quadratic polynomial.

S122, obtaining the Heisen matrix corresponding to the quadratic polynomial based on the coefficients of the quadratic polynomial.

And S123, performing eigenvalue decomposition on the hessian matrix to obtain the principal curvature of the point.

In this embodiment, the projection information based on all neighboring points is used for the quadratic polynomialz=ax ² +bxy+cy ² The fitting mode achieves the determination of the principal curvature.

Because of the local coordinate systemxyThe plane is a plane perpendicular to the normal vector of the point, i.e., a tangent plane to the point on the curved surface. After the adjacent points are projected in the local coordinate system, the projection information is inxShaft and method for producing the sameyThe difference vector of the projection on the axis is a value in 2 tangential directions, and the projection information is thatzThe projection values on the axes are the values of the difference vector in the normal direction, which are the values of the second order vector, so that a second order polynomial can be used for fitting.

And, because the projection is performed on the tangential plane, the adopted second order polynomial pair xAndyshould be 0, i.e. this second order polynomialxAndythe former coefficient is 0 and the constant term is 0, so the quadratic polynomial used in this embodiment isz=ax ² +bxy+cy ² 。

By fitting the quadratic polynomial, coefficients in the polynomial of the degree can be determined, includinga、b、c。

In the present embodiment, the neighboring point pj is set in the local coordinate systemxA shaft(s),yShaft and method for producing the samezProjection coefficients on the axes are respectivelyu _j, v _j ,w _j Determining the coefficienta、b、cThe equation of (2) is as follows:

wherein,Nrepresentation pointsp _i Is a number of adjacent points. Solving for coefficientsa、b、cAfter that, a Hessian matrix of the quadratic polynomial can be calculated, the definition of the Hessian matrix is:

for the second order polynomialz=ax ² +bxy+cy ² The hessian matrix is represented as follows:

and (3) performing eigenvalue decomposition on the Herson matrix of the quadratic polynomial to obtain 2 eigenvalues and corresponding eigenvectors, wherein the 2 eigenvalues are the principal curvatures of the points pi, and the 2 eigenvectors are the principal directions of the points pi.

That is, the principal curvatures of the points pi may include a first principal curvature and a second principal curvature, assuming that the first principal curvature is greater than or equal to the second principal curvature. The larger first main curvature corresponds to the curvature in the direction of the maximum curved surface bending degree, and the smaller second main curvature corresponds to the curvature in the direction of the minimum curved surface bending degree.

In this embodiment, principal curvature and principal direction can be obtained by directly decomposing the characteristic value of the Hessian matrix. Coefficients of the first and second basis forms of the surface corresponding to the polynomial, whereu=x,v=yThenr _u = (1, 0, 0),r _v =(0, 1, 0),r _uu =(0, 0, 2a),r _vv = (0, 0, 2c),r _uv = (0, 0,b),n= (0, 0, 1)。

E=<r _u ,r _u >= 1,F=<r _u ,r _v >= 0,G=<r _v ,r _v >= 1,L=<r _uu ,n>= 2a,M=<r _uv ,n>=b,N=<r _vv ,n>= 2cSubstituting into the coefficient matrix formula of the Weingarten transformation can find that the matrix is a Hessian matrix. Therefore, in this embodiment, the principal curvature and the principal direction can be obtained by directly performing eigenvalue decomposition on the Hessian matrix.

After the principal curvatures of the respective points are obtained, the surface type to which the points belong may be determined based on the principal curvatures, and specifically, the surface type to which the points belong may be determined according to the values of the first principal curvature and the second principal curvature, where the surface type is a plane type, a cylinder-like surface type, a sphere-like surface type, or a saddle-like surface type. The characteristics of various surface types, the main curvature, and the judgment method are shown in table 1.

TABLE 1 multiple facet types

The first principal curvature is noted ask ₁ The second principal curvature is noted ask ₂ Wherein, the method comprises the steps of, wherein,k ₁ for a larger principal curvature of the material,k ₂ is of smaller principal curvature. The planar type as shown in fig. 6, which has little curvature in both main directions, therefore,k ₁ ≈ 0、k ₂ 0, i.e. 1-k ₁ Very large, 1-k ₂ Large, e.g. 1 k ₁ 200 or more, 1%k ₂ 200 or more.

The cylinder-like surface type, as shown in fig. 7, has a large curvature in the first main direction and little curvature in the second main direction, and therefore,k ₁ >0、k ₂ 0, i.e. 1-k ₁ Below a value of 1 +.k ₂ Large, e.g. 1k ₁ Less than 200,1/|k ₂ And 200 or more.

The class of sphere as shown in fig. 8, which has a curvature in both main directions, has the same curvature direction, and therefore,k ₁ >0、k ₂ >0, i.e. 1-k ₁ Below a value of 1 +.k ₂ Below a value, for example 1 +.k ₁ In the presence of a catalyst in the amount of less than 200,k ₂ greater than 0 and 1-k ₂ Less than 200.

The saddle-like surface type, as shown in fig. 9, has bends in both main directions, and the directions of the bends are opposite, and therefore,k ₁ >0、k ₂ <0, i.e. 1-k ₁ Below a value 1/|k ₂ I is below a value, e.g. 1 +.k ₁ In the presence of a catalyst in the amount of less than 200,k ₂ less than 0 and 1-k ₂ <200。

After the surface types of all the points in the point cloud are determined in the above manner, the point cloud segmentation is executed for all the surface types. In this embodiment, for each surface type, by determining a seed point in the points of the surface type, region growth is performed from the seed point, and referring to fig. 10, the steps may include the following sub-steps:

s141, for each point in various surface types existing in the point cloud, adding the point into a surface type point area corresponding to the surface type when more than a preset number of adjacent points with the same surface type as the point exist.

S142, selecting any one point from the surface type point area as an initial seed point.

In this embodiment, each point in the point cloud is traversed, and the traversed point and the neighboring points thereof are counted, and when the method is implemented, the statistics can be performed through an array num_stat. num_stat is an array containing 4 elements. When the point and one of the adjacent points meet the plane condition, num_stat [0] is added with 1, when the point and one of the adjacent points meet the cylindrical-like surface condition, num_stat [1] is added with 1, when the point and one of the adjacent points meet the spherical-like surface condition, num_stat [2] is added with 1, and when the point and one of the adjacent points meet the saddle-like surface condition, num_stat [3] is added with 1.

The position and maximum of the maximum in the num_stat array is then found. When the position of the maximum value is 0 and the maximum value is greater than or equal to a preset number (for example, 3), the point and the adjacent points are similar to the plane characteristics, and the point is added into the plane type point area. When the maximum value is 1 and the maximum value is greater than or equal to 3, which means that the point and its neighboring points are similar to the cylindrical surface feature, the point is added to the cylindrical surface-like type point region. When the maximum value is 2 and the maximum value is greater than or equal to 3, which means that the point and its neighboring points are similar to spherical features, the point is added to the sphere-like type point region. When the maximum value is 3 and the maximum value is greater than or equal to 3, the point and the adjacent points are similar to the saddle surface characteristics, and the point is added into the saddle surface type-like point area. In this way, the points added to the various surface type point areas can be used as seed points subsequently, and the subsequent area growth from the seed points is facilitated.

It can be seen that, for a certain point, if the plane type to which the point belongs is a plane type, the point may be added to the plane type point area when the point has more than a preset number of neighboring points of the same plane type to which the point belongs. The judgment principle is similar for points of other surface types.

On this basis, for each face type point region, one point may be arbitrarily selected from the face type point region as an initial seed point from which region growth is performed.

Referring to fig. 11, in the present embodiment, the process of performing region growing based on the initial seed point may be implemented by:

and S143, sequentially searching adjacent points of the initial seed points, and taking the adjacent points as new seed points if the adjacent points have normal consistency and surface consistency with the initial seed points.

And S144, for the new seed points, searching new adjacent points of the new seed points in sequence, and if the new adjacent points have normal consistency and surface consistency with the new seed points, taking the new adjacent points as the new seed points until all the new seed points are searched out so as to determine the corresponding surfaces.

In this embodiment, the search for the neighboring points of the initial seed point is started with the initial seed point as the starting point. For the searched adjacent point, if the adjacent point has normal consistency and surface consistency with the initial seed point, the adjacent point is indicated to be in the same surface type as the initial seed point and is in the same surface (plane or curved surface). In order to continue the extension to the periphery to search out the entire surface, the search can be continued with the neighboring point as a new seed point and the new seed point as a new starting point.

Similarly, if the new seed point and its new neighboring point satisfy the condition of normal consistency and face consistency, the new neighboring point will serve as the new seed point, and the search is continued with the new seed point. Until the searched neighboring points no longer meet the conditions of normal consistency and surface consistency, i.e. no new neighboring points are used as new seed points, it is indicated that the surface boundary is searched at this time, i.e. the corresponding surface, e.g. a plane or a curved surface, can be determined based on the searched points.

The area growth is carried out by adopting the mode aiming at each surface type existing in the point cloud, so that each surface existing in the point cloud can be determined, namely, the point cloud segmentation is realized.

In this embodiment, when determining whether the neighboring point and the initial seed point have normal consistency and surface consistency, the normal consistency condition and the surface consistency condition may be properly relaxed.

Specifically, if the included angle between the normal vector of the adjacent point and the normal vector of the initial seed point is smaller than the preset included angle, it can be determined that the adjacent point has normal consistency with the initial seed point.

The preset included angle may be, for example, 10 °, that is, when the included angle between the adjacent point and the normal vector of the initial seed point is smaller than 10 °, it may be determined that the adjacent point has normal consistency with the initial seed point.

In addition, if the included angle between the difference vector between the adjacent point and the initial seed point and the normal vector of the initial seed point is within the preset range, determining that the adjacent point and the initial seed point have surface consistency.

An error threshold value, for example, may be set at 5 ° when the surface consistency condition is determined, that is, if the angle between the difference vector between the adjacent point and the initial seed point and the normal vector of the initial seed point is within the range of 85 ° -95 °, it may be determined that the adjacent point and the initial seed point have surface consistency.

In addition, the normal consistency and the face consistency between the new neighboring points and the new seed points are judged in the same manner as described above, and will not be described in detail herein.

In this embodiment, during the region growing process of the curved surface, it can be found that the local neighborhood is still very close to a plane, so that the normal consistency condition and the plane consistency condition after the above-mentioned proper relaxation are used, and the region growing process can be ensured not to be interrupted.

In order to make the point cloud segmentation process provided in this embodiment more clear, the overall process of the point cloud segmentation process will be described below with reference to fig. 12.

A point cloud of the target object is obtained. The target object may be a workpiece model, which may be photographed by a 3D camera to obtain a three-dimensional point cloud.

And filtering the point cloud. In the step, noise points in the point cloud, such as isolated points, miscellaneous points, points floating on the point cloud and the like, can be filtered.

The principal curvature and principal direction of each point in the point cloud are calculated.

The type of face to which the point belongs is determined based on the principal curvature of the point. In this step, the determined surface type includes a plane type, a cylinder-like surface type, a sphere-like surface type, and a saddle-like surface type.

Determining points from the points of each surface type, adding the points into the surface type point area, and selecting initial seed points from the points.

And aiming at the plane type point area, starting from the initial seed point, carrying out area growth, and determining a corresponding plane.

And aiming at the similar cylindrical surface type point area, starting from the initial seed point, carrying out area growth, and determining the corresponding similar cylindrical surface.

And aiming at the sphere-like type point area, starting from the initial seed point, carrying out area growth, and determining the corresponding sphere-like surface.

And aiming at the saddle-like surface type point area, performing area growth from the initial seed point, and determining the corresponding saddle-like surface.

With reference to fig. 13, the above process of filtering the point cloud is described:

for point i in the point cloud, a normal vector for point i is obtained. Wherein, the number of points in the point cloud may be numbered from 0, for example, the total number of points in the point cloud is N, and the number of points may be 0- (N-1) in sequence.

And judging whether the number i is smaller than the total number of points in the point cloud, if so, executing the subsequent flow, and if so, ending the filtering processing flow.

The position and normal vector of point i are determined.

Searching for a plurality of neighboring points of the acquisition point i. Wherein, the numbers of the plurality of adjacent points can be denoted as j, and the numbers start from 0, for example, the total number of the adjacent points is M, and then the numbers of the adjacent points can be 0- (M-1) in sequence.

Judging whether the number j is smaller than the total number of adjacent points, if so, executing the subsequent procedure in sequence, and if so, jumping to the subsequent step to judge whether the number of adjacent points meeting the condition is larger than or equal to the preset number.

The location of the neighboring point j is determined.

The distance between the local planes of the neighboring point j and point i is calculated.

Judging whether the distance is smaller than or equal to a preset distance, wherein the preset distance can be 1.5, if the distance is smaller than or equal to the preset distance, adding 1 to the count of adjacent points meeting the condition, and if the distance is larger than the preset distance, repeating the steps of j=j+1.

And judging whether the number of the adjacent points meeting the preset distance condition is larger than or equal to the preset number when the number j of the adjacent points j is equal to the total number of the adjacent points, if so, adding the point i into the filtered point set, namely, if so, filtering the point i.

Let i=i+1, and return to the above step of determining whether i is smaller than the total number of points in the point cloud, until i is equal to the total number of points in the point cloud, and end the filtering process flow.

Referring to fig. 14 in combination, the following describes the implementation process of calculating the principal curvatures and principal directions of each point in the point cloud in fig. 12:

for point i in the point cloud, a normal vector for point i is determined.

Whether the number i of the point i is smaller than the total point number of the point cloud is judged.

If the number i is smaller than the total point number of the point cloud, determining the position of the point i and determining the normal vector of the point i. If the number i is equal to the total point number of the point cloud, ending the calculation flow.

A local coordinate system is constructed from the normal vector of point i and the local plane.

Searching for a plurality of neighboring points of point i.

For the adjacent point j, judging whether the number j of the adjacent point j is smaller than the total point number of the adjacent point.

If the number j is smaller than the total number of the adjacent points, calculating a difference vector between the point i and the adjacent point j, and obtaining projection information of the difference vector in a local coordinate system. And j=j+1, repeating the above process until the number j is equal to the total number of adjacent points, and jumping to the subsequent process flow.

And fitting to obtain coefficients of a quadratic polynomial based on projection information of all adjacent points.

The hessian matrix is calculated based on coefficients of the polynomial of the degree.

And decomposing the characteristic value of the Heisen matrix, and obtaining the principal curvature and principal direction of the point i according to the obtained characteristic value and the characteristic vector.

Let i=i+1, and return to execute the above procedure until i is equal to the total point number of the point cloud, and jump out of the whole procedure.

Referring to fig. 15 in combination, the following describes the implementation procedure of determining a point-to-face type point region from among the points of each face type based on the face type to which the principal curvature determination point belongs in fig. 12:

for the point i, judging whether the number i of the point i is smaller than the total point number of the point cloud.

If the total number of points is smaller than the total number of points of the point cloud, two principal curvatures of the point i are obtained.

Searching for a plurality of neighboring points of the acquisition point i.

And judging whether the number j of the adjacent point j is smaller than the total point number of the adjacent point.

If the total number of points is smaller than that of the adjacent points, two principal curvatures of the adjacent points are obtained. Otherwise, the process jumps to the subsequent processing flow.

The situation of the point i and its neighbors is counted by the value num_stat.

After traversing the point j adjacent to the point i, the position of the maximum value of the numerical value num_stat is obtained.

If the position of the maximum value of the numerical value num_stat is 0 and the maximum value is greater than a preset number (e.g., 3), the point i is added to the plane type point area.

If the position of the maximum value of the numerical value num_stat is 1 and the maximum value is larger than the preset number, adding the point i into the cylindrical-like surface type point area.

If the position of the maximum value of the numerical value num_stat is 2 and the maximum value is larger than the preset number, adding the point i into the sphere-like type point area.

If the position of the maximum value of the numerical value num_stat is 3 and the maximum value is larger than the preset number, adding the point i into the saddle-surface-like type point area.

Let i=i+1, and return to execute the above steps until i is equal to the total point number of the point cloud, and jump out of the whole flow.

The point cloud segmentation method provided by the embodiment can process the condition that the point cloud contains a plurality of surface types, and only points conforming to a flat area are selected to be added to a seed point each time in the existing area growth segmentation algorithm, so that only the flat area in the point cloud can be segmented. In the point cloud segmentation method provided by the embodiment, the seed points are determined from the points of the various surface types by determining the various surface types in the point cloud, so that the areas of the various surface types can be segmented.

In addition, in this embodiment, the point cloud is filtered first to remove points on the surface of the non-point cloud, such as isolated points, miscellaneous points, points floating on the point cloud, and the like. After filtering treatment, the point cloud only contains points on the surface of the object, so that the influence on subsequent treatment is avoided.

And calculating the principal curvature and principal direction of the points by adopting a local neighborhood 2-order surface fitting mode, and dividing the types of the points by using the principal curvature value to obtain seed points of various surface types.

When a surface of a certain type is segmented, the area growth proceeds from the initial seed point of the surface type, and only the point of the surface type is added and converted into the seed point to continue the growth in the growth process, so that the area growth does not occur on the surfaces of other surface types. Therefore, each surface can be well divided, and the surface of each surface type obtains the corresponding region growth segmentation result, so that the effect of processing the point cloud containing the curved surface is achieved.

Based on the same inventive concept, please refer to fig. 16, which is a schematic diagram illustrating functional modules of the point cloud segmentation apparatus 110 according to the embodiment of the present application, the embodiment may divide the functional modules of the point cloud segmentation apparatus 110 according to the above-mentioned method embodiment. For example, each functional module may be divided corresponding to each function, or two or more functions may be integrated in one processing module. The integrated modules may be implemented in hardware or in software functional modules. It should be noted that, in the embodiment of the present application, the division of the modules is schematic, which is merely a logic function division, and other division manners may be implemented in actual implementation.

For example, in the case of dividing each functional module by the corresponding each function, the point cloud dividing apparatus 110 shown in fig. 16 is only one apparatus schematic. The point cloud segmentation apparatus 110 may include a construction module 111, a fitting module 112, a determination module 113, and a growth module 114, and functions of each functional module of the point cloud segmentation apparatus 110 are described in detail below.

A construction module 111, configured to construct, for each point in a point cloud of a target object, a local coordinate system based on the point, and obtain projection information of a neighboring point adjacent to the point in the local coordinate system; it will be appreciated that the building block 111 may be adapted to perform the above step S11, and reference may be made to the details of the implementation of the building block 111 with respect to the above step S11.

A fitting module 112, configured to obtain a principal curvature of the point according to projection information of points adjacent to the point; it will be appreciated that the fitting module 112 may be used to perform step S12 described above, and reference may be made to the details of step S12 regarding the implementation of the fitting module 112.

A determining module 113, configured to determine, according to the principal curvature, a surface type to which the point belongs; it will be appreciated that the determination module 113 may be used to perform the above step S13, and reference may be made to the details of the implementation of the determination module 113 in relation to the above step S13.

And the growing module 114 is configured to select, for each surface type existing in the point cloud, any one of the points belonging to the surface type as an initial seed point, and determine a corresponding surface by performing region growth from the initial seed point and according to the point belonging to the surface type, so as to implement segmentation of the point cloud. It will be appreciated that the growth module 114 may be used to perform step S14 described above, and reference may be made to step S14 described above for detailed implementation of the growth module 114.

In one possible implementation manner, the point cloud segmentation apparatus 110 further includes a filtering module configured to perform filtering processing on the point cloud in advance, where the filtering module may specifically be configured to:

Constructing a local plane of each point in the point cloud; calculating the distance between each adjacent point of the points and the local plane, and taking the adjacent point with the distance smaller than a preset distance value as a target adjacent point; if the number of the target adjacent points is greater than or equal to a preset threshold value, reserving the points; and if the number of the target adjacent points is smaller than the preset threshold value, filtering the points.

In one possible implementation, the above-mentioned construction module 111 may be used to:

constructing a local plane of the point, and obtaining a normal vector of the point; and constructing and obtaining a local coordinate system based on the normal vector and the two mutually perpendicular direction vectors on the local plane.

calculating, for each neighboring point adjacent to the point, a difference vector between the neighboring point and the point; and projecting the difference vector to each coordinate axis of the local coordinate system to obtain projection information on each coordinate axis.

In one possible implementation, the fitting module 112 may be configured to:

fitting the constructed quadratic polynomial according to projection information of all adjacent points of the points, and determining coefficients in the quadratic polynomial; based on the coefficient of the quadratic polynomial, obtaining a hessian matrix corresponding to the quadratic polynomial; and decomposing the characteristic value of the hessian matrix to obtain the principal curvature of the point.

In one possible implementation, the principal curvatures include a first principal curvature and a second principal curvature, and the determining module 113 may be configured to:

In one possible implementation, the growth module 114 may be configured to:

for each point in various surface types existing in the point cloud, adding the point into a surface type point area corresponding to the surface type when more than a preset number of adjacent points with the same surface type as the point exist; and selecting any one point from the surface type point area as an initial seed point.

In one possible implementation, the growth module 114 may be specifically configured to:

searching adjacent points of the initial seed points in sequence, and taking the adjacent points as new seed points if the adjacent points have normal consistency and surface consistency with the initial seed points; and sequentially searching new adjacent points of the new seed points aiming at the new seed points, and taking the new adjacent points as the new seed points if the new adjacent points have normal consistency and surface consistency with the new seed points until all the new seed points are searched out so as to determine the corresponding surfaces.

In one possible implementation manner, if an included angle between the normal vector of the adjacent point and the normal vector of the initial seed point is smaller than a preset included angle, determining that the adjacent point has normal consistency with the initial seed point;

Referring to fig. 17, a block diagram of an electronic device provided in an embodiment of the present application may be a computer device, etc., and the electronic device includes a memory 130, a processor 120, and a communication module 140. The memory 130, the processor 120, and the communication module 140 are electrically connected directly or indirectly to each other to realize data transmission or interaction. For example, the components may be electrically connected to each other via one or more communication buses or signal lines.

Wherein the memory 130 is used for storing programs or data. The Memory 130 may be, but is not limited to, random access Memory (Random Access Memory, RAM), read Only Memory (ROM), programmable Read Only Memory (Programmable Read-Only Memory, PROM), erasable Read Only Memory (Erasable Programmable Read-Only Memory, EPROM), electrically erasable Read Only Memory (Electric Erasable Programmable Read-Only Memory, EEPROM), etc. The processor 120 is configured to read/write data or programs stored in the memory 130, and perform the point cloud segmentation method provided in any embodiment of the present application. The communication module 140 is used for establishing communication connection between the electronic device and other communication terminals through a network, and is used for receiving and transmitting data through the network.

It should be understood that the structure shown in fig. 17 is merely a schematic diagram of the structure of the electronic device, and the electronic device may further include more or fewer components than those shown in fig. 17, or have a different configuration than that shown in fig. 17.

Further, the embodiment of the application also provides a computer readable storage medium, and the computer readable storage medium stores machine executable instructions, which when executed, implement the point cloud segmentation method provided in the above embodiment.

In particular, the computer readable storage medium can be a general-purpose storage medium, such as a mobile disk, a hard disk, or the like, and the computer program on the computer readable storage medium can execute the above-described point cloud segmentation method when executed. With respect to the processes in the computer readable storage medium and the executable instructions thereof involved when executed, reference is made to the relevant descriptions of the above method embodiments, which are not described in detail herein.

In summary, according to the method and the device for dividing the point cloud provided by the embodiment of the application, for each point in the point cloud of the target object, a local coordinate system is constructed based on the point, and projection information of adjacent points adjacent to the point in the local coordinate system is obtained. And fitting according to projection information of adjacent points of the points to obtain the principal curvature of the points, determining the surface type of the points according to the principal curvature, selecting any one of the points of the surface type as an initial seed point aiming at various surface types existing in the point cloud, and starting from the initial seed point and carrying out region growth according to the points of the surface type to determine the corresponding surface so as to realize the segmentation of the point cloud. In the scheme, the area of various surface types can be segmented by determining the surface type and selecting the seed point from the points of the surface type to perform area growth, and in the area growth process, only the points of the same surface type are added and cannot grow into the areas of other surface types, so that the point cloud comprising the various surface types can be segmented well.

The foregoing is merely illustrative of the present invention, and the present invention is not limited thereto, and any changes or substitutions easily contemplated by those skilled in the art within the scope of the present invention should be included in the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method of point cloud segmentation, the method comprising:

determining the surface type of the point according to the principal curvature;

2. The method of point cloud segmentation according to claim 1, further comprising filtering the point cloud in advance, the step comprising:

Constructing a local plane of each point in the point cloud;

3. The point cloud segmentation method according to claim 1, characterized in that the step of constructing a local coordinate system based on the points includes:

4. The point cloud segmentation method according to claim 1, characterized in that the step of obtaining projection information of neighboring points adjacent to the point in the local coordinate system includes:

5. The method according to claim 1, wherein the fitting the principal curvature of the point according to the projection information of the neighboring points of the point comprises:

6. The point cloud segmentation method of claim 1, wherein the principal curvatures comprise a first principal curvature and a second principal curvature;

7. The point cloud segmentation method according to claim 1, wherein the step of selecting, for each of the face types existing in the point cloud, any one of the points belonging to the face type as an initial seed point, comprises:

8. The method of point cloud segmentation as defined in claim 7, wherein said step of determining a corresponding surface by region growing from said initial seed point and from a point of said surface type comprises:

9. The point cloud segmentation method according to claim 8, wherein if an included angle between a normal vector of the neighboring point and a normal vector of the initial seed point is smaller than a preset included angle, determining that the neighboring point has normal consistency with the initial seed point;

10. A point cloud segmentation apparatus, the apparatus comprising: