CN115424265A - Point cloud semantic segmentation and labeling method and system - Google Patents
Point cloud semantic segmentation and labeling method and system Download PDFInfo
- Publication number
- CN115424265A CN115424265A CN202211025642.8A CN202211025642A CN115424265A CN 115424265 A CN115424265 A CN 115424265A CN 202211025642 A CN202211025642 A CN 202211025642A CN 115424265 A CN115424265 A CN 115424265A
- Authority
- CN
- China
- Prior art keywords
- laser
- points
- labeling
- point
- frame
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/70—Labelling scene content, e.g. deriving syntactic or semantic representations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/20—Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/64—Three-dimensional objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
Abstract
The invention discloses a point cloud semantic segmentation and labeling method and a point cloud semantic segmentation and labeling system, wherein the method comprises the steps of firstly, constructing a three-dimensional scene corresponding to laser radar scanning, and establishing a three-dimensional coordinate system with the laser radar as an origin in the three-dimensional scene; then obtaining the coordinates of the laser point in a three-dimensional coordinate system; and then, selecting the laser points in a rectangular frame or curve frame mode, and finally marking the laser points selected by the frame. Aiming at the characteristics of disorder, sparseness, uneven distribution and the like of data annotation, two point cloud semantic segmentation methods and a point cloud semantic deleting man-machine interaction annotation method are designed, so that an annotator can better accurately and quickly divide point clouds in a three-dimensional scene on a two-dimensional screen, and the problems of label missing, label missing and low efficiency of the annotation method in the prior art are solved.
Description
Technical Field
The invention relates to the technical field of point cloud annotation in data processing, in particular to a point cloud semantic segmentation and annotation method and system.
Background
In recent years, 3D sensors are continuously appearing and developing, so that the application scenes of point cloud data are more and more extensive. Particularly in the field of automatic driving, a widely adopted solution is to obtain point cloud by using a vehicle-mounted laser radar and drive an algorithm model to iterate by using data, so that an automobile has strong perception capability and processes more and more complex scenes, wherein three-dimensional point cloud data labeling is one of indispensable links.
After the spatial coordinates of each sampling point on the surface of the object are obtained, a point set, namely a point cloud, is obtained. Including three-dimensional coordinates, laser reflection intensity, color information, and the like. The point cloud semantic segmentation labeling task is to visualize a labeled object to perform more detailed detection and classification, so that accurate dimension segmentation is obtained. And (3) endowing each point in the point cloud with a corresponding semantic label, then inputting the processed point cloud data into a deep neural network, and training a model. Two modes of manual labeling and semi-automatic labeling are generally adopted: the manual marking completely depends on a large amount of human resources, and the accuracy can be ensured; semi-automatic labeling such as a method, a device, equipment and a product for labeling laser point cloud data of patent application CN202111013628.1 can improve labeling efficiency to a great extent, but the obtained result is not completely ideal, automatic driving has strict requirements on training data in order to ensure safety, and manual adjustment is needed in the later period.
The existing labeling platform mainly aims at two-dimensional image labeling, the three-dimensional point cloud labeling platform is also a blue sea, only a few open source point cloud labeling platforms only have simple rendering and interaction, a labeling worker cannot perform efficient labeling, and more labeling research is a target detection labeling task for a three-dimensional frame, for example, a method and a system for point cloud and image fusion labeling of patent CN202110139777.6, a point Yun Yuyi is divided into the tasks, fine to atomic level labeling research is less, in addition, a laser point cloud labeling method and a device of patent CN201710262366.X provide two labeling modes, the first mode is to click to obtain rays from a camera to a screen click point, obtain points labeled in a section, and then judge whether the distance from the laser point to the rays in the section is smaller than a distance threshold value; the second is to draw a three-dimensional frame to see whether the point is in the frame; according to the first scheme, only the local point is marked, and the second scheme searches whether the laser point falls into the local space or not by drawing a three-dimensional frame, so that the marking efficiency is low.
Therefore, the method in the prior art has the technical problem of low labeling efficiency.
Disclosure of Invention
The invention provides a point cloud semantic segmentation labeling method and a point cloud semantic segmentation labeling system, which are used for solving or at least partially solving the technical problem of low labeling efficiency of the method in the prior art.
In order to solve the technical problem, a first aspect of the present invention provides a point cloud semantic segmentation labeling method, including:
constructing a three-dimensional scene corresponding to laser radar scanning, and establishing a three-dimensional coordinate system with the laser radar as an origin in the three-dimensional scene;
acquiring the coordinates of a laser point in a three-dimensional coordinate system;
adopt the mode of rectangle frame or curve frame to frame the laser point and select, wherein, adopt the mode of rectangle frame to frame the laser point and select and include: responding to the clicking operation of a mouse, determining a rectangular frame in a canvas of a screen according to two points of opposite angles and the direction, converting the coordinate of a laser point in a three-dimensional coordinate system into a two-dimensional screen coordinate, and judging whether the laser point is selected by the rectangular frame according to the relation between the points and the rectangle; adopt the mode of curve frame to carry out the frame selection to the laser point, include: responding to the clicking operation of a mouse, forming a curve in a canvas of a screen, regularly acquiring points of the mouse on the curve by monitoring a moving event of the mouse, replacing a section of curve with line segments within a preset interval, sequentially connecting the points on the curve to form a polygon, converting the coordinate of a laser point in a three-dimensional coordinate system into a two-dimensional screen coordinate, and judging whether the laser point is framed by the polygon corresponding to the curve according to the relation between the point and the polygon;
and marking the laser points selected by the frame.
In one embodiment, converting the coordinates of the laser point in a three-dimensional coordinate system to two-dimensional screen coordinates comprises:
converting the coordinates of the laser point in the three-dimensional coordinate system into the coordinates of standard equipment through camera observation transformation;
and converting the standard equipment coordinates into two-dimensional screen coordinates.
In one embodiment, the determining whether the laser spot is framed by a rectangle frame according to the relationship between the spot and the rectangle includes:
and judging whether the x component of the two-dimensional screen coordinate of the converted laser point is between the minimum value and the maximum value of the rectangular frame in the horizontal x direction and whether the y component of the two-dimensional screen coordinate of the converted laser point is between the minimum value and the maximum value of the rectangular frame in the horizontal y direction, and if the x component and the y component are both in the range, indicating that the laser point is selected by the rectangular frame.
In one embodiment, the method for determining whether the laser point is framed by the polygon corresponding to the curve uses a rotation number algorithm, which includes:
connecting the laser points to be determined with all the points of the polygon;
calculating included angles of two adjacent sides according to a preset sequence, wherein the included angles are directed angles, positive anticlockwise and negative clockwise, and the sides with the included angles smaller than 180 degrees are taken;
accumulating all included angles;
if the accumulated sum of the included angles is 0, the laser point to be determined is positioned outside the polygon and is not selected by a frame; if the accumulated sum of the included angles is integral multiple of 360, the laser point to be determined is located in the polygon and is selected by a frame.
In one embodiment, marking the laser points selected by the frame includes:
and labeling the type of the label and rendering the color of the laser point according to the type of the laser point and the color corresponding to the type.
In one embodiment, after labeling the selected laser points of the frame, the method further comprises: and checking whether the conditions of label missing and label missing occur in a local labeling space formed by the frame selection area, performing frame selection again if the conditions of label missing occur, and deleting the labeling information of the laser points with label missing if the conditions of label missing occur.
Based on the same inventive concept, the second aspect of the present invention provides a point cloud semantic segmentation labeling system, which includes:
the scene construction module is used for constructing a three-dimensional scene corresponding to the scanning of the laser radar and establishing a three-dimensional coordinate system with the laser radar as an origin in the three-dimensional scene;
the three-dimensional coordinate acquisition module is used for acquiring the coordinates of the laser point in a three-dimensional coordinate system;
the frame selection module is used for performing frame selection on the laser points in a rectangular frame or curve frame mode, wherein the frame selection on the laser points in the rectangular frame mode comprises the following steps: responding to the clicking operation of a mouse, determining a rectangular frame in a canvas of a screen according to two points of opposite angles and the direction, converting the coordinate of a laser point in a three-dimensional coordinate system into a two-dimensional screen coordinate, and judging whether the laser point is framed and selected by the rectangular frame according to the relation between the points and the rectangle; the method for selecting the laser points by adopting a curve frame mode comprises the following steps: responding to the clicking operation of a mouse, forming a curve in a canvas of a screen, regularly acquiring points of the mouse on the curve by monitoring a moving event of the mouse, replacing a section of curve with line segments within a preset interval, sequentially connecting the points on the curve to form a polygon, converting the coordinate of a laser point in a three-dimensional coordinate system into a two-dimensional screen coordinate, and judging whether the laser point is framed by the polygon corresponding to the curve according to the relation between the point and the polygon;
and the marking module is used for marking the laser points selected by the frame.
In an embodiment, the system further comprises a labeling data interaction module, which performs visual rendering on the received laser point cloud data and the corresponding label information and color information, and feeds back a labeling result to the server after the labeling is completed.
Based on the same inventive concept, a third aspect of the present invention provides a computer-readable storage medium having stored thereon a computer program which, when executed, performs the method of the first aspect.
Based on the same inventive concept, a fourth aspect of the present invention provides a computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the method of the first aspect when executing the program.
Compared with the prior art, the invention has the advantages and beneficial technical effects as follows:
the invention provides a point cloud semantic segmentation and labeling method, which comprises the steps of constructing a three-dimensional coordinate system with a laser radar as an origin, acquiring coordinates of a laser point in the three-dimensional coordinate system, adopting two different frame selection modes, converting the coordinates of the laser point in the three-dimensional coordinate system into two-dimensional screen coordinates by the rectangular frame selection mode, determining a rectangular frame according to two diagonal points and the direction between the two points, converting any curve into a polygon by the curve frame selection mode, namely, drawing a two-dimensional closed graph (a rectangular frame or a curve) in a two-dimensional screen to find out whether the laser point falls into the graph or not, on one hand, directly drawing the approximate outline of the point cloud of a scanned target object, on the other hand, the efficiency of the interaction mode of the two-dimensional graph in the two-dimensional screen is higher, and therefore, the labeling efficiency is integrally improved.
Drawings
In order to more clearly illustrate the embodiments or technical solutions of the present invention, the drawings used in the embodiments or technical solutions in the prior art are briefly introduced below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to these drawings without creative efforts.
FIG. 1 is a schematic overall flow chart of a point cloud semantic segmentation labeling method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram illustrating selectable interactive tool icons provided in the point cloud semantic segmentation labeling system according to an embodiment of the present invention;
FIG. 3 is a schematic diagram illustrating interaction of rectangular box semantic segmentation in the point cloud semantic segmentation labeling system according to an embodiment of the present invention;
FIG. 4 is an interactive schematic diagram of rectangular box semantic deletion in the point cloud semantic segmentation labeling system according to the embodiment of the invention;
FIG. 5 is a diagram illustrating interaction of curve box semantic segmentation in a point cloud semantic segmentation labeling system according to an embodiment of the present invention;
FIG. 6 is a flow chart of labeling with a point cloud semantic segmentation labeling system in an exemplary embodiment;
FIG. 7 is a point cloud semantic segmentation annotation interface provided in an embodiment.
Detailed Description
The inventor of the application finds out through a great deal of research and practice that:
the patent with the application number of 201710262366.X discloses a laser point cloud labeling method and device, and two labeling modes are provided, wherein the first mode is that rays from a camera to a screen click point are obtained through clicking, a point labeled in an interval is obtained, and whether the distance between a laser point and the rays in the interval is smaller than a distance threshold value or not is judged; the second is to draw a three-dimensional frame and determine whether the point is inside the frame. Although the first method can accurately pick up the points, the target objects in the real scene are relatively concentrated and have a certain outline, and the labeling efficiency is low only by picking up a certain number of points; the second method searches whether the laser point falls into the local space or not by drawing a three-dimensional frame, although a certain outline can be drawn, the interaction mode of drawing the three-dimensional frame is more complicated, because a camera visual angle can cause a certain visual error, and the outline of a joint object needs to be removed by adjusting in the later stage, so that the labeling efficiency is not high.
Based on the point cloud semantic segmentation and labeling method, the three-dimensional coordinates of the point cloud are firstly converted into two-dimensional screen coordinates, a two-dimensional closed graph (a rectangular frame or a curve) is drawn in a two-dimensional screen to find whether a laser point falls into the graph or not, on one hand, the approximate outline of the point cloud of a scanned target object can be directly drawn, and on the other hand, the interaction mode of drawing the two-dimensional graph in the two-dimensional screen is better in experience and higher in efficiency. The point cloud semantic segmentation algorithm needs a large amount of real and accurate semantic segmentation data and needs a high-efficiency and humanized interaction scheme. On the other hand, from the development of labeling engineering, the dependence of the automatic driving enterprises on data reaches the stage of mass production, high efficiency and low cost, the individualized and individual data production of workshops cannot meet the existing data requirements, the efficiency of labeling is urgently needed to be improved through industrialized, flow-based and large-scale data product production, and the labeling platform can integrate the database design, data processing, quality control, data safety management and the like, so that the management and execution of one-stop processing of point cloud data are integrated. Therefore, the invention also provides a point cloud semantic segmentation and labeling system.
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Example one
The embodiment of the invention provides a point cloud semantic segmentation and labeling method, which comprises the following steps:
constructing a three-dimensional scene corresponding to laser radar scanning, and establishing a three-dimensional coordinate system with the laser radar as an origin in the three-dimensional scene;
acquiring the coordinates of a laser point in a three-dimensional coordinate system;
adopt the mode of rectangle frame or curve frame to carry out the frame selection to the laser point, wherein, adopt the mode of rectangle frame to carry out the frame selection to the laser point and include: responding to the clicking operation of a mouse, determining a rectangular frame in a canvas of a screen according to two points of opposite angles and the direction, converting the coordinate of a laser point in a three-dimensional coordinate system into a two-dimensional screen coordinate, and judging whether the laser point is selected by the rectangular frame according to the relation between the points and the rectangle; the method for selecting the laser points by adopting a curve frame mode comprises the following steps: responding to the clicking operation of a mouse, forming a curve in a canvas of a screen, regularly acquiring points of the mouse on the curve by monitoring a moving event of the mouse, replacing a section of curve with line segments within a preset interval, sequentially connecting the points on the curve to form a polygon, converting the coordinate of a laser point in a three-dimensional coordinate system into a two-dimensional screen coordinate, and judging whether the laser point is framed by the polygon corresponding to the curve according to the relation between the point and the polygon;
and marking the laser points selected by the frame.
The invention aims to provide a solution for three-dimensional point cloud semantic segmentation and labeling, and designs two point cloud semantic segmentation methods (namely rectangular frame selection and curve frame selection) and a point cloud semantic deletion man-machine interaction labeling method aiming at the characteristics of disorder, sparseness, uneven distribution and the like of data labeling, so that a labeler can divide point clouds in a three-dimensional scene accurately and quickly on a two-dimensional screen better, and the problems of label missing, label error and low efficiency generated by a labeling method in the prior art are solved; correspondingly, the embodiment of the invention also provides a complete labeling system which can meet the labeling requirement of the laser radar point cloud semantic segmentation task in the current automatic driving.
Please refer to fig. 1, which is a schematic overall flow chart of a point cloud semantic segmentation labeling method according to an embodiment of the present invention.
Specifically, the coordinates of the laser spot in the three-dimensional coordinate system can be converted into two-dimensional screen coordinates through coordinate conversion, and after the two-dimensional screen coordinates of the laser spot are obtained, whether the laser spot falls in the rectangular frame can be judged according to the relationship between the two-dimensional screen coordinates of the laser spot and a drawn rectangle.
The time interval can be set according to the actual situation, for example, 20ms is selected. The preset interval can also be set according to actual conditions.
In one embodiment, converting the coordinates of the laser point in a three-dimensional coordinate system to two-dimensional screen coordinates comprises:
converting the coordinates of the laser point in the three-dimensional coordinate system into the coordinates of standard equipment through camera observation transformation;
and converting the standard equipment coordinates into two-dimensional screen coordinates.
In a specific implementation process, the coordinate transformation is realized as follows:
1. converting the three-dimensional coordinates through a camera view to obtain the coordinates of the standard equipment
First, a View transformation is performed, and the model is projected onto the camera using an observation (View) matrix, denoted as V. Defining the observation matrix (i.e. the camera state), the following parameters are required:
(1) camera position Pos = [ x = [ x ] p ，y p ，z p ] T
(2) Above camera U = [ x ] u ，y u ，z u ] T
(3) Camera viewing direction L = [ x = l ，y l ，z l ] T
(4) Right camera R = L × U = [ x = [ x ] r ，y r ，z r ] T
Wherein X, Y and Z represent X-axis, Y-axis and Z-axis three-dimensional coordinates, pos, U, L and R respectively represent camera position coordinates, upper vector, observation direction vector and right vector, and subscripts p, U, L and R also representCorresponding to these 4 quantities, e.g. x p ，y p ，z p Three-axis coordinates representing the camera position.
To sense the translation and rotation transformation of the object, the observation matrix needs to be reversed, i.e. from the observation matrix to the initial observation matrix (denoted as V) 0 ) And (4) inverse transforming. For the translation matrix V T It is written in homogeneous form as follows:
for the rotation matrix V R The inverse transformation is obtained as follows:
R：[x l ，y l ，z l ] T ×[x u ，y u ，z u ] T ＝[x r ，y r ，z r ] T →[1，0，0] T (X-axis direction)
U：[x u ，y u ，z u ] T →[0，1，0] T (Y-axis direction)
L：[-x l ，-y l ，-z l ] T →[0，0，1] T (Z-axis direction)
Therefore, there are:
the rotation matrix is again an orthogonal matrix, and the obtained observation matrix is:
secondly, performing projection transformation to convert the image into an observation space coordinate system with the camera as an origin, rendering and seeing only an object positioned on a view cone by the camera, and eliminating fixed points outside the range to be divided into orthogonal projection and perspective projection O Comprises the following steps:
wherein l and r respectively represent the positions of the left plane and the right plane of the view cone on the X axis, t and b respectively represent the positions of the upper plane and the lower plane of the view cone on the Y axis, and n and f respectively represent the positions of the near plane and the far plane of the view cone on the Z axis.
The perspective projection is changed into orthogonal projection by extruding a far plane f, and a perspective projection matrix P P Comprises the following steps:
p denotes the projection matrix, the subscripts O and P denote the orthogonal projection and the perspective projection, respectively, and P → O denotes the transition from the perspective projection to the orthogonal projection, i.e., P O Representing an orthogonal projection matrix, P P Representing a perspective projection matrix, P P→O Representing a projection matrix that is transformed from perspective projection to orthogonal projection.
In summary, the view space is converted to the clipping space such that the transformed x, y, z components are all located at [ -w, w]Within this range, it trims the coordinate C uniformly c Comprises the following steps:
c denotes the coordinates, x, y, z, w denote the homogeneous coordinate components, the subscript clip denotes the clip coordinates, and the subscript p denotes the laser point coordinates. And finally mapping the clipping coordinates to a space of [ -1,1] by perspective division, wherein the standardized equipment coordinates are as follows:
the subscript ndc denotes the normalized device coordinates.
2. Converting standardized device coordinates to screen coordinates
The second step is to perform real projection after finishing all the cutting and mapping operations, i.e. projection to the screen space, so as to obtain real two-dimensional pixel coordinates:
Screen x ＝(1+C ndc .x)/2×PixelWidth
Screen y ＝(1+C ndc .y)/2×PixelHeight
screen indicates Screen coordinates (Screen) x 、Screen y Respectively the abscissa and the ordinate of the screen), pixelWidth and PixelHeight indicate the width and height of the window, respectively. C ndc X and C ndc Y are the x-axis and y-axis coordinates of the normalized coordinates, respectively.
In one embodiment, the determining whether the laser spot is framed by a rectangle frame according to the relationship between the spot and the rectangle includes:
and judging whether the x component of the two-dimensional screen coordinate of the converted laser point is between the minimum value and the maximum value of the rectangular frame in the horizontal x direction and whether the y component of the two-dimensional screen coordinate of the converted laser point is between the minimum value and the maximum value of the rectangular frame in the horizontal y direction, and if the x component and the y component are both in the range, indicating that the laser point is selected by the rectangular frame.
In the specific implementation process, after the two-dimensional screen coordinates of the laser point are obtained, whether the laser point falls into the rectangular frame is judged according to the following formula:
rectangle.x min ＜Screen x ＜rectangle.x max
&&rectangle.y min ＜Screen y ＜rectangle.y max
rectangle represents a rectangular box, and subscripts min and max represent the small and large values of the rectangular box in the horizontal x-direction and vertical y-direction, respectively.
In one embodiment, the method for determining whether the laser point is framed by a polygon corresponding to the curve uses a rotation number algorithm, which includes:
connecting the laser points to be determined with all the points of the polygon;
calculating included angles of two adjacent sides according to a preset sequence, wherein the included angles are directed angles, positive anticlockwise and negative clockwise, and the sides with the included angles smaller than 180 degrees are taken;
accumulating all included angles;
if the accumulated sum of the included angles is 0, the laser point to be determined is positioned outside the polygon and is not selected by a frame; if the accumulated sum of the included angles is integral multiple of 360, the laser point to be determined is located in the polygon and is selected by a frame.
Specifically, the interactive mode of curve box selection is to draw any curve box in a canvas of a two-dimensional screen through a mouse to select a point cloud. The curve is described into a polygon by using the idea of 'replacing the curve with straight curve', then a three-dimensional coordinate point is converted into a two-dimensional screen coordinate, and whether the point is in the curve or not is judged by an algorithm of whether the point is in a closed polygon or not.
Firstly, a curve is drawn on a canvas of a screen through mouse operation, points of the mouse on the curve are obtained at regular time by monitoring moving events of the mouse in the process of drawing the curve, in the embodiment, the time interval is selected to be 20 milliseconds, a segment of the curve is replaced by a line segment in a smaller interval, and finally a polygon formed by a plurality of points can be used for approximating the closed curve. If the plotted curve does not form a closed area, the first point and the last point are connected.
Besides describing a curve, it is also necessary to know how to draw the curve and record information such as the shape and position of the curve. In the present embodiment, a high-order bezier curve is adopted, and key points in the set are taken, and in order to avoid the problem of factorial multiplication, the number of the key points is also taken within the control range, and the key points are grouped.
Second, it is determined whether the decision point is within the closed polygon. The prior art generally implements two algorithms: ray casting algorithm (also called even-odd algorithm) and round-robin algorithm (winding number algorithm). The ray casting algorithm is based on the theorem of the approximate curve: a simple closed polygon must divide a plane into an interior and an exterior, and a line segment connecting any point inside and any point outside must pass through the edge of the closed figure. The ray casting algorithm is to guide a ray from a point to infinity, calculate the times of intersection of the ray and the edge of the polygon, if the times are odd, the point is inside the polygon, and if the times are even, the point is outside the polygon. The algorithm cannot process the special situations that the ray intersects with the vertex of the polygon and the ray coincides with the edge of the polygon, so that an improved ray projection algorithm is provided, but the performance consumption is high, and the number of calculation points is large and the requirement of a marker on real-time interaction experience is high, so that the rotation number algorithm is adopted in the implementation mode, and the calculation efficiency is high. The principle is the total number of times the closed curve in the plane has passed around this point counterclockwise. When the number of revolutions equals 0, the point is outside the curve. This algorithm is also referred to as non-zero rule (non-rule). The calculation steps are as follows:
(1) connecting the point with all points of the polygon;
(2) calculating the included angles of two adjacent sides in sequence;
(3) the specified included angles are directed angles, the anticlockwise included angles are positive, the clockwise included angles are negative, and the sides with included angles smaller than 180 degrees are taken;
(4) accumulating all included angles;
(5) if the sum of the included angles is 0, the point is out of the polygon; if the sum of the angles is an integer multiple of 360, then it is within the polygon.
Finally, all the laser points are traversed cyclically, converting them from three-dimensional coordinates to two-dimensional coordinates, all the conversion processes of which have been described previously, through the points falling within the closed curve by the revolution number algorithm.
The selection of points can be completed through two interactions of rectangular frame selection and curve frame selection, then corresponding semantic tags are attached, and if modification requirements exist, one interaction mode can be selected optionally to re-cover the frame selection.
In one embodiment, marking the laser points selected by the frame includes:
and labeling the type of the label and rendering the color of the laser point according to the type of the laser point and the color corresponding to the type.
In one embodiment, after marking the laser point selected by the frame, the method further includes: and checking whether the conditions of label missing and label missing occur in a local labeling space formed by the frame selection area, performing frame selection again if the conditions of label missing occur, and deleting the labeling information of the laser points of the label missing if the conditions of label missing occur.
Example two
Based on the same inventive concept, the embodiment provides a point cloud semantic segmentation labeling system, which includes:
the scene construction module is used for constructing a three-dimensional scene corresponding to the scanning of the laser radar and establishing a three-dimensional coordinate system with the laser radar as an origin in the three-dimensional scene;
the three-dimensional coordinate acquisition module is used for acquiring the coordinates of the laser point in a three-dimensional coordinate system;
the frame selection module is used for performing frame selection on the laser points in a rectangular frame or curve frame mode, wherein the frame selection on the laser points in the rectangular frame mode comprises the following steps: converting the coordinate of the laser spot in a three-dimensional coordinate system into a two-dimensional screen coordinate, responding to the clicking operation of a mouse, determining a rectangular frame in the canvas of the screen according to two points of opposite angles and the direction, and judging whether the laser spot is framed and selected by the rectangular frame according to the relation between the points and the rectangular frame; the method for selecting the laser points by adopting a curve frame mode comprises the following steps: responding to the clicking operation of a mouse, forming a curve in a canvas of a screen, regularly acquiring points of the mouse on the curve by monitoring a moving event of the mouse, replacing a section of curve with line segments within a preset interval, sequentially connecting the points on the curve to form a polygon, converting the coordinate of a laser point in a three-dimensional coordinate system into a two-dimensional screen coordinate, and judging whether the laser point is framed by the polygon corresponding to the curve according to the relation between the point and the polygon;
and the marking module is used for marking the laser points selected by the frame.
Please refer to fig. 6, which is a flowchart illustrating labeling performed by the point cloud semantic segmentation labeling system in an embodiment.
Specifically, the process of semantic annotation by using the point cloud semantic segmentation and annotation system is as follows:
step 1: and uploading the point cloud data files in batches, such as the pcd format point cloud.
Step 2: and (4) creating a point cloud semantic segmentation labeling task by a project manager, subpackaging the uploaded point cloud task, and distributing to different labeling personnel.
And step 3: and configuring a template, setting different label names and identifiers according to the predefined point cloud semantic types, and selecting corresponding colors of the labels, so that the labels can be distinguished conveniently in the labeling process.
And 4, step 4: reading and loading the point cloud files with corresponding formats, constructing a three-dimensional scene corresponding to laser radar scanning in a terminal (front end), establishing a three-dimensional coordinate system with the laser radar as an origin, traversing all laser points to acquire corresponding position information, and visualizing the position information through a particle system. The embodiment of the invention does not strictly limit the way of constructing the three-dimensional scene, and can adopt various technical schemes such as WebGL, OSG, STK and the like.
And 5: point cloud semantic labeling, which is mainly used for labeling and dyeing laser points, and labeling personnel can intuitively and quickly judge and select the laser points of each semantic type. The whole labeling interface is divided into three large blocks: 1) The main operation area provides three-dimensional construction rendering and interaction of point clouds on a two-dimensional screen; 2) The attribute column is divided into a label column and a list column, the label column is all semantic labels configured for the template, semantic information is attached to each interactive operation in the process of marking, and the list column is an operation record list and supports addition, deletion, modification and check; 3) The tool bar comprises four marking tools designed in the front, a cloud point drawing tool, a bird's-eye view/free visual angle switching tool, a point particle size adjusting tool, a frame control tool and the like.
Fig. 2 to 5 are schematic diagrams of selectable interactive tool icons, rectangular box semantic segmentation interaction, rectangular box semantic deletion interaction and curve box semantic segmentation interaction provided in the point cloud semantic segmentation labeling system in the embodiment of the present invention, respectively. FIG. 7 is a point cloud semantic segmentation annotation interface provided in an embodiment.
The specific labeling function can be realized through the interaction steps of S1-S4:
s1, selecting a marking tool in a toolbar to frame laser points;
s2, setting the label type of the selected laser point in a label column, and distinguishing according to the marking color corresponding to the target type;
optionally, a flexible labeling sequence is provided, the label type is selected first, and then the framing and dyeing of the laser points are performed.
S3, checking whether the local labeling space has a missing mark and a wrong mark, wherein the missing mark can be selected again in a frame mode, and the wrong mark can be selected to be an erasing tool;
preferably, the orthogonal/projection camera is selected through aerial view/free visual angle switching, so that a marker can understand the three-dimensional space conveniently, the position, the direction and the visual angle of the camera can be adjusted to check the laser point of the three-dimensional scene, a cloud point drawing tool is selected for zooming, the marker can be positioned to a position to be marked conveniently, and the interaction experience of the marker is enhanced.
And S4, repeating the steps to ensure that the semantic information of the laser point is accurate and comprehensive as much as possible, checking the marking record of the list column, and confirming that the next frame is clicked by no mistake until the task is completed.
Step 6: and (3) performing annotation data interaction, storing in a form of label type-color-point sequence group, and rendering all points visually by receiving the laser point cloud data sent by the server and the corresponding label information and color information at the front end. Meanwhile, in the labeling process, the result is fed back to the server, and a reasonable and efficient interface is designed for data interaction in consideration of large data interaction amount and uniqueness of point cloud semantics, so that the labeling experience is prevented from being influenced by network delay.
And 7: and (4) exporting the labeling result, wherein the point sequence invariance of the original point cloud file is considered, each point semantic labeling result is corresponded to the original file through a serial number, and finally, the point sequence invariance is exported in a specified format.
Generally speaking, the invention designs various point cloud semantic man-machine interaction labeling methods aiming at the characteristics of disorder, sparseness, uneven distribution and the like of data labeling, so that a labeling person can better accurately and quickly divide point clouds in a three-dimensional scene on a two-dimensional screen, and the problems of label missing, label error and low efficiency caused by manual labeling are solved; the embodiment of the invention provides a set of complete marking system which is put into production of data products and provides basic data service for vehicle enterprises. Million-level point cloud labeling is supported, the labeling interaction delay is within 200ms, the corresponding speed of a conventional data interface is within 500ms, the point cloud operating tools are 4, various semantic type labeling is supported, the platform starting speed is 3s, the maximum number of online people to be labeled is 200, and the labeling accuracy can reach more than 98%.
Since the system introduced in the second embodiment of the present invention is a system adopted for implementing the point cloud semantic segmentation labeling method in the first embodiment of the present invention, based on the method introduced in the first embodiment of the present invention, those skilled in the art can understand the specific structure and deformation of the system, and thus, details are not described here. All systems adopted by the method in the first embodiment of the invention belong to the protection scope of the invention.
EXAMPLE III
Based on the same inventive concept, the present invention also provides a computer-readable storage medium, on which a computer program is stored, which when executed, implements the method as described in the first embodiment.
Since the computer-readable storage medium introduced in the third embodiment of the present invention is a computer-readable storage medium used for implementing the point cloud semantic segmentation labeling method in the first embodiment of the present invention, based on the method introduced in the first embodiment of the present invention, persons skilled in the art can understand the specific structure and deformation of the computer-readable storage medium, and thus details are not described here. Any computer readable storage medium used in the method of the first embodiment of the present invention falls within the intended scope of the present invention.
Example four
Based on the same inventive concept, the present application further provides a computer device, which includes a storage, a processor, and a computer program stored on the storage and running on the processor, and when the processor executes the computer program, the method in the first embodiment is implemented.
Since the computer device introduced in the fourth embodiment of the present invention is a computer device used for implementing the point cloud semantic segmentation labeling method in the first embodiment of the present invention, based on the method introduced in the first embodiment of the present invention, those skilled in the art can understand the specific structure and deformation of the computer device, and thus, details are not described herein again. All the computer devices used in the method in the first embodiment of the present invention are within the scope of the present invention.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including preferred embodiments and all such alterations and modifications as fall within the scope of the invention.
It will be apparent to those skilled in the art that various modifications and variations can be made in the embodiments of the present invention without departing from the spirit or scope of the embodiments of the invention. Thus, if such modifications and variations of the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to encompass such modifications and variations.
Claims (10)
1. A point cloud semantic segmentation labeling method is characterized by comprising the following steps:
constructing a three-dimensional scene corresponding to laser radar scanning, and establishing a three-dimensional coordinate system with the laser radar as an origin in the three-dimensional scene;
acquiring the coordinates of a laser point in a three-dimensional coordinate system;
adopt the mode of rectangle frame or curve frame to carry out the frame selection to the laser point, wherein, adopt the mode of rectangle frame to carry out the frame selection to the laser point and include: responding to the clicking operation of a mouse, determining a rectangular frame in a canvas of a screen according to two points of opposite angles and the direction, converting the coordinate of a laser point in a three-dimensional coordinate system into a two-dimensional screen coordinate, and judging whether the laser point is framed and selected by the rectangular frame according to the relation between the points and the rectangle; the method for selecting the laser points by adopting a curve frame mode comprises the following steps: forming a curve in a canvas of a screen in response to clicking operation of a mouse, acquiring points of the mouse on the curve at regular time by monitoring a moving event of the mouse, replacing a section of curve with line segments within a preset interval, sequentially connecting the points on the curve to form a polygon, converting coordinates of a laser point in a three-dimensional coordinate system into coordinates of a two-dimensional screen, and judging whether the laser point is framed and selected by the polygon corresponding to the curve according to the relationship between the points and the polygon;
and marking the laser points selected by the frame.
2. The point cloud semantic segmentation labeling method of claim 1, wherein converting coordinates of the laser point in a three-dimensional coordinate system into two-dimensional screen coordinates comprises:
converting the coordinates of the laser point in the three-dimensional coordinate system into the coordinates of standard equipment through camera observation transformation;
and converting the standard equipment coordinates into two-dimensional screen coordinates.
3. The point cloud based semantic segmentation labeling method of claim 1, wherein judging whether the laser point is framed by a rectangular frame according to the relation between the point and the rectangle comprises:
and judging whether the x component of the two-dimensional screen coordinate of the converted laser point is between the minimum value and the maximum value of the rectangular frame in the horizontal x direction and whether the y component of the two-dimensional screen coordinate of the converted laser point is between the minimum value and the maximum value of the rectangular frame in the horizontal y direction, and if the x component and the y component are both in the range, indicating that the laser point is selected by the rectangular frame.
4. The point cloud-based semantic segmentation labeling method of claim 1, wherein the method for judging whether the laser point is framed by a polygon corresponding to the curve adopts a revolution number algorithm, and the algorithm comprises the following steps:
connecting the laser points to be determined with all the points of the polygon;
calculating included angles of two adjacent sides according to a preset sequence, wherein the included angles are directed angles, positive anticlockwise and negative clockwise, and the sides with the included angles smaller than 180 degrees are taken;
accumulating all included angles;
if the accumulated sum of the included angles is 0, the laser point to be determined is positioned outside the polygon and is not selected by a frame; if the accumulated sum of the included angles is integral multiple of 360, the laser point to be determined is located in the polygon and is selected by a frame.
5. The point cloud based semantic segmentation labeling method of claim 1, wherein labeling the laser points selected from the frame comprises:
and labeling the type of the label and rendering the color of the laser point according to the type of the laser point and the color corresponding to the type.
6. The point cloud based semantic segmentation labeling method of claim 1, wherein after labeling the laser points selected by the box, the method further comprises: and checking whether the conditions of label missing and label missing occur in a local labeling space formed by the frame selection area, performing frame selection again if the conditions of label missing occur, and deleting the labeling information of the laser points of the label missing if the conditions of label missing occur.
7. A point cloud semantic segmentation labeling system is characterized by comprising:
the scene construction module is used for constructing a three-dimensional scene corresponding to the scanning of the laser radar and establishing a three-dimensional coordinate system with the laser radar as an origin in the three-dimensional scene;
the three-dimensional coordinate acquisition module is used for acquiring the coordinates of the laser point in a three-dimensional coordinate system;
the frame selection module is used for performing frame selection on the laser points in a rectangular frame or curve frame mode, wherein the frame selection on the laser points in the rectangular frame mode comprises the following steps: responding to the clicking operation of a mouse, determining a rectangular frame in a canvas of a screen according to two points of opposite angles and the direction, converting the coordinate of a laser point in a three-dimensional coordinate system into a two-dimensional screen coordinate, and judging whether the laser point is selected by the rectangular frame according to the relation between the points and the rectangle; the method for selecting the laser points by adopting a curve frame mode comprises the following steps: forming a curve in a canvas of a screen in response to clicking operation of a mouse, acquiring points of the mouse on the curve at regular time by monitoring a moving event of the mouse, replacing a section of curve with line segments within a preset interval, sequentially connecting the points on the curve to form a polygon, converting coordinates of a laser point in a three-dimensional coordinate system into coordinates of a two-dimensional screen, and judging whether the laser point is framed and selected by the polygon corresponding to the curve according to the relationship between the points and the polygon;
and the marking module is used for marking the laser points selected by the frame.
8. The point cloud semantic segmentation-based labeling system of claim 7, further comprising a labeling data interaction module for performing visual rendering on the received laser point cloud data and corresponding label information and color information, and feeding back a labeling result to the server after the labeling is completed.
9. A computer-readable storage medium, on which a computer program is stored, which program, when executed, carries out the method of any one of claims 1 to 6.
10. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method according to any of claims 1 to 6 when executing the program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211025642.8A CN115424265A (en) | 2022-08-25 | 2022-08-25 | Point cloud semantic segmentation and labeling method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211025642.8A CN115424265A (en) | 2022-08-25 | 2022-08-25 | Point cloud semantic segmentation and labeling method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115424265A true CN115424265A (en) | 2022-12-02 |
Family
ID=84198736
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211025642.8A Withdrawn CN115424265A (en) | 2022-08-25 | 2022-08-25 | Point cloud semantic segmentation and labeling method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115424265A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116309962A (en) * | 2023-05-10 | 2023-06-23 | 倍基智能科技(四川)有限公司 | Laser radar point cloud data labeling method, system and application |
| CN116597149A (en) * | 2023-06-07 | 2023-08-15 | 宁波博登智能科技有限公司 | Point cloud semantic segmentation system and method based on industrial production |
-
2022
- 2022-08-25 CN CN202211025642.8A patent/CN115424265A/en not_active Withdrawn
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116309962A (en) * | 2023-05-10 | 2023-06-23 | 倍基智能科技(四川)有限公司 | Laser radar point cloud data labeling method, system and application |
| CN116309962B (en) * | 2023-05-10 | 2023-09-26 | 倍基智能科技(四川)有限公司 | Laser radar point cloud data labeling method, system and application |
| CN116597149A (en) * | 2023-06-07 | 2023-08-15 | 宁波博登智能科技有限公司 | Point cloud semantic segmentation system and method based on industrial production |
| CN116597149B (en) * | 2023-06-07 | 2024-02-06 | 宁波博登智能科技有限公司 | Point cloud semantic segmentation system and method based on industrial production |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108648269B (en) | Method and system for singulating three-dimensional building models | |
| Han et al. | Potential of big visual data and building information modeling for construction performance analytics: An exploratory study | |
| US11074701B2 (en) | Interior photographic documentation of architectural and industrial environments using 360 panoramic videos | |
| US10854006B2 (en) | AR-enabled labeling using aligned CAD models | |
| CN110176078B (en) | Method and device for labeling training set data | |
| CN115424265A (en) | Point cloud semantic segmentation and labeling method and system | |
| US11200457B2 (en) | System and method using augmented reality for efficient collection of training data for machine learning | |
| EP1267309B1 (en) | 3D Computer Modelling Apparatus | |
| CN111300416B (en) | Modularized reconfigurable robot planning simulation method and system based on augmented reality | |
| Zollmann et al. | Interactive 4D overview and detail visualization in augmented reality | |
| WO2020179065A1 (en) | Image processing device, image processing method, and recording medium | |
| EP3971829B1 (en) | Cutting method, apparatus and system for point cloud model | |
| CN112633657A (en) | Construction quality management method, device, equipment and storage medium | |
| CN115345822A (en) | Automatic three-dimensional detection method for surface structure light of aviation complex part | |
| CN110443862A (en) | Lithologic map filling method and system based on unmanned aerial vehicle and electronic equipment | |
| EP4068210A1 (en) | System and method for automated estimation of 3d orientation of a physical asset | |
| CN114882272A (en) | Fusion analysis method for aerial manufacturing full-angle projection image surface patch attributes | |
| CN114612393A (en) | Monocular vision-based reflective part pose estimation method | |
| CN113870432A (en) | Bus-based non-cooperative ship target full-automatic three-dimensional reconstruction system | |
| Pal et al. | Activity-level construction progress monitoring through semantic segmentation of 3D-informed orthographic images | |
| CN108876906A (en) | The method and device of virtual three-dimensional model is established based on the global plane optimizing of cloud | |
| CN116978010A (en) | Image labeling method and device, storage medium and electronic equipment | |
| CN110334818B (en) | Method and system for automatically identifying pipeline | |
| CN115546422A (en) | Building three-dimensional model construction method and system and electronic equipment | |
| CN113808186B (en) | Training data generation method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20221202 |
|
| WW01 | Invention patent application withdrawn after publication |