CN113160327A - Method and system for realizing point cloud completion - Google Patents
Method and system for realizing point cloud completion Download PDFInfo
- Publication number
- CN113160327A CN113160327A CN202110381744.2A CN202110381744A CN113160327A CN 113160327 A CN113160327 A CN 113160327A CN 202110381744 A CN202110381744 A CN 202110381744A CN 113160327 A CN113160327 A CN 113160327A
- Authority
- CN
- China
- Prior art keywords
- point cloud
- sparse
- depth map
- training
- laser
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 46
- 238000012549 training Methods 0.000 claims description 84
- 238000012545 processing Methods 0.000 claims description 16
- 238000004364 calculation method Methods 0.000 claims description 14
- 238000013527 convolutional neural network Methods 0.000 claims description 12
- 238000005457 optimization Methods 0.000 claims description 9
- 238000010276 construction Methods 0.000 claims description 6
- 230000000694 effects Effects 0.000 description 6
- 238000013473 artificial intelligence Methods 0.000 description 5
- 238000003860 storage Methods 0.000 description 5
- 238000001514 detection method Methods 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000009826 distribution Methods 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 230000000295 complement effect Effects 0.000 description 2
- 238000005034 decoration Methods 0.000 description 2
- 238000000280 densification Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 238000009825 accumulation Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000002349 favourable effect Effects 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
- G06T7/521—Depth or shape recovery from laser ranging, e.g. using interferometry; from the projection of structured light
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- General Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Biophysics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Molecular Biology (AREA)
- Health & Medical Sciences (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Biology (AREA)
- Optics & Photonics (AREA)
- Optical Radar Systems And Details Thereof (AREA)
- Traffic Control Systems (AREA)
Abstract
The invention provides a method and a system for realizing point cloud completion, wherein the method comprises the following steps: acquiring an acquired two-dimensional picture and a point cloud sparse depth map of the same target object; and inputting the point cloud sparse depth map and the two-dimensional picture into a pre-trained regression model so as to output a corresponding predicted dense point cloud. According to the invention, the sparse laser point cloud is subjected to deep completion to obtain the high-precision and dense predicted dense point cloud, and the effectiveness and the real-time performance of the deep completion are improved.
Description
Technical Field
The invention relates to the technical field of data processing, in particular to a method and a system for realizing point cloud completion.
Background
In practical use, the robot generally needs to sense, position, detect and avoid obstacles.
Indoor robots typically require the use of sensors such as depth cameras to provide dense depth maps or point clouds, however depth cameras are sensitive to changes in illumination and to areas where features degrade and errors in measured depth information can occur. The laser radar can generate accurate 3D point cloud, which is a good choice for robot application, however, if the robot platform moves, and if dense point cloud information is to be obtained, the inter-frame movement of the robot and a moving target in the environment need to be accurately measured, so that the accurate dense point cloud data can be obtained, which undoubtedly increases the system complexity.
Therefore, when the laser radar is used for sensing, positioning and detecting obstacle avoidance, it is very important to complement the depth information of the laser radar, and how to complement the sparse point cloud quickly and accurately is a technical problem that needs to be solved urgently by technical personnel in the field.
Disclosure of Invention
The invention aims to provide a method and a system for realizing point cloud completion, which are used for realizing the deep completion of sparse laser point cloud to obtain high-precision and dense predicted dense point cloud and improving the effectiveness and the real-time performance of the deep completion.
The technical scheme provided by the invention is as follows:
the invention provides a method for realizing point cloud completion, which comprises the following steps:
acquiring an acquired two-dimensional picture and a point cloud sparse depth map of the same target object;
and inputting the point cloud sparse depth map and the two-dimensional picture into a pre-trained regression model so as to output a corresponding predicted dense point cloud.
Further, the step of obtaining the two-dimensional image and the point cloud sparse depth map of the same target object comprises the following steps:
shooting the target object to obtain a two-dimensional picture, and scanning the target object through a laser radar to obtain corresponding sparse laser point cloud;
and searching a sparse laser point cloud corresponding to the target time frame according to the target time frame of the two-dimensional picture, and acquiring a point cloud sparse depth map according to the sparse laser point cloud.
Further, the step of obtaining a point cloud sparse depth map according to the sparse laser point cloud comprises the following steps:
and projecting the sparse laser point cloud to a pixel coordinate system where the two-dimensional picture is located according to the external reference calibration relation between the laser radar and the camera so as to generate a corresponding point cloud sparse depth map.
Further, the method also comprises the following steps:
establishing a training set according to the training data pair; the training set comprises a plurality of training data pairs, and each training data pair comprises a sample image and a sparse point cloud sample image of the same subframe for training;
and training according to the training data pair and the real dense point cloud to obtain the regression model.
Further, the training to obtain the regression model according to the training data pair and the real dense point cloud comprises the following steps:
adopting a convolutional neural network to construct a generator; the sample image and the point cloud sparse depth map are used as the input of a generator, and the dense point cloud is predicted to be used as the output of the generator;
constructing a classifier by adopting a convolutional neural network; predicting dense point cloud, real dense point cloud and sample image as input of a classifier, and taking a loss function of the classifier as output of the classifier;
carrying out weighted calculation on the point cloud difference value and the loss function of the classifier, and reversely transmitting the calculation result to a generator for parameter optimization to obtain the regression model; and the point cloud difference is the depth difference between the predicted dense point cloud and the real dense point cloud.
The invention also provides a system for realizing point cloud completion, which comprises:
the acquisition module is used for acquiring an acquisition two-dimensional picture and a point cloud sparse depth map of the same target object;
and the processing module is used for inputting the point cloud sparse depth map and the two-dimensional picture into a pre-trained regression model so as to output the corresponding predicted dense point cloud.
Further, the obtaining module includes:
the data acquisition unit is used for shooting the target object to acquire a two-dimensional picture and scanning the target object through a laser radar to acquire corresponding sparse laser point cloud;
the searching unit is used for searching the sparse laser point cloud corresponding to the target time frame according to the target time frame of the two-dimensional picture;
and the projection processing unit is used for acquiring a point cloud sparse depth map according to the sparse laser point cloud.
Further, the projection processing unit includes:
the acquisition subunit is used for acquiring an external parameter calibration relation between the laser radar and the camera;
and the generating subunit is used for projecting the sparse laser point cloud to a pixel coordinate system where the two-dimensional picture is located according to the external reference calibration relation between the laser radar and the camera so as to generate a corresponding point cloud sparse depth map.
Further, the method also comprises the following steps:
the sample acquisition module is used for establishing a training set according to the training data pair; the training set comprises a plurality of training data pairs, and each training data pair comprises a sample image and a sparse point cloud sample image of the same subframe for training;
and the model training module is used for training according to the training data pair and the real dense point cloud to obtain the regression model.
Further, the model training module comprises:
the network construction unit is used for constructing a generator by adopting a convolutional neural network; the sample image and the point cloud sparse depth map are used as the input of a generator, and the dense point cloud is predicted to be used as the output of the generator;
the network construction unit is also used for constructing a classifier by adopting a convolutional neural network; predicting dense point cloud, real dense point cloud and sample image as input of a classifier, and taking a loss function of the classifier as output of the classifier;
the optimization training unit is used for carrying out weighted calculation on the point cloud difference value and the loss function of the classifier, and reversely transmitting the calculation result to the generator for parameter optimization to obtain the regression model;
and the point cloud difference value is a depth difference value of the predicted dense point cloud and the real dense point cloud.
By the method and the system for realizing point cloud completion, provided by the invention, the sparse laser point cloud can be subjected to deep completion to obtain high-precision and dense predicted dense point cloud, and the effectiveness and the real-time performance of the deep completion are improved.
Drawings
The above features, technical features, advantages and implementations of a method and system for point cloud completion are further described in the following detailed description of preferred embodiments in a clearly understandable manner, with reference to the accompanying drawings.
FIG. 1 is a flow chart of an embodiment of a method for point cloud completion according to the present invention;
FIG. 2 is a flow chart of an embodiment of a method for point cloud completion of the present invention;
FIG. 3 is a schematic diagram of a regression standard of an implementation method of point cloud completion according to the present invention;
FIG. 4 is a flowchart of an embodiment of a method for implementing point cloud completion according to the present invention.
Detailed Description
In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system structures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.
It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
For the sake of simplicity, the drawings only schematically show the parts relevant to the present invention, and they do not represent the actual structure as a product. In addition, in order to make the drawings concise and understandable, components having the same structure or function in some of the drawings are only schematically illustrated or only labeled. In this document, "one" means not only "only one" but also a case of "more than one".
It should be further understood that the term "and/or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.
In addition, in the description of the present application, the terms "first", "second", and the like are used only for distinguishing the description, and are not intended to indicate or imply relative importance.
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following description will be made with reference to the accompanying drawings. It is obvious that the drawings in the following description are only some examples of the invention, and that for a person skilled in the art, other drawings and embodiments can be derived from them without inventive effort.
The time frame refers to the time interval of the Livox laser radar for emitting laser at equal intervals to obtain the laser point information, the time frame comprises a plurality of subframes with equal duration, each subframe can obtain a sparse laser point cloud fed back by aiming at a certain part of the same target object, and the Livox laser radar can form the overall part appearance information of the same target object through the sparse laser point cloud fed back by the time frame.
One embodiment of the present invention, as shown in fig. 1, is a method for implementing point cloud completion, including:
s100, acquiring a two-dimensional image and a point cloud sparse depth map of the same target object;
specifically, the target object includes, but is not limited to, a wall, a door, and the like, and even a driving floor. The camera is used for photographing the target object in the same posture in a time frame to obtain a corresponding two-dimensional picture, and in addition, the laser radar is used for scanning and performing point cloud processing on the target object in the same posture in the time frame to obtain a corresponding point cloud sparse depth map.
S200, inputting the point cloud sparse depth map and the two-dimensional picture into a pre-trained regression model to output a corresponding predicted dense point cloud.
Specifically, because the depth distribution of the scene and the feature distribution of the image data have strong correlation, the depth values of the pixels on the same target object are often similar or close to each other. According to the characteristic, the depth completion of the sparse laser point cloud captured by the laser radar is carried out by extracting the pixel characteristics of the two-dimensional picture, namely the two-dimensional picture captured by the camera and the point cloud sparse depth map captured by the laser radar are fused, so that the predicted dense point cloud corresponding to the pixel points of the two-dimensional picture one by one is obtained.
The method disclosed by the invention fuses information acquired by the camera and the laser radar to perform depth completion on the sparse laser point cloud to obtain the high-precision and dense predicted dense point cloud, so that the effectiveness and the real-time performance of the depth completion are improved, and more reliable support is provided for more advanced visual tasks such as subsequent three-dimensional reconstruction, automatic driving and intelligent robots.
An embodiment of the present invention, as shown in fig. 2, is a method for implementing point cloud completion, including:
s010 establishes a training set according to the training data pair; the training set comprises a plurality of training data pairs, and each training data pair comprises a sample image and a sparse point cloud sample image of the same subframe for training;
s020 training according to the training data pair and the real dense point cloud to obtain the regression model;
specifically, the processor is connected with Livox laser radar and camera respectively, can acquire and store the two-dimensional picture of the sparse laser point cloud that Livox laser radar gathered and camera collection, and the processor can acquire the two-dimensional picture as the sample image with historical collection. Because the Livox laser radar can acquire a plurality of groups of sparse laser point clouds in a time frame, the corresponding point cloud sparse depth map can be obtained by processing the selected sparse laser point clouds, and the processor can use the historically acquired point cloud sparse depth map as a sparse point cloud sample map.
In addition, the processor can acquire a corresponding current time frame according to the current subframe of the selected sparse laser point cloud, so as to acquire a two-dimensional picture matched with the current time frame, the two-dimensional picture corresponding to the current time frame and the sparse point cloud sample picture are a training data pair, and by analogy, training data pairs of different subframes can be acquired, so that the acquisition of the training data pair in one time frame is completed. Similarly, training data segments of a plurality of time frames can be acquired by multi-frame acquisition, and a training set is established according to a plurality of training data pairs. Therefore, the processor can take the two-dimensional pictures historically acquired before the current moment as sample images and the point cloud sparse depth maps historically acquired before the current moment as sparse point cloud sample maps, and thus the processor can acquire training data pairs of different time frames. After the processor obtains the training set in the above mode, the processor trains the real dense point cloud and the training data in the training set to obtain a regression model.
The sparse point cloud sample map (or point cloud sparse depth map) comprises coordinate values and depth values of laser data points on a pre-established laser coordinate system (which is a three-dimensional coordinate system). The two-dimensional picture (or sample image) includes coordinate values and depth values of the pixel keypoints on a pixel coordinate system (which is a two-dimensional coordinate system).
Preferably, the Livox laser radar has a timestamp when acquiring the laser point cloud, and the camera also has a timestamp when shooting and acquiring the two-dimensional picture, so that when the processor acquires the laser point cloud and the two-dimensional picture from the Livox laser radar and the camera and stores the laser point cloud and the two-dimensional picture, all the laser point cloud and all the images acquired by the same time frame are respectively stored in the same storage area, and are named according to the time frame. Thus, the training data pairs can be conveniently searched subsequently. It should be noted that the time lengths corresponding to the respective time frames are equal, and the intervals between the adjacent time frames are also equal.
S110, shooting the target object to obtain a two-dimensional picture, and scanning the target object through a laser radar to obtain corresponding sparse laser point cloud;
s120, searching a sparse laser point cloud corresponding to a target time frame according to the target time frame of the two-dimensional picture, and acquiring a point cloud sparse depth map according to the sparse laser point cloud;
specifically, laser is emitted to a target object through the Livox laser radar to obtain laser point cloud, and when a beam of laser irradiates the surface of the object, the reflected laser carries information such as direction and distance. If the laser beam is scanned according to a certain track, the reflected laser point data information is recorded while scanning, and a large amount of laser point data can be obtained due to extremely fine scanning, so that sparse laser point clouds can be formed according to the large amount of laser point data, a plurality of sparse laser point clouds can be obtained through accumulation for a period of time, and then the dense laser point clouds can be obtained according to the sparse laser point clouds. It should be understood that the time frame for the Livox lidar to perform point cloud collection includes a plurality of subframes, and each subframe can collect a sparse laser point cloud, so that when a camera collects a two-dimensional picture of a target object in one time frame, the Livox lidar can collect the sparse laser point clouds with the same number as the subframes in the time frame. Wherein the number of sub-frames is related to the acquisition accuracy of the Livox lidar.
The sparse laser point cloud comprises laser point data about the appearance surface of the target object, and each laser point data comprises information such as an xyz coordinate, depth and distance.
Preferably, the two-dimensional picture is an RGB picture taken using a monocular camera, a binocular camera, or an RGB-D camera. The camera and the Livox laser radar are installed on a mobile device (such as a robot, an unmanned vehicle and the like), and the overlapping of the camera and the Livox laser radar in the visual field range is ensured, so that the laser point cloud and the two-dimensional picture are synchronously acquired.
Preferably, the two-dimensional picture is a color image, that is, an RGB picture, which may be obtained by shooting a target object with an RGB-D camera disposed on a mobile robot platform or an unmanned vehicle mobile platform in advance, that is, in a motion process of the mobile robot and the unmanned vehicle, the color image of a scene where the mobile robot and the unmanned vehicle are located is acquired by the RGB-D camera. The RGB-D camera may be a camera capable of acquiring RGB pictures with high resolution, and accordingly, the color image is an RGB picture with high resolution. The method uses the color image for assisting in depth completion, and can play a guiding role in scene identification due to the abundant and dense information of the color image so as to output and obtain the high-precision and dense predicted dense point cloud.
S200, inputting the point cloud sparse depth map and the two-dimensional picture into a pre-trained regression model to output a corresponding predicted dense point cloud.
Specifically, the same portions of this embodiment as those of the above embodiment are referred to the above embodiment, and are not described in detail here. The novel Livox laser radar adopted by the invention is a laser radar sensor which is low in price and can generate accurate 3D point cloud, and the Livox laser radar is a good choice for robot application. Livox has a special scanning mode, and can obtain very dense point cloud data by integrating over time, however, if the robot platform moves, if dense point cloud information is to be obtained, the inter-frame motion of the robot and a moving target in the environment need to be accurately measured, so that the more accurate dense point cloud data can be obtained, which undoubtedly increases the system complexity. Therefore, when Livox is adopted, the depth information is complemented to obtain real-time depth data, and the real-time depth data can help the robot to avoid obstacles.
The point cloud sparse depth map is subjected to depth completion, a regression model is built through an anti-neural network, the RGB image and the sparse depth map are used as the input of the regression model, a dense depth map is obtained, and the densification of the point cloud sparse depth map is realized. The constructed regression model is favorable for obtaining the predicted dense point cloud on the basis of the known point cloud sparse depth map and the RGB image to realize the densification of the point cloud sparse depth map.
In an embodiment of the present invention, a method for implementing point cloud completion includes:
s010 establishes a training set according to the training data pair; the training set comprises a plurality of training data pairs, and each training data pair comprises a sample image and a sparse point cloud sample image of the same subframe for training;
s021, constructing a generator by adopting a convolutional neural network; the sample image and the point cloud sparse depth map are used as the input of a generator, and the dense point cloud is predicted to be used as the output of the generator;
s022, constructing a classifier by adopting a convolutional neural network; predicting dense point cloud, real dense point cloud and sample image as input of a classifier, and taking a loss function of the classifier as output of the classifier;
s023, performing weighted calculation on the point cloud difference value and a loss function of the classifier, and reversely transmitting a calculation result to a generator to perform parameter optimization to obtain the regression model; the point cloud difference is a depth difference between the predicted dense point cloud and the real dense point cloud;
specifically, the process of manufacturing the training data is to continue the above embodiment, and acquire the two-dimensional picture and the Dense laser point cloud data of the same time frame, so that { RGB _ image (color image), Dense _ point cloud (Dense point cloud), point cloud1 (first sparse point cloud), point cloud2 (second sparse point cloud). } can be formed. Wherein, the sense _ point cloud can be used as the true value data of the depth map in the training data, and then a plurality of training data pairs are formed, such as [ { RGB _ image, point cloud1}, { RGB _ image, point cloud2} ]. The sets of data pairs serve as training data. Thus, each segment is collected, a plurality of training data pairs are formed.
For example, assuming that the first time frame t1 includes a plurality of subframes, for example, a first subframe t11 and a second subframe t12, … …, the two-dimensional picture obtained during the first time frame t1 (i.e., the first subframe t11 and the second subframe t12, … …) is RGB _ image, the sparse laser point cloud obtained during the first subframe t11 is point cloud1, and the sparse laser point cloud obtained during the second subframe t12 is point cloud 2. Then, the sparse laser point cloud point 1 and the two-dimensional picture RGB _ image collected by the first sub-frame t11 are stored in the storage area D1, and the storage area D1 is named according to the first sub-frame t 11. Similarly, the sparse laser point cloud point 2 and the two-dimensional picture RGB _ image collected by the second sub-frame t12 are stored in the storage area D2, and the storage area D2 is named according to the second sub-frame t 12. By analogy, two-dimensional pictures and sparse laser point clouds in different time frames can be stored in groups, and then the sparse laser point clouds are processed to obtain a point cloud sparse depth map.
When the processor trains the regression model, the relative relationship between the true value point and the closest point of the physical space is utilized to carry out regression training. Specifically, as shown in fig. 3, the x and y directions are based on the closest point in the pixels in the same row, and the z direction is based on the closest point in the pixels in the same column.
S110, shooting the target object to obtain a two-dimensional picture, and scanning the target object through a laser radar to obtain corresponding sparse laser point cloud;
s121, searching a sparse laser point cloud corresponding to a target time frame according to the target time frame of the two-dimensional picture;
s122, according to the external reference calibration relation between the laser radar and the camera, projecting the sparse laser point cloud to a pixel coordinate system where the two-dimensional picture is located to generate a corresponding point cloud sparse depth map;
specifically, laser point data included in the sparse laser point cloud are respectively projected to a pixel coordinate system, so that a corresponding point cloud sparse depth map is obtained. The point cloud sparse depth map is a two-dimensional projection map obtained by projecting laser point cloud collected by a Livox laser radar to a plane where a gray level image is located, namely a pixel coordinate system, and is a single-channel image, and the numerical value of the point cloud sparse depth map is the depth value of the corresponding laser point data in the point cloud.
As shown in fig. 4, the laser point cloud collected by the Livox lidar is used as a regression reference of the whole model, and each frame of laser data needs a corresponding sparse depth map (i.e. the point cloud sparse depth map of the present invention) and an RGB image (i.e. one of the two-dimensional images of the present invention) to form a corresponding dense depth map (i.e. the predicted dense point cloud of the present invention). The generation of the sparse depth map needs to project the sparse laser point cloud acquired by Livox into a two-dimensional image, namely, the depth value of the laser point of the sparse laser point cloud is converted into depth information under a pixel coordinate system according to the external reference calibration relation between the Livox laser radar and the camera. The HRNet is preferentially adopted by the main network of the regression model, and mainly depends on the output high-resolution result, and certainly, the main network can also adopt other modes to obtain the high-resolution result, and the method is also applicable.
S200, inputting the point cloud sparse depth map and the two-dimensional picture into a pre-trained regression model to output a corresponding predicted dense point cloud.
Specifically, the same portions of this embodiment as those of the above embodiment are referred to the above embodiment, and are not described in detail here. The point cloud completion method provided by the invention can be applied to artificial intelligence platforms, such as a mobile robot platform, an unmanned vehicle mobile platform and the like, and is used for enabling the artificial intelligence platform to sense a three-dimensional scene structure so as to predict surrounding scenes of the artificial intelligence platform, and realizing motion planning and the like of the artificial intelligence platform in the three-dimensional scene. The method can fully utilize the potential correlation between the laser radar point cloud and the camera image, so that the point cloud sparse depth map is subjected to depth completion effectively and accurately to obtain the predicted dense point cloud, the problems that the point cloud in a partial region is too sparse and is difficult to complete by completing only based on the point cloud are solved, the corresponding two-dimensional image is provided as a reference, the information content is increased, and the point cloud completion effect is effectively improved. The invention is based on an unsupervised network architecture, not only solves the problem of dependence on labels, but also can better complete the depth map, so that the robot can better avoid obstacles and build the map in the following process.
Based on the laser points and the pixel points which are matched in the two-dimensional picture and the point cloud sparse depth map of the same target object corresponding to the same space range, advantage complementation is carried out through a camera with dense but relatively inaccurate pixel depth information and a laser radar with accurate but relatively sparse point cloud depth information, and therefore dense and relatively accurate predicted dense point cloud is obtained through blank point filling completion on the original point cloud sparse depth map. According to the invention, the point cloud density and processing are carried out on the two-dimensional picture and the point cloud sparse depth map, so that the depth information precision is improved, and meanwhile, the calculation time is effectively shortened, thereby effectively reducing the equipment and calculation cost, and being beneficial to popularization and application of products and methods. Therefore, the environment detection sensing method can greatly meet the requirement that the robot, the automatic driving and other artificial intelligence carries out accurate and efficient environment detection sensing, and provides accurate environment detection results for follow-up automatic driving, sensing positioning and detection obstacle avoidance.
In an embodiment of the present invention, a system for implementing point cloud completion includes:
the acquisition module is used for acquiring an acquisition two-dimensional picture and a point cloud sparse depth map of the same target object;
and the processing module is used for inputting the point cloud sparse depth map and the two-dimensional picture into a pre-trained regression model so as to output the corresponding predicted dense point cloud.
Specifically, this embodiment is a system embodiment corresponding to the above method embodiment, and specific effects refer to the above method embodiment, which is not described in detail herein.
Based on the foregoing embodiment, the obtaining module includes:
the data acquisition unit is used for shooting the target object to acquire a two-dimensional picture and scanning the target object through a laser radar to acquire corresponding sparse laser point cloud;
the searching unit is used for searching the sparse laser point cloud corresponding to the target time frame according to the target time frame of the two-dimensional picture;
and the projection processing unit is used for acquiring a point cloud sparse depth map according to the sparse laser point cloud.
Specifically, this embodiment is a system embodiment corresponding to the above method embodiment, and specific effects refer to the above method embodiment, which is not described in detail herein.
Based on the foregoing embodiment, the projection processing unit includes:
the acquisition subunit is used for acquiring an external parameter calibration relation between the laser radar and the camera;
and the generating subunit is used for projecting the sparse laser point cloud to a pixel coordinate system where the two-dimensional picture is located according to the external reference calibration relation between the laser radar and the camera so as to generate a corresponding point cloud sparse depth map.
Specifically, this embodiment is a system embodiment corresponding to the above method embodiment, and specific effects refer to the above method embodiment, which is not described in detail herein.
Based on the foregoing embodiment, further comprising:
the sample acquisition module is used for establishing a training set according to the training data pair; the training set comprises a plurality of training data pairs, and each training data pair comprises a sample image and a sparse point cloud sample image of the same subframe for training;
and the model training module is used for training according to the training data pair and the real dense point cloud to obtain the regression model.
Specifically, this embodiment is a system embodiment corresponding to the above method embodiment, and specific effects refer to the above method embodiment, which is not described in detail herein.
Based on the foregoing embodiments, the model training module includes:
the network construction unit is used for constructing a generator by adopting a convolutional neural network; the sample image and the point cloud sparse depth map are used as the input of a generator, and the dense point cloud is predicted to be used as the output of the generator;
the network construction unit is also used for constructing a classifier by adopting a convolutional neural network; predicting dense point cloud, real dense point cloud and sample image as input of a classifier, and taking a loss function of the classifier as output of the classifier;
the optimization training unit is used for carrying out weighted calculation on the point cloud difference value and the loss function of the classifier, and reversely transmitting the calculation result to the generator for parameter optimization to obtain the regression model;
and the point cloud difference value is a depth difference value of the predicted dense point cloud and the real dense point cloud.
Specifically, this embodiment is a system embodiment corresponding to the above method embodiment, and specific effects refer to the above method embodiment, which is not described in detail herein.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-described division of program modules is illustrated, and in practical applications, the above-described distribution of functions may be performed by different program modules, that is, the internal structure of the apparatus may be divided into different program units or modules to perform all or part of the above-described functions. Each program module in the embodiments may be integrated in one processing unit, or each unit may exist alone physically, or two or more units are integrated in one processing unit, and the integrated unit may be implemented in a form of hardware, or may be implemented in a form of software program unit. In addition, the specific names of the program modules are only used for distinguishing the program modules from one another, and are not used for limiting the protection scope of the application.
In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and reference may be made to the related descriptions of other embodiments for parts that are not described or recited in detail in a certain embodiment.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus/terminal device and method may be implemented in other ways. For example, the above-described embodiments of the apparatus/terminal device are merely illustrative, and for example, the division of the modules or units is only one logical division, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
It should be noted that the above embodiments can be freely combined as necessary. The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.
Claims (10)
1. A method for realizing point cloud completion is characterized by comprising the following steps:
acquiring an acquired two-dimensional picture and a point cloud sparse depth map of the same target object;
and inputting the point cloud sparse depth map and the two-dimensional picture into a pre-trained regression model so as to output a corresponding predicted dense point cloud.
2. The method for implementing point cloud completion according to claim 1, wherein the step of obtaining the two-dimensional image and the point cloud sparse depth map of the same target object comprises the steps of:
shooting the target object to obtain a two-dimensional picture, and scanning the target object through a laser radar to obtain corresponding sparse laser point cloud;
and searching a sparse laser point cloud corresponding to the target time frame according to the target time frame of the two-dimensional picture, and acquiring a point cloud sparse depth map according to the sparse laser point cloud.
3. The method for implementing point cloud completion according to claim 2, wherein the step of obtaining a point cloud sparse depth map according to the sparse laser point cloud comprises the steps of:
and projecting the sparse laser point cloud to a pixel coordinate system where the two-dimensional picture is located according to the external reference calibration relation between the laser radar and the camera so as to generate a corresponding point cloud sparse depth map.
4. The method for realizing point cloud completion according to any one of claims 1 to 3, further comprising the steps of:
establishing a training set according to the training data pair; the training set comprises a plurality of training data pairs, and each training data pair comprises a sample image and a sparse point cloud sample image of the same subframe for training;
and training according to the training data pair and the real dense point cloud to obtain the regression model.
5. The method for implementing point cloud completion according to claim 4, wherein the training of the regression model according to the training data pairs and the real dense point cloud comprises the following steps:
adopting a convolutional neural network to construct a generator; the sample image and the point cloud sparse depth map are used as the input of a generator, and the dense point cloud is predicted to be used as the output of the generator;
constructing a classifier by adopting a convolutional neural network; predicting dense point cloud, real dense point cloud and sample image as input of a classifier, and taking a loss function of the classifier as output of the classifier;
carrying out weighted calculation on the point cloud difference value and the loss function of the classifier, and reversely transmitting the calculation result to a generator for parameter optimization to obtain the regression model; and the point cloud difference is the depth difference between the predicted dense point cloud and the real dense point cloud.
6. A system for realizing point cloud completion is characterized by comprising:
the acquisition module is used for acquiring an acquisition two-dimensional picture and a point cloud sparse depth map of the same target object;
and the processing module is used for inputting the point cloud sparse depth map and the two-dimensional picture into a pre-trained regression model so as to output the corresponding predicted dense point cloud.
7. The system of claim 6, wherein the acquisition module comprises:
the data acquisition unit is used for shooting the target object to acquire a two-dimensional picture and scanning the target object through a laser radar to acquire corresponding sparse laser point cloud;
the searching unit is used for searching the sparse laser point cloud corresponding to the target time frame according to the target time frame of the two-dimensional picture;
and the projection processing unit is used for acquiring a point cloud sparse depth map according to the sparse laser point cloud.
8. The system of claim 7, wherein the projection processing unit comprises:
the acquisition subunit is used for acquiring an external parameter calibration relation between the laser radar and the camera;
and the generating subunit is used for projecting the sparse laser point cloud to a pixel coordinate system where the two-dimensional picture is located according to the external reference calibration relation between the laser radar and the camera so as to generate a corresponding point cloud sparse depth map.
9. The system for implementing point cloud completion according to any one of claims 6 to 8, further comprising:
the sample acquisition module is used for establishing a training set according to the training data pair; the training set comprises a plurality of training data pairs, and each training data pair comprises a sample image and a sparse point cloud sample image of the same subframe for training;
and the model training module is used for training according to the training data pair and the real dense point cloud to obtain the regression model.
10. The system of claim 9, wherein the model training module comprises:
the network construction unit is used for constructing a generator by adopting a convolutional neural network; the sample image and the point cloud sparse depth map are used as the input of a generator, and the dense point cloud is predicted to be used as the output of the generator;
the network construction unit is also used for constructing a classifier by adopting a convolutional neural network; predicting dense point cloud, real dense point cloud and sample image as input of a classifier, and taking a loss function of the classifier as output of the classifier;
the optimization training unit is used for carrying out weighted calculation on the point cloud difference value and the loss function of the classifier, and reversely transmitting the calculation result to the generator for parameter optimization to obtain the regression model;
and the point cloud difference value is a depth difference value of the predicted dense point cloud and the real dense point cloud.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110381744.2A CN113160327A (en) | 2021-04-09 | 2021-04-09 | Method and system for realizing point cloud completion |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110381744.2A CN113160327A (en) | 2021-04-09 | 2021-04-09 | Method and system for realizing point cloud completion |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113160327A true CN113160327A (en) | 2021-07-23 |
Family
ID=76888930
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110381744.2A Pending CN113160327A (en) | 2021-04-09 | 2021-04-09 | Method and system for realizing point cloud completion |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113160327A (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113409227A (en) * | 2021-08-19 | 2021-09-17 | 深圳市信润富联数字科技有限公司 | Point cloud picture repairing method and device, electronic equipment and storage medium |
| CN113628140A (en) * | 2021-08-12 | 2021-11-09 | 浙江大学 | Point cloud dense completion method based on deep learning |
| CN113780349A (en) * | 2021-08-09 | 2021-12-10 | 深圳奥锐达科技有限公司 | Method for acquiring training sample set, model training method and related device |
| CN114677315A (en) * | 2022-04-11 | 2022-06-28 | 探维科技(北京)有限公司 | Image fusion method, device, equipment and medium based on image and laser point cloud |
| WO2023044605A1 (en) * | 2021-09-22 | 2023-03-30 | 深圳先进技术研究院 | Three-dimensional reconstruction method and apparatus for brain structure in extreme environments, and readable storage medium |
| WO2023164845A1 (en) * | 2022-03-02 | 2023-09-07 | 深圳市大疆创新科技有限公司 | Three-dimensional reconstruction method, device, system, and storage medium |
| CN117351310A (en) * | 2023-09-28 | 2024-01-05 | 山东大学 | Multi-mode 3D target detection method and system based on depth completion |
| WO2024109055A1 (en) * | 2022-11-25 | 2024-05-30 | 华为技术有限公司 | Laser radar point cloud processing method and apparatus |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110097589A (en) * | 2019-04-29 | 2019-08-06 | 广东工业大学 | A kind of depth complementing method applied to sparse map denseization |
| US20200082207A1 (en) * | 2018-09-07 | 2020-03-12 | Baidu Online Network Technology (Beijing) Co., Ltd. | Object detection method and apparatus for object detection |
| CN111563923A (en) * | 2020-07-15 | 2020-08-21 | 浙江大华技术股份有限公司 | Method for obtaining dense depth map and related device |
| CN112102472A (en) * | 2020-09-01 | 2020-12-18 | 北京航空航天大学 | Sparse three-dimensional point cloud densification method |
-
2021
- 2021-04-09 CN CN202110381744.2A patent/CN113160327A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20200082207A1 (en) * | 2018-09-07 | 2020-03-12 | Baidu Online Network Technology (Beijing) Co., Ltd. | Object detection method and apparatus for object detection |
| CN110097589A (en) * | 2019-04-29 | 2019-08-06 | 广东工业大学 | A kind of depth complementing method applied to sparse map denseization |
| CN111563923A (en) * | 2020-07-15 | 2020-08-21 | 浙江大华技术股份有限公司 | Method for obtaining dense depth map and related device |
| CN112102472A (en) * | 2020-09-01 | 2020-12-18 | 北京航空航天大学 | Sparse three-dimensional point cloud densification method |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113780349A (en) * | 2021-08-09 | 2021-12-10 | 深圳奥锐达科技有限公司 | Method for acquiring training sample set, model training method and related device |
| CN113780349B (en) * | 2021-08-09 | 2023-07-11 | 深圳奥锐达科技有限公司 | Training sample set acquisition method, model training method and related device |
| CN113628140B (en) * | 2021-08-12 | 2023-08-08 | 浙江大学 | Point cloud dense completion method based on deep learning |
| CN113628140A (en) * | 2021-08-12 | 2021-11-09 | 浙江大学 | Point cloud dense completion method based on deep learning |
| CN113409227A (en) * | 2021-08-19 | 2021-09-17 | 深圳市信润富联数字科技有限公司 | Point cloud picture repairing method and device, electronic equipment and storage medium |
| WO2023044605A1 (en) * | 2021-09-22 | 2023-03-30 | 深圳先进技术研究院 | Three-dimensional reconstruction method and apparatus for brain structure in extreme environments, and readable storage medium |
| WO2023164845A1 (en) * | 2022-03-02 | 2023-09-07 | 深圳市大疆创新科技有限公司 | Three-dimensional reconstruction method, device, system, and storage medium |
| CN114677315A (en) * | 2022-04-11 | 2022-06-28 | 探维科技(北京)有限公司 | Image fusion method, device, equipment and medium based on image and laser point cloud |
| CN114677315B (en) * | 2022-04-11 | 2022-11-29 | 探维科技(北京)有限公司 | Image fusion method, device, equipment and medium based on image and laser point cloud |
| US11954835B2 (en) | 2022-04-11 | 2024-04-09 | Tanway Technology (beijing) Co., Ltd. | Methods, devices, apparatuses, and media for image fusion utilizing images and LiDAR point clouds |
| WO2024109055A1 (en) * | 2022-11-25 | 2024-05-30 | 华为技术有限公司 | Laser radar point cloud processing method and apparatus |
| CN117351310A (en) * | 2023-09-28 | 2024-01-05 | 山东大学 | Multi-mode 3D target detection method and system based on depth completion |
| CN117351310B (en) * | 2023-09-28 | 2024-03-12 | 山东大学 | Multi-mode 3D target detection method and system based on depth completion |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113160327A (en) | Method and system for realizing point cloud completion | |
| US10937231B2 (en) | Systems and methods for updating a high-resolution map based on binocular images | |
| CA3028653C (en) | Methods and systems for color point cloud generation | |
| CN110675418B (en) | Target track optimization method based on DS evidence theory | |
| CN112419494B (en) | Obstacle detection and marking method and device for automatic driving and storage medium | |
| CN110689562A (en) | Trajectory loop detection optimization method based on generation of countermeasure network | |
| CN111060924B (en) | SLAM and target tracking method | |
| US20220051425A1 (en) | Scale-aware monocular localization and mapping | |
| CN113240813B (en) | Three-dimensional point cloud information determining method and device | |
| JP2017181476A (en) | Vehicle location detection device, vehicle location detection method and vehicle location detection-purpose computer program | |
| CN114494466B (en) | External parameter calibration method, device and equipment and storage medium | |
| Li et al. | Durlar: A high-fidelity 128-channel lidar dataset with panoramic ambient and reflectivity imagery for multi-modal autonomous driving applications | |
| CN113608234A (en) | City data acquisition system | |
| CN110136186B (en) | Detection target matching method for mobile robot target ranging | |
| CN116630528A (en) | Static scene reconstruction method based on neural network | |
| CN113196341A (en) | Method for detecting and modeling objects on the surface of a road | |
| CN113052846B (en) | Multi-line radar point cloud densification method and device | |
| CN112364741B (en) | Monocular remote obstacle detection method and device for unmanned aerial vehicle and unmanned aerial vehicle | |
| CN113792645A (en) | AI eyeball fusing image and laser radar | |
| CN113874681B (en) | Evaluation method and system for point cloud map quality | |
| CN114359891A (en) | Three-dimensional vehicle detection method, system, device and medium | |
| CN113433566A (en) | Map construction system and map construction method | |
| Agrawal et al. | RWU3D: Real World ToF and Stereo Dataset with High Quality Ground Truth | |
| Berrio et al. | Semantic sensor fusion: From camera to sparse LiDAR information | |
| TW202349343A (en) | Depth estimation method, device, electronic device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |