CN115205610A - Training method and training device for perception model and electronic equipment - Google Patents
Training method and training device for perception model and electronic equipment Download PDFInfo
- Publication number
- CN115205610A CN115205610A CN202110378156.3A CN202110378156A CN115205610A CN 115205610 A CN115205610 A CN 115205610A CN 202110378156 A CN202110378156 A CN 202110378156A CN 115205610 A CN115205610 A CN 115205610A
- Authority
- CN
- China
- Prior art keywords
- point cloud
- target
- cloud data
- data
- training
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Optical Radar Systems And Details Thereof (AREA)
Abstract
The method for training the perception model comprises the following steps: acquiring actual point cloud data continuously acquired by a laser radar; carrying out target modeling according to the actual point cloud data to obtain a target model; calibrating and measuring the beam information of the laser radar to obtain the distribution of laser beams; obtaining simulation point cloud data according to the reflected laser points of the laser beam distributed on the target model; and obtaining fusion training data according to the actual point cloud data and the simulation point cloud data, and training the perception model by using the fusion training data. According to the scheme, the generation efficiency and diversity of training data for perception model training can be considered, and the training effect of the perception model is improved.
Description
Technical Field
The embodiment of the specification relates to the technical field of point cloud data processing, in particular to a method and a device for training a perception model and electronic equipment.
Background
Lidar is a sensor widely used in the field of autopilot and provides accurate distance measurement and three-dimensional geometric information. However, the sparsity of the lidar point cloud makes target detection difficult. In order to guarantee the performance of the detection algorithm, especially the detection task of long distance, occlusion and special target scenes, a large amount of training and test data is needed. In order to meet the requirement of the laser radar perception model on a large amount of training data, data augmentation is generally performed.
At present, there are two common data augmentation methods for laser point cloud:
one is to use real point cloud data to perform data augmentation, and the data augmentation is realized through sampling, rotation, mirroring, partial deletion, copying and pasting and the like. On one hand, however, since the distribution of the point cloud changes with the changes of the distance and the direction, the use of actual point cloud data is greatly limited in order to ensure the consistency and the authenticity of the data; on the other hand, different laser radars and different sensor arrangement schemes generate different point clouds in the same scene, so that actual point cloud data acquired by the laser radars cannot be used as training data after the sensors are replaced.
The other is to use simulation data for data augmentation, i.e., a target model is constructed by modeling software, the target model is placed in a simulation environment, and a simulated lidar beam is used to generate a reflected laser spot. The method has the advantages that the generated training data is not limited by the target distance, direction and pose, the laser radar wire harnesses distributed randomly can be simulated, and the method is more flexible to use. However, since it is very time-consuming to model a fine target (for example, complex surface modeling is required for a vehicle), batch modeling cannot be realized, and if a model existing in a virtual scene such as a game is used, available models (for example, models for different vehicle models) are very limited, and it is difficult to meet the requirement of a detection algorithm on the diversity of training data.
Disclosure of Invention
In view of this, embodiments of the present disclosure provide a training method and a training apparatus for a perceptual model, and an electronic device, which can improve the training effect of the perceptual model by taking into account the generation efficiency and the diversity of training data for training the perceptual model.
The embodiment of the specification discloses a method for training a perception model, which comprises the following steps:
acquiring actual point cloud data continuously acquired by a laser radar;
carrying out target modeling according to the actual point cloud data to obtain a target model;
calibrating and measuring the beam information of the laser radar to obtain the distribution of laser beams;
obtaining simulation point cloud data according to the reflected laser points of the laser beam distributed on the target model;
and obtaining fusion training data according to the actual point cloud data and the simulation point cloud data, and training the perception model by using the fusion training data.
Optionally, the performing target modeling according to the actual point cloud data to obtain a target model includes:
intercepting the actual point cloud data to obtain a target point cloud;
registering multiple frames of target point clouds to obtain dense point clouds;
and based on the dense point cloud, obtaining the target model through point cloud surface reconstruction.
Optionally, before performing the point cloud surface reconstruction, the method further includes:
and denoising, ground removing and/or smoothing the dense point cloud to obtain a smooth dense point cloud for point cloud surface reconstruction.
Optionally, the registering multiple frames of target point clouds to obtain a dense point cloud includes:
and selecting a first frame from the multi-frame target point cloud as a basic frame according to the acquisition sequence, and sequentially matching and overlapping subsequent frames in the multi-frame target point cloud to the overlapping result of the previous frame according to the acquisition sequence to obtain the dense point cloud.
Optionally, the sequentially matching and superimposing subsequent frames in the multiple frames of target point clouds on the superimposed result of the previous frame according to the collecting sequence to obtain the dense point cloud includes:
and sequentially carrying out global matching on subsequent frames in the multi-frame target point cloud and the corresponding superposition result of the previous frame, then carrying out local matching according to the decreasing step length until the matching degree reaches a preset threshold value, and then carrying out matching superposition on the subsequent frame and the current superposition result until the matching superposition of the last frame is completed.
Optionally, the registering multiple frames of target point clouds to obtain a dense point cloud includes:
dividing the multi-frame target point cloud into a plurality of subsequences according to the acquisition sequence;
respectively taking the first frame in the subsequences as the basic frame of the corresponding subsequence, and respectively matching and overlapping the subsequent frames to the overlapping result of the previous frame according to the acquisition sequence for any subsequence until the matching and overlapping of the last frame in the subsequences are completed;
and sequentially matching and overlapping the overlapping results of the plurality of subsequences according to the acquisition sequence to obtain the dense point cloud.
Optionally, the obtaining simulation point cloud data according to the reflected laser points of the laser beam distributed on the target model includes:
obtaining the simulation point cloud data according to the reflected laser points of the laser beam distributed on the target model in different scenes;
and/or the presence of a gas in the atmosphere,
according to the reflected laser points of the laser beam distributed on the target model under different laser radars and different sensor setting parameters, obtaining the simulation point cloud data;
and/or the presence of a gas in the gas,
and obtaining the simulated point cloud data according to the reflected laser points of the laser beam distributed on the target model under different poses of the target model.
Optionally, the obtaining of fusion training data according to the actual point cloud data and the simulated point cloud data includes:
and determining the using quantity and the fusion proportion of the actual point cloud data and the simulated point cloud data according to the required data quantity of the perception model to obtain the fusion training data.
The embodiment of the present specification further provides a device for training a perceptual model, wherein the device includes:
the actual point cloud acquisition unit is suitable for acquiring actual point cloud data continuously acquired by the laser radar;
the modeling unit is suitable for carrying out target modeling according to the actual point cloud data to obtain a target model;
the calibration measuring unit is suitable for calibrating and measuring the beam information of the laser radar to obtain the distribution of laser beams;
the simulation point cloud obtaining unit is suitable for obtaining simulation point cloud data according to the reflected laser points of the laser beam distributed on the target model;
and the fusion training unit is suitable for obtaining fusion training data according to the actual point cloud data and the simulation point cloud data and training a perception model by using the fusion training data.
The present specification further provides an electronic device, including a memory and a processor, where the memory stores computer instructions executable on the processor, and the processor executes the computer instructions to perform the steps of the method according to any one of the foregoing embodiments.
By adopting the method for training the perception model in the embodiment of the specification, firstly, the target model is constructed on the basis of actual point cloud data continuously acquired by the laser radar, so that a large number of refined target models can be constructed rapidly and accurately in a large scale. Furthermore, the distribution of the laser beams is obtained by calibrating and measuring the beam information of the laser radar, and the simulated point cloud data can be obtained according to the reflected laser points of the laser beams distributed on the target model without the limitation of the target distance, direction and pose, and the laser beams distributed by the laser radar at will can be simulated, so that a large amount of simulated point cloud data can be generated in batch, and the requirements on the quantity and diversity of training data in the process of training the laser radar perception model can be efficiently met. Therefore, the actual point cloud data and the simulation point cloud data are fused, and the perception model is trained by using the obtained fusion training data, so that the diversity and the accuracy of the training data can be considered, and the training effect of the perception model is improved.
Furthermore, the actual point cloud data is intercepted to obtain a target point cloud, then multi-frame target point clouds are registered to obtain dense point clouds, the target model is obtained through point cloud surface reconstruction based on the dense point clouds, and accurate, rapid and large-scale target modeling can be achieved.
Further, before point cloud surface reconstruction is carried out, denoising, ground removing and/or smoothing are carried out on the dense point cloud to obtain a smooth dense point cloud for point cloud surface reconstruction, so that data noise can be reduced, and the accuracy of a target model obtained through reconstruction is further improved.
Further, dividing the multi-frame target point cloud into a plurality of subsequences according to an acquisition sequence, taking a first frame in the subsequences as a basic frame of the corresponding subsequence, sequentially matching and overlapping subsequent frames to an overlapping result of a previous frame according to the acquisition sequence for any subsequence until the matching and overlapping of a last frame in the subsequences are completed, and then sequentially matching and overlapping the overlapping results of the subsequences according to the acquisition sequence to obtain the dense point cloud. By adopting the segmented superposition method, superposition matching errors can be reduced, and registration accuracy is improved.
Further, the simulation point cloud data can be obtained according to the reflected laser points of the laser beams distributed on the target model under different scenes; and/or obtaining the simulation point cloud data according to the reflected laser points of the laser beam distributed on the target model under different laser radars and different sensor setting parameters; and/or obtaining the simulation point cloud data according to the reflected laser points of the laser beams distributed on the target model at different poses of the target model. Accordingly, the perception performance of the perception model to different scenes can be improved, the generated simulation point cloud data is not limited by the distance, direction, pose and the like of a target, and the capability of the perception model for solving the problem of medium and long tails is improved.
Furthermore, the using quantity and the fusion proportion of the actual point cloud data and the simulated point cloud data are determined according to the required data quantity of the perception model, the fusion training data are obtained, the simulated point cloud data and the actual point cloud data can be controllably selected according to requirements, the data amplification effect of the simulated point cloud data is fully exerted, the capability of the target model for solving the problems of medium and long tails is improved, and the perception performance of the perception model under different scenes (including small probability scenes) is improved.
Drawings
FIG. 1 is a flow chart illustrating a method for training a perceptual model in an embodiment of the present specification;
FIG. 2 is a flow chart of a method for obtaining an object model in an embodiment of the present description;
FIG. 3 illustrates a cloud of registered dense spots in a particular scenario in an embodiment of the present description;
FIG. 4 illustrates a smoothed dense point cloud obtained in a particular scenario in an embodiment of the present description;
FIG. 5 is a diagram illustrating an effect of a target model reconstructed in a specific scenario according to an embodiment of the present disclosure;
FIG. 6 is a diagram illustrating an effect of simulated point clouds for a specific scene obtained in an embodiment of the present disclosure;
FIG. 7 is a schematic structural diagram illustrating a training apparatus for a perception model in an embodiment of the present disclosure;
fig. 8 shows a schematic structural diagram of an electronic device in an embodiment of the present specification.
Detailed Description
The Point Cloud is a mass Point set which expresses target space distribution and target surface special effects in the same space reference system, and the Point Cloud is a Point set obtained after the spatial coordinates of each sampling Point on the surface of the object are obtained, and is called Point Cloud. The point cloud obtained by laser measurement includes information such as three-dimensional coordinates (XYZ) and laser reflection Intensity (Intensity), wherein the laser reflection Intensity information is related to the surface material, roughness and incident angle direction of the target, and the transmitting capability and laser wavelength of the laser radar.
As described in the background, the sparsity of the lidar point cloud presents certain difficulties for target detection. In order to guarantee the performance of the detection algorithm, especially the detection task of long distance, occlusion and special target scenes, a large amount of training and test data is needed. At present, there are two common data augmentation methods for laser point cloud:
one is to use actual point cloud data to perform data augmentation, and is realized by sampling, rotating, mirroring, partial deleting, copying, pasting and the like. On one hand, however, since the distribution of the point cloud changes with the change of the distance and the direction, the use of actual point cloud data is greatly limited in order to ensure the consistency and the authenticity of the data; on the other hand, different radars and different sensor arrangement schemes generate different point clouds for the same scene, so that after the sensor is replaced, the previously acquired actual point cloud data cannot be used as training data.
The other is data augmentation using simulation data, i.e., a target model is built by modeling software, placed in a simulation environment, and a simulated laser beam is used to generate a reflected laser spot. The method has the advantages that the generated training data is not limited by the distance, the direction and the pose of the target, the laser beams distributed randomly can be simulated, and the method is more flexible to use. However, since it is very time-consuming to model a fine target (for example, complex surface modeling is required for a vehicle), batch modeling cannot be realized, and if a model existing in a virtual scene such as a game is used, available models (for example, models for different vehicle models) are very limited, and it is difficult to meet the requirement of a detection algorithm on the diversity of training data.
In view of the above problems, the embodiments of the present specification provide a training scheme for a corresponding perceptual model. Firstly, a target model is constructed based on actual point cloud data continuously acquired by a laser radar, so that a large number of refined target models can be constructed rapidly and accurately in a large scale. Furthermore, the distribution of the laser beams is obtained by calibrating and measuring the beam information of the laser radar, and the obtained simulated point cloud data can simulate the laser beams randomly distributed by the laser radar without the limitation of the target distance, direction and pose according to the reflected laser points distributed on the target model of the laser beams, so that a large amount of simulated point cloud data can be generated in batch, and the requirements on the quantity and diversity of training data in the process of training the laser radar perception model can be effectively met. Therefore, the actual point cloud data and the simulation point cloud data are fused, and the perception model is trained by using the obtained fusion training data, so that the diversity and the accuracy of the training data can be considered, and the training effect of the perception model is improved.
For those skilled in the art to better understand the technical concepts, technical principles and advantages of the embodiments of the present disclosure, and to better implement the embodiments of the present disclosure, the following detailed description of the training scheme of the perceptual model employed in the embodiments of the present disclosure is made with reference to the accompanying drawings through specific scenarios and application examples.
Referring to the flowchart of the training method of the perceptual model described in fig. 1, in the embodiment of the present specification, the following steps may be adopted to train the perceptual model.
And S11, acquiring actual point cloud data continuously acquired by the laser radar.
The actual point cloud data is derived from actual data actually acquired by the laser radar, and in the embodiment of the description, the type and arrangement scheme of the laser radar for acquiring the actual point cloud data are not limited.
In a specific implementation, the actual point cloud data may include multiple frames of point cloud information and annotation information. In order to improve the utilization efficiency of the actual point cloud data, the actual point cloud data may include multiple frames of continuously acquired point cloud information and labeling information. Based on the difference of the target object and the difference of the application requirement, the corresponding labeling information may be different. For a vehicle, the annotation information may include 7-dimensional coordinate data of a point cloud including vehicle center point coordinates (XYZ) and length, width, height and heading angle according to the autopilot requirement.
In a specific implementation, an existing open source data set may be used as the actual point cloud data.
And S12, carrying out target modeling according to the actual point cloud data to obtain a target model.
In a particular implementation, a corresponding object model may be created based on object objects appearing in the scene. In embodiments of the present description, target modeling may be performed for a rigid object (e.g., a vehicle) that does not deform during operation. In the observation process of the laser radar, each frame of point cloud data is used for accurately measuring the object at different angles.
And S13, calibrating the beam information of the laser radar to obtain the distribution of the laser beams.
In specific implementation, the distribution of laser beams of the laser radar in a simulation environment can be obtained by calibrating and measuring the beam information of the real laser radar.
As an alternative example, calibration measurements may be performed for lidar with different arrangements (e.g. with different numbers, different distribution positions, etc.) respectively, resulting in a distribution of the laser beam of the respective lidar.
And S14, obtaining simulation point cloud data according to the reflected laser points of the laser beam distributed on the target model.
In specific implementation, simulation point cloud data can be obtained in various ways according to the reflected laser points of the laser beam distributed on the target model. Some example ways are given below:
1) Obtaining the simulation point cloud data according to the reflected laser points of the laser beam distributed on the target model in different scenes;
2) According to the reflected laser points of the laser beam distributed on the target model under different laser radars and different sensor setting parameters, obtaining the simulation point cloud data;
3) And obtaining the simulation point cloud data according to the reflected laser points of the laser beam distributed on the target model under different poses of the target model.
In specific implementation, the simulation point cloud data can be obtained by adopting any one or more modes, so that abundant simulation point cloud data can be obtained and used as a training sample of a laser radar perception model.
In a specific application process, because the actual point cloud data acquired by the laser radar does not cover small-probability scenes such as long distance, shielding and special targets, correspondingly, the actual point cloud data is adopted to train the laser radar perception model, the obtained perception model has limited target recognition capability in the small-probability scenes, and even the data in the small-probability scenes can be ignored as noise in the training process, so that the perception model is difficult to recognize the targets in the small-probability scenes. Therefore, in specific implementation, simulation point cloud data corresponding to a target model in a corresponding scene can be controllably generated according to requirements by changing at least one of sensor setting parameters, target placement poses and the like of the laser radar according to different scenes, especially small probability scenes, so that different laser sensor schemes are adapted, the flexibility is strong, the recognition capability of a laser radar perception model in different scenes, especially small probability scenes, can be improved, and the detection performance of the laser radar perception model is comprehensively improved.
And S15, obtaining fusion training data according to the actual point cloud data and the simulation point cloud data, and training the perception model by using the fusion training data.
In specific implementation, the using quantity and the fusion proportion of the actual point cloud data and the simulated point cloud data can be determined according to the required data quantity of the perception model, and the fusion training data can be obtained, so that the accuracy and the diversity of the training data can be considered, and a better training effect can be obtained.
When the perception model is trained, reasonable training strategies can be formulated according to needs, sampling modes and loss function weights of simulation point cloud data and actual point cloud data are selected according to task requirements, the data amplification effect of the simulation point cloud data is fully exerted, the capability of the perception model for solving the problems of medium and long tail regions is improved, and the perception performance under a small probability scene is improved.
And testing the trained perception model by adopting an actual point cloud data set until the perception model meets the testing requirement to obtain a target perception model which can be used for target detection.
The actual point cloud dataset used for testing may include point cloud information and annotation information, where annotation information may include target pose information. In specific implementation, the point cloud information of the actual point cloud data set is input into the trained perceptual model for testing, whether the perceptual model meets the performance requirement is determined, and if the perceptual model does not meet the performance requirement, the training method in the foregoing embodiment of the present specification is continuously adopted to train the perceptual model until the test is passed.
More specifically, in the process of training the perception model, whether the identified target in the point cloud of the corresponding frame is consistent with the corresponding labeling information is determined, and whether the perception model meets preset test requirements, such as overall accuracy, accuracy in various specific scenes and the like, is determined according to the test result of the test data set. If the test requirement is not met, inputting the training data to the perception model again, and here, adjusting the proportion and the data quantity of the simulation point cloud data and the actual point cloud data in the training data or increasing the data quantity of the training data in some specific scenes and the like according to the early-stage test performance.
By adopting the embodiment of the specification, firstly, the target model is constructed based on the actual point cloud data, and a large number of refined target models can be constructed rapidly and accurately in a large scale; furthermore, the distribution of laser beams is obtained by calibrating and measuring the beam information of the laser radar, and the simulated point cloud data is obtained according to the reflected laser points of the laser beams distributed on the target model, so that the laser beams of the laser radar distributed at will can be simulated in a simulated environment and are not limited by the distance, direction and pose of a target, a large amount of simulated point cloud data can be generated in batches, the requirements on the quantity and diversity of training data in the process of training the perception model of the laser radar can be met efficiently, and therefore, the actual point cloud data and the simulated point cloud data are used for obtaining fusion point cloud data and participate in the training of the perception model together, the training effect of the perception model can be improved comprehensively, and the detection performance of the perception model can be improved comprehensively.
In specific implementation, the training method of the perceptual model may be further optimized and extended as needed, and is described in detail below by specific examples and specific application scenarios.
Taking the modeled object as a vehicle as an example, the objective modeling principle of the embodiment of the present specification is explained: assuming that the modeled vehicle is a rigid object and cannot deform in the operation process, in the observation process of the laser radar, each frame of laser point cloud is used for accurately measuring the vehicle at different angles, and if the observation point clouds of multiple frames of target vehicles are overlapped at a specific relative pose, the intensive target surface measurement at multiple angles can be obtained. Based on the principle, the embodiment of the specification utilizes the superposition of the measurement values under different observation angles to realize denser geometric measurement, so that the difficulty brought to the modeling of the target object by the sparsity of the laser point cloud can be overcome, and the accurate and efficient modeling of the target model is realized.
Referring to the flowchart of the method for acquiring the target model shown in fig. 2, as for step S12, a specific method for obtaining the target model is shown below through specific steps.
And S121, intercepting the actual point cloud data to obtain a target point cloud.
In specific implementation, the target point cloud can be obtained by intercepting input multi-frame actual point cloud data. The intercepting range can be determined based on the marking information corresponding to the actual point cloud data, such as a marking frame, the observation point cloud in the marking frame is intercepted, and the target point cloud can be obtained.
In addition, in specific implementation, for the input actual laser point cloud, if the actual laser point cloud is acquired by a plurality of laser radars, the laser point clouds acquired by the plurality of laser radars may be fused to obtain a frame of observation point cloud with a larger view field, and the fused observation point cloud is intercepted to obtain the target point cloud.
And S122, registering the multi-frame target point cloud to obtain a dense point cloud.
In specific implementation, the dense point clouds can be obtained by registering through a point cloud registration algorithm, and a user can select a proper point cloud registration algorithm to implement according to needs.
In some embodiments of the present specification, a first frame is selected from the multiple frames of target point clouds as a base frame according to an acquisition order, and subsequent frames in the multiple frames of target point clouds are sequentially matched and superimposed on a superimposition result of a previous frame according to the acquisition order to obtain the target point cloud. More specifically, for multiple frames of target point clouds, a second frame of the multiple frames of target point clouds and a first frame serving as a base frame may be superimposed to obtain a superimposed result of the second frame and the first frame, and then a third frame of target point clouds and the superimposed result are matched and superimposed, and so on until each frame of the multiple frames of target point clouds is matched and superimposed.
For an overlapping process of any two frames of target point clouds, for example, overlapping a second frame of target point cloud and a first frame of target point cloud in a plurality of frames of target point clouds, or overlapping an ith frame and an (i-1) th overlapping result, wherein i is a natural number and is not less than 2, in specific implementation, matching and overlapping can be performed by firstly adopting global matching and then adopting a local matching mode.
Specifically, the subsequent frames in the multi-frame target point cloud may be sequentially subjected to global matching with the superposition result of the previous frame corresponding thereto, then subjected to local matching according to the decreasing step length until the matching degree reaches the preset threshold, and then subjected to matching superposition of the subsequent frame and the current superposition result until the matching superposition of the last frame is completed. In the process of local matching, the matching step length can be decreased progressively, i.e. rough matching can be carried out with a larger step length, and if the matching degree reaches a corresponding threshold value, the matching is finished; otherwise, continuously reducing the step length, and performing more refined matching until the matching degree reaches the corresponding threshold value.
In the global matching process, if the mirror image superposition is determined, coordinate matching is firstly carried out. In the local matching process, local features of two frames of point clouds can be extracted for matching to obtain a coordinate transformation matrix, and then registration can be carried out based on the coordinate transformation matrix.
In the local matching process, a more accurate position can be quickly reached through rough matching with a larger step length, further more refined matching is carried out through a smaller step length, the matching precision can be improved, and therefore the registration precision and the registration efficiency can be considered at the same time.
In specific implementation, in order to reduce the overlay error in the registration process, the multi-frame target point cloud may be divided into a plurality of subsequences according to the acquisition order; then, respectively taking the first frame in the plurality of subsequences as the basic frame of the corresponding subsequence, and respectively matching and overlapping subsequent frames to the overlapping result of the previous frame according to the acquisition sequence for any subsequence until the matching and overlapping of the last frame in the subsequence are completed; furthermore, the superposition results of the plurality of subsequences can be sequentially matched and superposed according to the acquisition order to obtain the dense point cloud.
For the matching process of two frames of target point clouds in each subsequence, reference may be made to an overlapping process of any two frames of target point clouds, or a global matching and then local matching may be adopted, which may refer to the detailed description of the foregoing embodiment.
In addition, after matching and overlaying of each subsequence is completed, in order to fill an observation angle which may be missed, since the vehicle is generally symmetrical, the overlaying result can be mirrored once again, and therefore the overlaying result after mirroring can be overlaid once again with the overlaying result before mirroring to serve as the dense point cloud.
In an example scenario of the present specification, a schematic diagram of a point cloud effect obtained after superimposing a second frame of target point cloud and a first frame of target point cloud in multiple frames of target point clouds is shown in fig. 3, and it can be seen that more precise surface measurement can be obtained by superimposing and registering different target point clouds.
S123, denoising, ground removing and/or smoothing the dense point cloud to obtain a smooth dense point cloud for point cloud surface reconstruction.
For the dense point cloud obtained by registration, there may be a noise point cloud, and for this purpose, the optional step S123 may be adopted to perform filtering processing.
Specifically, superimposing multiple frames of target point clouds may introduce noise point clouds and ground point clouds, and may use the height information and the local relative distance to perform judgment, so that the ground point clouds and the noise point clouds may be removed to obtain smooth dense point clouds, such as a smooth dense point cloud effect diagram shown in fig. 4.
And S124, based on the dense point cloud, reconstructing the surface of the point cloud to obtain the target model.
On the basis of the dense point cloud obtained in step S122 or the smooth dense point cloud obtained in step S123, a triangular patch can be obtained through surface reconstruction, and simulation reconstruction based on actual point cloud data is completed to obtain a target model. In specific implementation, the dense point cloud data may be input into modeling and simulation software, and the target model may be obtained through point cloud generation. For example, the smooth point cloud shown in fig. 4, and the target model effect obtained after the point cloud surface reconstruction is shown in fig. 5. Fig. 6 shows simulated point cloud data obtained by the method of the embodiment of the present disclosure.
In the embodiment, the simulation point cloud data generated based on the actual point cloud data is used for laser radar perception model training, and the simulation point cloud data can be generated in batches and has diversity, so that after the perception model is trained by the actual point cloud data and the simulation point cloud data, the perception model is used for target detection, the diversity and accuracy requirements of a detection algorithm on the point cloud data can be met, and the detection performance can be improved due to the small probability scene.
In a specific implementation, the trained perceptual model may be applied to an autonomous vehicle, the autonomous vehicle may be equipped with a laser radar and a target detection system, and the target detection system may include the perceptual model trained by using the training method in the foregoing embodiments of the present specification. Real point cloud data are continuously collected through the laser radar, and are input into the perception model for target recognition, so that a target detection result can be obtained.
In a specific implementation, the target detection result may specifically be based on the identified target object, and output detection information included therein, such as the target object and pose information thereof. As a specific application example, the data may be further processed according to a specific service requirement, or the detection result may be further input to a downstream application to be used as a decision judgment of a specific service.
For example, the detection result may be displayed as an image, or may be displayed as a map. For another example, the detection result may be output to an automatic driving system for determining a corresponding driving strategy, including driving speed, whether to avoid an obstacle, and the like.
Before the adopted laser radar perception model is trained, firstly, a target model is obtained based on actual point cloud data, the beam information of the laser radar is calibrated and measured, the distribution of laser beams is obtained, simulated point cloud data is obtained according to the reflected laser points distributed on the target model by the laser beams, the simulated point cloud data obtained by the amplification and the actual point cloud data are fused and used for the training of the perception model, the scale and the diversity of data used for perception model training can be greatly expanded, and therefore the obtained perception model has better robustness, and the laser radar perception model is used for target detection, so that the target detection performance can be improved, including the identification capability in a small probability scene.
The embodiment of the present specification further provides product embodiments corresponding to the above training method for the perceptual model, and the product embodiments are correspondingly described below with reference to the accompanying drawings.
Referring to the schematic structural diagram of the training apparatus for perceptual model shown in fig. 7, in this embodiment, the training apparatus 70 for perceptual model may include: an actual point cloud obtaining unit 71, a modeling unit 72, a calibration measuring unit 73, a simulation point cloud obtaining unit 74 and a fusion training unit 75, wherein:
the actual point cloud obtaining unit 71 is adapted to obtain actual point cloud data continuously collected by the laser radar;
the modeling unit 72 is suitable for performing target modeling according to the actual point cloud data to obtain a target model;
the calibration measuring unit 73 is adapted to calibrate and measure the beam information of the laser radar to obtain the distribution of the laser beams;
the simulated point cloud obtaining unit 74 is adapted to obtain simulated point cloud data according to the reflected laser points of the laser beam distributed on the target model;
the fusion training unit 75 is adapted to obtain fusion training data according to the actual point cloud data and the simulated point cloud data, and train a perception model by using the fusion training data.
In a specific implementation, the modeling unit 72 is adapted to intercept the actual point cloud data to obtain a target point cloud, register multiple frames of target point clouds to obtain a dense point cloud, and reconstruct a point cloud surface based on the dense point cloud to obtain the target model.
As an alternative example, the modeling unit 72 may further perform denoising, land removal and/or smoothing on the dense point cloud before performing the point cloud surface reconstruction, so as to obtain a smooth dense point cloud for the point cloud surface reconstruction.
The method for registering multiple frames of target point clouds by the modeling unit 72 to obtain a dense point cloud may specifically refer to the foregoing embodiments, and a description thereof is not repeated here.
In a specific implementation, the simulated point cloud obtaining unit 74 obtains simulated point cloud data in a plurality of ways according to the reflected laser points of the laser beam distributed on the target model, for example, the simulated point cloud data may be obtained in at least one of the following ways:
1) According to the reflected laser points of the laser beam distributed on the target model in different scenes, obtaining the simulation point cloud data;
2) According to the reflected laser points distributed on the target model under different laser radars and different sensor setting parameters of the laser beam, obtaining the simulation point cloud data;
3) And obtaining the simulation point cloud data according to the reflected laser points of the laser beam distributed on the target model under different poses of the target model.
In a specific implementation, the fusion training unit 75 is adapted to determine the usage amount and the fusion ratio of the actual point cloud data and the simulated point cloud data according to the required data amount of the perception model, so as to obtain the fusion training data.
In specific implementation, the training method for the perceptual model according to the embodiment may be run on an electronic device, such as a computer terminal, including a vehicle-mounted terminal, a personal computer, a tablet computer, or the like, or may be run on a server, a cloud, or through a computer cluster, and the training apparatus for the perceptual model according to the embodiment may also be installed, stored, and run on the electronic device.
Referring to a schematic structural diagram of an electronic device shown in fig. 8, an embodiment of the present specification provides an electronic device, as shown in fig. 8, an electronic device 80 includes a memory 81 and a processor 82, where the memory 81 stores computer instructions executable on the processor 82, and when the processor 82 executes the computer instructions, the steps of the method according to any one of the foregoing embodiments are performed.
In a specific implementation, the electronic device may further include a display 83 adapted to display the operation result, and may also output and display the intermediate execution process.
In specific implementation, the real point cloud data or the training data may be obtained through the input interface 84, or the actual point cloud data or the training data may be obtained through the communication interface 85.
In one implementation, the memory 81, the processor 82, the display 83, the input interface 84, and the communication interface 85 may communicate with each other via a bus 86.
The present specification also provides a computer readable storage medium, on which computer instructions are stored, wherein the computer instructions can execute the steps of the method of any one of the foregoing embodiments when executed.
In particular implementations, the computer-readable storage medium may be a variety of suitable readable storage media such as an optical disk, a mechanical hard disk, a solid state disk, and so on.
As described in the previous embodiment, the aforementioned perception model may correspond to an autonomous automobile. Corresponding algorithm modules may be provided in the autonomous vehicle for autonomous driving decisions. Of course, these algorithm modules may vary depending on the type of autonomous vehicle. For example, different algorithm modules may be involved for logistics vehicles, public service vehicles, medical service vehicles, terminal service vehicles. The algorithm modules are illustrated below for these four autonomous vehicles, respectively:
the logistics vehicle refers to a vehicle used in a logistics scene, and may be, for example, a logistics vehicle with an automatic sorting function, a logistics vehicle with a refrigeration and heat preservation function, and a logistics vehicle with a measurement function. These logistics vehicles may involve different algorithm modules.
For example, the logistics vehicles can be provided with an automatic sorting device which can automatically take out, convey, sort and store the goods after the logistics vehicles reach the destination. This relates to an algorithm module for sorting goods, which mainly implements logic control of goods taking out, carrying, sorting and storing.
For another example, in a cold chain logistics scenario, the logistics vehicle may further include a refrigeration and insulation device, and the refrigeration and insulation device may implement refrigeration or insulation of transported fruits, vegetables, aquatic products, frozen foods, and other perishable foods, so that the transportation environment is in a proper temperature environment, and the long-distance transportation problem of perishable foods is solved. The algorithm module is mainly used for dynamically and adaptively calculating the proper temperature of cold meal or heat preservation according to the information such as the property, the perishability, the transportation time, the current season, the climate and the like of food (or articles), and automatically adjusting the cold-storage heat preservation device according to the proper temperature, so that a transport worker does not need to manually adjust the temperature when the vehicle transports different foods or articles, the transport worker is liberated from the complicated temperature regulation and control, and the efficiency of cold-storage heat preservation transportation is improved.
For another example, in most logistics scenarios, the fee is charged according to the volume and/or weight of the parcel, but the number of the logistics parcels is very large, and the measurement of the volume and/or weight of the parcel by only depending on a courier is very inefficient and has high labor cost. Therefore, in some logistics vehicles, a measuring device is additionally arranged, so that the volume and/or the weight of the logistics packages can be automatically measured, and the cost of the logistics packages can be calculated. This relates to an algorithm module for logistics package measurement, which is mainly used to identify the type of logistics package, determine the measurement mode of logistics package, such as volume measurement or weight measurement or combined measurement of volume and weight, and can complete the measurement of volume and/or weight according to the determined measurement mode and complete the cost calculation according to the measurement result.
The public service vehicle is a vehicle providing some public service, and may be, for example, a fire truck, an ice removal truck, a watering cart, a snow scraper, a garbage disposal vehicle, a traffic guidance vehicle, and the like. These public service vehicles may involve different algorithm modules.
For example, in the case of an autonomous fire fighting vehicle, the main task is to perform a reasonable fire fighting task for the fire scene, which involves an algorithm for the fire fighting task, which at least requires logic for the identification of the fire situation, the planning of the fire fighting scheme and the automatic control of the fire fighting equipment.
For another example, for an ice removing vehicle, the main task is to remove ice and snow on the road surface, which involves an algorithm module for ice removal, the algorithm module at least needs to realize the recognition of the ice and snow condition on the road surface, formulate an ice removal scheme according to the ice and snow condition, such as which road sections need to be deiced, which road sections need not to be deiced, whether a salt spreading manner, the salt spreading gram number, and the like are adopted, and the logic of automatic control of a deicing device under the condition of determining the ice removal scheme.
The medical service vehicle is an automatic driving vehicle capable of providing one or more medical services, the vehicle can provide medical services such as disinfection, temperature measurement, dispensing and isolation, and the algorithm module relates to algorithm modules for providing various self-service medical services, the algorithm modules mainly realize identification of disinfection requirements and control of a disinfection device so that the disinfection device can disinfect patients, or identify the positions of the patients, control the temperature measurement device to automatically press close to the forehead and the like of the patients to measure the temperature of the patients, or is used for realizing judgment of symptoms, giving out prescriptions according to judgment results and realizing identification of medicine/medicine containers and control of a medicine taking manipulator so that the medicine taking manipulator can grab medicines for the patients according to the prescriptions, and the like.
The terminal service vehicle is a self-service type automatic driving vehicle which can replace some terminal devices to provide certain convenient service for users, and for example, the vehicles can provide services such as printing, attendance checking, scanning, unlocking, payment and retail for the users.
For example, in some application scenarios, a user often needs to go to a specific location to print or scan a document, which is time consuming and labor intensive. Therefore, a terminal service vehicle capable of providing printing/scanning service for a user appears, the service vehicles can be interconnected with user terminal equipment, the user sends a printing instruction through the terminal equipment, the service vehicle responds to the printing instruction, documents required by the user are automatically printed, the printed documents can be automatically sent to the position of the user, the user does not need to queue at a printer, and the printing efficiency can be greatly improved. Or, the scanning instruction sent by the user through the terminal equipment can be responded, the scanning vehicle is moved to the position of the user, the user places the document to be scanned on the scanning tool of the service vehicle to complete scanning, queuing at the printer/scanner is not needed, and time and labor are saved. This involves an algorithm module providing print/scan services that needs to identify at least the interconnection with the user terminal equipment, the response to print/scan instructions, the positioning of the user's location, and travel control.
For another example, with the development of new retail business, more and more e-commerce delivers goods to various large office buildings and public areas by means of vending machines, but the vending machines are placed at fixed positions and are not movable, and users need to go by the vending machines to purchase the needed goods, which is still poor in convenience. Therefore, self-service driving vehicles capable of providing retail services appear, the service vehicles can carry commodities to move automatically and can provide corresponding self-service shopping Applications (APPs) or shopping entrances, a user can place an order for the self-service driving vehicles capable of providing retail services through the APPs or shopping entrances by means of a terminal such as a mobile phone, the order comprises names and the number of commodities to be purchased and the position of the user, after the vehicle receives an order placing request, whether the current remaining commodities have the commodities purchased by the user and the number of the commodities is enough can be determined, the commodities can be carried to the position of the user automatically and can be provided for the user when the commodities which are purchased by the user and the number of the commodities are enough, the shopping convenience of the user is further improved, the time of the user is saved, and the time of the user is used for more important things. This involves algorithm modules that provide retail services that implement logic primarily to respond to customer order requests, order processing, merchandise information maintenance, customer location, payment management, etc.
Although the embodiments of the present invention are disclosed above, the present invention is not limited thereto. Various changes and modifications may be effected therein by one skilled in the art without departing from the spirit and scope of the invention as defined in the appended claims.
Claims (10)
1. A method for training a perception model comprises the following steps:
acquiring actual point cloud data continuously acquired by a laser radar;
carrying out target modeling according to the actual point cloud data to obtain a target model;
calibrating and measuring the beam information of the laser radar to obtain the distribution of laser beams;
obtaining simulation point cloud data according to the reflected laser points of the laser beam distributed on the target model;
and obtaining fusion training data according to the actual point cloud data and the simulation point cloud data, and training the perception model by using the fusion training data.
2. The method of claim 1, wherein the modeling the target from the actual point cloud data to obtain a target model comprises:
intercepting the actual point cloud data to obtain a target point cloud;
registering multiple frames of target point clouds to obtain dense point clouds;
and based on the dense point cloud, obtaining the target model through point cloud surface reconstruction.
3. The method of claim 2, wherein prior to performing point cloud surface reconstruction, further comprising:
and denoising, ground removing and/or smoothing the dense point cloud to obtain a smooth dense point cloud for point cloud surface reconstruction.
4. The method of claim 2, wherein the registering the multiple frames of target point clouds to obtain a dense point cloud comprises:
and selecting a first frame from the multi-frame target point cloud as a basic frame according to the acquisition sequence, and sequentially matching and overlapping subsequent frames in the multi-frame target point cloud to the overlapping result of the previous frame according to the acquisition sequence to obtain the dense point cloud.
5. The method according to claim 4, wherein the sequentially matching and superimposing subsequent frames in the plurality of frames of the target point cloud on the superimposed result of the previous frame according to the collecting order to obtain the dense point cloud comprises:
and sequentially carrying out global matching on the subsequent frames in the multi-frame target point cloud and the superposition result of the corresponding previous frame, then carrying out local matching according to the decreasing step length until the matching degree reaches a preset threshold value, and then carrying out matching superposition on the subsequent frame and the current superposition result until the matching superposition of the last frame is completed.
6. The method of claim 2, wherein the registering the multiple frames of target point clouds to obtain a dense point cloud comprises:
dividing the multi-frame target point cloud into a plurality of subsequences according to the acquisition sequence;
respectively taking the first frame in the subsequences as the basic frame of the corresponding subsequence, and sequentially matching and overlapping the subsequent frames to the overlapping result of the previous frame according to the acquisition sequence for any subsequence until the matching and overlapping of the last frame in the subsequences are completed;
and sequentially matching and overlapping the overlapping results of the plurality of subsequences according to the acquisition sequence to obtain the dense point cloud.
7. The method of claim 1, wherein said deriving simulated point cloud data from reflected laser points of said laser beam distributed on said target model comprises:
according to the reflected laser points of the laser beam distributed on the target model in different scenes, obtaining the simulation point cloud data;
and/or the presence of a gas in the gas,
according to the reflected laser points distributed on the target model under different laser radars and different sensor setting parameters of the laser beam, obtaining the simulation point cloud data;
and/or the presence of a gas in the atmosphere,
and obtaining the simulation point cloud data according to the reflected laser points of the laser beam distributed on the target model under different poses of the target model.
8. The method of claim 1, wherein the deriving fused training data from the actual point cloud data and the simulated point cloud data comprises:
and determining the using quantity and the fusion proportion of the actual point cloud data and the simulated point cloud data according to the required data quantity of the perception model to obtain the fusion training data.
9. A training apparatus for a perceptual model, comprising:
the actual point cloud acquisition unit is suitable for acquiring actual point cloud data continuously acquired by the laser radar;
the modeling unit is suitable for carrying out target modeling according to the actual point cloud data to obtain a target model;
the calibration measuring unit is suitable for calibrating and measuring the beam information of the laser radar to obtain the distribution of laser beams;
the simulated point cloud acquisition unit is suitable for acquiring simulated point cloud data according to the reflected laser points of the laser beam distributed on the target model;
and the fusion training unit is suitable for obtaining fusion training data according to the actual point cloud data and the simulation point cloud data and training a perception model by using the fusion training data.
10. An electronic device comprising a memory and a processor, the memory having stored thereon computer instructions executable on the processor, wherein the processor, when executing the computer instructions, performs the steps of the method of any of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110378156.3A CN115205610A (en) | 2021-04-08 | 2021-04-08 | Training method and training device for perception model and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110378156.3A CN115205610A (en) | 2021-04-08 | 2021-04-08 | Training method and training device for perception model and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115205610A true CN115205610A (en) | 2022-10-18 |
Family
ID=83571166
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110378156.3A Pending CN115205610A (en) | 2021-04-08 | 2021-04-08 | Training method and training device for perception model and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115205610A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115542338A (en) * | 2022-11-30 | 2022-12-30 | 湖南大学 | Laser radar data learning method based on point cloud space distribution mapping |
| CN116071621A (en) * | 2023-03-15 | 2023-05-05 | 中汽智联技术有限公司 | Training sample generation and verification method, device and medium for perception algorithm |
| CN116071618A (en) * | 2023-02-23 | 2023-05-05 | 深圳佑驾创新科技有限公司 | 3D point cloud generation method, device, equipment and medium based on simulation data |
| CN116152770A (en) * | 2023-04-19 | 2023-05-23 | 深圳佑驾创新科技有限公司 | 3D target matching model building method and device |
| WO2024156093A1 (en) * | 2023-01-28 | 2024-08-02 | Huawei Technologies Co., Ltd. | Systems, methods, and media for generating point cloud frame training data |
| WO2024187340A1 (en) * | 2023-03-13 | 2024-09-19 | 北京易控智驾科技有限公司 | Training method and apparatus, target detection method, electronic device, and storage medium |
-
2021
- 2021-04-08 CN CN202110378156.3A patent/CN115205610A/en active Pending
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115542338A (en) * | 2022-11-30 | 2022-12-30 | 湖南大学 | Laser radar data learning method based on point cloud space distribution mapping |
| WO2024156093A1 (en) * | 2023-01-28 | 2024-08-02 | Huawei Technologies Co., Ltd. | Systems, methods, and media for generating point cloud frame training data |
| CN116071618A (en) * | 2023-02-23 | 2023-05-05 | 深圳佑驾创新科技有限公司 | 3D point cloud generation method, device, equipment and medium based on simulation data |
| CN116071618B (en) * | 2023-02-23 | 2023-06-20 | 深圳佑驾创新科技有限公司 | 3D point cloud generation method, device, equipment and medium based on simulation data |
| WO2024187340A1 (en) * | 2023-03-13 | 2024-09-19 | 北京易控智驾科技有限公司 | Training method and apparatus, target detection method, electronic device, and storage medium |
| CN116071621A (en) * | 2023-03-15 | 2023-05-05 | 中汽智联技术有限公司 | Training sample generation and verification method, device and medium for perception algorithm |
| CN116152770A (en) * | 2023-04-19 | 2023-05-23 | 深圳佑驾创新科技有限公司 | 3D target matching model building method and device |
| CN116152770B (en) * | 2023-04-19 | 2023-09-22 | 深圳佑驾创新科技股份有限公司 | 3D target matching model building method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN115205610A (en) | Training method and training device for perception model and electronic equipment | |
| CN110059608B (en) | Object detection method and device, electronic equipment and storage medium | |
| CN109844807B (en) | Method, system and apparatus for segmenting and sizing objects | |
| US20180330207A1 (en) | Deep Image-to-Image Network Learning for Medical Image Analysis | |
| CN112749594B (en) | Information completion method, lane line identification method, intelligent driving method and related products | |
| EP3203264B1 (en) | Method of imaging an object for tracking and documentation during transportation and storage | |
| Qin | A critical analysis of satellite stereo pairs for digital surface model generation and a matching quality prediction model | |
| CN102982560B (en) | According to the surface segmentation of RGB and depth image | |
| US20180173971A1 (en) | Pedestrian detection neural networks | |
| WO2018125928A1 (en) | Multi-channel sensor simulation for autonomous control systems | |
| CN111241969A (en) | Target detection method and device and corresponding model training method and device | |
| KR102308456B1 (en) | Tree species detection system based on LiDAR and RGB camera and Detection method of the same | |
| McPhee et al. | Live animal assessments of rump fat and muscle score in Angus cows and steers using 3-dimensional imaging | |
| CN111814739B (en) | Method, device, equipment and storage medium for detecting express package volume | |
| CN111080171A (en) | Logistics allocation method based on logistics allocation algorithm | |
| CN110135396A (en) | Recognition methods, device, equipment and the medium of surface mark | |
| Milella et al. | Towards intelligent retail: Automated on-shelf availability estimation using a depth camera | |
| CN116596835A (en) | Express package defect detection reminding method, device, equipment and storage medium | |
| Potó et al. | Laser scanned point clouds to support autonomous vehicles | |
| JP2021154935A (en) | Vehicle simulation system, vehicle simulation method and computer program | |
| CN112489240B (en) | Commodity display inspection method, inspection robot and storage medium | |
| JP2020109602A (en) | Model generation device, vehicle simulation system, model generation method and computer program | |
| Li et al. | Research on a random sampling method for bulk grain based on the M-Unet and SGBM algorithms | |
| JP6894395B2 (en) | Information acquisition device, information aggregation system, and information aggregation device | |
| CN115171061A (en) | Vehicle detection method, device and equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20230712 Address after: Room 437, Floor 4, Building 3, No. 969, Wenyi West Road, Wuchang Subdistrict, Yuhang District, Hangzhou City, Zhejiang Province Applicant after: Wuzhou Online E-Commerce (Beijing) Co.,Ltd. Address before: Room 01, 45 / F, AXA building, 8 Shanton Road, Singapore Applicant before: Alibaba Singapore Holdings Ltd. |
|
| TA01 | Transfer of patent application right |