CN115994589A - Training method and device, target detection method, electronic device and storage medium - Google Patents
Training method and device, target detection method, electronic device and storage medium Download PDFInfo
- Publication number
- CN115994589A CN115994589A CN202310287318.1A CN202310287318A CN115994589A CN 115994589 A CN115994589 A CN 115994589A CN 202310287318 A CN202310287318 A CN 202310287318A CN 115994589 A CN115994589 A CN 115994589A
- Authority
- CN
- China
- Prior art keywords
- training
- data set
- training data
- target
- lidar unit
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000012549 training Methods 0.000 title claims abstract description 327
- 238000001514 detection method Methods 0.000 title claims abstract description 290
- 238000000034 method Methods 0.000 title claims abstract description 57
- 238000003860 storage Methods 0.000 title claims abstract description 14
- 230000006870 function Effects 0.000 claims description 20
- 238000005457 optimization Methods 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 9
- 230000008901 benefit Effects 0.000 description 6
- 230000008569 process Effects 0.000 description 6
- 238000000280 densification Methods 0.000 description 5
- 238000012545 processing Methods 0.000 description 5
- 238000005070 sampling Methods 0.000 description 4
- 239000013589 supplement Substances 0.000 description 4
- 238000004519 manufacturing process Methods 0.000 description 3
- 239000000203 mixture Substances 0.000 description 3
- 230000004075 alteration Effects 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000005065 mining Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000007792 addition Methods 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000000354 decomposition reaction Methods 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000000047 product Substances 0.000 description 1
- 238000005215 recombination Methods 0.000 description 1
- 230000006798 recombination Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Optical Radar Systems And Details Thereof (AREA)
Abstract
The present disclosure relates to a training method and apparatus, a target detection method, an electronic device, and a storage medium. The training method is used for training an object detection device at least comprising a plurality of laser radar units with different wire harnesses. The training method comprises the following steps: generating a third training data set based on the first training data set and the second training data set; training the initial detection model by using the first training data set and the third training data set which is randomly sampled, and generating a first detection model; training is performed on the first detection model by using the third training data set and the randomly sampled first training data set, a second detection model is generated, the first wire harness of the first lidar unit is higher than the second wire harness of the second lidar unit, and the first field of view of the first lidar unit is smaller than the second field of view of the second lidar unit. Meanwhile, accurate target detection of a front-view long distance and a near-view short distance is realized, and the robustness of the target detection device is ensured.
Description
Technical Field
The present disclosure relates to the field of unmanned technologies, and in particular, to a training method and apparatus, a target detection method, an electronic device, and a storage medium.
Background
Currently, 3D object detection has been widely used in the field of autopilot and unmanned, for providing accurate object size, category and location. For example, in the field of unmanned mine cars, an all-weather high-efficiency operation is required, and a general camera cannot provide accurate target detection information due to the influence of low illuminance at night. Therefore, in all-weather operation scenarios such as unmanned mine cars, lidar is often selected as a sensor for 3D object detection.
Further, unmanned mining vehicles require both forward looking remote accurate target detection and look-around near accurate target detection, and it is desirable that the target detection device be robust, i.e., that the target detection device still be able to operate robustly when either the forward looking radar or the look-around radar is damaged. At present, the target detection device of the existing unmanned mine car is difficult to realize accurate target detection of a long forward looking distance and a short around looking distance simultaneously, and the robustness of the target detection device is guaranteed.
Disclosure of Invention
The present disclosure has been made in view of the above-described problems. The present disclosure provides a training method and apparatus for an object detection apparatus of an unmanned vehicle, an object detection method, an electronic device, and a storage medium.
According to one aspect of the present disclosure, there is provided a training method for an object detection apparatus including at least a plurality of lidar units having different harnesses, including: generating a third training data set for the second lidar unit based on the first training data set for the first lidar unit and the second training data set for the second lidar unit; training the initial detection model using the first training data set and the randomly sampled third training data set to generate a first detection model for the first lidar unit; training is performed on the first detection model using the third training data set and the randomly sampled first training data set to generate a second detection model for the second lidar unit, wherein the first beam of the first lidar unit is higher than the second beam of the second lidar unit and the first field of view of the first lidar unit is less than the second field of view of the second lidar unit.
Furthermore, according to a training method of one aspect of the present disclosure, generating a third training data set for the second lidar unit based on the first training data set for the first lidar unit and the second training data set for the second lidar unit comprises: determining target objects matched with each other in the first training data set and the second training data set; taking target data corresponding to the target object in the second training data set as training input, taking target data corresponding to the target object in the first training data set as training truth value, and executing training to obtain a training data set generation model; and generating a third training data set using the training data generation model based on the second training data set.
Furthermore, a training method according to an aspect of the present disclosure further includes: performing training on the second detection model using the second training dataset, generating an optimized second detection model for the second lidar unit; and performing training on the optimized second detection model using the first training dataset, generating an optimized first detection model for the first lidar unit.
Furthermore, a training method according to an aspect of the present disclosure further includes: performing target matching on a first target detection result and a second target detection result of the first detection model and the second detection model aiming at the same training data set, and determining a corresponding target pair; determining a consistency function of the first target detection result and the second target detection result aiming at the corresponding target pair; and adjusting the first detection model and the second detection model based on the consistency function.
According to another aspect of the present disclosure, there is provided an object detection method for a vehicle including at least an object detection device configured by a plurality of lidar units having different wire harnesses, including: detecting working state information of a plurality of laser radar units; acquiring running state information of a vehicle and running environment information of the vehicle; and controlling the plurality of lidar units to perform target detection based on the operating state information, and the operating environment information, wherein the plurality of lidar units includes a first lidar unit and a second lidar unit, a first wire harness of the first lidar unit is higher than a second wire harness of the second lidar unit, and a first field of view of the first lidar unit is less than a second field of view of the second lidar unit.
Further, according to another aspect of the present disclosure, an object detection method, using the training method as above, trains an object detection device.
According to yet another aspect of the present disclosure, there is provided a training device for an object detection device including at least a plurality of lidar units having different wire harnesses, including: a training data set generation unit configured to generate a third training data set for the second lidar unit based on the first training data set for the first lidar unit and the second training data set for the second lidar unit; a first detection model training unit configured to perform training on the initial detection model using the first training data set and the randomly sampled third training data set, generating a first detection model for the first lidar unit; and a second detection model training unit configured to perform training on the first detection model using the third training data set and the randomly sampled first training data set, to generate a second detection model for the second lidar unit, wherein the first beam of the first lidar unit is higher than the second beam of the second lidar unit, and the first field of view of the first lidar unit is less than the second field of view of the second lidar unit.
Furthermore, according to a training apparatus of still another aspect of the present disclosure, the training data set generating unit is configured to: determining target objects matched with each other in the first training data set and the second training data set; taking target data corresponding to the target object in the second training data set as training input, taking target data corresponding to the target object in the first training data set as training truth value, and executing training to obtain a training data set generation model; and generating a third training data set using the training data generation model based on the second training data set.
Furthermore, according to a training device of a further aspect of the present disclosure, the second detection model training unit is further configured to perform training on the second detection model using the second training data set, generating a second detection model for optimization of the second lidar unit; and the first detection model training unit is further configured to perform training on the optimized second detection model using the first training data set, generating an optimized first detection model for the first lidar unit.
Furthermore, the training device according to still another aspect of the present disclosure further includes a target matching training unit configured to: performing target matching on a first target detection result and a second target detection result of the first detection model and the second detection model aiming at the same training data set, and determining a corresponding target pair; determining a consistency function of the first target detection result and the second target detection result aiming at the corresponding target pair; and adjusting the first detection model and the second detection model based on the consistency function.
According to still another aspect of the present disclosure, there is provided an electronic device including: a memory for storing computer readable instructions; and a processor for executing the computer readable instructions to cause the electronic device to perform the training method or the target detection method as above.
According to yet another aspect of the present disclosure, there is provided a non-transitory computer readable storage medium storing computer readable instructions which, when executed by a processor, cause the processor to perform the training method or the target detection method as above.
According to yet another aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements a training method or an object detection method as above.
As will be described in detail below, according to the training method and apparatus, the target detection method, the electronic device, and the storage medium of the target detection apparatus of the embodiments of the present disclosure, by taking a front view dense training target as a training truth value of a surround view sparse training target in a front view and surround view overlapping area, training a point cloud dense model, and then dense all surround view training target point clouds using the dense model, the problem of accuracy degradation caused by the surround view training target point clouds becoming sparse is solved; the front training targets and the around training targets are respectively and alternately randomly placed into the around and front scenes after being sampled, so that the quantity and the richness of the training targets are enhanced, a detection model for around is initialized by using a detection model trained by a front training data set, after the detection model for around is trained, the weight is used for the front detection model, and finally, the front model is finely adjusted, so that iterative training is realized; finally, the front and around detection models are trained and adjusted by using the consistency function by using the same targets in the front and around overlapping areas, which should have the same category, size and distance attributes. In an unmanned mine car for example, with the object detection device including the forward-looking and through-looking detection models thus obtained, accurate object detection at both the forward-looking long distance and the through-looking short distance can be achieved, and the robustness of the object detection device is ensured.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and are intended to provide further explanation of the technology claimed.
Drawings
The above and other objects, features and advantages of the present disclosure will become more apparent by describing in more detail embodiments thereof with reference to the attached drawings. The accompanying drawings are included to provide a further understanding of embodiments of the disclosure, and are incorporated in and constitute a part of this specification, illustrate embodiments of the disclosure and together with the description serve to explain the disclosure, without limitation to the disclosure. In the drawings, like reference numerals generally refer to like parts or steps.
FIG. 1 is a schematic diagram outlining an object detection apparatus according to an embodiment of the present disclosure;
FIG. 2 is a flow chart illustrating a training method of an object detection device according to an embodiment of the present disclosure;
FIG. 3 is a flow chart further illustrating the generation of a densified training data set in a training method of an object detection apparatus according to an embodiment of the present disclosure;
FIG. 4 is a flow chart further illustrating a training method of the object detection apparatus according to an embodiment of the present disclosure;
FIG. 5 is a flow chart further illustrating a training method of the object detection apparatus according to an embodiment of the present disclosure;
FIG. 6 is a flowchart illustrating a target detection method according to an embodiment of the present disclosure;
FIG. 7 is a functional block diagram illustrating a training device of the object detection device according to an embodiment of the present disclosure;
FIG. 8 is a hardware block diagram illustrating an electronic device according to an embodiment of the disclosure; and
fig. 9 is a schematic diagram illustrating a computer-readable storage medium according to an embodiment of the present disclosure.
Detailed Description
In order to make the objects, technical solutions and advantages of the present disclosure more apparent, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. It should be apparent that the described embodiments are only some of the embodiments of the present disclosure and not all of the embodiments of the present disclosure, and that the present disclosure is not limited by the example embodiments described herein.
First, an object detection apparatus according to an embodiment of the present disclosure is summarized with reference to fig. 1.
As shown in fig. 1, an object detection device 10 according to an embodiment of the present disclosure is configured in a vehicle 1. In one embodiment of the present disclosure, the vehicle 1 may be, for example, a transportation device in a mine work area for transporting mining products out of the work area, such as an unmanned mine car. It is to be readily understood that the present disclosure is not limited thereto, and the vehicle 1 may be any other unmanned, assisted-driving, and manual-driving vehicle provided with the object detection device 10.
Further, as shown in fig. 1, the object detection device 10 according to the embodiment of the present disclosure includes a first lidar unit a disposed at the head top of the vehicle 1, and second lidar units B1, B2, and B3 disposed at both sides of the head and the tail of the vehicle 1. In the embodiment of the present disclosure, the first lidar unit a is used as a front view main lidar unit of the target detection apparatus 10, and the second lidar units B1, B2, and B3 are used as look-around blind supplement lidar units of the target detection apparatus 10. The first wire harness (e.g., 64 lines, 128 lines, 512 lines) of the first lidar unit a is higher than the second wire harness (e.g., 6 lines, 8 lines, 16 lines) of the second lidar units B1, B2, and B3. The first field angle 111 (e.g., about 120 ° in the horizontal direction) of the first lidar unit a is smaller than the second field angle 112 (e.g., about 360 ° in the total field angle in the horizontal direction) of the second lidar units B1, B2, and B3.
The object detection device 10 thus configured, by covering the far forward looking region in the traveling direction of the vehicle 1 with the high-harness first lidar unit a, and covering the near looking region by about 360 ° with the low-harness second lidar units B1, B2, and B3, ensures both accurate detection of the object and environmental information of the far forward looking region and no dead angle coverage for the near around the vehicle. It will be readily appreciated that the configuration of fig. 1 is merely exemplary, that the high beam first lidar unit may be more than one unit, that the low beam second lidar unit may be more or less than three units, and that the particular configuration locations of the first lidar unit and the second lidar unit are not limited to the locations shown in fig. 1.
Thus, accurate target detection at a far distance in front view and full-view target detection at a near distance in whole view are realized simultaneously. In addition, in order for the target detection device to be capable of having robustness when the forward looking main lidar unit or the look-around blind-supplement lidar unit is damaged, the forward looking main lidar unit and the look-around blind-supplement lidar unit will be respectively configured with independent target detection models, so that even if one of the forward looking main lidar unit and the look-around blind-supplement lidar unit fails, the other radar unit can be relied on at least to keep the vehicle running.
Hereinafter, a training method for an object detection apparatus and an object detection method according to embodiments of the present disclosure will be further described with reference to the accompanying drawings.
Fig. 2 is a flowchart illustrating a training method of an object detection apparatus according to an embodiment of the present disclosure.
As shown in fig. 2, the training method of the object detection device according to the embodiment of the present disclosure includes the following steps.
In step S201, a third training data set for the second lidar unit is generated based on the first training data set for the first lidar unit and the second training data set for the second lidar unit.
As described above, since the first lidar unit a is a high-beam lidar unit and the second lidar units B1, B2, and B3 are low-beam lidar units, the target objects in the first training data set for the first lidar unit a are denser than the target objects in the second training data set for the second lidar units B1, B2, and B3. In target detection of an actual lidar unit, the performance of target detection depends on the number of point clouds of the target. The denser the target point cloud, the better the target detection performance. Thus, in a training method of an object detection device according to an embodiment of the present disclosure, a third training data set for a second lidar unit is generated based on a first training data set for the first lidar unit and a second training data set for the second lidar unit. That is, the target objects in the second training data set are densified based on the first training data set with the target objects being denser, thereby obtaining a third training data set with the target objects being denser for training the second lidar unit. Hereinafter, a flow of generating a densified training data set in a training method of an object detection apparatus according to an embodiment of the present disclosure will be specifically described with reference to fig. 3.
As shown in fig. 3, the flow of generating a densified training data set in the training method of the object detection apparatus according to the embodiment of the present disclosure includes the following steps.
In step S301, a target object in which the first training data set and the second training data set match each other is determined. For example, as shown in fig. 1, there is a partial overlap of the first field of view of the first lidar unit a with the second fields of view of the second lidar units B1, B2, and B3, so that the same target object exists in the first training data set for the first lidar unit and the second training data set for the second lidar unit.
In step S302, training is performed with the target data corresponding to the target object in the second training data set as a training input and the target data corresponding to the target object in the first training data set as a training truth value to acquire a training data set generation model.
In step S303, a third training data set is generated using the training data set generation model based on the second training data set. By the training data set generation model acquired in step S302 as above, a densification process may be performed on the second training data set for the second lidar unit, thereby acquiring a third training data set for the second lidar unit, the target object in the third training data set being denser than the target object in the second training data set.
Next, a training method of the object detection apparatus according to an embodiment of the present disclosure is described with continued reference back to fig. 2.
In step S202, training is performed on the initial detection model using the first training data set and the randomly sampled third training data set, generating a first detection model for the first lidar unit.
In embodiments of the present disclosure, the initial detection model for the first lidar unit may be any 3D target detection model. By randomly sampling the third training data set obtained in step S201 and adding the result of random sampling to the first training data set to perform training on the initial detection model, the number and the richness of the categories of the target objects in the training data set are further improved, and thus the detection accuracy of the first detection model obtained by training is improved.
In step S203, training is performed on the first detection model using the third training data set and the randomly sampled first training data set, generating a second detection model for the second lidar unit.
In an embodiment of the present disclosure, the initial detection model for the second lidar unit may be the first detection model acquired in step S202. Training is performed on the initial detection model (i.e., the first detection model) of the second lidar unit by randomly sampling the first training data set and adding the result of the random sampling to the densified third training data set acquired in step S201, thereby acquiring a second detection model for the second lidar unit.
In the training method of the target detection apparatus according to the embodiment of the present disclosure described with reference to fig. 2 and 3, the third training data set for densification of the second lidar unit is generated by training the training data set generation model that performs the training target densification process, and the detection accuracy of the detection model obtained by training is improved by cross-initializing the first detection model for the first lidar unit and the second detection model for the second lidar unit in the training phase to perform iterative training.
Fig. 4 is a flowchart further illustrating a training method of the object detection apparatus according to an embodiment of the present disclosure. Steps S401 to S403 shown in fig. 4 are the same as S201 to S203 shown in fig. 2, respectively, and repeated description thereof will be omitted herein.
In step S404, training is performed on the second detection model using the second training data set, generating an optimized second detection model for the second lidar unit.
As described above, the second detection model is obtained by training with the densified third training data set and the randomly sampled first training data set in step S403. In an actual target detection scenario, the second detection model is full view target detection for a near field of view. Therefore, in order to further adapt to the actual target detection scene, training adjustment is further performed on the second detection model by using the relatively sparse second training data set, so as to obtain an optimized second detection model more conforming to the all-view target detection scene in the near-looking range.
In step S405, training is performed on the optimized second detection model using the first training dataset, generating an optimized first detection model for the first lidar unit.
Similar to the description in step S404, in an actual target detection scenario, the first detection model is an accurate target detection for forward looking distance. Therefore, in order to further adapt the actual target detection scenario, training adjustment is further performed on the optimized second detection model using the first training data set for forward looking target detection, thereby obtaining an optimized first detection model that more conforms to the forward looking remote accurate target detection scenario.
In the training method of the object detection apparatus according to the embodiment of the present disclosure described with reference to fig. 4, by further performing training and adjustment using training data sets conforming to respective actual object detection scenes, an object detection model realizing higher detection performance in the actual object detection scenes is obtained.
Fig. 5 is a flowchart further illustrating a training method of the object detection apparatus according to an embodiment of the present disclosure. In an embodiment of the present disclosure, after the first and second target detection models are obtained as described above with reference to fig. 2 and the optimized first and second target detection models are obtained as described above with reference to fig. 4, the obtained target detection models may be further adjusted and optimized.
In step S501, object matching is performed on the first object detection result and the second object detection result of the first detection model and the second detection model for the same training data set, and corresponding object pairs are determined.
In the region of partial overlap of the first field of view of the first lidar unit a and the second field of view of the second lidar units B1, B2 and B3, the detection results of the first detection model and the second detection model should be identical for the same target. Thus, such a corresponding target pair may be determined in step S501.
In step S502, a consistency function of the first target detection result and the second target detection result is determined for the corresponding target pair. In embodiments of the present disclosure, the consistency of the first target detection result and the second target detection result may be characterized as corresponding to the same target using, for example, L1, L2, KL divergence, JS divergence function as a consistency function.
In step S503, the first detection model and the second detection model are adjusted based on the consistency function. In an embodiment of the disclosure, the first detection model and the second detection model are adjusted such that the first target detection result and the second target detection result, which are characterized by the consistency function, tend to be completely consistent in an ideal state, thereby obtaining a more accurate first detection model and second detection model.
In the training method of the object detection apparatus according to the embodiment of the present disclosure described with reference to fig. 5, by utilizing the overlapping regions of the front view and the around view, different detection models should have the same category, size, and distance attribute for the same object, and adjusting the front view and the around view detection models by utilizing the consistency function training, higher detection performance is achieved.
Above, a training method of an object detection apparatus according to an embodiment of the present disclosure is described with reference to the accompanying drawings. Hereinafter, an object detection method according to an embodiment of the present disclosure will be further described.
Fig. 6 is a flowchart illustrating a target detection method according to an embodiment of the present disclosure. As shown in fig. 6, the object detection method according to the embodiment of the present disclosure includes the following steps.
In step S601, operation state information of a plurality of lidar units is detected. In an embodiment of the present disclosure, the operational status information indicates, for example, whether the first lidar unit and the second lidar unit are operating properly.
In step S602, running state information of the vehicle and running environment information of the vehicle are acquired. In an embodiment of the present disclosure, the operating state information indicates, for example, an operating speed of the vehicle. The operating environment information indicates, for example, a geographic location in which the vehicle is located, and further indicates, for example, a current operating environment of the vehicle by loading scenario information associated with the current address location. For example, in an operating scenario such as a mine work area, the operating environment information indicates whether the vehicle is currently in a traffic area where little additional obstruction is present or in a queuing area, a waiting area, and a loading area where other vehicles or work equipment such as an excavator may be present.
In step S603, the plurality of lidar units are controlled to perform target detection based on one or more of the operating state information, and the operating environment information.
For example, in an embodiment of the present disclosure, when the operation state information indicates that one of the first and second lidar units is in a failure state, the other lidar unit in the operation state is controlled to continue to operate, and it may be determined whether the operation speed of the vehicle needs to be reduced accordingly based on the operation speed indicated by the operation state information accordingly.
For another example, in the embodiment of the present disclosure, when the running environment information indicates that the vehicle is in a traffic area where little additional obstacle occurs, one or more of the first lidar unit and the second lidar unit may be controlled to be in a standby state without frequently performing the target detection process. When the operation environment information indicates that the vehicle is in a queuing area, a waiting area, and a loading area where other vehicles or working equipment such as an excavator may exist, the first lidar unit and the second lidar unit may be controlled to simultaneously perform target detection, thereby ensuring the working safety of the working area.
In the embodiment of the present disclosure, the object detection apparatus performing the object detection method described above with reference to fig. 6 is trained by using the training method described above with reference to fig. 2 to 5.
Fig. 7 is a functional block diagram illustrating a training device of the object detection device according to an embodiment of the present disclosure. As shown in fig. 7, a training apparatus 700 of the target detection apparatus according to the embodiment of the present disclosure includes a training data set generating unit 701, a first detection model training unit 702, a second detection model training unit 703, and a target matching training unit 704. The training apparatus 700 may perform the respective steps of the training method of the object detection apparatus according to the embodiment of the present disclosure as described above with reference to fig. 2 to 5. Those skilled in the art will readily understand that: these unit modules may be implemented in various manners by hardware alone, by software alone, or by a combination thereof, and the present disclosure is not limited to any one of them. Training apparatus 700 according to embodiments of the present disclosure may be configured at a remote dispatch command system and/or at a transportation device end.
Specifically, the training data set generating unit 701 is configured to generate a third training data set for the second lidar unit based on the first training data set for the first lidar unit and the second training data set for the second lidar unit.
More specifically, the training data set generating unit 701 is configured to determine target objects in which the first training data set and the second training data set match each other; taking target data corresponding to the target object in the second training data set as training input, taking target data corresponding to the target object in the first training data set as training truth value, and executing training to obtain a training data set generation model; and generating a third training data set using the training data generation model based on the second training data set.
First detection model training unit 702 is configured to perform training on the initial detection model using the first training data set and the randomly sampled third training data set to generate a first detection model for the first lidar unit.
The second detection model training unit 703 is configured to perform training on the first detection model using the third training data set and the randomly sampled first training data set, generating a second detection model for the second lidar unit.
More specifically, the second detection model training unit 703 is further configured to perform training on the second detection model using the second training data set, generating an optimized second detection model for the second lidar unit.
The first detection model training unit 702 is further configured to perform training on the optimized second detection model using the first training data set, generating an optimized first detection model for the first lidar unit.
Furthermore, the target matching training unit 704 is configured to perform target matching on the first detection model and the second detection model for the first target detection result and the second target detection result of the same training data set, and determine a corresponding target pair; determining a consistency function of the first target detection result and the second target detection result aiming at the corresponding target pair; and adjusting the first detection model and the second detection model based on the consistency function.
According to the training device of the target detection device, the training data set generation model for training target densification processing is trained to generate the third training data set for densification of the second laser radar unit, and the first detection model for the first laser radar unit and the second detection model for the second laser radar unit are cross-initialized in the training stage to execute iterative training, so that the detection precision of the detection model obtained by training is improved. Further, the training device of the object detection device according to the embodiment of the present disclosure acquires an object detection model that achieves higher detection performance in an actual object detection scene by further performing training and adjustment using training data sets conforming to respective actual object detection scenes. Further, the training device of the object detection device according to the embodiment of the present disclosure achieves higher detection performance by training and adjusting the forward-looking and backward-looking detection models with a consistency function by utilizing the overlapping regions of the forward-looking and backward-looking, different detection models should have the same category, size, and distance attribute for the same object.
Fig. 8 is a hardware block diagram illustrating an electronic device 800 according to an embodiment of the disclosure. An electronic device according to an embodiment of the present disclosure includes at least a processor; and a memory for storing computer readable instructions. When loaded and executed by a processor, the processor performs the training method or the target detection method for the target detection apparatus as described above.
The electronic device 800 shown in fig. 8 specifically includes: a Central Processing Unit (CPU) 801, a Graphics Processing Unit (GPU) 802, and a main memory 803. These units are interconnected by a bus 804. A Central Processing Unit (CPU) 801 and/or a Graphics Processing Unit (GPU) 802 may be used as the above-described processor, and a main memory 803 may be used as the above-described memory storing computer readable instructions. Furthermore, the electronic device 800 may also comprise a communication unit 805, a storage unit 806, an output unit 807, an input unit 808 and an external device 809, which units are also connected to the bus 804.
Fig. 9 is a schematic diagram illustrating a computer-readable storage medium according to an embodiment of the present disclosure. As shown in fig. 9, a computer-readable storage medium 900 according to an embodiment of the present disclosure has computer-readable instructions 901 stored thereon. When the computer readable instructions 901 are executed by a processor, a training method or an object detection method for an object detection apparatus according to an embodiment of the present disclosure described with reference to the above drawings is performed. The computer-readable storage medium includes, but is not limited to, for example, volatile memory and/or nonvolatile memory. The volatile memory may include, for example, random Access Memory (RAM) and/or cache memory (cache), and the like. The non-volatile memory may include, for example, read Only Memory (ROM), hard disk, flash memory, optical disk, magnetic disk, and the like.
In the above, the training method and apparatus, the target detection method, the electronic device, and the storage medium of the target detection apparatus according to the embodiments of the present disclosure are described with reference to the accompanying drawings, by taking the front view dense training target as the training truth value of the around view sparse training target in the front view and around view overlapping area, the point cloud dense model is trained, and then all around view training target point clouds are dense by using the dense model, so that the problem of accuracy drop caused by the around view training target point clouds sparse is solved; the front training targets and the around training targets are respectively and alternately randomly placed into the around and front scenes after being sampled, so that the quantity and the richness of the training targets are enhanced, a detection model for around is initialized by using a detection model trained by a front training data set, after the detection model for around is trained, the weight is used for the front detection model, and finally, the front model is finely adjusted, so that iterative training is realized; finally, the front and around detection models are trained and adjusted by using the consistency function by using the same targets in the front and around overlapping areas, which should have the same category, size and distance attributes. In an unmanned mine car for example, with the object detection device including the forward-looking and through-looking detection models thus obtained, accurate object detection at both the forward-looking long distance and the through-looking short distance can be achieved, and the robustness of the object detection device is ensured.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.
The basic principles of the present disclosure have been described above in connection with specific embodiments, however, it should be noted that the advantages, benefits, effects, etc. mentioned in the present disclosure are merely examples and not limiting, and these advantages, benefits, effects, etc. are not to be considered as necessarily possessed by the various embodiments of the present disclosure. Furthermore, the specific details disclosed herein are for purposes of illustration and understanding only, and are not intended to be limiting, since the disclosure is not necessarily limited to practice with the specific details described.
The block diagrams of the devices, apparatuses, devices, systems referred to in this disclosure are merely illustrative examples and are not intended to require or imply that the connections, arrangements, configurations must be made in the manner shown in the block diagrams. As will be appreciated by one of skill in the art, the devices, apparatuses, devices, systems may be connected, arranged, configured in any manner. Words such as "including," "comprising," "having," and the like are words of openness and mean "including but not limited to," and are used interchangeably therewith. The terms "or" and "as used herein refer to and are used interchangeably with the term" and/or "unless the context clearly indicates otherwise. The term "such as" as used herein refers to, and is used interchangeably with, the phrase "such as, but not limited to.
In addition, as used herein, the use of "or" in the recitation of items beginning with "at least one" indicates a separate recitation, such that recitation of "at least one of A, B or C" for example means a or B or C, or AB or AC or BC, or ABC (i.e., a and B and C). Furthermore, the term "exemplary" does not mean that the described example is preferred or better than other examples.
It is also noted that in the systems and methods of the present disclosure, components or steps may be disassembled and/or assembled. Such decomposition and/or recombination should be considered equivalent to the present disclosure.
Various changes, substitutions, and alterations are possible to the techniques described herein without departing from the teachings of the techniques defined by the appended claims. Furthermore, the scope of the claims of the present disclosure is not limited to the particular aspects of the process, machine, manufacture, composition of matter, means, methods and acts described above. The processes, machines, manufacture, compositions of matter, means, methods, or acts, presently existing or later to be developed that perform substantially the same function or achieve substantially the same result as the corresponding aspects described herein may be utilized. Accordingly, the appended claims are intended to include within their scope such processes, machines, manufacture, compositions of matter, means, methods, or acts.
The previous description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
The foregoing description has been presented for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of the disclosure to the form disclosed herein. Although a number of example aspects and embodiments have been discussed above, a person of ordinary skill in the art will recognize certain variations, modifications, alterations, additions, and subcombinations thereof.
Claims (12)
1. A training method for an object detection device including at least a plurality of lidar units having different harnesses, comprising:
generating a third training data set for a second lidar unit based on the first training data set for the first lidar unit and the second training data set for the second lidar unit;
Performing training on an initial detection model using the first training data set and the randomly sampled third training data set to generate a first detection model for the first lidar unit;
performing training on the first detection model using the third training data set and the randomly sampled first training data set, generating a second detection model for the second lidar unit,
wherein the first beam of the first lidar unit is higher than the second beam of the second lidar unit, and the first field of view of the first lidar unit is smaller than the second field of view of the second lidar unit.
2. The training method of claim 1, wherein the generating a third training data set for a second lidar unit based on the first training data set for the first lidar unit and the second training data set for the second lidar unit comprises:
determining target objects matched with each other in the first training data set and the second training data set;
taking the target data corresponding to the target object in the second training data set as training input, taking the target data corresponding to the target object in the first training data set as training truth value, and executing training to obtain a training data set generation model; and
Generating a model using the training data based on the second training data set, generating the third training data set.
3. The training method of claim 1 or 2, further comprising:
performing training on the second detection model using the second training dataset, generating a second detection model for optimization of the second lidar unit; and
training the optimized second detection model using the first training dataset, generating an optimized first detection model for the first lidar unit.
4. The training method of claim 1 or 2, further comprising:
performing target matching on the first target detection result and the second target detection result of the first detection model and the second detection model aiming at the same training data set, and determining corresponding target pairs;
determining a consistency function of the first target detection result and the second target detection result aiming at the corresponding target pair; and
the first detection model and the second detection model are adjusted based on the consistency function.
5. An object detection method for a vehicle including at least an object detection device configured by a plurality of lidar units having different wire harnesses, comprising:
Detecting working state information of the laser radar units;
acquiring running state information of the vehicle and running environment information of the vehicle; and
controlling the plurality of lidar units to perform target detection based on one or more of the operating state information, and the operating environment information,
wherein the plurality of lidar units includes a first lidar unit and a second lidar unit, a first beam of the first lidar unit is higher than a second beam of the second lidar unit, and a first field of view of the first lidar unit is less than a second field of view of the second lidar unit.
6. The target detection method of claim 5, further comprising:
training the object detection device using the training method according to claim 1 or 2.
7. A training device for an object detection device comprising at least a plurality of lidar units having different harnesses, comprising:
a training data set generation unit configured to generate a third training data set for a second lidar unit based on the first training data set for the first lidar unit and the second training data set for the second lidar unit;
A first detection model training unit configured to perform training on an initial detection model using the first training data set and the third training data set randomly sampled, generating a first detection model for the first lidar unit; and
a second detection model training unit configured to perform training on the first detection model using the third training data set and the randomly sampled first training data set, generate a second detection model for the second lidar unit,
wherein the first beam of the first lidar unit is higher than the second beam of the second lidar unit, and the first field of view of the first lidar unit is smaller than the second field of view of the second lidar unit.
8. The training apparatus of claim 7, wherein the training data set generation unit is configured to:
determining target objects matched with each other in the first training data set and the second training data set;
taking the target data corresponding to the target object in the second training data set as training input, taking the target data corresponding to the target object in the first training data set as training truth value, and executing training to obtain a training data set generation model; and
Generating a model using the training data based on the second training data set, generating the third training data set.
9. The training device of claim 7 or 8,
the second detection model training unit is further configured to perform training on the second detection model using the second training dataset, generating a second detection model for optimization of the second lidar unit; and
the first detection model training unit is further configured to perform training on the optimized second detection model using the first training dataset, generating an optimized first detection model for the first lidar unit.
10. The training apparatus of claim 7 or 8, further comprising a target matching training unit configured to:
performing target matching on the first target detection result and the second target detection result of the first detection model and the second detection model aiming at the same training data set, and determining corresponding target pairs;
determining a consistency function of the first target detection result and the second target detection result aiming at the corresponding target pair; and
The first detection model and the second detection model are adjusted based on the consistency function.
11. An electronic device, comprising:
a memory for storing computer readable instructions; and
a processor for executing the computer readable instructions to cause the electronic device to perform the training method of any one of claims 1-4 or the target detection method of claim 5 or 6.
12. A non-transitory computer readable storage medium storing computer readable instructions which, when executed by a processor, cause the processor to perform the training method of any of claims 1-4 or the target detection method of claim 5 or 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310287318.1A CN115994589B (en) | 2023-03-23 | 2023-03-23 | Training method and device, target detection method, electronic device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310287318.1A CN115994589B (en) | 2023-03-23 | 2023-03-23 | Training method and device, target detection method, electronic device and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115994589A true CN115994589A (en) | 2023-04-21 |
| CN115994589B CN115994589B (en) | 2023-05-23 |
Family
ID=85992438
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310287318.1A Active CN115994589B (en) | 2023-03-23 | 2023-03-23 | Training method and device, target detection method, electronic device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115994589B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180348346A1 (en) * | 2017-05-31 | 2018-12-06 | Uber Technologies, Inc. | Hybrid-View Lidar-Based Object Detection |
| US20210255300A1 (en) * | 2020-01-31 | 2021-08-19 | Metawave Corporation | Gan-based data synthesis for semi-supervised learning of a radar sensor |
| WO2022088720A1 (en) * | 2020-10-30 | 2022-05-05 | 上海商汤临港智能科技有限公司 | Sample generation method and apparatus, neural network training method and apparatus, and data processing method and apparatus |
| US20220198806A1 (en) * | 2020-12-21 | 2022-06-23 | Beihang University | Target detection method based on fusion of prior positioning of millimeter-wave radar and visual feature |
| CN115761392A (en) * | 2022-10-14 | 2023-03-07 | 际络科技(上海)有限公司 | Laser radar target detection model training method, detection method and device |
-
2023
- 2023-03-23 CN CN202310287318.1A patent/CN115994589B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20180348346A1 (en) * | 2017-05-31 | 2018-12-06 | Uber Technologies, Inc. | Hybrid-View Lidar-Based Object Detection |
| US20210255300A1 (en) * | 2020-01-31 | 2021-08-19 | Metawave Corporation | Gan-based data synthesis for semi-supervised learning of a radar sensor |
| WO2022088720A1 (en) * | 2020-10-30 | 2022-05-05 | 上海商汤临港智能科技有限公司 | Sample generation method and apparatus, neural network training method and apparatus, and data processing method and apparatus |
| US20220198806A1 (en) * | 2020-12-21 | 2022-06-23 | Beihang University | Target detection method based on fusion of prior positioning of millimeter-wave radar and visual feature |
| CN115761392A (en) * | 2022-10-14 | 2023-03-07 | 际络科技(上海)有限公司 | Laser radar target detection model training method, detection method and device |
Non-Patent Citations (1)
| Title |
|---|
| 陆峰;徐友春;李永乐;王德宇;谢德胜;: "基于信息融合的智能车障碍物检测方法", 计算机应用 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115994589B (en) | 2023-05-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3397554B1 (en) | System and method for utilization of multiple-camera network to capture static and/or motion scenes | |
| US10754354B2 (en) | Hover control | |
| WO2019015158A1 (en) | Obstacle avoidance method for unmanned aerial vehicle, and unmanned aerial vehicle | |
| US10929986B2 (en) | Techniques for using a simple neural network model and standard camera for image detection in autonomous driving | |
| CN111291650B (en) | Automatic parking assisting method and device | |
| US20220246040A1 (en) | Control method and device for unmanned aerial vehicle, and computer readable storage medium | |
| US20170066135A1 (en) | Enhanced system and method for control of robotic devices | |
| WO2020258217A1 (en) | State estimation method and system for movable platform, movable platform, and storage medium | |
| WO2021016854A1 (en) | Calibration method and device, movable platform, and storage medium | |
| CN109073390B (en) | Positioning method and device, electronic equipment and readable storage medium | |
| US20200082181A1 (en) | Method and system for sensing an obstacle, and storage medium | |
| US9812020B2 (en) | Electronic device and unmanned aerial vehicle control method | |
| CN109509236A (en) | Vehicle bounding box generation method, device and storage medium in unmanned scene | |
| CN111427355A (en) | Obstacle data processing method, device, equipment and storage medium | |
| Er et al. | Development control and navigation of Octocopter | |
| US20210110173A1 (en) | System and method for tracking objects using multi-edge bounding box factors | |
| CN115994589B (en) | Training method and device, target detection method, electronic device and storage medium | |
| CN109073398B (en) | Map establishing method, positioning method, device, terminal and storage medium | |
| CN111712687A (en) | Aerial survey method, aircraft and storage medium | |
| CN113031641A (en) | Unmanned aerial vehicle control method and device, storage medium and unmanned aerial vehicle | |
| CN117055618A (en) | Unmanned aerial vehicle and control method thereof | |
| CN113705272A (en) | Method, device, equipment and storage medium for detecting travelable area | |
| US20220277480A1 (en) | Position estimation device, vehicle, position estimation method and position estimation program | |
| KR102071720B1 (en) | Method for matching radar target list and target of vision image | |
| US11587331B2 (en) | Lane keeping for autonomous vehicles |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| REG | Reference to a national code |
Ref country code: HK Ref legal event code: DE Ref document number: 40090419 Country of ref document: HK |