CN116383760A - Target type fusion method and device, electronic equipment and storage medium - Google Patents
Target type fusion method and device, electronic equipment and storage medium Download PDFInfo
- Publication number
- CN116383760A CN116383760A CN202310324994.1A CN202310324994A CN116383760A CN 116383760 A CN116383760 A CN 116383760A CN 202310324994 A CN202310324994 A CN 202310324994A CN 116383760 A CN116383760 A CN 116383760A
- Authority
- CN
- China
- Prior art keywords
- target
- sequence
- type
- laser radar
- hidden markov
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000007500 overflow downdraw method Methods 0.000 title abstract description 15
- 230000004927 fusion Effects 0.000 claims abstract description 67
- 238000000034 method Methods 0.000 claims abstract description 67
- 238000012545 processing Methods 0.000 claims description 13
- 238000004364 calculation method Methods 0.000 claims description 8
- 230000008569 process Effects 0.000 abstract description 16
- 230000008447 perception Effects 0.000 abstract description 10
- 238000010586 diagram Methods 0.000 description 12
- 238000004590 computer program Methods 0.000 description 11
- 238000001514 detection method Methods 0.000 description 9
- 238000004422 calculation algorithm Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 5
- 230000000007 visual effect Effects 0.000 description 5
- 230000002159 abnormal effect Effects 0.000 description 4
- 230000006399 behavior Effects 0.000 description 4
- 238000009434 installation Methods 0.000 description 4
- 239000000969 carrier Substances 0.000 description 3
- 238000013135 deep learning Methods 0.000 description 3
- 230000007547 defect Effects 0.000 description 3
- 238000005259 measurement Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000007613 environmental effect Effects 0.000 description 2
- 238000011156 evaluation Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 238000013179 statistical model Methods 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 239000002184 metal Substances 0.000 description 1
- 238000011022 operating instruction Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000011144 upstream manufacturing Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N7/00—Computing arrangements based on specific mathematical models
- G06N7/01—Probabilistic graphical models, e.g. probabilistic networks
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Abstract
The application discloses a target type fusion method, a target type fusion device, electronic equipment and a storage medium, wherein the method comprises the steps of receiving a laser radar target sequence and a camera target sequence; according to the laser radar target sequence and the camera target sequence, a target matching pair is obtained through association; and calculating and updating the target type fusion result in the target matching pair according to the hidden Markov model. The method and the device realize the fusion of the target types, simultaneously reduce the complexity of the fusion perception process, and improve the stability and the instantaneity of the automatic driving system.
Description
Technical Field
The present disclosure relates to the field of autopilot technologies, and in particular, to a method and apparatus for fusing object types, an electronic device, and a storage medium.
Background
Environmental perception is one of key technologies for realizing an automatic driving function of an automobile, and a single sensor has great limitation in acquiring environmental information, identifying a target and the like due to different properties of each sensor.
In the related art, in the process of sensing fusion of multiple sensors, (1) information such as the position, the type and the speed of a target can be effectively detected according to point cloud data of a laser radar, but the recognition effect of small obstacles such as pedestrians, awls (traffic roadblocks) and the like is unstable. (2) The category information of the target can be effectively identified according to the image information for the vision sensor, but the distance detection and the speed estimation of the target have larger errors.
Therefore, for the automatic driving scene with various traffic participants, the accurate identification of the type attribute information of the target can reduce a lot of false detections, provide more reliable information for the decision control of the automatic driving vehicle, and can adopt different avoidance schemes according to different types of the target.
Disclosure of Invention
The embodiment of the application provides a target type fusion method and device, electronic equipment and storage medium, so as to reduce fusion complexity and meet the requirement of real-time performance.
The embodiment of the application adopts the following technical scheme:
in a first aspect, an embodiment of the present application provides a target type fusion method, where the method includes:
receiving a laser radar target sequence and a camera target sequence;
according to the laser radar target sequence and the camera target sequence, a target matching pair is obtained through association;
and calculating and updating the target type fusion result in the target matching pair according to the hidden Markov model.
In some embodiments, the classification of the types of the road traffic participants in advance at least comprises five types of target types of vehicles, pedestrians, riders, cones and unknowns, the calculation and updating of the target type fusion result in the target matching pair according to the hidden markov model comprises the following steps:
calculating the probability value of the target type by using a hidden Markov model according to the target matching pair;
and updating the attribute of the target type according to the probability value of the target type.
In some embodiments, the computing the probability value for the target type using a hidden markov model comprises:
establishing a state sequence I= { I according to probability values of the vehicle, the pedestrian, the rider, the cone and the unknown object 1 ，i 2 ，...，i T }；
Setting a model lambda= (A, B, pi), and observing a sequence Q= { Q 1 ，q 2 ，...，q T And calculates the probability of P (i|q, λ) at maximum I.
In some embodiments, the calculating the probability value of the target type using the hidden markov model further includes:
setting probability values when the states at different moments are i, wherein t represents the moment, and i represents the state;
taking the probability value of the initial type of the target as an initial vector: delta 1 (i)＝π i b i q 1 ，i＝1，2，...，N；
The states of T at different instants t=2 according to the initial vector recursion:
and calculates the vector delta when T time is maximum T (i)：
According to the vector delta when T time is maximum T (i) Backtracking the optimal path t=t-1, T-2, 1, resulting in an optimal pathA kind of electronic device
As a probability value for the target type.
In some embodiments, the receiving a lidar target sequence and a camera target sequence comprises:
and receiving a laser radar target sequence and a camera target sequence for processing target level fusion.
In some embodiments, the correlating the target matching pair according to the lidar target sequence and the camera target sequence includes:
converting the coordinates of the mounting positions of the laser radar and the vision sensor on the vehicle to the same vehicle body coordinate system, and searching targets of the laser radar and the vision sensor when the time stamps are similar or identical to obtain target position information of the same frame;
calculating similarity according to the mahalanobis distance based on the target position information of the same frame, and finding out associated target matching pairs;
and updating the associated target matching pair after re-association according to the target tracking ID, the target transverse distance, the target speed similarity and the dynamic threshold of the space distance.
In some embodiments, after calculating and updating the target type fusion result in the target matching pair according to the hidden markov model, the method further includes:
and according to the target category determined by the target type fusion result, adopting a corresponding avoidance strategy.
In a second aspect, embodiments of the present application further provide a target type fusion device, where the device includes:
the receiving module is used for receiving the laser radar target sequence and the camera target sequence;
the association matching module is used for obtaining an object matching pair according to the laser radar object sequence and the camera object sequence in an association mode;
and the calculation updating module is used for calculating and updating the target type fusion result in the target matching pair according to the hidden Markov model.
In a third aspect, embodiments of the present application further provide an electronic device, including: a processor; and a memory arranged to store computer executable instructions that, when executed, cause the processor to perform the above method.
In a fourth aspect, embodiments of the present application also provide a computer-readable storage medium storing one or more programs that, when executed by an electronic device comprising a plurality of application programs, cause the electronic device to perform the above-described method.
The above-mentioned at least one technical scheme that this application embodiment adopted can reach following beneficial effect: receiving a laser radar target sequence and a camera target sequence; and then, according to the laser radar target sequence and the camera target sequence, obtaining a target matching pair through association, and finally, calculating and updating a target type fusion result in the target matching pair by using a hidden Markov model. In the process of multi-sensor fusion, a hidden Markov model is adopted to calculate the type of the target and update the type of the target is realized. Meanwhile, more accurate type information can be obtained through multi-sensor fusion calculation of type probability, and false detection conditions are reduced.
Drawings
The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute an undue limitation to the application. In the drawings:
FIG. 1 is a schematic flow chart of a target type fusion method in an embodiment of the present application;
FIG. 2 is a schematic structural diagram of a target type fusion device according to an embodiment of the present application;
FIG. 3 is a schematic diagram of the implementation principle of the object type fusion method in the embodiment of the present application;
fig. 4 is a schematic structural diagram of an electronic device in an embodiment of the present application.
Detailed Description
For the purposes, technical solutions and advantages of the present application, the technical solutions of the present application will be clearly and completely described below with reference to specific embodiments of the present application and corresponding drawings. It will be apparent that the described embodiments are only some, but not all, of the embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.
The following describes in detail the technical solutions provided by the embodiments of the present application with reference to the accompanying drawings.
The embodiment of the application provides a target type fusion method, as shown in fig. 1, and provides a flow chart of the target type fusion method in the embodiment of the application, where the method at least includes the following steps S110 to S130:
step S110, a laser radar target sequence and a camera target sequence are received.
A plurality of laser radars and visual sensors are deployed on the vehicle and serve as hardware carriers for multi-sensor fusion sensing. Point cloud data of obstacles in the environment can be received by the laser radar, and the obstacles in the environment existing in the image can be identified by the vision sensor. The data generated by the vision sensor is a 2D image, and the perception accuracy of the shape and the category of the object is high. The success of deep learning techniques stems from computer vision tasks, and many successful algorithms are also based on the processing of image data, so image-based perception techniques are now relatively mature. The laser radar overcomes the defects of the camera to a certain extent, can accurately sense the distance of an object, and meanwhile, the 3D point cloud generated by the laser radar is sparse (for example, the scanning line is only 32 lines, 64 lines or 128 lines). For long-distance objects or small objects, the number of reflection points is relatively small, and the method is unfavorable for obtaining complete obstacle point cloud data.
In the specific implementation, the specific number of the laser radars or the specific positions of the vision sensors are not specifically limited, and according to the actual needs, a person skilled in the art can convert the laser radars and the vision sensors to the same vehicle body coordinate system according to the installation positions of the laser radars and the vision sensors on the vehicle, and find targets with similar time stamps of the two sensors, so that the targets can be conveniently compared. I.e. to align the time stamps.
And step S120, obtaining a target matching pair by association according to the laser radar target sequence and the camera target sequence.
And finding out a matching pair according to the laser radar target sequence and the camera target sequence.
For example, the similarity may be calculated according to the mahalanobis distance based on the target position information of the two sensors in the same frame, so as to find the associated matching pair.
Preferably, according to the difference of the properties of the two sensors, the longitudinal distance of the remote visual target detection is quite different from that of the laser radar, if the distance similarity calculation is simply relied on at the moment, the false association or the missed association is caused, so that the association matching pair is updated according to the tracking ID, the transverse distance, the speed similarity and the dynamic threshold of the space distance, the result of the association matching pair is more accurate, and finally the association is carried out to obtain the target matching pair.
And step S130, calculating and updating the target type fusion result in the target matching pair according to the hidden Markov model.
According to the target matching pair obtained by association in the steps, the type probability value can be calculated by using the hidden Markov model, and meanwhile, the target type attribute is updated according to the type probability result, so that the final target fusion result is more accurate.
Hidden markov models, (Hidden Markov Model, HMM) are statistical models that are used to describe a markov process with hidden unknown parameters. The difficulty is to determine the implicit parameters of the process from the observable parameters and then use these parameters for further analysis. For example, a Hidden Markov Model (HMM) based evaluation of the behavior of a driving vehicle may be employed to detect abnormal situations. The observed state of the Markov model is provided by a correlation tracking algorithm, including speed and vehicle position. The framework derives these observations based on the point cloud and odometry data. Therefore, dangerous and abnormal driving behaviors in the simulated multi-lane expressway scene can be reliably inferred by adopting the hidden Markov model, wherein the hidden Markov model comprises vehicles except two self-vehicles.
In the implementation, when calculating according to the hidden markov model, the target type of the fusion sensing result of the sensor needs to be specifically divided in advance, such as a vehicle, a pedestrian, a rider, a cone barrel and the like, and then a state sequence is established according to the probability value of the target. And then, giving a model and an observation sequence, and obtaining a probability maximum sequence according to the probability of the initial type of the target. And finding out the optimal target type according to the maximum sequence, and updating the perception fusion result.
The method is different from the method of combining the millimeter wave radar with the vision sensor fusion in the related technology, the millimeter wave radar can stably detect the speed information of the moving target, has low cost, is complementary with the vision sensor, and is easy to realize by utilizing the target level fusion method. According to the hidden Markov model, the type attribute of the traffic participant is calculated, and compared with other types judging methods in the fusion scheme, the algorithm is high in accuracy and small in code complexity, and the real-time requirement of a vehicle-mounted automatic driving system is met.
Therefore, when the method is adopted for fusing the types of the targets, the type information of the targets cannot be accurately detected by the millimeter wave radar, the targets are sensitive to metal, and the recognition effect on the targets of pedestrians is poor. Therefore, the problem that the millimeter wave radar easily detects one target into a plurality of targets, and error correlation is easy to occur, so that fusion result errors are caused is avoided. Because the hidden Markov model can calculate probability values for five types of traffic participants including, but not limited to, vehicles, pedestrians, riders, cones, unknowns, the accuracy of classification can be improved under subdivision of target types.
The method is different from a method based on stereoscopic vision and a laser radar in the related art, and utilizes a vision sensor and the laser radar to perform joint calibration, establishes a matching relationship between image information and point cloud information, and detects a target based on the image and the point cloud. The method improves the accuracy of target type identification by a target level fusion method based on target detection of the laser radar and the millimeter wave radar, and improves the type judgment of small obstacles such as pedestrians, cones and the like.
Therefore, when the method is adopted to carry out target type fusion, the problems that the combined calibration of the stereoscopic vision and the laser radar depends on the calibration precision, the difficulty of data matching association is high, the follow-up target detection algorithm is complex and the instantaneity is poor are avoided.
In one embodiment of the present application, the classification of the types of the road traffic participants in advance includes at least five types of target types of vehicles, pedestrians, riders, cones and unknowns, the calculating and updating the fusion result of the target types in the target matching pair according to the hidden markov model includes: calculating the probability value of the target type by using a hidden Markov model according to the target matching pair; and updating the attribute of the target type according to the probability value of the target type.
According to the main types of road traffic participants, besides vehicles, pedestrians and riders, the target types of the cone barrel and the unknown object are subdivided, according to the target matching pair, the probability value of the target type is calculated by using a hidden Markov model, and then the target type attribute is updated according to the probability value of the target type. Because the probability value of the target type can change in the fusion sensing process, the attribute of the target type also needs to be updated, and the real-time requirement of an automatic driving system is met.
In one embodiment of the present application, the calculating the probability value of the target type using the hidden markov model includes: establishing a state sequence I= { I according to probability values of the vehicle, the pedestrian, the rider, the cone and the unknown object 1 ，i 2 ，...，i T -a }; setting a model lambda= (A, B, pi), and observing a sequence Q= { Q 1 ，q 2 ，...，q T And calculates the probability of P (i|q, λ) at maximum I.
In specific implementation, in order to calculate the probability value of the target type, a state sequence i= { I is first established according to the probability values of the vehicle, the pedestrian, the rider, the cone and the unknown object 1 ，i 2 ，...，i T "where i represents a state。
The setting model λ= (a, B, pi) is set when the actual modeling needs are made.
Observation sequence q= { Q 1 ，q 2 ，...，q T As is well known in the art, can be set according to the actual scenario.
And calculating the probability of P (I|Q, lambda) at the maximum time I according to the model, the observation sequence and the state sequence.
In one embodiment of the present application, the calculating the probability value of the object type using the hidden markov model further includes: setting probability values when the states at different moments are i, wherein t represents the moment, and i represents the state; taking the probability value of the initial type of the target as an initial vector: delta 1 (i)＝π i b i q 1 I=1, 2,; the states of T at different instants t=2 according to the initial vector recursion:
and calculates the vector delta when T time is maximum T (i)：/>
According to the vector delta when T time is maximum T (i) Backtracking the optimal path t=t-1, T-2,..1, resulting in an optimal +.>
As a probability value for the target type.
First, an initial vector is obtained by taking a target initial type probability value as an initial vector, namely a probability value representing a state i at different moments: delta 1 (i)＝π i b i q 1 ，i＝1，2，...，N。
Then, the states at different times are recursively deduced, and then the maximum target type probability value at the time T is calculated.
Finally, according to the steps, the probability maximum sequence can be obtained according to the probability of the target initial type, so that the optimal target type is found out to update the fusion result.
The method utilizes the hidden Markov model to calculate the type probabilities of the two sensor targets, reduces the complexity of the system, and improves the stability and the instantaneity of the system.
In one embodiment of the present application, the receiving a lidar target sequence and a camera target sequence includes: and receiving a laser radar target sequence and a camera target sequence for processing target level fusion.
Specifically, the input terminal includes a laser radar target sequence and a vision sensor target sequence, the input target information includes information of time stamp, ID, position, type probability value, speed, acceleration, length, width, height, etc., and the information is provided by the sensor terminal (laser radar and vision sensor) to process the target level fusion.
And updating the target type by a multi-sensor fusion method. And moreover, for the cone or pedestrian target single sensor, the single sensor may be erroneously detected to be of an UNKNOWN type (UNKNOWN), and more accurate type information can be obtained through multi-sensor fusion calculation type probability, so that the false detection is reduced. Meanwhile, the fusion system is low in complexity and good in instantaneity, and the vehicle-mounted requirement of automatic driving is met.
In one embodiment of the present application, the obtaining the target matching pair according to the association between the laser radar target sequence and the camera target sequence includes: converting the coordinates of the mounting positions of the laser radar and the vision sensor on the vehicle to the same vehicle body coordinate system, and searching targets of the laser radar and the vision sensor when the time stamps are similar or identical to obtain target position information of the same frame; calculating similarity according to the mahalanobis distance based on the target position information of the same frame, and finding out associated target matching pairs; and updating the associated target matching pair after re-association according to the target tracking ID, the target transverse distance, the target speed similarity and the dynamic threshold of the space distance.
In the implementation, firstly, the coordinates of the installation positions of the laser radar and the vision sensor on the vehicle are converted into the same vehicle body coordinate system, targets of the laser radar and the vision sensor when the time stamps are similar or identical are searched to obtain target position information of the same frame, and then, the mahalanobis distance between the two targets or other ways for determining similarity are calculated to find out related target matching pairs.
Further, in order to improve accuracy of the matching pair, the associated target matching pair may be updated after re-association according to dynamic thresholds of the target tracking ID, the target lateral distance, the target speed similarity, and the spatial distance.
The target tracking ID is a unique ID of the target tracking process for distinguishing between different targets.
The target lateral distance is related to the target heading angle.
The target speed similarity is the relative speed between the vehicle and the obstacle.
Dynamic thresholds for spatial distance include, but are not limited to, drivable zones, lane lines, speed limit signs, and other road constraints.
It should be noted that any one or more of the above-mentioned dynamic thresholds of the target tracking ID, target lateral distance, target speed similarity, and spatial distance may be used to re-associate and update the associated target matching pair.
In one embodiment of the present application, after calculating and updating the target type fusion result in the target matching pair according to the hidden markov model, the method further includes: and according to the target category determined by the target type fusion result, adopting a corresponding avoidance strategy.
And adopting a corresponding avoidance strategy in the automatic driving decision module according to the target category determined by the target type fusion result. It can be understood that more accurate type information can be obtained by calculating the update target type probability through multi-sensor fusion, and false detection in the fusion sensing process is reduced. And the accuracy in the automatic driving decision module is improved.
The embodiment of the present application further provides a target type fusion device 200, as shown in fig. 2, and provides a schematic structural diagram of the target type fusion device in the embodiment of the present application, where the target type fusion device 200 at least includes: a receiving module 210, an association matching module 220, and a computing update module 230, wherein:
in one embodiment of the present application, the receiving module 210 is specifically configured to: a lidar target sequence and a camera target sequence are received.
A plurality of laser radars and visual sensors are deployed on the vehicle and serve as hardware carriers for multi-sensor fusion sensing. Point cloud data of obstacles in the environment can be received by the laser radar, and the obstacles in the environment existing in the image can be identified by the vision sensor. The data generated by the vision sensor is a 2D image, and the perception accuracy of the shape and the category of the object is high. The success of deep learning techniques stems from computer vision tasks, and many successful algorithms are also based on the processing of image data, so image-based perception techniques are now relatively mature. The laser radar overcomes the defects of the camera to a certain extent, can accurately sense the distance of an object, and meanwhile, the 3D point cloud generated by the laser radar is sparse (for example, the scanning line is only 32 lines, 64 lines or 128 lines). For long-distance objects or small objects, the number of reflection points is relatively small, and the method is unfavorable for obtaining complete obstacle point cloud data.
In the specific implementation, the specific number of the laser radars or the specific positions of the vision sensors are not specifically limited, and according to the actual needs, a person skilled in the art can convert the laser radars and the vision sensors to the same vehicle body coordinate system according to the installation positions of the laser radars and the vision sensors on the vehicle, and find targets with similar time stamps of the two sensors, so that the targets can be conveniently compared. I.e. to align the time stamps.
In one embodiment of the present application, the association matching module 220 is specifically configured to: and obtaining a target matching pair by correlation according to the laser radar target sequence and the camera target sequence.
And finding out a matching pair according to the laser radar target sequence and the camera target sequence.
For example, the similarity may be calculated according to the mahalanobis distance based on the target position information of the two sensors in the same frame, so as to find the associated matching pair.
Preferably, according to the difference of the properties of the two sensors, the longitudinal distance of the remote visual target detection is quite different from that of the laser radar, if the distance similarity calculation is simply relied on at the moment, the false association or the missed association is caused, so that the association matching pair is updated according to the tracking ID, the transverse distance, the speed similarity and the dynamic threshold of the space distance, the result of the association matching pair is more accurate, and finally the association is carried out to obtain the target matching pair.
In one embodiment of the present application, the computing update module 230 is specifically configured to: and calculating and updating the target type fusion result in the target matching pair according to the hidden Markov model.
According to the target matching pair obtained by association in the steps, the type probability value can be calculated by using the hidden Markov model, and meanwhile, the target type attribute is updated according to the type probability result, so that the final target fusion result is more accurate.
Hidden markov models, (Hidden Markov Model, HMM) are statistical models that are used to describe a markov process with hidden unknown parameters. The difficulty is to determine the implicit parameters of the process from the observable parameters and then use these parameters for further analysis. For example, a Hidden Markov Model (HMM) based evaluation of the behavior of a driving vehicle may be employed to detect abnormal situations. The observed state of the Markov model is provided by a correlation tracking algorithm, including speed and vehicle position. The framework derives these observations based on the point cloud and odometry data. Therefore, dangerous and abnormal driving behaviors in the simulated multi-lane expressway scene can be reliably inferred by adopting the hidden Markov model, wherein the hidden Markov model comprises vehicles except two self-vehicles.
In the implementation, when calculating according to the hidden markov model, the target type of the fusion sensing result of the sensor needs to be specifically divided in advance, such as a vehicle, a pedestrian, a rider, a cone barrel and the like, and then a state sequence is established according to the probability value of the target. And then, giving a model and an observation sequence, and obtaining a probability maximum sequence according to the probability of the initial type of the target. And finding out the optimal target type according to the maximum sequence, and updating the perception fusion result.
It can be understood that the above-mentioned object type fusion device can implement each step of the object type fusion method provided in the foregoing embodiment, and the relevant explanation about the object type fusion method is applicable to the object type fusion device, which is not described herein.
The above-described object type fusion device 200 may be used in a fusion awareness module of an autopilot system and as an upstream module of a decision module.
Fig. 3 is a schematic implementation diagram of a target type fusion method in the embodiment of the present application, where the implementation principle includes the following implementation flows:
step S310, a laser radar target sequence is acquired.
Step S320, a camera target sequence is acquired.
Step S330, alignment in space time.
A plurality of laser radars and visual sensors are deployed on the vehicle and serve as hardware carriers for multi-sensor fusion sensing. Point cloud data of obstacles in the environment can be received by the laser radar, and the obstacles in the environment existing in the image can be identified by the vision sensor. The data generated by the vision sensor is a 2D image, and the perception accuracy of the shape and the category of the object is high. The success of deep learning techniques stems from computer vision tasks, and many successful algorithms are also based on the processing of image data, so image-based perception techniques are now relatively mature. The laser radar overcomes the defects of the camera to a certain extent, can accurately sense the distance of an object, and meanwhile, the 3D point cloud generated by the laser radar is sparse (for example, the scanning line is only 32 lines, 64 lines or 128 lines). For long-distance objects or small objects, the number of reflection points is relatively small, and the method is unfavorable for obtaining complete obstacle point cloud data.
In the specific implementation, the specific number of the laser radars or the specific positions of the vision sensors are not specifically limited, and according to the actual needs, a person skilled in the art can convert the laser radars and the vision sensors to the same vehicle body coordinate system according to the installation positions of the laser radars and the vision sensors on the vehicle, and find targets with similar time stamps of the two sensors, so that the targets can be conveniently compared. I.e. to align the time stamps.
Step S340, creating or deleting the track.
Step S350, matching the association, and for unassociated tracks and measurements, proceeding to step S360. Step S370 is entered for the associated track and measurement.
Step S360, re-association, and step S370 is performed for the matched pair in re-association. The process returns to step S340 for unassociated tracks and measurements.
Converting the coordinates of the mounting positions of the laser radar and the vision sensor on the vehicle to the same vehicle body coordinate system, and searching targets of the laser radar and the vision sensor when the time stamps are similar or identical to obtain target position information of the same frame; calculating similarity according to the mahalanobis distance based on the target position information of the same frame, and finding out associated target matching pairs; and updating the associated target matching pair after re-association according to the target tracking ID, the target transverse distance, the target speed similarity and the dynamic threshold of the space distance.
In step S370, type fusion calculation is performed.
Calculating the probability value for the object type using the hidden Markov model includes:
establishing a state sequence I= { I according to probability values of the vehicle, the pedestrian, the rider, the cone and the unknown object 1 ，i 2 ，...，i T }；
Setting a model lambda= (A, B, pi), and observing a sequence Q= { Q 1 ，q 2 ，...，q T And calculates the probability of P (i|q, λ) at maximum I.
The calculating the probability value of the target type by using the hidden Markov model further comprises:
setting probability values when the states at different moments are i, wherein t represents the moment, and i represents the state;
taking the probability value of the initial type of the target as an initial vector: delta 1 (i)＝π i b i q 1 ，i＝1，2，...，N；
The states of T at different instants t=2 according to the initial vector recursion:
and calculates the vector delta when T time is maximum T (i)：
According to the vector delta when T time is maximum T (i) Backtracking the optimal path t=t-1, T-2, 1, resulting in an optimal
As a probability value for the target type.
In step S380, the target type is updated.
And updating the target type after calculating the probability value of the target type according to the probability value.
Fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present application. Referring to fig. 4, at the hardware level, the electronic device includes a processor, and optionally an internal bus, a network interface, and a memory. The Memory may include a Memory, such as a Random-Access Memory (RAM), and may further include a non-volatile Memory (non-volatile Memory), such as at least 1 disk Memory. Of course, the electronic device may also include hardware required for other services.
The processor, network interface, and memory may be interconnected by an internal bus, which may be an ISA (Industry Standard Architecture ) bus, a PCI (Peripheral Component Interconnect, peripheral component interconnect standard) bus, or EISA (Extended Industry Standard Architecture ) bus, among others. The buses may be classified as address buses, data buses, control buses, etc. For ease of illustration, only one bi-directional arrow is shown in FIG. 4, but not only one bus or type of bus.
And the memory is used for storing programs. In particular, the program may include program code including computer-operating instructions. The memory may include memory and non-volatile storage and provide instructions and data to the processor.
The processor reads the corresponding computer program from the nonvolatile memory into the memory and then runs the computer program to form the target type fusion device on a logic level. The processor is used for executing the programs stored in the memory and is specifically used for executing the following operations:
receiving a laser radar target sequence and a camera target sequence;
according to the laser radar target sequence and the camera target sequence, a target matching pair is obtained through association;
and calculating and updating the target type fusion result in the target matching pair according to the hidden Markov model.
The method performed by the object type fusion device disclosed in the embodiment shown in fig. 1 of the present application may be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware in a processor or by instructions in the form of software. The processor may be a general-purpose processor, including a central processing unit (Central Processing Unit, CPU), a network processor (Network Processor, NP), etc.; but also digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The disclosed methods, steps, and logic blocks in the embodiments of the present application may be implemented or performed. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of a method disclosed in connection with the embodiments of the present application may be embodied directly in hardware, in a decoded processor, or in a combination of hardware and software modules in a decoded processor. The software modules may be located in a random access memory, flash memory, read only memory, programmable read only memory, or electrically erasable programmable memory, registers, etc. as well known in the art. The storage medium is located in a memory, and the processor reads the information in the memory and, in combination with its hardware, performs the steps of the above method.
The electronic device may further execute the method executed by the target type fusion device in fig. 1, and implement the function of the target type fusion device in the embodiment shown in fig. 1, which is not described herein.
The embodiments of the present application also provide a computer readable storage medium storing one or more programs, where the one or more programs include instructions, which when executed by an electronic device including a plurality of application programs, enable the electronic device to perform a method performed by the object type fusion apparatus in the embodiment shown in fig. 1, and specifically are configured to perform:
receiving a laser radar target sequence and a camera target sequence;
according to the laser radar target sequence and the camera target sequence, a target matching pair is obtained through association;
and calculating and updating the target type fusion result in the target matching pair according to the hidden Markov model.
It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.
Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The foregoing is merely exemplary of the present application and is not intended to limit the present application. Various modifications and changes may be made to the present application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. which are within the spirit and principles of the present application are intended to be included within the scope of the claims of the present application.
Claims (10)
1. A method of object type fusion, wherein the method comprises:
receiving a laser radar target sequence and a camera target sequence;
according to the laser radar target sequence and the camera target sequence, a target matching pair is obtained through association;
and calculating and updating the target type fusion result in the target matching pair according to the hidden Markov model.
2. The method of claim 1, wherein the pre-classifying the types of road traffic participants into at least five categories of target types including vehicles, pedestrians, riders, cones, unknowns, the computing and updating the target type fusion result in the target match pair according to a hidden markov model, comprising:
calculating the probability value of the target type by using a hidden Markov model according to the target matching pair;
and updating the attribute of the target type according to the probability value of the target type.
3. The method of claim 2, wherein the calculating the probability value for the object type using a hidden markov model comprises:
establishing a state sequence I= { I according to probability values of the vehicle, the pedestrian, the rider, the cone and the unknown object 1 ，i 2 ，...，i T }；
Setting a model lambda= (A, B, pi), and observing a sequence Q= { Q 1 ，q 2 ，...，q T And calculates the probability of P (i|q, λ) at maximum I.
4. The method of claim 3, wherein the calculating the probability value of the object type using a hidden markov model further comprises:
setting probability values when the states at different moments are i, wherein t represents the moment, and i represents the state;
taking the probability value of the initial type of the target as an initial vector: delta 1 (i)＝π i b i q 1 ，i＝1，2，...，N；
Recursively estimating the state of T at different moments t=2 according to the initial vector：
And calculates the vector delta when T time is maximum T (i)：/>
According to the vector delta when T time is maximum T (i) Backtracking the optimal path t=t-1, T-2, 1, resulting in an optimal
As a probability value for the target type.
5. The method of claim 1, wherein the receiving a lidar target sequence and a camera target sequence comprises:
and receiving a laser radar target sequence and a camera target sequence for processing target level fusion.
6. The method of claim 1, wherein the correlating the lidar target sequence and the camera target sequence to obtain a target matching pair comprises:
converting the coordinates of the mounting positions of the laser radar and the vision sensor on the vehicle to the same vehicle body coordinate system, and searching targets of the laser radar and the vision sensor when the time stamps are similar or identical to obtain target position information of the same frame;
calculating similarity according to the mahalanobis distance based on the target position information of the same frame, and finding out associated target matching pairs;
and updating the associated target matching pair after re-association according to the target tracking ID, the target transverse distance, the target speed similarity and the dynamic threshold of the space distance.
7. The method of any one of claims 1 to 6, wherein after the computing and updating the target type fusion result in the target match pair according to the hidden markov model, further comprising:
and according to the target category determined by the target type fusion result, adopting a corresponding avoidance strategy.
8. A target type fusion device, wherein the device comprises:
the receiving module is used for receiving the laser radar target sequence and the camera target sequence;
the association matching module is used for obtaining an object matching pair according to the laser radar object sequence and the camera object sequence in an association mode;
and the calculation updating module is used for calculating and updating the target type fusion result in the target matching pair according to the hidden Markov model.
9. An electronic device, comprising:
a processor; and
a memory arranged to store computer executable instructions which, when executed, cause the processor to perform the method of any of claims 1 to 7.
10. A computer readable storage medium storing one or more programs, which when executed by an electronic device comprising a plurality of application programs, cause the electronic device to perform the method of any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310324994.1A CN116383760A (en) | 2023-03-29 | 2023-03-29 | Target type fusion method and device, electronic equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310324994.1A CN116383760A (en) | 2023-03-29 | 2023-03-29 | Target type fusion method and device, electronic equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116383760A true CN116383760A (en) | 2023-07-04 |
Family
ID=86960953
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310324994.1A Pending CN116383760A (en) | 2023-03-29 | 2023-03-29 | Target type fusion method and device, electronic equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116383760A (en) |
-
2023
- 2023-03-29 CN CN202310324994.1A patent/CN116383760A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Holder et al. | Real-time pose graph SLAM based on radar | |
| CN109800658B (en) | Parking space type online identification and positioning system and method based on neural network | |
| JP6747269B2 (en) | Object recognition device | |
| US11719555B2 (en) | Map information system | |
| CN108573611B (en) | Speed limit sign fusion method and speed limit sign fusion system | |
| US11442456B2 (en) | Apparatus for determining lane change path of autonomous vehicle and method thereof | |
| EP3819663A1 (en) | Method for determining a position of a vehicle | |
| CN114495064A (en) | Monocular depth estimation-based vehicle surrounding obstacle early warning method | |
| CN112793567A (en) | Driving assistance method and system based on road condition detection | |
| US20220215197A1 (en) | Data processing method and apparatus, chip system, and medium | |
| US11487293B2 (en) | Map-information obstacle-tracking system and method | |
| US20230094836A1 (en) | Method for Detecting Moving Objects in the Surroundings of a Vehicle, and Motor Vehicle | |
| CN111723724B (en) | Road surface obstacle recognition method and related device | |
| CN111612818A (en) | Novel binocular vision multi-target tracking method and system | |
| JP6555132B2 (en) | Moving object detection device | |
| CN112801024A (en) | Detection information processing method and device | |
| CN110411499B (en) | Evaluation method and evaluation system for detection and identification capability of sensor | |
| CN116383760A (en) | Target type fusion method and device, electronic equipment and storage medium | |
| US11915489B2 (en) | Method, device, computer program and computer program product for operating a vehicle | |
| Guo et al. | Understanding surrounding vehicles in urban traffic scenarios based on a low-cost lane graph | |
| US11983918B2 (en) | Platform for perception system development for automated driving system | |
| EP4047514B1 (en) | Platform for perception system development for automated driving system | |
| Hsu et al. | Object detection and recognition by using sensor fusion | |
| US20230143958A1 (en) | System for neural architecture search for monocular depth estimation and method of using | |
| US20240078814A1 (en) | Method and apparatus for modeling object, storage medium, and vehicle control method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |