CN117333846A - Detection method and system based on sensor fusion and incremental learning in severe weather - Google Patents
Detection method and system based on sensor fusion and incremental learning in severe weather Download PDFInfo
- Publication number
- CN117333846A CN117333846A CN202311474818.2A CN202311474818A CN117333846A CN 117333846 A CN117333846 A CN 117333846A CN 202311474818 A CN202311474818 A CN 202311474818A CN 117333846 A CN117333846 A CN 117333846A
- Authority
- CN
- China
- Prior art keywords
- sensor
- data
- vision sensor
- fusion
- coordinate system
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 64
- 230000004927 fusion Effects 0.000 title claims abstract description 62
- 238000012549 training Methods 0.000 claims abstract description 22
- 238000012545 processing Methods 0.000 claims abstract description 17
- 239000011159 matrix material Substances 0.000 claims description 51
- 238000000034 method Methods 0.000 claims description 24
- 238000013519 translation Methods 0.000 claims description 16
- 230000000007 visual effect Effects 0.000 claims description 16
- 238000004364 calculation method Methods 0.000 claims description 15
- 238000010586 diagram Methods 0.000 claims description 14
- 230000015654 memory Effects 0.000 claims description 13
- 238000010606 normalization Methods 0.000 claims description 12
- 230000006870 function Effects 0.000 claims description 10
- 230000009466 transformation Effects 0.000 claims description 10
- 230000008569 process Effects 0.000 claims description 9
- 238000004422 calculation algorithm Methods 0.000 claims description 8
- 238000004590 computer program Methods 0.000 claims description 7
- 238000000605 extraction Methods 0.000 claims description 4
- 230000010354 integration Effects 0.000 claims description 4
- 230000017105 transposition Effects 0.000 claims description 4
- 238000006243 chemical reaction Methods 0.000 claims description 3
- 238000009826 distribution Methods 0.000 claims description 3
- 238000007637 random forest analysis Methods 0.000 claims description 3
- 238000012937 correction Methods 0.000 claims description 2
- 238000003384 imaging method Methods 0.000 claims description 2
- 238000005259 measurement Methods 0.000 claims description 2
- 230000003287 optical effect Effects 0.000 claims description 2
- 230000005855 radiation Effects 0.000 claims description 2
- 230000011218 segmentation Effects 0.000 claims description 2
- 230000001360 synchronised effect Effects 0.000 claims description 2
- 238000005516 engineering process Methods 0.000 description 6
- 238000011161 development Methods 0.000 description 4
- 230000018109 developmental process Effects 0.000 description 4
- 230000019771 cognition Effects 0.000 description 3
- 238000013135 deep learning Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 238000005728 strengthening Methods 0.000 description 1
- 239000002023 wood Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/07—Target detection
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- Artificial Intelligence (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Radar Systems Or Details Thereof (AREA)
Abstract
The invention discloses a detection method and a detection system based on sensor fusion and incremental learning in severe weather, comprising the following steps: the radar and the vision sensor respectively acquire data of the driving environment to obtain driving environment information; processing the obtained two kinds of sensor data according to a space-time alignment rule; extracting bottom layer characteristics of the vision sensor data and performing data level fusion with the radar sensor data; and selectively storing the fused data according to different weather conditions, and training according to the rule of incremental learning to obtain a target detection result. The invention effectively improves the target detection precision and robustness of the vehicle in severe weather, and provides a reliable basis for the next-stage task, thereby achieving the purpose of improving the driving safety.
Description
Technical Field
The invention relates to the technical field of intelligent driving, in particular to a detection method and a detection system based on sensor fusion and incremental learning in severe weather, and especially relates to a target detection method and a detection system based on sensor fusion and incremental learning technology in severe weather.
Background
The intelligent driving has great significance for the future development of the automobile industry, represents the development level of the intelligent mobile industry in China, and specifically, the intelligent driving is realized effectively and safely by comprehensively analyzing the data obtained by the sensors and combining with a high-precision map, making decisions and estimating the state of the vehicle independently under different driving scenes, and purposefully completing tracking control and cooperative control so as to reduce the probability of various road accidents caused by human factors and improve the industrial development level in China.
With the development of artificial intelligence technology, vehicle target detection becomes one of the important points of research. The deep learning has a great effect, and the sensor data is processed in an off-line or on-line mode, so that the intelligent vehicle can accurately identify objects in the driving environment, and a foundation is provided for tasks such as subsequent obstacle avoidance and path planning. However, when the prior art is applied to target detection in severe weather, accuracy and robustness are reduced, safe driving cannot be realized, and the method becomes one of key problems in intelligent driving technology research.
The patent document with the publication number of CN110008843B discloses a vehicle target joint cognition method and system based on point cloud and image data, which comprises a data level joint module, a deep learning target detection wood block and a joint cognition module, wherein the data level joint module acquires three-dimensional point cloud data and image data and is used for fusing the point cloud data and the image data, the fused data is summarized in the deep learning target detection module to carry out feature level detection and identification, a detection result is output, and the joint cognition module adopts an evidence theory method to judge the feature level fusion detection result and the data level fusion detection result to obtain a reliability distribution as output. However, this patent document does not combine the sensor with bad weather, and cannot improve accuracy and robustness.
Disclosure of Invention
Aiming at the defects in the prior art, the invention aims to provide a detection method and a detection system based on sensor fusion and incremental learning in severe weather.
The invention provides a detection method based on sensor fusion and incremental learning in severe weather, which comprises the following steps:
the acquisition step: data acquisition is carried out on the driving environment through a radar sensor and a vision sensor respectively, so that radar sensor data and vision sensor data are obtained;
the processing steps are as follows: processing the radar sensor and vision sensor data according to a space-time alignment rule to obtain time-synchronous radar sensor and vision sensor data, space-aligned binocular vision sensor data and space-aligned radar sensor and vision sensor data;
and (3) a fusion step: extracting bottom layer characteristics of the vision sensor data and carrying out data level fusion on the bottom layer characteristics and the radar sensor data to obtain a data information stream;
the detection step comprises: and selectively storing the data fused in the fusion step according to different weather conditions, and training according to the rule of incremental learning to obtain a target detection result.
Preferably, the space-time alignment rule in the processing step includes:
at t of the radar sensor d Taking the time stamp t of the visual sensor with the time threshold value delta t before and after the time stamp c The selection rule is that
|t c -t d |≤Δt
To obtain two sensor data that are time synchronized;
the coordinate of the corner point on the visual sensor calibration plate under the world coordinate system is p w After vision sensor correction of rigid transformation is carried out on vision sensor images with initial optical centers not parallel to each other, the same corner point p w The imaging points of the (a) fall at the same height of the left and right vision sensor images to realize the space alignment of the vision sensors, the vision sensor images are transformed by the rigid body to convert the vision sensor coordinate system into the radar sensor coordinate system, and the space alignment of the radar sensor and the vision sensor is realized.
Preferably, the data-level fusion in the fusing step includes the steps of:
the integration step: reading an image of a visual sensor, converting image data into a data tensor with 3 channels, connecting the data of the left and right visual sensors through a filter, and integrating the data of the left and right visual sensors;
the extraction step: the data integrated in the integration step is subjected to convolution network and normalization processing, the bottom layer characteristics of the vision sensor image are extracted, and detailed data information is reserved;
normalization: the radar data tensor is normalized, and original characteristics of the radar data tensor are reserved;
and (3) data fusion: and (3) performing filter connection with the image tensor of the vision sensor subjected to feature extraction, and fusing the original data features of the radar and the vision sensor which belong to different distributions to obtain normalized data information flows under different days.
Preferably, the incremental learning training in the detecting step includes the steps of:
the segmentation step: weather data belonging to different domains is continuously segmented; wherein, the data stream batches near the dividing points of different weather data streams comprise data on the left side and the right side;
an input step: the image data which are continuously segmented are sent into a model for training in the form of tensors, and M tensors are fixedly stored in the training process;
training: the similarity of the two tensors is measured through style measurement, and if the similarity is high, the old tensor with the lowest similarity in the storage is replaced by the new tensor with high similarity; if the similarity is low, performing pseudo-domain detection on tensors with the styles being too different from the existing domain styles by using a random forest algorithm and a sparse random projection algorithm, wherein tensors which do not pass through detection are additionally stored, and added into M memories when a certain number is accumulated, and meanwhile, the same number is removed from the M memories, so that the balance of the number of the memories and the number of the data in different domains is maintained.
Preferably, the spatial alignment process includes:
correcting the binocular vision sensor, and estimating a rotation matrix R and a translation matrix T of the right vision sensor relative to the left vision sensor according to the rotation and translation matrices of the left and right vision sensors;
the rotation matrix is calculated by:
wherein R is l 、R r Is the rotation matrix of the left and right side view sensor relative to the world coordinate system, the subscripts l and r respectively represent the left and right sides, the superscript T represents the matrix transposition, and the requirements are satisfiedThe superscript-1 denotes the inverse of the matrix;
the translation matrix is calculated by:
T＝T r -RT l
wherein T is l 、T r Is a translation matrix of the left and right view sensors relative to the world coordinate system;
corner point p of left calibration plate w The coordinates (U, V) are transformed from the world coordinate system to the left view sensor coordinate system through rigid transformation to obtain the coordinates (X) of the point under the left view sensor coordinate system l ,Y l ,Z l )；
Coordinates (X) l ,Y l ,Z l ) The calculation of (2) is obtained by:
wherein R is l 、T l The rotation matrix and the translation matrix of the left view sensor coordinate system relative to the world coordinate system are respectively;
coordinates (X) in the left vision sensor coordinate system l ,Y l ,Z l ) Conversion of rigid body transformation, perspective projection and radiation transformation to images obtained from right-side view sensorIn the pixel coordinate system, the pixel coordinates (u 'of the point are obtained' r ,v′ r ) I.e. corner p w Projection coordinates of the coordinates in a pixel coordinate system;
the projection coordinates are calculated by the following equation:
wherein K is r Is an internal reference matrix of the right-side view sensor;
taking n calibration plate corner points on M images, optimizing an objective function, and optimizing all matrix parameters by using an L-M algorithm to minimize projection errors of the corner points;
the calculation of the objective function is derived from the following equation:
in the formula (u) r ,v r ) The actual position of the corner in the pixel coordinate system of the image obtained by the right view sensor is shown;
for left-side vision sensor, in spatial alignment of radar sensor and vision sensor, according to its rotation matrix R relative to radar sensor lra And a translation matrix T lra The conversion of a coordinate system is realized, and points in the left view sensor coordinate system are converted into a radar sensor coordinate system;
the coordinate system transformation is obtained by:
wherein (X) l ,Y l ,Z l ) Is the coordinates in the left vision sensor coordinate system, (X) ra ,Y ra ,Z ra ) Is the coordinates in the corresponding radar sensor coordinate system.
Preferably, the step of fusing data includes:
two data tensors f of left and right vision sensor l 、f r Obtaining a single Zhang Shijiao sensor characteristic diagram through filter connection; channel fusion is carried out on the vision sensor characteristic diagram through 1X 1 convolution, different channel information is integrated, and the number of channels is changed through 3X 3 convolution, so that the vision sensor characteristic diagram f is obtained c ；
The calculation of the visual sensor profile is made by:
in the method, in the process of the invention,indicating filter connection, W 1 、W 2 Respectively represent convolution of the first layer network and the second layer network, BN 1 、BN 2 The Batch normalization respectively represents the first layer network and the second layer network;
radar data tensor f ra Normalized and normalized vision sensor characteristic diagram f c Realizing data level fusion through filter connection to obtain a feature diagram f;
the calculation of the feature map is derived from the following equation:
wherein F is N Representing a normalization function, and scaling the feature map elements to 0-1;
the calculation of the normalization function is derived from the following equation:
where f=f-min (f), f is a two-dimensional tensor.
Preferably, the training step includes:
the three-dimensional feature map f is flattened to obtain a two-dimensional feature map x, the channel number is kept unchanged, and a gram matrix G is obtained from the two-dimensional feature map x;
the calculation of the gram matrix is given by:
G＝x T ×x
wherein x represents matrix multiplication and T represents matrix transposition;
the gram matrix G is used to measure the difference between the two feature maps, denoted by d;
the calculation of the variability is obtained by:
where, l represents a layer-one network,the i, j-th elements representing two gram matrices G, A, G, A are the calculated gram matrices for two different feature maps, respectively, f=h×w representing the gram matrix size, where h and w are the length and width of the feature maps, respectively.
The invention also provides a detection system based on sensor fusion and incremental learning in severe weather, which comprises:
and the acquisition module is used for: data acquisition is carried out on the driving environment through a radar sensor and a vision sensor respectively, so that radar sensor data and vision sensor data are obtained;
the processing module is used for: processing the radar sensor and vision sensor data according to a space-time alignment rule to obtain time-synchronous radar sensor and vision sensor data, space-aligned binocular vision sensor data and space-aligned radar sensor and vision sensor data;
and a fusion module: extracting bottom layer characteristics of the vision sensor data and carrying out data level fusion on the bottom layer characteristics and the radar sensor data to obtain a data information stream;
and a detection module: and selectively storing the data fused in the fusion module according to different weather conditions, and training according to the rule of incremental learning to obtain a target detection result.
The invention also provides a computer readable storage medium storing a computer program which when executed by a processor realizes the steps of the detection method based on sensor fusion and incremental learning in severe weather.
The invention also provides an electronic device, which comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, wherein the computer program is executed by the processor to realize the steps of the detection method based on sensor fusion and incremental learning in severe weather.
Compared with the prior art, the invention has the following beneficial effects:
1) The invention provides a more perfect sensor fusion and increment learning technology, namely, multi-sensor data are processed through space-time alignment rules, fusion is carried out according to data level fusion rules, then storage training is carried out on the data through increment learning rules, and finally, a target detection result in severe weather is obtained through analysis, so that the target detection precision in severe weather is improved, and the robustness is high;
2) The invention provides the target detection method with high accuracy and robustness, which reduces the probability of missed detection or error of target detection when the vehicle runs in bad weather, avoids safety accidents caused by the error of target detection, and improves the driving safety and road passing efficiency of the intelligent vehicle.
Drawings
Other features, objects and advantages of the present invention will become more apparent upon reading of the detailed description of non-limiting embodiments, given with reference to the accompanying drawings in which:
FIG. 1 is a flow chart of a detection method based on sensor fusion and incremental learning in severe weather.
Detailed Description
The present invention will be described in detail with reference to specific examples. The following examples will assist those skilled in the art in further understanding the present invention, but are not intended to limit the invention in any way. It should be noted that variations and modifications could be made by those skilled in the art without departing from the inventive concept. These are all within the scope of the present invention.
As shown in fig. 1, the embodiment discloses a detection method based on sensor fusion and incremental learning in severe weather, which comprises the following steps:
the radar and the vision sensor respectively acquire data of the driving environment to obtain driving environment information;
processing the obtained two kinds of sensor data according to a space-time alignment rule;
extracting bottom layer characteristics of the vision sensor data and performing data level fusion with the radar sensor data;
and selectively storing the fused data according to different weather conditions, and training according to the rule of incremental learning to obtain a target detection result.
The in-vehicle sensor of the host vehicle includes a binocular vision sensor and a radar sensor.
The detection process of the host vehicle is as follows:
under severe weather, the host vehicle captures driving environment information through the vehicle-mounted sensor, analyzes the environment information through the sensor fusion and incremental learning technology, identifies a target to be detected, outputs a detection result, and then analyzes and identifies the next target to be detected in the driving environment through the sensor fusion and incremental learning technology, and outputs the detection result. The host vehicle continuously repeats the steps, and target detection under severe weather is achieved.
Specifically, the radar and the vision sensor respectively acquire data of driving environment to obtain driving environment information, and the method comprises the following steps:
in severe weather, the radar sensor of the host vehicle is at t d High-resolution distance-azimuth images are acquired at moment, and a binocular vision sensor is used for measuring the distance and azimuth of the images at t c Color images are collected at the moment to form information data of the running environment of the host vehicle, a radar sensor image, a left vision sensor image and a right vision sensor image are obtained respectively, and a data set is formed.
Specifically, the processing the two kinds of sensor data according to the space-time alignment rule includes:
and (3) performing time synchronization on data obtained by the radar and the binocular vision sensors according to the time stamp interval, selecting sensor data matched one by one, correcting by using the vision sensors, and realizing the spatial alignment of the same object.
It should be noted that, the sampling frequencies of the radar and the binocular vision sensor are different, the obtained original driving environment data are not matched, and the rule of selecting the time matched different sensor data is as follows:
|t c -t d |≤Δt
where Δt is a time threshold, and the visual sensor map exceeding the time threshold is discarded based on the radar sensor time stamp, thereby obtaining sensor data of one-to-one matching.
It should be further noted that, in the spatial calibration alignment rule, n calibration plate corner points on m images are taken, (u) r ,v r ) For the actual position of the corner in the pixel coordinate system of the right-side view sensor, (u) ′ r ,v r ′ ) For its estimate, the projection error is calculated as follows:
estimated value of corner projection (u ′ r ,v r ′ ) Reference matrix K of right vision sensor r Rotation matrix R of right side vision sensor relative to left side vision sensor, translation matrix T, rotation matrix R of left side vision sensor coordinate system relative to world coordinate system l Translation matrix T l And (5) determining. The rotation matrix R and the translation matrix T are calculated by the rotation matrix R of the left and right view sensor relative to the world coordinate system l 、R r Translation matrix T l 、T r The method can obtain:
in order to solve the matrix parameters, the projection errors are optimized by using an L-M (Levenberg-Marquardt) algorithm, so that the projection errors of the corner points are minimized.
Specifically, the extracting the bottom layer features of the vision sensor data and performing data level fusion with the radar sensor data includes:
the data sets are sorted by weather and delivered to a network that continuously segments the data sets by a fixed lot size such that the lots near the dividing line contain data of different weather on both the left and right sides.
Vision sensor image x via bilinear interpolation s ＝(x d +0.5)×w x -0.5, into an image of the same size as the radar sensor map. w (w) x Is a scale factor, is determined by the size of an original vision sensor image and the size of a radar sensor image, and is x d Is the original image pixel point.
Two data tensors f of left and right vision sensor l 、f r Obtaining a single Zhang Shijiao sensor characteristic diagram through filter connectionTo integrate the features of the two visual sensor maps, convolution and Batch normalization are applied to f c . First by 1X 1 convolution W 1 Channel fusion is carried out, different channel information is integrated, and one layer of Batch normalized BN is adopted 1 Obtaining f c ：
f c ＝BN 1 [W 1 (f c )]
Then is convolved with 3X 3W 2 And one layer of Batch normalized BN 2 Changing the number of channels, and finally normalizing
Scaling the feature map element to 0-1 to obtainPre-fusion visual sensor profile f c ：
f c ＝F N (BN 2 [W 2 (f c )])
In the process, the size of the characteristic diagram of the visual sensor is kept unchanged, and only the number of channels is changed so as to keep the original information to the maximum extent.
Radar data tensor f ra Through normalization of f ra ＝F N (f ra ) And f is as above c Obtaining a characteristic diagram after data level fusion through filter connection
Specifically, according to different weather conditions, the fused data is selectively stored, training is performed according to an increment learning rule, and a target detection result is obtained, including:
feature map of each batchFrom three dimensions Zhang Liangbian to two dimensions tensor->Calculating the gram matrix g=x T X. Assuming a layer I network, the gram matrixes of two different feature graphs are G respectively l ＝{g l } ij And A l ＝{a l } ij The dissimilarity of the feature map styles can be obtained by measuring their dissimilarity. The calculation of the variability is obtained by:
the data input to the network will be stored until the storage capacity is reached, with a fixed storage size M. If the storage capacity is full, the feature map which is closest to the new feature map style in the storage is replaced by calculating the difference of the feature map styles. The new style is found by random forest and sparse random projection algorithm, so that the memory contains characteristic diagrams of different weather.
The data of each round of training comprises new data and history data randomly selected from the storage, and the memory is continuously updated in the training process and participates in the next round of training so as to realize continuous strengthening of history memory while learning new knowledge.
The invention also provides a detection system based on sensor fusion and incremental learning in severe weather, which can be realized by executing the flow steps of the detection method based on sensor fusion and incremental learning in severe weather, namely, a person skilled in the art can understand the detection method based on sensor fusion and incremental learning in severe weather as a preferred implementation mode of the detection system based on sensor fusion and incremental learning in severe weather.
A detection system based on sensor fusion and incremental learning in severe weather, comprising: and the acquisition module is used for: the radar sensor and the vision sensor respectively acquire data of driving environment to obtain driving environment information; the processing module is used for: processing the two kinds of sensor data obtained in the acquisition module according to a space-time alignment rule; and a fusion module: extracting bottom layer characteristics of the vision sensor data and performing data level fusion with the radar sensor data; and a detection module: and selectively storing the fused data according to different weather conditions, and training according to the rule of incremental learning to obtain a target detection result.
Those skilled in the art will appreciate that the invention provides a system and its individual devices, modules, units, etc. that can be implemented entirely by logic programming of method steps, in addition to being implemented as pure computer readable program code, in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc. Therefore, the system and various devices, modules and units thereof provided by the invention can be regarded as a hardware component, and the devices, modules and units for realizing various functions included in the system can also be regarded as structures in the hardware component; means, modules, and units for implementing the various functions may also be considered as either software modules for implementing the methods or structures within hardware components.
In the description of the present application, it should be understood that the terms "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," and the like indicate orientations or positional relationships based on the orientations or positional relationships illustrated in the drawings, merely to facilitate description of the present application and simplify the description, and do not indicate or imply that the devices or elements being referred to must have a specific orientation, be configured and operated in a specific orientation, and are not to be construed as limiting the present application.
The foregoing describes specific embodiments of the present invention. It is to be understood that the invention is not limited to the particular embodiments described above, and that various changes or modifications may be made by those skilled in the art within the scope of the appended claims without affecting the spirit of the invention. The embodiments of the present application and features in the embodiments may be combined with each other arbitrarily without conflict.
Claims (10)
1. The detection method based on sensor fusion and incremental learning in severe weather is characterized by comprising the following steps of:
the acquisition step: data acquisition is carried out on the driving environment through a radar sensor and a vision sensor respectively, so that radar sensor data and vision sensor data are obtained;
the processing steps are as follows: processing the radar sensor and vision sensor data according to a space-time alignment rule to obtain time-synchronous radar sensor and vision sensor data, space-aligned binocular vision sensor data and space-aligned radar sensor and vision sensor data;
and (3) a fusion step: extracting bottom layer characteristics of the vision sensor data and carrying out data level fusion on the bottom layer characteristics and the radar sensor data to obtain a data information stream;
the detection step comprises: and selectively storing the data fused in the fusion step according to different weather conditions, and training according to the rule of incremental learning to obtain a target detection result.
2. The method of sensor fusion and incremental learning based detection in severe weather of claim 1, wherein the spatiotemporal alignment rules in the processing step include:
at t of the radar sensor d Taking the time stamp t of the visual sensor with the time threshold value delta t before and after the time stamp c Its selection rule is |t c -t d |≤Δt
To obtain two sensor data that are time synchronized;
the coordinate of the corner point on the visual sensor calibration plate under the world coordinate system is p w After vision sensor correction of rigid transformation is carried out on vision sensor images with initial optical centers not parallel to each other, the same corner point p w The imaging points of the (a) fall at the same height of the left and right vision sensor images to realize the space alignment of the vision sensors, the vision sensor images are transformed by the rigid body to convert the vision sensor coordinate system into the radar sensor coordinate system, and the space alignment of the radar sensor and the vision sensor is realized.
3. The method for detecting bad weather based on sensor fusion and incremental learning according to claim 1, wherein the data level fusion in the fusion step comprises the steps of:
the integration step: reading an image of a visual sensor, converting image data into a data tensor with 3 channels, connecting the data of the left and right visual sensors through a filter, and integrating the data of the left and right visual sensors;
the extraction step: the data integrated in the integration step is subjected to convolution network and normalization processing, the bottom layer characteristics of the vision sensor image are extracted, and detailed data information is reserved;
normalization: the radar data tensor is normalized, and original characteristics of the radar data tensor are reserved;
and (3) data fusion: and (3) performing filter connection with the image tensor of the vision sensor subjected to feature extraction, and fusing the original data features of the radar and the vision sensor which belong to different distributions to obtain normalized data information flows under different days.
4. The method for detecting bad weather based on sensor fusion and incremental learning according to claim 1, wherein the incremental learning training in the detecting step comprises the steps of:
the segmentation step: weather data belonging to different domains is continuously segmented; wherein, the data stream batches near the dividing points of different weather data streams comprise data on the left side and the right side;
an input step: the image data which are continuously segmented are sent into a model for training in the form of tensors, and M tensors are fixedly stored in the training process;
training: the similarity of the two tensors is measured through style measurement, and if the similarity is high, the old tensor with the lowest similarity in the storage is replaced by the new tensor with high similarity; if the similarity is low, performing pseudo-domain detection on tensors with the styles being too different from the existing domain styles by using a random forest algorithm and a sparse random projection algorithm, wherein tensors which do not pass through detection are additionally stored, and added into M memories when a certain number is accumulated, and meanwhile, the same number is removed from the M memories, so that the balance of the number of the memories and the number of the data in different domains is maintained.
5. The method for sensor fusion and incremental learning based detection in severe weather according to claim 2, wherein the spatial alignment process comprises:
correcting the binocular vision sensor, and estimating a rotation matrix R and a translation matrix T of the right vision sensor relative to the left vision sensor according to the rotation and translation matrices of the left and right vision sensors;
the rotation matrix is calculated by:
wherein R is l 、R r Is the rotation matrix of the left and right side view sensor relative to the world coordinate system, the subscripts l and r respectively represent the left and right sides, the superscript T represents the matrix transposition, and the requirements are satisfiedThe superscript-1 denotes the inverse of the matrix;
the translation matrix is calculated by:
T＝T r -RT l
wherein T is l 、T r Is a translation matrix of the left and right view sensors relative to the world coordinate system;
corner point p of left calibration plate w The coordinates (U, V) are transformed from the world coordinate system to the left view sensor coordinate system through rigid transformation to obtain the coordinates (X) of the point under the left view sensor coordinate system l ,Y l ,Z l )；
Coordinates (X) l ,Y l ,Z l ) The calculation of (2) is obtained by:
wherein R is l 、T l The rotation matrix and the translation matrix of the left view sensor coordinate system relative to the world coordinate system are respectively;
coordinates (X) in the left vision sensor coordinate system l ，Y l ，z l ) The image is converted into a pixel coordinate system of an image obtained by a right-side view sensor through rigid body transformation, perspective projection and radiation transformation, and the pixel coordinate (u 'of the point is obtained' r ，v′ r ) I.e. corner p w Projection coordinates of the coordinates in a pixel coordinate system;
the projection coordinates are calculated by the following equation:
wherein K is r Is an internal reference matrix of the right-side view sensor;
taking n calibration plate corner points on M images, optimizing an objective function, and optimizing all matrix parameters by using an L-M algorithm to minimize projection errors of the corner points;
the calculation of the objective function is derived from the following equation:
in the formula (u) r ，v r ) The actual position of the corner in the pixel coordinate system of the image obtained by the right view sensor is shown;
for left-side vision sensor, in spatial alignment of radar sensor and vision sensor, according to its rotation matrix R relative to radar sensor lra And a translation matrix T lra The conversion of a coordinate system is realized, and points in the left view sensor coordinate system are converted into a radar sensor coordinate system;
the coordinate system transformation is obtained by:
wherein (X) l ，Y l ，Z l ) Is the coordinates in the left vision sensor coordinate system, (X) ra ，Y ra ，Z ra ) Is the coordinates in the corresponding radar sensor coordinate system.
6. The method for sensor fusion and incremental learning based detection in severe weather according to claim 3, wherein the step of fusing data comprises:
two data tensors f of left and right vision sensor l 、f r Obtaining a single Zhang Shijiao sensor characteristic diagram through filter connection; channel fusion is carried out on the vision sensor characteristic diagram through 1X 1 convolution, different channel information is integrated, and the number of channels is changed through 3X 3 convolution, so that the vision sensor characteristic diagram f is obtained c ；
The calculation of the visual sensor profile is made by:
in the method, in the process of the invention,indicating filter connection, W 1 、W 2 Respectively represent convolution of the first layer network and the second layer network, BN 1 、BN 2 The Batch normalization respectively represents the first layer network and the second layer network;
radar data tensor f ra Normalized and normalized vision sensor characteristic diagram f c Realizing data level fusion through filter connection to obtain a feature diagram f;
the calculation of the feature map is derived from the following equation:
wherein F is N Representing a normalization function, and scaling the feature map elements to 0-1;
the calculation of the normalization function is derived from the following equation:
where f=f-min (f), f is a two-dimensional tensor.
7. The method for detecting bad weather based on sensor fusion and incremental learning according to claim 4, wherein the training step comprises:
the three-dimensional feature map f is flattened to obtain a two-dimensional feature map x, the channel number is kept unchanged, and a gram matrix G is obtained from the two-dimensional feature map x;
the calculation of the gram matrix is given by:
G＝x T ×x
wherein x represents matrix multiplication and T represents matrix transposition;
the gram matrix G is used to measure the difference between the two feature maps, denoted by d;
the calculation of the variability is obtained by:
where, l represents a layer-one network,the i, j-th elements representing two gram matrices G, A, G, A are the calculated gram matrices for two different feature maps, respectively, f=h×w representing the gram matrix size, where h and w are the length and width of the feature maps, respectively.
8. A detection system based on sensor fusion and incremental learning in severe weather, comprising:
and the acquisition module is used for: data acquisition is carried out on the driving environment through a radar sensor and a vision sensor respectively, so that radar sensor data and vision sensor data are obtained;
the processing module is used for: processing the radar sensor and vision sensor data according to a space-time alignment rule to obtain time-synchronous radar sensor and vision sensor data, space-aligned binocular vision sensor data and space-aligned radar sensor and vision sensor data;
and a fusion module: extracting bottom layer characteristics of the vision sensor data and carrying out data level fusion on the bottom layer characteristics and the radar sensor data to obtain a data information stream;
and a detection module: and selectively storing the data fused in the fusion module according to different weather conditions, and training according to the rule of incremental learning to obtain a target detection result.
9. A computer readable storage medium storing a computer program, wherein the computer program when executed by a processor implements the steps of the sensor fusion and incremental learning based detection method in bad weather according to any one of claims 1 to 7.
10. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the computer program when executed by the processor realizes the steps of the sensor fusion and incremental learning based detection method in bad weather according to any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311474818.2A CN117333846A (en) | 2023-11-03 | 2023-11-03 | Detection method and system based on sensor fusion and incremental learning in severe weather |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311474818.2A CN117333846A (en) | 2023-11-03 | 2023-11-03 | Detection method and system based on sensor fusion and incremental learning in severe weather |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117333846A true CN117333846A (en) | 2024-01-02 |
Family
ID=89295543
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311474818.2A Pending CN117333846A (en) | 2023-11-03 | 2023-11-03 | Detection method and system based on sensor fusion and incremental learning in severe weather |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117333846A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117593717A (en) * | 2024-01-18 | 2024-02-23 | 武汉大学 | Lane tracking method and system based on deep learning |
-
2023
- 2023-11-03 CN CN202311474818.2A patent/CN117333846A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117593717A (en) * | 2024-01-18 | 2024-02-23 | 武汉大学 | Lane tracking method and system based on deep learning |
| CN117593717B (en) * | 2024-01-18 | 2024-04-05 | 武汉大学 | Lane tracking method and system based on deep learning |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111340797B (en) | Laser radar and binocular camera data fusion detection method and system | |
| CN108932736B (en) | Two-dimensional laser radar point cloud data processing method and dynamic robot pose calibration method | |
| CN111563442B (en) | Slam method and system for fusing point cloud and camera image data based on laser radar | |
| CN111429514A (en) | Laser radar 3D real-time target detection method fusing multi-frame time sequence point clouds | |
| Wulff et al. | Early fusion of camera and lidar for robust road detection based on U-Net FCN | |
| CN113313763B (en) | Monocular camera pose optimization method and device based on neural network | |
| CN111681259B (en) | Vehicle tracking model building method based on Anchor mechanism-free detection network | |
| CN113903011B (en) | Semantic map construction and positioning method suitable for indoor parking lot | |
| CN110487286B (en) | Robot pose judgment method based on point feature projection and laser point cloud fusion | |
| CN114359181B (en) | Intelligent traffic target fusion detection method and system based on image and point cloud | |
| CN115032651A (en) | Target detection method based on fusion of laser radar and machine vision | |
| CN108645375B (en) | Rapid vehicle distance measurement optimization method for vehicle-mounted binocular system | |
| CN111797684B (en) | Binocular vision ranging method for moving vehicle | |
| CN117333846A (en) | Detection method and system based on sensor fusion and incremental learning in severe weather | |
| CN113205604A (en) | Feasible region detection method based on camera and laser radar | |
| CN113592905A (en) | Monocular camera-based vehicle running track prediction method | |
| CN113313047A (en) | Lane line detection method and system based on lane structure prior | |
| CN113256731A (en) | Target detection method and device based on monocular vision | |
| CN114413958A (en) | Monocular vision distance and speed measurement method of unmanned logistics vehicle | |
| CN111583342A (en) | Target rapid positioning method and device based on binocular vision | |
| CN116978009A (en) | Dynamic object filtering method based on 4D millimeter wave radar | |
| WO2020113425A1 (en) | Systems and methods for constructing high-definition map | |
| US20220164595A1 (en) | Method, electronic device and storage medium for vehicle localization | |
| CN109785388B (en) | Short-distance accurate relative positioning method based on binocular camera | |
| CN115187959B (en) | Method and system for landing flying vehicle in mountainous region based on binocular vision |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |