WO2023283987A1 - Sensor security detection method and device for unmanned system, and storage medium - Google Patents

Abstract

Disclosed in the present invention is a sensor security detection method for an unmanned system, comprising: dividing sensors in the unmanned system into two groups according to the positioning and detection functions, and respectively searching the cross correlation between the sensors in each group; screening out suspicious sensors according to the difference between sense data of the sensors with cross correlation in each group of sensors; and determining whether the suspicious sensor is an abnormal sensor or not according to the correlation of the sense data of each suspicious sensor at two adjacent moments. Also disclosed in the present invention are a sensor security detection device for an unmanned system and a storage medium. According to the present invention, the sensors are divided into two groups according to two functions, i.e., sensor positioning and detection, the cross correlation between the sensors in the group and the autocorrelation of each sensor in the time sequence are searched, such that the suspicious sensors are screened out, and the abnormal sensor is determined and screened from the suspicious sensors, to realize real-time detection of sensor security in the unmanned system, thereby accurately positioning the abnormal sensor.

Description

Sensor safety detection method, device and storage medium for unmanned system technical field

The invention relates to the technical field of automatic driving and sensing, in particular to a sensor safety detection method, equipment and storage medium for an unmanned system.

Background technique

Unmanned driving technology is a comprehensive technology involving many cutting-edge technologies such as artificial intelligence, sensing technology, map technology and computers. On-board sensors are important measurement devices that self-driving cars rely on. The function of the sensor is like the eyes and ears of human beings, and it is mainly composed of laser radar, visual camera, millimeter wave radar, global positioning system (GPS) and other equipment. Vehicle-mounted sensors provide unmanned vehicles with detailed environmental information around them, and provide rich data for planning and decision-making modules, including the location, shape, category, and speed information of obstacles, as well as semantic understanding of some special scenarios (such as construction areas, traffic lights and traffic signs, etc.).

Unmanned driving is very dependent on on-board sensors, and the sensors are extremely vulnerable to external attacks or interference and become unreliable. The security of the planning and decision-making process of unmanned vehicles is based on the safety of sensors. The attack of sensors is a simple, direct, violent and effective method. Because it does not need to enter the interior of the unmanned driving system, the threshold of this attack method is Very low, it poses a huge security threat to unmanned systems. Once the sensor of the unmanned vehicle is attacked or fails, it will lead to distortion of the information obtained by the sensor and wrong recognition results, and then plan an incorrect driving strategy, which is very likely to cause a car accident and cause serious loss of life and property.

At present, the research on unmanned driving technology mainly focuses on the functional level, such as environmental perception technology, sensor fusion technology, etc., but the security research on unmanned vehicle close-range attack is still in a relatively early stage, mainly through the single sensor. Some improvement and defense measures are taken to enhance the robustness of sensors and reduce the impact of attacks to a certain extent, but a real-time and systematic abnormal sensor detection method has not yet been formed.

Contents of the invention

In view of the deficiencies in the prior art, the present invention provides a sensor safety detection method, equipment and storage medium for unmanned systems, which can detect the safety of sensors in unmanned systems in real time and accurately locate abnormal sensors.

In order to achieve the above object, the present invention adopts the following technical solutions:

A sensor safety detection method for an unmanned system, comprising:

The sensors in the unmanned system are divided into two groups, the first group is the sensor for positioning, and the second group is the sensor for detecting the shape characteristics of the object;

Find the cross-correlation between the sensors in each group separately;

Screen out suspicious sensors based on the difference between the sensing data of sensors with cross-correlation in each group of sensors;

According to the correlation of sensing data of each suspicious sensor at two adjacent moments, it is judged whether it is an abnormal sensor.

As one of the implementation manners, the process of respectively finding the cross-correlation between the sensors in each group includes:

Unify the sensory data of sensors with correlation in each group into the same feature dimension;

The process of screening out suspicious sensors according to the difference between the sensory data of sensors with cross-correlation in each group of sensors includes:

Under the same feature dimension, establish a distance model to measure the distance between the sensory data of the sensors with mutual correlation in each group;

The distance between the corresponding sensing data is calculated in real time according to the distance model, and compared with the respective first distance thresholds, and the sensors corresponding to the sensing data exceeding the respective first distance thresholds are suspicious sensors.

As one of the implementations, the first group of sensors includes GPS (Global Positioning System, Global Positioning System), IMU (Inertial Measurement Unit, inertial measurement unit) and Lidar (ie laser radar);

The process of unifying the sensory data of the first group of correlated sensors into the same feature dimension, including: laser radar and IMU fusion positioning to form a fusion feature vector

The original feature vector of GPS is transformed into GPS feature vector after projective transformation

In the process of establishing the first distance model Distance (GPS, Lidar) to measure the distance between the sensing data of the sensors with mutual correlation in the first group, the norm is used to characterize the distance between the sensing data,

n is a non-negative integer.

As one of the implementation methods, when laser radar and IMU are fused and positioned, it includes:

IMU measurement data between two time points

perform pre-integration;

Using continuous feature point cloud sequence {P _k }, {P _k+1 } and IMU pre-integration results

Estimate the relative motion of the carrier.

As one of the implementations, the use of continuous feature point cloud sequences {P _k }, {P _k+1 } and IMU pre-integration results

The process of estimating the relative motion of the carrier includes:

Using IMU preintegration results

Carry out coordinate transformation on the feature point cloud sequence {P _k+1 }, so that it is in the same coordinate system as the feature point cloud sequence {P _k };

p _j ∈ P _k+1 , p _i ∈ P _k , the sum of the distances between p _j and p _i is denoted as d, and the distance from the carrier coordinate system (B) to the world coordinate system (W) when d is minimized is solved by using the Levenberg-Maquardt algorithm amount of rotation

and translation

According to the amount of rotation

and the translation amount

Get the positioning formula of lidar and IMU fusion

As one of the implementation manners, the second group of sensors includes a laser radar and a camera;

The process of unifying the perception data of the sensors with correlation in the second group into the same feature dimension, including:

Perception data for lidar:

After the transformation of the coordinate system, any three-dimensional point (X _L , Y _L , Z _L ) in the three-dimensional point cloud of the perception data is projected to a two-dimensional point (i, j) on the two-dimensional coordinate system;

Perform feature extraction on the obtained tensor to obtain the lidar feature vector

[X _L ,Y _L ] ^T represents the position of the detection target relative to the vehicle body coordinate system [0,0] ^T , [h _L ,w _L ] ^T represents the height and width of the detection frame, θ _L represents the detection target within the detection frame confidence level;

Perception data for camera:

Call the deep convolutional neural network algorithm to extract the features of the collected RGB images, and get the camera feature vector

[X _v Y _v ] ^T represents the coordinate value of the target object on the image coordinate system, [h _v , w _v ,] ^T represents the height and width of the detection frame, and p represents the confidence of the target object.

As one of the implementations, in the process of establishing the second distance model Distance(Camera, Lidar) to measure the distance between the sensing data of sensors with cross-correlation in the second group, the norm is used to characterize the distance between the sensing data distance,

n is a non-negative integer.

As one of the implementation manners, the process of determining a suspicious sensor includes:

A normal distribution of the distance D between the sensing data of the sensors A and B with mutual correlation in at least one of the first group and the second group is established:

D～N(ε,σ ² );

Among them, expect

variance

Distance _i (A, B) is the distance between the sensing data of two sensors A and B in the i-th sample, m is the total number of normal samples, and m is an integer greater than 0;

Determine whether the confidence interval M satisfies the conditions:

If not, the two sensors A, B are suspect sensors.

As one of the implementation manners, the process of judging whether each suspicious sensor is an abnormal sensor according to the correlation of sensing data at two adjacent moments includes:

Determine the feature vector representation for each suspect sensor

Represents the feature vector of the jth sensor at the ith moment;

Establish a third distance model Difference _sensor-ji (t _i , t _i+1 ) of the time series, the third distance model indicates that the perception data of the jth sensor is at two adjacent moments: the ith moment and the i+th 1 moment distance;

According to the third distance model, the distance between the feature vectors of each suspicious sensor at two adjacent moments is calculated respectively, and compared with the respective second distance thresholds, the sensors corresponding to the sensing data exceeding the respective second distance thresholds It is an abnormal sensor.

As one of the implementations, in the process of establishing the third distance model of the time series, the norm is used to characterize the distance between the perception data:

n is a non-negative integer,

Represents the feature vector of the jth sensor at the i+1th moment.

As one of the implementation manners, the process of determining the abnormal sensor includes:

Establish the normal distribution of the distance _D3 of the feature vector of the suspicious sensor at adjacent moments:

D ₃ ～N(τ,σ ₃ ² );

Among them, expect

variance

m ₃ represents the total number of adjacent time periods involved in the calculation, and m ₃ is an integer greater than 0;

Determine whether the confidence interval _M3 satisfies the conditions:

If not, the suspect sensor is an abnormal sensor.

Another object of the present invention is to provide a storage medium in which a plurality of instructions are stored, and the instructions are suitable for being loaded by a processor and performing the sensor safety detection of the unmanned system described in any one of the above method steps.

Another object of the present invention is to provide a sensor safety detection device for an unmanned system, which includes the above-mentioned storage medium and a processor suitable for implementing various instructions.

Aiming at the security problem of sensors, the present invention divides the sensors into two groups based on the two functions of sensor positioning and detection, and searches for the mutual correlation between the sensors in the group and the autocorrelation of each sensor in the time series, so that according to the The difference between the sensing data of the sensors of cross-correlation screens out suspicious sensors, and judges whether the sensing data of the sensors are safe and reliable according to the correlation of the sensing data of each suspicious sensor at two adjacent moments, so as to detect the unmanned system in real time. Sensor safety, accurate positioning of abnormal sensors.

Description of drawings

Fig. 1 is a flow chart of a sensor safety detection method of an unmanned system according to an embodiment of the present invention;

Fig. 2 is a functional block diagram of a sensor safety detection method of an unmanned system according to an embodiment of the present invention;

3 is a schematic diagram of an inertial measurement unit according to an embodiment of the present invention;

Fig. 4 is a functional block diagram of a multi-sensor fusion positioning according to an embodiment of the present invention;

Fig. 5 is the frequency relationship diagram of the IMU and Lidar of the embodiment of the present invention;

6 is a schematic diagram of the principle of spherical projection of the laser radar point cloud according to the embodiment of the present invention;

7 is a schematic diagram of the azimuth φ and vertex angle θ of the laser radar according to the embodiment of the present invention;

8 is a schematic diagram of the transformation relationship between the spherical coordinates and the rectangular coordinates of the laser radar according to the embodiment of the present invention;

Fig. 9 is a GPS, laser radar and IMU fusion positioning track diagram before the analog sensor of the embodiment of the present invention is attacked by the outside world;

Fig. 10 is a GPS, laser radar and IMU fusion positioning track diagram after the analog sensor of the embodiment of the present invention is attacked by the outside world;

Fig. 11 is a positioning track diagram of the laser radar according to the embodiment of the present invention after being tampered with and attacked;

Fig. 12 is a structural block diagram of a sensor safety detection device of an unmanned system according to an embodiment of the present invention.

detailed description

In the present invention, the terms "arranged", "provided", and "connected" should be interpreted broadly. For example, it may be a fixed connection, a detachable connection, or an integral structure; it may be a mechanical connection or an electrical connection; it may be a direct connection or an indirect connection through an intermediary; internal connectivity. Those of ordinary skill in the art can understand the specific meanings of the above terms in the present invention according to specific situations.

Moreover, some of the above terms may be used to indicate other meanings besides orientation or positional relationship, for example, the term "upper" may also be used to indicate a certain attachment relationship or connection relationship in some cases. Those skilled in the art can understand the specific meanings of these terms in the present invention according to specific situations.

In order to make the object, technical solution and advantages of the present invention more clear, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

Referring to Fig. 1 and Fig. 2, an embodiment of the present invention provides a sensor safety detection method for an unmanned system, including:

S01. Divide the sensors in the unmanned system into two groups, and unify the feature representation. The first group is the sensor for positioning, and the second group is the sensor for detecting the shape feature of the object.

Exemplarily, the on-board sensors in the unmanned system may include IMU, lidar, GPS, visual camera, millimeter-wave radar and other equipment.

As shown in Figure 3, the IMU is a device that measures the three-axis attitude angle (or angular rate) and acceleration of the carrier. The IMU is usually composed of 3 accelerometers, 3 gyroscopes, and 3 magnetometer groups, which are used to measure the angular velocity and acceleration of the object in three-dimensional space, and then calculate the attitude of the carrier. Accelerometers, gyroscopes and magnetometers are installed on mutually perpendicular measurement axes, and their output can be regarded as the output of acceleration, angular velocity and magnetic field strength in three directions with a total of 9 degrees of freedom (DOF), expressed as IMU=[a _x ,a _y ,a _z ,ω _x ,ω _y ,ω _z ,g _x ,g _y ,g _z ] ^T .

Lidar is a radar system that emits a laser beam to detect the position, speed and other characteristics of the target. Its working principle is to emit a detection laser beam to the target, and then combine the received signal (target echo) reflected from the target with the transmitted signal. To obtain relevant information of the target, such as target distance, azimuth, height, speed, attitude, and even shape parameters, so as to detect, track and identify the target and use simultaneous localization and mapping (SLAM) technology to estimate the carrier's position and posture.

The basic principle of the GPS navigation system is to measure the distance between the known satellite and the user's receiver, and then integrate the data of multiple satellites to determine the specific position of the receiver. Its output is usually longitude, latitude and altitude, expressed as GPS=[lon,lat,alt] ^T , where lon indicates longitude, lat indicates latitude, and alt indicates altitude.

The visual perception part includes target detection and positioning. The sensors can be divided into the following two groups according to the target detection and positioning functions: the first group of sensors can include GPS, IMU and lidar, and the second group of sensors can include lidar and cameras.

The first set of sensors is used for positioning. Since GPS is based on the global coordinate system, IMU and lidar are based on the carrier coordinate system, so in the positioning task, the positioning results of GPS and the positioning results of lidar and IMU can be unified in the world coordinate system. In the SLAM algorithm of fusion positioning, the two types of sensors are tightly coupled and related. Figure 4 shows the joint positioning framework of GPS, lidar and IMU.

A second set of sensors is used for detection. In the detection task, the camera and lidar are used to detect the target object, and the position coordinates and height and width information of the object are given. The point cloud of lidar in three-dimensional space can be projected into a spherical coordinate system, and the points in this spherical coordinate system can be transformed into a two-dimensional coordinate system. In the two-dimensional coordinate system, according to the distribution of points and the lidar target detection network, the feature vector of the detected target can be obtained, which contains the coordinates of the target and the size information of the detection frame. Similarly, the camera can also obtain the coordinates of the detection target and the size of the detection frame through the RGB image and the visual detection network.

S02. Search for the cross-correlation between the sensors in each group respectively.

The process of finding the cross-correlation between sensors in each group can specifically be to unify the sensory data of the correlated sensors in each group into the same feature dimension, as shown in Figure 2, in this process, the positioning task The processes of feature unification and feature unification in detection tasks are performed separately.

[The process of unified feature representation in positioning tasks]

Specifically, in the positioning task, it is first necessary to unify the perception data of the sensors with correlation in the first group into the same feature dimension. This process specifically includes:

1) Laser radar and IMU fusion positioning form a fusion feature vector

IMU original feature vector IMU＝[a _x , a _y , a _z ,ω _x ,ω _y ,ω _z ,g _x ,g _y ,g _z ] ^T , because in the positioning task, more attention is paid to the velocity and Acceleration state, so the IMU output characteristics can be defined as:

The acceleration is a three-dimensional vector in Euclidean space

angular acceleration

Velocity is a three-dimensional vector in Euclidean space

gyroscope bias

and accelerometer bias

Composition, the zero bias term

Lidar is a collection of point clouds in the original feature, which can be expressed as a vector

(64-line lidar, each line scans 1800 point clouds), where X represents the abscissa of the point cloud on the horizontal plane, Y represents the vertical coordinate of the point cloud on the horizontal plane, Z represents the height of the point cloud, and intensity represents the reflection of the point cloud strength. For the positioning task, the pose of the lidar carrier is solved by the SLAM algorithm, and its feature vector can be expressed as

The relationship between the lidar and IMU feature vectors: the carrier coordinate system is defined as B, which is consistent with the IMU coordinate system; the world coordinate system is defined as W, and the origin is the carrier body center when the system is initialized. Let the starting time of the i-th scan of Lidar be t _i , and all the point clouds obtained by the scan are P _i , where any point is recorded as p _n ∈ P _i ; the data collected by the IMU in [t _i , t _i+1 ] is I _(i,i+1) , i is an integer greater than 0. Since the IMU output frequency is higher than Lidar, I _(i,i+1) contains n groups of acceleration and angular velocity in the carrier coordinate system B

The state of the system at time t _i includes attitude, position, velocity and IMU bias. Among them, the pose transformation constitutes a special Euclidean group [R _i ,t _i ]∈SE(3); the velocity is a three-dimensional vector in Euclidean space

gyroscope bias

and accelerometer bias

constitute

According to the working principle of IMU, its observation model can be known as:

in,

η～N(0,Σ) represents the observation noise,

is the rotation matrix from the carrier coordinate system B to the world coordinate system W,

is the translation vector from the carrier coordinate system B to the world coordinate system W.

is the acceleration of gravity, according to the dynamic model of the IMU

and

Integrate within the IMU sampling interval time δt

get:

As shown in Figure 5, it is the frequency relationship diagram between IMU and Lidar. The laser odometer module uses the continuous feature point cloud sequence {P _k }, {P _k+1 } and IMU pre-integration results

Estimate the relative motion of the carrier, and its output frequency is consistent with the sampling frequency of Lidar.

Therefore, when using lidar and IMU fusion positioning, the following steps can be specifically included:

S021, the IMU measurement data between two moments

Perform pre-integration to obtain the pose change of the moving object between these two moments;

S022. Using the continuous feature point cloud sequence {P _k }, {P _k+1 } and IMU pre-integration results

Estimate the relative motion of the carrier.

In step S022, use the continuous feature point cloud sequence {P _k }, {P _k+1 } and IMU pre-integration results

The process of estimating the relative motion of the carrier may specifically include:

S0221. Use the IMU pre-integration result

Carry out coordinate transformation on the feature point cloud sequence {P _k+1 }, so that it is in the same coordinate system as the feature point cloud sequence {P _k }. Among them, p _j ∈ P _k+1 , p _i ∈ P _k , the sum of the distance between p _j and p _i is recorded as d, j and k are integers greater than 0, f(x) represents the distance expression of the point cloud, Have:

S0222. Make d the smallest, and use the Levenberg-Maquardt algorithm to solve the rotation amount from the carrier coordinate system B to the world coordinate system W when d is the smallest

and translation

Therefore, according to the amount of rotation

and translation

The positioning formula of lidar and IMU fusion can be obtained:

2) The original feature vector of GPS is transformed into GPS feature vector after projection transformation

The original feature vector of GPS GPS=[lon,lat,alt] ^T is latitude and longitude information based on WGS-84 coordinates, where lon is longitude, lat is latitude, and alt represents altitude. For the positioning task of automatic driving, since the GPS coordinate system needs to be projected into a flat map, dead reckoning can be done. Therefore, the feature vector output by the GPS positioning algorithm can be defined as

GPS positioning can construct the relationship between the world coordinate system W where the carrier is located and the earth coordinate system (WGS-84), and the original feature vector output at time i+1 can be expressed as

The original GPS output positioning is based on the latitude and longitude information of the WGS-84 coordinates. The GPS coordinate system needs to be projected into a flat map before dead reckoning can be done. The commonly used projection methods are Gauss Kruger projection, UTM projection, Mercator projection and so on.

In this embodiment, the GPS coordinates can be converted using the Mercator projection, and the conversion method is as follows:

z _G,i+1 = alt _i+1 ;

Among them, x _G,i+1 , y _G,i+1 , z _G,i+1 are the ground coordinate system coordinates of the original GPS feature vector projected by Mercator, EARTH_RAD is the radius of the earth, and the value is 6378137 meters. SCALE represents the map scale. Therefore, the feature vector output by the GPS positioning algorithm at time i+1

[The process of unified feature representation in detection tasks]

Specifically, in the detection task, the traditional CNN (Convolutional Neural Networks, convolutional neural network) detection task design is mostly used for two-dimensional image pattern recognition (width × height × number of channels), considering that the three-dimensional point cloud data format is different It conforms to this pattern, and the point cloud data is sparse and irregular, which is not good for feature extraction. Therefore, it is first necessary to unify the sensory data of the correlated sensors in the second group into the same feature dimension, and this process will be described in detail below.

Before inputting point cloud data into CNN, the data is first spherically projected to obtain a dense, two-dimensional data. The process of spherical projection to a two-dimensional image plane is shown in Figure 6.

(1) LiDAR

Fig. 7 is a schematic diagram of the azimuth φ and the vertex θ of the lidar, where φ and θ in the figure represent the azimuth and the azimuth of the point respectively. Usually, the azimuth is the angle relative to the true north direction, but under the Lidar coordinate system of the present embodiment, the azimuth is the angle relative to the x direction (directly ahead of the vehicle), and the calculation formulas of φ and θ are: :

Among them, (X, Y, Z) is the coordinate of each point in the 3D point cloud. Therefore, according to the above formula, for each point in the point cloud, its (θ, φ) can be calculated through its coordinates (X, Y, Z), thus projecting the points in the three-dimensional space coordinate system to a spherical coordinate Tie.

As shown in Figure 8, the transformation relationship between spherical coordinates and rectangular coordinates is shown. By differentiating the angle (θ, φ), a two-dimensional rectangular coordinate system can be obtained:

According to the above formula, each point (X _L , Y _L , Z _L ) in the spherical coordinate system can be represented by a point in the rectangular coordinate system.

That is to say, for the perception data of lidar, when the perception data is unified under the same feature dimension, the following process needs to be carried out:

1) After the transformation of the coordinate system, any 3D point (X _L , Y _L , Z _L ) in the 3D point cloud of the perception data is projected to a 2D point (i, j) on the 2D coordinate system to obtain a Tensor of size (W, H, C), W represents the number of grids divided by the laser perception range in the horizontal direction; H represents the number of lines of the laser radar; C represents the laser point feature vector

of dimensions.

Here, the 5 features of each point in the point cloud:

is put into the corresponding two-dimensional coordinates (i, j), intensity represents the reflection intensity of the point cloud, and range represents the distance from the viewpoint to the sampling point.

2) Through the lidar target detection network to extract the features of the obtained tensor, the lidar feature vector can be obtained:

Among them, [X _L , Y _L ] ^T represents the position of the detection target relative to the vehicle body coordinate system [0,0] ^T , [h _L , w _L ] ^T represents the height and width of the detection frame, θ _L represents the Confidence in detecting objects.

(2) Camera

The self-driving vehicle collects images around the vehicle through the on-board camera, and the collected pictures are input into the target detection system in RGB format. The detection system calls the deep convolutional neural network algorithm to extract the features of the RGB image. The common structure in the neural network algorithm is convolution layer, pooling layer, activation layer, dropout layer, BN (batch normalization) layer, fully connected layer, etc., and finally the features extracted from the picture can effectively describe the information of the target object.

Therefore, for the sensory data of the camera, when the sensory data is unified into the same feature dimension, the following process is specifically required: call the deep convolutional neural network algorithm to perform feature extraction on the collected RGB images, and obtain the camera feature vector:

Among them, [X _v Y _v ] ^T represents the coordinate value of the target object on the image coordinate system, [h _v , w _v ,] ^T represents the height and width of the detection frame, and p represents the confidence of the target object.

S03. Screen out suspicious sensors according to the difference between the sensing data of the sensors with mutual correlation in each group of sensors.

The process of screening suspicious sensors can specifically include:

S031. Under the same feature dimension, establish a distance model for measuring the distance between the sensing data of the sensors with mutual correlation in each group.

S032. Calculate the distance between the corresponding sensing data in real time according to the distance model, and compare it with the respective first distance thresholds, and the sensors corresponding to the sensing data exceeding the respective first distance thresholds are suspicious sensors.

In step S031, similarly, the establishment of the distance model corresponding to the first group of positioning sensors and the establishment of the distance model corresponding to the second group of detection sensors are performed independently, and the specific process is as follows.

[The process of establishing the distance model in the positioning task]

According to the conversion relationship between GPS positioning and lidar and IMU fusion positioning, the perception information of GPS, lidar and IMU can be unified under the same feature representation, and the distance model of the two can be established under the same feature.

In the process of establishing the first distance model Distance (GPS, Lidar) to measure the distance between the sensing data of the sensors with cross-correlation in the first group, this embodiment uses the norm to represent the distance between the sensing data, then GPS The output feature distance of the localization method fused with lidar and IMU can be expressed as:

Among them, n is a non-negative integer.

[The process of establishing the distance model in the detection task]

Theoretically, the lidar feature vector

with camera character vector

The detection results should be consistent, which contains two meanings: one is that the size of the detection frame is consistent, and the other is that the coordinates are consistent, that is, the detection frame completely overlaps. However, due to the influence of noise and sensor measurement accuracy, there will be a certain normal offset between these two feature vectors. built on the same characteristics

The distance model between the two can be used to calculate the distance between the detection frame and the coordinates of the two.

In the process of establishing the second distance model Distance(Camera, Lidar) to measure the distance between the sensing data of the sensors with cross-correlation in the second group, the present embodiment uses a norm to characterize the distance between sensing data:

Among them, n is a non-negative integer.

In step S032, the process of determining suspicious sensors may specifically include:

S0321. Establish a normal distribution of the distance D between the sensing data of the sensors A and B with mutual correlation in at least one of the first group and the second group:

D～N(ε,σ ² );

Among them, expect

variance

Distance _i (A, B) is the distance between the sensing data of two sensors A and B in the i-th sample, m is the total number of normal samples, and m is an integer greater than 0;

S0322. Determine whether the confidence interval M satisfies the condition

n is a constant that varies according to the confidence level. If not, the two sensors A and B are abnormal, and they are suspicious sensors; otherwise, the two sensors are normal.

For example, if the distance between the two sensors can be judged to be normal with a confidence level of 99%, the corresponding confidence interval M should satisfy the condition:

which is,

Indicates that sensors A and B have no abnormalities at a confidence level of 99%, otherwise it indicates that sensors A and B are abnormal.

It can be understood that, in other implementation manners, the confidence level can be set as required, and is not necessarily limited to 99%. For example, when finding the 90% confidence interval, n=1.65,

When seeking the 95% confidence interval, n=1.96,

Similarly, as shown in FIG. 2 , the abnormality of each group of sensors is detected in the positioning task and the detection task, which will be introduced in detail below.

[Detection process of abnormal sensors in positioning tasks]

In the positioning task, since the lidar and the inertial sensor in the first group of sensors have certain measurement errors, under normal circumstances, affected by factors such as measurement error, measurement accuracy, and environmental noise, the features of the two in the same dimension are also There is a certain deviation or distance. Since the distance between the perception data of the lidar and the inertial sensor is affected by many factors, here, it can be considered that the distance between the two perception data obeys a normal distribution. We define the mean distance between the two under normal circumstances (that is, the distance expectation) as ε ₁ , and σ ₁ ² as the variance, and the expected ε ₁ and variance σ ₁ ² can be calculated through the sample:

Among them, Distance _i (GPS, Lidar) is the distance between GPS and Lidar in the i-th sample, and m ₁ represents the total number of normal samples.

Establish a normal distribution of the distance D ₁ of the lidar and GPS:

D ₁ ～N(ε ₁ ,σ ₁ ² )

Assuming that the distance of this group of sensors can be judged to be normal with _a confidence level of 99%, the corresponding confidence interval M1 should satisfy the condition:

That is, if

It means that there is no abnormality in the GPS and lidar in the positioning sensor with a confidence level of 99%, otherwise it means that there is an abnormality in the IMU or lidar in the positioning sensor.

[Detection process of abnormal sensor in detection task]

In the detection task, ideally, the detection frames of the lidar and the camera in the second group of sensors should completely coincide. The offset (including the offset of the coordinates and the offset of the length and width of the detection frame), and the offset distance is within a certain range. Since the distance between the lidar and the camera perception data is affected by many factors, it can be considered that the distance between the two perception data obeys the normal distribution. We define ε ₂ as the distance expectation, σ ₂ ² as the variance, and the expected ε ₂ and variance σ ₂ ² can be calculated through the sample:

Among them, Difference _i (Camera, Lidar) is the offset of the detection frame of the camera and lidar in the i-th sample, and m ₂ represents the total number of normal samples.

Establish the normal distribution of the distance D ₂ between the lidar and the camera:

D ₂ ～N(ε ₂ ,σ ₂ ² )

Assuming that the distance of this group of sensors can be judged to be normal with a confidence level of 99%, then the corresponding confidence interval _M2 satisfies the condition:

That is, if

Indicates that there is no abnormality in the camera and lidar in the detection task with a 99% confidence level, otherwise it indicates that there is an abnormality in the camera or lidar in the detection task.

S04. According to the correlation of the sensing data of each suspicious sensor at two adjacent moments, determine whether it is an abnormal sensor.

The process of judging abnormal sensors may specifically include:

S041. Determine the feature vector representation of each suspicious sensor

Represents the feature vector of the jth sensor at the ith moment.

After the suspicious sensors are screened out in step S03, it is necessary to further detect the autocorrelation of the abnormal sensors in the group, that is, check whether a single sensor is abnormal in time series, and finally locate the abnormal sensor. Specifically, if the positioning function is abnormal, it is necessary to detect the correlation of the first group of sensors: lidar and IMU in time series, and locate the abnormal sensor. If the detection function is abnormal, it is necessary to detect the correlation of the second group of sensors: lidar and camera in time series, and locate the abnormal sensor.

In order to calculate the distance between the sensing data of the sensor at two adjacent moments, it is first necessary to determine the feature representation of the sensor. Considering the calculation efficiency and calculation accuracy comprehensively, the eigenvectors of the sensors used for time series correlation calculation of each group of sensors are expressed as follows:

The original feature vector output by GPS positioning at time i can be expressed as

The output feature vector after Mercator projection

The initial feature vector output by the lidar at time t _i is

Where X represents the abscissa of the point cloud on the horizontal plane, Y represents the ordinate of the point cloud on the horizontal plane, Z represents the height of the point cloud, and intensity represents the reflection intensity of the point cloud. The feature vector here is the original feature vector of the lidar, regardless of whether it is a detection task or a positioning task.

Solve the feature vector at time i through the positioning algorithm of lidar

Among them, x _l,i represents the abscissa of the horizontal plane where the carrier equipped with lidar is located at time t _i , y _l,i represents the vertical coordinate of the horizontal plane where the carrier is located, and z _l,i represents the height of the carrier. In the detection task, use the laser detection algorithm to solve the feature vector at time i

Where [X _L,i ,Y _L,i ] ^T represents the position of the detection target relative to the vehicle body coordinate system [0 0] ^T , [h _L,i ,w _L,i ] ^T represents the height and width of the detection frame, θ _L is the confidence level of the detection target within the detection frame.

The initial feature vector output by the IMU at time t _i

The acceleration is a three-dimensional vector in Euclidean space

angular acceleration

Velocity is a three-dimensional vector in Euclidean space

gyroscope bias

Zero and accelerometer bias

constitute

Camera target object detection result with vector

express. Among them, X _v,i Y _v,i ] ^T represents the coordinate value of the target object on the image coordinate system, [h _i w _i ] ^T represents the height and width of the detection frame, and p _i represents the confidence of the target object. In time series correlation analysis, the confidence level does not participate in the correlation analysis.

S042. Establish the third distance model Difference _sensor-ji (t _i , t _i+1 ) of the time series, the third distance model indicates that the sensory data of the jth sensor is at two adjacent moments: the i-th moment and the i+th 1 moment of distance.

Under normal circumstances, the information sensed by the sensor at two adjacent moments must have a large area of overlap, that is, there is an obvious correlation in the time series, and the j-th sensor is at two adjacent moments (i-th and i-th The distance at time +1) can be expressed as: Difference _sensor-ji (t _i ,t _i-1 ).

In the process of establishing the third distance model Difference _sensor-ji (t _i , t _i+1 ) of the time series, the norm is also used in this embodiment to calculate the characteristic distance of sensor perception information at two adjacent moments:

Among them, L _n (t _i ,t _i+1 ) represents the distance between the characteristic vector of the sensor at the i-th moment and the i+1 moment,

Represents the feature vector of the jth sensor at the ith moment,

Represents the feature vector of the jth sensor at the time i+1, n is a non-negative integer, and can be 0, 1, 2, 3....

S043. According to the third distance model, respectively calculate the distance between the feature vectors of each suspicious sensor at two adjacent moments, and compare them with the respective second distance thresholds, the sensor corresponding to the sensing data exceeding the respective second distance thresholds It is an abnormal sensor.

After the third distance model Difference _sensor-ji (t _i ,t _i+1 ) is established, according to the suspicious sensor to be further judged, the feature vector of the suspicious sensor can be substituted into the third distance model to calculate the suspicious The characteristic distance of sensor perception information.

Here, the second distance threshold corresponding to the characteristic distance of each sensor perception information is not always a fixed value. Considering that under normal circumstances, at two adjacent moments, the data sensed by the sensors must have most overlapping areas, that is, in There is an obvious correlation in the time series, so at the two moments before and after, the distance of the sensor feature vector must be within a certain threshold range. According to this idea, we judge whether the sensor is abnormal by calculating whether the distance between sensor perception data at adjacent moments is within the threshold range.

Since the distance between adjacent moments of a single sensor is affected by many factors such as environmental noise, measurement error, and feature extraction algorithm, we believe that the distance between the two obeys a normal distribution.

The process of identifying suspicious sensors can be done in the following ways:

S0431. Establish the normal distribution of the distance _D3 of the feature vector of the suspicious sensor at adjacent moments:

D ₃ ～N(τ,σ ₃ ² );

Among them, expect

variance

m ₃ represents the total number of adjacent time periods involved in the calculation, and m ₃ is an integer greater than 0. The expectation of the distance between adjacent moments of each sensor is different, which is related to the length of time between two adjacent moments of the sensor.

To judge whether the confidence interval M3 satisfies the condition, the confidence interval M3 with a confidence degree _of 99 _% is:

That is, if

Indicates that the feature data sensed by the jth sensor at the i+1th moment is normal with a 99% confidence level, and if not, the suspicious sensor is an abnormal sensor.

It can be understood that, in other implementation manners, the confidence level can be set as required, and is not necessarily limited to 99%. For example, when finding the 90% confidence interval, n=1.65,

When seeking the 95% confidence interval, n=1.96,

The sensor safety detection method of the unmanned system in this embodiment has been simulated, verified, and used through experiments, and can detect the safety of the sensors in the unmanned system in real time and accurately locate abnormal sensors. The specific verification process is as follows.

(1) Experiment preparation

The multi-sensor cross-correlation positioning experiment is carried out on the ROS (Robot Operating System) robot operating system, using the KITTI dataset as test data. This dataset is used to evaluate algorithms such as lidar odometry, visual odometry, stereo, optical flow, 3D object detection and 3D tracking Performance in an in-vehicle environment. KITTI contains real image data collected in scenes such as urban areas, rural areas, and highways.

In the data acquisition platform, the sensor includes a 64-line laser radar, which is located in the center of the roof. A color camera and a black and white camera are placed on both sides of the laser radar, a total of four cameras. At the left rear of the radar, there is an integrated navigation system (OXTS RT 3003), which can output RTK/IMU integrated navigation results (including longitude, latitude and attitude), and also output IMU raw data.

(2) The overall idea of the experiment

In the experiment, the KITTI data set was played back offline, and the vehicle pose estimated by the lidar and IMU and the pose estimated by GPS were unified into the world coordinate system, and the eigenvalues of the lidar at different times were tampered with (simulated sensors were attacked by the outside world) , using the cross-correlation of the eigenvectors output by lidar, IMU and GPS, and the autocorrelation of each sensor to find the moment when the data is tampered with and attack, and locate the specific sensor being attacked.

Output features based on lidar and IMU positioning

and the characteristics of the GPS positioning output

Since only the movement of the vehicle on the plane is considered, in the lidar and IMU positioning, the value of z _l,i is a preset fixed height value, and the value of z _G,i measured by satellite reflects the height of the real geographic location , there is no correlation between the two, so the unmanned vehicle moving in the plane can only consider the output features

As shown in Figure 9, the dotted line represents the GPS positioning output x _G,i ,y _G,i in the spatial domain, and the solid line represents the positioning of the lidar and IMU in the spatial domain about x _l,i ,y _l,i Output the result.

(3) Establish cross-correlation model and autocorrelation model

Calculate the distance of the cross-correlation according to the second distance model of the sensor

Calculates the mean of two sets of work features. Every 10 seconds, two groups of positioning features that have not been tampered with by the attack are selected as test samples, and the mean value of the distance of the positioning feature sequence is calculated. Wherein the mean value ε ₁ =8.32, and the variance σ ₁ ² =42.65. Calculate the threshold for anomaly detection

Indicates that the test sample can be considered with a 99% confidence level that there is no abnormality in the GPS, lidar and IMU in the positioning sensor, otherwise it indicates that there is an abnormality in the GPS, IMU or lidar in the positioning sensor.

For the moment when the cross-correlation distance is outside the interval [2.37, 14.28], it is marked as the time t _i when the sensor may fail. Calculate the autocorrelation distance according to the third distance model of the sensor: Difference _sensor (t _i ,t _i+1 ), every 10s, calculate the expected τ and variance σ of the distances of GPS, Lidar and IMU at time i and i+1, respectively ₃ ² , calculate the average distance of the GPS test samples as ε ₁ =8.3239, and the variance σ ₁ ² =42.6508. Calculate the distance threshold for GPS anomalies

Indicates that the feature data of the GPS sensor sample at time i+1 is normal at a 99% confidence level, otherwise it is abnormal.

The average distance of the lidar test samples is calculated as ε ₂ =0.9, and the variance σ ₂ ² =0.3769. Substituting and calculating the distance threshold of lidar abnormality, we get

Indicates that the characteristic data of the lidar sensor sample at the i+1th time is normal at a 99% confidence level, otherwise it is abnormal.

In the same way, the average distance of the IMU test samples is calculated as ε ₃ =1.02402, and the variance σ ₃ ² =0.000201. Substituting into the calculation of the abnormal distance threshold of the IMU

Indicates that the feature data of the IMU sensor sample at time i+1 is normal at a 99% confidence level, otherwise it is abnormal.

(4) Wrong note experiment

Select 5 groups of attack points to simulate the sensor attack test, inject the error point cloud data of the lidar positioning feature at the time series index of 0, 1000, 2000, 3000, and 4000 respectively, and simulate the sensor being attacked at this moment.

Figure 10 is the location trajectory output in the space domain after the laser radar attack data is injected; Figure 11 shows the waveform of the location feature [x,y] ^T in the time domain, where the solid line represents the time of 0, 1000, 2000, 3000, and 4000 The characteristic waveform of injected lidar attack data, and the dotted line indicates the reference waveform of GPS positioning. The circle marks the positioning waveform at the moment when the lidar is attacked.

Table 1 below records the interaction between the laser radar and IMU fusion positioning and GPS positioning output at the time series index of 0, 1000, 2000, 3000, 4000 and the time series index of 500, 1500, 2500, 3500 normal time. According to the statistics of correlation and autocorrelation distance, it is found that at the moment of attack injection, the positioning output of laser radar and IMU fusion has a large deviation from the GPS reference waveform, and the average value of the deviation is more than 10 meters, which is much larger than the normal threshold

upper limit. At this time, the distance will be greater than

The data (such as the cells marked with gray background color in Table 1) are marked as possible failure of the sensor, and the corresponding time is t _i =0, 1000, 2000, 3000, 4000, and the Lidar, GPS, and IMU features are calculated at i The distance between time i and time i+1, after calculation, the distance between time i and time i+1 of the test sample injected into the attacking lidar is greater than the upper limit of Difference _lidar (t _i , t _i+1 ), and the measured The Difference _IMU (t _i ,t _i+1 )∈[1.01082,1.03722] and the Difference _GPS (t _i ,t _i+1 )∈[0.29,2.36] of the sample obtained, so it can be argued that the 99% confidence level is At time 0, 1000, 2000, 3000, and 4000, the GPS and IMU are normal but the lidar sensor is abnormal, which may be attacked.

Table 1. Test results

It can be seen that the sensor safety detection method for the unmanned system of this embodiment can very accurately locate the abnormal sensor.

In addition, as shown in FIG. 12 , this embodiment also provides a computer-readable storage medium 10 and a sensor safety detection device for an unmanned system. There are multiple instructions stored in the storage medium 10, and the instructions are suitable for use by The processor 20 loads and executes the steps of the above-mentioned sensor safety detection method for an unmanned system, and the storage medium 10 is a part of the detection device. In some embodiments, the processor may be a central processing unit (Central Processing Unit, CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor is typically used to control the overall operation of the computing device. In this embodiment, the processor is configured to run program codes stored in the storage medium or process data.

It should be noted that, in this document, the term "comprising", "comprising" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article or apparatus comprising a set of elements includes not only those elements, It also includes other elements not expressly listed, or elements inherent in the process, method, article, or device. Without further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus comprising that element.

Through the description of the above embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus a necessary general-purpose hardware platform, and of course also by hardware, but in many cases the former is better implementation. Based on such an understanding, the essence of the technical solution of the present invention or the part that contributes to the prior art can be embodied in the form of software products, and the computer software products are stored in a storage medium (such as ROM/RAM, disk, CD) contains several instructions to make a terminal (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) execute the methods described in various embodiments of the present invention.

To sum up, the embodiment of the present invention aims at the problem of sensor safety, starting from the two major functions of sensor positioning and detection, dividing the sensors into two groups, and unifying the sensing data into the same feature according to the cross-correlation between sensor sensing signals Under the representation, the distance model and distance threshold between sensors are constructed to measure the distance between sensors, and this is used as a basis for judgment to detect abnormalities in sensors. Secondly, for suspicious sensors, by analyzing and calculating the correlation of a single sensor itself in the time domain, the distance function and distance threshold of adjacent moments are established, and the normal distribution and confidence level are used to judge whether the sensor is abnormal, so as to accurately judge and Locate anomaly sensors. The present invention is not only aimed at a single sensor anomaly detection method, but a complete set of sensor safety detection methods for unmanned systems, not only suitable for unmanned vehicles, but also for other unmanned systems, such as unmanned aerial vehicles, Unmanned ships etc.

The above description is only the specific implementation of the present application. It should be pointed out that for those of ordinary skill in the art, without departing from the principle of the present application, some improvements and modifications can also be made. It should be regarded as the protection scope of this application.

Claims (20)

1. A sensor safety detection method for an unmanned system, including:

   The sensors in the unmanned system are divided into two groups, the first group is the sensor for positioning, and the second group is the sensor for detecting the shape characteristics of the object;

   Find the cross-correlation between the sensors in each group separately;

   Screen out suspicious sensors based on the difference between the sensing data of sensors with cross-correlation in each group of sensors;

   According to the correlation of sensing data of each suspicious sensor at two adjacent moments, it is judged whether it is an abnormal sensor.
2. The sensor safety detection method of unmanned system according to claim 1, wherein,

   The process of finding the cross-correlation between the sensors in each group respectively includes:

   Unify the sensory data of sensors with correlation in each group into the same feature dimension;

   The process of screening out suspicious sensors according to the difference between the sensory data of sensors with cross-correlation in each group of sensors includes:

   Under the same feature dimension, establish a distance model to measure the distance between the sensory data of the sensors with mutual correlation in each group;

   The distance between the corresponding sensing data is calculated in real time according to the distance model, and compared with the respective first distance thresholds, and the sensors corresponding to the sensing data exceeding the respective first distance thresholds are suspicious sensors.
3. The sensor safety detection method of unmanned system according to claim 2, wherein, the first group of sensors comprises GPS, IMU and lidar;

   The process of unifying the sensory data of the first group of correlated sensors into the same feature dimension, including: laser radar and IMU fusion positioning to form a fusion feature vector

   

   The original feature vector of GPS is transformed into GPS feature vector after projective transformation

   

   In the process of establishing the first distance model Distance (GPS, Lidar) to measure the distance between the sensing data of the sensors with mutual correlation in the first group, the norm is used to characterize the distance between the sensing data,

   

   n is a non-negative integer.
4. The sensor safety detection method of an unmanned system according to claim 3, wherein, when the lidar and the IMU are fused and positioned, it includes:

   IMU measurement data between two time points

   

   perform pre-integration;

   Using continuous feature point cloud sequence {P _k }, {P _k+1 } and IMU pre-integration results

   

   Estimate the relative motion of the carrier.
5. The sensor safety detection method of an unmanned system according to claim 4, wherein said utilizing continuous feature point cloud sequences {P _k }, {P _k+1 } and IMU pre-integration results

   

   The process of estimating the relative motion of the carrier includes:

   Using IMU preintegration results

   

   Carry out coordinate transformation on the feature point cloud sequence {P _k+1 }, so that it is in the same coordinate system as the feature point cloud sequence {P _k };

   p _j ∈ P _k+1 , p _i ∈ P _k , the sum of the distances between p _j and p _i is denoted as d, and the distance from the carrier coordinate system (B) to the world coordinate system (W) when d is minimized is solved by using the Levenberg-Maquardt algorithm amount of rotation

   

   and translation

   

   According to the amount of rotation

   

   and the translation amount

   

   Get the positioning formula of lidar and IMU fusion

   
6. The sensor safety detection method of an unmanned system according to claim 2, wherein the second group of sensors includes a laser radar and a camera;

   The process of unifying the perception data of the sensors with correlation in the second group into the same feature dimension, including:

   Perception data for lidar:

   After the transformation of the coordinate system, any three-dimensional point (X _L , Y _L , Z _L ) in the three-dimensional point cloud of the perception data is projected to a two-dimensional point (i, j) on the two-dimensional coordinate system;

   Perform feature extraction on the obtained tensor to obtain the lidar feature vector

   

   [X _L ,Y _L ] ^T represents the position of the detection target relative to the vehicle body coordinate system [0,0] ^T , [h _L ,w _L ] ^T represents the height and width of the detection frame, θ _L represents the detection target within the detection frame confidence level;

   Perception data for camera:

   Call the deep convolutional neural network algorithm to extract the features of the collected RGB images and get the camera feature vector

   

   [X _v Y _v ] ^T represents the coordinate value of the target object on the image coordinate system, [h _v , w _v ,] ^T represents the height and width of the detection frame, and p represents the confidence of the target object.
7. The sensor safety detection method of unmanned system according to claim 6, wherein, in establishing the second distance model Distance (Camera, Lidar) of the distance between the sensing data of the sensor with cross-correlation in the measurement second group In the process, the norm is used to represent the distance between the perception data,

   

   

   n is a non-negative integer.
8. The sensor safety detection method of unmanned system according to claim 2, wherein, the process of determining suspicious sensor comprises:

   A normal distribution of the distance D between the sensing data of the sensors A and B with mutual correlation in at least one of the first group and the second group is established:

   D～N(ε,σ ² );

   Among them, expect

   

   variance

   

   Distance _i (A, B) is the distance between the sensing data of two sensors A and B in the i-th sample, m is the total number of normal samples, and m is an integer greater than 0;

   Determine whether the confidence interval M satisfies the conditions:

   

   If not, the two sensors A, B are suspect sensors.
9. The sensor safety detection method of an unmanned system according to claim 1, wherein, according to the correlation of sensing data of each suspicious sensor at two adjacent moments, the process of judging whether it is an abnormal sensor comprises:

   Determine the feature vector representation for each suspect sensor

   

   

   Represents the feature vector of the jth sensor at the ith moment;

   Establish a third distance model Difference _sensor-ji (t _i , t _i+1 ) of the time series, the third distance model indicates that the perception data of the jth sensor is at two adjacent moments: the ith moment and the i+th 1 moment distance;

   According to the third distance model, the distance between the feature vectors of each suspicious sensor at two adjacent moments is calculated respectively, and compared with the respective second distance thresholds, the sensors corresponding to the sensing data exceeding the respective second distance thresholds It is an abnormal sensor.
10. The sensor safety detection method of an unmanned system according to claim 9, wherein,

    In the process of establishing the third distance model of the time series, the norm is used to characterize the distance between the perception data:

    

    n is a non-negative integer,

    

    Represents the feature vector of the jth sensor at the time i+1;

    The process of identifying abnormal sensors includes:

    Establish the normal distribution of the distance _D3 of the feature vector of the suspicious sensor at adjacent moments:

    D ₃ ～N(τ,σ ₃ ² );

    Among them, expect

    

    variance

    

    m ₃ represents the total number of adjacent time periods involved in the calculation, and m ₃ is an integer greater than 0;

    Determine whether the confidence interval _M3 satisfies the conditions:

    

    If not, the suspect sensor is an abnormal sensor.
11. The sensor safety detection method of an unmanned system according to claim 2, wherein the process of judging whether the suspicious sensor is an abnormal sensor according to the correlation of the sensing data of each suspicious sensor at two adjacent moments comprises:

    Determine the feature vector representation for each suspect sensor

    

    

    Represents the feature vector of the jth sensor at the ith moment;

    Establish a third distance model Difference _sensor-ji (t _i , t _i+1 ) of the time series, the third distance model indicates that the perception data of the jth sensor is at two adjacent moments: the ith moment and the i+th 1 moment distance;

    According to the third distance model, the distance between the feature vectors of each suspicious sensor at two adjacent moments is calculated respectively, and compared with the respective second distance thresholds, the sensors corresponding to the sensing data exceeding the respective second distance thresholds is the abnormal sensor;

    In the process of establishing the third distance model of the time series, the norm is used to characterize the distance between the perception data:

    

    n is a non-negative integer,

    

    Represents the feature vector of the jth sensor at the i+1th moment.
12. The sensor safety detection method of an unmanned system according to claim 11, wherein the process of determining an abnormal sensor comprises:

    Establish the normal distribution of the distance _D3 of the feature vector of the suspicious sensor at adjacent moments:

    D ₃ ～N(τ,σ ₃ ² );

    Among them, expect

    

    variance

    

    m ₃ represents the total number of adjacent time periods involved in the calculation, and m ₃ is an integer greater than 0;

    Determine whether the confidence interval _M3 satisfies the conditions:

    

    If not, the suspect sensor is an abnormal sensor.
13. The sensor safety detection method of an unmanned system according to claim 4, wherein, according to the correlation of sensing data of each suspicious sensor at two adjacent moments, the process of judging whether it is an abnormal sensor comprises:

    Determine the feature vector representation for each suspect sensor

    

    

    Represents the feature vector of the jth sensor at the ith moment;

    Establish a third distance model Difference _sensor-ji (t _i , t _i+1 ) of the time series, the third distance model indicates that the perception data of the jth sensor is at two adjacent moments: the ith moment and the i+th 1 moment distance;

    According to the third distance model, the distance between the feature vectors of each suspicious sensor at two adjacent moments is calculated respectively, and compared with the respective second distance thresholds, the sensors corresponding to the sensing data exceeding the respective second distance thresholds is the abnormal sensor;

    In the process of establishing the third distance model of the time series, the norm is used to characterize the distance between the perception data:

    

    n is a non-negative integer,

    

    Represents the feature vector of the jth sensor at the time i+1;

    The process of identifying abnormal sensors includes:

    Establish the normal distribution of the distance _D3 of the feature vector of the suspicious sensor at adjacent moments:

    D ₃ ～N(τ,σ ₃ ² );

    Among them, expect

    

    variance

    

    m ₃ represents the total number of adjacent time periods involved in the calculation, and m ₃ is an integer greater than 0;

    Determine whether the confidence interval _M3 satisfies the conditions:

    

    If not, the suspect sensor is an abnormal sensor.
14. The sensor safety detection method of an unmanned system according to claim 6, wherein, according to the correlation of sensing data of each suspicious sensor at two adjacent moments, the process of judging whether it is an abnormal sensor comprises:

    Determine the feature vector representation for each suspect sensor

    

    

    Represents the feature vector of the jth sensor at the ith moment;

    Establish a third distance model Difference _sensor-ji (t _i , t _i+1 ) of the time series, the third distance model indicates that the perception data of the jth sensor is at two adjacent moments: the ith moment and the i+th 1 moment distance;

    According to the third distance model, the distance between the feature vectors of each suspicious sensor at two adjacent moments is calculated respectively, and compared with the respective second distance thresholds, the sensors corresponding to the sensing data exceeding the respective second distance thresholds is the abnormal sensor;

    In the process of establishing the third distance model of the time series, the norm is used to characterize the distance between the perception data:

    

    n is a non-negative integer,

    

    Represents the feature vector of the jth sensor at the time i+1;

    The process of identifying abnormal sensors includes:

    Establish the normal distribution of the distance _D3 of the feature vector of the suspicious sensor at adjacent moments:

    D ₃ ～N(τ,σ ₃ ² );

    Among them, expect

    

    variance

    

    m ₃ represents the total number of adjacent time periods involved in the calculation, and m ₃ is an integer greater than 0;

    Determine whether the confidence interval _M3 satisfies the conditions:

    

    If not, the suspect sensor is an abnormal sensor.
15. The sensor safety detection method of an unmanned system according to claim 8, wherein, according to the correlation of sensing data of each suspicious sensor at two adjacent moments, the process of judging whether it is an abnormal sensor comprises:

    Determine the feature vector representation for each suspect sensor

    

    

    Represents the feature vector of the jth sensor at the ith moment;

    Establish a third distance model Difference _sensor-ji (t _i , t _i+1 ) of the time series, the third distance model indicates that the perception data of the jth sensor is at two adjacent moments: the ith moment and the i+th 1 moment distance;

    According to the third distance model, the distance between the feature vectors of each suspicious sensor at two adjacent moments is calculated respectively, and compared with the respective second distance thresholds, the sensors corresponding to the sensing data exceeding the respective second distance thresholds is the abnormal sensor;

    In the process of establishing the third distance model of the time series, the norm is used to characterize the distance between the perception data:

    

    n is a non-negative integer,

    

    Represents the feature vector of the jth sensor at the time i+1;

    The process of identifying abnormal sensors includes:

    Establish the normal distribution of the distance _D3 of the feature vector of the suspicious sensor at adjacent moments:

    D ₃ ～N(τ,σ ₃ ² );

    Among them, expect

    

    variance

    

    m ₃ represents the total number of adjacent time periods involved in the calculation, and m ₃ is an integer greater than 0;

    Determine whether the confidence interval _M3 satisfies the conditions:

    

    If not, the suspect sensor is an abnormal sensor.
16. A storage medium, wherein a plurality of instructions are stored in the storage medium, and the instructions are suitable for being loaded by a processor and executing the steps of a sensor safety detection method for an unmanned system, and the sensor safety detection method for an unmanned system Detection methods, including:

    The sensors in the unmanned system are divided into two groups, the first group is the sensor for positioning, and the second group is the sensor for detecting the shape characteristics of the object;

    Find the cross-correlation between the sensors in each group separately;

    Screen out suspicious sensors based on the difference between the sensing data of sensors with cross-correlation in each group of sensors;

    According to the correlation of the sensing data of each suspicious sensor at two adjacent moments, it is judged whether it is an abnormal sensor.
17. The storage medium according to claim 16, wherein the process of respectively finding the cross-correlation between the sensors in each group comprises:

    Unify the sensory data of sensors with correlation in each group into the same feature dimension;

    The process of screening out suspicious sensors according to the difference between the sensory data of sensors with cross-correlation in each group of sensors includes:

    Under the same feature dimension, establish a distance model to measure the distance between the sensory data of the sensors with mutual correlation in each group;

    The distance between the corresponding sensing data is calculated in real time according to the distance model, and compared with the respective first distance thresholds, and the sensors corresponding to the sensing data exceeding the respective first distance thresholds are suspicious sensors.
18. The storage medium according to claim 17, wherein,

    The first group of sensors includes GPS, IMU, and lidar. The process of unifying the sensory data of the first group of relevant sensors into the same feature dimension includes: lidar and IMU fusion positioning to form a fusion feature vector

    

    The original feature vector of GPS is transformed into GPS feature vector after projective transformation

    

    In the process of establishing the first distance model Distance (GPS, Lidar) to measure the distance between the sensing data of the sensors with mutual correlation in the first group, the norm is used to characterize the distance between the sensing data,

    

    n is a non-negative integer;

    When laser radar and IMU are fused and positioned, it includes:

    IMU measurement data between two time points

    

    perform pre-integration;

    Using continuous feature point cloud sequence {P _k }, {P _k+1 } and IMU pre-integration results

    

    Estimate the relative motion of the carrier;

    The second group of sensors includes lidar and cameras. The process of unifying the sensory data of the second group of sensors with correlations into the same feature dimension includes:

    Perception data for lidar:

    After the transformation of the coordinate system, any three-dimensional point (X _L , Y _L , Z _L ) in the three-dimensional point cloud of the perception data is projected to a two-dimensional point (i, j) on the two-dimensional coordinate system;

    Perform feature extraction on the obtained tensor to obtain the lidar feature vector

    

    [X _L ,Y _L ] ^T represents the position of the detection target relative to the vehicle body coordinate system [0,0] ^T , [h _L ,w _L ] ^T represents the height and width of the detection frame, θ _L represents the detection target within the detection frame confidence level;

    Perception data for camera:

    Call the deep convolutional neural network algorithm to extract the features of the collected RGB images, and get the camera feature vector

    

    [X _v Y _v ] ^T represents the coordinate value of the target object on the image coordinate system, [h _v , w _v ,] ^T represents the height and width of the detection frame, and p represents the confidence of the target object;

    In the process of establishing the second distance model Distance(Camera, Lidar) measuring the distance between the sensing data of the sensors with cross-correlation in the second group, the norm is used to represent the distance between the sensing data,

    

    

    n is a non-negative integer.
19. A sensor safety detection device for an unmanned system, which includes a storage medium and a processor adapted to implement instructions; multiple instructions are stored in the storage medium, and the instructions are suitable for being loaded by the processor and executed without The step of the sensor security detection method of human system, the sensor security detection method of described unmanned system, comprises:

    The sensors in the unmanned system are divided into two groups, the first group is the sensor for positioning, and the second group is the sensor for detecting the shape characteristics of the object;

    Find the cross-correlation between the sensors in each group separately;

    Screen out suspicious sensors based on the difference between the sensing data of sensors with cross-correlation in each group of sensors;

    According to the correlation of sensing data of each suspicious sensor at two adjacent moments, it is judged whether it is an abnormal sensor.
20. The sensor safety detection device for an unmanned system according to claim 19, wherein the process of respectively finding the mutual correlation between the sensors in each group comprises:

    Unify the sensory data of sensors with correlation in each group into the same feature dimension;

    The process of screening out suspicious sensors according to the difference between the sensory data of sensors with cross-correlation in each group of sensors includes:

    Under the same feature dimension, establish a distance model to measure the distance between the sensory data of the sensors with mutual correlation in each group;

    Calculate the distance between the corresponding sensing data in real time according to the distance model, and compare it with the respective first distance threshold, and the sensor corresponding to the sensing data exceeding the respective first distance threshold is a suspicious sensor;

    The first group of sensors includes GPS, IMU, and lidar. The process of unifying the sensory data of the first group of relevant sensors into the same feature dimension includes: lidar and IMU fusion positioning to form a fusion feature vector

    

    The original feature vector of GPS is transformed into GPS feature vector after projective transformation

    

    In the process of establishing the first distance model Distance (GPS, Lidar) to measure the distance between the sensing data of the sensors with mutual correlation in the first group, the norm is used to characterize the distance between the sensing data,

    

    n is a non-negative integer;

    The second group of sensors includes lidar and cameras. The process of unifying the sensory data of the second group of sensors with correlations into the same feature dimension includes:

    Perception data for lidar:

    After the transformation of the coordinate system, any three-dimensional point (X _L , Y _L , Z _L ) in the three-dimensional point cloud of the perception data is projected to a two-dimensional point (i, j) on the two-dimensional coordinate system;

    Perform feature extraction on the obtained tensor to obtain the lidar feature vector

    

    [X _L ,Y _L ] ^T represents the position of the detection target relative to the vehicle body coordinate system [0,0] ^T , [h _L ,w _L ] ^T represents the height and width of the detection frame, θ _L represents the detection target within the detection frame confidence level;

    Perception data for camera:

    Call the deep convolutional neural network algorithm to extract the features of the collected RGB images, and get the camera feature vector

    

    [X _v Y _v ] ^T represents the coordinate value of the target object on the image coordinate system, [h _v , w _v ,] ^T represents the height and width of the detection frame, and p represents the confidence of the target object.

Images