WO2022126397A1 - 传感器的数据融合方法、设备及存储介质 - Google Patents

传感器的数据融合方法、设备及存储介质 Download PDF

Info

Publication number
WO2022126397A1
WO2022126397A1 PCT/CN2020/136622 CN2020136622W WO2022126397A1 WO 2022126397 A1 WO2022126397 A1 WO 2022126397A1 CN 2020136622 W CN2020136622 W CN 2020136622W WO 2022126397 A1 WO2022126397 A1 WO 2022126397A1
Authority
WO
WIPO (PCT)
Prior art keywords
value
predicted
observation value
noise
predicted observation
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2020/136622
Other languages
English (en)
French (fr)
Inventor
孙扬
蒋卓键
陈晓智
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
SZ DJI Technology Co Ltd
Original Assignee
SZ DJI Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by SZ DJI Technology Co Ltd filed Critical SZ DJI Technology Co Ltd
Priority to PCT/CN2020/136622 priority Critical patent/WO2022126397A1/zh
Publication of WO2022126397A1 publication Critical patent/WO2022126397A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods

Definitions

  • the present application relates to the technical field of data processing, and in particular, to a sensor data fusion method, a data fusion device, a movable platform, a control terminal, and a storage medium.
  • a Kalman filter is generally used to obtain the current moment's observations by fusing the predicted observations at the current moment with the historical observations at the previous moment, and fuse the predicted and historical observations.
  • the standard deviation (noise) of the current moment observation needs to be used.
  • the standard deviation of the observation at the current moment is set according to the experience adjustment parameters or determined according to the kinematic equation of the previous moment.
  • the empirical parameter adjustment setting or the kinematic equation determination at the previous moment cannot truly reflect the observed noise, thus reducing the accuracy of data fusion.
  • the embodiments of the present application provide a sensor data fusion method, a data fusion device, a movable platform, a control terminal, and a storage medium, so as to improve the accuracy of sensor data fusion.
  • an embodiment of the present application provides a data fusion method for a sensor, where the data fusion method includes:
  • the current observation value is generated according to the historical data sensed by the sensor, the predicted observation value corresponding to the current data, and the predicted observation noise corresponding to the predicted observation value.
  • an embodiment of the present application further provides a data fusion device, where the data fusion device includes a processor and a memory;
  • the memory is used to store computer programs
  • the processor is configured to execute the computer program and when executing the computer program, realize:
  • the current observation value is generated according to the historical data sensed by the sensor, the predicted observation value corresponding to the current data, and the predicted observation noise corresponding to the predicted observation value.
  • an embodiment of the present application further provides a movable platform, where the movable platform includes:
  • a sensor mounted on the platform body for data
  • the memory is used to store a computer program;
  • the processor is used to execute the computer program and implement the following steps when executing the computer program:
  • the current observation value is generated according to the historical data sensed by the sensor, the predicted observation value corresponding to the current data, and the predicted observation noise corresponding to the predicted observation value.
  • an embodiment of the present application further provides a control terminal, where the control terminal is communicatively connected to a movable platform, and the movable platform includes a processor and a memory;
  • the memory is used to store computer programs
  • the processor is configured to execute the computer program and when executing the computer program, realize:
  • the current observation value is generated according to the historical data sensed by the sensor, the predicted observation value corresponding to the current data, and the predicted observation noise corresponding to the predicted observation value.
  • an embodiment of the present application further provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor can be implemented as implemented in the present application.
  • the sensor data fusion method, data fusion device, movable platform, control terminal, and storage medium disclosed in the embodiments of the present application for the current data sensed by the sensor at the current moment, can use the neural network model to obtain the predicted observation value of the current data and Predicting the observation noise, combined with the historical data collected by the sensor, can obtain the current observation value, thereby improving the accuracy of data fusion.
  • FIG. 1 is a schematic structural diagram of an unmanned aerial vehicle provided by an embodiment of the present application.
  • FIG. 2 is a schematic block diagram of a flight control system of an unmanned aerial vehicle provided by an embodiment of the present application
  • FIG. 3 is a schematic flowchart of steps of a method for data fusion of sensors provided by an embodiment of the present application
  • FIG. 4 is a schematic diagram of a data fusion process of a sensor provided by an embodiment of the present application.
  • FIG. 5 is a schematic block diagram of a data fusion apparatus provided by an embodiment of the present application.
  • FIG. 6 is a schematic block diagram of a movable platform provided by an embodiment of the present application.
  • FIG. 7 is a schematic block diagram of a control terminal provided by an embodiment of the present application.
  • a Kalman filter is used to fuse the predicted observation value of the current moment with the history of the previous moment. Observation value, get the current observation value, and fusing the predicted observation value and the historical observation value need to use the predicted observation noise (standard deviation) at the current moment.
  • the standard deviation of the observation at the current moment is determined according to the empirical parameter adjustment setting or the kinematic equation of the previous moment. However, if deep learning is used for observation, if the empirical parameter adjustment setting or the kinematic equation of the previous moment is also used, it cannot be determined. It really reflects the noise of observations, thus reducing the accuracy of data fusion.
  • the movable platform needs to be based on the speed of the surrounding target, the distance between the target and the movable platform, and the distance of the target relative to the movable platform. Location and other status information to drive.
  • the movable platform mainly collects the point cloud data around it through sensors (visual sensor, radar or TOF, etc.), and determines the predicted observation value of the state information of the target at the current moment through the point cloud data, and then the predicted observation value of the state information. It is fused with the estimated value of the state information of the target at the previous moment to obtain the state information (current observation value) of the target at the current moment.
  • the predicted observation value of the fusion state information and the estimated value of the state information need to use the currently observed predicted observation noise and the estimated noise predicted at the previous moment.
  • the predicted observation noise is also determined by using the empirical parameter adjustment setting or the kinematic equation of the previous moment, it cannot really reflect the observation noise, so the state information of the target object will be reduced.
  • the accuracy of the mobile platform cannot guarantee the driving safety of the movable platform.
  • the embodiments of the present application provide a sensor data fusion method, a data fusion device, a movable platform and a storage medium, by acquiring the current data sensed by the sensor; the predicted observation value corresponding to the current data and the predicted observation noise corresponding to the predicted observation value; and according to the historical data sensed by the sensor, the predicted observation value corresponding to the current data, and the predicted observation value corresponding to the predicted observation value Noise, generate the current observation value, the current observation value is the fusion data.
  • the accuracy of data fusion can be improved.
  • the sensor data fusion method provided in the embodiment of the present application may be applied to a data fusion apparatus, or may also be applied to a movable platform or a control terminal.
  • the movable platform includes at least one of an aircraft, a robot and an unmanned vehicle, and the aircraft includes an unmanned aircraft and a manned aircraft;
  • the control terminal includes a remote control, a smart phone, a tablet computer, a notebook computer, a desktop computer or a wearable equipment and other electronic equipment.
  • the senor includes one of a vision sensor, a radar, an inertial measurement device, a TOF device and an odometer, and the vision sensor includes a camera.
  • a vision sensor a radar
  • an inertial measurement device a TOF device
  • an odometer a odometer
  • the vision sensor includes a camera.
  • the following will take the mobile platform as an example of a UAV, and the sensor as a visual sensor or radar as an example.
  • FIG. 1 shows the structure of an unmanned aerial vehicle 100 provided by an embodiment of the present application
  • FIG. 2 shows a structural framework of a flight control system of the unmanned aerial vehicle 100 provided by an embodiment of the present application.
  • the UAV 100 may include a frame 10 , a power system 11 , a control system 12 and a radar 20 .
  • the frame 10 may include a fuselage and a foot frame (also known as a landing gear).
  • the fuselage may include a center frame and one or more arms connected to the center frame, the one or more arms extending radially from the center frame.
  • the tripod is connected with the fuselage, and is used for supporting when the drone 100 is landed.
  • the radar 20 can be installed on the UAV, specifically, can be installed on the rack 10 of the UAV 100. During the flight of the UAV 100, it is used to measure the surrounding environment of the UAV 100, such as obstacles, etc. to ensure flight safety. In the embodiment of the present application, the radar 20 is used to scan and measure the target around the UAV 100 to obtain corresponding data, so as to determine the state information of the target.
  • the radar 20 is installed on the tripod of the UAV 100 , the radar 20 is connected to the control system 12 in communication, and the radar 20 transmits the collected data to the control system 12 for processing by the control system 12 .
  • the UAV 100 may include two or more tripods, and the radar 20 is mounted on one of the tripods.
  • the radar 20 may also be mounted on other positions of the UAV 100, which is not specifically limited.
  • the radar 20 mainly includes a radio frequency front-end module and a signal processing module.
  • the radio frequency front-end module may include a transmitting antenna and a receiving antenna.
  • the transmitting antenna is used to send signals to the target, and the receiving antenna is used to receive the signal reflected by the target.
  • the signal processing module is responsible for Generate modulated signals and process and analyze the collected intermediate frequency signals, where the target objects are buildings, vehicles, pedestrians, mountains, buildings or other drones.
  • the power system 11 may include one or more electronic governors (referred to as ESCs for short), one or more propellers, and one or more motors corresponding to the one or more propellers, wherein the motors are connected between the electronic governors and the one or more propellers. Between the propellers, the motor and the propeller are arranged on the arm of the UAV 100; the electronic governor is used to receive the driving signal generated by the control system, and provide driving current to the motor according to the driving signal to control the speed of the motor.
  • ESCs electronic governors
  • the motor is used to drive the propeller to rotate, thereby providing power for the flight of the UAV 100, and the power enables the UAV 100 to achieve one or more degrees of freedom movement.
  • the drone 100 may rotate about one or more axes of rotation.
  • the above-mentioned rotation axes may include a roll axis, a yaw axis, and a pitch axis.
  • the motor may be a DC motor or a permanent magnet synchronous motor.
  • the motor may be a brushless motor or a brushed motor.
  • Control system 12 may include a controller and a sensing system.
  • the controller is used to control the flight of the UAV 100, for example, the flight of the UAV 100 can be controlled according to the attitude information measured by the sensing system. It should be understood that the controller can control the UAV 100 according to pre-programmed instructions.
  • the sensing system is used to measure the attitude information of the UAV 100, that is, the position information and state information of the UAV 100 in space, such as three-dimensional position, three-dimensional angle, three-dimensional velocity, three-dimensional acceleration, and three-dimensional angular velocity.
  • the sensing system may include at least one of a gyroscope, an ultrasonic sensor, an electronic compass, an inertial measurement unit (Inertial Measurement Unit, IMU), a visual sensor, a global navigation satellite system, a barometer, and other sensors.
  • the global navigation satellite system may be the Global Positioning System (GPS).
  • GPS Global Positioning System
  • the sensor in the sensing system may also use the sensor data fusion method provided by the embodiment of the present application to improve the accuracy of the data. So that the control system 12 performs flight control according to the fused data, thereby providing flight safety.
  • the controller may include one or more processors and memory.
  • the processor may be, for example, a micro-controller unit (Micro-controller Unit, MCU), a central processing unit (Central Processing Unit, CPU), or a digital signal processor (Digital Signal Processor, DSP), and the like.
  • the memory may be a Flash chip, a read-only memory (ROM, Read-Only Memory) disk, an optical disk, a U disk, or a mobile hard disk.
  • the UAV 100 may include a rotary-wing UAV, such as a quad-rotor UAV, a hexa-rotor UAV, an octa-rotor UAV, a fixed-wing UAV, or a rotary-wing and fixed-wing UAV.
  • a rotary-wing UAV such as a quad-rotor UAV, a hexa-rotor UAV, an octa-rotor UAV, a fixed-wing UAV, or a rotary-wing and fixed-wing UAV.
  • the combination of machines is not limited here.
  • the embodiment of the present application also provides a flight system, where the flight system includes an unmanned aerial vehicle 100 and a control terminal.
  • the control terminal is located on the ground end of the flight system, and can communicate with the UAV 100 wirelessly, so as to remotely control the UAV 100 .
  • the unmanned aerial vehicle 100 specifically the controller of the control system of the unmanned aerial vehicle 100, can be used to execute the data fusion method of any one of the sensors provided in the embodiments of the present application, so as to improve the accuracy of data fusion, and then Ensure the safety of drone flight.
  • the controller is used to: obtain the current data sensed by the sensor; determine the predicted observation value corresponding to the current data and the predicted observation noise corresponding to the predicted observation value based on the pre-trained neural network model;
  • the historical data sensed by the sensor, the predicted observation value corresponding to the current data, and the predicted observation noise corresponding to the predicted observation value are used to generate the current observation value, that is, the fusion result.
  • the current observation value is the state information of the target object at the current moment, so the UAV determines the state information of the target object according to the fusion result, and then uses the state information to avoid obstacles or plan the flight path, etc., which can improve the flight performance. safety.
  • control terminal of the flight system can also be used to execute the data fusion method of any one of the sensors provided in the embodiments of the present application, so as to improve the safety of UAV flight.
  • the control terminal is used to: obtain the current data sensed by a sensor (such as a visual sensor mounted on an unmanned aerial vehicle); based on a pre-trained neural network model, determine the predicted observation value corresponding to the current data and the The predicted observation noise corresponding to the predicted observation value; the current observation value, that is, the fusion result, is generated according to the historical data sensed by the sensor, the predicted observation value corresponding to the current data, and the predicted observation noise corresponding to the predicted observation value.
  • a sensor such as a visual sensor mounted on an unmanned aerial vehicle
  • the current observation value that is, the fusion result
  • the current observation value that is, the fusion result
  • the UAV can use the state information to avoid obstacles or plan the flight path, which can improve the safety of flight.
  • the neural network model can obtain the predicted observation value corresponding to the current data sensed by the sensor and the predicted observation value The corresponding predicted observation noise can truly reflect the observed noise, thereby improving the accuracy of data fusion.
  • the pre-trained neural network model provided by the embodiment of the present application is introduced.
  • the neural network model by setting the loss function of the neural network model, the neural network model can output the predicted observation value corresponding to the current data sensed by the sensor and the predicted observation noise corresponding to the predicted observation value.
  • the loss function of the neural network model is determined according to the fact that both the predicted observation value and the predicted observation noise obey a normal distribution, and the loss function is used to correlate the predicted observation value and the predicted observation noise.
  • the neural network model can output the predicted observation value and the observation noise corresponding to the current data at the same time.
  • the neural network model can be obtained. loss function.
  • the normal distribution between the predicted observation value and the predicted observation noise is:
  • Equation (1) p is the probability density function
  • w is the weight of the neural network model
  • x is the input sample
  • y is the output sample (observed value)
  • ⁇ is the observation noise
  • the maximum negative log-likelihood of equation (3) is used as the loss function of the neural network model.
  • the loss function is in is the labeled predicted observation value
  • i is the ith sample data.
  • the corresponding sample data can be selected for model training of the neural network to obtain a pre-trained neural network model.
  • the process of pre-training the neural network model may be: obtaining sample data, where the sample data includes point cloud data of the target object, labeled predicted observation values and predicted observation noise; Initialize to obtain the neural network model to be trained, and train the neural network model to be trained based on the sample data until the trained neural network model converges.
  • the neural network model in the process of training the neural network model, each time the neural network model is trained, the neural network model can output the predicted observation value and predicted observation noise of the target object, and based on the preset loss function, according to the output predicted observation value and predicted observation Noise and the labeled predicted observation value and predicted observation noise, determine the loss value of the neural network model, if the loss value is less than or equal to the preset loss value, it can be determined that the trained neural network model is converged, and then stop training the neural network model, If the loss value is greater than the preset loss value, update the weight of the neural network model, and after updating the weight, continue to train the neural network model.
  • the neural network model After training the neural network model, if the sensor data fusion method is applied to the mobile platform, the neural network model can be stored in the memory of the mobile platform, and of course, the neural network model can also be processed first. Compression processing, such as performing pruning processing, saves the compressed neural network model in the memory of the mobile platform, thereby saving the storage space of the mobile platform.
  • FIG. 3 is a schematic flowchart of steps of a method for data fusion of sensors provided by an embodiment of the present application.
  • the data fusion method is applied to a movable platform, and the platform body of the movable platform is provided with a sensor.
  • the movable platform also stores a pre-trained neural network model in advance, and the neural network model can output the predicted observation value corresponding to the data sensed by the sensor and the predicted observation noise corresponding to the predicted observation value.
  • the data fusion method of the sensor includes steps S101 to S103.
  • the senor includes one of a vision sensor, a radar, an inertial measurement device, a TOF device, and an odometer
  • the radar includes a lidar or a millimeter-wave radar.
  • the sensor is installed on the movable platform, and is used for sensing data of the target (object) around the movable platform.
  • the current data sensed by the sensor includes the point cloud data corresponding to the target object sensed by the radar at the current moment, the angular velocity and acceleration of the movable platform collected by the inertial measurement unit, and the speed of the movable platform collected by the odometer.
  • the target can include cars, pedestrians, obstacles, etc.
  • the pre-trained neural network model is obtained by training the neural network model through a preset loss function.
  • the preset loss function is determined according to a normal distribution relationship between the predicted observation value and the predicted observation noise, and the loss function is determined according to maximizing the log-likelihood of the normal distribution. Therefore, the neural network model can output the predicted observation value of the current data and the predicted observation noise corresponding to the predicted observation value.
  • the neural network model may include a convolutional neural network model (Convolutional Neural Networks, CNN), a recurrent neural network model (Recurrent Neural Network, RNN) and a deep convolutional neural network model (Deep Convolutional Neural Networks, DCNN), of course, can also It includes other network models, which are not specifically limited in this embodiment of the present application.
  • CNN convolutional Neural Networks
  • RNN recurrent neural network model
  • DCNN Deep Convolutional Neural Networks
  • the current data sensed by the sensor is input into the preset trained neural network model, and the current data is learned through the neural network model, and the predicted observation value corresponding to the current data and the corresponding predicted observation value can be obtained. Predict observation noise.
  • the current data includes point cloud data
  • the predicted observation value of the target object and the predicted observation noise corresponding to the predicted observation value can be determined according to the point cloud data through the neural network model, wherein the predicted observation value can describe
  • the state information of the target object can specifically include the length, width, height, three-dimensional position coordinates, the predicted value corresponding to the yaw offset angle, etc.
  • the corresponding predicted observation noise includes the length, width, height, three-dimensional position coordinates, and yaw offset angle. and the corresponding noise value.
  • the predicted observation noise includes the variance or standard deviation corresponding to the predicted observation value.
  • the predicted observation value corresponding to the current data and the predicted observation noise corresponding to the predicted observation value are determined.
  • the current data may be input into the pre-trained A neural network model is used to obtain the predicted observation value and output noise corresponding to the current data; and the predicted observation noise corresponding to the predicted observation value is determined according to the output noise.
  • the output noise is s
  • the predicted observation noise is ⁇
  • the predicted observation noise is ⁇
  • the historical data sensed by the sensor is the data before the current moment, such as the data of the previous moment, of course, it can also be the data corresponding to multiple historical moments before the current moment.
  • the historical observation value may be determined according to the historical data sensed by the sensor; and then the current observation value may be generated according to the historical observation value, the predicted observation value corresponding to the current data, and the predicted observation noise corresponding to the predicted observation value. value.
  • the current observation can be used as the historical observation value at the next moment.
  • the estimated value corresponding to the current moment and the estimated noise corresponding to the estimated value may also be estimated according to historical data sensed by the sensor, and then based on the predicted observation value, predicted observation noise and the estimated value, Estimate noise for data fusion to generate current observations.
  • the estimated value of the state information of the target corresponding to the current moment and the estimated noise corresponding to the estimated value are estimated according to historical data obtained by the sensor sensing the state information of the target at the previous moment.
  • the estimated value includes the length, width, height, three-dimensional position coordinates, yaw offset angle, etc. of the target object, and the estimated noise includes the variance or standard deviation corresponding to the estimated value.
  • the estimated value and estimated noise can be determined using kinematic equations.
  • a kinematic equation corresponding to the target object of the sensor may be determined, and the kinematic equation is used to describe the motion trajectory of the target object; based on the kinematic equation, the estimated value corresponding to the current moment and The estimated noise corresponding to the estimated value.
  • the kinematic equation corresponding to the target can be determined based on the state information of the target at different times before, and can be used to determine the current state information of the target according to the data of the previous moment.
  • the kinematic equation of the target is a linear motion equation, and of course it can also be a curvilinear motion equation. Therefore, based on the linear motion equation, the position information of the current moment (the moment when the sensor senses the data) can be estimated according to the position information of the previous moment, that is, the estimated value (position information) corresponding to the current data and the corresponding estimated noise.
  • data fusion is performed according to the predicted observation value, the predicted observation noise, and the estimated value and the estimated noise to generate the current observation value.
  • a Kalman filter can be used to perform data fusion according to the predicted observation value, predicted observation noise, and the estimated value and estimated noise to generate the current observation value.
  • a correction amount may be determined according to the predicted observation value, the predicted observation noise, the estimated value, and the estimated noise, and a fusion result may be determined according to the estimated value and the correction amount.
  • the correction amount is determined according to the predicted observation value, the predicted observation noise and the estimated value, and the estimated noise, specifically, a correction coefficient is determined according to the predicted observation noise and the estimated noise, and a correction coefficient is determined according to the predicted observation value and the estimated noise. value to determine a difference value; and a correction amount is determined according to the correction coefficient and the difference value.
  • the fusion result (current observation value) of the coordinate value can be expressed as the following formula:
  • y′ represents the fusion result
  • y m represents the estimated value
  • ⁇ m represents the estimated noise
  • y m represents the predicted observation value
  • ⁇ p represents the predicted observation noise
  • the correction amount can be determined according to the predicted observation value, the predicted observation noise, the estimated value and the estimated noise; according to the predicted observation value and the correction amount, the fusion result of the target object is determined, that is, the sum of the observed value and the correction amount is used as The fusion result can be used to describe the state information of the target.
  • the correction amount may be a positive value or a negative value, and the correction amount includes a correction value of at least one of the length, width, height, three-dimensional position coordinates and yaw offset angle of the target.
  • the predicted observation value can be corrected by the correction amount, the target state information of the target object can be obtained, and the accuracy of the target state information of the target object can be improved.
  • the current data sensed by the sensor such as the data used to represent the state information of the target object
  • the current data is acquired, and the current data is input into the neural network model, and the neural network model can output the
  • the predicted observation value and predicted observation noise corresponding to the current data; the estimated value and estimated noise corresponding to the current time are determined by using historical data (data at the previous moment) through kinematic equations; finally, according to the predicted observation value, predicted observation value
  • the noise is fused with the estimated value and the estimated noise to generate the current observation value, that is, the fusion result.
  • the predicted observation noise used is not set based on empirical parameter adjustment or determined according to the kinematic equation at the previous moment, but the noise related to the predicted observation value output by the neural network model, which can truly It reflects the noise of observations, thereby improving the accuracy of data fusion.
  • the last moment is a moment separated from the current moment by a collection period, for example, the current moment is T i , the last moment is T i-1 , and T is the collection period of the sensor, and the specific size is not limited here.
  • the kinematic equation can be used to determine the corresponding estimated value and estimated noise, and for other information describing the target, other fitting equations similar to the kinematic equation can be used to determine.
  • the senor may be a visual sensor, such as a binocular camera, or a combination of a monocular camera and a ranging device, and correspondingly, current depth image data sensed by the visual sensor may be acquired.
  • a visual sensor such as a binocular camera, or a combination of a monocular camera and a ranging device, and correspondingly, current depth image data sensed by the visual sensor may be acquired.
  • the neural network model may be a 3D box model trained based on a convolutional neural network, and the 3D box model is used to output the predicted observation value corresponding to the data sensed by the visual sensor and the corresponding predicted observation value. Predict observation noise.
  • the predicted observation value includes the position observation value (x, y, z) of the target object in the world coordinate system of the vision sensor, the size observation value (l, h, w) of the target object, and the The angle observation value ⁇ of the target object, the size observation value includes the length observation value l, the width observation value w and the height observation value h; correspondingly, the predicted observation noise includes the predicted observation noise corresponding to the position observation value, the size observation value The predicted observation noise corresponding to the observation value and the predicted observation noise corresponding to the angle observation value.
  • the depth image data is the data collected in the image coordinate system, in order to speed up the processing speed and improve the processing efficiency, the depth image data can also be converted into image data in the world coordinate system;
  • the image data in the world coordinate system is input into the pre-trained neural network model, and the predicted observation value corresponding to the image data and the predicted observation noise corresponding to the predicted observation value are obtained.
  • the fusion method provided by the above embodiments can use the neural network model to obtain the observed value and the observed noise of the data, and can truly reflect the observed noise, and then combine the estimated value corresponding to the data and the estimated noise corresponding to the estimated value to perform the data analysis. Fusion, thereby improving the accuracy of data fusion
  • FIG. 5 is a schematic block diagram of a data fusion apparatus provided by an embodiment of the present application.
  • the data fusion apparatus 200 further includes at least one or more processors 201 and a memory 202 .
  • the processor 201 may be, for example, a Micro-controller Unit (MCU, MCU), a Central Processing Unit (Central Processing Unit, CPU), or a Digital Signal Processor (Digital Signal Processor, DSP) or the like.
  • MCU Micro-controller Unit
  • CPU Central Processing Unit
  • DSP Digital Signal Processor
  • the memory 202 may be a Flash chip, a read-only memory (ROM, Read-Only Memory) magnetic disk, an optical disk, a U disk, a mobile hard disk, and the like.
  • ROM Read-Only Memory
  • the memory 202 is used for storing a computer program; the processor 201 is used for executing the computer program and when executing the computer program, executes any one of the sensor data fusion methods provided in the embodiments of the present application, so as to improve the fusion accuracy.
  • the processor is configured to execute the computer program and implement the following steps when executing the computer program:
  • the loss function of the neural network model is determined according to the fact that both the predicted observation value and the predicted observation noise obey a normal distribution, and the loss function is used to compare the predicted observation value and the predicted observation value.
  • the predicted observation noise is correlated.
  • the loss function of the neural network model is obtained by maximizing the log-likelihood of the normal distribution corresponding to the normal distribution.
  • determining the predicted observation value corresponding to the current data and the predicted observation noise corresponding to the predicted observation value based on the pre-trained neural network model includes:
  • the processor is configured to: determine a historical observation value according to historical data sensed by the sensor; according to the historical observation value, the predicted observation value corresponding to the current data, and the corresponding predicted observation value The predicted observation noise of , yields the current observation.
  • the processor is configured to: determine the estimated value corresponding to the current moment and the estimated noise corresponding to the estimated value according to the historical observation value; value and estimated noise for data fusion to generate the current observation value.
  • the processor is used to implement:
  • kinematic equation corresponding to the target object of the sensor Determine the kinematic equation corresponding to the target object of the sensor, and the kinematic equation is used to describe the motion trajectory of the target object; based on the kinematic equation, determine the estimation corresponding to the current moment according to the historical observation value value and the estimated noise corresponding to the estimated value.
  • the kinematic equations include linear motion equations or curvilinear motion equations.
  • the processor is configured to: utilize a Kalman filter to perform data fusion according to the predicted observation value, the predicted observation noise, and the estimated value and the estimated noise to generate the current observation value.
  • performing data fusion according to the predicted observation value, predicted observation noise, and the estimated value and estimated noise to generate the current observation value includes:
  • a correction amount is determined according to the predicted observation value, the predicted observation noise and the estimated value, and the estimated noise, and a fusion result is determined according to the estimated value and the correction amount to obtain the current observation value.
  • the determining the correction amount according to the predicted observation value, the predicted observation noise and the estimated value and the estimated noise includes:
  • a correction coefficient is determined according to the predicted observation noise and the estimated noise, a difference value is determined according to the predicted observation value and the estimated value; and a correction amount is determined according to the correction coefficient and the difference value.
  • the senor includes one of a vision sensor, a radar, an inertial measurement device, a TOF device.
  • the predicted observation noise includes a variance or standard deviation corresponding to the predicted observation value
  • the estimated noise includes a variance or standard deviation corresponding to the estimated value
  • the acquiring the current data sensed by the sensor includes: acquiring the current depth image data sensed by the vision sensor.
  • the neural network model is a 3D box model trained based on a convolutional neural network, and the 3D box model is used to output the predicted observation value corresponding to the current depth image data sensed by the visual sensor and the The predicted observation noise corresponding to the predicted observation.
  • the predicted observation value includes a position observation value of the target object of the vision sensor in the world coordinate system, a size observation value of the target object, and an angle observation value of the target object
  • the size observation value Values include length observations, width observations, and height observations
  • the predicted observation noise includes predicted observation noise corresponding to the position observation value, predicted observation noise corresponding to the size observation value, and predicted observation noise corresponding to the angle observation value.
  • the current depth image data is data collected in an image coordinate system
  • the processor is configured to:
  • determining the predicted observation value corresponding to the current depth image data and the predicted observation noise corresponding to the predicted observation value based on the pre-trained neural network model includes: converting the image data in the world coordinate system Input to the pre-trained neural network model to obtain the predicted observation value corresponding to the image data and the predicted observation noise corresponding to the predicted observation value.
  • FIG. 6 is a schematic block diagram of a movable platform provided by an embodiment of the present application.
  • the movable platform further includes at least one or more processors 301 , a memory 302 and a sensor 303 , and the sensor 303 is arranged on the platform body of the movable platform.
  • the processor 301 may be, for example, a micro-controller unit (Micro-controller Unit, MCU), a central processing unit (Central Processing Unit, CPU), or a digital signal processor (Digital Signal Processor, DSP) or the like.
  • MCU Micro-controller Unit
  • CPU Central Processing Unit
  • DSP Digital Signal Processor
  • the memory 302 may be a Flash chip, a read-only memory (ROM, Read-Only Memory) magnetic disk, an optical disk, a U disk, or a removable hard disk, and the like.
  • ROM Read-Only Memory
  • the memory 302 is used for storing a computer program; the processor 301 is used for executing the computer program and when executing the computer program, executes any one of the sensor data fusion methods provided in the embodiments of this application, so as to improve the Sensor data fusion results.
  • the processor is configured to execute the computer program and implement the following steps when executing the computer program:
  • the loss function of the neural network model is determined according to the normal distribution of the predicted observation value and the predicted observation noise, and the loss function is used to compare the predicted observation value and the predicted observation value.
  • the predicted observation noise is correlated.
  • the loss function of the neural network model is obtained by maximizing the log-likelihood of the normal distribution corresponding to the normal distribution.
  • determining the predicted observation value corresponding to the current data and the predicted observation noise corresponding to the predicted observation value based on the pre-trained neural network model including:
  • the processor is configured to implement: determining a historical observation value according to historical data sensed by the sensor; according to the historical observation value, the predicted observation value corresponding to the current data, and the corresponding predicted observation value The predicted observation noise of , yields the current observation.
  • the processor is configured to: determine the estimated value corresponding to the current moment and the estimated noise corresponding to the estimated value according to the historical observation value; value and estimated noise for data fusion to generate the current observation value.
  • the processor is used to implement:
  • kinematic equation corresponding to the target object of the sensor Determine the kinematic equation corresponding to the target object of the sensor, and the kinematic equation is used to describe the motion trajectory of the target object; based on the kinematic equation, determine the estimation corresponding to the current moment according to the historical observation value value and the estimated noise corresponding to the estimated value.
  • the kinematic equations include linear motion equations or curvilinear motion equations.
  • the processor is configured to: utilize a Kalman filter to perform data fusion according to the predicted observation value, the predicted observation noise, and the estimated value and the estimated noise to generate the current observation value.
  • performing data fusion according to the predicted observation value, predicted observation noise, and the estimated value and estimated noise to generate the current observation value includes:
  • a correction amount is determined according to the predicted observation value, the predicted observation noise and the estimated value, and the estimated noise, and a fusion result is determined according to the estimated value and the correction amount to obtain the current observation value.
  • the determining the correction amount according to the predicted observation value, the predicted observation noise and the estimated value and the estimated noise includes:
  • a correction coefficient is determined according to the predicted observation noise and the estimated noise, a difference value is determined according to the predicted observation value and the estimated value; and a correction amount is determined according to the correction coefficient and the difference value.
  • the senor includes one of a vision sensor, a radar, an inertial measurement device, a TOF device.
  • the predicted observation noise includes a variance or standard deviation corresponding to the predicted observation value
  • the estimated noise includes a variance or standard deviation corresponding to the estimated value
  • the acquiring the current data sensed by the sensor includes: acquiring the current depth image data sensed by the vision sensor.
  • the neural network model is a 3D box model trained based on a convolutional neural network, and the 3D box model is used to output the predicted observation value corresponding to the current depth image data sensed by the visual sensor and the The predicted observation noise corresponding to the predicted observation.
  • the predicted observation value includes a position observation value of the target object of the vision sensor in the world coordinate system, a size observation value of the target object, and an angle observation value of the target object
  • the size observation value Values include length observations, width observations, and height observations
  • the predicted observation noise includes predicted observation noise corresponding to the position observation value, predicted observation noise corresponding to the size observation value, and predicted observation noise corresponding to the angle observation value.
  • the current depth image data is data collected in an image coordinate system
  • the processor is configured to:
  • determining the predicted observation value corresponding to the current depth image data and the predicted observation noise corresponding to the predicted observation value based on the pre-trained neural network model includes: converting the image data in the world coordinate system Input to the pre-trained neural network model to obtain the predicted observation value corresponding to the image data and the predicted observation noise corresponding to the predicted observation value.
  • the movable platform includes one of an aircraft, a robot, and an unmanned vehicle.
  • FIG. 7 is a schematic block diagram of a control terminal provided by an embodiment of the present application.
  • the control terminal 400 includes at least one or more processors 401 and a memory 402 , and the control terminal 400 is connected to the movable platform in communication for controlling the movement of the movable platform.
  • the processor 401 may be, for example, a Micro-controller Unit (MCU), a Central Processing Unit (CPU), a Digital Signal Processor (DSP), or the like.
  • MCU Micro-controller Unit
  • CPU Central Processing Unit
  • DSP Digital Signal Processor
  • the memory 402 may be a Flash chip, a read-only memory (ROM, Read-Only Memory) magnetic disk, an optical disk, a U disk, a mobile hard disk, and the like.
  • ROM Read-Only Memory
  • the memory 402 is used for storing a computer program; the processor 401 is used for executing the computer program, and when executing the computer program, executes any one of the sensor data fusion methods provided in the embodiments of this application, so as to improve the Sensor data fusion results.
  • Embodiments of the present application further provide a computer-readable storage medium, where the computer-readable storage medium stores a computer program, the computer program includes program instructions, and the processor executes the program instructions to implement the above implementation The steps of any one of the sensor data fusion methods provided in the example.
  • the computer-readable storage medium may be an internal storage unit of the movable platform described in any of the foregoing embodiments, such as a memory or internal memory of the movable platform.
  • the computer-readable storage medium can also be an external storage device of the removable platform, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital) equipped on the removable platform , SD) card, flash memory card (Flash Card), etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Traffic Control Systems (AREA)

Abstract

一种传感器的数据融合方法、数据融合装置、可移动平台、控制终端以及存储介质，其中，该方法包括：获取传感器感测的当前数据(S101)；基于预先训练好的神经网络模型，确定所述当前数据对应的预测观测值和所述预测观测值对应的预测观测噪声(S102)；根据所述传感器感测的历史数据、所述当前数据对应的预测观测值和所述预测观测值对应的预测观测噪声，生成当前观测值(S103)。

Description

传感器的数据融合方法、设备及存储介质 技术领域
本申请涉及数据处理技术领域，尤其涉及一种传感器的数据融合方法、数据融合装置、可移动平台、控制终端以及存储介质。
背景技术
对于传感器数据融合，比如视觉传感器，一般使用卡尔曼(kalman)滤波器通过融合当前时刻的预测观测值与上一时刻的历史观测值，得到当前时刻的观测值，融合预测观测值和历史观测值需要使用当前时刻观测的标准差(噪声)。目前，当前时刻观测的标准差是根据经验调参设置或者是根据上一时刻的运动学方程确定的，然而利用深度学习进行观测得到当前时刻的预测观测值进行数据融合时，若标准差还采用经验调参设置或者上一时刻的运动学方程确定，不能真正的反应观测的噪声，因此会降低数据融合的准确度。
发明内容
本申请实施例供了一种传感器的数据融合方法、数据融合装置、可移动平台、控制终端以及存储介质，以提高传感器数据融合的准确性。
第一方面，本申请实施例提供了一种传感器的数据融合方法，所述数据融合方法包括：
获取传感器感测的当前数据；
基于预先训练好的神经网络模型，确定所述当前数据对应的预测观测值和所述预测观测值对应的预测观测噪声；
根据所述传感器感测的历史数据、所述当前数据对应的预测观测值和所述预测观测值对应的预测观测噪声，生成当前观测值。
第二方面，本申请实施例还提供了一种数据融合装置，所述数据融合装置包括处理器和存储器；
所述存储器用于存储计算机程序；
所述处理器，用于执行所述计算机程序并在执行所述计算机程序时，实现：
获取传感器感测的当前数据；
基于预先训练好的神经网络模型，确定所述当前数据对应的预测观测值和所述预测观测值对应的预测观测噪声；
根据所述传感器感测的历史数据、所述当前数据对应的预测观测值和所述预测观测值对应的预测观测噪声，生成当前观测值。
第三方面，本申请实施例还提供了一种可移动平台，所述可移动平台包括：
平台本体；
传感器，所述传感器安装在所述平台本体上，用于数据；
处理器和存储器；
其中，所述存储器用于存储计算机程序；所述处理器，用于执行所述计算机程序并在执行所述计算机程序时，实现如下步骤：
获取传感器感测的当前数据；
基于预先训练好的神经网络模型，确定所述当前数据对应的预测观测值和所述预测观测值对应的预测观测噪声；
根据所述传感器感测的历史数据、所述当前数据对应的预测观测值和所述预测观测值对应的预测观测噪声，生成当前观测值。
第四方面、本申请实施例还提供了一种控制终端，所述控制终端与可移动平台通信连接，所述可移动平台包括处理器和存储器；
所述存储器用于存储计算机程序；
所述处理器，用于执行所述计算机程序并在执行所述计算机程序时，实现：
获取传感器感测的当前数据；
基于预先训练好的神经网络模型，确定所述当前数据对应的预测观测值和所述预测观测值对应的预测观测噪声；
根据所述传感器感测的历史数据、所述当前数据对应的预测观测值和所述预测观测值对应的预测观测噪声，生成当前观测值。
第五方面，本申请实施例还提供了一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，所述计算机程序被处理器执行时使所述处理器实现如本申请实施例提供的任一项所述的数据融合方法方法的步骤。
本申请实施例公开的传感器的数据融合方法、数据融合装置、可移动平台、控制终端以及存储介质，对于传感器感测的当前时刻的当前数据，能够利用神经网络模型得到当前数据的预测观测值和预测观测噪声，再结合该传感器采集的历史数据，可以得到当前观测值，由此提高了数据融合的准确性。
应当理解的是，以上的一般描述和后文的细节描述仅是示例性和解释性的，并不能限制本申请。
附图说明
为了更清楚地说明本申请实施例技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1是本申请实施例提供的一种无人机的结构示意图；
图2是本申请实施例提供的无人机的飞行控制系统的示意性框图；
图3是本申请实施例提供的一种传感器的数据融合方法的步骤示意流程图；
图4是本申请实施例提供的传感器的数据融合过程的示意图；
图5是本申请实施例提供的一种数据融合装置的示意框图；
图6是本申请实施例提供的一种可移动平台的示意框图；
图7是本申请实施例提供的一种控制终端的示意框图。
具体实施方式
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
还应当理解，在此本申请说明书中所使用的术语仅仅是出于描述特定实施例的目的而并不意在限制本申请。如在本申请说明书和所附权利要求书中所使用的那样，除非上下文清楚地指明其它情况，否则单数形式的“一”、“一个”及“该”意在包括复数形式。
还应当进一步理解，在本申请说明书和所附权利要求书中使用的术语“和/或”是指相关联列出的项中的一个或多个的任何组合以及所有可能组合，并且包括这些组合。
附图中所示的流程图仅是示例说明，不是必须包括所有的内容和操作/步骤，也不是必须按所描述的顺序执行。例如，有的操作/步骤还可以分解、组合或部分合并，因此实际执行的顺序有可能根据实际情况改变。
目前，为了提高传感器感测的数据的准确性，需要对传感器感测的数据进行融合，比如视觉传感器，一般使用卡尔曼(kalman)滤波器通过融合当前时刻的预测观测值与上一时刻的历史观测值，得到当前观测值，融合预测观测值和历史观测值需要使用当前时刻的预测观测噪声(标准差)。当前时刻观测的标准差是根据经验调参设置或者是根据上一时刻的运动学方程确定的，然而利用深度学习进行观测，若还采用经验调参设置或者上一时刻的运动学方程确定，不能真正的反应观测的噪声，因此会降低数据融合的准确度。
示例性的，比如应用在自动驾驶领域的传感器，具体应用于可移动平台，可移动平台需要根据周围目标物的速度、目标物与可移动平台之间的距离、目标物相对于可移动平台的位置等状态信息来行驶。可移动平台主要是通过传感器(视觉传感器、雷达或TOF等)采集其周围的点云数据，通过点云数据确定目标物在当前时刻的状态信息的预测观测值，之后对状态信息的预测观测值与目标物在上一时刻的状态信息的估计值进行融合，得到目标物在当前时刻的状态信息(当前观测值)。
然而，融合状态信息的预测观测值和状态信息的估计值需要使用当前观测到的预测观测噪声和上一时刻预测到的估计噪声，但预测观测噪声是根据经验调参设置或者是根据上一时刻的运动学方程确定的，然而利用深度学习进行观测，若还采用经验调参设置或者上一时刻的运动学方程确定预测观测噪声，不能真正的反应观测的噪声，因此会降低目标物的状态信息的准确度，无法保证可移动平台的行驶安全。
为解决上述问题，本申请实施例提供了一种传感器的数据融合方法、数据融合装置、可移动平台以及存储介质，通过获取传感器感测的当前数据；基于预先训练好的神经网络模型，确定所述当前数据对应的预测观测值和所述预测观测值对应的预测观测噪声；并根据所述传感器感测的历史数据、所述当前数 据对应的预测观测值和所述预测观测值对应的预测观测噪声，生成当前观测值，该当前观测值即为融合数据。由此能够提高数据融合的准确度。
下面结合附图，对本申请的一些实施方式作详细说明。在不冲突的情况下，下述的实施例及实施例中的特征可以相互组合。
需要说明的是，本申请实施例提供的传感器的数据融合方法，可以应用于数据融合装置中，或者也可以应用于可移动平台或者控制终端中。其中，可移动平台包括飞行器、机器人和无人驾驶车辆中的至少一项，飞行器包括无人飞行器和载人飞行器；控制终端包括遥控器、智能手机、平板电脑、笔记本电脑、台式电脑或者可穿戴设备等电子设备。
其中，所述传感器包括视觉传感器、雷达、惯性测量装置、TOF装置和里程计中的一种，视觉传感器包括摄像头。以下将以可移动平台为无人机为例，以及以传感器为视觉传感器或雷达为例进行介绍。
请参阅图1和图2，图1示出了本申请实施例提供的一种无人机100的结构，图2示出了本申请实施例提供的无人机100的飞行控制系统的结构框架。如图1和图2所示，无人机100可以包括机架10、动力系统11、控制系统12和雷达20。
机架10可以包括机身和脚架(也称为起落架)。机身可以包括中心架以及与中心架连接的一个或多个机臂，一个或多个机臂呈辐射状从中心架延伸出。脚架与机身连接，用于在无人机100着陆时起支撑作用。
雷达20，可以安装在无人机上，具体可以安装在无人机100的机架10上，在无人机100的飞行过程中，用于测量无人机100的周围环境，比如障碍物等，以确保飞行的安全性。在本申请的实施例中，雷达20用于对无人机100周围的目标物进行扫描测量得到相应的数据，以确定目标物的状态信息。
雷达20安装在无人机100的脚架上，该雷达20与控制系统12通信连接，雷达20将采集到的数据传输至控制系统12，由控制系统12进行处理。
需要说明的是，无人机100可以包括两个或两个以上脚架，雷达20搭载在其中一个脚架上。雷达20也可以搭载在无人机100的其他位置，对此不作具体限定。
雷达20主要包括射频前端模块和信号处理模块，射频前端模块可以包括发射天线和接收天线，发射天线用于向目标物发送信号，接收天线用于接收被目 标物反射回来的信号，信号处理模块负责产生调制信号以及对采集的中频信号进行处理分析，其中目标物比如为建筑物、车辆、行人、山体、建筑物或其他无人机等。
动力系统11可以包括一个或多个电子调速器(简称为电调)、一个或多个螺旋桨以及与一个或多个螺旋桨相对应的一个或多个电机，其中电机连接在电子调速器与螺旋桨之间，电机和螺旋桨设置在无人机100的机臂上；电子调速器用于接收控制系统产生的驱动信号，并根据驱动信号提供驱动电流给电机，以控制电机的转速。
电机用于驱动螺旋桨旋转，从而为无人机100的飞行提供动力，该动力使得无人机100能够实现一个或多个自由度的运动。在某些实施例中，无人机100可以围绕一个或多个旋转轴旋转。例如，上述旋转轴可以包括横滚轴、偏航轴和俯仰轴。应理解，电机可以是直流电机，也可以是永磁同步电机。或者，电机可以是无刷电机，也可以是有刷电机。
控制系统12可以包括控制器和传感系统。控制器用于控制无人机100的飞行，例如，可以根据传感系统测量的姿态信息控制无人机100的飞行。应理解，控制器可以按照预先编好的程序指令对无人机100进行控制。传感系统用于测量无人机100的姿态信息，即无人机100在空间的位置信息和状态信息，例如，三维位置、三维角度、三维速度、三维加速度和三维角速度等。
传感系统例如可以包括陀螺仪、超声传感器、电子罗盘、惯性测量单元(Inertial Measurement Unit，IMU)、视觉传感器、全球导航卫星系统和气压计等传感器中的至少一种。例如，全球导航卫星系统可以是全球定位系统(Global Positioning System，GPS)。需要说明的是，传感系统中的传感器也可以用本申请实施例提供的传感器的数据融合方法，以提高数据的准确性。以便控制系统12根据融合后的数据进行飞行控制，由此提供了飞行的安全性。
控制器可以包括一个或多个处理器和存储器。处理器例如可以是微控制单元(Micro-controller Unit，MCU)、中央处理单元(Central Processing Unit，CPU)或数字信号处理器(Digital Signal Processor，DSP)等。存储器可以是Flash芯片、只读存储器(ROM，Read-Only Memory)磁盘、光盘、U盘或移动硬盘等。
无人机100可以包括旋翼型无人机，例如四旋翼无人机、六旋翼无人机、 八旋翼无人机，也可以是固定翼无人机，还可以是旋翼型与固定翼无人机的组合，在此不作限定。
本申请实施例还提供了一种飞行系统，该飞行系统包括无人机100和控制终端。控制终端为位于飞行系统的地面端，可以通过无线方式与无人机100进行通信，用于对无人机100进行远程操纵。
应理解，上述对于无人机100各组成部分的命名仅是出于标识的目的，并不应理解为对本说明书的实施例的限制。该无人机100，具体为无人机100的控制系统的控制器，可以用于执行本申请实施例提供的任一项所述的传感器的数据融合方法，以提高数据融合的准确性，进而确保无人机飞行的安全性。
示例性的，比如控制器用于：获取传感器感测的当前数据；基于预先训练好的神经网络模型，确定所述当前数据对应的预测观测值和所述预测观测值对应的预测观测噪声；根据所述传感器感测的历史数据、所述当前数据对应的预测观测值和所述预测观测值对应的预测观测噪声，生成当前观测值，即融合结果。比如，当前观测值为目标物的当前时刻的状态信息，由此无人机根据融合结果确定目标物的状态信息，再利用该状态信息进行避障或者规划飞行路径等，由此可以提高飞行的安全性。
还应理解，飞行系统的控制终端也可以用于执行本申请实施例提供的任一项所述的传感器的数据融合方法，以提高无人机飞行的安全性。
示例性的，比如控制终端用于：获取传感器(比如无人机搭载的视觉传感器)感测的当前数据；基于预先训练好的神经网络模型，确定所述当前数据对应的预测观测值和所述预测观测值对应的预测观测噪声；根据所述传感器感测的历史数据、所述当前数据对应的预测观测值和所述预测观测值对应的预测观测噪声，生成当前观测值，即融合结果。以便根据融合结果确定目标物的状态信息，并将目标物的状态信息发送给无人机，以使无人机利用该状态信息进行避障或者规划飞行路径等，由此可以提高飞行的安全性。
由于本申请实施例提供的传感器的数据融合方法，均使用了预先训练好的神经网络模型实现的，该神经网络模型可以得到传感器感测到的当前数据对应的预测观测值以及所述预测观测值对应的预测观测噪声，可以真实地反应观测的噪声，由此可以提高数据融合的准确性。
为了便于理解，在介绍传感器的数据融合方法之前，先介绍本申请实施例 提供的预先训练好的神经网络模型。在本申请的实施例中，通过设置神经网络模型的损失函数，使得该神经网络模型能够输出传感器感测的当前数据对应的预测观测值以及所述预测观测值对应的预测观测噪声。
具体地，该神经网络模型的损失函数是根据预测观测值和预测观测噪声均服从正态分布确定的，所述损失函数用于将所述预测观测值和所述预测观测噪声进行相关。由此可以实现神经网络模型可以同时输出所述当前数据对应的预测观测值及观测噪声。
具体地，在所述预测观测值和所述预测观测噪声均服从正态分布时，通过最大化所述正态分布对应的正态分布式的对数似然，即可以得到该神经网络模型的损失函数。
示例性的，由于所述预测观测值和预测观测噪声服从正态分布，即所述预测观测值和预测观测噪声之间的正态分布式为：
p(y∣fw(x))＝N(fw(x),σ 2)    (1)
在式(1)中，p为概率密度函数，w为神经网络模型的权重，x为输入样本，y为输出样本(观测值)，σ为观测噪声，因为正态分布式取对数单调性不变，对式(1)进行最大化的对数似然可以得到下式：
Figure PCTCN2020136622-appb-000001
在式(2)中，σ为预测观测噪声，对数函数自变量为正数，则令s＝logσ 2，因此式(2)可以变形为：
Figure PCTCN2020136622-appb-000002
由于要求神经网络的损失函数越小越好，则将式(3)的最大负对数似然作为神经网络模型的损失函数。
例如，损失函数为
Figure PCTCN2020136622-appb-000003
其中，
Figure PCTCN2020136622-appb-000004
为标注的预测观测值，i为第i个样本数据。
在设定好神经网络的损失函数后，可选择相应的样本数据对应该神经网络 进行模型训练，以得到预先训练好的神经网络模型。
示例性的，对神经网络模型进行预训练的过程可以为：获取样本数据，其中，该样本数据包括目标物的点云数据、标注的预测观测值和预测观测噪声；对神经网络模型的权重进行初始化，得到待训练的神经网络模型，并基于该样本数据对该待训练的神经网络模型进行训练，直至训练后的神经网络模型收敛。其中，在训练神经网络模型的过程中，每次训练神经网络模型时，神经网络模型能够输出目标物的预测观测值和预测观测噪声，并基于预设损失函数根据输出的预测观测值和预测观测噪声以及标注的预测观测值和预测观测噪声，确定神经网络模型的损失值，如果该损失值小于或等于预设损失值，则可以确定训练后的神经网络模型收敛，进而停止训练神经网络模型，如果该损失值大于预设损失值，则更新神经网络模型的权重，并在更新权重后，继续训练神经网络模型。
在训练完神经网络模型之后，若将传感器的数据融合方法应用在可移动平台上，则可以将该神经网络模型保存在所述可移动平台的存储器中，当然也可以先对该神经网络模型进行压缩处理，比如进行剪枝处理，在将压缩处理后的神经网络模型保存在所述可移动平台的存储器中，由此可以节省可移动平台的存储空间。
请参阅图3，图3是本申请实施例提供的一种传感器的数据融合方法的步骤示意流程图。在本申请的实施例中，该数据融合方法应用于可移动平台中，该可移动平台的平台本体设有传感器。
其中，该可移动平台中还预先保存有预先训练好的神经网络模型，该神经网络模型能够输出传感器感测的数据对应的预测观测值和所述预测观测值对应的预测观测噪声。
如图3所示，该传感器的数据融合方法包括步骤S101至步骤S103。
S101、获取传感器感测的当前数据。
在本申请的实施例中，所述传感器包括视觉传感器、雷达、惯性测量装置、TOF装置和里程计中的一种，所述雷达包括激光雷达或毫米波雷达。传感器安装在可移动平台上，用于感测可移动平台周围目标(目标物)的数据。
示例性的，传感器感测的当前数据包括雷达感测到的目标物在当前时刻对应的点云数据、惯性测量单元采集到的可移动平台的角速度和加速度、里程计 采集到的可移动平台的里程等，目标物可以包括汽车、行人、障碍物等。
S102、基于预先训练好的神经网络模型，确定所述当前数据对应的预测观测值和所述预测观测值对应的预测观测噪声。
在本申请的实施例中，预先训练好的神经网络模型是通过预设损失函数对神经网络模型进行训练得到的。所述预设损失函数是根据预测观测值和预测观测噪声之间的正态分布关系式确定的，所述损失函数是根据最大化所述正态分布式的对数似然确定的。从而使得该神经网络模型可以输出当前数据的预测观测值和所述预测观测值对应的预测观测噪声。
其中，神经网络模型可以包括卷积神经网络模型(Convolutional Neural Networks，CNN)、循环神经网络模型(Recurrent Neural Network，RNN)和深度卷积神经网络模型(Deep Convolutional Neural Networks，DCNN)，当然还可以包括其他网络模型，本申请实施例对此不做具体限定。
具体地，将传感器感测到的当前数据输入至预设训练好的神经网络模型，通过神经网络模型对当前数据进行学习，可以得到该当前数据对应的预测观测值以及所述预测观测值对应的预测观测噪声。
示例性的，比如所述当前数据包括点云数据，可以根据该点云数据通过神经网络模型确定目标物的预测观测值以及所述预测观测值对应的预测观测噪声，其中该预测观测值可以描述目标物的状态信息，具体可以包括长度、宽度、高度、三维位置坐标、yaw偏移角等对应的预测值等，相应的预测观测噪声包括长度、宽度、高度、三维位置坐标、yaw偏移角等对应的噪声值。具体地，所述预测观测噪声包括所述预测观测值对应的方差或标准差。
在一些实施例中，基于预先训练好的神经网络模型，确定所述当前数据对应的预测观测值和所述预测观测值对应的预测观测噪声，具体可以将所述当前数据输入至预先训练好的神经网络模型，得到所述当前数据对应的预测观测值和输出噪声；根据所述输出噪声确定所述预测观测值对应的预测观测噪声。
具体地，输出噪声为s，预测观测噪声为σ，其中输出噪声和预测观测噪声满足s＝logσ 2，由此可以根据所述输出噪声确定所述预测观测值对应的预测观测噪声。
S103、根据所述传感器感测的历史数据、所述当前数据对应的预测观测值和所述预测观测值对应的预测观测噪声，生成当前观测值。
其中，传感器感测到的历史数据为当前时刻之前的数据，比如为上一时刻的数据，当然也可以为当前时刻之前多个历史时刻对应的数据。
具体地，可以根据所述传感器感测的历史数据确定历史观测值；再根据所述历史观测值、所述当前数据对应的预测观测值和所述预测观测值对应的预测观测噪声，生成当前观测值。
示例性的，比如用上一时刻的历史数据对应的观测值作为历史观测值，根据所述历史观测值、所述当前数据对应的预测观测值和所述预测观测值对应的预测观测噪声，生成当前观测值。其中，该当前观测值可以作为下一时刻的历史观测值。
在一些实施例中，还可以根据传感器感测到历史数据，估计当前时刻对应的估计值和所述估计值对应的估计噪声，再根据所述预测观测值、预测观测噪声和所述估计值、估计噪声进行数据融合，生成当前观测值。
比如，根据传感器在上一时刻对目标物的状态信息进行感测而得到的历史数据，来估计所述当期时刻对应的目标物的状态信息的估计值和所述估计值对应的估计噪声。
其中，所述估计值包括目标物的长度、宽度、高度、三维位置坐标、yaw偏移角等值，估计噪声包括估计值对应的方差或标准差。
在一些实施例中，若目标物为运动目标，可以利用运动学方程确定估计值和估计噪声。具体地，可以确定所述传感器的目标物对应的运动学方程，所述运动学方程用于描述所述目标物的运动轨迹；基于所述运动学方程，确定所述当前时刻对应的估计值和所述估计值对应的估计噪声。其中，目标物对应的运动学方程可以基于目标物在之前不同时刻的状态信息确定，可以用于根据上一时刻的数据确定当前目标物的状态信息。
示例性的，比如目标物的运动学方程为直线运动方程，当然也可以为曲线运动方程。由此可以基于该直线运动方程根据上一时刻的位置信息估计当前时刻(传感器感测到数据的时刻)的位置信息，即所述当前数据对应的估计值(位置信息)以及对应的估计噪声。
在一些实施例中，根据所述预测观测值、预测观测噪声和所述估计值、估计噪声进行数据融合，生成当前观测值。具体可以利用Kalman滤波器，根据所述预测观测值、预测观测噪声和所述估计值、估计噪声进行数据融合，生成 当前观测值。
在一些实施例中，可以先根据所述预测观测值、预测观测噪声和所述估计值、估计噪声确定修正量，以及根据所述估计值和所述修正量确定融合结果。
在根据所述预测观测值、预测观测噪声和所述估计值、估计噪声确定修正量，具体为根据所述预测观测噪声和所述估计噪声确定修正系数，根据所述预测观测值和所述估计值确定差异值；以及根据所述修正系数和所述差异值确定修正量。
示例性的，以三维位置坐标中坐标值y值为例进行介绍，该坐标值的融合结果(当前观测值)可以表示为下式：
Figure PCTCN2020136622-appb-000005
在式(4)中，y′表示融合结果，y m表示估计值，σ m表示估计噪声，y m表示预测观测值，σ p表示预测观测噪声，
Figure PCTCN2020136622-appb-000006
表示修正系数，
Figure PCTCN2020136622-appb-000007
表示修正量。
示例性的，可以根据预测观测值、预测观测噪声、估计值和估计噪声，确定修正量；根据该预测观测值和修正量，确定目标物的融合结果，即将该观测值与修正量之和作为融合结果，可以用于描述目标物的状态信息。其中，修正量可以为正值，也可以负值，修正量包括目标物的长度、宽度、高度、三维位置坐标和yaw偏移角中的至少一项的修正值。通过修正量可以对预测观测值进行修正，可以得到目标物的目标状态信息，提高了目标物的目标状态信息的准确性。
示例性的，如图4所示，获取传感器感测到的当前数据，比如用于表示目标物的状态信息的数据，并将该当前数据输入至神经网络模型，该神经网络模型能够输出所述当前数据对应的预测观测值以及预测观测噪声；在利用历史数据(上一时刻的数据)通过运动学方程确定所述当前时刻对应的估计值以及估计噪声；最后根据所述预测观测值、预测观测噪声和所述估计值、估计噪声进行数据融合，生成当前观测值，即融合结果。由于该数据融合方式，使用的预 测观测噪声不是根据经验调参设置的或者是根据上一时刻的运动学方程确定的，而是通过神经网络模型输出的与预测观测值相关的噪声，可以真实地反应观测的噪声，由此提高了数据融合的准确率。
需要说明的是，上一时刻为与当前时刻间隔一个采集周期的时刻，比如当前时刻为T i，上一时刻为T i-1，T为传感器的采集周期，具体大小在此不做限定。对于描述目标物的状态信息，可以用运动学方程确定对应的估计值和估计噪声，对于描述目标物的其他信息，则可以利用类似与运动学方程的其他拟合方程进行确定。
在一些实施例中，所述传感器可以为视觉传感器，比如为双目摄像头，或者单目摄像头与测距装置的组合，相应地可以获取视觉传感器感测的当前深度图像数据。
相应地，所述神经网络模型可以为基于卷积神经网络训练的3D box模型，所述3D box模型用于输出所述视觉传感器感测的数据对应的预测观测值和所述预测观测值对应的预测观测噪声。
具体地，所述预测观测值包括所述视觉传感器的目标物在世界坐标系的位置观测值(x,y,z)、所述目标物的尺寸观测值(l,h,w)和所述目标物的角度观测值θ，所述尺寸观测值包括长度观测值l、宽度观测值w和高度观测值h；相应地，所述预测观测噪声包括位置观测值对应的预测观测噪声、所述尺寸观测值对应的预测观测噪声和所述角度观测值对应的预测观测噪声。
需要说明的是，由于所述深度图像数据为在图像坐标系下采集的数据，为了加快处理速度提高处理效率，还可以将所述深度图像数据转换为世界坐标系下的图像数据；再将所述世界坐标系下的图像数据输入至预先训练好的神经网络模型，得到所述图像数据对应的预测观测值和所述预测观测值对应的预测观测噪声。
上述各实施例提供的融合方法，能够利用神经网络模型得到数据的观测值和观测噪声，可以真正地反应的观测的噪声，再结合数据对应的估计值和所述估计值对应的估计噪声进行数据融合，由此提高了数据融合的准确性
请参阅图5，图5是本申请实施例提供的一种数据融合装置的示意性框图。如图5所示，该数据融合装置200还至少包括一个或多个处理器201和存储器202。
其中，处理器201例如可以是微控制单元(Micro-controller Unit，MCU)、中央处理单元(Central Processing Unit，CPU)或数字信号处理器(Digital Signal Processor，DSP)等。
存储器202可以是Flash芯片、只读存储器(ROM，Read-Only Memory)磁盘、光盘、U盘或移动硬盘等。
其中，存储器202用于存储计算机程序；处理器201用于执行所述计算机程序并在执行所述计算机程序时，执行本申请实施例提供的任一项所述的传感器的数据融合方法，以提高融合的准确性。
示例性的，所述处理器，用于执行所述计算机程序并在执行所述计算机程序时，实现如下步骤：
获取传感器感测的当前数据；基于预先训练好的神经网络模型，确定所述当前数据对应的预测观测值和所述预测观测值对应的预测观测噪声；根据所述传感器感测的历史数据、所述当前数据对应的预测观测值和所述预测观测值对应的预测观测噪声，生成当前观测值。
在一些实施例中，所述神经网络模型的损失函数是根据所述预测观测值和所述预测观测噪声均服从正态分布确定的，所述损失函数用于将所述预测观测值和所述预测观测噪声进行相关。
在一些实施例中，所述神经网络模型的损失函数为最大化所述正态分布对应的正态分布式的对数似然得到的。
在一些实施例中，所述基于预先训练好的神经网络模型，确定所述当前数据对应的预测观测值和所述预测观测值对应的预测观测噪声，包括：
将所述当前数据输入至预先训练好的神经网络模型，得到所述当前数据对应的预测观测值和输出噪声；根据所述输出噪声确定所述预测观测值对应的预测观测噪声。
在一些实施例中，所述处理器用于实现：根据所述传感器感测的历史数据确定历史观测值；根据所述历史观测值、所述当前数据对应的预测观测值和所述预测观测值对应的预测观测噪声，生成当前观测值。
在一些实施例中，所述处理器用于实现：根据所述历史观测值确定当前时刻对应的估计值和所述估计值对应的估计噪声；根据所述预测观测值、预测观测噪声和所述估计值、估计噪声进行数据融合，生成当前观测值。
在一些实施例中，所述处理器用于实现：
确定所述传感器的目标物对应的运动学方程，所述运动学方程用于描述所述目标物的运动轨迹；基于所述运动学方程，根据所述历史观测值确定所述当前时刻对应的估计值和所述估计值对应的估计噪声。
在一些实施例中，所述运动学方程包括直线运动方程或曲线运动方程。
在一些实施例中，所述处理器用于实现：利用Kalman滤波器，根据所述预测观测值、预测观测噪声和所述估计值、估计噪声进行数据融合，生成当前观测值。
在一些实施例中，所述根据所述预测观测值、预测观测噪声和所述估计值、估计噪声进行数据融合，生成当前观测值，包括：
根据所述预测观测值、预测观测噪声和所述估计值、估计噪声确定修正量，以及根据所述估计值和所述修正量确定融合结果，得到当前观测值。
在一些实施例中，所述根据所述预测观测值、预测观测噪声和所述估计值、估计噪声确定修正量，包括：
根据所述预测观测噪声和所述估计噪声确定修正系数，根据所述预测观测值和所述估计值确定差异值；以及根据所述修正系数和所述差异值确定修正量。
在一些实施例中，所述传感器包括视觉传感器、雷达、惯性测量装置、TOF装置中的一种。
在一些实施例中，所述预测观测噪声包括所述预测观测值对应的方差或标准差，所述估计噪声包括所述估计值对应的方差或标准差。
在一些实施例中，所述获取传感器感测的当前数据，包括：获取视觉传感器感测的当前深度图像数据。
在一些实施例中，所述神经网络模型为基于卷积神经网络训练的3D box模型，所述3D box模型用于输出所述视觉传感器感测的当前深度图像数据对应的预测观测值和所述预测观测值对应的预测观测噪声。
在一些实施例中，所述预测观测值包括所述视觉传感器的目标物在世界坐标系的位置观测值、所述目标物的尺寸观测值和所述目标物的角度观测值，所述尺寸观测值包括长度观测值、宽度观测值和高度观测值；
所述预测观测噪声包括位置观测值对应的预测观测噪声、所述尺寸观测值对应的预测观测噪声和所述角度观测值对应的预测观测噪声。
在一些实施例中，所述当前深度图像数据为在图像坐标系下采集的数据，所述处理器用于实现：
将所述当前深度图像数据转换为世界坐标系下的图像数据；
相应地，所述基于预先训练好的神经网络模型，确定所述当前深度图像数据对应的预测观测值和所述预测观测值对应的预测观测噪声，包括：将所述世界坐标系下的图像数据输入至预先训练好的神经网络模型，得到所述图像数据对应的预测观测值和所述预测观测值对应的预测观测噪声。
请参阅图6，图6是本申请实施例提供的一种可移动平台的示意性框图。如图6所示，该可移动平台还至少包括一个或多个处理器301、存储器302和传感器303，传感器303设置在可移动平台的平台本体上。
其中，处理器301例如可以是微控制单元(Micro-controller Unit，MCU)、中央处理单元(Central Processing Unit，CPU)或数字信号处理器(Digital Signal Processor，DSP)等。
存储器302可以是Flash芯片、只读存储器(ROM，Read-Only Memory)磁盘、光盘、U盘或移动硬盘等。
其中，存储器302用于存储计算机程序；处理器301用于执行所述计算机程序并在执行所述计算机程序时，执行本申请实施例提供的任一项所述的传感器的数据融合方法，以提高传感器的数据融合结果。
示例性的，所述处理器，用于执行所述计算机程序并在执行所述计算机程序时，实现如下步骤：
获取传感器感测的当前数据；基于预先训练好的神经网络模型，确定所述当前数据对应的预测观测值和所述预测观测值对应的预测观测噪声；根据所述传感器感测的历史数据、所述当前数据对应的预测观测值和所述预测观测值对应的预测观测噪声，生成当前观测值。
在一些实施例中，所述神经网络模型的损失函数是根据所述预测观测值和所述预测观测噪声均服从正态分布确定的，所述损失函数用于将所述预测观测值和所述预测观测噪声进行相关。
在一些实施例中，所述神经网络模型的损失函数为最大化所述正态分布对应的正态分布式的对数似然得到的。
在一些实施例中，所述基于预先训练好的神经网络模型，确定所述当前数 据对应的预测观测值和所述预测观测值对应的预测观测噪声，包括：
将所述当前数据输入至预先训练好的神经网络模型，得到所述当前数据对应的预测观测值和输出噪声；根据所述输出噪声确定所述预测观测值对应的预测观测噪声。
在一些实施例中，所述处理器用于实现：根据所述传感器感测的历史数据确定历史观测值；根据所述历史观测值、所述当前数据对应的预测观测值和所述预测观测值对应的预测观测噪声，生成当前观测值。
在一些实施例中，所述处理器用于实现：根据所述历史观测值确定当前时刻对应的估计值和所述估计值对应的估计噪声；根据所述预测观测值、预测观测噪声和所述估计值、估计噪声进行数据融合，生成当前观测值。
在一些实施例中，所述处理器用于实现：
确定所述传感器的目标物对应的运动学方程，所述运动学方程用于描述所述目标物的运动轨迹；基于所述运动学方程，根据所述历史观测值确定所述当前时刻对应的估计值和所述估计值对应的估计噪声。
在一些实施例中，所述运动学方程包括直线运动方程或曲线运动方程。
在一些实施例中，所述处理器用于实现：利用Kalman滤波器，根据所述预测观测值、预测观测噪声和所述估计值、估计噪声进行数据融合，生成当前观测值。
在一些实施例中，所述根据所述预测观测值、预测观测噪声和所述估计值、估计噪声进行数据融合，生成当前观测值，包括：
根据所述预测观测值、预测观测噪声和所述估计值、估计噪声确定修正量，以及根据所述估计值和所述修正量确定融合结果，得到当前观测值。
在一些实施例中，所述根据所述预测观测值、预测观测噪声和所述估计值、估计噪声确定修正量，包括：
根据所述预测观测噪声和所述估计噪声确定修正系数，根据所述预测观测值和所述估计值确定差异值；以及根据所述修正系数和所述差异值确定修正量。
在一些实施例中，所述传感器包括视觉传感器、雷达、惯性测量装置、TOF装置中的一种。
在一些实施例中，所述预测观测噪声包括所述预测观测值对应的方差或标准差，所述估计噪声包括所述估计值对应的方差或标准差。
在一些实施例中，所述获取传感器感测的当前数据，包括：获取视觉传感器感测的当前深度图像数据。
在一些实施例中，所述神经网络模型为基于卷积神经网络训练的3D box模型，所述3D box模型用于输出所述视觉传感器感测的当前深度图像数据对应的预测观测值和所述预测观测值对应的预测观测噪声。
在一些实施例中，所述预测观测值包括所述视觉传感器的目标物在世界坐标系的位置观测值、所述目标物的尺寸观测值和所述目标物的角度观测值，所述尺寸观测值包括长度观测值、宽度观测值和高度观测值；
所述预测观测噪声包括位置观测值对应的预测观测噪声、所述尺寸观测值对应的预测观测噪声和所述角度观测值对应的预测观测噪声。
在一些实施例中，所述当前深度图像数据为在图像坐标系下采集的数据，所述处理器用于实现：
将所述当前深度图像数据转换为世界坐标系下的图像数据；
相应地，所述基于预先训练好的神经网络模型，确定所述当前深度图像数据对应的预测观测值和所述预测观测值对应的预测观测噪声，包括：将所述世界坐标系下的图像数据输入至预先训练好的神经网络模型，得到所述图像数据对应的预测观测值和所述预测观测值对应的预测观测噪声。
在一些实施例中，所述可移动平台包括飞行器、机器人和无人驾驶车辆中的一种。
请参阅图7，图7是本申请实施例提供的一种控制终端的示意性框图。如图7所示，该控制终端400至少包括一个或多个处理器401和存储器402，控制终端400与可移动平台通信连接，用于控制可移动平台移动。
其中，处理器401例如可以是微控制单元(Micro-controller Unit，MCU)、中央处理单元(Central Processing Unit，CPU)或数字信号处理器(Digital Signal Processor，DSP)等。
存储器402可以是Flash芯片、只读存储器(ROM，Read-Only Memory)磁盘、光盘、U盘或移动硬盘等。
其中，存储器402用于存储计算机程序；处理器401用于执行所述计算机程序并在执行所述计算机程序时，执行本申请实施例提供的任一项所述的传感器的数据融合方法，以提高传感器的数据融合结果。
本申请的实施例中还提供一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，所述计算机程序中包括程序指令，所述处理器执行所述程序指令，实现上述实施例提供的任一种所述的传感器的数据融合方法的步骤。
其中，所述计算机可读存储介质可以是前述任一实施例所述的可移动平台的内部存储单元，例如所述可移动平台的存储器或内存。所述计算机可读存储介质也可以是所述可移动平台的外部存储设备，例如所述可移动平台上配备的插接式硬盘，智能存储卡(Smart Media Card，SMC)，安全数字(Secure Digital，SD)卡，闪存卡(Flash Card)等。
以上所述，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本申请揭露的技术范围内，可轻易想到各种等效的修改或替换，这些修改或替换都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以权利要求的保护范围为准。

Claims (54)

  1. 一种传感器的数据融合方法，其特征在于，包括：
    获取传感器感测的当前数据；
    基于预先训练好的神经网络模型，确定所述当前数据对应的预测观测值和所述预测观测值对应的预测观测噪声；
    根据所述传感器感测的历史数据、所述当前数据对应的预测观测值和所述预测观测值对应的预测观测噪声，生成当前观测值。
  2. 根据权利要求1所述的方法，其特征在于，所述神经网络模型的损失函数是根据所述预测观测值和所述预测观测噪声均服从正态分布确定的，所述损失函数用于将所述预测观测值和所述预测观测噪声进行相关。
  3. 根据权利要求2所述的方法，其特征在于，所述神经网络模型的损失函数为最大化所述正态分布对应的正态分布式的对数似然得到的。
  4. 根据权利要求1所述的方法，其特征在于，所述基于预先训练好的神经网络模型，确定所述当前数据对应的预测观测值和所述预测观测值对应的预测观测噪声，包括：
    将所述当前数据输入至预先训练好的神经网络模型，得到所述当前数据对应的预测观测值和输出噪声；
    根据所述输出噪声确定所述预测观测值对应的预测观测噪声。
  5. 根据权利要求1所述的方法，其特征在于，所述方法包括：
    根据所述传感器感测的历史数据确定历史观测值；
    根据所述历史观测值、所述当前数据对应的预测观测值和所述预测观测值对应的预测观测噪声，生成当前观测值。
  6. 根据权利要求5所述的方法，其特征在于，所述方法包括：
    根据所述历史观测值确定当前时刻对应的估计值和所述估计值对应的估计噪声；
    根据所述预测观测值、预测观测噪声和所述估计值、估计噪声进行数据融合，生成当前观测值。
  7. 根据权利要求6所述的方法，其特征在于，所述方法包括：
    确定所述传感器的目标物对应的运动学方程，所述运动学方程用于描述所述目标物的运动轨迹；
    基于所述运动学方程，根据所述历史观测值确定所述当前时刻对应的估计值和所述估计值对应的估计噪声。
  8. 根据权利要求7所述的方法，其特征在于，所述运动学方程包括直线运动方程或曲线运动方程。
  9. 根据权利要求6所述的方法，其特征在于，所述方法包括：
    利用Kalman滤波器，根据所述预测观测值、预测观测噪声和所述估计值、估计噪声进行数据融合，生成当前观测值。
  10. 根据权利要求6所述的方法，其特征在于，所述根据所述预测观测值、预测观测噪声和所述估计值、估计噪声进行数据融合，生成当前观测值，包括：
    根据所述预测观测值、预测观测噪声和所述估计值、估计噪声确定修正量，以及根据所述估计值和所述修正量确定融合结果，得到当前观测值。
  11. 根据权利要求10所述的方法，其特征在于，所述根据所述预测观测值、预测观测噪声和所述估计值、估计噪声确定修正量，包括：
    根据所述预测观测噪声和所述估计噪声确定修正系数，根据所述预测观测值和所述估计值确定差异值；以及
    根据所述修正系数和所述差异值确定修正量。
  12. 根据权利要求1-11任一项所述的方法，其特征在于，所述传感器包括视觉传感器、雷达、惯性测量装置、TOF装置中的一种。
  13. 根据权利要求1-11任一项所述的方法，其特征在于，所述预测观测噪声包括所述预测观测值对应的方差或标准差，所述估计噪声包括所述估计值对应的方差或标准差。
  14. 根据权利要求1-11任一项所述的方法，其特征在于，所述获取传感器感测的当前数据，包括：
    获取视觉传感器感测的当前深度图像数据。
  15. 根据权利要求14所述的方法，其特征在于，所述神经网络模型为基于卷积神经网络训练的3D box模型，所述3D box模型用于输出所述视觉传感器感测的当前深度图像数据对应的预测观测值和所述预测观测值对应的预测观测 噪声。
  16. 根据权利要求15所述的方法，其特征在于，所述预测观测值包括所述视觉传感器的目标物在世界坐标系的位置观测值、所述目标物的尺寸观测值和所述目标物的角度观测值，所述尺寸观测值包括长度观测值、宽度观测值和高度观测值；
    所述预测观测噪声包括位置观测值对应的预测观测噪声、所述尺寸观测值对应的预测观测噪声和所述角度观测值对应的预测观测噪声。
  17. 根据权利要求14所述的方法，其特征在于，所述当前深度图像数据为在图像坐标系下采集的数据，所述方法包括：
    将所述当前深度图像数据转换为世界坐标系下的图像数据；
    所述基于预先训练好的神经网络模型，确定所述当前深度图像数据对应的预测观测值和所述预测观测值对应的预测观测噪声，包括：
    将所述世界坐标系下的图像数据输入至预先训练好的神经网络模型，得到所述图像数据对应的预测观测值和所述预测观测值对应的预测观测噪声。
  18. 一种数据融合装置，其特征在于，所述数据融合装置包括处理器和存储器；
    所述存储器用于存储计算机程序；
    所述处理器，用于执行所述计算机程序并在执行所述计算机程序时，实现：
    获取传感器感测的当前数据；
    基于预先训练好的神经网络模型，确定所述当前数据对应的预测观测值和所述预测观测值对应的预测观测噪声；
    根据所述传感器感测的历史数据、所述当前数据对应的预测观测值和所述预测观测值对应的预测观测噪声，生成当前观测值。
  19. 根据权利要求18所述的装置，其特征在于，所述神经网络模型的损失函数是根据所述预测观测值和所述预测观测噪声均服从正态分布确定的，所述损失函数用于将所述预测观测值和所述预测观测噪声进行相关。
  20. 根据权利要求19所述的装置，其特征在于，所述神经网络模型的损失函数为最大化所述正态分布对应的正态分布式的对数似然得到的。
  21. 根据权利要求18所述的装置，其特征在于，所述基于预先训练好的神经网络模型，确定所述当前数据对应的预测观测值和所述预测观测值对应的预 测观测噪声，包括：
    将所述当前数据输入至预先训练好的神经网络模型，得到所述当前数据对应的预测观测值和输出噪声；
    根据所述输出噪声确定所述预测观测值对应的预测观测噪声。
  22. 根据权利要求18所述的装置，其特征在于，所述处理器用于实现：
    根据所述传感器感测的历史数据确定历史观测值；
    根据所述历史观测值、所述当前数据对应的预测观测值和所述预测观测值对应的预测观测噪声，生成当前观测值。
  23. 根据权利要求22所述的装置，其特征在于，所述处理器用于实现：
    根据所述历史观测值确定当前时刻对应的估计值和所述估计值对应的估计噪声；
    根据所述预测观测值、预测观测噪声和所述估计值、估计噪声进行数据融合，生成当前观测值。
  24. 根据权利要求23所述的装置，其特征在于，所述处理器用于实现：
    确定所述传感器的目标物对应的运动学方程，所述运动学方程用于描述所述目标物的运动轨迹；
    基于所述运动学方程，根据所述历史观测值确定所述当前时刻对应的估计值和所述估计值对应的估计噪声。
  25. 根据权利要求24所述的装置，其特征在于，所述运动学方程包括直线运动方程或曲线运动方程。
  26. 根据权利要求23所述的装置，其特征在于，所述处理器用于实现：
    利用Kalman滤波器，根据所述预测观测值、预测观测噪声和所述估计值、估计噪声进行数据融合，生成当前观测值。
  27. 根据权利要求23所述的装置，其特征在于，所述根据所述预测观测值、预测观测噪声和所述估计值、估计噪声进行数据融合，生成当前观测值，包括：
    根据所述预测观测值、预测观测噪声和所述估计值、估计噪声确定修正量，以及根据所述估计值和所述修正量确定融合结果，得到当前观测值。
  28. 根据权利要求27所述的装置，其特征在于，所述根据所述预测观测值、预测观测噪声和所述估计值、估计噪声确定修正量，包括：
    根据所述预测观测噪声和所述估计噪声确定修正系数，根据所述预测观测 值和所述估计值确定差异值；以及
    根据所述修正系数和所述差异值确定修正量。
  29. 根据权利要求18-28任一项所述的装置，其特征在于，所述传感器包括视觉传感器、雷达、惯性测量装置、TOF装置中的一种。
  30. 根据权利要求18-28任一项所述的装置，其特征在于，所述预测观测噪声包括所述预测观测值对应的方差或标准差，所述估计噪声包括所述估计值对应的方差或标准差。
  31. 根据权利要求18-28任一项所述的装置，其特征在于，所述获取传感器感测的当前数据，包括：
    获取视觉传感器感测的当前深度图像数据。
  32. 根据权利要求31所述的装置，其特征在于，所述神经网络模型为基于卷积神经网络训练的3D box模型，所述3D box模型用于输出所述视觉传感器感测的当前深度图像数据对应的预测观测值和所述预测观测值对应的预测观测噪声。
  33. 根据权利要求32所述的装置，其特征在于，所述预测观测值包括所述视觉传感器的目标物在世界坐标系的位置观测值、所述目标物的尺寸观测值和所述目标物的角度观测值，所述尺寸观测值包括长度观测值、宽度观测值和高度观测值；
    所述预测观测噪声包括位置观测值对应的预测观测噪声、所述尺寸观测值对应的预测观测噪声和所述角度观测值对应的预测观测噪声。
  34. 根据权利要求31所述的装置，其特征在于，所述当前深度图像数据为在图像坐标系下采集的数据，所述处理器用于实现：
    将所述当前深度图像数据转换为世界坐标系下的图像数据；
    所述基于预先训练好的神经网络模型，确定所述当前深度图像数据对应的预测观测值和所述预测观测值对应的预测观测噪声，包括：
    将所述世界坐标系下的图像数据输入至预先训练好的神经网络模型，得到所述图像数据对应的预测观测值和所述预测观测值对应的预测观测噪声。
  35. 一种可移动平台，其特征在于，所述可移动平台包括处理器和存储器；
    所述存储器用于存储计算机程序；
    所述处理器，用于执行所述计算机程序并在执行所述计算机程序时，实现：
    获取传感器感测的当前数据；
    基于预先训练好的神经网络模型，确定所述当前数据对应的预测观测值和所述预测观测值对应的预测观测噪声；
    根据所述传感器感测的历史数据、所述当前数据对应的预测观测值和所述预测观测值对应的预测观测噪声，生成当前观测值。
  36. 根据权利要求35所述的可移动平台，其特征在于，所述神经网络模型的损失函数是根据所述预测观测值和所述预测观测噪声均服从正态分布确定的，所述损失函数用于将所述预测观测值和所述预测观测噪声进行相关。
  37. 根据权利要求36所述的可移动平台，其特征在于，所述神经网络模型的损失函数为最大化所述正态分布对应的正态分布式的对数似然得到的。
  38. 根据权利要求35所述的可移动平台，其特征在于，所述基于预先训练好的神经网络模型，确定所述当前数据对应的预测观测值和所述预测观测值对应的预测观测噪声，包括：
    将所述当前数据输入至预先训练好的神经网络模型，得到所述当前数据对应的预测观测值和输出噪声；
    根据所述输出噪声确定所述预测观测值对应的预测观测噪声。
  39. 根据权利要求35所述的可移动平台，其特征在于，所述处理器用于实现：
    根据所述传感器感测的历史数据确定历史观测值；
    根据所述历史观测值、所述当前数据对应的预测观测值和所述预测观测值对应的预测观测噪声，生成当前观测值。
  40. 根据权利要求39所述的可移动平台，其特征在于，所述处理器用于实现：
    根据所述历史观测值确定当前时刻对应的估计值和所述估计值对应的估计噪声；
    根据所述预测观测值、预测观测噪声和所述估计值、估计噪声进行数据融合，生成当前观测值。
  41. 根据权利要求40所述的可移动平台，其特征在于，所述处理器用于实现：
    确定所述传感器的目标物对应的运动学方程，所述运动学方程用于描述所 述目标物的运动轨迹；
    基于所述运动学方程，根据所述历史观测值确定所述当前时刻对应的估计值和所述估计值对应的估计噪声。
  42. 根据权利要求41所述的可移动平台，其特征在于，所述运动学方程包括直线运动方程或曲线运动方程。
  43. 根据权利要求40所述的可移动平台，其特征在于，所述处理器用于实现：
    利用Kalman滤波器，根据所述预测观测值、预测观测噪声和所述估计值、估计噪声进行数据融合，生成当前观测值。
  44. 根据权利要求40所述的可移动平台，其特征在于，所述根据所述预测观测值、预测观测噪声和所述估计值、估计噪声进行数据融合，生成当前观测值，包括：
    根据所述预测观测值、预测观测噪声和所述估计值、估计噪声确定修正量，以及根据所述估计值和所述修正量确定融合结果，得到当前观测值。
  45. 根据权利要求44所述的可移动平台，其特征在于，所述根据所述预测观测值、预测观测噪声和所述估计值、估计噪声确定修正量，包括：
    根据所述预测观测噪声和所述估计噪声确定修正系数，根据所述预测观测值和所述估计值确定差异值；以及
    根据所述修正系数和所述差异值确定修正量。
  46. 根据权利要求35-45任一项所述的可移动平台，其特征在于，所述传感器包括视觉传感器、雷达、惯性测量装置、TOF装置中的一种。
  47. 根据权利要求35-45任一项所述的可移动平台，其特征在于，所述预测观测噪声包括所述预测观测值对应的方差或标准差，所述估计噪声包括所述估计值对应的方差或标准差。
  48. 根据权利要求35-45任一项所述的可移动平台，其特征在于，所述获取传感器感测的当前数据，包括：
    获取视觉传感器感测的当前深度图像数据。
  49. 根据权利要求48所述的可移动平台，其特征在于，所述神经网络模型为基于卷积神经网络训练的3D box模型，所述3D box模型用于输出所述视觉传感器感测的当前深度图像数据对应的预测观测值和所述预测观测值对应的预 测观测噪声。
  50. 根据权利要求49所述的可移动平台，其特征在于，所述预测观测值包括所述视觉传感器的目标物在世界坐标系的位置观测值、所述目标物的尺寸观测值和所述目标物的角度观测值，所述尺寸观测值包括长度观测值、宽度观测值和高度观测值；
    所述预测观测噪声包括位置观测值对应的预测观测噪声、所述尺寸观测值对应的预测观测噪声和所述角度观测值对应的预测观测噪声。
  51. 根据权利要求48所述的可移动平台，其特征在于，所述当前深度图像数据为在图像坐标系下采集的数据，所述处理器用于实现：
    将所述当前深度图像数据转换为世界坐标系下的图像数据；
    所述基于预先训练好的神经网络模型，确定所述当前深度图像数据对应的预测观测值和所述预测观测值对应的预测观测噪声，包括：
    将所述世界坐标系下的图像数据输入至预先训练好的神经网络模型，得到所述图像数据对应的预测观测值和所述预测观测值对应的预测观测噪声。
  52. 根据权利要求45所述的可移动平台，其特征在于，所述可移动平台包括飞行器、机器人和无人驾驶车辆中的一种。
  53. 一种控制终端，其特征在于，所述控制终端与可移动平台通信连接，所述可移动平台包括处理器和存储器；
    所述存储器用于存储计算机程序；
    所述处理器，用于执行所述计算机程序并在执行所述计算机程序时，实现：
    获取传感器感测的当前数据；
    基于预先训练好的神经网络模型，确定所述当前数据对应的预测观测值和所述预测观测值对应的预测观测噪声；
    根据所述传感器感测的历史数据、所述当前数据对应的预测观测值和所述预测观测值对应的预测观测噪声，生成当前观测值。
  54. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质存储有计算机程序，所述计算机程序被处理器执行时使所述处理器实现如权利要求1至17任一项所述的数据融合方法的步骤。
PCT/CN2020/136622 2020-12-15 2020-12-15 传感器的数据融合方法、设备及存储介质 Ceased WO2022126397A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/CN2020/136622 WO2022126397A1 (zh) 2020-12-15 2020-12-15 传感器的数据融合方法、设备及存储介质

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2020/136622 WO2022126397A1 (zh) 2020-12-15 2020-12-15 传感器的数据融合方法、设备及存储介质

Publications (1)

Publication Number Publication Date
WO2022126397A1 true WO2022126397A1 (zh) 2022-06-23

Family

ID=82059874

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/136622 Ceased WO2022126397A1 (zh) 2020-12-15 2020-12-15 传感器的数据融合方法、设备及存储介质

Country Status (1)

Country Link
WO (1) WO2022126397A1 (zh)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116108762A (zh) * 2023-04-13 2023-05-12 南京航空航天大学 一种利用力传感器的大型复合材料构件装配变形预测方法
CN117034121A (zh) * 2023-08-18 2023-11-10 南京中旭电子科技有限公司 霍尔传感器故障显示方法、装置、设备及存储介质
CN117150428A (zh) * 2023-08-14 2023-12-01 中国电子科技集团公司第五十二研究所 一种多类传感器数据融合方法、装置及电子设备
CN117990112A (zh) * 2024-04-03 2024-05-07 中国人民解放军海军工程大学 基于鲁棒无迹卡尔曼滤波的无人机光电平台目标定位方法
CN118642045A (zh) * 2024-08-15 2024-09-13 四川腾盾科技有限公司 固定翼集群无人机目标融合定位方法、装置及系统
CN119845261A (zh) * 2025-03-24 2025-04-18 天津云圣智能科技有限责任公司 一种基于传感器的物理参数估计方法及装置

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5373456A (en) * 1992-11-02 1994-12-13 The United States Of America As Represented By The Secretary Of The Navy Expert system for assessing accuracy of models of physical phenomena and for selecting alternate models in the presence of noise
CN102316496A (zh) * 2011-09-07 2012-01-11 上海交通大学 无线传感器网络中基于卡尔曼滤波的数据融合方法
CN108759846A (zh) * 2018-05-29 2018-11-06 东南大学 自适应扩展卡尔曼滤波噪声模型建立方法
CN111814976A (zh) * 2020-07-14 2020-10-23 西安建筑科技大学 一种空调系统传感器故障误差再学习方法及系统

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5373456A (en) * 1992-11-02 1994-12-13 The United States Of America As Represented By The Secretary Of The Navy Expert system for assessing accuracy of models of physical phenomena and for selecting alternate models in the presence of noise
CN102316496A (zh) * 2011-09-07 2012-01-11 上海交通大学 无线传感器网络中基于卡尔曼滤波的数据融合方法
CN108759846A (zh) * 2018-05-29 2018-11-06 东南大学 自适应扩展卡尔曼滤波噪声模型建立方法
CN111814976A (zh) * 2020-07-14 2020-10-23 西安建筑科技大学 一种空调系统传感器故障误差再学习方法及系统

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116108762A (zh) * 2023-04-13 2023-05-12 南京航空航天大学 一种利用力传感器的大型复合材料构件装配变形预测方法
CN116108762B (zh) * 2023-04-13 2023-06-27 南京航空航天大学 一种利用力传感器的大型复合材料构件装配变形预测方法
CN117150428A (zh) * 2023-08-14 2023-12-01 中国电子科技集团公司第五十二研究所 一种多类传感器数据融合方法、装置及电子设备
CN117150428B (zh) * 2023-08-14 2025-07-29 中国电子科技集团公司第五十二研究所 一种多类传感器数据融合方法、装置及电子设备
CN117034121A (zh) * 2023-08-18 2023-11-10 南京中旭电子科技有限公司 霍尔传感器故障显示方法、装置、设备及存储介质
CN117990112A (zh) * 2024-04-03 2024-05-07 中国人民解放军海军工程大学 基于鲁棒无迹卡尔曼滤波的无人机光电平台目标定位方法
CN118642045A (zh) * 2024-08-15 2024-09-13 四川腾盾科技有限公司 固定翼集群无人机目标融合定位方法、装置及系统
CN119845261A (zh) * 2025-03-24 2025-04-18 天津云圣智能科技有限责任公司 一种基于传感器的物理参数估计方法及装置

Similar Documents

Publication Publication Date Title
US11822353B2 (en) Simple multi-sensor calibration
US11604479B2 (en) Methods and system for vision-based landing
CN112912811B (zh) 一种无人机路径规划方法、装置及无人机
EP3158293B1 (en) Sensor fusion using inertial and image sensors
EP3158417B1 (en) Sensor fusion using inertial and image sensors
EP3158412B1 (en) Sensor fusion using inertial and image sensors
EP3158411B1 (en) Sensor fusion using inertial and image sensors
CN109388150B (zh) 多传感器环境地图构建
US20190050000A1 (en) Image space motion planning of an autonomous vehicle
US20170153122A1 (en) System and method for automatic sensor calibration
CN112136137A (zh) 一种参数优化方法、装置及控制设备、飞行器
JP2017065467A (ja) 無人機およびその制御方法
CN106155075B (zh) 一种可分离式无人机控制系统
US20210240185A1 (en) Shooting control method and unmanned aerial vehicle
CN109508036B (zh) 一种中继点生成方法、装置和无人机
AU2017251682A1 (en) Systems and methods for establishing a flight pattern adjacent to a target for a vehicle to follow
US20240019866A1 (en) Aerial vehicle control method and apparatus, aerial vehicle, and storage medium
WO2022094962A1 (zh) 飞行器的悬停方法、飞行器及存储介质
WO2022227096A1 (zh) 点云数据的处理方法、设备及存储介质
US12481074B2 (en) Motion-based calibration of an aerial device
CN215416387U (zh) 一种无人机

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20965415

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20965415

Country of ref document: EP

Kind code of ref document: A1