CN116824433A - Visual-inertial navigation-radar fusion self-positioning method based on self-supervision neural network - Google Patents
Visual-inertial navigation-radar fusion self-positioning method based on self-supervision neural network Download PDFInfo
- Publication number
- CN116824433A CN116824433A CN202310495293.4A CN202310495293A CN116824433A CN 116824433 A CN116824433 A CN 116824433A CN 202310495293 A CN202310495293 A CN 202310495293A CN 116824433 A CN116824433 A CN 116824433A
- Authority
- CN
- China
- Prior art keywords
- self
- features
- network
- radar
- fusion
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000004927 fusion Effects 0.000 title claims abstract description 61
- 238000013528 artificial neural network Methods 0.000 title claims abstract description 26
- 238000000034 method Methods 0.000 title claims abstract description 22
- 230000000007 visual effect Effects 0.000 claims abstract description 32
- 238000000605 extraction Methods 0.000 claims abstract description 17
- 230000009466 transformation Effects 0.000 claims abstract description 17
- 239000011159 matrix material Substances 0.000 claims abstract description 9
- 238000009499 grossing Methods 0.000 claims description 11
- 238000005457 optimization Methods 0.000 claims description 7
- 239000000284 extract Substances 0.000 claims description 4
- 230000006870 function Effects 0.000 description 16
- 238000005516 engineering process Methods 0.000 description 5
- 238000012360 testing method Methods 0.000 description 4
- 238000012549 training Methods 0.000 description 4
- 230000002902 bimodal effect Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000005286 illumination Methods 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 238000013135 deep learning Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 238000009825 accumulation Methods 0.000 description 1
- 230000004913 activation Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 238000007500 overflow downdraw method Methods 0.000 description 1
- 230000008092 positive effect Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000002310 reflectometry Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
- 230000001502 supplementing effect Effects 0.000 description 1
Landscapes
- Image Analysis (AREA)
Abstract
The application relates to a vision-inertial navigation-radar fusion self-positioning method based on a self-supervision neural network, which comprises the following steps: acquiring a video frame sequence, inertial navigation data and a laser radar point cloud picture; inputting the video frame sequence, the inertial navigation data and the laser radar point cloud image into a pose estimation network model to obtain the relative pose estimation of the input video frame; wherein, the pose estimation network model includes: the depth prediction network is used for obtaining a depth map according to an input video frame sequence; the feature extraction network is used for extracting feature information from an input video frame sequence, inertial navigation data and a laser radar point cloud image respectively to obtain visual features, momentum features and radar features; the feature fusion network is used for fusing the visual features and the radar features to obtain pre-fusion features, and fusing the pre-fusion features and the momentum features to obtain fusion features; and the pose estimation network is used for predicting a pose transformation matrix according to the fusion characteristics. The application improves the positioning precision of the self-supervision depth positioning algorithm.
Description
Technical Field
The application relates to the technical field of self-positioning, in particular to a vision-inertial navigation-radar fusion self-positioning method based on a self-supervision neural network.
Background
The self-positioning technique is widely used in the fields of automatic driving, SLAM and the like. In some scenes (buildings in cities, tunnels, dense forests in mountains, mountain holes or satellite refusal sites), it is difficult to use satellite, beidou, GPS and other radio wave technologies for positioning, and at the moment, autonomous positioning is required by means of cameras carried by automobiles, unmanned aerial vehicles or robots. The traditional self-positioning algorithm based on the vision sensor comprises the following basic flow: feature extraction, feature matching, pose calculation and back-end optimization. The traditional self-positioning algorithm can naturally obtain higher positioning precision by means of complete global map construction optimization, but the calculation process is complex, the calculation speed is slow, and real-time reasoning is difficult to realize. With the popularity and continuous development of deep learning, a neural network-based visual self-positioning algorithm, namely a deep visual odometer VO, can train out a model offline by utilizing a data set in advance, and the trained model is directly used for reasoning, so that the algorithm does not need characteristic matching and rear-end optimization steps, and real-time reasoning is easy to realize. The depth VO improves positioning performance by training the optimized objective loss function through supervised or unsupervised learning. The model can be trained without acquiring a real pose label by non-supervision (or self-supervision) learning, so that a large amount of unlabeled data can be utilized, and the training cost is lower compared with that of a supervision mode.
Self-monitoring VO was first proposed by tinchui methou in 2017. The self-supervision VO firstly sends the input continuous frame images into a neural network for pose estimation and a neural network for depth estimation to calculate pose transformation between a depth image of the input images and the images, then uses the depth image and the pose to calculate the reprojection error between the continuous frame images through the projection relation established by epipolar geometry, and finally calculates the gradient of the error relative to the pose estimation and the depth estimation neural network and counter-propagates the updated parameters so as to achieve the effect of optimizing estimation.
However, locating by only single modality data may suffer from data loss, insufficient information, and the like. In the present big data age, more and more sensors are invented, more and more data types can be acquired, and multi-mode sensor fusion is a main trend of future positioning technology development. Compared with the Shan Motai sensor positioning technology, the multi-mode sensor fusion positioning technology can integrate the advantages of all mode data to perform information fusion and information compensation, so that positioning with higher accuracy is realized. The video frame signal can provide the most intuitive RGB time sequence information, which is important for neural network learning feature detection, however, dynamic objects may exist in the video frame so as to cause misjudgment of the motion speed, and the monocular video frame signal lacks depth information; the inertial navigation signal can provide direct information of individual acceleration and speed, but various parameter biases exist in the inertial navigation signal, and error accumulation can be generated, so that the inertial navigation signal is not beneficial to long-time work; lidar signals provide depth map information but lack color visual features, so that lidar signals may be a good information supplement to monocular video frame signals that lack depth information. Binocular video signals, although also providing depth information, require the help of left and right eye matching algorithms, so the computational complexity is high, while binocular cameras are very sensitive to illumination changes and texture details, and inconsistent left and right eye illumination and too monotonous texture scenes may cause matching failure. Various bimodal sensor fusion positioning algorithms based on neural networks exist. For example: depth vision-momentum meter VIO, depth radar vision meter VLO and depth momentum radar meter LIO, which lack more or less input of some important information based on bimodal sensor fusion, and research of three and more modal sensor fusion positioning technologies is still in the traditional algorithm stage.
Disclosure of Invention
The application provides a vision-inertial navigation-radar fusion self-positioning method based on a self-supervision neural network, which solves the problem of insufficient self-positioning information depending on a single mode or a double mode.
The technical scheme adopted for solving the technical problems is as follows: the visual-inertial navigation-radar fusion self-positioning method based on the self-supervision neural network comprises the following steps:
acquiring a video frame sequence, inertial navigation data and a laser radar point cloud picture;
inputting the video frame sequence, the inertial navigation data and the laser radar point cloud image into a pose estimation network model to obtain relative pose estimation of an input video frame;
wherein, the pose estimation network model comprises:
the depth prediction network is used for obtaining a depth map according to the input video frame sequence;
the feature extraction network is used for extracting feature information from an input video frame sequence, inertial navigation data and a laser radar point cloud image respectively to obtain visual features, momentum features and radar features;
the feature fusion network is used for fusing the visual features and the radar features to obtain pre-fused corrected visual features and corrected radar features, and fusing the pre-fused corrected visual features and corrected radar features with the momentum features to obtain fusion features;
the pose estimation network is used for predicting a pose transformation matrix according to the fusion characteristics;
and the parameter optimization module is used for calculating a loss function according to the depth map, the pose transformation matrix and the video frame, and adjusting parameters of the pose estimation network model according to the loss function.
The feature extraction network includes:
a first feature extraction section that extracts visual features from the video frame sequence using a first convolutional network;
a second feature extraction part for extracting the motion feature from the inertial navigation data by adopting an LSTM network;
the third feature extraction part projects the laser radar point cloud image to a 2D plane, codes the laser radar point cloud image projected to the 2D plane in a three-channel coding mode, and extracts radar features by adopting a second convolution network;
the first convolution network and the second convolution network have the same structure and share the weight of all network layers except the BN layer;
the first convolution network and the second convolution network have the same structure and share the weight of all network layers except the BN layer.
The feature fusion network comprises:
the first fusion part is used for fusing the visual characteristics and the radar characteristics by adopting a channel exchange strategy to obtain pre-fused corrected visual characteristics and corrected radar characteristics;
and the second fusion part is used for carrying out channel splicing on the presplitting corrected visual features and the corrected radar features and the momentum features to obtain fusion features.
The channel switching policy is thatWherein V 'is' k,c Visual characteristics of the c-th channel output by the k-th layer convolution layer after the exchange strategy are represented, V k,c Visual characteristics of the c-th channel representing the output of the k-th convolutional layer, L k,c A radar feature representing the c-th channel of the k-th convolutional layer output, a v,k,c 、b v,k,c 、σ v,k,c Sum mu v,k,c Respectively representing the slope, bias, mean and variance of BN layer in the first convolution network; a, a l,k,c 、b l,k,c 、σ l,k,c Sum mu l,k,c Respectively representing the slope, bias, mean and variance of BN layer in the second convolution network; delta is a threshold.
The loss function includes a reconstruction errorDepth smoothing loss and geometric consistency loss, expressed as:wherein L is all As a loss function, L pe Representing reconstruction error, L smooth Representing depth smoothing loss, L geo Represents the geometric consistency loss, l represents the scale number, ω 1 ,ω 2 ,ω 3 Weights representing reconstruction error, depth smoothing loss, and geometric consistency loss, respectively.
The expression of the reconstruction error is:wherein I is s P is the source image s For source image I s Point on I t For the target image, p' s For source image I s Point p on s Corresponding to the target image I t SSIM () is a structural similarity function, lambda 1 And lambda (lambda) 2 Is a weight coefficient.
The expression of the reconstruction error is: l (L) pe ＝λ 1 |I′ s -I s |+λ 2 SSIM(I′ s ,I s ) Wherein I s For source image, I' s Is based on the target image I t Reconstructed source image, SSIM () is a structural similarity function, λ 1 And lambda (lambda) 2 Is a weight coefficient.
The expression of the depth smoothing loss is:wherein D is t For the depth map corresponding to the image at time t in the video frame sequence S, I t For the target image +.>And->Representing partial derivatives of the x-direction and y-direction of the two-dimensional image coordinates, respectively.
Table of the geometric consistency lossThe expression is:wherein D is t For the depth map corresponding to the image at time t in the sequence of video frames S, D' t And representing the depth map of the current t moment which is generated by reconstructing the pose transformation by utilizing the depth map of the subsequent moment.
Advantageous effects
Due to the adoption of the technical scheme, compared with the prior art, the application has the following advantages and positive effects: according to the application, through designing the feature extraction network of each mode and the multimode data fusion network, the full interaction and perfection between the multimode information are realized by utilizing the similar-mode priority fusion strategy and the channel exchange strategy, and the positioning precision of the self-supervision depth positioning algorithm is improved.
Drawings
FIG. 1 is a flow chart of a vision-inertial navigation-radar fusion self-positioning method based on a self-supervising neural network in accordance with an embodiment of the present application;
FIG. 2 is a framework diagram of a pose estimation network model in an embodiment of the application;
FIG. 3 is a schematic diagram of feature fusion in an embodiment of the application;
fig. 4 is a graph comparing the path trace and the true path of the test output on the KITTI data sets seq09 and seq10 according to an embodiment of the application.
Detailed Description
The application will be further illustrated with reference to specific examples. It is to be understood that these examples are illustrative of the present application and are not intended to limit the scope of the present application. Furthermore, it should be understood that various changes and modifications can be made by one skilled in the art after reading the teachings of the present application, and such equivalents are intended to fall within the scope of the application as defined in the appended claims.
The embodiment of the application relates to a vision-inertial navigation-radar fusion self-positioning method based on a self-supervision neural network, which is shown in fig. 1 and comprises the following steps:
step 1, acquiring a video frame sequence, inertial navigation data and a laser radar point cloud picture;
and step 2, inputting the video frame sequence, the inertial navigation data and the laser radar point cloud image into a pose estimation network model to obtain the relative pose estimation of the input video frame.
The pose estimation network model is shown in fig. 2, takes a video frame sequence, inertial navigation data and a laser radar point cloud image as input, and outputs the pose estimation network model as relative pose estimation of each moment of the input sequence. For an input video frame sequence, three parallel processing branches are provided, wherein one branch is a depth map D obtained through a depth prediction network DepthNet; secondly, inputting inertial navigation data and a laser radar point cloud image into a multi-mode feature extraction network, and then obtaining relative pose estimation (pose transformation matrix T) at corresponding moment through a pose estimation network PoseNet with a feature fusion function; thirdly, the loss function is calculated by combining the depth map D and the pose transformation T. The pose estimation network model calculates a self-supervision loss function by utilizing a video frame, a depth map and a pose and optimizes network parameters through gradient descent back propagation. Multiple training iterations are performed until the model converges.
The pose estimation network model in the present embodiment includes: the system comprises a depth prediction network, a feature extraction network, a feature fusion network, a pose estimation network and a parameter optimization module.
The feature extraction network is used for extracting the features of each mode.
The visual features are extracted for a sequence of video frames from a monocular camera using a seven-layer convolutional network (ConvNet), each of which is followed by a batch normalization layer (BN layer) and a nonlinear activation function layer (relu layer). Each frame of the input video frame sequence is set to be video frame F t And its next frame F t+1 After being spliced according to the channels, the video signal is input into a network to extract space information and time sequence information, and visual characteristics V are obtained t ：
V t ＝ConvNet(F t ,F t+1 )
For IMU signals X from inertial navigation t Extracting time domain information by adopting an LSTM network, and taking the value of the last hidden layer of the LSTM network as an output momentum characteristic I t ：
I t ＝LSTM(X t )
For point cloud signals with the data format of (x, y, z, i) from the laser radar, wherein x, y, z represent three-dimensional space coordinates, i represents intensity (related to object surface illuminance and reflectivity), a sparse unordered point cloud is projected onto a 2D plane by adopting a cylindrical coordinate projection mode:
alpha and beta represent pitch and yaw angles, respectively, of the cylindrical coordinates, and are also the abscissas and abscissas of the planar projection view. When a plurality of 3D points are mapped to the same coordinates, only the 3D point nearest to the origin of coordinates (laser radar sensor center) is taken as the corresponding projection point. The projected 2D projection image is encoded into the following steps by adopting a three-channel encoding mode(three channel pixel values at each coordinate in the projection view). If some coordinates do not have any 3D points corresponding to them, the channel value at these coordinates is set to the default value of 0. Direct adoption of ConvNet network pair projected radar projection map M identical to video signal processing t Extracting features L t Wherein the weights of all networks except the BN layer are shared:
L t ＝ConvNet(M t ,M t+1 )
the common multi-mode feature fusion method based on the neural network is that features of each mode data are extracted by using different feature encoders and aligned according to a certain dimension, and then the mode features are spliced according to the dimension and then sent to a fusion network for processing. This is most common in depth odometers based on bimodal fusion. However, in the tri-modal fusion, the method of directly using tri-modal data channel stitching needs to learn, for the model, when a certain modality should be fused with which modality (or fused with two other modalities simultaneously) (i.e. the fusion sequence) in addition to how the certain modality is fused with two other modalities (i.e. the fusion mode), so that training of the model becomes more difficult and slow.
The feature fusion network in this embodiment follows a fixed fusion order: the fusion between the simpler and visual similar modes is firstly carried out, and then the fusion between the complex and abstract dissimilar modes is carried out. The radar point cloud data can be converted into 2D image representation through projection transformation, then the 2D image representation can be fused with the characteristics of video frame sequence signals in similar modes, and then the fusion result is further fused with the characteristics of IMU signals with larger data format differences.
When the fusion of the video features and the radar features is carried out, firstly, data of two similar modes are respectively input into a feature extraction network (ConvNet) with shared weights except a BN layer, and a channel switching strategy is adopted:
the two rows above represent the visual characteristic V of the c-th channel output by the k-th convolution layer, respectively k,c And radar feature L k,c Expression (a) of passing BN layer v,k,c 、b v,k,c 、σ v,k,c Sum mu v,k,c Respectively representing the slope, bias, mean and variance of BN layer in the first convolution network; a, a l,k,c 、b l,k,c 、σ l,k,c Sum mu l,k,c Respectively representing the slope, bias, mean and variance of the BN layer in the second convolutional network), the channel switching strategy judges the importance and integrity of each channel information of the output feature map of the previous layer according to the slope a in the weight value of the BN layer, and an excessively small slope value (here, the threshold value is set as delta) means that the parameter updating weight of the model at the channel is small when the gradient return parameter is updated, namely, the information of the channel is incomplete or unimportant, so that the components of the feature map output by the convolutional layer at the channel are replaced at the next iterationThe channel component at the corresponding position of the feature map of another similar mode (the condition corresponding to the second row in the above formula is that the visual feature channel is replaced by the radar feature channel) so as to achieve the effect of supplementing the missing information, because the feature component at the corresponding channel of the feature map output after the similar mode experiences some identical network layers (weight sharing) should contain similar semantic information (illumination, depth and the like), and therefore, the information among the modes can be mutually supplemented and perfected.
Obtaining pre-fused corrected visual characteristics V 'through similar modal fusion' t And radar feature L' t Then the presegregated visual features and radar features are extracted from the IMU features to obtain momentum features I output by the network t And splicing on the channels to obtain fusion characteristics.
The pose estimation network in this embodiment is composed of a PoseNet composed of an SE module and an LSTM, and takes the fusion characteristics as input to obtain a predicted pose transformation matrix T t ：
T t ＝PoseNet(V′ t ,L′ t ,I t )
The depth prediction network in this embodiment gives a video frame image I when predicting t It and its next frame image I s The depth prediction network (DepthNet) is input into the splice, and the depth prediction network calculates I by comparing the images of the previous and the next frames t Depth map D of (2) t ：
D t ＝DepthNet(I t ,I s )
The parameter optimization module in this embodiment calculates a loss function according to the depth map, the pose transformation matrix and the video frame, and adjusts parameters of the pose estimation network model according to the loss function. The loss function includes reconstruction error, depth smoothing loss, and geometric consistency loss, among others.
In calculating the reconstruction error, two consecutive frame pictures I are given t (target image) and I s (source image) now have their depth map D obtained through a depth prediction network (DepthNet) and a pose estimation network (pousenet) t And D s And pose transformation T between them t→s For I s Any point p in (2) s ∈I s The pixel-camera coordinate transformation formula can be used for obtaining the pixel-camera coordinate transformation formula in I t Position p 'on' s ：
Wherein, the representative position corresponds to I t Point p 'on' s And I s Point p on s Is the same point in the real world, K represents the camera reference matrix. Similarly, all I s All points on are projected to I t All I's are obtained by going up (ignoring points beyond the boundary after projection) s Point on I t The position of the upper part can be according to I t Reconstructing the RGB value of the pixel point at the corresponding position to reconstruct I' s And then reconstruct I' s And I s Comparing and calculating reconstruction error L pe Reconstruction error L pe The method comprises the following steps:
or L pe ＝λ 1 |I′ s -I s |+λ 2 SSIM(I′ s ,I s )
Wherein SSIM () is a structural similarity function, lambda 1 And lambda (lambda) 2 As the weight coefficient, I' s The method meets the following conditions:
in order to make the depth map output by the depth prediction network DepthNet have smoothness and scale consistency, a depth smoothing loss L needs to be adopted smooth And geometric consistency loss L geo Additional constraints are placed on DepthNet:
wherein D is t For the depth map corresponding to the image at time t in the sequence of video frames S,and->Respectively representing partial derivatives of the two-dimensional image coordinates in the x-direction and the y-direction, D' t And representing the depth map of the current t moment which is generated by reconstructing the pose transformation by utilizing the depth map of the subsequent moment.
To reduce invalid point matches during projection, the final loss function L all The expression of (2) is:
wherein l represents a scale number, ω 1 ,ω 2 ,ω 3 Weights representing reconstruction error, depth smoothing loss, and geometric consistency loss, respectively. By calculating L all The back propagation updates the network parameters with respect to the gradients of all network parameters, through multiple iterations until the model converges.
To verify the effectiveness of this embodiment, the performance of this embodiment and other unsupervised deep learning based positioning algorithms over the KITTI data set in recent years was compared, as indicated in tables 1 and 2.
Table 1 comparison of average translational error and average rotational error for various unsupervised neural network positioning algorithms
TABLE 2 comparison of absolute track errors and relative pose errors for various unsupervised neural network positioning algorithms
Table 1 is the test results on the KITTI data sets seq09 and seq10, all of which were trained using seq00 through seq08 of the KITTI data set, tested with seq09 and seq10, where t rel Representing rotational offset error, t tel Representing translational offset errors, representing how many meters of translational and rotational offset, respectively, are produced per 100 meters of travel on average. (unit: m/100 m).
Table 2 shows the results of the test of this embodiment and other algorithms on KITTI datasets seq09 and seq10, where ATE represents the root mean square of the predicted camera pose and the true camera pose difference, and RPE represents the relative pose error from frame to frame.
Fig. 4 is a schematic diagram comparing the path trace and the real path of the test output on the KITTI data sets seq09 and seq10 according to the present embodiment, and it can be seen from fig. 4 that the two curves are substantially identical, which indicates that the positioning of the present embodiment is accurate.
Claims (9)
1. The vision-inertial navigation-radar fusion self-positioning method based on the self-supervision neural network is characterized by comprising the following steps of:
acquiring a video frame sequence, inertial navigation data and a laser radar point cloud picture;
inputting the video frame sequence, the inertial navigation data and the laser radar point cloud image into a pose estimation network model to obtain relative pose estimation of an input video frame;
wherein, the pose estimation network model comprises:
the depth prediction network is used for obtaining a depth map according to the input video frame sequence;
the feature extraction network is used for extracting feature information from an input video frame sequence, inertial navigation data and a laser radar point cloud image respectively to obtain visual features, momentum features and radar features;
the feature fusion network is used for fusing the visual features and the radar features to obtain pre-fused corrected visual features and corrected radar features, and fusing the pre-fused corrected visual features and corrected radar features with the momentum features to obtain fusion features;
the pose estimation network is used for predicting a pose transformation matrix according to the fusion characteristics;
and the parameter optimization module is used for calculating a loss function according to the depth map, the pose transformation matrix and the video frame, and adjusting parameters of the pose estimation network model according to the loss function.
2. The self-supervised neural network based vision-inertial navigation-radar fusion self-localization method of claim 1, wherein the feature extraction network comprises:
a first feature extraction section that extracts visual features from the video frame sequence using a first convolutional network;
a second feature extraction part for extracting the motion feature from the inertial navigation data by adopting an LSTM network;
the third feature extraction part projects the laser radar point cloud image to a 2D plane, codes the laser radar point cloud image projected to the 2D plane in a three-channel coding mode, and extracts radar features by adopting a second convolution network; the first convolution network and the second convolution network have the same structure and share the weight of all network layers except the BN layer.
3. The self-supervision neural network-based vision-inertial navigation-radar fusion self-localization method according to claim 1 or 2, wherein the feature fusion network comprises:
the first fusion part is used for fusing the visual characteristics and the radar characteristics by adopting a channel exchange strategy to obtain pre-fused corrected visual characteristics and corrected radar characteristics;
and the second fusion part is used for carrying out channel splicing on the presplitting corrected visual features and the corrected radar features and the momentum features to obtain fusion features.
4. According to the weightsThe self-supervised neural network-based vision-inertial navigation-radar fusion self-localization method of claim 3, wherein the channel switching strategy isWherein V 'is' k,c Visual characteristics of the c-th channel output by the k-th layer convolution layer after the exchange strategy are represented, V k,c Visual characteristics of the c-th channel representing the output of the k-th convolutional layer, L k,c A radar feature representing the c-th channel of the k-th convolutional layer output, a v,k,c 、b v,k,c 、σ v,k,c Sum mu v,k,c Respectively representing the slope, bias, mean and variance of BN layer in the first convolution network; a, a l,k,c 、b l,k,c 、σ l,k,c Sum mu l,k,c Respectively representing the slope, bias, mean and variance of BN layer in the second convolution network; delta is a threshold.
5. The self-supervised neural network based vision-inertial navigation-radar fusion self-localization method of claim 1, wherein the loss function comprises reconstruction errors, depth smoothing losses, and geometric consistency losses, expressed as:wherein L is all As a loss function, L pe Representing reconstruction error, L smooth Representing depth smoothing loss, L geo Represents the geometric consistency loss, l represents the scale number, ω 1 ,ω 2 ,ω 3 Weights representing reconstruction error, depth smoothing loss, and geometric consistency loss, respectively.
6. The self-supervised neural network based vision-inertial navigation-radar fusion self-localization method of claim 5, wherein the reconstruction error is expressed as:wherein I is s P is the source image s For source image I s Point on I t For the target image, p' s For source image I s Point p on s Corresponding to the target image I t SSIM () is a structural similarity function, lambda 1 And lambda (lambda) 2 Is a weight coefficient.
7. The self-supervised neural network based vision-inertial navigation-radar fusion self-localization method of claim 5, wherein the reconstruction error is expressed as: l (L) pe ＝λ 1 |I′ s -I s |+λ 2 SSIM(I′ s ,I s ) Wherein I s For source image, I' s Is based on the target image I t Reconstructed source image, SSIM () is a structural similarity function, λ 1 And lambda (lambda) 2 Is a weight coefficient.
8. The self-supervised neural network based vision-inertial navigation-radar fusion self-localization method of claim 5, wherein the expression of the depth smoothing loss is:wherein D is t For the depth map corresponding to the image at time t in the video frame sequence S, I t For the target image +.>And->Representing partial derivatives of the x-direction and y-direction of the two-dimensional image coordinates, respectively.
9. The self-supervising neural network-based vision-inertial navigation-radar fusion self-positioning method of claim 5, wherein the expression of the geometric consistency loss is:wherein D is t For the depth map corresponding to the image at time t in the sequence of video frames S, D t ' represents the depth map at the current t-time generated by pose transformation reconstruction using the subsequent time depth map.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310495293.4A CN116824433A (en) | 2023-05-05 | 2023-05-05 | Visual-inertial navigation-radar fusion self-positioning method based on self-supervision neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310495293.4A CN116824433A (en) | 2023-05-05 | 2023-05-05 | Visual-inertial navigation-radar fusion self-positioning method based on self-supervision neural network |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116824433A true CN116824433A (en) | 2023-09-29 |
Family
ID=88111682
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310495293.4A Pending CN116824433A (en) | 2023-05-05 | 2023-05-05 | Visual-inertial navigation-radar fusion self-positioning method based on self-supervision neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116824433A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117058474A (en) * | 2023-10-12 | 2023-11-14 | 南昌航空大学 | Depth estimation method and system based on multi-sensor fusion |
-
2023
- 2023-05-05 CN CN202310495293.4A patent/CN116824433A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117058474A (en) * | 2023-10-12 | 2023-11-14 | 南昌航空大学 | Depth estimation method and system based on multi-sensor fusion |
| CN117058474B (en) * | 2023-10-12 | 2024-01-12 | 南昌航空大学 | Depth estimation method and system based on multi-sensor fusion |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110956651B (en) | Terrain semantic perception method based on fusion of vision and vibrotactile sense | |
| US20210142095A1 (en) | Image disparity estimation | |
| Vaudrey et al. | Differences between stereo and motion behaviour on synthetic and real-world stereo sequences | |
| CN112258618A (en) | Semantic mapping and positioning method based on fusion of prior laser point cloud and depth map | |
| CN110675418A (en) | Target track optimization method based on DS evidence theory | |
| Wang et al. | DV-LOAM: Direct visual lidar odometry and mapping | |
| CN112556719B (en) | Visual inertial odometer implementation method based on CNN-EKF | |
| US11948310B2 (en) | Systems and methods for jointly training a machine-learning-based monocular optical flow, depth, and scene flow estimator | |
| CN111860651A (en) | Monocular vision-based semi-dense map construction method for mobile robot | |
| Zhao et al. | RTSfM: Real-time structure from motion for mosaicing and DSM mapping of sequential aerial images with low overlap | |
| CN116824433A (en) | Visual-inertial navigation-radar fusion self-positioning method based on self-supervision neural network | |
| CN112258565B (en) | Image processing method and device | |
| CN116402876A (en) | Binocular depth estimation method, binocular depth estimation device, embedded equipment and readable storage medium | |
| Zhang et al. | Improved feature point extraction method of ORB-SLAM2 dense map | |
| Liu et al. | Real-time dense construction with deep multi-view stereo using camera and IMU sensors | |
| CN116740488B (en) | Training method and device for feature extraction model for visual positioning | |
| Leite et al. | Dense disparity maps from RGB and sparse depth information using deep regression models | |
| CN116630528A (en) | Static scene reconstruction method based on neural network | |
| Mathew et al. | Monocular depth estimation with SPN loss | |
| CN115482282A (en) | Dynamic SLAM method with multi-target tracking capability in automatic driving scene | |
| CN112069997B (en) | Unmanned aerial vehicle autonomous landing target extraction method and device based on DenseHR-Net | |
| Cai et al. | Bi-direction direct RGB-D visual odometry | |
| Yu et al. | Scene-aware refinement network for unsupervised monocular depth estimation in ultra-low altitude oblique photography of UAV | |
| CN114894191B (en) | Unmanned aerial vehicle navigation method suitable for dynamic complex environment | |
| Wei et al. | DeepSFM: Robust Deep Iterative Refinement for Structure From Motion |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |