CN112115786A - Monocular vision odometer method based on attention U-net - Google Patents

Abstract

The present application discloses a monocular visual odometry method and device based on attention U-net. The method will acquire a monocular image sequence, pass several adjacent images through the lens boundary recognition algorithm in turn, and identify the lens boundary from consecutive frames. , reducing the amount of subsequent calculations. Using an attention-based local feature enhancement method, an attention mechanism is added to the U‑net auto-encoding network, and the key frame sequence is input into the network. First, an attention-based method is used to distinguish texture areas and smooth areas. When locating high-frequency When the location of details, the attention mechanism acts as a feature selector, which enhances high-frequency features and suppresses noise in smooth regions. Input the feature-enhanced sequence into the final Bi‑LSTM (Bi‑directional Long Short‑Term Memory), and at each timestamp, roughly infer the t _n+1 frame according to the t _n frame. The image is used as input to the reverse sequence, and the camera pose for each timestamp is obtained according to the context.

Description

A monocular visual odometry method based on attention U-net

technical field

The present application relates to the fields of image enhancement and visual odometry, in particular to a monocular visual odometry method and system based on attention U-net.

Background technique

To complete autonomous navigation, a mobile robot first needs to determine its own position and attitude, that is, positioning. Visual Odometry (VO) is also proposed, which only uses the adjacent frame image streams obtained by a single or multiple cameras to estimate intelligence. Body posture, which can reconstruct the environment. VO mostly estimates the pose of the current frame by calculating the motion between frames. The purpose of VO is to calculate the motion trajectory of the camera between frames, thereby reducing drift for back-end closed-loop detection and mapping. The visual odometry based on deep learning does not need complex geometric operations, and the end-to-end operation form makes the method based on deep learning more concise.

On these basis, the researchers try to explore a new intelligent image homography calculation method. By collecting image sequences in real time, enriching the understanding of the image through the learning of neural network, obtaining the feature matching of adjacent frames, and obtaining the camera pose. Konda et al. were the first to implement deep learning-based VO by extracting visual motion and depth information. After estimating depth information using stereo images, a convolutional neural network (CNN) predicts changes in camera speed and orientation through a softmax function. Kendall et al. used CNN to implement an end-to-end localization system with RGB images as input and camera pose as output. The system proposes a 23-layer deep convolutional network PoseNet, which utilizes transfer learning to apply a database of classification problems to solve complex image regression problems. Compared with traditional local visual features, the trained features are more robust to illumination, motion blur, and camera intrinsic parameters. Costante et al. replaced RGB images with dense optical flow as the input of CNN. The system designs three different CNN architectures for feature learning of VO, and achieves the robustness of the algorithm under conditions such as image blur and underexposure. However, the experimental results also show that the training data has a great influence on the algorithm. When the motion between frames of the image sequence is large, the algorithm has a large error, which is mainly due to the lack of high-speed training samples in the training data.

In order to solve the above problems, in the existing feature extraction network, CNN is generally used to extract instance representations, but when encountering complex lighting and complex texture environments, it is difficult to extract effective features or the extracted features are not prominent enough. There is a large error; in the common CNN processing process, high-frequency information is easily lost in the latter layer, and the use of residual links can alleviate this loss and further enhance the high-frequency signal; in addition, in terms of data association, the existing visual odometry The method generally only considers the propagation of the forward sequence when processing the image sequence, and often ignores the role of the reverse sequence, and does not fully exploit the contextual relationship.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a monocular visual odometry method based on attention U-net, which is to overcome the above-mentioned problems or at least partially solve or alleviate the above-mentioned problems.

According to one aspect of the present invention, a sequence of monocular images is obtained, and several adjacent images are sequentially passed through the lens boundary recognition algorithm to identify the lens boundary from consecutive frames. Dimension reduction is performed to reduce the amount of subsequent calculations.

Among them, the key frame sequence method is used to identify the shot boundary of each frame by dividing each frame into 16×16 non-overlapping grids. The chi-square distance is used to calculate the corresponding grid histogram difference d between two adjacent frames:

H _i represents the histogram of the ith frame, and H _i+1 represents the histogram of the (i+1)th frame. I represents an image block at the same location in both frames. The histogram mean difference between two consecutive frames is calculated as follows

D is the average histogram difference of two consecutive frames, and d _k is the card variance between the kth image block. N represents the total number of image patches in the image. Identify shot boundaries on frames with histogram differences greater than a threshold Tshot:

The acquired image sequence is reduced to 1/4 of the original image by using the existing Gaussian pyramid and convolution with a stride of 2.

According to another aspect of the present invention, feature reconstruction identifies and enhances multi-texture regions to enhance high-frequency details. L _i-1 represents the input of the i-th convolutional layer, and the output of the i-th layer is expressed as:

L _i =σ(W _i *L _i-1 +b _i )

*Refers to the operation of convolution, σ refers to nonlinear activation (ReLU), and the feature reconstruction network consists of a convolutional layer for feature extraction, multiple stacked dense blocks, and a sub-pixel convolutional layer as an upsampling module. The dense block Composed of residual blocks (Resblock), it shows strong learning ability for object recognition. Let H _i be the input of the i-th residual module, and the output F _i can be expressed as:

F _i =φ _i (H _i ,W _i )+H _i

The residual block contains two convolutional layers. Specifically, the residual module function can be expressed as follows:

φ _i (H _i ; Wi )=σ ₂ (W _i ² *σ ₁ (W _i ¹ *H _{i )} )

Wherein W _i ¹ and W _i ² are the weights of the two convolutional layers respectively, σ ₁ and σ ₂ represent normalization, and the input of the i-th residual module H _i is the concatenation of the output of the previous residual module. Use a convolutional layer with a 1×1 kernel to control how much of the previous state should be retained. It adaptively learns the weights of different states. The input of the i-th residual block is represented as:

H _i =σ ₀ (W _i ⁰ *[F ₁ ,F ₂ ,...,F _i-1 ])

_Wi ⁰ represents a 1×1 convolution weight, and σ ₀ represents ReLU normalization.

Generate attention to determine the exact location of the texture. The U-net-like architecture is adopted, and the convolutional layers are replaced by dense blocks. In the compression path, the convolutional layers are first used to perform low-level feature extraction on the interpolated image. Then use 2×2 max pooling to reduce the dimensionality of the data and get a larger receptive field, using two pooling in the compression path. In the expansion path, a deconvolution layer is added to upsample the existing feature maps. By combining low-level features and high-level features in the expansion path, the output can accurately segment whether the region is textured and whether it needs a feature reconstruction network for repair. The feature channel output by the network is 1, and in the last layer, an activation sigmoid is used to control the output mask from 0 to 1. The higher the probability that the pixel belongs to the texture region, the closer the mask value is to 1, which means that these pixels need more attention, if not, the mask value will be closer to 0.

Residual learning based on attention, through the output of feature reinforcement network and the generation of mask, the image residual after reinforcement is obtained, and the feature before reinforcement is used as the input of attention generation network to interpolate to obtain the final attention Force produces results. Through the output of the feature reconstruction network and the point generation of the mask value, the residual of the ILR image is obtained, and the final attention generation result is obtained by adding the interpolated HR image as the input of the attention generation network. It can be expressed as:

HRc(i,j)= ^Fc (i,j)×M(i,j)+ ^{ILRc (} i,j)

where F=[F ¹ ; F ² ; F ³ ] is the output of the feature reconstruction network, and the number of output channels is 3. M is the mask value. ILR=[ILR ¹ , ILR ² , ILR ³ ] is the interpolated image, and HR=[HR ¹ , HR ² , HR ³ ] is the final enhancement result of the method. i and j represent the pixel position in each channel, and c represents the channel index.

According to yet another aspect of the present invention, the pose of each time-stamped camera is obtained from the augmented image sequence. Using a CNN network containing 4 convolutions, one layer of 7 × 7, two layers of 5 × 5, and finally using a 3 × 3 convolution kernel, the output of the CNN is modeled by Bi-LSTM sequence, given the features at time t , then the Bi-LSTM at time t is updated as:

s _t =f(Ux _t +Ws _t-1 )

s' _t =f(U'x _t +W's _t+1 )

A _t =f(WA _t-1 +Ux _t )

A' _t =f(W'A' _t+1 +U'x _t )

y _t =g(VA ₂ +V'A' ₂ )

where s _t and s' _t represent the forward and reverse memory variables at time t, A _t and A' _t represent the forward and reverse hidden layer variables at time t, and y _t represents the output variable. In addition, f and g represent nonlinear activation functions, and U, U', W, W', V, and V' represent weight matrices corresponding to their respective variables. Then two fully connected layers are added behind Bi-LSTM to integrate the trained image sequence information into a final accurate 6-DOF pose, including three rotations and three translations. What the inner camera does is linear motion:

Among them, P represents the interception rate of the image center at the current time t, v represents the current speed, T represents the sampling period of the camera, α is a constant, n represents the nth frame at the current time, and 2.57 is the empirical value obtained from experiments.

Description of drawings

Hereinafter, some specific embodiments of the present application will be described in detail by way of example and not limitation with reference to the accompanying drawings. The same reference numbers in the figures designate the same or similar parts or parts. It will be understood by those skilled in the art that the drawings are not necessarily to scale. In the attached picture:

Fig. 1 is the schematic flow sheet of the present application method;

2 is a schematic structural block diagram of a monocular visual odometry method based on attention U-net according to an embodiment of the present application;

FIG. 3 schematically shows the feature enhancement model structure of the preferred embodiment of the present invention;

Figure 4 shows the context-based Bi-LSTM model structure;

FIG. 5 is a computing device provided by an embodiment of the present application;

FIG. 6 is a computer-readable storage medium provided by an embodiment of the present application;

FIG. 7 is an odometer operation result generated based on the KITTI data set 01 sequence and 07 sequence in an embodiment of the present application.

Detailed ways

The present invention will be described below with reference to the accompanying drawings and specific embodiments.

FIG. 1 is a schematic flowchart of a monocular visual odometry method based on attention U-net according to an embodiment of the present application. Referring to FIG. 1 , a monocular visual odometry method and system based on attention U-net provided by the embodiment of the present application may include:

Step S1: Obtain a monocular image sequence, filter out the key frame sequence through the lens boundary recognition method, and reduce the dimension of the image through the Gaussian pyramid.

Step S2: Using a convolutional neural network, a fully convolutional network model composed of multiple stacked residual blocks and sub-pixel deconvolution to reconstruct the dimensionality-reduced key frame.

Step S3: Using a U-net-like structure, replace the convolution block with a residual block to solve the problem of gradient disappearance due to depth, input the key frame after dimension reduction, and generate a texture mask of the image.

Step S4: Complete the texture alignment of the entire learning scene instance using the corresponding entity feature consistency theory in the scene, and use the obtained alignment relationship to fuse the data, so as to realize the authenticity of the visual input to strengthen the local scene features.

Step S5 : Based on the enhanced image, combined with the proposed artificial image synthesis method, according to the image at time t, an image at time t+n (t≤3) is generated to provide reverse data for pose estimation.

Step S6: Using a CNN network containing 4 convolutions, fuse the enhanced image sequence and artificially synthesized data and reduce the dimension, and perform the sequence modeling of Bi-LSTM on the output of the CNN, and finally infer the bit of each timestamp. Posture changes.

The embodiment of the present application provides a monocular visual odometry method based on the attention U-net. In the method provided by the embodiment of the present application, a real-time image sequence is obtained first, and a shot boundary recognition method is used to screen out key frames, and the The key frame sequence is dimensionally reduced, and then the feature reconstruction network is used to restore high-frequency details and reconstruct the visual image. In addition, the key frame sequence uses an attention-based U-net-like network to distinguish texture areas and smooth areas. When locating high-frequency details location, the attention mechanism acts as a feature selector, which enhances high-frequency features and suppresses noise in smooth regions, generating texture masks. Through the output of the feature enhancement network and the generation of the mask, the image residual after enhancement is obtained, and the feature before enhancement is used as the input of the attention generation network to interpolate to obtain the final attention generation result.

The experimental data set used in the method is the KITTI data set (co-founded by Karlsruhe Institute of Technology in Germany and Toyota American Institute of Technology), which is currently the largest international computer vision algorithm evaluation data in autonomous driving scenarios. set. The KITTI data acquisition platform includes 2 grayscale cameras, 2 color cameras, a Velodyne3D lidar, 4 optical lenses, and a GPS navigation system. The entire dataset consists of 389 pairs of stereo images and optical flow maps (each image contains up to 15 vehicles and 30 pedestrians with varying degrees of occlusion), 39.2 kilometers of visual odometry sequences, and more than 200,000 images of 3D annotated objects.

S1. Identify the shot boundary of each frame by dividing each frame into non-overlapping grids of size 16×16. The chi-square distance is used to calculate the corresponding grid histogram difference d between two adjacent frames:

H _i represents the histogram of the ith frame, and H _i+1 represents the histogram of the (i+1)th frame. I represents an image block at the same location in both frames. The histogram mean difference between two consecutive frames is calculated as follows:

D is the average histogram difference of two consecutive frames, and d _k is the card variance between the kth image block. N represents the total number of image patches in the image. Identify shot boundaries on frames with histogram differences greater than a threshold Tshot:

The acquired image sequence is reduced to 1/4 of the original image by using the existing Gaussian pyramid and convolution with a stride of 2.

S2, feature reconstruction, identify and strengthen multi-texture areas, and strengthen high-frequency details. L _i-1 represents the input of the i-th convolutional layer, and the output of the i-th layer is expressed as:

L _i =σ(W _i *L _i-1 +b _i )

*Refers to the operation of convolution, σ refers to nonlinear activation (ReLU), and the feature reconstruction network consists of a convolutional layer for feature extraction, multiple stacked dense blocks, and a sub-pixel convolutional layer as an upsampling module. The dense block Composed of residual blocks (Resblock), it shows strong learning ability for object recognition. Let H _i be the input of the i-th residual module, and the output F _i can be expressed as:

F _i =φ _i (H _i ,W _i )+H _i

The residual block contains two convolutional layers. Specifically, the residual module function can be expressed as follows

φ _i (H _i ; Wi )=σ ₂ (W _i ² *σ ₁ (W _i ¹ *H _{i )} )

Wherein W _i ¹ and W _i ² are the weights of the two convolutional layers respectively, σ ₁ and σ ₂ represent normalization, and the input of the i-th residual module H _i is the concatenation of the output of the previous residual module. Use a convolutional layer with a 1×1 kernel to control how much of the previous state should be retained. It adaptively learns the weights of different states. The input of the i-th residual block is represented as:

H _i =σ ₀ (W _i ⁰ *[F ₁ ,F ₂ ,...,F _i-1 ])

_Wi ⁰ represents a 1×1 convolution weight, and σ ₀ represents ReLU normalization.

S3. The network consists of shrinking paths, expanding paths and skip connections, and it takes a bilinearly interpolated original image (at the desired size) as input. The redundancy added by interpolation can reduce the information loss of forward propagation, which is beneficial to the accurate segmentation of textured and smooth regions. Using a U-net-like structure, the convolution blocks are replaced with residual blocks to solve the problem of gradient disappearance due to depth. Due to the reusability of residual networks, stacking residual blocks can greatly reduce the number of parameters. The specific implementation process of real-time scene feature texture recovery based on attention mechanism is as follows:

1) In the compression path, low-level features are first extracted using convolutional layers. Then use max pooling to reduce the dimensionality of the data and get a larger receptive field. By using pooling twice in the compression path, the network can use a larger area to predict whether a pixel belongs to a high frequency region.

2) In the expansion path, a deconvolution layer is added to upsample the existing feature maps. Low-level features contain a lot of useful information, and a lot of information is lost during forward propagation. By combining low-level features and high-level features in the expansion path, the output can accurately segment whether the region is textured and whether it needs a feature reconstruction network for repair. The last layer of the network uses a sigmoid activation function to control the output mask from 0 to 1. If a pixel has a higher probability of belonging to a textured region, the mask value will be closer to 1, which means those pixels need more attention; if not, the mask value will be closer to 0.

S4. Through the output of the feature reconstruction network and the point generation of the mask value, the residual of the HR image is obtained, and the final attention generation result is obtained by adding the interpolated LR image as the input of the attention generation network. It can be expressed as:

HRc(i,j)= ^Fc (i,j)×M(i,j)+ ^{ILRc (} i,j)

where F=[F ¹ ; F ² ; F ³ ] is the output of the feature reconstruction network, and the number of output channels is 3. M is the mask value. ILR=[ILR ¹ , ILR ² , ILR ³ ] is the interpolated image. HR=[HR ¹ , HR ² , HR ³ ] is the final reinforcement result of the method. i and j represent the pixel position in each channel, and c represents the channel index. The attention generating network makes the residual values from textured regions larger, while the residual values from non-textured regions tend to be zero. Mask M is a feature selector that enhances high-frequency features and suppresses noise, so in the output image, high-frequency details will be recovered, and where smooth, noise will be removed.

S5. A method of artificially synthesizing pictures is proposed, and it is believed that the camera moves in a straight line in a short period of time:

Among them, P represents the interception rate of the image center at the current time t, v represents the current speed, T represents the sampling period of the camera, α is a constant, n represents the nth frame at the current time, and 2.57 is the empirical value obtained from experiments.

S6. Obtain the pose of each timestamped camera according to the enhanced image sequence. Using a CNN network containing 4 convolutions, one layer of 7 × 7, two layers of 5 × 5, and finally using a 3 × 3 convolution kernel, the output of the CNN is modeled by Bi-LSTM sequence, given the features at time t , then the Bi-LSTM at time t is updated as:

s _t =f(Ux _t +Ws _t-1 )

s' _t =f(U'x _t +W's _t+1 )

A _t =f(WA _t-1 +Ux _t )

A' _t =f(W'A' _t+1 +U'x _t )

y _t =g(VA ₂ +V'A' ₂ )

where s _t and s' _t represent the forward and reverse memory variables at time t, A _t and A' _t represent the forward and reverse hidden layer variables at time t, and y _t represents the output variable. In addition, f and g represent nonlinear activation functions, and U, U', W, W', V, and V' represent weight matrices corresponding to their respective variables. The Bi-LSTM is then followed by two fully connected layers to integrate the trained image sequence information into a final accurate 6DOF pose, including three rotations and three translations.

FIG. 2 is a schematic structural block diagram of a monocular visual odometry method based on attention U-net according to an embodiment of the present application. The apparatus may generally include a preprocessing module, a feature reconstruction module, an attention enhancement module, a residual learning module, an artificial data synthesis module, and a context-based pose inference module.

Figure 3 shows the structure of a feature enhancement model according to a preferred embodiment of the present invention, the model consists of a feature reconstruction module, an attention-based local feature enhancement module, and a residual learning module

Figure 4 shows the context-based Bi-LSTM model structure, taking the output of the previous CNN model as the Bi-LSTM input to obtain the camera pose estimate for each timestamp.

The present invention aims to protect a local feature enhancement method, a context-based pose estimation method, and a deep learning-based visual odometry method, which mainly estimates the camera pose based on local representation alignment. In environments with complex textures or lighting, the texture alignment is not accurate enough. The method proposed in this scheme enhances high-frequency details through feature reconstruction, identifies and strengthens multi-texture areas, and generates textures by adding attention to U-net. The mask determines the exact location of the texture. In order to enhance as many texture details as possible, an attention mechanism that requires a large receptive field is provided, and then superimposed with the reconstructed image to complete feature enhancement. In addition, camera pose estimation is a serialization problem. The existing method only considers the forward sequence of images. This scheme uses artificial synthetic data, adds reverse constraints, and fully mines context constraints to obtain more accurate cameras at each timestamp. pose.

The above and other objects, advantages and features of the present application will be more apparent to those skilled in the art from the following detailed description of the specific embodiments of the present application in conjunction with the accompanying drawings.

An embodiment of the present application further provides a computing device, referring to FIG. 5 , the computing device includes a memory 520, a processor 510, and a computer program stored in the memory 520 and executable by the processor 510, the computer program Space 530 stored in memory 520 for program code which, when executed by processor 510, implements for performing any one of the methods 531 according to the invention.

Embodiments of the present application also provide a computer-readable storage medium. Referring to Figure 6, the computer-readable storage medium comprises a storage unit for program codes provided with a program 531' for performing the method steps according to the invention, the program being executed by a processor.

Embodiments of the present application also provide a computer program product including instructions. The computer program product, when run on a computer, causes the computer to perform the method steps according to the invention.

In the above-mentioned embodiments, it may be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented in software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer loads and executes the computer program instructions, all or part of the processes or functions described in the embodiments of the present application are generated. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions may be downloaded from a website site, computer, server or data center Transmission to another website site, computer, server, or data center is by wire (eg, coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (eg, infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, a data center, or the like that includes an integration of one or more available media. The usable media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, DVD), or semiconductor media (eg, Solid State Disk (SSD)), among others.

Professionals should be further aware that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. Interchangeability, the above description has generally described the components and steps of each example in terms of function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each particular application, but such implementations should not be considered beyond the scope of this application.

Those of ordinary skill in the art can understand that all or part of the steps in the method of implementing the above embodiments can be completed by instructing the processor through a program, and the program can be stored in a computer-readable storage medium, and the storage medium is non-transitory ( English: non-transitory) media, such as random access memory, read only memory, flash memory, hard disk, solid state disk, magnetic tape (English: magnetic tape), floppy disk (English: floppy disk), optical disc (English: optical disc) and any combination thereof.

The final example effect is shown in Figure 7. Based on the operation results of the 01 sequence and the 07 sequence of the KITTI data set, it shows the comparison between the camera pose motion trajectory estimated by the model and the real trajectory.

The above are only the preferred specific embodiments of the present application, but the protection scope of the present application is not limited to this. Substitutions should be covered within the protection scope of this application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims (8)

1. A monocular visual odometry method based on attention U-net, comprising: Obtain a monocular image sequence, pass several adjacent images through the lens boundary recognition algorithm in turn, and identify the lens boundary from consecutive frames. The entire module uses key frames to perform operations, and uses the Gaussian pyramid to reduce the dimension of the original image and reduce subsequent calculations. quantity. Using an attention-based local feature enhancement method, an attention mechanism is added to the U-net self-encoding network, and the key frame sequence is input into the network. First, an attention-based method is used to distinguish texture areas and smooth areas. When locating high-frequency When the location of details, the attention mechanism acts as a feature selector, which enhances high-frequency features and suppresses noise in smooth regions. Input the feature-enhanced sequence into the final Bi-LSTM, at each timestamp, roughly infer the image of t _n+1 frame as the input of the reverse sequence according to t _n frame, and obtain the camera position of each timestamp. posture. 2 . The method of claim 1 , wherein the calculating key frame sequence method identifies the shot boundary of each frame by dividing each frame into non-overlapping grids of size 16×16. 3 . The chi-square distance is used to calculate the corresponding grid histogram difference d between two adjacent frames:

H _i represents the histogram of the ith frame, and H _i+1 represents the histogram of the (i+1)th frame. I represents an image block at the same location in both frames. The histogram mean difference between two consecutive frames is calculated as follows

D is the average histogram difference of two consecutive frames, and d _k is the card variance between the kth image block. N represents the total number of image patches in the image. Identify shot boundaries on frames with histogram differences greater than a threshold Tshot:

The acquired image sequence is reduced to 1/4 of the original image by using the existing Gaussian pyramid and convolution with a stride of 2. 3. The method according to claim 1, reconstruction of features, identification and enhancement of multi-texture regions, and enhancement of high-frequency details. L _i-1 represents the input of the i-th convolutional layer, and the output of the i-th layer is expressed as: L _i =σ(W _i *L _i-1 +b _i ) *Refers to the operation of convolution, σ refers to nonlinear activation (ReLU), and the feature reconstruction network consists of a convolutional layer for feature extraction, multiple stacked dense blocks, and a sub-pixel convolutional layer as an upsampling module. The dense block Composed of residual blocks (Resblock), it shows strong learning ability for object recognition. Let H _i be the input of the i-th residual module, and the output F _i can be expressed as: F _i =φ _i (H _i ,W _i )+H _i The residual block contains two convolutional layers. Specifically, the residual module function can be expressed as follows φ _i (H _i ; Wi )=σ ₂ (W _i ² *σ ₁ (W _i ¹ *H _{i )} ) Wherein W _i ¹ and W _i ² are the weights of the two convolutional layers respectively, and σ ₁ and σ ₂ indicate that the input of the normalized ith residual module _Hi is the concatenation of the output of the previous residual module. Use a convolutional layer with a 1×1 kernel to control how much of the previous state should be retained. It adaptively learns the weights of different states. The input of the i-th residual block is represented as: H _i =σ ₀ (W _i ⁰ *[F ₁ ,F ₂ ,...,F _i-1 ]) _Wi ⁰ represents a 1×1 convolution weight, and σ ₀ represents ReLU normalization. 4. The method of claim 1, wherein attention is generated to determine the exact location of the texture. The U-net-like architecture is adopted, and the convolutional layers are replaced by dense blocks. In the compression path, the convolutional layers are first used to perform low-level feature extraction on the interpolated image. Then use 2×2 max pooling to reduce the dimensionality of the data and get a larger receptive field, using two pooling in the compression path. In the expansion path, a deconvolution layer is added to upsample the existing feature maps. By combining low-level features and high-level features in the expansion path, the output can accurately segment whether the region is textured and whether it needs a feature reconstruction network for repair. The feature channel output by the network is 1, and in the last layer, an activation sigmoid is used to control the output mask from 0 to 1. The higher the probability that the pixel belongs to the texture region, the closer the mask value is to 1, which means that these pixels need more attention, if not, the mask value will be closer to 0. 5 . The method according to claim 1 , wherein the residual learning based on attention obtains the image residual after enhancement through the output of the feature enhancement network and the generation of the mask, and at the same time, the feature before enhancement is obtained. 6 . Interpolate as the input of the attention generation network to get the final attention generation result. Through the output of the feature reconstruction network and the point generation of the mask value, the residual of the HR image is obtained, and the final attention generation result is obtained by adding the interpolated LR image as the input of the attention generation network. It can be expressed as: HRc(i,j)= ^Fc (i,j)×M(i,j)+ ^{ILRc (} i,j) where F=[F ¹ ; F ² ; F ³ ] is the output of the feature reconstruction network, and the number of output channels is 3. M is the mask value. ILR=[ILR ¹ , ILR ² , ILR ³ ] is the interpolated image. HR=[HR ¹ , HR ² , HR ³ ] is the final reinforcement result of the method. i and j represent the pixel position in each channel, and c represents the channel index. 6. The method of claim 1, wherein the pose of each time-stamped camera is obtained from the enhanced image sequence. Using a CNN network containing 4 convolutions, one layer of 7 × 7, two layers of 5 × 5, and finally using a 3 × 3 convolution kernel, the input of the CNN is modeled by Bi-LSTM sequence, given the features at time t , then the Bi-LSTM at time t is updated as: s _t =f(Ux _t +Ws _t-1 ) s' _t =f(U'x _t +W's _t+1 ) A _t =f(WA _t-1 +Ux _t ) A' _t =f(W'A' _t+1 +U'x _t ) y _t =g(VA ₂ +V'A' ₂ ) where s _t and s' _t represent the forward and reverse memory variables at time t, A _t and A' _t represent the forward and reverse hidden layer variables at time t, and y _t represents the output variable. In addition, f and g represent nonlinear activation functions, and U, U', W, W', V, and V' represent weight matrices corresponding to their respective variables. The Bi-LSTM is then followed by two fully connected layers to integrate the trained image sequence information into a final accurate 6DOF pose, including three rotations and three translations. 7. The method according to claim 1 and claim 6, characterized in that, in practical application, it is unrealistic to obtain a reverse sequence, and in order to add a reverse constraint, a method for artificially synthesizing pictures is proposed, and it is considered that the camera does the work in a short time. The linear motion is:

Among them, P represents the interception rate of the image center at the current time t, v represents the current speed, T represents the sampling period of the camera, α is a constant, n represents the nth frame at the current time, and 2.57 is the empirical value obtained from experiments. 8 . The method according to claim 1 , wherein, in a laboratory environment, the data set used in the method is the KITTI data set. 9 .

Images