CN114998411B - Self-supervision monocular depth estimation method and device combining space-time enhancement luminosity loss - Google Patents

Abstract

The invention relates to a self-supervised monocular depth estimation method and device combined with spatiotemporal enhanced photometric loss, wherein the method includes: acquiring several adjacent frame images in an image sequence; inputting the images into a trained deep learning network Depth information and pose information are obtained, in which the photometric loss information of the deep learning network is obtained based on the spatial transformation model of depth sensing pixel correspondence, and an omnidirectional automatic mask is used to avoid the pixels of moving objects from participating in the calculation of photometric errors. The present invention can improve the accuracy of photometric loss and thereby better supervise the learning of deep networks.

Description

Self-supervised monocular depth estimation method and device combined with spatiotemporal enhanced photometric loss

Technical field

The present invention relates to the field of computer vision technology, and in particular to a self-supervised monocular depth estimation method and device combined with spatiotemporal enhanced photometric loss.

Background technique

Estimating the depth information of a scene from an image, that is, image depth estimation, is a basic and very important task in current computer vision. A good image depth estimation algorithm can be applied to outdoor driving scenes and indoor small robots, etc., and has huge application value. When a robot or self-driving car is working, it uses a depth estimation algorithm to obtain scene depth information to assist the robot in path planning or obstacle avoidance for its next movement.

Depth estimation using images is divided into supervised and self-supervised methods. Supervised methods mainly use neural networks to establish the mapping between images and depth maps, and train under the supervision of true values, so that the network gradually has the ability to fit the depth. However, due to the higher true value cost of supervised methods, self-supervised methods have gradually become mainstream in recent years. Compared with methods that require binocular image pairs for training, methods based on sequence images have become widely concerned by researchers because of their wider scope of application.

The self-supervised monocular depth framework based on sequence images mainly includes a depth estimation network and a pose estimation network, which respectively predict the depth of the target frame and the pose transformation of the target frame and source frame. Combining the estimated depth and pose, the source frame can be transformed into the coordinate system of the target frame to obtain the reconstructed image. The difference in photometric loss between the target frame and the reconstructed image can be used to supervise the simultaneous training of the two networks. As the photometric loss decreases, the depth estimated by the network becomes increasingly accurate.

A spatial transformation model needs to be used when generating photometric loss. Although the existing spatial transformation model conforms to the theoretical rigid body transformation method, due to the error of the translation vector in the pose during the calculation process, a certain depth estimation error will occur, that is to say , the greater the depth, the greater the depth estimation error. In addition, in order to solve the problem of inaccurate photometric loss caused by moving pixels that violate photometric consistency in the image, the main idea of the existing method is to find and filter out pixels whose brightness does not become smaller from one frame to another during the training process. , the generated binary mask, but this binary mask can only identify objects in the same direction as the camera movement.

Contents of the invention

The inventor of the present invention found that the greater the depth, the greater the depth estimation error is for the following reasons: the purpose of spatial transformation is to make the corresponding pixels in the target frame and the source frame coincide on the pixel plane after spatial transformation. If the nearby point P _N is used to solve the corresponding relationship between the corresponding pixels p _t and p _s , as shown in Figure 1. The principle of self-supervised depth estimation is to make the estimated pose and depth more accurate by minimizing the photometric error of p _t and p _s . For the near area, as shown in Figure 1, with a certain number of points, only when p _t and the transformed point p _F are relatively coincident, the estimated pose can be more accurate and the depth performance can be better. For distant areas, as shown in Figure 2, only the predicted rotation matrix needs to be accurate to ensure that the photometric error of p _t and p _s becomes smaller. Therefore, if the distance is not distinguished and the estimated rotation matrix and translation vector are used to construct the photometric error , the photometric error uncertainty will be greatly increased, resulting in poor depth estimation results.

The technical problem to be solved by the present invention is to provide a self-supervised monocular depth estimation method and device that combines spatiotemporal enhanced photometric loss, which can improve the accuracy of photometric loss and thereby better supervise the learning of deep networks.

The technical solution adopted by the present invention to solve the technical problem is to provide a self-supervised monocular depth estimation method combined with spatiotemporal enhanced photometric loss, which includes the following steps:

Obtain several adjacent frames of images in the image sequence;

The image is input into the trained deep learning network to obtain depth information and pose information. The photometric loss information of the deep learning network is obtained based on the spatial transformation model of depth sensing pixel correspondence, and uses omnidirectional automatic masking. Membrane to prevent the pixels of moving objects from participating in the calculation of photometric errors.

The photometric loss information is obtained based on the spatial transformation model corresponding to the depth sensing pixels as follows:

Use the homography matrix to perform spatial transformation on the distant area, and construct a first reconstruction map; wherein, the far area regards the far area as an infinite plane;

Use the basic matrix to perform spatial transformation and construct a second reconstruction map;

The photometric error map based on the first reconstructed image and the photometric error map based on the second reconstructed image are solved through two pixel correspondences, and then the minimum value is selected pixel by pixel to obtain the final photometric loss information.

The calculation of using omnidirectional automatic mask to avoid the participation of pixels of moving objects in the photometric error is as follows:

Predict the initial depth and initial pose of the target frame through the pre-trained network, and generate an initial reconstruction map;

Add interference terms to the initial pose, and use spatial transformation to obtain several hypothetical reconstructed frames; use the hypothetical reconstructed frames, combined with the luminosity of the target frame, to generate multiple photometric error maps, and use the Multiple photometric error maps obtain multiple binary masks;

Select the minimum value from the plurality of binary masks as the final mask.

The interference term is a translational disturbance term, including: [t _max ,0,0], [-t _max ,0,0], [0,0,t _max ] and [0,0,-t _max ], where , t _max represents the maximum value in the initialized translation vector.

The technical solution adopted by the present invention to solve the technical problem is to provide a self-supervised monocular depth estimation device that combines spatiotemporal enhanced photometric loss, including:

The acquisition module is used to acquire several adjacent frame images in the image sequence;

An estimation module, used to input the image into a trained deep learning network to obtain depth information and pose information; the photometric loss information of the deep learning network is obtained based on the spatial transformation model of the depth sensing pixel correspondence module, and uses The omnidirectional automatic mask module prevents the pixels of moving objects from participating in the calculation of photometric errors.

The depth sensing pixel correspondence module includes:

The first construction unit is used to use the homography matrix to perform spatial transformation on the far area and construct the first reconstruction map; wherein the far area regards the far area as an infinite plane;

The second construction unit is used to perform spatial transformation using the basic matrix and construct the second reconstruction map;

The photometric loss information acquisition unit is used to solve the photometric error map based on the first reconstructed image and the photometric error map based on the second reconstructed image through two pixel correspondences, and then select the minimum value pixel by pixel to obtain the final Photometric loss information.

The omnidirectional automatic mask module includes:

An initial reconstruction map generation unit is used to predict the initial depth and initial pose of the target frame through the pre-trained network, and generate an initial reconstruction map;

A binarized mask generation unit is used to add interference terms to the initial pose, and use spatial transformation to obtain several hypothetical reconstructed frames; use the hypothetical reconstructed frames, combined with the luminosity of the target frame, to generate A plurality of photometric error maps, and using the multiple photometric error maps to obtain a plurality of binary masks;

A mask selection unit is used to select a minimum value from the plurality of binary masks as the final mask.

The interference term is a translational disturbance term, including: [t _max ,0,0], [-t _max ,0,0], [0,0,t _max ] and [0,0,-t _max ], where , t _max represents the maximum value in the initialized translation vector.

beneficial effects

Due to the adoption of the above technical solution, the present invention has the following advantages and positive effects compared with the existing technology: the present invention uses a depth-sensing pixel correspondence method to mine pixel correspondences in distant areas, improving the The problem of inaccurate regional pixel correspondence, and the omnidirectional automatic masking method is used to obtain an omnidirectional binary mask to avoid the pixels of moving objects from participating in the calculation of photometric errors. The present invention improves the accuracy of photometric loss by improving spatial transformation and generating automatic masks for dynamic objects, thereby better supervising the learning of deep networks.

Description of the drawings

Figure 1 is a schematic diagram of the near point pose solution;

Figure 2 is a schematic diagram for solving the pose of a distant point;

Figure 3 is a schematic diagram of the basic framework of Monodepth2;

Figure 4 is a schematic diagram of the generation of photometric loss in the first embodiment of the present invention;

Figure 5 is a schematic diagram of an omnidirectional automatic mask in the first embodiment of the present invention.

Detailed ways

The present invention will be further described below in conjunction with specific embodiments. It should be understood that these examples are only used to illustrate the invention and are not intended to limit the scope of the invention. In addition, it should be understood that after reading the teachings of the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms also fall within the scope defined by the appended claims of this application.

The first embodiment of the present invention relates to a self-supervised monocular depth estimation method combined with spatiotemporal enhanced photometric loss, which includes the following steps: acquiring several adjacent frame images in an image sequence; inputting the images into trained deep learning Depth information and pose information are obtained from the network. The photometric loss information of the deep learning network is obtained based on the spatial transformation model corresponding to the depth sensing pixels, and an omnidirectional automatic mask is used to avoid the pixels of moving objects from participating in the calculation of photometric errors. .

The method of this implementation mode can be directly used in general self-supervised monocular depth estimation. Any work based on the SfMLearner framework as the implementation principle can use the method of this implementation mode. It only needs to adopt the space transformation model based on depth sensing pixel correspondence of this embodiment for the spatial transformation part in the original framework, and adopt the omnidirectional automatic mask of the present application for the automatic mask part.

The following takes the basic framework of Monodepth2 of Godard et al. as an example to further illustrate the present invention.

In order to make it easier to understand, the overall framework of Monodepth2 is introduced below, as shown in Figure 3. Its input is the RGB image of three adjacent frames in the sequence; the output is the depth of the target frame, and the pose transformation between the target frame and the source frame.

The basic framework of this embodiment is the same as Figure 3 . Since this implementation mainly improves the method of spatial transformation to generate photometric loss and automatic masking, we will first focus on these two parts of Monodepth2:

Monodepth2 uses the same spatial transformation model as SfMLearner, based on the depth D _t of the target frame I _t and the pose T _{t→s of} the target frame I _t and the source frame I s = [R _t→s | _{t t→s} ]. For the corresponding pixels p _t and p _s between the target frame and the source frame, if they correspond to the same 3D point, they should satisfy:

D _s K ^-1 p _s =D _t K ^-1 p _t

Among them, K is the internal parameter of the camera. Since monocular depth has scale ambiguity, it can be transformed into the following formula for spatial transformation:

p _s ~KT _t→s D _t K ^-1 p _t

In the spatial geometric transformation, KT _t→s K ^-1 is defined as the basic matrix F, which is used to correspond to the pixels between frames. This relationship can then be used to construct a reconstructed frame

According to the target frame and reconstructed frame, the photometric loss pe can be constructed, which consists of L1 error and structural similarity (SSIM) error, as follows:

α is a hyperparameter, Monodepth2 is set to 0.85

The automatic masking of Monodepth2 is mainly to solve the problem of inaccurate photometric loss caused by moving pixels that violate photometric consistency in the image. The main idea is to filter out pixels whose brightness does not become smaller from one frame to another during the training process. The generated binary mask μ is as follows:

[] is Iversonbracket, used to generate binary masks. I _t is the target frame, I _s is the source frame, is the reconstructed frame obtained by spatial transformation.

Regarding the photometric loss generated by spatial transformation, this embodiment obtains the photometric loss amount based on the spatial transformation model corresponding to the depth sensing pixel. As shown in Figure 4, the details are as follows:

In the process of space transformation, the far enough area can be regarded as an infinite plane, and the plane satisfies:

n ^T P+D＝0

Among them, n is the normal vector of the plane, P is a three-dimensional point on the plane, and D is the depth of the point. After transformation, it can be obtained:

Putting it into the relationship of spatial transformation, we can get:

When D _t is infinite, that is, for the infinite plane:

p _s ~KR _t→s D _t K ^-1 p _t

KR _t→s K ^-1 is defined as the homography matrix H∞ at infinity, so for distant areas only the rotation matrix is used to perform spatial transformation and then construct the reconstruction map In order to distinguish, the reconstructed graph obtained by using the basic matrix is expressed as Since the depth estimated by monocular scale estimation has scale ambiguity, it is impossible to directly select the two pixel correspondences through the predicted depth. Therefore, this embodiment designs an adaptive selection method. Specifically, two photometric error maps are obtained through two pixel correspondences, and then the minimum value is selected pixel by pixel, that is, the final photometric error is:

For omnidirectional automatic masking, this implementation directly inputs the image sequence into the module. After obtaining the masking result, it is applied to the photometric error to block out the unreliable parts, as shown in Figure 5. The details are as follows:

This implementation introduces a Monodepth2 pre-training network to predict the initial depth D _init and initial frame pose T _init of the target frame, and further generates an initial reconstruction map I _init . Since the depth and pose are already relatively accurate, the photometric error in areas that meet photometric consistency is already smaller, but the areas that do not meet photometric consistency have the potential to become smaller.

In response to this idea, by adding interference terms to the initial pose, some post-interference poses are introduced, and some hypothetical reconstructed frames are obtained by using spatial transformation. Using these reconstructed frames I _i , where i∈{1,2,…}, combined with the luminosity of the target frame, multiple photometric error maps can be generated, and multiple binary masks can be obtained by using the sizes of these photometric error values. , corresponding to the pixels of moving objects in various directions, as follows:

M _i =[pe(I _t ,I _init ),pe(I _t ,I _i )]

In order to capture objects moving in all directions, the minimum value of each generated mask is taken to obtain the final mask, that is:

_MoA =min(M ₁ ,M ₂ ,…)

During the implementation process of this implementation, only the translation vector is perturbed. The specific translation perturbation term t _{i is} : t ₁ =[t _max ,0,0], t ₂ =[-t _max ,0,0], t ₃ =[0,0,t _max ] and t ₄ =[0,0,-t _max ], where t _max is the maximum value of the initialized translation vector.

It is not difficult to find that the present invention uses a depth-sensing pixel correspondence method to mine pixel correspondences in distant areas, improves the problem of inaccurate pixel correspondence in distant areas, and uses omnidirectional automatic masking to obtain an omnidirectional pixel correspondence. The binary mask is used to prevent the pixels of moving objects from participating in the calculation of photometric errors. The present invention improves the accuracy of photometric loss by improving spatial transformation and generating automatic masks for dynamic objects, thereby better supervising the learning of deep networks. Therefore, by applying the depth-sensing pixel correspondence and omnidirectional automatic masking of this embodiment to the Monodepth2 framework of Godard et al., monocular depth estimation results with higher accuracy can be obtained.

The second embodiment of the present invention relates to a self-supervised monocular depth estimation device combined with spatiotemporal enhanced photometric loss, including: an acquisition module for acquiring several adjacent frame images in an image sequence; and an estimation module for converting the The image is input into the trained deep learning network to obtain depth information and pose information; the photometric loss information of the deep learning network is obtained based on the spatial transformation model of the depth sensing pixel correspondence module, and the omnidirectional automatic mask module is used to avoid Pixels of moving objects participate in the calculation of photometric errors.

The depth-sensing pixel correspondence module includes: a first construction unit for spatially transforming the distant area using a homography matrix and constructing a first reconstruction map; wherein the distant area regards the distant area as a a plane at infinity; a second construction unit for using the basic matrix to perform spatial transformation and construct a second reconstruction map; a photometric loss information acquisition unit for solving the problem based on the first reconstruction map through two pixel correspondences The photometric error map and the photometric error map based on the second reconstructed image are then selected pixel by pixel to obtain the final photometric loss information.

The omnidirectional automatic mask module includes: an initial reconstruction map generation unit, used to predict the initial depth and initial pose of the target frame through a pre-trained network, and generate an initial reconstruction map; a binary mask generation unit, used to Interference terms are added to the initial pose, and spatial transformation is used to obtain several hypothetical reconstructed frames; multiple photometric error maps are generated using the hypothetical reconstructed frames, combined with the luminosity of the target frame, and the multiple photometric error maps are generated. A plurality of binary masks are obtained from each photometric error map; a mask selection unit is used to select a minimum value from the plurality of binary masks as the final mask. Among them, the interference term is a translational disturbance term, including: [t _max ,0,0], [-t _max ,0,0], [0,0,t _max ] and [0,0,-t _max ] , where t _max represents the maximum value in the initialized translation vector.

Claims (6)

1. A method for self-supervising monocular depth estimation in combination with space-time enhancement luminosity loss, comprising the steps of:

acquiring a plurality of adjacent frame images in an image sequence;

inputting the image into a trained deep learning network to obtain depth information and pose information, wherein luminosity loss information of the deep learning network is obtained based on a space transformation model of a depth perception pixel corresponding relation, and pixels of a moving object are prevented from participating in luminosity error calculation by using an omnidirectional automatic mask; the luminosity loss information is obtained based on a space transformation model of depth perception pixel corresponding relation specifically as follows:

carrying out space transformation on a remote area by utilizing a homography matrix, and constructing a first reconstruction map; wherein the remote area treats the remote area as an infinitely distant plane;

performing space transformation by utilizing the basic matrix, and constructing a second reconstruction map;

and solving a luminosity error graph based on the first reconstruction graph and a luminosity error graph based on the second reconstruction graph through the corresponding relation of the two pixels, and then selecting the minimum value pixel by pixel to obtain final luminosity loss information.

2. The method for self-supervising monocular depth estimation combined with space-time enhancement luminosity loss of claim 1 wherein the calculation of avoiding pixel participation luminosity errors of a moving object by using an omnidirectional automatic mask is specifically:

predicting the initial depth and the initial pose of a target frame through a pre-training network, and generating an initial reconstruction map;

adding an interference item to the initial pose, and obtaining a plurality of assumed reconstructed frames by utilizing space transformation; generating a plurality of luminosity error maps by using the hypothesized reconstructed frame and combining luminosity of the target frame, and obtaining a plurality of binarization masks by using the plurality of luminosity error maps;

and selecting a minimum value from the plurality of binarized masks as a final mask.

3. The method of self-supervising monocular depth estimation combining spatio-temporal enhancement luminosity losses of claim 2 wherein the interference term is a translational disturbance term comprising: [ t ] _max ,0,0]、[-t _max ,0,0]、[0,0,t _max ]And [0, -t _max ]Wherein t is _max Representing the maximum value in the initialized translation vector.

4. A self-supervising monocular depth estimation device incorporating space-time enhancement luminosity loss, comprising:

the acquisition module is used for acquiring a plurality of adjacent frame images in the image sequence;

the estimation module is used for inputting the image into a trained deep learning network to obtain depth information and pose information; the luminosity loss information of the deep learning network is obtained based on a space transformation model of a depth perception pixel corresponding relation module, and pixels of a moving object are prevented from participating in luminosity error calculation by using an omnidirectional automatic mask module; the depth perception pixel correspondence module comprises:

the first construction unit is used for carrying out space transformation on the remote area by utilizing a homography matrix and constructing a first reconstruction map;

wherein the remote area treats the remote area as an infinitely distant plane;

a second construction unit for performing spatial transformation using the base matrix and constructing a second reconstruction map;

and the luminosity loss information acquisition unit is used for solving a luminosity error graph based on the first reconstruction graph and a luminosity error graph based on the second reconstruction graph through the corresponding relation of the two pixels, and then selecting the minimum value pixel by pixel to obtain final luminosity loss information.

5. The self-supervising monocular depth estimation apparatus incorporating spatio-temporal enhancement luminosity losses of claim 4 wherein the omnidirectional automatic masking module includes:

the initial reconstruction map generating unit is used for predicting the initial depth and the initial pose of the target frame through the pre-training network and generating an initial reconstruction map;

the binarization mask generating unit is used for adding an interference item to the initial pose and obtaining a plurality of assumed reconstruction frames by utilizing space transformation; generating a plurality of luminosity error maps by combining the luminosity of the target frame by using the hypothesized reconstructed frame, and obtaining a plurality of binarized masks by using the plurality of luminosity error maps

And the mask selecting unit is used for selecting the minimum value from the plurality of binarized masks as a final mask.

6. The self-supervising monocular depth estimation apparatus incorporating spatio-temporal enhancement luminosity losses of claim 5 wherein the interference term is a translational disturbance term comprising: [ t ] _max ,0,0]、[-t _max ,0,0]、[0,0,t _max ]And [0, -t _max ]Wherein t is _max Representing the maximum value in the initialized translation vector.