CN112085776B - A direct method for scene depth estimation in unsupervised monocular images - Google Patents

Abstract

The invention discloses a direct method unsupervised monocular image scene depth estimation method, which belongs to the field of computer vision and depth estimation and comprises the following steps: constructing a neural network; calculating an image reprojection error; image masking calculation and camera pose updating. The method overcomes the defects that monocular image depth estimation has high requirement on environment, is easily interfered by low texture regions, has poor camera pose estimation precision and the like, and not only obviously improves the depth estimation precision but also has auxiliary action on positioning and navigation of a moving vehicle by combining the traditional monocular depth estimation problem with a visual odometer; the method has the advantages of high precision, strong flexibility, wide application range and the like, can be used for sensing, preventing collision and positioning navigation of the surrounding environment of equipment such as an automatic driving vehicle, a mobile robot and the like, and has various application scenes.

Description

A direct method for scene depth estimation in unsupervised monocular images

technical field

The invention belongs to the fields of computer vision, neural network and depth estimation, and more particularly, relates to a direct method unsupervised monocular image scene depth estimation method.

Background technique

At present, depth estimation has developed rapidly, driven by related technologies such as neural networks and sensors, and has been widely used in intelligent robots, pedestrian recognition, face unlocking, VR applications, and autonomous driving. The primary task of depth estimation is to estimate the distance from the object ahead to the camera based on a single color image captured by the camera.

There are two main ways to obtain the three-dimensional information corresponding to the scene from the real scene: one is to use a sensor that can perceive the three-dimensional depth information of the scene to collect the depth information in the scene, and the other is to restore the three-dimensional image from the corresponding two-dimensional image of the scene. in-depth information. Structured light and time-of-flight (ToF) are commonly used sensor-based depth estimation algorithms. Structured light technology consists of a special projector and a camera. The camera obtains three-dimensional information of the scene by collecting the specific light information emitted by the projector and projecting it into the scene. Structured light uses special sensors to collect high-precision three-dimensional scene information, and is currently widely used in face unlocking, secure payment and other fields. However, due to the limitation of technical principles, it can only be used for distance measurement of close-range objects and small scene ranges, so it is not suitable for depth estimation in road scenes. ToF is another commonly used depth information acquisition technique, which utilizes the round-trip flight time of the signal between two transceivers to obtain depth information. The depth cameras and lidar ranging devices commonly used in mobile phones use ToF technology to obtain high-precision depth maps of the scene. ToF technology has a wide range of applications in the fields of AR, somatosensory games and autonomous driving, but the lidar and sensors required by ToF The price is expensive, and the roof lidar applied to autonomous driving has limitations due to its large volume, so the recovery of 3D depth information from the 2D image corresponding to the scene has gradually become the mainstream.

Image-based depth estimation can be divided into supervised learning and unsupervised learning. Supervised learning methods all rely on the depth 3D map corresponding to the training image. Usually, deep 3D maps are collected by radar sensors, so the training datasets of neural network-based supervised learning methods are usually small in size, and the cost of dataset acquisition is also high, which greatly limits the availability of supervised learning methods. Mobility and adaptability.

Unsupervised learning can usually be divided into multi-camera, monocular and binocular depth estimation methods according to the number of cameras needed to recover depth. The traditional depth estimation method is mainly based on the geometric constraints of feature point matching and environmental assumptions. The binocular method and the multi-eye depth estimation method require accurate external parameters of the camera, and cannot eliminate the error caused by the change of external parameters. The monocular image depth estimation method only needs the internal parameters of the camera and does not require the feature matching process. The algorithm is more concise and has a wider range of adaptability.

SUMMARY OF THE INVENTION

In view of the above technical problems existing in the prior art, the present invention proposes a direct method for unsupervised monocular image depth estimation, which not only significantly improves the depth estimation accuracy, but also has an auxiliary effect on the positioning and navigation of moving vehicles, and the design is reasonable. , overcomes the deficiencies of the prior art, and has a good effect.

In order to achieve the above object, the present invention adopts the following technical solutions:

A direct method for unsupervised monocular image scene depth estimation method, comprising the following steps:

Step 1: Build a depth estimation neural network, take the monocular continuous image as the input image, and use the depth estimation neural network to output the depth estimation image;

Step 2: Calculate the initial camera pose, use the previous frame image of the input image, the depth estimation neural image and the camera pose to calculate the reprojection image, and calculate the reprojection error between the reprojection image and the current frame image, and use back propagation. Update the parameters of the depth estimation neural network to obtain a new depth estimation image;

Step 3: Use the reprojected image and the input image to calculate the image mask, update the camera pose estimation between the two frames of images before and after, and repeat iterative steps 2 and 3.

Preferably, in step 1, it specifically includes the following steps:

Step 1.1: The constructed depth estimation neural network is fully convolutional U-shaped, and the convolutional network in the Res-Net18 network structure is used as the main structure network in the convolution part;

Step 1.2: The deconvolution part includes several layers of deconvolution layers and ReLu activation layers superimposed as the main structure. Each deconvolution layer is connected to the last layer of convolutional layers in the convolutional block with the same scale in the convolutional part. , forming the final deconvolution layer;

Step 1.3: Take the monocular continuous image as the training data set, and perform sample expansion and generalization operations such as image inversion, gamma transformation, and color channel change before inputting to the depth estimation neural network.

Preferably, in step 2, it specifically includes the following steps:

Step 2.1: pre-calibrate the internal parameters of the camera or obtain the internal parameters of the camera from the data set;

Step 2.2: Using the current frame image and the previous frame image, use the direct method to calculate the initial camera pose;

Step 2.3: The reprojection image will be calculated using the previous frame image of the input image, the depth estimation image and the camera pose;

Step 2.4: Calculate the reprojection error between the current frame image and the reprojection image;

Step 2.5: Use the depth estimation image output by the depth estimation neural network and the image reprojection error, and update the parameters of the depth estimation neural network through backpropagation to obtain a new depth estimation image.

Preferably, in step 3, it specifically includes the following steps:

Step 3.1: Calculate the similarity error according to the reprojected image and the current frame image to obtain an image mask;

Step 3.2: Calculate the Jacobian matrix between the previous frame image and the obtained camera pose;

Step 3.3: Multiply the image mask with the Jacobian matrix to obtain the improved Jacobian matrix;

Step 3.4: Using the improved Jacobian matrix, update the camera pose estimation between the two frames of images before and after.

Step 3.5: Repeat iterative steps 2 and 3.

Preferably, the convolutional part uses the convolutional network in the Res-Net18 network structure as the main structure network, which consists of 5 convolutional blocks and 5 deconvolutional blocks, and the first convolutional block contains 1 volume Product group, the convolution group contains 1 convolution layer, the input of the convolution layer is a 3-channel color image, and the output is 64 channels. The 2nd, 3rd, 4th, and 5th convolution blocks respectively contain 2 convolutions Each convolution group contains 2 convolution layers respectively, and the number of channels contained in the 2nd, 3rd, 4th, and 5th convolution blocks are 64, 128, 256, and 512, respectively. The number of deconvolution groups is twice the number of convolution groups in the corresponding scale convolution block, and the number of channels contained in each convolution group in each deconvolution block is 512, 256, 128, 64, and 1, respectively.

Preferably, the previous convolutional block and the next convolutional block are connected through a max pooling operation, and the max pooling operation also scales the size of the next convolutional block in the two adjacent convolutional blocks to the previous convolutional block One-half of the accumulation block, the scale of the next deconvolution block in the two adjacent convolution blocks is twice the scale of the previous deconvolution block, and the convolution in each convolution group and deconvolution group The layer and the deconvolution layer are connected by 3*3 convolution. In addition, the content of the second convolution block is copied to the fourth deconvolution block, and the content of the fifth convolution block is copied to the first convolution block. in three deconvolution blocks.

Beneficial technical effects brought by the present invention:

The present invention obtains a two-dimensional image of the environment through a monocular camera, and then uses a designed fully convolutional neural network to obtain a three-dimensional depth map corresponding to the two-dimensional image. When training the network, the present invention uses the designed image reprojection error and image mask, which increases training efficiency and depth estimation accuracy. The image mask used in the method can effectively remove interference factors such as ground texture areas and moving vehicles in the road environment, greatly improving the accuracy of monocular depth estimation, while reducing the requirements for the environment and training costs; , because the positioning method provided by the present invention can obtain the depth map in front of the camera in real time, the method can not only be used for the navigation of ground mobile robots and automatic driving cars, but also suitable for drones flying in the air.

The method of the invention not only overcomes the defects of the traditional depth estimation method, such as strong dependence on radar sensors, high requirements for the environment, and inflexibility, but also effectively overcomes the problem of holes in the monocular depth estimation due to low texture areas. The positioning and navigation of UAV is also very suitable; it has many advantages such as high precision, strong flexibility, and wide application range, and can be used for navigation and obstacle avoidance of intelligent mobile devices such as mobile robots and UAVs in indoor environments, broadening the application. Scenes.

Description of drawings

1 is a schematic diagram of an unsupervised monocular depth estimation module based on a fully convolutional neural network of the present invention;

Fig. 2 is the training process schematic diagram of the fully convolutional neural network of the present invention;

Detailed ways

The present invention is described in further detail below in conjunction with the accompanying drawings and specific embodiments:

As shown in Figure 1, a direct method for unsupervised scene depth estimation from monocular images includes:

Step 1: The constructed depth estimation neural network is a fully convolutional U-shaped, including a convolution part and a deconvolution part. The depth estimation neural network takes the monocular continuous image as the input image, and uses the depth estimation neural network to output the depth estimation image.

Step 1 includes the following sub-steps:

Step 1.1: The convolutional part uses the convolutional network in the Res-Net18 network structure as the main structure network. The convolution part consists of several convolution blocks. The previous convolution block and the next convolution block are connected by a maximum pool operation. Each convolution block contains several convolution groups. It contains several convolutional layers, and at the same time, the maximum pooling operation also scales the size of the next convolutional block in two adjacent convolutional blocks to one-half of the previous convolutional block. As shown in FIG. 2 , in the embodiment of the present invention, the number of convolution blocks is 5, and each convolution module includes several different convolution groups. The first convolution block contains 1 convolution group, and the convolution group contains 1 convolution layer. The input of the convolution layer is a 3-channel color image and the output is 64 channels; the second, third, fourth, and fifth Each convolution block contains 2 convolution groups, and each convolution group contains 2 convolution layers respectively. The number of channels contained in the 2nd, 3rd, 4th, and 5th convolution blocks are 64, 128, 256 and 512. The 7*7 convolution kernel is used in the first convolution block, the 3*3 convolution kernel is used in the 2nd, 3rd, 4th, and 5th convolution blocks, and the convolution layer in each convolution block is Linear correction units (Rectified Linear Units, ReLu) are used as the activation function.

Step 1.2: The deconvolution part consists of several deconvolution blocks. The deconvolution block of the same scale in the deconvolution part consists of two parts, namely the base block of the same scale in the convolution part as the current deconvolution part and the deconvolution block of the current scale. The scale of the next deconvolution block is twice the scale of the previous deconvolution block. Each deconvolution block contains several deconvolution groups. As shown in Figure 2, the number of deconvolution blocks in the present invention is 5, and the number of deconvolution groups in the deconvolution block is twice the number of convolution groups in the corresponding scale convolution block. The number of convolutional layers contained in it is the same as the number of convolutional layers in the corresponding scale convolutional group, and the number of channels contained in each convolutional group in each deconvolution block is 512, 256, 128, 64, 1 .

Step 1.3: Use the convolution part and the deconvolution part to construct a depth estimation neural network, take the monocular continuous image as the training data set, input the constructed neural network for depth estimation, and perform image inversion before inputting the depth estimation neural network, Sample expansion and generalization operations such as gamma transform, color channel change, etc.

Step 2: Calculate the reprojection image using the previous frame image of the input image, the depth estimation image and the camera pose, and calculate the reprojection error between the reprojection image and the current frame image to obtain the reprojection error of the image, and use back propagation. Perform parameter update of the depth estimation neural network to obtain a new depth estimation image.

Step 2 includes the following sub-steps:

Step 2.1: Pre-calibrate the camera's internal parameters or obtain the camera's internal parameters from the data set. The camera's internal parameters are calibrated off-line, using Zhang Zhengyou's camera calibration method, and using a checkerboard.

Step 2.2: Using the current frame image and the previous frame image, use the direct method to calculate the initial camera pose.

Step 2.3: The reprojection image will be calculated using the previous frame of the input image, the depth estimation image and the camera pose;

Step 2.4: Calculate the reprojection error between the current frame image and the reprojection image. The reprojection error is the loss function of the depth estimation neural network.

The loss function in the depth estimation neural network uses the error between the reprojected image and the original image, which is defined as:

分别表示输入图像和重投影图像的像素值，i表示像素值的索引。Among them, I _i ,

represent the pixel values of the input image and the reprojected image, respectively, and i represents the index of the pixel value.

Step 2.5: Use the depth estimation image output by the depth estimation neural network and the image reprojection error to update the parameters of the depth estimation network through backpropagation to obtain a new depth estimation image.

Step 3: Use the reprojected image and the input image to calculate the image mask, update the camera pose estimation between the two frames of images before and after, and repeat iterative steps 2 and 3.

Step 3 includes the following sub-steps:

Step 3.1: Calculate the similarity error according to the reprojected image and the current frame image to obtain an image mask.

Specifically, the image mask is a binary image. The image mask is set to 1 when the reprojection error is less than the difference between the images before and after the frame, otherwise, it is set to 0. In the method of the present invention, the image mask of the low texture area and the moving vehicle area is 0, and the image mask of other places is 1.

Step 3.3: Calculate the Jacobian matrix between the previous frame image and the obtained camera pose.

Step 3.4: Multiply the image mask with the Jacobian matrix to obtain the improved Jacobian matrix.

Step 3.5: Using the improved Jacobian matrix, update the camera pose estimation between the two frames of images before and after.

Step 3.6: Repeat iterative steps 2 and 3.

Setting the number of iterations to 60,000, the model saved at the 50,000th training achieved the best estimate, with a total training time of 40 hours.

The above is the complete implementation process of this embodiment.

Of course, the above description is not intended to limit the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by those skilled in the art within the essential scope of the present invention should also belong to the present invention. the scope of protection of the invention.

Claims (5)

1. A direct method unsupervised monocular image scene depth estimation method is characterized by comprising the following steps: the method comprises the following steps:

step 1: constructing a depth estimation neural network, taking a monocular continuous image as an input image, and outputting a depth estimation image by using the depth estimation neural network;

step 2: calculating an initial camera pose, calculating a re-projection image by using a previous frame image, a depth estimation image and the camera pose of an input image, calculating a re-projection error of the re-projection image and a current frame image, and updating parameters of a depth estimation network by using back propagation to obtain a new depth estimation image;

and step 3: calculating an image mask by utilizing the reprojected image and the input image, updating the camera pose estimation between the front frame image and the rear frame image, and repeating the iteration steps 2 and 3;

in step 3, the method specifically comprises the following steps:

step 3.1: calculating a similarity error according to the reprojected image and the current frame image to obtain an image mask;

step 3.2: calculating a Jacobian matrix between the last frame of image and the obtained camera pose;

step 3.3: multiplying the image mask by the Jacobian matrix to obtain an improved Jacobian matrix;

step 3.4: updating the camera pose estimation between the front frame image and the rear frame image by using the improved Jacobian matrix;

Step 3.5: and repeating the iteration steps 2 and 3.

2. A direct method unsupervised monocular image scene depth estimation method according to claim 1, characterized in that: in step 1, the method specifically comprises the following steps:

step 1.1: the constructed depth estimation neural network is in a full convolution U type, and the convolution part uses a convolution network in a Res-Net18 network structure as a main structure network;

step 1.2: the deconvolution part comprises a plurality of deconvolution layers and ReLu activated layer stacking as a main body structure, and each deconvolution layer is connected with the last convolution layer in the convolution blocks with the same mesoscale in the convolution part to form a final deconvolution layer;

step 1.3: monocular continuous images are taken as a training data set, and sample expansion and generalization operations including image inversion, gamma transformation and color channel change are carried out before the input of the depth estimation neural network.

3. The direct method unsupervised monocular image scene depth estimation method of claim 1, characterized by: in the step 2, the method specifically comprises the following steps:

step 2.1: pre-calibrating intrinsic parameters of the camera or obtaining the intrinsic parameters of the camera from the data set;

step 2.2: calculating the initial camera pose by using the current frame image and the previous frame image and adopting a direct method;

Step 2.3: calculating a re-projection image by using a last frame image and a depth estimation image of the input image and the camera pose;

step 2.4: calculating a reprojection error between the current frame image and the reprojected image;

step 2.5: and updating parameters of the depth estimation neural network by utilizing the depth estimation image and the image reprojection error output by the depth estimation neural network through back propagation to obtain a new depth estimation image.

4. A direct method unsupervised monocular image scene depth estimation method according to claim 2, characterized in that: the convolution part uses a convolution network in a Res-Net18 network structure as a main structure network, and comprises 5 convolution blocks and 5 deconvolution blocks, wherein the 1 st convolution block comprises 1 convolution group, the convolution group comprises 1 convolution layer, the input of the convolution layer is a 3-channel color image, the output is 64 channels, the 2 nd, 3 rd, 4 th and 5 th convolution blocks respectively comprise 2 convolution groups, each convolution group comprises 2 convolution layers, the number of channels contained in the 2 nd, 3 th, 4 th and 5 th convolution blocks is respectively 64, 128, 256 and 512, the number of deconvolution groups in the deconvolution blocks is 2 times of the number of convolution groups in the corresponding scale convolution blocks, and the number of channels contained in each convolution group in each deconvolution block is respectively 512, 256, 128, 64 and 1.

5. A direct method unsupervised monocular image scene depth estimation method according to claim 2, characterized in that: the last volume block and the next volume block are connected through a maximum pool operation, the maximum pool operation also scales the size of the next volume block in the two adjacent volume blocks to be one half of the size of the last volume block, the size of the next deconvolution block in the two adjacent volume blocks is 2 times of the size of the last deconvolution block, the convolution layer and the deconvolution layer in each volume unit and the deconvolution group are connected through a 3 x 3 convolution mode, in addition, the content of the second volume block is copied into the fourth deconvolution block, and the content of the fifth volume block is copied into the third deconvolution block.

Images