JP2023502793A - Method, device and storage medium for generating panoramic image with depth information - Google Patents

Abstract

A method and apparatus for generating panoramic images with depth information are provided. A method for generating a panoramic image with depth information is to obtain a 2D image of the current scene based on spherical projection, and connect a preset number of 2D images horizontally end-to-end. determining the depth information of the intermediate image using a trained neural network model for predicting image depth; and based on the intermediate position in the horizontal direction of the intermediate image. Obtaining an image equal to the length of the two-dimensional image by cropping the intermediate image, determining the depth information of the cropped image, and determining the cropped image with depth information as the panoramic image of the current scene. and [Selection drawing] Fig. 1

Description

TECHNICAL FIELD The present disclosure relates to the technical field of 3D model reconstruction, and more particularly to a method, apparatus and storage medium for generating a panoramic image with depth information.

3D model reconstruction occupies an important position in the fields of industrial detection, quality control and machine vision. In the field of 3D reconstruction of indoor and outdoor scenes, the depth data should be modeled by forming a point cloud, and based on the distance information of the point cloud, the point clouds of different positions obtained by the sensors should be stitched together. There is However, the acquisition of depth data for three-dimensional reconstruction typically requires expensive dedicated depth sensors such as structured light, lasers based on the Time Of Flight principle, etc., which are expensive. The cost is too high for large-scale industrial practical use.

An embodiment of the present disclosure provides a method of generating a panoramic image with depth information, which comprises acquiring a spherical projection-based two-dimensional image of a current scene, Determining the depth information of the intermediate image by constructing an intermediate image by horizontally end-to-end tangent of the dimensional images and utilizing a trained neural network model for predicting image depth. and trimming the intermediate image based on the intermediate position in the horizontal direction of the intermediate image to obtain an image equal to the length of the two-dimensional image, determining the depth information of the trimmed image, and cropping with depth information and determining the resulting image as a panoramic image of the current scene.

In one example of the present disclosure, a neural network model for predicting image depth includes a pre-trained convolutional neural network as an initial backbone network, structurally adjusting the initial backbone network according to a U-Net network structure, A plurality of depth-informed color 3D images are trained by training the initial adjusted backbone network to generate a neural network model for predicting the image depth.

In one example of the present disclosure, the pre-trained convolutional neural network is an ImageNet pre-trained DenseNet network. Structurally adjusting the initial backbone network according to the U-Net network structure,
removing all connectivity layers of a DenseNet network;
Based on the U-Net network structure, add multiple upsampling layers after the last layer of the DenseNet network with all connected layers removed, and add a superposition layer corresponding to each upsampling layer in the multiple upsampling layers, respectively. where the number of input channels of each upsampling layer is a preset multiple of its number of output channels, and in each added upsampling layer, the input information of the upsampling layer is converted to a preset resolution Upsampling by a factor, convolving the upsampling result with the output data of the convolution layer corresponding to the upsampling layer, performing at least one convolution operation on the convolution result, and using a preset activation function. linearly correcting the result of the convolution operation by
Performing one convolution operation for outputting depth information and one convolution operation for outputting reliability information on the output of the last upsampling layer in a plurality of upsampling layers. include.

In one example of this disclosure, the DenseNet network is a DenseNet-169 network and the plurality of upsampling layers is four upsampling layers. The number of input channels for each upsampling layer is twice its number of output channels. For each added upsampling layer, the input information of the upsampling layer is upsampled at a double resolution, and the at least one convolution operation may be two convolution operations.

In one example of the present disclosure, for four upsampling layers, according to the order from the first layer to the last layer of the four upsampling layers, the convolution layers corresponding to the four upsampling layers sequentially are: pool3_pool, pool2_pool, pool1, conv1/relu.

In one example of the present disclosure, prior to training the initial adjusted backbone network utilizing multiple depth-informed color 3D images, a method according to an embodiment of the present disclosure includes:
In the step of preprocessing the plurality of color 3D images with depth information and preprocessing the plurality of color 3D images with depth information, at least one color 3D image with depth information includes depth information If it is determined that there is a hole pointing to a pixel point for which depth information cannot be determined in the color three-dimensional image with the marking, the method further includes not performing the hole-filling process.

In one example of the present disclosure, the process of training an initial backbone network that is tuned using multiple depth-informed color 3D images is trained using a supervised learning method, where The loss function obtained is a function of the depth and confidence estimates for each pixel based on the neural network model.

In one example of the present disclosure, after obtaining a spherical projection-based two-dimensional image of a current scene, a method according to embodiments of the present disclosure, in response to determining that the two-dimensional image has a visual angle blind region, and filling the viewing angle blind area with black.

In one example of the present disclosure, a method according to an embodiment of the present disclosure comprises forming an intermediate image by connecting a predetermined number of the two-dimensional images in a horizontal direction end-to-end, and then Further including trimming the edges.

In one example of the present disclosure, trimming the upper and lower edges of the intermediate image includes cropping an image from the upper and lower edges of the intermediate image to a height of a preset ratio of the height of the intermediate image, respectively. Including.

In one example of the present disclosure, the preset number is 3 and the preset ratio is 15%.

An embodiment of the present disclosure further provides an apparatus for generating a panoramic image with depth information, the apparatus comprising: an acquisition unit for acquiring a spherical projection-based two-dimensional image of a current scene; Utilizing a stitching unit for tangling a number of said two-dimensional images horizontally end-to-end to construct an intermediate image and a trained neural network model for predicting image depth, said A processing unit for determining the depth information of the intermediate image and cropping the intermediate image based on the intermediate position in the horizontal direction of the intermediate image to obtain an image equal to the length of the two-dimensional image, and the cropped image a cropping unit for determining depth information and determining the cropped image with depth information as a panoramic image of the current scene.

In one example of this disclosure, a neural network model for predicting image depth is:
A pre-trained convolutional neural network is used as an initial backbone network, the initial backbone network is structurally adjusted according to the U-Net network structure, and a plurality of depth-informed color 3D images are used to obtain the adjusted It is trained by training an initial backbone network and generating a neural network model for predicting the image depth.

In one example of the present disclosure, the pretrained convolutional neural network is an ImageNet pretrained DenseNet-169 network.

In one example of the present disclosure, the processing unit comprises:
a deletion subunit for deleting all connectivity layers of a DenseNet-169 network;
Based on the U-Net network structure, add 4 upsampling layers after the last layer of the DenseNet-169 network with all connected layers removed, and superimpose each corresponding to each upsampling layer in the 4 upsampling layers setting a layer, where the number of input channels in each upsampling layer is twice the number of output channels, and in each added upsampling layer, upsampling the input information of the upsampling layer at twice the resolution and then convolving the upsampling result with the output data of the convolution layer corresponding to the upsampling layer, successively performing two convolution operations on the convolution result, and using a preset activation function an additive subunit for linearly correcting the result of the convolution operation;
An operation for performing one convolution operation for outputting depth information and one convolution operation for outputting reliability information on the output of the last upsampling layer in the four upsampling layers. subunits.

In one example of the present disclosure, for four upsampling layers, the convolution layers corresponding to the four upsampling layers sequentially are pool3_pool, pool2_pool, pool1, according to the order from the first layer to the last layer of the four upsampling layers. , conv1/relu.

In one example of the present disclosure, the processing unit further:
In the step of preprocessing the plurality of color 3D images with depth information and preprocessing the plurality of color 3D images with depth information, at least one color 3D image with depth information includes depth information This is used to avoid filling the hole when it is determined that there is a hole pointing to a pixel point for which depth information cannot be determined in the color 3D image with .

In one example of the present disclosure, in the step of training the initial adjusted backbone network using the plurality of depth-informed color 3D images, training using a supervised learning method, wherein depth estimation The loss function used in is a function of depth and confidence estimates for each pixel based on a neural network model.

In one example of the present disclosure, the acquisition unit is further used to black fill a visual angle blind area in response to determining that there is a visual angle blind area in the two-dimensional image.

In one example of the present disclosure, the stitching unit is further used to trim the top and bottom edges of the intermediate image.

In one example of the present disclosure, the stitching unit is further used to crop the image from the top and bottom edges of the intermediate image to a height of a preset ratio of the height of the intermediate image, respectively.

In one example of the present disclosure, the preset number value is 3 and the preset ratio is 15%.

An embodiment of the present disclosure is a non-transitory computer readable storage storing a computer program containing instructions that, when executed by a processor of an electronic device, causes the processor to perform the method of generating a panoramic image with depth information described above. It also provides a medium.

Embodiments of the disclosure further provide an electronic device including a memory and a processor. The memory is used to store a computer program executable by a processor, and when the processor executes the program, it implements the method of generating a panoramic image with depth information described above.

With the technical solution according to the embodiments of the present disclosure, after obtaining the two-dimensional image of the current scene based on spherical projection, the intermediate image is obtained by connecting the multiple two-dimensional images horizontally end-to-end. Constitute. A trained neural network model for predicting image depth is then utilized to determine the depth information of the intermediate images. An image equal to the length of the two-dimensional image is obtained by trimming the intermediate image based on the intermediate position in the horizontal direction of the intermediate image. Having determined the depth information of the intermediate image, the image cropped from the intermediate image also has corresponding depth information and can be a depth-informed panoramic image of the current scene. Applying the technical solution according to the embodiments of the present disclosure, the panoramic image with depth information of the current scene can be obtained without using a depth camera, thus reducing the cost. In addition, since the depth information is determined for an intermediate image obtained by joining a plurality of two-dimensional images, there is no loss of information in the connecting portions between the ends of the two-dimensional images. can be made more accurate.

The following drawings are not intended to limit the scope of the present disclosure, but merely to exemplify and interpret the present disclosure.
4 is a flowchart of a method of generating a panoramic image with depth information according to some embodiments of the present disclosure; FIG. 4 is a schematic diagram of an intermediate image constructed by joining a plurality of two-dimensional images end-to-end in contact with some embodiments of the present disclosure; FIG. 4B is a schematic diagram of the result of cropping the upper and lower edges of an intermediate image according to some embodiments of the present disclosure; 1 is a schematic block diagram of an apparatus for generating panoramic images with depth information according to some embodiments of the present disclosure; FIG. 4 is a schematic configuration diagram of an apparatus for generating a panoramic image with depth information according to some other embodiments of the present disclosure; FIG. 1 is a schematic block diagram of an electronic device according to some embodiments of the present disclosure; FIG.

In order to make the objectives, technical solutions and advantages of the present disclosure clearer, the technical solutions of the present disclosure are described in detail below along with the drawings and examples.

In the embodiments of the present disclosure, a normal camera can be used to capture and obtain a two-dimensional image of the current scene. Construct an intermediate image by joining multiple 2D images of the same current scene horizontally edge-to-edge, and utilize a trained neural network model to predict the depth of the intermediate image Information can be confirmed. Subsequently, the intermediate image is trimmed based on the intermediate position in the horizontal direction of the intermediate image to obtain an image equal to the length of the two-dimensional image of the current scene. The image cropped from the intermediate image has corresponding depth information, so it can be a depth-informed panoramic image of the current scene.

FIG. 1 is a flow chart of a method for generating a panoramic image with depth information according to some embodiments of the present disclosure, as shown in FIG. 1, the method mainly includes the following steps.

In step 101, a two-dimensional image of the current scene based on spherical projection is obtained.

Embodiments of the present disclosure may obtain a spherical projection-based two-dimensional image of the current scene from the image capture device. A two-dimensional image can only have RGB information of a panoramic image, but it does not have depth information of a panoramic image. Any mobile device with a camera, such as a mobile phone, may be used. The image capture device used does not require a depth camera, so the cost of acquiring panoramic images is low.

In embodiments of the present disclosure, for the top and bottom edges of the captured current scene 2D image, the neural network model does not need to align the perfect viewing angle for inferring the depth information of the 2D image. , with sufficient texture, line, and object information at vertical viewing angles.

For example, if the full viewing angle is not aligned, the viewing angle blind area in the two-dimensional image can be filled with black. According to an exemplary embodiment, after obtaining a spherical projection-based two-dimensional image of the current scene, it can be further determined whether there is a visual angle blind region in the two-dimensional image. In response to determining that there is a visual angle blind area in the two-dimensional image, the visual angle blind area in the two-dimensional image is painted black.

In step 102, an intermediate image is constructed by connecting a predetermined number of two-dimensional images in the horizontal direction end-to-end.

In embodiments of the present disclosure, the value of the preset number N is an integer greater than 1, for example, the value of N may be 3.

A two-dimensional image based on spherical projection touches its own information end-to-end. Constructing an intermediate image by joining a preset number of two-dimensional images horizontally end-to-end connects the tail of the first two-dimensional image to the head of the second two-dimensional image. and the tail of the second two-dimensional image is connected to the head of the third two-dimensional image, and so on, and the head of the first two-dimensional image is connected to the tail of the last two-dimensional image. can point to not The preset number of two-dimensional images may be multiple identical images, ie, all two-dimensional images of the current scene, based on spherical projection. For example, as shown in FIG. 2, a schematic diagram of an intermediate image composed by three two-dimensional images is shown.

In some examples, a two-dimensional image based on spherical projection has a length (horizontal length) to width (vertical length, also called height) ratio of 2:1. Therefore, an intermediate image constructed by connecting N two-dimensional images horizontally end-to-end has a length to width ratio of 2N:1. For example, when N=3, the aspect ratio of the intermediate image is 6:1.

In practical applications, there may be distortions in the upper and lower edge parts of 2D images based on spherical projection, and the distortions affect subsequent convolutional neural network training and depth estimation. In order to reduce this effect, in the embodiments of the present disclosure, an intermediate image is formed by connecting a predetermined number of two-dimensional images in the horizontal direction end-to-end, and then the upper and lower edges of the intermediate image are You can also trim. For example, the image can be cropped to a proportional height from the top edge of the intermediate image, and the image to a proportional height from the bottom edge of the intermediate image. The trimming ratio for the top edge and the trimming ratio for the bottom edge may be the same or different.

In an embodiment of the present disclosure, trimming the upper and lower edges of the intermediate image means trimming the height of the image from the upper and lower edges of the intermediate image to a preset ratio of the height of the intermediate image, respectively. Including, here, the preset ratio may be 15% as long as it can guarantee that the lack of texture, line, and object information in the two-dimensional image does not exceed a certain threshold, and other values There may be. This threshold can be determined according to specific needs, and the present disclosure is not limited thereto. For example, for an intermediate image with an aspect ratio of 6:1, after trimming the top and bottom edges of the intermediate image by 15% respectively, the aspect ratio becomes 60:7. As shown in FIG. 3, an example of an intermediate image with top and bottom edges trimmed is shown, where the scribe portion is the trimmed image portion.

At step 103, depth information for intermediate images is determined using a trained neural network model for predicting image depth.

Embodiments of the present disclosure may obtain a neural network model for predicting image depth through training. For example, a large number of color 3D images with depth information of actual scenes may be pre-collected as training samples. The training samples require pixel-level alignment and include a variety of indoor and outdoor scenes, such as wall corners, cars, ceilings, ground, windows, doors, and other elements.

In general, there are holes in depth-informed color 3D images of real scenes, and in the embodiments of the present disclosure, a large number of collected training samples (i.e., depth-informed color 3D images) are used in advance. Can be processed (Gaussian filter, size adjustment, etc.). What should be explained is that when training samples are preprocessed, if there are holes in the training samples, the holes are not filled. A hole here refers to a pixel point in an image for which depth information cannot be determined. For such pixel points whose depth information cannot be determined, in the embodiments of the present disclosure, the depth information is still unknown, and the depth value is obtained by pre-estimating or other methods to realize the filling process. never

After collecting a large amount of training samples, the training samples can be used to train a neural network model to obtain a neural network model for predicting image depth.

In embodiments of the present disclosure, a neural network model for predicting image depth may be trained by the following scheme. The pre-trained convolutional neural network is taken as the initial backbone network, and the initial backbone network is structurally adjusted according to the U-Net network structure. Multiple depth-informed color 3D images (ie, training samples) are utilized to train an initial tuned backbone network to generate a neural network model for predicting image depth.

The pre-trained convolutional neural network may be a DenseNet network pre-trained with ImageNet. In some examples, the DenseNet network may be a DenseNet-169 network.

Structurally adjusting the initial backbone network according to the U-Net network structure can include the following steps.

Remove all connectivity layers of the DenseNet network. Based on the U-Net network structure, add multiple upsampling layers after the last layer of the DenseNet network with all connected layers removed, and add a superposition layer corresponding to each upsampling layer in the multiple upsampling layers, respectively. set. In some examples, for example, if the DenseNet network is a DenseNet-169 network, four upsampling layers can be added after the last layer of the DenseNet-169 network.

Illustratively, the pre-trained convolutional neural network can be a model such as Google Inception, ResNet, VGG, etc. pre-trained with ImageNet. If the pre-trained convolutional neural network is ImageNet pre-trained Google Inception, ResNet, or VGG, add an upsampling layer based on the above U-Net network structure, and for each upsampling layer, a corresponding superposition The implementation of setting the layers may differ, eg the name of the added layers may change.

In an embodiment of the present disclosure, the four added upsampling layers can be, for example, one layer each in the DenseNet-169 network. Assume that the four upsampling layers added are, in order from the first layer to the last layer: 1st upsampling layer, 2nd upsampling layer, 3rd upsampling layer, 4th upsampling layer Then the output of the last layer of the DenseNet-169 network with all connected layers removed is the input of the first upsampling layer, the output of the first upsampling layer is the input of the second upsampling layer, and the output of the first upsampling layer is the input of the second upsampling layer. The output of the upsampling layer of is the input of the third upsampling layer, and the output of the third upsampling layer is the input of the fourth upsampling layer. Also, the superimposed layers corresponding to the first upsampling layer, the second upsampling layer, the third upsampling layer, and the fourth upsampling layer are sequentially configured as pool3_pool, pool2_pool, pool1, conv1/relu. be able to.

The number of input channels for each upsampling layer may be a preset multiple of its number of output channels, such as twice. In each added upsampling layer, the input information of the upsampling layer is upsampled at a preset resolution multiple (for example, twice the resolution), and the upsampling result is the superposition layer corresponding to the upsampling layer. Convolved with the output data, the convolution result is subjected to at least one convolution operation (for example, the at least one convolution operation may be two convolution operations. If there are two convolution operations, the first and perform a second convolution operation on the result of the first convolution operation, each convolution operation may be a two-dimensional convolution operation with a 3×3 convolution kernel). and linearly correct the convolution result using a preset activation function (eg, relu activation function). Here, upsampling the input information of the upsampling layer to double the resolution can refer to upsampling done by expanding the number of rows and columns of pixels of the input information to twice the original number, respectively. can.

One convolution operation for outputting depth information and one convolution operation for outputting reliability information are performed on the output of the last upsampling layer among the plurality of upsampling layers. Alternatively, the output of the last upsampling layer is subjected to one convolution operation with two output channels, which output confidence information and depth information respectively. In some examples, the convolution operation may be a two-dimensional convolution operation with a 3×3 convolution kernel.

Depth information D may be in meters. Reliability information refers to the reliability of the predicted depth value of each pixel point in the intermediate image. A high confidence value for a pixel point indicates that the predicted depth value for this pixel point is close to the actual depth value, and a low confidence value indicates that the predicted depth value for this pixel point is close to the actual depth value. A not-so-close confidence value of 0 indicates that this pixel point is a hole and its depth value cannot be determined/predicted.

According to some illustrative examples, in the course of training an initial backbone network that is tuned using multiple depth-informed color 3D images, it can be trained using a supervised learning method. , the loss function used for depth estimation is a function of depth and confidence estimates for each pixel based on a trained neural network model. In some embodiments, the loss function may be a combination of the following three functions.

Function 1, function f1(x) of the depth estimate value of each pixel x based on the neural network model: mask filtering the absolute value of the difference between the depth estimate value of each pixel x and the true depth value of the neural network model;
Function 2, Gradient function f2(x) of the depth estimation value of each pixel x based on the neural network model: Mask the absolute value of the difference value between the gradient of the depth estimation value of each pixel x by the neural network model and the gradient of the true depth value filtering,
Function 3, function f3(x) of the reliability estimated value of each pixel x based on the neural network model: the absolute value of the difference between the reliability estimated value of each pixel x based on the neural network model and the true reliability value.

The true confidence value can be determined by adopting the following method: if there is no depth estimate of pixel x by the neural network model, determine the true confidence value as 0; If a depth estimate exists, determine the true confidence value using the following formula: true confidence value=1−preset adjustment factor (eg, 0.02)×(pixels by neural network model× is the depth estimate of -the depth true value of pixel x).

In the embodiment of the present disclosure, the corresponding weighted average results of the above three functions for each pixel in the image are cumulatively added and then the average value is calculated, and the calculated average value is the loss value of the loss function. can be done. Note that the above true depth value refers to the actual depth value of a pixel in an image.

For the functions f1 and f2 described above, mask filtering makes it possible to ignore depth estimation values in portions where there is a hole in the true depth value. The function f3 described above uses the L1 absolute value loss for confidence estimation, and by adopting this function, the confidence true value of the portion of the hole in the depth true value is set to 0, For pixel points where the depth estimate is far away from the true depth value, the confidence estimate should be close to 0, and for pixel points where the depth estimate and the true depth value are relatively close, the confidence estimate The value should be close to 1.

In step 104, the intermediate image is trimmed based on the intermediate position in the horizontal direction of the intermediate image to obtain an image equal to the length of the two-dimensional image, the depth information of the trimmed image is determined, and the depth information with the depth information is obtained. Accept the cropped image as a panoramic image of the current scene.

In embodiments of the present disclosure, a trained neural network model for predicting image depth is used to determine the depth information of intermediate images, and then an intermediate image with depth information and confidence information is obtained. be done. By trimming the intermediate image based on the intermediate position in the horizontal direction of the intermediate image (for example, the midpoint position), an image equal to the length of the two-dimensional image of the current scene can be obtained. For example, as shown in FIG. 2, when connecting three 2D images of the same current scene horizontally end-to-end to form an intermediate image, the horizontal midpoint position of the intermediate image is used as a reference. A two-dimensional image of the current scene obtained by defining two image regions with lengths corresponding to 50% of the length of the two-dimensional image respectively toward the left and right, and cropping these two image regions. The image can be made to correspond to the second two-dimensional image in the intermediate image, equal to the length of . Depth information for the cropped image can be determined, and the cropped image with depth information can be determined as the panoramic image of the current scene.

In practice, the depth information of intermediate images determined using a trained neural network model for predicting image depth includes depth information for each pixel point in the intermediate images. Therefore, the depth information of each pixel point in the image equal to the length of the two-dimensional image obtained by trimming the intermediate image based on the intermediate position in the horizontal direction of the intermediate image can be directly determined. Similarly, the confidence information of each pixel point in the cropped image can also be determined directly.

Depth estimates whose corresponding confidence estimates are greater than a preset confidence threshold (eg, 0.8) can be used as reliable depth data sources in the panoramic image of the current scene. The confidence threshold value can be adjusted, for example, depending on whether the final application requires more depth data or depth data with higher confidence.

After determining the depth-informed panoramic image of the current scene, embodiments of the present disclosure use this depth information in subsequent processes such as pixel alignment, image stitching, etc. for high-precision pixel alignment and image stitching. Operations such as matching can be supported. At the same time, this depth information can be converted to a single-point point cloud for subsequent 3D reconstruction tasks across indoor and outdoor scenes, such as triangular meshing, texture mapping, etc. can also

The above describes in detail the method for generating a panoramic image with depth information according to an embodiment of the present disclosure, and the embodiment of the present disclosure further provides an apparatus for generating a panoramic image with depth information, hereinafter referring to FIG. A detailed explanation will be given together.

FIG. 4 is a schematic block diagram of an apparatus 400 for generating panoramic images with depth information according to some embodiments of the present disclosure. As shown in FIG. 4, the apparatus 400 can include an acquisition unit 401, a stitching unit 402, a processing unit 403 and a trimming unit 404. As shown in FIG.

Acquisition unit 401 is used to acquire a two-dimensional image of the current scene based on spherical projection.

The stitching unit 402 is used to construct an intermediate image by stitching a preset number of two-dimensional images horizontally end-to-end.

A processing unit 403 is used to determine the depth information of the intermediate images using a trained neural network model for predicting image depth.

The trimming unit 404 trims the intermediate image based on the intermediate position in the horizontal direction of the intermediate image to obtain an image equal to the length of the two-dimensional image, determines the depth information of the trimmed image, and determines the depth information of the trimmed image. is used to establish the cropped image of the current scene as the panorama image of the current scene.

FIG. 5 is a schematic block diagram of an apparatus 500 for generating panoramic images with depth information according to some further embodiments of the present disclosure. As shown in FIG. 5, the apparatus 500 includes an acquisition unit 501, a stitching unit 502, a processing unit 503 and a trimming unit 504. As shown in FIG. In some embodiments, the acquiring unit 501, the stitching unit 502, the processing unit 503 and the trimming unit 504 are implemented based on the above acquiring unit 401, the stitching unit 402, the processing unit 403 and the trimming unit 404 respectively. be able to.

In some embodiments, the neural network model for predicting image depth includes a pre-trained convolutional neural network as an initial backbone network, structurally adjusting the initial backbone network according to the U-Net network structure, It can be trained by utilizing multiple depth-informed color 3D images to generate a neural network model for predicting image depth by training an initial backbone network that is tuned.

In some embodiments, the pre-trained convolutional neural network is an ImageNet pre-trained DenseNet network. For example, the DenseNet network may be a DenseNet-169 network.

In some embodiments, the processing unit 503 can include a deletion subunit 5031 , an addition subunit 5032 and an arithmetic subunit 5033 .

Deletion subunit 5031 is used to delete all connectivity layers of the DenseNet network.

The adding subunit 5032 adds multiple upsampling layers after the last layer of the DenseNet network with all connected layers removed, based on the U-Net network structure, and each upsampling layer in the multiple upsampling layers respectively. is used to set the superposition layer corresponding to . The number of input channels for each upsampling layer is a preset multiple of the number of output channels. for each added upsampling layer, upsampling the input information of the upsampling layer at a preset resolution multiple, superimposing the upsampling result with the output data of the superimposed layer corresponding to the upsampling layer, Perform at least one convolution operation on the convolution result and linearly modify the convolution result using a preset activation function.

In some examples, the DenseNet network is a DenseNet-169 network. Four upsampling layers can be added after the last layer of the DenseNet-169 network. The number of input channels for each upsampling layer is twice the number of output channels. In some examples, at each added upsampling layer, the input information for that upsampling layer is upsampled at twice the resolution. In some examples, two convolution operations are performed on the convolution result.

The arithmetic subunit 5033 performs one convolution operation for outputting depth information and one convolution operation for outputting reliability information on the output of the last upsampling layer in the plurality of upsampling layers. It is used to perform

In some embodiments, the DenseNet network is a DenseNet-169 network. For four upsampling layers, according to the order from the first layer to the last layer of the four upsampling layers, the convolution layers corresponding to the four upsampling layers sequentially are pool3_pool, pool2_pool, pool1, conv1/relu, respectively. may be

In some embodiments, the processing unit 503 further preprocesses the plurality of depth-informed color 3D images, and in preprocessing the plurality of depth-informed color 3D images, preprocesses at least one depth-informed color 3D image. When it is determined that the color 3D image with depth information has a hole pointing to a pixel point for which depth information cannot be determined in the color 3D image with depth information, it is used to prevent the hole from being filled.

In some embodiments, the processing unit 503 further trains using a supervised learning method in the course of training the initial adjusted backbone network utilizing the plurality of depth-informed color 3D images, The loss function used for depth estimation is used because it is a function of the depth and confidence estimates for each pixel based on the neural network model.

In some embodiments, the acquisition unit 501 is further used to black fill the visual angle blind area in the two-dimensional image in response to determining that there is a visual angle blind area in the two-dimensional image.

In some embodiments, stitching unit 502 is also used to trim the top and bottom edges of the intermediate image.

In some embodiments, the stitching unit 502 trims the top and bottom edges of the intermediate image by cropping the images from the top and bottom edges of the intermediate image to a height that is a preset percentage of the height of the intermediate image, respectively. used to trim the

In some embodiments, the preset number value may be 3 and the preset percentage may be 15%.

Embodiments of the present disclosure provide a non-transitory computer program stored with a computer program that, when executed by a processor of an electronic device, causes the processor to perform a method of generating a panoramic image with depth information according to embodiments of the present disclosure. A computer-readable storage medium is further provided.

Embodiments of the present disclosure further provide an electronic device. FIG. 6 is a schematic diagram of an electronic device 600 according to an embodiment of the present disclosure. As shown in FIG. 6, electronic device 600 includes memory 601 and processor 602 . Memory 601 and processor 602 can be connected via a bus. The memory 601 is used to store a computer program executable by the processor 602, which, when executed by the processor 602, can implement a method of generating a panoramic image with depth information according to embodiments of the present disclosure. can.

The foregoing are merely illustrative examples of the present disclosure and are not intended to limit the content of the present disclosure. Any modification, equivalent change, improvement, etc. made without departing from the spirit and principle of this disclosure shall fall within the protection scope of this disclosure.

Claims (20)

A method for generating a panoramic image with depth information, comprising:
obtaining a two-dimensional image of the current scene based on spherical projection;
forming an intermediate image by connecting a predetermined number of the two-dimensional images in the horizontal direction end-to-end;
determining depth information for the intermediate image using a trained neural network model for predicting image depth;
trimming the intermediate image based on the intermediate position in the horizontal direction of the intermediate image to obtain an image equal to the length of the two-dimensional image; determining depth information of the trimmed image; establishing the cropped image as a panoramic image of a current scene. The neural network model for predicting image depth comprises:
taking a pre-trained convolutional neural network as an initial backbone network, and structurally adjusting the initial backbone network according to the U-Net network structure;
2. Trained by utilizing a plurality of depth-informed color 3D images to train the initial adjusted backbone network to generate a neural network model for predicting the image depth. described method. the pre-trained convolutional neural network is an ImageNet pre-trained DenseNet network;
Structurally adjusting the initial backbone network according to the U-Net network structure includes:
removing all connectivity layers of the DenseNet network;
Based on the U-Net network structure, adding a plurality of upsampling layers after the last layer of the DenseNet network from which the all-connected layer is removed, each corresponding to each upsampling layer in the plurality of upsampling layers. setting up superposition layers, the number of input channels of each upsampling layer being a preset multiple of its number of output channels;
for each added upsampling layer, upsampling the input information of the upsampling layer at a preset resolution multiple, superimposing the upsampling result with the output data of the superimposed layer corresponding to the upsampling layer, performing at least one convolution operation on the convolution result and linearly modifying the convolution operation result using a preset activation function;
Performing one convolution operation for outputting depth information and one convolution operation for outputting reliability information on the output of the last upsampling layer among the plurality of upsampling layers. 3. The method of claim 2, comprising: the DenseNet network is a DenseNet-169 network, the plurality of upsampling layers is four upsampling layers;
the number of input channels for each upsampling layer is twice its number of output channels;
for each added upsampling layer, upsampling the input information for that upsampling layer at twice the resolution;
4. The method of claim 3, wherein said at least one convolution operation is two convolution operations. For the four upsampling layers, according to the order from the first layer to the last layer of the four upsampling layers, the convolution layers sequentially corresponding to the four upsampling layers are pool3_pool, pool2_pool, pool1, respectively. 5. The method of claim 4, wherein conv1/relu. Prior to training the initial adjusted backbone network utilizing multiple depth-informed color 3D images, the method comprises:
In the process of pre-processing the plurality of depth-information-attached color 3D images, and pre-processing the plurality of depth-information-attached color 3-D images, at least one depth-information-attached color 3-D image includes the depth 3. The method of claim 2, further comprising, when determining that there is a hole pointing to a pixel point for which depth information cannot be determined in the color three-dimensional image with information, not performing hole-filling processing on the hole. training using a supervised learning method in the process of training the initial adjusted backbone network using the plurality of depth-informed color 3D images;
3. The method of claim 2, wherein the loss function used for depth estimation is a function of depth and confidence estimates for each pixel based on a neural network model. further comprising: after obtaining a spherical projection-based two-dimensional image of a current scene, in response to determining that there is a visual angle blind area in the two-dimensional image, filling the visual angle blind area with black. Item 8. The method according to any one of Items 1 to 7. The method according to any one of claims 1 to 7, further comprising trimming the upper and lower edges of the intermediate image after constructing an intermediate image by connecting a preset number of the two-dimensional images horizontally end-to-end. A method according to any one of paragraphs. Trimming the upper and lower edges of the intermediate image includes:
10. The method of claim 9, comprising cropping an image of height a preset percentage of the height of the intermediate image from the top and bottom edges of the intermediate image, respectively. A device for generating a panoramic image with depth information,
an acquisition unit for acquiring a two-dimensional image of the current scene based on spherical projection;
a stitching unit for constructing an intermediate image by stitching a preset number of said two-dimensional images horizontally end-to-end;
a processing unit for determining depth information for said intermediate image utilizing a trained neural network model for predicting image depth;
trimming the intermediate image based on the intermediate position in the horizontal direction of the intermediate image to obtain an image equal to the length of the two-dimensional image; determining depth information of the trimmed image; a cropping unit for establishing the cropped image as a panoramic image of a current scene. The neural network model for predicting image depth comprises:
taking a pre-trained convolutional neural network as an initial backbone network, and structurally adjusting the initial backbone network according to the U-Net network structure;
12. Trained by utilizing a plurality of depth-informed color 3D images to train the initial adjusted backbone network to generate a neural network model for predicting the image depth. Apparatus as described. the pre-trained convolutional neural network is an ImageNet pre-trained DenseNet network;
The processing unit is
a deletion subunit for deleting all connectivity layers of said DenseNet network;
Based on the U-Net network structure, adding a plurality of upsampling layers after the last layer of the DenseNet network from which the all-connected layer is removed, each corresponding to each upsampling layer in the plurality of upsampling layers. setting up a superposition layer, the number of input channels of each upsampling layer is a preset multiple of its output channel number, and in each added upsampling layer, the input information of the upsampling layer is converted to a preset resolution Upsampling by a factor, convolving the upsampling result with the output data of the convolution layer corresponding to the upsampling layer, performing at least one convolution operation on the convolution result, and using a preset activation function. an additive subunit for linearly correcting the convolution operation result by
for performing one convolution operation for outputting depth information and one convolution operation for outputting reliability information on the output of the last upsampling layer among the plurality of upsampling layers; 13. The apparatus of claim 12, comprising a computational subunit. the DenseNet network is a DenseNet-169 network, the plurality of upsampling layers is four upsampling layers;
the number of input channels for each upsampling layer is twice its number of output channels;
for each added upsampling layer, upsampling the input information for that upsampling layer at twice the resolution;
14. The apparatus of claim 13, wherein said at least one convolution operation is two convolution operations. The processing unit further comprises:
In the process of pre-processing the plurality of depth-information-attached color 3D images, and pre-processing the plurality of depth-information-attached color 3-D images, at least one depth-information-attached color 3-D image includes the depth 13. Apparatus according to claim 12, wherein when it is determined that there is a hole pointing to a pixel point for which depth information cannot be determined in the color three-dimensional image with information, the hole is not filled. The acquisition unit further comprises:
Apparatus according to any one of claims 11 to 15, adapted for black-filling the visual angle blind area in response to determining that there is a visual angle blind area in the two-dimensional image. The splicing unit further comprises:
A device according to any one of claims 11 to 15, used for trimming the upper and lower edges of said intermediate image. The splicing unit further comprises:
Apparatus according to any one of claims 11 to 15, used for cropping an image with a height of a preset ratio of the height of the intermediate image from the upper and lower edges of said intermediate image respectively. . A non-transitory computer readable storage medium having stored thereon a computer program comprising instructions which, when executed by a processor of an electronic device, cause said processor to perform the method of any one of claims 1 to 10. an electronic device,
a processor;
a memory for storing a computer program executable by said processor;
Electronic equipment, wherein the computer program, when executed by the processor, causes the processor to perform the method according to any one of claims 1 to 10.

Images