CN114820344A - Depth map enhancement method and device - Google Patents

Abstract

The application discloses a depth map enhancement method and device, wherein the method comprises the following steps: acquiring an initial depth map from original visual data, performing multi-scale feature extraction of an alternating convolution and deconvolution module on the initial depth map to obtain a feature view, and performing scale compression and convolution to obtain two-stage feature vectors in sequence; and carrying out spatial transformation and feature extraction on the initial depth map to obtain depth map features, strengthening the depth map features based on the feature vectors of the two stages, recovering the depth structure, generating a feature map, and mapping by a multilayer perceptron to obtain a final depth map. Therefore, the technical problems that in the related technology, due to the fact that a target object needs to meet certain specific geometric symmetry, feature extraction and prediction are conducted, in the process of deep acquisition, completion and densification of geometric features are difficult to conduct through a universal enhancement model, and the conditions of information deficiency and low accuracy exist in acquisition are solved.

Description

Depth map enhancement method and device

technical field

The present application relates to the technical field of computer vision, and in particular, to a depth map enhancement method and device.

Background technique

With the advancement of technology, the acquisition and acquisition of stereo object models has become easier and easier, especially depth maps have attracted more and more research attention due to their applications in various fields such as autonomous driving and robotics.

Due to many factors such as low resolution and occlusion of the device, the quality of depth data directly obtained by depth scanning devices such as lidar and depth cameras is usually poor. Low-quality data is mainly manifested in incomplete depth spatial structure and information redundancy, etc. Therefore, enhancement methods and systems for depth maps are becoming more and more important in practical engineering applications.

In the related art, the completion and densification methods based on geometric features are used, which require the target object to satisfy a certain geometric symmetry, and then use the acquired images for feature extraction and prediction. However, the geometric differences between real-world objects are quite large, and it is difficult to use a general enhancement model to complete and densify geometric features in the process of depth acquisition, resulting in incomplete information and low accuracy in the acquisition, which needs to be improved urgently. .

SUMMARY OF THE INVENTION

The present application provides a depth map enhancement method and device to solve the problem that in the related art, since the target object needs to meet a certain geometric symmetry before feature extraction and prediction, it is difficult to use a general enhancement in the process of depth acquisition. The model performs the completion and densification of geometric features, which causes technical problems such as incomplete information and low accuracy in the acquisition.

The embodiment of the first aspect of the present application provides a depth map enhancement method, including the following steps: obtaining an initial depth map from original visual data; performing multi-scale feature extraction on the initial depth map by alternating convolution and deconvolution modules, The deconvolution module performs step-by-step recovery to obtain a three-channel feature view with the three-dimensional coordinates of the depth map; scale compression and convolution are performed on the feature view, and two-stage feature vectors are obtained in turn; The image is subjected to spatial transformation and feature extraction to obtain depth map features, and the depth map features are enhanced based on the two-stage feature vectors, and the depth structure is restored to generate a first three-channel feature map; The second three-channel feature map is obtained by multi-layer perceptron mapping, and the first three-channel feature map and the second three-channel feature map are fused, and the final depth map is obtained by the multi-layer perceptron mapping.

Optionally, in an embodiment of the present application, the obtaining an initial depth map from raw visual data includes: obtaining a low-quality depth map that satisfies a preset condition from the raw visual data; image and infrared and depth map raw data for preprocessing and feature extraction to generate the initial depth map.

Optionally, in an embodiment of the present application, the obtaining a low-quality depth map that satisfies a preset condition from the raw visual data includes: collecting the raw visual data based on a preset collection viewing angle threshold, The low quality depth map is obtained.

Optionally, in an embodiment of the present application, performing spatial transformation and feature extraction on the initial depth map to obtain depth map features, and enhancing the depth map features based on the two-stage feature vectors , and restore the depth structure to generate a first three-channel feature map, including: performing spatial three-dimensional transformation on the initial depth map to obtain a three-dimensional transformed depth; extracting a first feature based on the three-dimensional transformed depth, and converting the The first feature is fused with the view feature vector of the feature view to obtain a first-stage fusion feature; a second feature is extracted based on the first-stage fusion feature, and the second feature is fused with the view feature vector, The second-stage fusion feature is obtained; the first three-channel feature map is obtained by the multi-layer perceptron mapping of the second-stage fusion feature.

An embodiment of the second aspect of the present application provides a depth map enhancement device, including: an acquisition module for acquiring an initial depth map from original visual data; a feature extraction module for performing alternate convolution and inversion on the initial depth map The multi-scale feature extraction of the convolution module is gradually restored by the deconvolution module, and the feature view with the three-dimensional coordinates of the depth map as the three-channel is obtained; the calculation module is used to perform scale compression and convolution on the feature view. , and obtain two-stage feature vectors in turn; the enhancement module is used to perform spatial transformation and feature extraction on the initial depth map to obtain depth map features, and strengthen the depth map features based on the two-stage feature vectors, and restore the depth structure to generate a first three-channel feature map; and a fusion module for obtaining a second three-channel feature map from the feature view multi-layer perceptron mapping, and fuse the first three-channel feature map and the The second three-channel feature map is mapped by a multilayer perceptron to obtain the final depth map.

Optionally, in an embodiment of the present application, the acquisition module includes: an acquisition unit, configured to obtain a low-quality depth map that satisfies a preset condition from the original visual data; a preprocessing unit, configured to The low-quality depth map and the infrared and depth map raw data are preprocessed and feature extracted to generate the initial depth map.

Optionally, in an embodiment of the present application, the acquiring unit is further configured to acquire the low-quality depth map by acquiring the raw visual data based on a preset acquisition viewing angle threshold.

Optionally, in an embodiment of the present application, the enhancement module includes: a three-dimensional transformation unit, configured to perform spatial three-dimensional transformation on the initial depth map to obtain a three-dimensional transformed depth; a first fusion unit, configured to The first feature is extracted based on the depth after the three-dimensional transformation, and the first feature is fused with the view feature vector of the feature view to obtain the first-stage fusion feature; the second fusion unit is used for The stage fusion feature extracts the second feature, and fuses the second feature with the view feature vector to obtain the second stage fusion feature; the mapping unit is used to map the second stage fusion feature multi-layer perceptron to obtain the second stage fusion feature. The first three-channel feature map is described.

An embodiment of a third aspect of the present application provides an electronic device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor executes the program to achieve The depth map enhancement method described in the above embodiments.

Embodiments of the fourth aspect of the present application provide a computer-readable storage medium on which a computer program is stored, and the program is executed by a processor, so as to implement the depth map enhancement method according to any one of claims 1-4 .

In this embodiment of the present application, an initial depth map can be obtained from the original visual data, two-stage feature vectors and depth map features can be extracted from the initial depth map and enhanced, the depth structure can be restored, and a final depth map can be obtained through multi-layer perceptron mapping , while maintaining the original depth geometry, it can complete the missing shape areas in the original data, and densify the overall depth map. It has the ability to adapt to shape defects and sparseness of different degrees and angles. Improve the effective output accuracy of low-precision acquisition equipment and improve the quality of data acquisition. Therefore, in the related art, since the target object needs to meet a certain geometric symmetry, and then perform feature extraction and prediction, it is difficult to use a general enhancement model to complete and dense geometric features in the process of depth acquisition. The technical problems of incomplete information and low precision exist in the collection.

Additional aspects and advantages of the present application will be set forth, in part, in the following description, and in part will be apparent from the following description, or learned by practice of the present application.

Description of drawings

The above and/or additional aspects and advantages of the present application will become apparent and readily understood from the following description of embodiments taken in conjunction with the accompanying drawings, wherein:

1 is a flowchart of a depth map enhancement method provided according to an embodiment of the present application;

FIG. 2 is a schematic diagram of the principle of a depth map enhancement method according to an embodiment of the present application;

3 is a schematic diagram of the principle of an enhancement network of a depth map enhancement method according to an embodiment of the present application;

4 is a schematic diagram of a generator of a depth map enhancement method according to an embodiment of the present application;

5 is a flowchart of a depth map enhancement method according to a specific embodiment of the present application;

FIG. 6 is a schematic structural diagram of a depth map enhancement apparatus provided according to an embodiment of the present application;

FIG. 7 is a schematic structural diagram of an electronic device provided according to an embodiment of the present application.

Detailed ways

The following describes in detail the embodiments of the present application, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals refer to the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary, and are intended to be used to explain the present application, but should not be construed as a limitation to the present application.

The depth map enhancement method and apparatus according to the embodiments of the present application are described below with reference to the accompanying drawings. In the related art mentioned in the above-mentioned background technology center, since the target object needs to meet a certain geometric symmetry, and then perform feature extraction and prediction, it is difficult to use a general enhancement model to complement the geometric features in the process of depth acquisition. Full-sum densification leads to the technical problems of incomplete information and low precision in acquisition. The present application provides a depth map enhancement method. In this method, an initial depth map can be obtained from the original visual data. In the figure, the two-stage feature vector and depth map features are extracted and enhanced to restore the depth structure, and the final depth map is obtained through multi-layer perceptron mapping. Completing and densifying the overall depth map has the ability to adapt to shape defects and sparseness of different degrees and angles, which can effectively improve the effective output accuracy of low-precision acquisition equipment and improve the quality of data acquisition. Therefore, in the related art, since the target object needs to meet a certain geometric symmetry, and then perform feature extraction and prediction, it is difficult to use a general enhancement model to complete and dense geometric features in the process of depth acquisition. The technical problems of incomplete information and low precision exist in the collection.

Specifically, FIG. 1 is a schematic flowchart of a depth map enhancement method provided by an embodiment of the present application.

As shown in Figure 1, the depth map enhancement method includes the following steps:

In step S101, an initial depth map is obtained from the original visual data.

In the actual implementation process, the embodiments of the present application may use a depth camera and a camera to perform matching acquisition to obtain raw visual data, and after obtaining views and sparse and incomplete low-quality depth maps from the raw visual data, processes including but not limited to data processing Then get the initial depth map.

Optionally, in an embodiment of the present application, obtaining an initial depth map from raw visual data includes: obtaining a low-quality depth map that meets preset conditions from the raw visual data; The raw data is preprocessed and feature extracted to generate an initial depth map.

As a possible implementation manner, in this embodiment of the present application, a low-quality depth map that satisfies preset conditions can be obtained from raw visual data, and the raw data of view, infrared, and depth maps obtained from the raw data can be preprocessed and feature extracted. , to generate an initial depth map for processing in subsequent steps. The embodiments of the present application use infrared data as an aid to enhance the data output of a low-power and low-performance depth camera and a lidar, which will be described in detail below.

It should be noted that the preset condition of the low-quality depth map can be set by those skilled in the art according to the actual situation, and no specific limitation is made here.

Optionally, in an embodiment of the present application, obtaining a low-quality depth map that satisfies a preset condition from raw visual data includes: collecting the raw visual data based on a preset collection viewing angle threshold to obtain a low-quality depth map.

Specifically, in this embodiment of the present application, a depth camera and a camera may be used for matching collection, and raw visual data is collected based on a preset collection viewing angle threshold to obtain a low-quality depth map. The embodiment of the present application performs original visual data collection based on a preset collection viewing angle threshold, which is beneficial to the subsequent enhancement and densification of low-quality depth maps, and lays a foundation for generating high-quality images.

It should be noted that the preset viewing angle threshold will change correspondingly according to different acquisition targets, and the viewing angle threshold may be set by those skilled in the art according to the actual situation, which is not specifically limited here.

In step S102, multi-scale feature extraction is performed on the initial depth map by alternating convolution and deconvolution modules, and the deconvolution module performs stepwise restoration to obtain a feature view with three-channel depth map three-dimensional coordinates.

Further, in this embodiment of the present application, multi-scale feature extraction by alternating convolution and deconvolution modules can be performed on the initial depth map obtained through the above steps, and the deconvolution module can perform stepwise recovery to obtain the three-dimensional coordinates of the depth map as three dimensions. For the feature view of the channel, the embodiment of the present application can extract multi-modal features by gradually extracting geometric features at different levels, so as to facilitate subsequent feature fusion and restore and generate high-precision images.

In step S103, scale compression and convolution are performed on the feature view, and two-stage feature vectors are obtained in sequence.

In the actual implementation process, the embodiment of the present application can perform scale compression and convolution on the feature view and infrared features, so as to obtain two-stage feature vectors, realize multi-modal feature extraction, and facilitate subsequent feature fusion and recovery generation. High-precision images.

In step S104, spatial transformation and feature extraction are performed on the initial depth map to obtain depth map features, and the depth map features are enhanced based on the two-stage feature vectors, and the depth structure is restored to generate a first three-channel feature map.

Specifically, this embodiment of the present application may perform spatial transformation and feature extraction on the initial depth map obtained in step S101, and input the two-stage feature vector obtained in step S103 to enhance the depth map features and restore the depth structure, and then generate The first three-channel feature map. By strengthening the depth map features and restoring the depth structure, the embodiment of the present application is beneficial to complete the missing shape regions in the original data while maintaining the original depth geometric structure, and to densify the overall depth map. The ability to adapt to incomplete and sparse shapes of different degrees and angles can effectively improve the effective output accuracy of low-precision acquisition devices and improve the quality of data acquisition.

Optionally, in an embodiment of the present application, spatial transformation and feature extraction are performed on the initial depth map to obtain depth map features, and the depth map features are enhanced based on the two-stage feature vector, and the depth structure is restored to generate the first depth map feature. A three-channel feature map, including: performing spatial three-dimensional transformation on the initial depth map to obtain a three-dimensional transformed depth; extracting a first feature based on the three-dimensional transformed depth, and fusing the first feature with the view feature vector of the feature view to obtain The first-stage fusion feature; the second feature is extracted based on the first-stage fusion feature, and the second feature is fused with the view feature vector to obtain the second-stage fusion feature; the second-stage fusion feature multi-layer perceptron mapping is used to obtain the first and third stage. Channel feature map.

For example, in the actual execution process, the steps of performing deep feature enhancement in this embodiment of the present application are as follows:

1. Perform spatial 3D transformation on the initial depth map to obtain the 3D transformed depth;

2. Extract the first feature according to the depth after the three-dimensional transformation, and fuse the first feature with the view feature vector of the feature view to obtain the first-stage fusion feature;

3. Extract the second feature through the first-stage fusion feature, and fuse the second feature with the view feature vector to obtain the second-stage fusion feature;

4. The first three-channel feature map is obtained by the second-stage fusion feature multi-layer perceptron mapping.

For example, as shown in Figure 2, the method of deep feature enhancement is:

The point branch extracts features from points and generates enhanced point features using an attention mask generated from the image features in the image branch;

The augmented point features are then forwarded to a fully connected layer to reconstruct a point set representing the global geometry that contributes to the final prediction of another subset of depths.

Specifically, the embodiments of the present application can convert the original N input points represented in 3D in the Euclidean space (N×3) into the fixed dimension C in the feature space (N×C), and use the EdgeConv learning module proposed in DGCNN. Extract point features.

In the spatial transformation layer, the embodiment of the present application can use the estimated 3×3 matrix to align the input point set to the canonical space. In order to estimate the 3×3 matrix, the embodiment of the present application can use a tensor to align the coordinates of each point and The coordinate differences of its k adjacent points are connected, and the image feature point feature ARC global feature CA×R is the feature enhancement module in the point branch of Figure 2.

Understandably, the feature augmentation module fuses global features from view modalities and geometric local features F _p extracted from point modalities via an attention mechanism. Specifically, the K-dimensional feature vector (shown in the top row of Fig. 2) from the view branch of the first enhancement unit, after concatenating the features with N times repeated points, is compressed into an Nx1-dimensional vector using MLP, which is converted by the sigmoid function to This vector is normalized to the range of [0, 1], _an attention mask ma (Nx1) can be obtained, after which the enhanced local point feature F is achieved by performing element- _wise multiplication of ma and the point feature F′ _p .

Further, the embodiments of the present application can obtain local features enhanced by global image features:

The final enhanced point feature _Fe (N×2×C) can be achieved by N-fold repeated concatenation of the local enhanced point feature F′ and the global point feature obtained by average pooling on _Fp .

In step S105, the second three-channel feature map is obtained from the feature view multi-layer perceptron mapping, and the first three-channel feature map and the second three-channel feature map are fused, and the multi-layer perceptron is mapped to obtain the final depth map.

As a possible implementation manner, in this embodiment of the present application, a multi-layer perceptron mapping may be performed on the compressed feature vector obtained in the above steps to obtain a second three-channel feature map, and the first and third channel feature maps obtained in the above steps may be The channel feature map and the second and third channel feature maps are fused, and then mapped by a multi-layer perceptron to obtain a restored dense and complete depth map. The embodiment of the present application can maintain the original depth geometric structure, complete the shape missing area in the original data, and densify the overall depth map, and has the ability to adapt to shape defects and sparseness of different degrees and angles , which can effectively improve the effective output accuracy of low-precision acquisition equipment and improve the quality of data acquisition.

Hereinafter, with reference to FIGS. 2 to 5 , an embodiment of the present application will be described in detail by taking an embodiment.

The embodiment of the present application includes the following steps:

Step S501: raw visual data acquisition. In the actual implementation process, the embodiments of the present application may use a depth camera and a camera, and perform matching collection of raw visual data based on a preset collection angle of view threshold to obtain raw visual data. Views and sparse and incomplete low-level data are obtained from the raw visual data. Quality depth map.

It should be noted that the preset viewing angle threshold will change correspondingly according to different acquisition targets, and the viewing angle threshold may be set by those skilled in the art according to the actual situation, which is not specifically limited here.

Step S502: preprocessing and feature extraction of the original data. As a possible implementation manner, in this embodiment of the present application, a low-quality depth map that satisfies preset conditions can be obtained from raw visual data, and the raw data of view, infrared, and depth maps obtained from the raw data can be preprocessed and feature extracted. , to generate an initial depth map for processing in subsequent steps.

It should be noted that the preset condition of the low-quality depth map can be set by those skilled in the art according to the actual situation, and no specific limitation is made here.

Step S503: Acquire a feature view using the three-dimensional coordinates of the depth map as three channels. Further, in this embodiment of the present application, multi-scale feature extraction by alternating convolution and deconvolution modules can be performed on the initial depth map obtained through the above steps, and the deconvolution module can perform stepwise recovery to obtain the three-dimensional coordinates of the depth map as three dimensions. For the feature view of the channel, the embodiment of the present application can extract multi-modal features by gradually extracting geometric features at different levels, so as to facilitate subsequent feature fusion and restore and generate high-precision images.

Step S504: Obtain two-stage feature vectors. In the actual implementation process, the embodiment of the present application can perform scale compression and convolution on the feature view and infrared features, so as to obtain two-stage feature vectors, realize multi-modal feature extraction, and facilitate subsequent feature fusion and recovery generation. High-precision images.

Step S505: Enhance the depth map feature and restore the depth structure. Specifically, this embodiment of the present application may perform spatial transformation and feature extraction on the initial depth map obtained in step S101, and input the two-stage feature vector obtained in step S103 to enhance the depth map features and restore the depth structure, and then generate The first three-channel feature map. By strengthening the depth map features and restoring the depth structure, the embodiment of the present application is beneficial to complete the missing shape regions in the original data while maintaining the original depth geometric structure, and to densify the overall depth map. The ability to adapt to incomplete and sparse shapes of different degrees and angles can effectively improve the effective output accuracy of low-precision acquisition devices and improve the quality of data acquisition.

In the actual execution process, the steps of deep feature enhancement in this embodiment of the present application are as follows:

1. Perform spatial 3D transformation on the initial depth map to obtain the 3D transformed depth;

2. Extract the first feature according to the depth after the three-dimensional transformation, and fuse the first feature with the view feature vector of the feature view to obtain the first-stage fusion feature;

3. Extract the second feature through the first-stage fusion feature, and fuse the second feature with the view feature vector to obtain the second-stage fusion feature;

4. The first three-channel feature map is obtained by the second-stage fusion feature multi-layer perceptron mapping.

For example, as shown in Figure 2, the method of deep feature enhancement is:

The point branch extracts features from points and generates enhanced point features using an attention mask generated from the image features in the image branch;

The augmented point features are then forwarded to a fully connected layer to reconstruct a point set representing the global geometry that contributes to the final prediction of another subset of depths.

Specifically, the embodiments of the present application can convert the original N input points represented in 3D in the Euclidean space (N×3) into the fixed dimension C in the feature space (N×C), and use the EdgeConv learning module proposed in DGCNN. Extract point features.

In the spatial transformation layer, the embodiment of the present application can use the estimated 3×3 matrix to align the input point set to the canonical space. In order to estimate the 3×3 matrix, the embodiment of the present application can use a tensor to align the coordinates of each point and The coordinate differences of its k adjacent points are connected, and the image feature point feature ARC global feature CA×R is the feature enhancement module in the point branch of Figure 2.

Understandably, the feature augmentation module fuses global features from view modalities and geometric local features F _p extracted from point modalities via an attention mechanism. Specifically, the K-dimensional feature vector (shown in the top row of Fig. 2) from the view branch of the first enhancement unit, after concatenating the features with N times repeated points, is compressed into an Nx1-dimensional vector using MLP, which is converted by the sigmoid function to This vector is normalized to the range of [0, 1], _an attention mask ma (Nx1) can be obtained, after which the enhanced local point feature F is achieved by performing element- _wise multiplication of ma with the point feature F ^′ _p .

Further, the embodiments of the present application can obtain local features enhanced by global image features:

The final enhanced point feature _Fe (N×2×C) can be achieved by N-fold repeated concatenation of the local enhanced point feature F′ and the global point feature obtained by average pooling on _Fp .

Step S506 : fuse the three-channel feature maps, and obtain a restored dense and complete depth map through multi-layer perceptron mapping. As a possible implementation manner, in this embodiment of the present application, a multi-layer perceptron mapping may be performed on the compressed feature vector obtained in the above steps to obtain a second three-channel feature map, and the first and third channel feature maps obtained in the above steps may be The channel feature map and the second and third channel feature maps are fused, and then mapped by a multi-layer perceptron to obtain a restored dense and complete depth map. The embodiment of the present application can maintain the original depth geometric structure, complete the shape missing area in the original data, and densify the overall depth map, and has the ability to adapt to shape defects and sparseness of different degrees and angles , which can effectively improve the effective output accuracy of low-precision acquisition equipment and improve the quality of data acquisition.

Further, the depth map enhancement method in the embodiment of the present application mainly includes the following aspects:

1. Multimodal feature extraction, fusion and restoration generation. Deep augmentation network is an adversarial architecture, including generator and discriminator, as shown in Figure 3, augmentation network is an adversarial architecture, including generator and discriminator. As shown in Fig. 4, the generator is a cascade of two enhancement units with similar structure, taking a low-quality depth X and a single-view reference image I as input, and each enhancement unit consists of three parallel functional branches: view branch, point branch and fusion branch, which predict point sets by processing view, infrared features, depth features and multimodal image point fusion features, respectively.

As shown in Figure 4, the depth generator has two cascaded enhancement units, the first unit takes the original image I and depth map X directly as input and transfers them into the latent embedding space, and then forwards the features to the next One element; the second element is functionally similar to the first element, but the output is a 3D point set X. From a data flow perspective, the generator consists of three parallel branches that predict depth from view, depth, and multimodal image point features, respectively, and the output depth is the union of the predicted point sets of the three branches.

Among them, the view branch takes images and infrared images as input and gradually extracts different levels of geometric features.

The depth branch, the point branch extracts features from the points and generates augmented point features using an attention mask generated from the image features in the image branch, thereby forwarding the augmented point features to a fully connected layer to reconstruct the points representing the global geometry , which contributes to the final prediction of another subset of depths.

Fusion branch, the fusion branch mainly fuses two feature streams.

2. Generate a deep discriminator. Adversarial training has facilitated the development of a range of representational and generative applications in image representation, but little work has been done on applying this architecture to depth completion, so embodiments of this application can add a discriminator to the framework Adversarial training is performed with the goal of identifying true and false depths. Embodiments of the present application can use a joint loss function that considers completeness and distribution uniformity to train the network to obtain a "coarse" full depth, then fine-tune the network with an adversarial loss to achieve a "fine" full depth, and use PointNet with A binary classification network that acts as a discriminator to distinguish predictions from either the generated set X or the real set Y. Embodiments of this application can be trained using adversarial losses in an improved Wasserstein GAN:

分别是从生成的数据和实际数据中提取的点集样本，第二项中的多项式是梯度罚分。Among them, D represents the set of 1-Lipschitz functions, y~PY and

are the point set samples drawn from the generated data and the real data, respectively, and the polynomial in the second term is the gradient penalty.

According to the depth map enhancement method proposed in the embodiment of the present application, an initial depth map can be obtained from the original visual data, and two-stage feature vectors and depth map features can be extracted from the initial depth map and enhanced to restore the depth structure. The layer perceptron maps to obtain the final depth map. While maintaining the original depth geometry, it can complete the shape missing areas in the original data, and densify the overall depth map, which can adapt to different degrees and different angles. and sparse ability, which can effectively improve the effective output accuracy of low-precision acquisition equipment and improve the quality of data acquisition. This solves the problem that in the related art, during data collection, due to the large geometric differences of objects, it is difficult to perform feature completion through geometric symmetry, so that the collected data has incomplete information and low precision, and it is difficult to achieve high precision. technical issues of image output.

Next, the depth map enhancement device proposed according to the embodiments of the present application will be described with reference to the accompanying drawings.

FIG. 6 is a schematic block diagram of a depth map enhancement apparatus according to an embodiment of the present application.

As shown in FIG. 6 , the depth map enhancement apparatus 10 includes: an acquisition module 100 , a feature extraction module 200 , a calculation module 300 , an enhancement module 400 and a fusion module 500 .

Specifically, the obtaining module 100 is configured to obtain the initial depth map from the original visual data.

The feature extraction module 200 is used to perform multi-scale feature extraction of alternating convolution and deconvolution modules on the initial depth map, and the deconvolution module performs gradual recovery to obtain a feature view with three-channel depth map three-dimensional coordinates.

The computing module 300 is configured to perform scale compression and convolution on the feature view, and obtain two-stage feature vectors in turn.

The enhancement module 400 is configured to perform spatial transformation and feature extraction on the initial depth map to obtain depth map features, and based on two-stage feature vectors to enhance the depth map features, restore the depth structure, and generate a first three-channel feature map.

The fusion module 500 is used to obtain the second three-channel feature map by mapping the feature view multi-layer perceptron, and fuse the first three-channel feature map and the second three-channel feature map, and map the multi-layer perceptron to obtain the final depth map.

Optionally, in an embodiment of the present application, the obtaining module 100 includes: an obtaining unit and a preprocessing unit.

Wherein, the obtaining unit is used to obtain a low-quality depth map that satisfies a preset condition from the original visual data.

A preprocessing unit for preprocessing and feature extraction on low-quality depth map and infrared and depth map raw data to generate an initial depth map.

Optionally, in an embodiment of the present application, the obtaining unit is further configured to collect raw visual data based on a preset collection viewing angle threshold to obtain a low-quality depth map.

Optionally, in an embodiment of the present application, the enhancement module 400 includes: a three-dimensional transformation unit, a first fusion unit, a second fusion unit, and a mapping unit.

The three-dimensional transformation unit is used to perform spatial three-dimensional transformation on the initial depth map to obtain the three-dimensional transformed depth.

The first fusion unit is configured to extract the first feature based on the depth after the three-dimensional transformation, and fuse the first feature with the view feature vector of the feature view to obtain the first-stage fusion feature.

The second fusion unit is configured to extract the second feature based on the first-stage fusion feature, and fuse the second feature with the view feature vector to obtain the second-stage fusion feature.

The mapping unit is used to obtain the first three-channel feature map by the second-stage fusion feature multi-layer perceptron mapping.

It should be noted that, the foregoing explanation of the embodiment of the depth map enhancement method is also applicable to the depth map enhancement apparatus of this embodiment, and details are not repeated here.

According to the depth map enhancement device proposed in the embodiment of the present application, an initial depth map can be obtained from the original visual data, two-stage feature vectors and depth map features can be extracted from the initial depth map and enhanced, and the depth structure can be restored. The layer perceptron is mapped to obtain the final depth map. While maintaining the original depth geometric structure, it can complete the shape missing area in the original data, and densify the overall depth map, which has the ability to adapt to different degrees and different angles. and sparse ability, which can effectively improve the effective output accuracy of low-precision acquisition equipment and improve the quality of data acquisition. This solves the problem that in the related art, during data collection, due to the large geometric differences of objects, it is difficult to perform feature completion through geometric symmetry, so that the collected data has incomplete information and low precision, and it is difficult to achieve high precision. technical issues of image output.

FIG. 7 is a schematic structural diagram of an electronic device provided by an embodiment of the present application. The electronic device may include:

Memory 701 , processor 702 , and computer programs stored on memory 701 and executable on processor 702 .

When the processor 702 executes the program, the depth map enhancement method provided in the above embodiments is implemented.

Further, the electronic device also includes:

The communication interface 703 is used for communication between the memory 701 and the processor 702 .

The memory 701 is used to store computer programs that can be executed on the processor 702 .

The memory 701 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk memory.

If the memory 701, the processor 702, and the communication interface 703 are independently implemented, the communication interface 703, the memory 701, and the processor 702 can be connected to each other through a bus and complete communication with each other. The bus may be an Industry Standard Architecture (referred to as ISA) bus, a Peripheral Component (referred to as PCI) bus, or an Extended Industry Standard Architecture (referred to as EISA) bus or the like. The bus can be divided into address bus, data bus, control bus and so on. For ease of presentation, only one thick line is used in FIG. 7, but it does not mean that there is only one bus or one type of bus.

Optionally, in terms of specific implementation, if the memory 701, the processor 702 and the communication interface 703 are integrated on one chip, the memory 701, the processor 702 and the communication interface 703 can communicate with each other through an internal interface.

The processor 702 may be a central processing unit (Central Processing Unit, CPU for short), or a specific integrated circuit (Application Specific Integrated Circuit, ASIC for short), or is configured to implement one or more of the embodiments of the present application integrated circuit.

This embodiment also provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, implements the above depth map enhancement method.

In the description of this specification, description with reference to the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples", etc., mean specific features described in connection with the embodiment or example , structure, material or feature is included in at least one embodiment or example of the present application. In this specification, schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the particular features, structures, materials or characteristics described may be combined in any suitable manner in any one or N of the embodiments or examples. Furthermore, those skilled in the art may combine and combine the different embodiments or examples described in this specification, as well as the features of the different embodiments or examples, without conflicting each other.

In addition, the terms "first" and "second" are only used for descriptive purposes, and should not be construed as indicating or implying relative importance or implying the number of indicated technical features. Thus, a feature delimited with "first", "second" may expressly or implicitly include at least one of that feature. In the description of the present application, "N" means at least two, such as two, three, etc., unless otherwise expressly and specifically defined.

Any process or method description in the flowchart or otherwise described herein may be understood to represent a module, segment or portion of code comprising one or N more executable instructions for implementing custom logical functions or steps of the process , and the scope of the preferred embodiments of the present application includes alternative implementations in which the functions may be performed out of the order shown or discussed, including performing the functions substantially concurrently or in the reverse order depending upon the functions involved, which should It is understood by those skilled in the art to which the embodiments of the present application belong.

Logic and/or steps represented in flowcharts or otherwise described herein, for example, may be considered an ordered listing of executable instructions for implementing the logical functions, may be embodied in any computer-readable medium, For use with, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can fetch instructions from and execute instructions from an instruction execution system, apparatus, or apparatus) or equipment. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport the program for use by or in connection with an instruction execution system, apparatus, or apparatus. More specific examples (non-exhaustive list) of computer readable media include the following: electrical connections with one or N wires (electronic devices), portable computer disk cartridges (magnetic devices), random access memory (RAM), Read Only Memory (ROM), Erasable Editable Read Only Memory (EPROM or Flash Memory), Fiber Optic Devices, and Portable Compact Disc Read Only Memory (CDROM). In addition, the computer readable medium may even be paper or other suitable medium on which the program may be printed, as the paper or other medium may be optically scanned, for example, followed by editing, interpretation, or other suitable medium as necessary process to obtain the program electronically and then store it in computer memory.

It should be understood that various parts of this application may be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, the N steps or methods may be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented by any one of the following techniques known in the art, or a combination thereof: discrete with logic gates for implementing logic functions on data signals Logic circuits, application specific integrated circuits with suitable combinational logic gates, Programmable Gate Arrays (PGA), Field Programmable Gate Arrays (FPGA), etc.

Those skilled in the art can understand that all or part of the steps carried by the methods of the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, and the program is stored in a computer-readable storage medium. When executed, one or a combination of the steps of the method embodiment is included.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing module, or each unit may exist physically alone, or two or more units may be integrated into one module. The above-mentioned integrated modules can be implemented in the form of hardware, and can also be implemented in the form of software function modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium.

The above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, and the like. Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limitations to the present application. Embodiments are subject to variations, modifications, substitutions and variations.

Claims (10)

1. a depth map enhancement method, is characterized in that, comprises the following steps: Obtain the initial depth map from the raw visual data; Perform multi-scale feature extraction of alternate convolution and deconvolution modules on the initial depth map, and gradually restore by the deconvolution module to obtain a three-channel feature view with the three-dimensional coordinates of the depth map; Scale compression and convolution are performed on the feature view, and two-stage feature vectors are obtained in turn; Spatial transformation and feature extraction are performed on the initial depth map to obtain depth map features, and the depth map features are enhanced based on the two-stage feature vectors, and the depth structure is restored to generate a first three-channel feature map; and A second three-channel feature map is obtained from the feature view multi-layer perceptron mapping, and the first three-channel feature map and the second three-channel feature map are fused, and the multi-layer perceptron is mapped to obtain a final depth map. 2. The method according to claim 1, wherein the obtaining an initial depth map from the original visual data comprises: obtaining a low-quality depth map that satisfies a preset condition from the original visual data; Perform preprocessing and feature extraction on the low-quality depth map and the infrared and depth map raw data to generate the initial depth map. 3. The method according to claim 1, wherein the obtaining a low-quality depth map that satisfies a preset condition from the original visual data comprises: The raw visual data is collected based on a preset collection viewing angle threshold to obtain the low-quality depth map. 4 . The method according to claim 1 , wherein the initial depth map is subjected to spatial transformation and feature extraction to obtain depth map features, and the depth map features are analyzed based on the two-stage feature vectors. 5 . Perform reinforcement and restore the deep structure to generate the first three-channel feature map, including: performing spatial three-dimensional transformation on the initial depth map to obtain a three-dimensional transformed depth; Extract the first feature based on the depth after the three-dimensional transformation, and fuse the first feature with the view feature vector of the feature view to obtain the first-stage fusion feature; Extract a second feature based on the first-stage fusion feature, and fuse the second feature with the view feature vector to obtain a second-stage fusion feature; The first three-channel feature map is obtained from the second-stage fusion feature multi-layer perceptron mapping. 5. A depth map enhancement device, comprising: an acquisition module for acquiring the initial depth map from the original visual data; The feature extraction module is used to perform multi-scale feature extraction of alternating convolution and deconvolution modules on the initial depth map, and is gradually restored by the deconvolution module to obtain the three-channel feature with the three-dimensional coordinates of the depth map. view; a computing module for performing scale compression and convolution on the feature view to obtain two-stage feature vectors in turn; The enhancement module is used to perform spatial transformation and feature extraction on the initial depth map to obtain depth map features, and strengthen the depth map features based on the feature vectors of the two stages, and restore the depth structure to generate the first three. channel feature maps; and The fusion module is used to obtain the second three-channel feature map from the feature view multi-layer perceptron mapping, and fuse the first three-channel feature map and the second three-channel feature map, and the multi-layer perceptron mapping obtains the final depth map. 6. The apparatus according to claim 5, wherein the acquiring module comprises: an acquisition unit, configured to obtain a low-quality depth map that satisfies a preset condition from the original visual data; A preprocessing unit, configured to perform preprocessing and feature extraction on the low-quality depth map and the original infrared and depth map data to generate the initial depth map. 7 . The apparatus according to claim 6 , wherein the acquiring unit is further configured to acquire the low-quality depth map by acquiring the raw visual data based on a preset acquisition viewing angle threshold. 8 . 8. The apparatus of claim 5, wherein the strengthening module comprises: a three-dimensional transformation unit, configured to perform spatial three-dimensional transformation on the initial depth map to obtain a three-dimensional transformed depth; a first fusion unit, configured to extract a first feature based on the depth after the three-dimensional transformation, and fuse the first feature with the view feature vector of the feature view to obtain a first-stage fusion feature; a second fusion unit, configured to extract a second feature based on the first-stage fusion feature, and fuse the second feature with the view feature vector to obtain a second-stage fusion feature; A mapping unit, configured to obtain the first three-channel feature map by mapping the second-stage fusion feature multi-layer perceptron. 9. An electronic device, characterized in that it comprises: a memory, a processor and a computer program stored on the memory and running on the processor, the processor executing the program to realize the method as claimed in the claim The depth map enhancement method described in any one of requirements 1-4. 10. A computer-readable storage medium on which a computer program is stored, characterized in that the program is executed by a processor to implement the depth map enhancement method according to any one of claims 1-4.

Images