RU2745010C1 - Methods for reconstruction of depth map and electronic computer device for their implementation - Google Patents

Abstract

FIELD: computing.

SUBSTANCE: invention relates to computing. A method for reconstructing a depth map includes the steps of obtaining an image and a sparse depth map containing depth data; the image feature tensor is calculated by the trained image encoder; calculating the trained depth encoder tensor of depth features; calculating the concatenated feature tensor by the converting module by concatenating the image feature tensor and the depth feature tensor and converting the concatenated features tensor by the transforming module using a predetermined parameter specifying the size of the transformed concatenated feature tensor; predicting a gradient depth map by a trained decoder using the transformed concatenated feature tensor and the tensor; and predicting the dense depth map by the iterative spatial distribution unit using the gradient depth map, the sparse depth map and the tensor obtained in the last upscaling block of the decoder data.

EFFECT: ensuring high quality depth map acquisition on low computing power devices in real time.

21 cl, 10 dwg

Description

BACKGROUND OF THE INVENTION

The technical field to which the invention relates

[0001] The present invention relates generally to the fields of computer vision, depth mapping using artificial intelligence and machine learning for reconstructing depth maps, in particular, to methods for reconstructing a depth map and an electronic computing device for their implementation.

Description of the prior art

[0002] Currently, there are several methods for reconstructing a depth map. Some methods use a high-precision multi-beam depth sensor and image to reconstruct the depth map on this image. A high-precision depth sensor produces high quality depth data that densely covers substantially the entire image field and from which a high quality depth map can be obtained. However, the use of a high-precision depth sensor in devices such as mobile phone, smartphone, tablet computer, robot vacuum cleaner, autonomous vehicle control device, etc. makes the device too expensive and can significantly increase the size of the device. Other methods use a simple low fidelity depth sensor and an image to reconstruct the depth map on that image. Such depth sensors produce low quality depth data with low data density, which is then processed by artificial intelligence to obtain a depth map of the required quality. Getting a high quality depth map from low quality depth data requires a device with high processing power, such as a computer with a high performance processor, high quality graphics card, and large memory. Devices such as mobile phone, smartphone, tablet computer, robot vacuum cleaner, autonomous vehicle control device, etc. do not have the characteristics of devices with high processing power, so it is difficult to obtain a high-quality depth map on them. In addition, the process of reconstructing a depth map with poor quality depth data is too slow to work in real time to produce an acceptable depth map.

[0003] In the prior art, technical solutions for depth map reconstruction are known, such as the solutions described below.

[0004] Patent Application CN102938142A, published 02/20/2013 and entitled "METHOD FOR FILLING INDOOR LIGHT DETECTION AND RANGING (LIDAR) MISSING DATA BASED ON KINECT", offers a technical solution to supplement the sparse depth data obtained by a laser depth sensor such as a lidar, depth data from the Kinect device. However, the Kinect device is expensive and the integration of the Kinect device into devices such as a mobile phone, smartphone, tablet computer, robot vacuum cleaner, autonomous vehicle control device, etc. will lead to a significant increase in the size of such devices. In addition, this technical solution uses a multi-beam lidar, as well as an iterative algorithm of the nearest points, which reduces the performance of the device and does not allow the reconstruction of the depth map in real time.

[0005] Patent application US20190004535A1, published 01/03/2019 and entitled "HIGH RESOLUTION 3D POINT CLOUDS GENERATION BASED ON CNN AND CRF MODELS", offers a technical solution for depth map reconstruction using a convolutional neural network that transforms the image obtained by the camera and corresponding sparse lidar point cloud into dense depth map. The proposed technical solution uses a ResNet-like architecture, which is too slow to implement this approach in real time. The architecture disclosed in the application is not intended to use pretrained image features, which reduces the possibility of training on a small dataset. In addition, this technical solution uses a high-performance graphics card and the post-processing of the proposed algorithm uses iterative processes that do not allow real-time computations.

[0006] Patent application CN108961390A, published 12/07/2018 and entitled "REAL-TIME 3D RECONSTRUCTION METHOD BASED ON DEPTH MAP" offers a technical solution that uses an RGB-D camera to extract sparse depth points that are converted into a dense depth map using texture information obtained by the RGB camera. Using an RGB-D camera in a device makes it more difficult and expensive, as an RGB-D camera requires the installation of an additional powerful LED light emitting diode, and the system needs to be calibrated for the entire device to work correctly. In addition, the proposed technical solution uses an algorithm without the use of artificial intelligence, which is adapted to the specific configuration of the device and is poorly generalized for input data from other depth sensors.

[0007] Patent Application CN106651925A, published 05/10/2017 and entitled "COLOR DEPTH IMAGE OBTAINING METHOD AND DEVICE" offers a technical solution in which data, which is a cloud of points covering the entire image, is obtained by a high quality depth sensor such as an RGB-D camera. and convert them to a dense depth map. The method mainly focuses on RGB and depth data alignment. Using an RGB-D camera in the device makes it more complex and expensive.

[0008] Current methods for reconstructing a depth map have the following disadvantages. Most methods use depth data that covers the entire image. Methods without the use of artificial intelligence tools, such as neural networks, are designed to use only a specific type of sensor and require calibration of the device performing this method. Artificial intelligence-based methods use depth data uniformly distributed over the image as input. Most artificial intelligence tools require a lot of training data. Most trained artificial intelligence tools contain a large number of parameters, such as weights, which require a lot of memory. To obtain highly accurate depth maps requires a complex architecture of the processing algorithm, which makes it impossible to obtain highly accurate depth maps in real time or essentially in real time.

[0009] The present invention has been made to overcome at least one of the above-described disadvantages and to provide at least one of the advantages described below.

SUMMARY OF THE INVENTION

[0010] An object of the present invention is to provide methods for reconstructing a depth map and an electronic computing device for their implementation, which provide at least one of the following advantages:

- obtaining a high quality depth map from sparse depth data that does not cover the entire image, in real time or essentially in real time with very low latency,

- simplification of the architecture of the artificial intelligence tools used,

- simplification of the device design by using a single-beam or double-beam depth sensor,

- obtaining high quality depth map on devices with low computing power in real time or essentially in real time with very low latency.

[0011] The foregoing advantages provide depth map reconstruction methods on low computing power devices to produce high quality depth maps from sparse depth data that does not cover the entire image, in real time or substantially in real time with very low latency.

[0012] One aspect of the present invention provides a method for reconstructing a depth map, said method comprising: acquiring (S101) an image and a sparse depth map containing depth data that are depths in no more than two planes perpendicular to the image plane, on the resulting image; calculating (S102) an image feature tensor by a trained image encoder comprising at least one convolutional layer, at least one activation layer and at least two data spatial resolution reduction layers by extracting image features from the obtained image; calculating (S103) a trained depth encoder containing at least one convolutional layer, at least one activation layer and at least two layers of data spatial resolution reduction, a depth feature tensor by extracting depth features from the obtained sparse depth map; calculating (S104) the concatenated feature tensor by concatenating the image feature tensor and the depth feature tensor, and converting the concatenated features tensor using a predetermined parameter specifying the size of the transformed concatenated feature tensor (S104); predict (S105) a gradient depth map by the trained decoder using the transformed concatenated feature tensor and the tensor obtained on each of at least two downsampling layers of the image encoder data except for the first downscaling layer of the image encoder data, the decoder comprising at least one a block for increasing spatial data resolution, containing layers in the following order: the first convolutional layer with a 1x1 kernel, the first activation layer, a layer for increasing the spatial data resolution, a convolutional layer of channel-by-channel 2D convolution, a second convolutional layer with a 1x1 kernel and a second activation layer, and a final enhancement layer spatial data resolution after the last block for increasing the spatial data resolution, and the number of layers for increasing the spatial resolution of the decoder data corresponds to the number of layers for reducing the spatial resolution of the data of the iso image, and the tensor obtained in the corresponding spatial resolution downscaling layer of the image encoder data starting from the last spatial resolution layer of the image encoder data is used in the corresponding spatial resolution upscaling unit of the decoder data starting from the first spatial resolution upscaling unit of the decoder data from at least one upscaling unit spatial resolution of the decoder data, and moreover, before entering into the first spatial resolution increasing unit of the decoder data from at least one spatial resolution increasing unit of the decoder data, the input tensor is obtained by concatenating the transformed tensor of concatenated features with the tensor obtained in the last spatial resolution reduction layer of the image encoder data , and before entering into each subsequent block for increasing spatial resolution of decoder data from at least one block for increasing spatial resolution is given The input tensor obtained in the previous decoder data upscaling block from at least one decoder data upscaling block with the tensor obtained in the corresponding downscaling layer of the image encoder data is obtained from the decoder data; and predicting (S106) by the iterative spatial distribution unit a dense depth map using a gradient depth map, a sparse depth map and a tensor obtained in the last upscaling block of decoder data from at least one upscaling block of decoder data, the prediction operation being dense depth maps are performed iteratively with a given number of iterations.

[0013] In one additional aspect, the image encoder is one of the standard pretrained classification networks MobileNet v1, MobileNet v2, InceptionNet, ResNet, R-CNN.

[0014] In another additional aspect, the predetermined parameter specifying the size of the transformed concatenated feature tensor is a multiple of 2 and preferably selected in the range 256-1024.

[0015] In yet another further aspect, the decoder further comprises at least one convolution block in front of each of the at least one convolutional block of data, wherein each convolution block of the at least one convolution block comprises layers in the following order: first convolutional layer with a 1x1 kernel, the first activation layer, a convolutional layer of channel-by-channel 2D convolution, a second convolutional layer with a 1x1 kernel, and a second activation layer, wherein step (S105) further comprises the steps of: dividing the input tensor into a set of n + 1 subtensors, where n is the number of convolution blocks; convolution of one subtenzor with a first convolution block of at least one convolution block; before each subsequent convolution block from at least one convolution block, the tensor obtained in the previous convolution block from at least one convolution block is divided into a set of k + 1 subtensions, where k is the number of remaining convolution blocks from at least one convolution block , and obtain a concatenated tensor by concatenating different nk subtenzors taken one from each of the nk sets of subtenzers obtained by splitting the tensors before each of the nk previous convolution blocks, to which no convolution was applied in the convolution blocks, with the subtensor obtained by dividing the tensor before each subsequent a convolution block to which convolution has not been applied in convolution blocks from at least one convolution block ; in each subsequent convolution block from at least one convolution block, convolution of the concatenated tensor obtained before this convolution block is performed; obtaining a concatenated tensor by concatenating the remaining subtenzers to which no convolution has been applied in convolution blocks from at least one convolution block; and processing by the block for increasing spatial resolution of the concatenated tensor data obtained by concatenating the remaining subtenzers to which convolution has not been applied in convolution blocks from at least one convolution block.

[0016] In yet another further aspect, each of the activation layers in the image encoder, the depth encoder, and the decoder is one of the layers ReLU, leaky ReLU, ReLU6, ELU.

[0017] Another aspect of the present invention provides a method for reconstructing a depth map, the method comprising: obtaining (S201) a sparse depth map containing depth data in an image; calculating (S203) by a trained depth encoder comprising at least one convolutional layer, at least one activation layer and at least two data spatial resolution reduction layers, a depth feature tensor by extracting depth features from the obtained sparse depth map; converting (S204) the conversion unit depth feature tensor using a predetermined parameter specifying the size of the transformed depth feature tensor; predicting (S205) a gradient depth map by a trained decoder using the transformed depth feature tensor as an input tensor, the decoder comprising at least one upscaling unit comprising layers in the following order: first convolutional layer with 1x1 core, first activation layer, a layer for enhancing the spatial data resolution, a convolutional layer of channel-by-channel two-dimensional convolution, a second convolutional layer with a 1x1 kernel and a second activation layer, and a final layer for enhancing the spatial data resolution after the last data upscaling block; and predicting (S206) by the iterative spatial distribution unit a dense depth map using a gradient depth map, a sparse depth map, and a tensor obtained in the last upscaling block of decoder data from at least one upscaling block of decoder data, the prediction operation being dense depth maps are performed iteratively with a given number of iterations.

[0018] In one additional aspect, the predetermined parameter specifying the size of the transformed depth cue tensor is a multiple of 2 and is preferably selected in the range 256-1024.

[0019] In another further aspect, the decoder further comprises at least one convolution block in front of each of the at least one convolution block, wherein each convolution block of the at least one convolution block comprises layers in the following order: a first convolutional layer with with a 1x1 kernel, a first activation layer, a convolutional layer of channel-by-channel 2D convolution, a second convolutional layer with a 1x1 kernel and a second activation layer, wherein step (S205) further comprises the steps of: dividing the input tensor into a set of n + 1 subtensors, where n is the number of convolution blocks; convolution of one subtenzor with a first convolution block of at least one convolution block; before each subsequent convolution block from at least one convolution block, the tensor obtained in the previous convolution block from at least one convolution block is divided into a set of k + 1 subtensions, where k is the number of remaining convolution blocks from at least one convolution block , and obtain a concatenated tensor by concatenating different nk subtenzors taken one from each of the nk sets of subtenzers obtained by splitting the tensors before each of the nk previous convolution blocks, to which no convolution was applied in the convolution blocks, with the subtensor obtained by dividing the tensor before each subsequent a convolution block to which convolution has not been applied in convolution blocks from at least one convolution block ; in each subsequent convolution block from at least one convolution block, convolution of the concatenated tensor obtained before this convolution block is performed; obtaining a concatenated tensor by concatenating the remaining subtenzers to which no convolution has been applied in convolution blocks from at least one convolution block; and processing by the block for increasing spatial resolution of the concatenated tensor data obtained by concatenating the remaining subtenzers to which convolution has not been applied in convolution blocks from at least one convolution block.

[0020] In yet another additional aspect, each of the activation layers in the depth encoder and decoder is one of the layers ReLU, leaky ReLU, ReLU6, ELU.

[0021] Another aspect provides a method for reconstructing a depth map, the method comprising: acquiring (S301) an image; calculating (S302) an image feature tensor by a trained image encoder comprising at least one convolutional layer, at least one activation layer and at least two data spatial resolution reduction layers by extracting image features from the obtained image; converting (S304) the image feature tensor using a predetermined parameter specifying the size of the transformed image feature tensor (S304); predict (S305) a gradient depth map by the trained decoder using the transformed image feature tensor and the tensor obtained on each of at least two downsampling layers of the image encoder data in addition to the first downscaling layer of the image encoder data, the decoder comprising at least one a block for increasing spatial data resolution, containing layers in the following order: the first convolutional layer with a 1x1 kernel, the first activation layer, a layer for increasing the spatial data resolution, a convolutional layer of channel-by-channel 2D convolution, a second convolutional layer with a 1x1 kernel and a second activation layer, and a final enhancement layer spatial data resolution after the last block for increasing the spatial data resolution, and the number of layers for increasing the spatial resolution of the data of the decoder corresponds to the number of layers for reducing the spatial resolution of the data of the encoder. and the tensor obtained in the corresponding spatial resolution downscaling layer of the image encoder data starting from the last spatial resolution layer of the image encoder data is used in the corresponding spatial resolution upscaling unit of the decoder data starting from the first spatial resolution upscaling unit of the decoder data from at least one upscaling unit spatial resolution of the decoder data, and moreover, before inputting the decoder data into the first spatial resolution increasing unit from at least one decoder data spatial resolution increasing unit, the input tensor is obtained by concatenating the transformed image feature tensor with the tensor obtained in the last spatial resolution reduction layer of the image encoder data , and before entering into each subsequent block for increasing the spatial resolution of the decoder data from at least one block for increasing the spatial resolution of the decoder data the input tensor is obtained by concatenating the tensor obtained in the previous decoder data upscaling block from at least one decoder data upscaling block with the tensor obtained in the corresponding downscaling layer of the image encoder data; and predicting (S306) a dense depth map by the iterative spatial distribution unit using the gradient depth map and the tensor obtained in the last upscaling block of the decoder data from at least one upscaling block of the decoder data, wherein the dense depth map prediction operation is performed iteratively with a given number of iterations.

[0022] In one additional aspect, the image encoder is one of the standard pretrained classification networks MobileNet v1, MobileNet v2, InceptionNet, ResNet, R-CNN.

[0023] In another additional aspect, the predetermined parameter specifying the size of the transformed image feature tensor is a multiple of 2 and is preferably selected in the range 256-1024.

[0024] In yet another further aspect, the decoder further comprises at least one convolution block in front of each of the at least one convolutional block of data, wherein each convolution block of the at least one convolution block comprises layers in the following order: first convolutional layer with a 1x1 kernel, the first activation layer, a convolutional layer of channel-by-channel two-dimensional convolution, a second convolutional layer with a 1x1 kernel and a second activation layer, wherein step (S305) further comprises the steps of: dividing the input tensor into a set of n + 1 subtensors, where n is the number of convolution blocks; convolution of one subtenzor with a first convolution block of at least one convolution block; before each subsequent convolution block from at least one convolution block, the tensor obtained in the previous convolution block from at least one convolution block is divided into a set of k + 1 subtensions, where k is the number of remaining convolution blocks from at least one convolution block , and obtain a concatenated tensor by concatenating different nk subtenzors taken one from each of the nk sets of subtenzers obtained by splitting the tensors before each of the nk previous convolution blocks, to which no convolution was applied in the convolution blocks, with the subtensor obtained by dividing the tensor before each subsequent a convolution block to which convolution has not been applied in convolution blocks from at least one convolution block ; in each subsequent convolution block from at least one convolution block, convolution of the concatenated tensor obtained before this convolution block is performed; obtaining a concatenated tensor by concatenating the remaining subtenzers to which no convolution has been applied in convolution blocks from at least one convolution block; and processing by the block for increasing spatial resolution of the concatenated tensor data obtained by concatenating the remaining subtenzers to which convolution has not been applied in convolution blocks from at least one convolution block.

[0025] In yet another further aspect, each of the activation layers in the image encoder and decoder is one of ReLU, leaky ReLU, ReLU6, ELU.

[0026] Another aspect provides a method for reconstructing a depth map, said method comprising: acquiring (S401) an image and a sparse depth map containing depth data that are depths of objects specified by the user in the acquired image; calculating (S402) an image feature tensor by a trained image encoder comprising at least one convolutional layer, at least one activation layer and at least two data spatial resolution reduction layers by extracting image features from the obtained image; calculating (S403) a trained depth encoder containing at least one convolutional layer, at least one activation layer and at least two data spatial resolution reduction layers, a depth feature tensor by extracting depth features from the obtained sparse depth map; calculating (S404) the concatenated feature tensor by concatenating the image feature tensor and the depth feature tensor, and converting the concatenated feature tensor using a predetermined parameter specifying the size of the transformed concatenated feature tensor (S404); predicting (S405) a gradient depth map by the trained decoder using the transformed concatenated feature tensor and the tensor obtained on each of the at least two downsampling layers of the image encoder data except for the first downscaling layer of the image encoder data, the decoder comprising at least one a block for increasing spatial data resolution, containing layers in the following order: the first convolutional layer with a 1x1 kernel, the first activation layer, a layer for increasing the spatial data resolution, a convolutional layer of channel-by-channel 2D convolution, a second convolutional layer with a 1x1 kernel and a second activation layer, and a final enhancement layer spatial data resolution after the last block for increasing the spatial data resolution, and the number of layers for increasing the spatial resolution of the decoder data corresponds to the number of layers for reducing the spatial resolution of the data of the iso image, and the tensor obtained in the corresponding spatial resolution downscaling layer of the image encoder data starting from the last spatial resolution layer of the image encoder data is used in the corresponding spatial resolution upscaling unit of the decoder data starting from the first spatial resolution upscaling unit of the decoder data from at least one upscaling unit spatial resolution of the decoder data, and moreover, before entering into the first spatial resolution increasing unit of the decoder data from at least one spatial resolution increasing unit of the decoder data, the input tensor is obtained by concatenating the transformed tensor of concatenated features with the tensor obtained in the last spatial resolution reduction layer of the image encoder data , and before entering into each subsequent block for increasing spatial resolution of decoder data from at least one block for increasing spatial resolution is given The input tensor obtained in the previous decoder data upscaling block from at least one decoder data upscaling block with the tensor obtained in the corresponding downscaling layer of the image encoder data is obtained from the decoder data; and predicting (S406) by the iterative spatial distribution unit a dense depth map using a gradient depth map, a sparse depth map, and a tensor obtained in the last upscaling block of decoder data from at least one upscaling block of the decoder data, the prediction operation being dense depth maps are performed iteratively with a given number of iterations.

[0027] In one additional aspect, the image encoder is one of the standard pre-trained classification networks MobileNet v1, MobileNet v2, InceptionNet, ResNet, R-CNN.

[0028] In another additional aspect, the predetermined parameter specifying the size of the transformed concatenated feature tensor is a multiple of 2 and is preferably selected in the range 256-1024.

[0029] In yet another further aspect, the decoder further comprises at least one convolution block in front of each of the at least one convolutional block of data, wherein each convolution block of the at least one convolution block comprises layers in the following order: first convolutional layer with a 1x1 kernel, the first activation layer, a convolutional layer of channel-by-channel 2D convolution, a second convolutional layer with a 1x1 kernel and a second activation layer, wherein step (S405) further comprises the steps of: dividing the input tensor into a set of n + 1 subtensors, where n is the number of convolution blocks; convolution of one subtenzor with a first convolution block of at least one convolution block; before each subsequent convolution block from at least one convolution block, the tensor obtained in the previous convolution block from at least one convolution block is divided into a set of k + 1 subtensions, where k is the number of remaining convolution blocks from at least one convolution block , and obtain a concatenated tensor by concatenating different nk subtenzors taken one from each of the nk sets of subtenzers obtained by splitting the tensors before each of the nk previous convolution blocks, to which no convolution was applied in the convolution blocks, with the subtensor obtained by dividing the tensor before each subsequent a convolution block to which convolution has not been applied in convolution blocks from at least one convolution block ; in each subsequent convolution block from at least one convolution block, convolution of the concatenated tensor obtained before this convolution block is performed; obtaining a concatenated tensor by concatenating the remaining subtenzers to which no convolution has been applied in convolution blocks from at least one convolution block; and processing by the block for increasing spatial resolution of the concatenated tensor data obtained by concatenating the remaining subtenzers to which convolution has not been applied in convolution blocks from at least one convolution block.

[0030] In yet another further aspect, each of the activation layers in the image encoder, the depth encoder, and the decoder is one of the layers ReLU, leaky ReLU, ReLU6, ELU.

[0031] Another aspect provides an electronic computing device, comprising: at least one processor; and a memory storing numerical parameters of a trained image encoder, a trained depth encoder, a transform unit, a trained decoder, an iterative spatial distribution unit, and instructions that, when executed by at least one processor, cause at least one processor to perform any of the above methods depth map reconstruction.

[0032] In a further aspect, the electronic computing device further comprises at least one of a camera, a depth sensor, and a user input device.

BRIEF DESCRIPTION OF DRAWINGS

[0033] The above and other aspects, features and advantages of the present invention will be better understood from the following detailed description, taken in conjunction with the accompanying drawings, in which:

[0034] FIG. 1 is a block diagram of an artificial intelligence means for implementing a first embodiment of a method 100 for reconstructing a depth map.

[0035] FIG. 2 is a block diagram of an artificial intelligence tool for implementing a second embodiment of a method 200 for reconstructing a depth map.

[0036] FIG. 3 is a block diagram of an artificial intelligence tool for implementing a third embodiment of a method 300 for reconstructing a depth map.

[0037] FIG. 4 is a block diagram of an artificial intelligence tool for implementing a fourth embodiment of a method 400 for reconstructing a depth map.

[0038] FIG. 5 is a schematic diagram of an interleaving operation using sub-tensor division in a decoder.

[0039] FIG. 6 is a flowchart of a first embodiment of a method 100 for reconstructing a depth map.

[0040] FIG. 7 is a flowchart of a second embodiment of a method 200 for reconstructing a depth map.

[0041] FIG. 8 is a flowchart of a third embodiment of a method 300 for reconstructing a depth map.

[0042] FIG. 9 is a flowchart of a fourth embodiment of a method 400 for reconstructing a depth map.

[0043] FIG. 10 is a block diagram of an electronic computing device 500 for performing any of the disclosed embodiments of a method for reconstructing a depth map.

[0044] In the following description, unless otherwise indicated, like reference numbers are used for like elements when shown in different drawings, and parallel descriptions thereof are omitted.

DETAILED DESCRIPTION OF EMBODIMENTS OF THE PRESENT INVENTION

[0045] The following description, with reference to the accompanying drawings, is provided to facilitate a thorough understanding of the various embodiments of the present invention as defined by the claims and their equivalents. The description includes various specific details to facilitate such understanding, but these details are to be considered only indicative. Accordingly, those skilled in the art will find that various changes and modifications to the various embodiments described herein can be devised without departing from the scope of the present invention. In addition, descriptions of well-known functions and structures may be omitted for clarity and conciseness.

[0046] The terms and wording used in the following description and claims are not limited to bibliographic meanings, but are simply used by the inventor of the present invention to provide a clear and consistent understanding of the present invention. Accordingly, those skilled in the art will appreciate that the following description of various embodiments of the present invention is provided for illustration only.

[0047] It should be understood that the singular includes plurality, unless the context clearly indicates otherwise.

[0048] Additionally, it should be understood that the terms "comprises", "comprising", "includes" and / or "including", when used in this application, means the presence of the recited features, meanings, operations, elements and / or components, but do not exclude the presence or addition of one or more other characteristics, values, operations, elements, components and / or their groups.

[0049] The proposed methods may be performed on an electronic computing device 500, which may be, for example, a mobile phone, a smartphone, a tablet computer, a robot vacuum cleaner, an autonomous vehicle control device, and so on. Electronic computing device 500 includes at least one or more processors 501 and memory 502. Electronic computing device 500 includes one or more modules for performing method steps. At least one of the plurality of modules may be implemented by an artificial intelligence (AI) model. AI related function can be performed using nonvolatile memory, volatile memory and processor.

[0050] One or more processors may include a general purpose processor such as a central processing unit (CPU), an application processor (AP), or the like, a graphics processor such as a graphics processing unit (GPU), an image processing unit (VPU) ) and / or a specialized AI processor such as a neural processor (NPU).

[0051] One or more processors control the processing of input data in accordance with a predetermined rule of operation or AI model stored in non-volatile memory and non-volatile memory. A given rule of operation or AI model is provided through training.

[0052] In this case, provision by training means that by applying a learning algorithm to a set of training data, a predetermined operation rule or AI model of the required characteristic is created. The training can be performed in the device itself, which contains the AI in accordance with the embodiment, and / or can be implemented via a separate server / system.

[0053] The AI model can be composed of multiple layers of a neural network. Each layer has multiple weights and performs a layer operation using the calculation of the previous layer and multiple weights. Examples of neural networks include, but are not limited to, Convolutional Neural Network (CNN), Deep Neural Network (DNN), Recurrent Neural Network (RNN), Restricted Boltzmann Machine (RBM), Deep Trust Network (DBN), Bidirectional Recurrent Deep neural network (BRDNN), generative adversarial networks (GAN) and deep Q-networks.

[0054] A training algorithm is a method for training a predetermined target device using a plurality of training data to cause, enable, or control a target device to perform a determination or prediction. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, partially supervised learning, or reinforcement learning.

[0055] Hereinafter, various embodiments of the present invention are described in more detail with reference to the accompanying drawings.

[0056] The present invention is for reconstructing a depth map from depth and / or image data. The present invention can be implemented on a low computing power electronic computing device such as a mobile phone, smartphone, tablet computer, robot vacuum cleaner, autonomous vehicle control device, etc. The present invention provides a high quality dense depth map from sparse depth data that does not cover the entire image, in real time or substantially in real time with very low latency. For example, a dense high quality depth map with a size of 512 × 256 pixels can be obtained on a Samsung smartphone with an Exynos 8895 processor in 10 milliseconds.

[0057] FIG. 1 shows a block diagram of an artificial intelligence tool for implementing a method 100 for reconstructing a depth map. The artificial intelligence tool contains an image encoder, a depth encoder, a transform module, a decoder, and an iterative spatial distribution module. The artificial intelligence is contained in an electronic computing device 500 that is configured to implement a method 100 for reconstructing a depth map. The artificial intelligence tool is pre-trained by inputting training data into it containing different images and sparse depth maps of these images. Such training is known in the art and therefore will not be described in detail in this disclosure.

[0058] The image encoder is designed to extract image features from an image entered into it and outputs an image feature tensor at the output. The image can be color or black and white. The image can be captured by a camera additionally contained in the electronic computing device 500, or retrieved from any available source such as the Internet, external storage, or the like. The image encoder contains at least one convolutional layer, at least one activation layer and at least two data reduction layers and can be any standard pretrained classification network, for example, MobileNet v1, MobileNet v2, InceptionNet, ResNet, R-CNN ...

[0059] The depth encoder is designed to extract depth features from a sparse depth map inserted into it and outputs a tensor of depth features. The sparse depth map can be obtained from a single beam or dual beam depth sensor additionally contained in electronic computing device 500, or retrieved from any available source such as the Internet, external storage, and the like. A sparse depth map contains depth data that is depths in no more than two planes perpendicular to the image plane in the image entered into the depth encoder.

[0060] The transform module is designed to obtain a transformed concatenated feature tensor. In the conversion module, the transformed concatenated feature tensor is obtained by concatenating the image feature tensor obtained by the image encoder and the depth feature tensor obtained by the depth encoder, and transforming the concatenated tensor using a predetermined parameter specifying the size of the transformed concatenated feature tensor. The parameter specifying the size of the transformed concatenated feature tensor is a multiple of 2. The value of the parameter specifying the size of the transformed concatenated feature tensor is selected based on the processing speed and quality requirements of the dense depth map obtained at the output of the artificial intelligence tool for reconstructing the depth map. The higher the value of the above parameter, the higher the quality of the obtained dense depth map and the lower the processing speed of the artificial intelligence, and the lower the value of the above parameter, the lower the quality of the obtained dense depth map and the higher the processing speed of the artificial intelligence. Preferably, the value of the parameter specifying the size of the transformed tensor of concatenated features is selected in the range 256-1024.

[0061] The decoder is for predicting a gradient depth map using the transformed concatenated feature tensor and the tensor obtained on each of at least two downsampling layers of the image encoder data except for the first downscaling layer of the image encoder data.

[0062] The decoder comprises at least one data upscaling block and a final data upscaling layer after the last data upscaling block. Each upscaling block contains layers in the following order: the first convolutional layer with a 1x1 kernel (pointwise convolution), the first activation layer, the spatial data upscaling layer, a depthwise convolution layer, the second convolutional layer with a 1x1 kernel, and second layer of activation. The number of downscaling layers of the decoder data corresponds to the number of downscaling layers of the image encoder data. Such a decoder structure makes it possible to increase the accuracy of data processing and reduce the processing time.

[0063] The tensor obtained in the corresponding downscaling layer of the image encoder data starting from the last spatial resolution layer of the image encoder data is used in the corresponding upscaling block of the decoder data starting from the first upscaling block of the decoder data from at least one upscaling block spatial resolution of the decoder data, and moreover, before entering into the first spatial resolution increasing unit of the decoder data from at least one spatial resolution increasing unit of the decoder data, the input tensor is obtained by concatenating the transformed tensor of concatenated features with the tensor obtained in the last spatial resolution reduction layer of the image encoder data , and before entering into each subsequent block for increasing the spatial resolution of decoder data from at least one block for increasing the spatial resolution of data d In the decoder, the input tensor is obtained by concatenating the tensor obtained in the previous decoder data upscaling block from at least one decoder data upscaling block with the tensor obtained in the corresponding downscaling layer of the image encoder data.

[0064] In an example in which the decoder comprises two data downscaling blocks and the image encoder comprises three data downscaling layers, the transformed concatenated feature tensor obtained in the transform unit is concatenated with the tensor obtained in the third downscaling layer of the image encoder data , and this concatenated tensor is processed by the first data upscaling unit. The tensor obtained in the first data upscaling unit is concatenated with the tensor obtained in the second data downscaling layer of the image encoder, and this concatenated tensor is processed by the second data upscaling unit. Then the tensor obtained in the second block of data spatial resolution is processed with the final data spatial enhancement layer to obtain a gradient depth map. This example is only intended to illustrate the operation of the decoder and is not intended to limit the invention, since the decoder may contain any number of spatial data upscaling units.

[0065] The decoder may further comprise at least one convolution block in front of each of the at least one data upscaling block. Each convolution block applies interleaving operations using tensor division into subtenzers and contains layers in the following order: first convolutional layer with 1x1 kernel, first activation layer, channel-by-channel 2D convolutional layer, second convolutional layer with 1x1 kernel, and second activation layer. Such a decoder structure insignificantly increases the complexity of the decoder, however, it makes it possible to further increase the accuracy of data processing and save a short processing time.

[0066] In the convolution blocks, the following interleaving operations are performed using the division of the tensor into subtensors: divide the input tensor into a set of n + 1 subtensions, where n is the number of convolution blocks; convolution of one subtenzor by the first block; before each subsequent convolution block, the tensor obtained in the previous convolution block is divided into a set of k + 1 subtensors, where k is the number of remaining convolution blocks, and a concatenated tensor is obtained by concatenating different nk subtensors, taken one from each of the nk sets of subtensors, obtained by splitting tensors before each of the nk previous convolution blocks, to which convolution was not applied in convolution blocks, with a subtensor obtained by dividing the tensor before each subsequent convolution block, to which convolution was not applied in convolution blocks; in each subsequent convolution block, convolution of the concatenated tensor obtained before this convolution block is performed.

[0067] The concatenated tensor is then obtained by concatenating the remaining subtenzers to which no convolution has been applied in the convolution blocks; and processing by the spatial resolution enhancing unit of the concatenated tensor data obtained by concatenating the remaining subtenzors to which convolution has not been applied in the convolution units.

[0068] FIG. 5 shows an example in which the decoder further comprises two convolution blocks in front of the spatial data upscaling block. This example, shown in FIG. 5 is only intended to explain the operation of the decoder and is not intended to limit the invention, since the decoder may comprise any number of convolution blocks before each upscaling block.

[0069] In the example shown in FIG. 5, the input tensor is divided into three subtensors 1, 2, 3. Subtensor 3 is processed by the first convolution block. The resulting tensor is divided into two subtensors 3.1 and 3.2. Subtensor 1 is concatenated with subtensor 3.1 and processed with the second convolution block. The resulting tensor 3.1.1 is concatenated with subtensor 3.2 and subtensor 2 and processed by a block for increasing the spatial data resolution.

[0070] Each of the activation layers in the artificial intelligence is one of the layers ReLU, leaky ReLU, ReLU6, ELU.

[0071] An iterative spatial distribution unit predicts a dense depth map using a gradient depth map, a sparse depth map, and a tensor obtained in the last upscaling block of the decoder data. The operation of predicting a dense depth map is performed iteratively with a given number of iterations. The number of iterations is set by the user and can be anything. Preferably, the number of iterations is 24.

[0072] The iterative spatial distribution unit can be any spatial distribution network (SPN). However, in the present invention, the known SPN is supplemented by the fact that during iterations of updating the dense depth map, the depth values in the dense depth map are replaced with known depth values from the sparse depth map input to the depth encoder. Replacing the depth values in the dense depth map with the known depth values from the sparse depth map provides a more accurate representation of the depth values in the dense depth map from the iterative spatial distribution module.

[0073] FIG. 2 shows a block diagram of an artificial intelligence tool for implementing a method 200 for reconstructing a depth map. The artificial intelligence tool contains a depth encoder, a transform module, a decoder, and an iterative spatial distribution module. The artificial intelligence is contained in an electronic computing device 500 that is configured to implement a method 200 for reconstructing a depth map. The artificial intelligence tool for implementing the method 200 is pre-trained in the same way as the artificial intelligence tool for implementing the method 100.

[0074] The structures of the depth encoder, transform module, decoder, and iterative artificial intelligence spatial distribution module for implementing method 200 correspond to the structures of the depth encoder, transform module, decoder, and iterative artificial intelligence spatial distribution module for implementing method 100.

[0075] The operations, inputs and outputs of the AI depth encoder for implementing the method 200 are the same as the operations, inputs and outputs of the AI depth encoder for implementing the method 100. Therefore, their detailed description will be omitted.

[0076] The transform module is designed to obtain the transformed depth feature tensor from the depth feature tensor obtained in the depth encoder using a predetermined parameter specifying the size of the transformed depth feature tensor. The conversion operation in the AI conversion unit for implementing the method 200 is the same as the conversion operation in the AI conversion unit for the implementation of the method 100. Therefore, a detailed description thereof will be omitted.

[0077] The decoder is for predicting a gradient depth map using the transformed depth cue tensor as an input tensor. In the artificial intelligence decoder for implementing the method 200, there are no concatenation operations with the tensor obtained on each of the at least two downsampling layers of the image encoder data. The remaining operations of the artificial intelligence decoder for implementing the method 200 are the same as those of the artificial intelligence decoder for implementing the method 100, so their detailed description will be omitted.

[0078] The operations, inputs and outputs of the iterative AI spatial distribution module for implementing the method 200 are the same as the operations, inputs and outputs of the iterative AI spatial distribution module for implementing the method 100. Therefore, their detailed description will be omitted.

[0079] FIG. 3 shows a block diagram of an artificial intelligence tool for implementing a method 300 for reconstructing a depth map. The artificial intelligence tool contains an image encoder, a transform module, a decoder, and an iterative spatial distribution module. The artificial intelligence is contained in an electronic computing device 500 that is configured to implement a method 300 for reconstructing a depth map. The artificial intelligence tool for implementing the method 300 is pre-trained in the same way as the artificial intelligence tool for implementing the method 100.

[0080] The structures of the image encoder, transform module, decoder and iterative artificial intelligence spatial distribution module for implementing the method 300 correspond to the structures of the image encoder, transform module, decoder, and iterative artificial intelligence spatial distribution module for implementing the method 100.

[0081] The operations, inputs and outputs of the artificial intelligence image encoder for implementing the method 300 are the same as the operations, inputs and outputs of the artificial intelligence image encoder for implementing the method 100. Therefore, their detailed description will be omitted.

[0082] The transform module is designed to obtain the transformed depth feature tensor from the image feature tensor obtained in the depth encoder using a predetermined parameter specifying the size of the transformed depth feature tensor. The conversion operation in the AI conversion unit for implementing the method 300 is the same as the conversion operation in the AI conversion unit for the implementation of the method 100. Therefore, a detailed description thereof will be omitted.

[0083] The decoder is for predicting a gradient depth map using the transformed image feature tensor and the tensor obtained on each of at least two downsampling layers of the image encoder data except for the first downscaling layer of the image encoder data.

[0084] The operations and outputs of the artificial intelligence decoder for implementing the method 300 are the same as the operations and outputs of the artificial intelligence decoder for implementing the method 100. Therefore, their detailed description will be omitted.

[0085] In the iterative spatial distribution module of the artificial intelligence tool for implementing the method 300, there is no operation of replacing depth values in the dense depth map with known depth values from the sparse depth map during iterations of updating the dense depth map. The rest of the operations of the iterative artificial intelligence spatial distribution module for implementing the method 300 are the same as those of the iterative artificial intelligence spatial distribution module for implementing the method 100, so their detailed description will be omitted.

[0086] FIG. 4 shows a block diagram of an artificial intelligence tool for implementing a method 400 for reconstructing a depth map. The artificial intelligence tool contains an image encoder, a depth encoder, a transform module, a decoder, and an iterative spatial distribution module. The artificial intelligence is contained in an electronic computing device 500 that is configured to implement a method 400 for reconstructing a depth map. The artificial intelligence tool for implementing the method 400 is pre-trained in the same way as the artificial intelligence tool for implementing the method 100.

[0087] Structures of an image encoder, a depth encoder, a transform module, a decoder and an iterative spatial distribution module of the artificial intelligence means for implementing the method 400 correspond to the structures of an image encoder, a depth encoder, a transform module, a decoder, and an iterative spatial distribution module of an artificial intelligence means for implementing the method 100 ...

[0088] The operations, inputs and outputs of the artificial intelligence image encoder for implementing the method 400 are the same as the operations, inputs and outputs of the artificial intelligence image encoder for implementing the method 100. Therefore, their detailed description will be omitted.

[0089] The depth encoder is designed to extract depth features from a sparse depth map inserted into it and outputs a depth feature tensor at the output. The sparse depth map contains depth data, which is the depths of objects specified by the user in the image entered into the image encoder. The sparse depth map can be generated by the user on any remote device containing a user input device and input into the depth encoder from said remote device or from any available source such as the Internet, external storage, and the like. A sparse depth map can also be generated on the electronic computing device 500 if the electronic computing device 500 further comprises a user input device.

[0090] The operations and output of the AI depth encoder for implementing the method 400 are the same as the operations and output of the AI depth encoder for the method 100. Therefore, their detailed description will be omitted.

[0091] The operations, inputs and outputs of the artificial intelligence decoder for implementing the method 400 are the same as the operations, inputs and outputs of the artificial intelligence decoder for implementing the method 100. Therefore, their detailed description will be omitted.

[0092] The operations, inputs and outputs of the AI converting unit for implementing the method 400 are the same as the operations, inputs and outputs of the AI converting unit for implementing the method 100. Therefore, their detailed description will be omitted.

[0093] The operations, inputs and outputs of the artificial intelligence decoder for implementing the method 400 are the same as the operations, inputs and outputs of the artificial intelligence decoder for implementing the method 100. Therefore, their detailed description will be omitted.

[0094] The operations, inputs, and outputs of the iterative AI spatial distribution unit for implementing the method 400 are the same as the operations, inputs and outputs of the iterative AI spatial distribution unit for implementing the method 100. Therefore, their detailed description will be omitted.

[0095] FIG. 6 is a flowchart illustrating a first embodiment of a method 100 for reconstructing a depth map. The method 100 for reconstructing the depth map is performed by the electronic computing device 500.

[0096] In step S101, an image and a sparse depth map are acquired. A sparse depth map contains depth data that is depths in no more than two planes perpendicular to the image plane in the resulting image.

[0097] In step S102, the trained image encoder calculates an image feature tensor by extracting image features from the image obtained in step S101. The trained image encoder comprises at least one convolutional layer, at least one activation layer, and at least two data downscaling layers.

[0098] In step S103, the trained depth encoder calculates a depth cue tensor by extracting depth cues from the sparse depth map obtained in step S101. The trained depth encoder comprises at least one convolutional layer, at least one activation layer, and at least two data downscaling layers.

[0099] In step S104, the transforming unit calculates the concatenated feature tensor by concatenating the image feature tensor calculated in step S102 and the depth feature tensor calculated in step S103, and converts the concatenated features tensor using a predetermined parameter specifying the size of the transformed concatenated feature tensor ...

[0100] In step S105, the trained decoder predicts a gradient depth map using the transformed concatenated feature tensor obtained in step S104 and the tensor obtained on each of at least two downsampling layers of the image encoder data except for the first downscaling layer of the encoder data Images. The decoder contains at least one block for increasing spatial data resolution, containing layers in the following order: first convolutional layer with a 1x1 kernel, first activation layer, layer for increasing spatial data resolution, convolutional layer of channel-by-channel two-dimensional convolution, second convolutional layer with a 1x1 kernel and a second layer activation, and the final data upscaling layer after the last data upscaling block. The number of downscaling layers of the decoder data corresponds to the number of downscaling layers of the image encoder data. The tensor obtained in the corresponding spatial resolution reduction layer of the image encoder data starting from the last spatial resolution layer of the image encoder data is used in the corresponding spatial resolution increasing unit of the decoder data starting from the first spatial resolution increasing unit of the decoder data from at least one data spatial resolution increasing unit decoder. Before entering the decoder data into the first downscaling unit from at least one decoder data upscaling unit, the input tensor is obtained by concatenating the transformed concatenated feature tensor with the tensor obtained in the last image coder data downscaling layer. Before entering into each subsequent block for increasing the spatial resolution of the decoder data from at least one block for increasing the spatial resolution of the decoder data, the input tensor is obtained by concatenating the tensor obtained in the previous block for increasing the spatial resolution of the decoder data from at least one block for increasing the spatial resolution of the decoder data, with the tensor obtained in the corresponding downsampling layer of the image encoder data.

[0101] In step S106, the iterative spatial distribution unit predicts a dense depth map using the gradient depth map predicted in step S105, the sparse depth map obtained in step S105, and the tensor obtained in the last upscaling block of the decoder data from at least one block for increasing the spatial resolution of the decoder data, while the dense depth map prediction operation is performed iteratively with a predetermined number of iterations.

[0102] The image encoder can be one of the standard pre-trained classification networks MobileNet v1, MobileNet v2, InceptionNet, ResNet, R-CNN.

[0103] The predetermined parameter specifying the size of the transformed concatenated feature tensor is a multiple of 2 and is preferably selected in the range 256-1024.

[0104] The decoder may further comprise at least one convolution block in front of each of the at least one data upscaling block. Each convolution block of at least one convolution block contains layers in the following order: a first convolutional layer with a 1x1 kernel, a first activation layer, a channel-by-channel 2D convolutional layer, a second convolutional layer with a 1x1 kernel, and a second activation layer. With this decoder configuration, step S105 further comprises the steps of: dividing the input tensor into a set of n + 1 subtensions, where n is the number of convolution blocks; convolution of one subtenzor with a first convolution block of at least one convolution block; before each subsequent convolution block from at least one convolution block, the tensor obtained in the previous convolution block from at least one convolution block is divided into a set of k + 1 subtensions, where k is the number of remaining convolution blocks from at least one convolution block , and obtain a concatenated tensor by concatenating different nk subtenzors taken one from each of the nk sets of subtenzers obtained by splitting the tensors before each of the nk previous convolution blocks, to which no convolution was applied in the convolution blocks, with the subtensor obtained by dividing the tensor before each subsequent a convolution block to which convolution has not been applied in convolution blocks from at least one convolution block ; in each subsequent convolution block of at least one convolution block, performing convolution of the concatenated tensor obtained before this convolution block; obtaining a concatenated tensor by concatenating the remaining subtenzers to which no convolution has been applied in the convolution blocks from at least one convolution block; and performing processing by the spatial resolution upscaling unit on the concatenated tensor data obtained by concatenating the remaining subtenzors to which no convolution has been applied in the convolution blocks from the at least one convolution block.

[0105] Each of the activation layers in the image encoder, the depth encoder and the decoder is one of the layers ReLU, leaky ReLU, ReLU6, ELU.

[0106] FIG. 7 is a flowchart illustrating a second embodiment of a method 200 for reconstructing a depth map. The method 200 for reconstructing the depth map is performed by the electronic computing device 500.

[0107] In step S201, a sparse depth map is obtained containing depth data in the image.

[0108] In step S203, the trained depth encoder calculates a depth cue tensor by extracting depth cues from the sparse depth map obtained in step S201. The trained depth encoder comprises at least one convolutional layer, at least one activation layer, and at least two data downscaling layers.

[0109] In step S204, the transforming unit converts the depth cue tensor calculated in step S203 using a predetermined parameter specifying the size of the transformed depth cue tensor.

[0110] In step S205, the trained decoder predicts the gradient depth map using the transformed depth cue tensor obtained in step S204 as an input tensor. The decoder contains at least one block for increasing spatial data resolution, containing layers in the following order: first convolutional layer with a 1x1 kernel, first activation layer, layer for increasing spatial data resolution, convolutional layer of channel-by-channel two-dimensional convolution, second convolutional layer with a 1x1 kernel and a second layer activation, and the final data upscaling layer after the last data upscaling block.

[0111] In step S206, the iterative spatial distribution unit predicts a dense depth map using the gradient depth map predicted in step S205, the sparse depth map obtained in step S201, and the tensor obtained in the last upscaling block of the decoder data from at least one block for increasing the spatial resolution of the decoder data. The operation of predicting a dense depth map is performed iteratively with a given number of iterations.

[0112] The predetermined parameter specifying the size of the transformed depth feature tensor is a multiple of 2 and is preferably selected in the range 256-1024.

[0113] The decoder may further comprise at least one convolution block in front of each of the at least one data upscaling block. Each convolution block of at least one convolution block contains layers in the following order: a first convolutional layer with a 1x1 kernel, a first activation layer, a channel-by-channel 2D convolutional layer, a second convolutional layer with a 1x1 kernel, and a second activation layer. With this decoder configuration, step S205 further comprises the steps of: dividing the input tensor into a set of n + 1 subtensions, where n is the number of convolution blocks; convolution of one subtenzor with a first convolution block of at least one convolution block; before each subsequent convolution block from at least one convolution block, the tensor obtained in the previous convolution block from at least one convolution block is divided into a set of k + 1 subtensions, where k is the number of remaining convolution blocks from at least one convolution block , and obtain a concatenated tensor by concatenating different nk subtenzors taken one from each of the nk sets of subtenzers obtained by splitting the tensors before each of the nk previous convolution blocks, to which no convolution was applied in the convolution blocks, with the subtensor obtained by dividing the tensor before each subsequent a convolution block to which convolution has not been applied in convolution blocks from at least one convolution block ; in each subsequent convolution block of at least one convolution block, performing convolution of the concatenated tensor obtained before this convolution block; obtaining a concatenated tensor by concatenating the remaining subtenzers to which no convolution has been applied in the convolution blocks from at least one convolution block; and performing processing by the spatial resolution upscaling unit on the concatenated tensor data obtained by concatenating the remaining subtenzors to which no convolution has been applied in the convolution blocks from the at least one convolution block.

[0114] Each of the activation layers in the depth encoder and decoder is one of the layers ReLU, leaky ReLU, ReLU6, ELU.

[0115] FIG. 8 is a flowchart illustrating a third embodiment of a method 300 for reconstructing a depth map. The method 300 for reconstructing the depth map is performed by the electronic computing device 500.

[0116] In step S301, an image is acquired.

[0117] In step S302, the trained image encoder calculates an image feature tensor by extracting image features from the image obtained in step S301. The trained image encoder comprises at least one convolutional layer, at least one activation layer, and at least two data downscaling layers.

[0118] In step S304, the transforming unit converts the image feature tensor calculated in step S302 using a predetermined parameter specifying the size of the transformed image feature tensor.

[0119] In step S305, the trained decoder predicts a gradient depth map using the transformed image feature tensor obtained in step S304 and the tensor obtained on each of at least two downsampling layers of the image encoder data except for the first downscaling layer of the encoder data Images. The decoder contains at least one block for increasing spatial data resolution, containing layers in the following order: first convolutional layer with a 1x1 kernel, first activation layer, layer for increasing spatial data resolution, convolutional layer of channel-by-channel two-dimensional convolution, second convolutional layer with a 1x1 kernel and a second layer activation, and the final data upscaling layer after the last data upscaling block. The number of downscaling layers of the decoder data corresponds to the number of downscaling layers of the image encoder data. The tensor obtained in the corresponding spatial resolution reduction layer of the image encoder data starting from the last spatial resolution layer of the image encoder data is used in the corresponding spatial resolution increasing unit of the decoder data starting from the first spatial resolution increasing unit of the decoder data from at least one data spatial resolution increasing unit decoder. Before entering the decoder data into the first downscaling unit from at least one decoder data upscaling unit, the input tensor is obtained by concatenating the transformed image feature tensor with the tensor obtained in the last image coder data downscaling layer. Before entering into each subsequent block for increasing the spatial resolution of the decoder data from at least one block for increasing the spatial resolution of the decoder data, the input tensor is obtained by concatenating the tensor obtained in the previous block for increasing the spatial resolution of the decoder data from at least one block for increasing the spatial resolution of the decoder data, with the tensor obtained in the corresponding downsampling layer of the image encoder data.

[0120] In step S306, the iterative spatial distribution unit predicts a dense depth map using the gradient depth map predicted in step S305 and the tensor obtained in the last upscaling block of the decoder data from at least one upscaling block of the decoder data. The operation of predicting a dense depth map is performed iteratively with a given number of iterations.

[0121] The image encoder can be one of the standard pre-trained classification networks MobileNet v1, MobileNet v2, InceptionNet, ResNet, R-CNN.

[0122] The predetermined parameter specifying the size of the transformed image feature tensor is a multiple of 2 and is preferably selected in the range of 256-1024.

[0123] The decoder may further comprise at least one convolution block in front of each of the at least one data upscaling block. Each convolution block of at least one convolution block contains layers in the following order: a first convolutional layer with a 1x1 kernel, a first activation layer, a channel-by-channel 2D convolutional layer, a second convolutional layer with a 1x1 kernel, and a second activation layer. With this decoder configuration, step S305 further comprises the steps of: dividing the input tensor into a set of n + 1 sub-tensors, where n is the number of convolution blocks; convolution of one subtenzor with a first convolution block of at least one convolution block; before each subsequent convolution block from at least one convolution block, the tensor obtained in the previous convolution block from at least one convolution block is divided into a set of k + 1 subtenzers, where k is the number of remaining convolution blocks from at least one convolution block , and obtain a concatenated tensor by concatenating different nk subtenzors taken one from each of the nk sets of subtenzers obtained by splitting the tensors before each of the nk previous convolution blocks, to which no convolution was applied in the convolution blocks, with the subtensor obtained by dividing the tensor before each subsequent a convolution block to which convolution has not been applied in convolution blocks from at least one convolution block ; in each subsequent convolution block of at least one convolution block, performing convolution of the concatenated tensor obtained before this convolution block; obtaining a concatenated tensor by concatenating the remaining subtenzers to which no convolution has been applied in the convolution blocks from at least one convolution block; and performing processing by the spatial resolution upscaling unit on the concatenated tensor data obtained by concatenating the remaining subtenzors to which no convolution has been applied in the convolution blocks from the at least one convolution block.

[0124] Each of the activation layers in the image encoder and decoder is one of the layers ReLU, leaky ReLU, ReLU6, ELU.

[0125] FIG. 9 is a flowchart illustrating a fourth embodiment of a method 400 for reconstructing a depth map. A method 400 for reconstructing a depth map is performed by an electronic computing device 500.

[0126] In step S401, an image and a sparse depth map are acquired. The sparse depth map contains depth data, which is the depth of objects specified by the user in the captured image.

[0127] In step S402, the trained image encoder calculates an image feature tensor by extracting image features from the image obtained in step S401. The trained image encoder comprises at least one convolutional layer, at least one activation layer, and at least two data downscaling layers.

[0128] In step S403, the trained depth encoder calculates a depth cue tensor by extracting depth cues from the sparse depth map obtained in step S401. The trained depth encoder comprises at least one convolutional layer, at least one activation layer, and at least two data downscaling layers.

[0129] In step S404, the transform module calculates the concatenated feature tensor by concatenating the image feature tensor calculated in step S402 and the depth feature tensor calculated in step S403, and converts the concatenated features tensor using a predetermined parameter specifying the size of the transformed concatenated feature tensor ...

[0130] In step S405, the trained decoder predicts a gradient depth map using the transformed concatenated feature tensor obtained in step S404 and the tensor obtained on each of at least two downsampling layers of the image encoder data other than the first downscaling layer of the encoder data Images. The decoder contains at least one block for increasing spatial data resolution, containing layers in the following order: first convolutional layer with a 1x1 kernel, first activation layer, layer for increasing spatial data resolution, convolutional layer of channel-by-channel two-dimensional convolution, second convolutional layer with a 1x1 kernel and a second layer activation, and the final data upscaling layer after the last data upscaling block. The number of downscaling layers of the decoder data corresponds to the number of downscaling layers of the image encoder data. The tensor obtained in the corresponding spatial resolution reduction layer of the image encoder data starting from the last spatial resolution layer of the image encoder data is used in the corresponding spatial resolution increasing unit of the decoder data starting from the first spatial resolution increasing unit of the decoder data from at least one data spatial resolution increasing unit decoder. Before entering the decoder data into the first downscaling unit from at least one decoder data upscaling unit, the input tensor is obtained by concatenating the transformed concatenated feature tensor with the tensor obtained in the last image coder data downscaling layer. Before entering into each subsequent block for increasing the spatial resolution of the decoder data from at least one block for increasing the spatial resolution of the decoder data, the input tensor is obtained by concatenating the tensor obtained in the previous block for increasing the spatial resolution of the decoder data from at least one block for increasing the spatial resolution of the decoder data, with the tensor obtained in the corresponding downsampling layer of the image encoder data.

[0131] In step S406, the iterative spatial distribution unit predicts a dense depth map using the gradient depth map predicted in step S405, the sparse depth map obtained in step S401, and the tensor obtained in the last spatial upscaling block of the decoder data from at least one block for increasing the spatial resolution of the decoder data. The operation of predicting a dense depth map is performed iteratively with a given number of iterations.

[0132] The image encoder can be one of the standard pre-trained classification networks MobileNet v1, MobileNet v2, InceptionNet, ResNet, R-CNN.

[0133] The predetermined parameter specifying the size of the transformed concatenated feature tensor is a multiple of 2 and is preferably selected in the range 256-1024.

[0134] The decoder may further comprise at least one convolution block in front of each of the at least one data upscaling block. Each convolution block of at least one convolution block contains layers in the following order: a first convolutional layer with a 1x1 kernel, a first activation layer, a channel-by-channel 2D convolutional layer, a second convolutional layer with a 1x1 kernel, and a second activation layer. With this decoder configuration, step S405 further comprises the steps of: dividing the input tensor into a set of n + 1 sub-tensors, where n is the number of convolution blocks; convolution of one subtenzor with a first convolution block of at least one convolution block; before each subsequent convolution block from at least one convolution block, the tensor obtained in the previous convolution block from at least one convolution block is divided into a set of k + 1 subtensions, where k is the number of remaining convolution blocks from at least one convolution block , and obtain a concatenated tensor by concatenating different nk subtenzors taken one from each of the nk sets of subtenzers obtained by splitting the tensors before each of the nk previous convolution blocks, to which no convolution was applied in the convolution blocks, with the subtensor obtained by dividing the tensor before each subsequent a convolution block to which convolution has not been applied in convolution blocks from at least one convolution block ; in each subsequent convolution block of at least one convolution block, performing convolution of the concatenated tensor obtained before this convolution block; obtaining a concatenated tensor by concatenating the remaining subtenzers to which no convolution has been applied in the convolution blocks from at least one convolution block; and performing processing by the spatial resolution upscaling unit on the concatenated tensor data obtained by concatenating the remaining subtenzors to which no convolution has been applied in the convolution blocks from the at least one convolution block.

[0135] Each of the activation layers in the image encoder, the depth encoder and the decoder is one of the layers ReLU, leaky ReLU, ReLU6, ELU.

[0136] FIG. 10 is a block diagram illustrating an electronic computing device 500 configured to perform any of methods 100, 200, 300, 400 for reconstructing a depth map. The electronic computing device 500 includes at least one processor 501 and a memory 502. The memory 502 stores the numerical parameters of a trained image encoder, a trained depth encoder, a transform unit, a trained decoder, an iterative spatial distribution unit, and instructions that when executed by at least one processor 501 cause at least one processor 501 to perform any of the depth map reconstruction methods 100, 200, 300, 400.

[0137] The electronic computing device may further comprise at least one of a camera, a depth sensor, and a user input device. The camera is intended at least for capturing the image supplied to the input of the image encoder. The depth sensor is designed to at least obtain a sparse depth map fed to the input of the depth encoder. The depth sensor can be any depth sensor such as a single beam or dual beam depth sensor. The user input device is intended for at least a user input of depth data by indicating the depth of objects in the image when generating a sparse depth map.

[0138] In addition, the methods disclosed herein may be implemented on a computer-readable medium that stores numeric parameters of trained artificial intelligence and computer-executable instructions that, when executed by a computer processor, cause a computer to perform the methods of the invention. Trained artificial intelligence and instructions for implementing the present methods can be downloaded to an electronic computing device via a network or from a medium.

[0139] The foregoing descriptions of embodiments of the invention are illustrative, and modifications to the configuration and implementation are within the scope of the present description. For example, although embodiments of the invention have been described generally in connection with Figures 1-10, the descriptions provided are exemplary. While the subject matter of the invention has been described in language characteristic of design features or methodological operations, it is understood that the subject matter of the appended claims is not necessarily limited to the specific features or operations described above. Moreover, the specific features and operations described above are disclosed as exemplary forms of implementing the claims. The invention is also not limited to the shown order of the method steps, the order can be modified by a person skilled in the art without innovative innovations. Some or all of the steps of the method can be performed sequentially or in parallel.

[0140] Accordingly, the scope of the embodiments is intended to be limited only by the following claims.

Claims (69)

1. A method for reconstructing a depth map, wherein said method comprises the steps at which: obtaining (S101) an image and a sparse depth map containing depth data that are depths in at most two planes perpendicular to the image plane in the acquired image; calculating (S102) an image feature tensor by a trained image encoder comprising at least one convolutional layer, at least one activation layer and at least two data spatial resolution reduction layers by extracting image features from the obtained image; calculating (S103) a trained depth encoder containing at least one convolutional layer, at least one activation layer and at least two layers of data spatial resolution reduction, a depth feature tensor by extracting depth features from the obtained sparse depth map; calculating (S104) the concatenated feature tensor by concatenating the image feature tensor and the depth feature tensor, and converting the concatenated feature tensor using a predetermined parameter specifying the size of the transformed concatenated feature tensor (S104); predicting (S105) a gradient depth map by the trained decoder using the transformed concatenated feature tensor and the tensor obtained on each of at least two downsampling layers of the image encoder data, except for the first downscaling layer of the image encoder data, the decoder comprising at least one block for increasing spatial data resolution, containing layers in the following order: first convolutional layer with a 1x1 kernel, first activation layer, a layer of increasing spatial data resolution, a convolutional layer of channel-by-channel 2D convolution, a second convolutional layer with a 1x1 kernel and a second activation layer and a final enhancement layer spatial data resolution after the last block for increasing the spatial data resolution, and the number of layers for increasing the spatial resolution of the decoder data corresponds to the number of layers for reducing the spatial resolution of the data of the iso image, and the tensor obtained in the corresponding downscaling layer of the image encoder data starting from the last downsizing layer of the image encoder data is used in the corresponding downscaling block of the decoder data starting from the first downscaling block of the decoder data from at least one block increasing the spatial resolution of the decoder data, and moreover, before entering into the first block for increasing the spatial resolution of the decoder data from at least one block for increasing the spatial resolution of the decoder data, the input tensor is obtained by concatenating the transformed tensor of concatenated features with the tensor obtained in the last layer of lowering the spatial resolution of the encoder data image, and before entering into each subsequent block for increasing the spatial resolution of decoder data from at least one block for increasing spatial resolution Decoding the decoder data, the input tensor is obtained by concatenating the tensor obtained in the previous decoder data upscaling block from at least one decoder data upscaling block with the tensor obtained in the corresponding downscaling layer of the image encoder data; and predicting (S106) a dense depth map by the iterative spatial distribution module using a gradient depth map, a sparse depth map and a tensor obtained in the last spatial upscaling block of decoder data from at least one upscaling block of decoder data, while the dense map prediction operation depth is performed iteratively with a given number of iterations. 2. The method according to claim 1, wherein the image encoder is one of the standard pretrained classification networks MobileNet v1, MobileNet v2, InceptionNet, ResNet, R-CNN. 3. The method according to claim 1, wherein the predetermined parameter specifying the size of the transformed concatenated feature tensor is a multiple of 2 and preferably selected in the range 256-1024. 4. The method according to claim. 1, in which the decoder further comprises at least one convolution block in front of each of the at least one block for increasing spatial data resolution, and each convolution block of the at least one convolution block contains layers in the following order: first a convolutional layer with a 1x1 kernel, a first activation layer, a channel-by-channel 2D convolutional layer, a second convolutional layer with a 1x1 kernel, and a second activation layer, wherein step (S105) further comprises: dividing the input tensor into a set of n + 1 sub-tensors, where n is the number of convolution blocks; convolution of one subtenzor with a first convolution block of at least one convolution block; before each subsequent convolution block from at least one convolution block, the tensor obtained in the previous convolution block from at least one convolution block is divided into a set of k + 1 subtensions, where k is the number of remaining convolution blocks from at least one convolution block , and obtain a concatenated tensor by concatenating different nk subtenzors taken one from each of the nk sets of subtenzers obtained by splitting the tensors before each of the nk previous convolution blocks, to which no convolution was applied in the convolution blocks, with the subtensor obtained by dividing the tensor before each subsequent a convolution block to which convolution has not been applied in convolution blocks from at least one convolution block; in each subsequent convolution block from at least one convolution block, convolution of the concatenated tensor obtained before this convolution block is performed; obtaining a concatenated tensor by concatenating the remaining subtenzers to which no convolution has been applied in convolution blocks from at least one convolution block; and processing by the block for increasing the spatial resolution of the concatenated tensor data obtained by concatenating the remaining subtenzors to which convolution was not applied in convolution blocks from at least one convolution block. 5. The method of claim 4, wherein each of the activation layers in the image encoder, the depth encoder, and the decoder is one of ReLU, leaky ReLU, ReLU6, ELU. 6. A method for reconstructing a depth map, wherein said method comprises the steps at which: obtaining (S201) a sparse depth map containing depth data in the image; calculating (S203) by a trained depth encoder comprising at least one convolutional layer, at least one activation layer and at least two data spatial resolution reduction layers, a depth feature tensor by extracting depth features from the obtained sparse depth map; converting (S204) the conversion unit depth feature tensor using a predetermined parameter specifying the size of the transformed depth feature tensor; predicting (S205) a gradient depth map by a trained decoder using the transformed depth feature tensor as an input tensor, the decoder comprising at least one upscaling unit comprising layers in the following order: first convolutional layer with 1x1 core, first activation layer, a layer for increasing the spatial resolution of data, a convolutional layer of channel-by-channel two-dimensional convolution, a second convolutional layer with a 1x1 kernel and a second activation layer and a final layer for increasing the spatial resolution of data after the last block for increasing the spatial data resolution; and predicting (S206) by the iterative spatial distribution unit a dense depth map using a gradient depth map, a sparse depth map and a tensor obtained in the last upscaling block of decoder data from at least one upscaling block of decoder data, the dense map prediction operation depth is performed iteratively with a given number of iterations. 7. The method according to claim 6, wherein the predetermined parameter specifying the size of the transformed depth feature tensor is a multiple of 2 and preferably selected in the range of 256-1024. 8. The method according to claim 6, wherein the decoder further comprises at least one convolution block in front of each of the at least one convolution block of data, and each convolution block of the at least one convolution block contains layers in the following order: first a 1x1 convolutional layer, a first activation layer, a channel-by-channel 2D convolutional layer, a 1x1 second convolutional layer, and a second activation layer, wherein step (S205) further comprises: dividing the input tensor into a set of n + 1 sub-tensors, where n is the number of convolution blocks; convolution of one subtenzor with a first convolution block of at least one convolution block; before each subsequent convolution block from at least one convolution block, the tensor obtained in the previous convolution block from at least one convolution block is divided into a set of k + 1 subtensions, where k is the number of remaining convolution blocks from at least one convolution block , and obtain a concatenated tensor by concatenating different nk subtenzors taken one from each of the nk sets of subtenzers obtained by splitting the tensors before each of the nk previous convolution blocks, to which no convolution was applied in the convolution blocks, with the subtensor obtained by dividing the tensor before each subsequent a convolution block to which convolution has not been applied in convolution blocks from at least one convolution block; in each subsequent convolution block from at least one convolution block, convolution of the concatenated tensor obtained before this convolution block is performed; obtaining a concatenated tensor by concatenating the remaining subtenzers to which no convolution has been applied in convolution blocks from at least one convolution block; and processing by the block for increasing the spatial resolution of the concatenated tensor data obtained by concatenating the remaining subtenzors to which convolution was not applied in convolution blocks from at least one convolution block. 9. The method of claim 8, wherein each of the activation layers in the depth encoder and decoder is one of ReLU, leaky ReLU, ReLU6, ELU. 10. A method for reconstructing a depth map, wherein said method comprises the steps at which: get (S301) an image; calculating (S302) an image feature tensor by a trained image encoder comprising at least one convolutional layer, at least one activation layer and at least two data spatial resolution reduction layers by extracting image features from the obtained image; converting (S304) the image feature tensor using a predetermined parameter specifying the size of the transformed image feature tensor (S304); predicting (S305) a gradient depth map by the trained decoder using the transformed image feature tensor and the tensor obtained on each of at least two downsampling layers of the image encoder data, except for the first downscaling layer of the image encoder data, the decoder comprising at least one block for increasing spatial data resolution, containing layers in the following order: first convolutional layer with a 1x1 kernel, first activation layer, a layer of increasing spatial data resolution, a convolutional layer of channel-by-channel 2D convolution, a second convolutional layer with a 1x1 kernel and a second activation layer and a final enhancement layer spatial data resolution after the last block for increasing the spatial data resolution, and the number of layers for increasing the spatial resolution of the decoder data corresponds to the number of layers for reducing the spatial resolution of the encoder data. and the tensor obtained in the corresponding spatial resolution reduction layer of the image encoder data starting from the last spatial resolution reduction layer of the image encoder data is used in the corresponding spatial resolution increasing unit of the decoder data starting from the first spatial resolution increasing unit of the decoder data from at least one block increasing the spatial resolution of the decoder data, and moreover, before entering into the first block for increasing the spatial resolution of the decoder data from at least one block for increasing the spatial resolution of the decoder data, the input tensor is obtained by concatenating the transformed tensor of the image features with the tensor obtained in the last layer of lowering the spatial resolution of the encoder data image, and before entering into each subsequent block for increasing the spatial resolution of decoder data from at least one block for increasing spatial resolution of the data x of the decoder, the input tensor is obtained by concatenating the tensor obtained in the previous decoder data upscaling block from at least one decoder data upscaling block with the tensor obtained in the corresponding downscaling layer of the image encoder data; and predict (S306) a dense depth map by an iterative spatial distribution unit using a gradient depth map and a tensor obtained in the last upscaling block of decoder data from at least one upscaling block of decoder data, wherein the dense depth map prediction operation is performed iteratively with a given number of iterations. 11. The method of claim 10, wherein the image encoder is one of the standard pretrained classification networks MobileNet v1, MobileNet v2, InceptionNet, ResNet, R-CNN. 12. The method of claim 10, wherein the predetermined parameter specifying the size of the transformed image feature tensor is a multiple of 2 and preferably selected in the range 256-1024. 13. The method according to claim 10, wherein the decoder further comprises at least one convolution block in front of each of the at least one convolution block of data, and each convolution block of the at least one convolution block contains layers in the following order: first a convolutional layer with a 1x1 kernel, a first activation layer, a channel-by-channel 2D convolutional layer, a second convolutional layer with a 1x1 kernel, and a second activation layer, wherein step (S305) further comprises: dividing the input tensor into a set of n + 1 sub-tensors, where n is the number of convolution blocks; convolution of one subtenzor with a first convolution block of at least one convolution block; before each subsequent convolution block from at least one convolution block, the tensor obtained in the previous convolution block from at least one convolution block is divided into a set of k + 1 subtensions, where k is the number of remaining convolution blocks from at least one convolution block , and obtain a concatenated tensor by concatenating different nk subtenzors taken one from each of the nk sets of subtenzers obtained by splitting the tensors before each of the nk previous convolution blocks, to which no convolution was applied in the convolution blocks, with the subtensor obtained by dividing the tensor before each subsequent a convolution block to which convolution has not been applied in convolution blocks from at least one convolution block; in each subsequent convolution block from at least one convolution block, convolution of the concatenated tensor obtained before this convolution block is performed; obtaining a concatenated tensor by concatenating the remaining subtenzers to which no convolution has been applied in convolution blocks from at least one convolution block; and processing by the block for increasing the spatial resolution of the concatenated tensor data obtained by concatenating the remaining subtenzors to which convolution was not applied in convolution blocks from at least one convolution block. 14. The method of claim 13, wherein each of the activation layers in the image encoder and decoder is one of ReLU, leaky ReLU, ReLU6, ELU. 15. A method for reconstructing a depth map, wherein said method comprises the steps at which: obtaining (S401) an image and a sparse depth map containing depth data that are depths of objects specified by the user in the acquired image; calculating (S402) an image feature tensor by a trained image encoder comprising at least one convolutional layer, at least one activation layer and at least two data spatial resolution reduction layers by extracting image features from the obtained image; calculating (S403) a trained depth encoder containing at least one convolutional layer, at least one activation layer and at least two data spatial resolution reduction layers, a depth feature tensor by extracting depth features from the obtained sparse depth map; calculating (S404) the concatenated feature tensor by concatenating the image feature tensor and the depth feature tensor, and converting the concatenated feature tensor using a predetermined parameter specifying the size of the transformed concatenated feature tensor (S404); predicting (S405) a gradient depth map by the trained decoder using the transformed concatenated feature tensor and the tensor obtained on each of at least two downsampling layers of the image encoder data, in addition to the first downscaling layer of the image encoder data, the decoder comprising at least one block for increasing spatial data resolution, containing layers in the following order: first convolutional layer with a 1x1 kernel, first activation layer, a layer of increasing spatial data resolution, a convolutional layer of channel-by-channel 2D convolution, a second convolutional layer with a 1x1 kernel and a second activation layer and a final enhancement layer spatial data resolution after the last block for increasing the spatial data resolution, and the number of layers for increasing the spatial resolution of the decoder data corresponds to the number of layers for reducing the spatial resolution of the data of the iso image, and the tensor obtained in the corresponding downscaling layer of the image encoder data starting from the last downsizing layer of the image encoder data is used in the corresponding downscaling block of the decoder data starting from the first downscaling block of the decoder data from at least one block increasing the spatial resolution of the decoder data, and moreover, before entering into the first block for increasing the spatial resolution of the decoder data from at least one block for increasing the spatial resolution of the decoder data, the input tensor is obtained by concatenating the transformed tensor of concatenated features with the tensor obtained in the last layer of lowering the spatial resolution of the encoder data image, and before entering into each subsequent block for increasing the spatial resolution of decoder data from at least one block for increasing spatial resolution Decoding the decoder data, the input tensor is obtained by concatenating the tensor obtained in the previous decoder data upscaling block from at least one decoder data upscaling block with the tensor obtained in the corresponding downscaling layer of the image encoder data; and predicting (S406) a dense depth map by the iterative spatial distribution unit using a gradient depth map, a sparse depth map and a tensor obtained in the last upscaling block of decoder data from at least one upscaling block of decoder data, the dense map prediction operation depth is performed iteratively with a given number of iterations. 16. The method of claim 15, wherein the image encoder is one of the standard pretrained classification networks MobileNet v1, MobileNet v2, InceptionNet, ResNet, R-CNN. 17. The method of claim 15, wherein the predetermined parameter specifying the size of the transformed concatenated feature tensor is a multiple of 2 and preferably selected in the range 256-1024. 18. The method according to claim 15, wherein the decoder further comprises at least one convolution block in front of each of the at least one convolution block of data, and each convolution block of the at least one convolution block contains layers in the following order: first a 1x1 convolutional layer, a first activation layer, a channel-by-channel 2D convolutional layer, a 1x1 second convolutional layer, and a second activation layer, wherein step (S405) further comprises: dividing the input tensor into a set of n + 1 sub-tensors, where n is the number of convolution blocks; convolution of one subtenzor with a first convolution block of at least one convolution block; before each subsequent convolution block from at least one convolution block, the tensor obtained in the previous convolution block from at least one convolution block is divided into a set of k + 1 subtensions, where k is the number of remaining convolution blocks from at least one convolution block , and obtain a concatenated tensor by concatenating different nk subtenzors taken one from each of the nk sets of subtenzers obtained by splitting the tensors before each of the nk previous convolution blocks, to which no convolution was applied in the convolution blocks, with the subtensor obtained by dividing the tensor before each subsequent a convolution block to which convolution has not been applied in convolution blocks from at least one convolution block; in each subsequent convolution block from at least one convolution block, convolution of the concatenated tensor obtained before this convolution block is performed; obtaining a concatenated tensor by concatenating the remaining subtenzers to which no convolution has been applied in convolution blocks from at least one convolution block; and processing by the block for increasing the spatial resolution of the concatenated tensor data obtained by concatenating the remaining subtenzors to which convolution was not applied in convolution blocks from at least one convolution block. 19. The method of claim 18, wherein each of the activation layers in the image encoder, depth encoder, and decoder is one of ReLU, leaky ReLU, ReLU6, ELU. 20. Electronic computing device containing: at least one processor and a memory that stores the numerical parameters of a trained image encoder, a trained depth encoder, a transform module, a trained decoder, an iterative spatial distribution module and instructions that, when executed by at least one processor, cause at least one processor to execute the method according to any one of claims. 1-19. 21. The electronic computing device of claim 20, further comprising at least one of a camera, a depth sensor, and a user input device.

Images