KR20210144519A - Method and apparatus of generating depth image - Google Patents

Abstract

A method and apparatus for generating a depth image according to an embodiment include: receiving an input image; extracting a feature corresponding to the input image; generating a feature for each depth resolution by decoding the feature using decoders corresponding to different depth resolutions; estimating a probability distribution for each depth resolution by gradually correcting the feature for each depth resolution; and generating a target depth image corresponding to the input image, based on a finally estimated probability distribution among the probability distributions for each depth resolution.

Description

METHOD AND APPARATUS OF GENERATING DEPTH IMAGE

The following embodiments relate to a method and an apparatus for generating a depth image.

The use of 3D information is important for image recognition or scene understanding. Adding depth information to spatial information in 2D makes it possible to effectively predict the spatial distribution of objects. In general, depth information can be known only by acquiring a depth image using a depth camera, and the quality of a depth image obtainable therefrom varies according to the performance of the depth camera. For example, the resolution or noise level of the acquired depth image may vary according to the performance of the depth camera. Since the accuracy of depth information greatly affects the quality of a result using the corresponding depth information, it is important to obtain a high-quality depth image.

The above-mentioned background art is possessed or acquired by the inventor in the process of deriving the disclosure of the present application, and it cannot be said that it is necessarily known technology disclosed to the general public prior to the present application.

According to an embodiment, a method for generating a depth image includes receiving an input image; extracting a feature corresponding to the input image; generating a feature for each depth resolution by decoding the feature using decoders corresponding to different depth resolutions; estimating a probability distribution for each depth resolution by progressively refining the feature for each depth resolution; and generating a target depth image corresponding to the input image based on a finally estimated probability distribution among the probability distributions for each depth resolution.

The generating of the feature for each depth resolution may include: generating the feature of the first depth resolution by using a first decoder corresponding to a first depth resolution among the depth resolutions; and generating a residual feature of the second depth resolution by using a second decoder corresponding to the second depth resolution among the depth resolutions.

The generating of the feature for each depth resolution may further include generating a residual feature of the third depth resolution by using a third decoder corresponding to a third depth resolution among the depth resolutions. have.

The generating of the feature for each depth resolution may include: generating the feature for each depth resolution by decoding the feature at a uniformly set depth interval; and generating the feature for each depth resolution by decoding the feature with a depth interval set differently based on a spacing-increasing discretization (SID) technique.

The estimating of the probability distribution for each depth resolution may include: estimating a first probability distribution corresponding to depth ranges of the first depth resolution based on a characteristic of the first depth resolution among the depth resolutions; and estimating a second probability distribution corresponding to depth ranges of a second depth resolution based on the first probability distribution and a residual feature of a second depth resolution among the depth resolutions.

The estimating of the second probability distribution includes estimating the second probability distribution corresponding to the depth ranges of the second depth resolution by correcting the first probability distribution by the residual feature of the second depth resolution. may include

The estimating of the second probability distribution may include: up-scaling the first probability distribution; and estimating the second probability distribution by correcting the up-scaled first probability distribution by a residual feature of the second depth resolution.

The estimating of the probability distribution for each depth resolution may include estimating a third probability distribution corresponding to depth ranges of a third depth resolution based on the second probability distribution and a residual feature of a third depth resolution among the depth resolutions. It may further include the step of

The estimating of the third probability distribution includes estimating the third probability distribution corresponding to the depth ranges of the third depth resolution by correcting the second probability distribution by the residual feature of the third depth resolution. may include

The estimating of the third probability distribution may include: up-scaling the second probability distribution; and estimating the third probability distribution by correcting the up-scaled second probability distribution by the residual feature of the third depth resolution.

The generating of the target depth image may include converting the finally estimated probability distribution into the target depth image.

The generating of the target depth image may include calculating an expected value of the finally estimated probability distribution; estimating a correction value of the expected value based on the finally estimated probability distribution; and generating a target depth image based on the expected value and the correction value.

The depth resolutions include at least two of a first depth resolution, a second depth resolution, and a third depth resolution, the first depth resolution having a coarse value compared to the second depth resolution, The second depth resolution may have a finer value than the first depth resolution, and the third depth resolution may have a finer value than the second depth resolution.

The method of generating the depth image may further include discretizing a depth range of depth values of pixels included in the input image and dividing the depth range into a plurality of sections.

The method of generating the depth image may further include outputting the target depth image.

The input image may include at least one of a single color image, an infrared image, and a depth image.

According to an embodiment, a method of generating a depth image includes: receiving a depth image; receiving a color image; generating a probability distribution of a first depth resolution by discretizing the depth image; extracting a feature corresponding to the color image; generating a feature for each second depth resolution by decoding the feature using at least one decoder corresponding to at least one second depth resolution; estimating a probability distribution for each depth resolution by gradually correcting the feature for each depth resolution; and generating a target depth image corresponding to the input image based on a finally estimated probability distribution among the probability distributions for each depth resolution.

According to an embodiment, an apparatus for generating a depth image includes a communication interface for receiving an input image; and extracting a feature corresponding to the input image, decoding the feature using decoders corresponding to different depth resolutions, generating a feature for each depth resolution, and gradually correcting the feature for each depth resolution. and a processor estimating a probability distribution for each resolution and generating a target depth image corresponding to the input image based on a finally estimated probability distribution among the probability distributions for each depth resolution.

The processor generates the feature of the first depth resolution by using a first decoder corresponding to a first depth resolution among the depth resolutions, and uses a second decoder corresponding to a second depth resolution among the depth resolutions. Thus, a residual feature of the second depth resolution may be generated.

The processor may generate a residual feature of the third depth resolution by using a third decoder corresponding to the third depth resolution among the depth resolutions.

The processor generates the feature for each depth resolution by decoding the feature at a uniformly set depth interval, or by decoding the feature by a depth interval set differently based on an increased spacing discretization (SID) technique. You can create features.

The processor estimates a first probability distribution corresponding to depth ranges of the first depth resolution based on a characteristic of a first depth resolution among the depth resolutions, and estimates a first probability distribution of the first probability distribution and the depth resolutions. A second probability distribution corresponding to depth ranges of the second depth resolution may be estimated based on the residual feature of the second depth resolution.

The processor may estimate the second probability distribution corresponding to the depth ranges of the second depth resolution by correcting the first probability distribution by the residual feature of the second depth resolution.

The processor may estimate the second probability distribution by up-scaling the first probability distribution and correcting the up-scaled first probability distribution by a residual feature of the second depth resolution.

The processor may estimate a third probability distribution corresponding to depth ranges of a third depth resolution based on the second probability distribution and a residual feature of a third depth resolution among the depth resolutions.

The processor may estimate the third probability distribution corresponding to the depth ranges of the third depth resolution by correcting the second probability distribution by the residual feature of the third depth resolution.

The processor may estimate the third probability distribution by up-scaling the second probability distribution and correcting the up-scaled second probability distribution by a residual feature of the third depth resolution.

The processor may convert the finally estimated probability distribution into the target depth image.

The processor calculates an expected value of the finally estimated probability distribution, estimates a correction value of the expected value based on the finally estimated probability distribution, and based on the expected value and the correction value, a target You can create a depth image.

the depth resolutions include at least two of a first depth resolution, a second depth resolution, and a third depth resolution, wherein the first depth resolution has a sparse value compared to the second depth resolution, wherein the second depth resolution is It may have a denser value than the first depth resolution, and the third depth resolution may have a denser value than the second depth resolution.

The processor may discretize a depth range of depth values of pixels included in the input image and divide it into a plurality of sections.

The communication interface may output the target depth image.

The input image may include at least one of a single color image, an infrared image, and a depth image.

A device for generating the depth image is a smart phone, a smart TV, a tablet, a head-up display (HUD), a 3D digital information display (DID), a 3D mobile device , and a smart vehicle (Smart Automobile) may include at least one.

1 is a flowchart illustrating a method of generating a depth image according to an exemplary embodiment;
FIG. 2 is a diagram for describing a configuration and operation of an apparatus for generating a depth image according to an exemplary embodiment;
FIG. 3 is a diagram for explaining a concept of substituting a pixel value with class labels of depth ranges to generate a depth image according to an exemplary embodiment;
4 is a diagram for explaining a concept of gradually correcting a feature for each depth resolution according to an exemplary embodiment;
5 is a flowchart illustrating a method of generating a feature for each depth resolution according to an embodiment.
6 is a flowchart illustrating a method of estimating a probability distribution for each depth resolution according to an embodiment;
7 is a diagram for describing a learning process of an apparatus for generating a depth image according to an exemplary embodiment;
8 is a diagram for describing a configuration and operation of an apparatus for generating a depth image according to another exemplary embodiment;
9 is a flowchart illustrating a method of generating a depth image according to another exemplary embodiment;
10 is a view for explaining a configuration and operation of an apparatus for generating a depth image according to another embodiment;
11 is a block diagram of an apparatus for generating a depth image according to an exemplary embodiment;

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, since various changes may be made to the embodiments, the scope of the patent application is not limited or limited by these embodiments. It should be understood that all modifications, equivalents and substitutes for the embodiments are included in the scope of the rights.

The terms used in the examples are used for description purposes only, and should not be construed as limiting. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present specification, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It should be understood that this does not preclude the existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiment belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present application. does not

In addition, in the description with reference to the accompanying drawings, the same components are given the same reference numerals regardless of the reference numerals, and the overlapping description thereof will be omitted. In the description of the embodiment, if it is determined that a detailed description of a related known technology may unnecessarily obscure the gist of the embodiment, the detailed description thereof will be omitted.

In addition, in describing the components of the embodiment, terms such as first, second, A, B, (a), (b), etc. may be used. These terms are only for distinguishing the components from other components, and the essence, order, or order of the components are not limited by the terms. When it is described that a component is “connected”, “coupled” or “connected” to another component, the component may be directly connected or connected to the other component, but another component is between each component. It will be understood that may also be "connected", "coupled" or "connected".

Components included in one embodiment and components having a common function will be described using the same names in other embodiments. Unless otherwise stated, descriptions described in one embodiment may be applied to other embodiments as well, and detailed descriptions within the overlapping range will be omitted.

1 is a flowchart illustrating a method of generating a depth image according to an embodiment, and FIG. 2 is a diagram for explaining a configuration and operation of an apparatus for generating a depth image according to an embodiment. Referring to FIGS. 1 and 2 , an apparatus for generating a depth image (hereinafter, 'generating apparatus') 200 according to an embodiment includes an input image 205 through steps 110 to 150 below. A target depth image 250 corresponding to may be generated. The generating apparatus 200 according to an embodiment may be configured as, for example, an encoder-decoder neural network including one encoder and three decoders. Hereinafter, for a more detailed description, an operation of the generating apparatus 200 configured with a deep neural network including one encoder and three decoders will be described, but the present invention is not limited thereto. The generating apparatus 200 may be configured as, for example, an encoder-decoder neural network including one encoder and two decoders.

In operation 110 , the generating device 200 receives the input image 205 . The input image 205 may include, for example, at least one of a single color image including RGB color information, an infrared image, and a depth image. A single color image may also be referred to as a 'single RGB image'. The single color image may be detected by, for example, an image sensor such as a complementary metal-oxide-semiconductor (CMOS) image sensor, a charge-coupled device (CCD) image sensor, or a stereo camera. The infrared image may be an image sensed by an infrared sensor. Here, the single color image, the infrared image, and/or the depth image sensed by each sensor may correspond to each other as images sensed by the same scene.

In operation 110 , the generating apparatus 200 may discretize a depth range of depth values of pixels included in the input image 205 and divide it into a plurality of sections. A process in which the generating apparatus 200 discretizes the depth range of depth values and divides it into a plurality of sections will be described in more detail with reference to FIG. 3 below.

In operation 120 , the generating device 200 extracts a feature corresponding to the input image 205 . The generating device 200 may extract features corresponding to the input image 205 through, for example, encoding by an encoder 210 such as SENet pre-trained for an image recognition task. . The encoder 210 extracts, for example, four features having different spatial sizes of 1/32, 1/16, 1/8 and 1/4 times the size of the input image 205 . can do. The four extracted features may be integrated into one through, for example, up-projection processes.

In step 130 , the generating apparatus 200 decodes the feature extracted in step 120 using the decoders 220 , 230 , and 240 corresponding to different depth resolutions, thereby performing a feature for each depth resolution. create Here, the different depth resolutions may include, for example, at least two of a first depth resolution, a second depth resolution, and a third depth resolution, but is not necessarily limited thereto. The depth resolutions may include more levels of values, such as a fourth depth resolution, a fifth depth resolution, and the like. In this specification, 'level' may be used interchangeably with 'resolution' or 'scale'.

For example, the first depth resolution may have a coarse value compared to the second depth resolution. Also, the second depth resolution may have a finer value than that of the first depth resolution. The third depth resolution may have a denser value than the second depth resolution. In other words, the depth resolutions may vary from a coarse value to a dense value in the order of the first depth resolution, the second depth resolution, and the third depth resolution.

The generating apparatus 200 may generate a feature for each depth resolution by, for example, decoding the feature extracted in step 120 by pre-trained classifiers or decoders 220 , 230 , and 240 . . In this case, the features extracted in step 120 may all be equally input to the decoders 220 , 230 , and 240 . Decoders 220 , 230 , 240 may decode the feature to different depth resolutions step by step, for example through different levels of quantization.

The generating apparatus 200 may generate a feature of the first depth resolution using, for example, the first decoder 220 corresponding to the first depth resolution among the depth resolutions. The generating apparatus 200 may generate a residual feature of the second depth resolution by using the second decoder 230 corresponding to the second depth resolution among the depth resolutions. The generating apparatus 200 may generate a residual feature of the third depth resolution by using the third decoder 240 corresponding to the third depth resolution among the depth resolutions.

The generating apparatus 200 may generate a feature for each depth resolution by, for example, decoding the feature at a uniformly set depth interval. Alternatively, the generating apparatus 200 may generate a feature for each depth resolution by, for example, decoding the feature by a different set depth interval. The generating device 200 decodes a feature by a depth interval set differently based on various spacing-increasing discretization (SID) techniques, such as a partially convex function, or a log function, for example. By doing so, it is possible to generate features for each depth resolution.

A process in which the generating device 200 generates features for each depth resolution will be described in more detail with reference to FIG. 5 below.

In step 140 , the generating apparatus 200 estimates a probability distribution for each depth resolution by progressively refining the feature for each depth resolution. According to an embodiment, the concept that the generating device 200 'gradually corrects features for each depth resolution' will be described in more detail with reference to FIG. 4 below.

In step 140 , the generating device 200 generates a first probability distribution corresponding to depth ranges of the first depth resolution, for example, based on the characteristic of the first depth resolution generated by the first decoder 220 . (P ₁ ) (225) can be estimated. _{The generating apparatus 200 corrects 227 the first probability distribution (P 1} ) 225 by the residual feature of the second depth resolution generated by the second decoder 230 to determine depth ranges of the second depth resolution. A second probability distribution (P ₂ ) 235 corresponding to may be estimated. The second decoder 230 may be learned to output a residual feature of the second depth resolution based on the feature extracted in step 120 , for example. The residual feature of the second depth resolution may include, for example, depth information of the intermediate resolution.

_{The generating device 200 corrects 237 the second probability distribution (P 2} ) 235 by the residual feature of the third depth resolution generated by the third decoder 240 to determine depth ranges of the third depth resolution. A third probability distribution (P ₃ ) 245 corresponding to may be estimated. The third decoder 240 may be, for example, learned to output the residual feature of the third depth resolution based on the feature extracted in step 120 . The residual feature of the third depth resolution may include, for example, depth information of a high resolution, and may correspond to a feature that relatively well represents an edge component of an object. The generating device 200 guides a residual component of depth information that was not estimated at the low depth resolution of the previous step to be processed in the current step, so that, in each step, the depth information corresponding to the depth resolution in charge is separated and independently can be estimated.

In one embodiment, the corrections 227 and 237 may include both an up-sampling operation and an integration operation with the probability distribution of the previous step. A method for the generating apparatus 200 to estimate a probability distribution for each depth resolution will be described in more detail with reference to FIG. 6 below.

In operation 150 , the generating apparatus 200 generates an input image 205 based on a finally estimated probability distribution (eg, a third probability distribution (P _{3 ) 245 ) among the probability distributions for each depth resolution.} A target depth image 250 corresponding to . In this case, the generating device 200 may convert the finally estimated probability distribution. The generating apparatus 200 may generate the target depth image 250 by converting _{the third probability distribution (P 3} ) 245 , which is a finally estimated discrete probability distribution, into a continuous value.

In operation 150 , the generating device 200 may calculate an expectation value of the finally estimated probability distribution. The expected value of the probability distribution may correspond to a weighted mean of the probability distribution. The generator 200 may estimate a correction value of the expected value based on the finally estimated probability distribution. The generating apparatus 200 may estimate the correction value of the expected value by, for example, a convolutional neural network (CNN), but is not limited thereto. The generating apparatus 200 may generate the target depth image 250 based on the expected value and the correction value.

The generating apparatus 200 may output the target depth image 250 generated in step 150 . The generating device 200 may implicitly output the target depth image 250 through an output device such as a display or the like, or externally to the outside of the generating device 200 through a communication interface. It can also be output explicitly.

The generating apparatus 200 according to an embodiment may finally generate the target depth image 250 through a structure in which depth information is gradually refined through segmentation for each depth resolution. The generating apparatus 200 may configure various depth resolutions or scales of the input image 205 , and may acquire different depth information from each depth resolution. The generating apparatus 200 may generate the final target depth image 250 by combining the acquired different depth information. Through this, the generating apparatus 200 may generate a high-quality depth image from a color image or an infrared image without using a separate depth sensor or a separate depth image.

The generating apparatus 200 may generate a large amount of depth images corresponding to a single RGB image or an infrared image by the above-described method and utilize it for supervised learning. The generating apparatus 200 recognizes, for example, a 3D object or 3D face included in the input image 205 by the target depth image 250 generated in response to the input image 205 , or a photo out-focusing, and/or functions such as a digital ruler may be performed.

In order to increase the utility of the depth image, it is important to use a high-resolution (or high-quality) depth image. In order to obtain a desirable result using a depth image, it is important to obtain a depth image in which a depth feature (eg, a depth feature for an edge of an object) is well represented. The generating apparatus 200 may generate a high-resolution and high-quality depth image by estimating depth information more precisely and accurately through the method of generating a multi-scale-based depth image described through this specification.

According to an embodiment, the generating apparatus 200 may generate a high-resolution (or high-quality) depth image from a relatively low-resolution (or low-quality) depth image. A method in which the generating apparatus 200 generates a high-resolution depth image from a low-resolution depth image will be described in detail with reference to FIGS. 9 to 10 below.

FIG. 3 is a diagram for explaining a concept of substituting a pixel value with class labels of depth ranges to generate a depth image, according to an exemplary embodiment. Referring to FIG. 3 , a depth image 310 corresponding to an input image and an image 330 to which class labels of depth ranges corresponding to the depth image 310 are assigned are shown according to an exemplary embodiment.

For example, when a single RGB image is given, the generating apparatus according to an embodiment may generate the depth image 310 based on multi-scale classification.

The generating apparatus may replace the task of generating (or estimating) the depth image by discretizing the depth image 310 corresponding to the single RGB image with the task of estimating the class of each pixel, such as the image 330 . .

The generating apparatus may generate the depth image 310 by estimating a depth value in units of pixels with respect to a single RGB image. In this case, the generating apparatus solves the problem of generating a depth image by dividing the range of depth values into N countable (eg, 16) sections through quantization of the depth values. can be converted into a problem of classifying Here, 'sections of ranges of depth values' correspond to 'classes', and the number N of sections may be set larger as the scale or depth resolution increases. In other words, since a more dense estimation is possible as the level or depth resolution increases, a target depth image having a dense value at the final level or final depth resolution may be generated.

For example, it is assumed that a depth value of one pixel corresponding to a sofa in the depth image 310 is 4.532, and the corresponding depth value corresponds to a tenth section among 16 sections. The generating apparatus may assign a label (eg, 10) to a class of corresponding pixels corresponding to the sofa like the image 330 through discretization of the depth image 310 . The generating apparatus may perform quantization for each multi-level or depth resolution to enable gradual correction of the probability distribution for each depth resolution.

4 is a diagram for explaining a concept of gradually correcting a feature for each depth resolution according to an exemplary embodiment. Referring to (a) of FIG. 4 , depth images in which the range of the depth value, that is, the label of the class, is gradually increased to, for example, 4, 8, and 16 through gradual correction of the depth values ( 410, 420, 430 are shown. In addition, referring to FIG. 4 (b), the class for the probability distribution of each pixel (eg, pixel A and pixel B) included in the depth image 410 of FIG. 4 (a) is labeled. Results are shown.

The generation device can increase the number of depth ranges (that is, class labels of depth resolution) to 4, 8, and 16 by finely refining the probability distribution for each depth resolution step by step in a coarse-to-fine method. can

For example, the generating apparatus upscales the first probability distribution corresponding to the four class labels in the depth image 410 , and then the second probability distribution corresponding to the eight class labels as in the depth image 420 . can be corrected with In addition, the generating apparatus up-scales the second probability distribution corresponding to 8 class labels as in the depth image 420 , and then up-scaling the third probability distribution corresponding to 16 class labels as in the depth image 430 . It can be corrected with a probability distribution.

In this case, both the pixel A and the pixel B included in the depth image 410 may belong to the class label 3 , but the probability that each pixel belongs to the class label 3 may be different from each other. A probability that pixel A belongs to class label 3 may be 0.41, and a probability that pixel B belongs to class label 3 may be 0.6. In this way, the generating device can reduce error propagation from the previous step to the current step by bringing all the information (probability distribution for each pixel) of the previous step to the current step. The generating apparatus may predict the target depth image from the probability distribution corresponding to the final level (eg, 16 class labels).

5 is a flowchart illustrating a method of generating a feature for each depth resolution according to an embodiment. Referring to FIG. 5 , the generating apparatus according to an embodiment may generate features for each depth resolution by performing steps 510 and 520 .

In operation 510 , the generating device generates a feature of the first depth resolution by using a first decoder corresponding to the first depth resolution among the depth resolutions.

In operation 520 , the generating apparatus generates a residual feature of the second depth resolution by using a second decoder corresponding to the second depth resolution among the depth resolutions. Residual features may also be referred to as 'residual probabilistic features'.

According to an embodiment, the generating apparatus may generate a residual feature of the third depth resolution by using a third decoder corresponding to the third depth resolution among the depth resolutions.

6 is a flowchart illustrating a method of estimating a probability distribution for each depth resolution according to an embodiment. Referring to FIG. 6 , the generating apparatus according to an embodiment may estimate a probability distribution for each depth resolution by performing steps 610 and 620 .

In operation 610 , the generating apparatus may estimate a first probability distribution corresponding to depth ranges of the first depth resolution based on the characteristic of the first depth resolution among the depth resolutions.

In operation 620 , the generating apparatus may estimate a second probability distribution corresponding to depth ranges of the second depth resolution based on a residual feature of the second depth resolution among the first probability distribution and the depth resolutions. The generating apparatus may estimate the second probability distribution corresponding to the depth ranges of the second depth resolution by, for example, correcting the first probability distribution by the residual feature of the second depth resolution.

In operation 620 , the generating device may up-scaling the first probability distribution by, for example, bilinear interpolation, but is not limited thereto. The generating apparatus may estimate the second probability distribution by correcting the up-scaled first probability distribution by the residual feature of the second depth resolution. Here, 'correcting' the first probability distribution up-scaled by the residual feature of the second depth resolution is understood to mean combining or summing the residual feature of the second depth resolution and the up-scaled first probability distribution. can Here, the combination may correspond to a summation or a weighted sum between depth values of pixel positions corresponding to each other.

According to an embodiment, the generating apparatus may estimate a third probability distribution corresponding to depth ranges of the third depth resolution based on a residual feature of the third depth resolution among the second probability distribution and the depth resolutions. The generating apparatus may estimate the third probability distribution corresponding to the depth ranges of the third depth resolution by correcting the second probability distribution by the residual feature of the third depth resolution. The generating device may up-scale the second probability distribution by, for example, but not limited to, bilinear interpolation. The generating apparatus may estimate the third probability distribution by correcting the second probability distribution up-scaled by the residual feature of the third depth resolution.

The generating apparatus according to an embodiment estimates a depth value for each pixel using a probability distribution such as, for example, an impulse function, and transmits the probability distribution itself (eg, a residual feature) to the next level of depth resolution. can The generating device may receive the probability distribution of the previous level as an input and correct or supplement the given probability distribution by adding or subtracting it to the depth resolution of the next level. In this case, the number of elements of the probability distribution may increase as the steps progress.

7 is a diagram for describing a learning process of an apparatus for generating a depth image according to an exemplary embodiment. Referring to the upper part of FIG. 7 , depth images estimated for each level are shown, and referring to the lower part of FIG. 7 , ground truth depth images corresponding to the depth images estimated for each level are shown.

The generating apparatus according to an embodiment provides, for example, a depth value corresponding to each level and a probability distribution of the depth value through correction in the probability distribution domain for each level or for each depth resolution through a deep neural network composed of encoder-decoders. can be estimated. The generator may train the classifiers or decoders for each level to minimize a difference between the depth values of pixels estimated at each level and the depth values of the same pixels in the correct answer depth image of the same level.

For example, the generating device may calculate a difference between the correct answer depth image and the estimated depth image by comparing the correct answer depth image corresponding to the ground truth of the depth information and the estimated depth image. The generating apparatus may adjust values of parameters constituting the neural network so that a difference between the correct answer depth image and the estimated depth image is reduced. For example, the generating apparatus may find an optimal parameter value in a direction in which a value of a loss function defining a difference between the correct depth image and the estimated depth image is minimized. Here, the loss function may be defined in various forms according to a classification method, a regression method, and the like. A method of adjusting a parameter value or a calibration process of depth information for generating an estimated depth image may vary depending on how the loss function is defined.

및 데이터 손실 함수(data loss function)

의 총합으로 정의될 수 있다. The loss function (L) is, for example, a classification loss function as shown in Equation 1 below.

and a data loss function.

can be defined as the sum of

는 올바른 클래스의 위치에서 '1'을 갖고, k번째 깊이 해상도에서 '0'을 갖는 원-핫 벡터(one-hot vector)를 나타낼 수 있다. 내고,

는 k번째 깊이 해상도에서의 정답 깊이 영상을 나타낼 수 있다. here,

may represent a one-hot vector having '1' at the position of the correct class and '0' at the kth depth resolution. pay,

may represent the correct answer depth image at the kth depth resolution.

The generator may find an optimal parameter value for each of the classifiers or decoders included in the neural network by repeatedly performing the above process for a large number of training images.

Through the above-described learning process, the second decoder is trained to output the residual feature of the second depth resolution based on the feature extracted by the encoder, and the third decoder is the residual feature of the third depth resolution based on the feature extracted by the encoder It can be learned to output.

8 is a diagram for describing a configuration and operation of an apparatus for generating a depth image according to another exemplary embodiment. Referring to FIG. 8 , the generating apparatus 800 according to an embodiment includes a resolution feature extraction (RFE) module including one encoder 810 and three decoders 821 , 823 , and 825 ( 820 , a probability distribution correction (PR) module 840 , and a depth transform unit 860 .

The resolution feature extraction (RFE) module 820 may include three decoders 821 , 823 , 825 that perform decoding at three depth resolutions. Also, the probability distribution correcting module (PR) module 840 may include a softmax performing unit 841 and probability distribution correcting units (PRs) 843 and 845 .

In FIG. 8 , R ₁ and R ₂ may represent residual features corresponding to the second depth resolution and the third depth resolution, respectively. P ₁ , P ₂ , and P ₃ may represent probability distributions corresponding to depth ranges of the first depth resolution, the second depth resolution, and the third depth resolution, respectively. Also, d ₁ , d ₂ , and d ₃ may represent the number of sections (or classes) of depth ranges of each of the first depth resolution, the second depth resolution, and the third depth resolution.

The generating apparatus 800 according to an embodiment may perform multi-scale depth classification from a single image by, for example, a deep neural network. A deep neural network can estimate, for example, a per-pixel depth value from coarse to dense. To this end, in one embodiment, the successive depth interval may be first quantized by several sets of individual labels with different granularities. Also, according to an embodiment, the probability distribution corresponding to the depth ranges of the predicted depth resolution provided by a series of classifiers may be subdivided, and the depth value of each pixel may be calculated by weighted sum of the subdivided probability distributions. In an embodiment, it is possible to effectively reduce quantization artifacts through gradual prediction and gradual depth correction while simplifying operations through such multi-scale classification.

The generating apparatus 800 may convert monocular depth estimation for a single RGB image into a multi-scale classification operation to predict depth values of all pixels. The generating device 800 predicts the probability distribution for the discretized depth value, and based on this, predicts the target depth image at the final level to generate a smooth depth map without discretization artifacts. can To achieve higher accuracy, the generating device 800 may incorporate a refinement module for post-processing to maximize the accuracy by adjusting the final probability distribution.

More specifically, the generating device 800 receiving the input image 805 (eg, a single RGB image) may encode the input image 805 through the encoder 810 . The encoder 810 may extract four features 815 having different spatial sizes of, for example, 1/32, 1/16, 1/8 and 1/4 times the size of the input image 805. can The four features 815 may be integrated into one, for example, via up-projection processes. The generating apparatus may perform decoding for three different depth resolutions on the integrated feature through the up-projection processor.

The resolution feature extraction (RFE) module 820 is configured with several extended convolutional filters with various dilation rates r ∈ {3, 6, 12, 18} to capture different levels of receptive fields. It can be carried out in a combination of branches. _{The residual features R 1} , R ₂ extracted by the resolution feature extraction (RFE) module 820 may be supplied to the probability distribution correction (PR) module 840 .

The probability distribution correction (PR) module 840 may estimate a probability distribution for each depth resolution by integrating features of previous levels. The densest probability distribution estimated by the probability distribution correction (PR) module 840 may be supplied to the depth converter 860 to finally generate a target depth image 870 or a final depth map.

를 추정할 수 있다. 확률 분포는 디코딩 과정에서 여러 단계의 정제 과정(refinement processes)을 통해 계층적으로 추정될 수 있다. 디코딩 단계에서 깊이 해상도 별 분류를 예측하기 위해, 일 실시예에서는 로그 함수를 사용하는 간격 증가 이산화(spacing-increasing discretization; SID)를 채택할 수 있다. 생성 장치(800)는 디코딩의 각 단계에서, 서로 다른 레벨들의 양자화를 사용할 수 있다. The generating device 800 provides, for example, a probability distribution corresponding to three different depth resolutions through the decoders 821 , 823 , and 825 .

can be estimated. The probability distribution may be hierarchically estimated through several refinement processes in the decoding process. In order to predict classification by depth resolution in the decoding step, in an embodiment, spacing-increasing discretization (SID) using a log function may be employed. The generating device 800 may use different levels of quantization in each step of decoding.

In one embodiment, for example, the entire depth interval [a, b] _{can be divided into three different numbers of depth ranges d 1} , d ₂ and d 3 , and the initial probability feature can be predicted through the neural network. have. The generating apparatus 800 may refine or correct values constituting the probability distribution as the depth resolution is gradually increased.

는

차원을 가질 수 있다. 이때,

이고,

일 수 있다. 여기서, H 및 W는 가장 조밀한 깊이 해상도 또는 가장 정밀한 레벨에서의 깊이 영상의 높이 및 폭을 나타낼 수 있다. 또한,

는 p 번째 픽셀의 깊이 값이 k 번째 깊이 해상도 또는 k 번째 레벨에서 특정 간격으로 존재할 가능성을 나타낼 수 있다. _{The probability distributions P 1} , P ₂ , and P ₃ for each depth resolution estimated by the generator 800 may have different spatial and channel resolutions. For example, if H and W represent the height and width of the depth image at the densest depth resolution,

Is

can have dimensions. At this time,

ego,

can be Here, H and W may represent the height and width of the depth image at the densest depth resolution or the most precise level. In addition,

may indicate the possibility that the depth value of the p-th pixel exists at a specific interval in the k-th depth resolution or the k-th level.

The generating device 800 may predict _{the first probability distribution P 1} from, for example, a feature of the first depth resolution generated by the first decoder 821 corresponding to the coarsest first depth resolution. _{According to an embodiment, the generating device 800 may predict the first probability distribution P 1} from the feature of the first depth resolution by the softmax performing unit 841 or directly without the softmax performing unit 841 . _{The first probability distribution P 1} may be predicted from the feature of the first depth resolution. In this case, the first probability distribution may correspond to depth ranges (eg, d ₁ = 4) of the first depth resolution.

The generating device 800 may predict _{the residual feature R 1} of the second depth resolution denser than the first depth resolution by the second decoder 823 . _{The generation device 800 aggregates the probability distribution P 1} of the previous step (first depth resolution) and the residual feature R ₁ of the second depth resolution by the probability distribution correcting unit 843 to obtain a second A probability distribution (P ₂ ) can be estimated. In this case, the second probability distribution P ₂ may correspond to depth ranges (eg, d ₂ = 8) of the second depth resolution.

Also, the generating device 800 may predict _{the residual feature R 2} of the third depth resolution that is denser than the second depth resolution by the third decoder 825 . The generating device 800 integrates the residual feature R _{2 of the third depth resolution with the probability distribution P 2} of the previous step (second depth resolution) by the probability distribution correcting unit 845 to obtain the third probability distribution ( P ₂ ) can be estimated. In this case, the third probability distribution P ₃ may correspond to depth ranges (eg, d ₃ =16) of the third depth resolution.

을 초기 값으로 예측한 후, 제1 깊이 해상도에서 예측된 거친 확률 분포(예를 들어, 제1 확률 분포

)에 더 미세한 구조를 추가하기 위해 제2 잔차 특징

을 예측할 수 있다. More specifically, the generating device 800 generates a first probability distribution

After predicting as an initial value, the coarse probability distribution predicted at the first depth resolution (eg, the first probability distribution

) to add a finer structure to the second residual feature

can be predicted.

및 제3 확률 분포

를 추정할 수 있다. 이때,

는 아래의 수학식 2에 기재된

로부터의 정보를 집계(aggregating)함으로써

로 정제될 수 있다. Thereafter, the generating device 800 sequentially generates a second probability distribution.

and a third probability distribution

can be estimated. At this time,

is described in Equation 2 below

by aggregating information from

can be purified with

Here, k ∈ {2, 3} may be.

의 업 스케일링 후, 업스케일링된 확률 분포

와 잔차 특징

을 합할 수 있다.

를 통한 업 스케일링 과정에서, 두 종류의 업 샘플링 프로세스가 수행될 수 있다. 이중 선형 보간은 공간 영역에서 수행되어 중간 특징이

과 동일한 공간 크기를 갖도록 할 수 있다. 이때, 특징의 채널 크기가

를 갖도록 채널 도메인에서 다른 보간이 수행될 수도 있다. The generating device 800 is, for example, a probability distribution through bilinear interpolation.

After upscaling of , the upscaled probability distribution

and residual features

can be summed

In the upscaling process through , two types of upsampling processes may be performed. Bilinear interpolation is performed in the spatial domain so that the intermediate features are

It can be made to have the same space size as . At this time, the channel size of the feature is

Another interpolation may be performed in the channel domain to have

를 잔차 특징

과 합산하기 전에 채널 방향의 모든 합산된 값이 1과 같도록

를 통해 업 스케일링된 확률 분포를 정규화될 수 있다. The generating device 800 is a probability distribution

Residual Features

Make sure that all summed values in the channel direction are equal to 1 before summing with

The upscaled probability distribution may be normalized through .

이 생성될 수 있다. A target depth image corresponding to the input image based on the finally estimated probability distribution among the probability distributions for each depth resolution

can be created.

는 깊이 영상

로 변환되고, 모든 깊이 해상도들에서의 정답 깊이들(ground-truth depths)과 비교될 수 있다. In this case, the probability distribution estimated at each depth resolution

is the depth image

, and can be compared with ground-truth depths at all depth resolutions.

The generating apparatus 800 according to an embodiment generates a more dense depth image by generating scale information of each stage, that is, features for different depth resolutions, in a multi-scale method. can do.

Also, the generating apparatus 800 according to an embodiment may reconstruct probability distributions for different depth resolutions based on features for different depth resolutions. More specifically, the module generating the depth resolution having a coarse value in the generating device concentrates on the global structure, and the modules generating the depth resolution having a dense value are high by generating the dense depth information. A target depth image having a resolution may be generated.

Also, in an embodiment, instead of correcting the depth value, the estimated probability distribution for each depth resolution is used for correction, so that the feature for each depth resolution may be more accurately corrected.

9 is a flowchart illustrating a method of generating a depth image according to another embodiment, and FIG. 10 is a diagram for explaining a configuration and operation of an apparatus for generating a depth image according to another embodiment.

9 and 10 , a process of generating a high-quality target depth image by enhancing a low-quality depth image by performing steps 910 to 970 by the generating apparatus according to an embodiment is illustrated. do.

In operation 910 , the generating device 1000 receives the depth image 1003 . The depth image 1003 may correspond to, for example, a low-quality depth image corresponding to a color image that is the input image 1001 received in operation 920 . The depth image 1003 may be sensed by a depth sensor that acquires a depth image representing depth information about an object, such as a kinect, a time-of-flight (TOF) depth camera, or an optical 3D scanner, for example. can The depth image 1003 is an image representing depth information, which is information about a depth (or distance) from a photographing position to an object.

In step 920 , the generating device receives an input image (eg, a color image) 1001 .

In operation 930 , the generating apparatus discretizes 1020 the depth image 1003 received in operation 910 , and generates a probability distribution P ₁ of the first depth resolution.

In operation 940 , the generating device extracts a feature corresponding to the color image 1001 received in operation 920 . The generating apparatus may extract a feature corresponding to the color image 1001 by, for example, the encoder 1010 .

In step 950 , the generating device generates a feature for each second depth resolution by decoding the feature extracted in step 940 using at least one decoder 1030 , 1050 corresponding to the at least one second depth resolution. do. Here, 'at least one second depth resolution' may be understood to include all of the depth resolutions having a denser depth value than the second depth resolution, such as the third depth resolution and the fourth depth resolution, in addition to the second depth resolution. can The generating apparatus may generate a residual feature of the second depth resolution using, for example, the second decoder 1030 corresponding to the second depth resolution. The generating apparatus may generate the residual feature of the third depth resolution by using the third decoder 1050 corresponding to the third depth resolution that is denser than the second depth resolution.

In operation 960 , the generating device estimates a probability distribution for each depth resolution by gradually correcting the feature for each depth resolution generated in operation 950 . _{The generating apparatus corrects 1040 the first probability distribution P 1} by the residual feature of the second depth resolution generated by the second decoder 1030 , thereby providing a second probability corresponding to depth ranges of the second depth resolution. The distribution (P ₂ ) can be estimated. _{The generating apparatus corrects ( 1060 ) the second probability distribution P 2} by the residual feature of the third depth resolution generated by the third decoder 1050 , thereby providing a third probability corresponding to the depth ranges of the third depth resolution. The distribution (P ₃ ) can be estimated.

In operation 970 , the generating device generates a target depth image corresponding to the input image 1001 based on a finally estimated probability distribution (eg, a third probability distribution P _{3 ) among probability distributions for each depth resolution.} (1080). _{The generating apparatus may generate the target depth image 1080 by converting the third probability distribution P 3} , which is the finally estimated discrete probability distribution, into a continuous value ( 1070 ). The target depth image 1080 may be used for object recognition such as 3D face recognition or photo effect processing such as defocusing. The target depth image 1080 may help improve performance of visual object recognition by determining a geometric relation between objects or providing 3D geometric information.

The generating device may output the target depth image generated in operation 970 .

11 is a block diagram of an apparatus for generating a depth image according to an exemplary embodiment. Referring to FIG. 11 , an apparatus (“generating device”) 1100 for generating a depth image according to an exemplary embodiment includes a communication interface 1110 , a processor 1130 , and a memory 1150 . The communication interface 1110 , the processor 1130 , and the memory 1150 may be connected to each other through the communication bus 1105 .

The communication interface 1110 receives an input image. Also, the communication interface 1110 may output the target depth image generated by the processor 1130 .

The processor 1130 extracts a feature corresponding to the input image. The processor 1130 generates a feature for each depth resolution by decoding the feature using decoders corresponding to different depth resolutions. The processor 1130 estimates a probability distribution for each depth resolution by gradually correcting a feature for each depth resolution. The processor 1130 generates a target depth image corresponding to the input image based on a finally estimated probability distribution among probability distributions for each depth resolution.

The memory 1150 may store an input image. Also, the memory 1150 may store the target depth image generated by the processor 1130 .

Also, the memory 1150 may store various pieces of information generated in the process of the processor 1130 described above. In addition, the memory 1150 may store various data and programs. The memory 1150 may include a volatile memory or a non-volatile memory. The memory 1150 may include a mass storage medium such as a hard disk to store various data.

According to an embodiment, the generating device 1100 may further include sensors such as an image sensor sensing or capturing an input image, a camera, a depth sensor sensing a depth value, and/or a depth camera. Alternatively, the generating device 1100 may further include a display device that outputs a target depth image corresponding to the input image.

Also, the processor 1130 may perform at least one method described above with reference to FIGS. 1 to 10 or an algorithm corresponding to the at least one method. The processor 1130 may be a hardware-implemented data processing device having a circuit having a physical structure for executing desired operations. For example, desired operations may include code or instructions included in a program. For example, a data processing device implemented as hardware includes a microprocessor, a central processing unit, a processor core, a multi-core processor, and a multiprocessor. , an Application-Specific Integrated Circuit (ASIC), and a Field Programmable Gate Array (FPGA).

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and carry out program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

The software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

As described above, although the embodiments have been described with reference to the limited drawings, those skilled in the art may apply various technical modifications and variations based on the above. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

1100: generating device
1105: communication bus
1110: communication interface
1130: Processor
1150: memory

Claims (35)

receiving an input image;
extracting a feature corresponding to the input image;
generating a feature for each depth resolution by decoding the feature using decoders corresponding to different depth resolutions;
estimating a probability distribution for each depth resolution by progressively refining the feature for each depth resolution; and
generating a target depth image corresponding to the input image based on a finally estimated probability distribution among the probability distributions for each depth resolution;
containing,
How to create a depth image. According to claim 1,
The step of generating the feature for each depth resolution includes:
generating a feature of the first depth resolution by using a first decoder corresponding to a first depth resolution among the depth resolutions; and
generating a residual feature of the second depth resolution by using a second decoder corresponding to a second depth resolution among the depth resolutions;
containing,
How to create a depth image. According to claim 1,
The step of generating the feature for each depth resolution includes:
generating a residual feature of the third depth resolution by using a third decoder corresponding to a third depth resolution among the depth resolutions;
further comprising,
How to create a depth image. According to claim 1,
The step of generating the feature for each depth resolution includes:
generating a feature for each depth resolution by decoding the feature at a uniformly set depth interval; and
Generating the feature for each depth resolution by decoding the feature with a depth interval set differently based on a spacing-increasing discretization (SID) technique
including any one of
How to create a depth image. According to claim 1,
The step of estimating the probability distribution for each depth resolution comprises:
estimating a first probability distribution corresponding to depth ranges of the first depth resolution based on a characteristic of the first depth resolution among the depth resolutions; and
estimating a second probability distribution corresponding to depth ranges of a second depth resolution based on the first probability distribution and a residual feature of a second depth resolution among the depth resolutions;
containing,
How to create a depth image. 6. The method of claim 5,
The step of estimating the second probability distribution comprises:
estimating the second probability distribution corresponding to depth ranges of the second depth resolution by correcting the first probability distribution by the residual feature of the second depth resolution;
containing,
How to create a depth image. 7. The method of claim 6,
The step of estimating the second probability distribution comprises:
up-scaling the first probability distribution; and
estimating the second probability distribution by correcting the up-scaled first probability distribution by a residual feature of the second depth resolution;
containing,
How to create a depth image. 6. The method of claim 5,
The step of estimating the probability distribution for each depth resolution comprises:
estimating a third probability distribution corresponding to depth ranges of a third depth resolution based on the second probability distribution and a residual feature of a third depth resolution among the depth resolutions;
further comprising,
How to create a depth image. 9. The method of claim 8,
The step of estimating the third probability distribution comprises:
estimating the third probability distribution corresponding to depth ranges of the third depth resolution by correcting the second probability distribution by the residual feature of the third depth resolution;
containing,
How to create a depth image. 9. The method of claim 8,
The step of estimating the third probability distribution comprises:
up-scaling the second probability distribution; and
estimating the third probability distribution by correcting the up-scaled second probability distribution by a residual feature of the third depth resolution;
containing,
How to create a depth image. According to claim 1,
The step of generating the target depth image includes:
converting the finally estimated probability distribution into the target depth image
containing,
How to create a depth image. According to claim 1,
The step of generating the target depth image includes:
calculating an expected value of the finally estimated probability distribution;
estimating a correction value of the expected value based on the finally estimated probability distribution; and
generating a target depth image based on the expected value and the correction value;
containing,
How to create a depth image. According to claim 1,
The depth resolutions are
at least two of a first depth resolution, a second depth resolution, and a third depth resolution;
The first depth resolution has a coarse value compared to the second depth resolution, the second depth resolution has a fine value compared to the first depth resolution, and the third depth resolution has a dense value compared to the second depth resolution,
How to create a depth image. According to claim 1,
dividing a depth range of depth values of pixels included in the input image into a plurality of sections by discretization;
further comprising,
How to create a depth image. According to claim 1,
outputting the target depth image
further comprising,
How to create a depth image. According to claim 1,
The input image is
comprising at least one of a single color image, an infrared image, and a depth image;
How to create a depth image. receiving a depth image;
receiving a color image;
generating a probability distribution of a first depth resolution by discretizing the depth image;
extracting a feature corresponding to the color image;
generating a feature for each second depth resolution by decoding the feature using at least one decoder corresponding to at least one second depth resolution;
estimating a probability distribution for each depth resolution by gradually correcting the feature for each depth resolution; and
generating a target depth image corresponding to the input image based on a finally estimated probability distribution among the probability distributions for each depth resolution;
containing,
How to create a depth image. A computer program stored on a medium for executing the method of any one of claims 1 to 17 in combination with hardware. a communication interface for receiving an input image; and
By extracting a feature corresponding to the input image, decoding the feature using decoders corresponding to different depth resolutions, generating a feature for each depth resolution, and gradually correcting the feature for each depth resolution, depth resolution A processor for estimating a star probability distribution and generating a target depth image corresponding to the input image based on a finally estimated probability distribution among the probability distributions for each depth resolution
containing,
A device that generates a depth image. 20. The method of claim 19,
the processor
generating a feature of the first depth resolution by using a first decoder corresponding to a first depth resolution among the depth resolutions;
generating a residual feature of the second depth resolution by using a second decoder corresponding to a second depth resolution among the depth resolutions;
A device that generates a depth image. 20. The method of claim 19,
the processor
generating a residual feature of the third depth resolution by using a third decoder corresponding to a third depth resolution among the depth resolutions;
A device that generates a depth image. 20. The method of claim 19,
the processor
The feature for each depth resolution is generated by decoding the feature at a uniformly set depth interval, or the feature for each depth resolution is generated by decoding the feature by a depth interval set differently based on an increased spacing discretization (SID) technique. doing,
A device that generates a depth image. 20. The method of claim 19,
the processor
estimating a first probability distribution corresponding to depth ranges of the first depth resolution based on a characteristic of a first depth resolution among the depth resolutions, and a second depth resolution of the first probability distribution and the depth resolutions estimating a second probability distribution corresponding to depth ranges of the second depth resolution based on the residual feature of
A device that generates a depth image. 24. The method of claim 23,
the processor
estimating the second probability distribution corresponding to depth ranges of the second depth resolution by correcting the first probability distribution by a residual feature of the second depth resolution;
A device that generates a depth image. 25. The method of claim 24,
the processor
up-scaling the first probability distribution and estimating the second probability distribution by correcting the up-scaled first probability distribution by a residual feature of the second depth resolution;
A device that generates a depth image. 24. The method of claim 23,
the processor
estimating a third probability distribution corresponding to depth ranges of a third depth resolution based on the second probability distribution and a residual feature of a third depth resolution among the depth resolutions;
A device that generates a depth image. 27. The method of claim 26,
the processor
estimating the third probability distribution corresponding to depth ranges of the third depth resolution by correcting the second probability distribution by a residual feature of the third depth resolution;
A device that generates a depth image. 28. The method of claim 27,
the processor
up-scaling the second probability distribution and estimating the third probability distribution by correcting the up-scaled second probability distribution by a residual feature of the third depth resolution;
A device that generates a depth image. 20. The method of claim 19,
the processor
converting the finally estimated probability distribution into the target depth image,
A device that generates a depth image. 20. The method of claim 19,
the processor
calculating an expected value of the finally estimated probability distribution, estimating a correction value of the expected value based on the finally estimated probability distribution, and generating a target depth image based on the expected value and the correction value generated,
A device that generates a depth image. 20. The method of claim 19,
The depth resolutions are
at least two of a first depth resolution, a second depth resolution, and a third depth resolution;
The first depth resolution has a coarse value compared to the second depth resolution, the second depth resolution has a coarse value compared to the first depth resolution, and the third depth resolution has a coarse value compared to the second depth resolution. having a dense value,
A device that generates a depth image. 20. The method of claim 19,
the processor
dividing a depth range of depth values of pixels included in the input image into a plurality of sections,
A device that generates a depth image. 20. The method of claim 19,
The communication interface is
outputting the target depth image,
A device that generates a depth image. 20. The method of claim 19,
The input image is
comprising at least one of a single color image, an infrared image, and a depth image;
A device that generates a depth image. 20. The method of claim 19,
The apparatus for generating the depth image
Smartphone, Smart-TV, Tablet, Head-up Display (HUD), 3D Digital Information Display (DID), 3D Mobile Device, and Smart Automobile comprising at least one of
A device that generates a depth image.

Images