JP2021174406A - Depth map super-resolution device, depth map super-resolution method, and depth map super-resolution program - Google Patents

Abstract

To obtain an appropriate super-resolution depth map having discrete boundaries in a simplified processing.SOLUTION: A tensor derivation part 103 derives, based on boundary positions and boundary directions as to a depth map in which each pixel has a pixel value correspondingly to a depth value of an object, a diffusion tensor for causing the pixel values at the boundary positions to diffuse correspondingly to the boundary directions. A smoothing part 104 generates, based on the depth map and the diffusion tensor, a super-resolution depth map obtained from applying super-resolution to the depth map.SELECTED DRAWING: Figure 4

Description

The disclosed technique relates to a depth map super-resolution device, a depth map super-resolution method, and a depth map super-resolution program.

Non-Patent Document 1 discloses that a super-resolution depth map is obtained by weighting and adding the observed depth values using weights calculated from the similarity between pixel positions and values. ..

Non-Patent Document 2 discloses that a super-resolution depth map that minimizes energy consisting of a smoothing term and a data term that evaluates whether the measured data can be reproduced is derived. Has been done. By using a smoothing term that takes into account the anisotropic diffusion tensor that reduces the effect of the smoothing term where the gradient of the image is large, we aim to change the depth discontinuously at the object boundary. There is.

Non-Patent Document 3 discloses that super-resolution of a depth map is realized by learning a neural network so as to estimate a dense depth map from an image and a sparse depth map. Since the method of Non-Patent Document 3 does not perform interpolation or smoothing, it is possible to output a super-resolution depth map that is discontinuous at the occlusion boundary.

Kopf, Johannes, et al. "Joint bilateral upsampling." ACM Transactions on Graphics (ToG). Vol. 26. No. 3. ACM, 2007. Ferstl, David, et al. "Image guided depth upsampling using anisotropic total generalized variation." Proceedings of the IEEE International Conference on Computer Vision. 2013. Chen, Yun, et al. "Learning Joint 2D-3D Representations for Depth Completion." Proceedings of the IEEE International Conference on Computer Vision. 2019.

The problem with the method of Non-Patent Document 1 is that the depth between the depth of the object in the foreground and the depth of the object in the background is taken near the occlusion boundary, so that the depth changes continuously near the occlusion boundary. It is to obtain an unnatural depth map.

The problem with the method of Non-Patent Document 2 is that each element of the anisotropic diffusion tensor controls the degree of smoothing, but the smoothing cannot be completely nullified. As a result, the depth is near the occlusion boundary. Is to obtain an unnatural depth map in which is continuously changing.

The problem with the method of Non-Patent Document 3 is that it is necessary to learn the neural network, and it is necessary to collect and learn the data in advance in the same scene as the environment to be applied. Such constraints impair the applicability of the method.

As described above, the methods of Non-Patent Documents 1 to 3 described above "as a result of smoothing and interpolation, the depth changes continuously at the occlusion boundary, so that an unnatural depth map can be obtained." Or, the problem is that "you must have learned in advance in the same scene as the application destination".

The disclosed technology has been made in view of the above points, and is a depth map super-resolution device, a depth map super-resolution device, which can obtain an appropriate super-resolution depth map having discrete boundaries by a simple process. It is an object of the present invention to provide a resolution method and a depth map super-resolution program.

The first aspect of the present disclosure is a depth map super-resolution device, in which each pixel has a pixel value corresponding to the depth value of an object in the boundary direction based on the boundary position and the boundary direction of the depth map. A tensor derivation unit that derives a diffusion tensor for diffusing pixel values at the boundary position, and a super-resolution depth map obtained by super-resolution of the depth map based on the depth map and the diffusion tensor. Includes a smoothing section to be generated.

The second aspect of the present disclosure is the depth map super-resolution method, in which the tensor derivation unit is based on the boundary position and the boundary direction of the depth map in which each pixel has a pixel value corresponding to the depth value of the object. , A diffusion tensor for diffusing the pixel values at the boundary position was derived according to the boundary direction, and the smoothing unit made the depth map super-resolution based on the depth map and the diffusion tensor. Includes generating a super-resolution depth map.

The third aspect of the present disclosure is a depth map super-resolution program, which is a program for causing a computer to function as the depth map super-resolution device of the first aspect.

According to the disclosed technique, an appropriate super-resolution depth map having discrete boundaries can be obtained by a simple process.

It is a schematic block diagram of an example of a computer functioning as a depth map super-resolution apparatus of this embodiment. It is a figure which shows an example of the input sparse depth map. It is a figure which shows an example of the input image. It is a block diagram which shows the structure of the depth map super-resolution apparatus of this embodiment. It is a figure which shows an example of the occlusion boundary. It is a figure which shows an example of the complementary depth map. It is a figure which shows an example of a super-resolution depth map. It is a flowchart which shows the depth map super-resolution processing routine of the depth map super-resolution apparatus of this embodiment.

Hereinafter, an example of an embodiment of the disclosed technology will be described with reference to the drawings. The same reference numerals are given to the same or equivalent components and parts in each drawing. In addition, the dimensional ratios in the drawings are exaggerated for convenience of explanation and may differ from the actual ratios.

<Outline of this embodiment>
In the present embodiment, a sparse depth map obtained by measurement with a LiDAR (Light Detection and Langing) sensor is upsampled using an image, and then discrete using a diffusion tensor derived using an occlusion boundary. An appropriate super-resolution depth map is obtained by smoothing the non-occlusion boundary while having a proper occlusion boundary.

<Configuration of depth map super-resolution device according to this embodiment>
FIG. 1 is a block diagram showing a hardware configuration of the depth map super-resolution device 10 of the present embodiment.

As shown in FIG. 1, the depth map super-resolution device 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, and a display unit 16. And has a communication interface (I / F) 17. The configurations are connected to each other via a bus 19 so as to be communicable with each other.

The CPU 11 is a central arithmetic processing unit that executes various programs and controls each part. That is, the CPU 11 reads the program from the ROM 12 or the storage 14, and executes the program using the RAM 13 as a work area. The CPU 11 controls each of the above configurations and performs various arithmetic processes according to the program stored in the ROM 12 or the storage 14. In the present embodiment, the ROM 12 or the storage 14 stores a depth map super-resolution program for learning a neural network. The depth map super-resolution program may be one program or a group of programs composed of a plurality of programs or modules.

The ROM 12 stores various programs and various data. The RAM 13 temporarily stores a program or data as a work area. The storage 14 is composed of an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and stores various programs including an operating system and various data.

The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used to perform various inputs including a sparse depth map and an image obtained by capturing an area corresponding to the depth map. For example, a sparse depth map as shown in FIG. 2 and an image of a region corresponding to the depth map as shown in FIG. 3 are input. FIG. 2 shows an example in which the sparse depth map displays only the pixels for which the depth value is obtained by the measurement by the LiDAR sensor in black. FIG. 3 shows an example of a color image or a grayscale image obtained by capturing a region corresponding to a sparse depth map.

Specifically, in the input image, each pixel taken by the camera has a pixel value represented by RGB or gray scale. Further, the sparse depth map is an array having a depth value of the same size as the image, which is acquired by projecting the measurement information measured by the LiDAR sensor onto the image. The value is zero for pixels for which the depth value has not been measured.

Here, it is assumed that the camera that captures the image and the LiDAR sensor that acquires the depth map are calibrated in advance, and the depth map is accurately projected on the image.

The display unit 16 is, for example, a liquid crystal display, and displays various information including a super-resolution depth map in which a sparse depth map is super-resolution. The display unit 16 may adopt a touch panel method and function as an input unit 15. Here, the super-resolution depth map is an array having a depth value of the same size as the image as a value, and has a non-zero value in all pixels.

In a sparse depth map, pixels whose depth value has not been measured may not have a value. In this case, in a super-resolution depth map, it is sparse that all pixels have some value. Different from depth map.

The communication interface 17 is an interface for communicating with other devices, and for example, standards such as Ethernet (registered trademark), FDDI, and Wi-Fi (registered trademark) are used.

Next, the functional configuration of the depth map super-resolution device 10 will be described. FIG. 4 is a block diagram showing an example of the functional configuration of the depth map super-resolution device 10.

The depth map super-resolution device 10 functionally includes an upsampling unit 101, a boundary derivation unit 102, a tensor derivation unit 103, and a smoothing unit 104, as shown in FIG.

The upsampling unit 101 uses an image representing a region corresponding to the sparse depth map received by the input unit 15 to obtain pixel values of pixels of the sparse depth map that do not have pixel values corresponding to the depth value. Generate a complementary depth map (see FIG. 6). FIG. 6 shows an example in which all the pixels of the complementary depth map display the values corresponding to the depth values in shades.

For example, a technique called Joint Bilateral Nearest Neighbor is used to complement the pixel values of pixels that do not have pixel values corresponding to the depth value. Specifically, as shown in the following equation (1), the depth value d'(x) of the complementary depth map at the position x is the depth value at the distance obtained by integrating the coordinate distance and the pixel value distance on the image plane. Takes the depth value of the nearest pixel among the pixels with.

（１）

(1)

In the equation (1), a and b are predetermined positive constants, and I (x) is a pixel value of the image at the position x. The color space used for I (x) is not limited, and is arbitrarily determined by the user. For example, any color space such as grayscale, RGB, XYZ, CIELAB, HSV, etc. is used.

The boundary derivation unit 102 derives an occlusion boundary representing the boundary position and the boundary direction from the complementary depth map output from the upsampling unit 101.

Here, the occlusion boundary is an array of the same size as the image as shown in FIG. 5, and the pixel values are four values (“not the boundary position”, “horizontal boundary position”, and “vertical direction”. Take either "boundary position" or "horizontal and vertical boundary position"). In FIG. 5, pixels whose occlusion boundary is not the boundary position are represented by white, the horizontal boundary is represented by a thick solid line, the vertical boundary is represented by a thin solid line, and the horizontal and vertical boundary is represented by a broken line. The example shown in is shown.

For example, with the complementary depth map as an input, the following determination formulas 1 and 2 are determined for each pixel x = (x_1, x_2) using a predetermined threshold value v.

(Judgment formula 1) | d (x_1 + 1, x_2) -d (x_1, x_2) |> v
(Judgment formula 2) | d (x_1, x_2 + 1) -d (x_1, x_2) |> v

Then, the occlusion boundary is created in each pixel x according to the following branches 1 to 4.

If branch 1: (determination formula 1) is false and (determination formula 2) is false, the pixel x at the occlusion boundary is a value indicating that it is not the boundary position.

Branch 2: If (judgment formula 1) is true and (judgment formula 2) is false, the pixel x at the occlusion boundary is a value indicating that the boundary position is in the horizontal direction.

Branch 3: If (judgment formula 1) is false and (judgment formula 2) is true, the pixel x at the occlusion boundary is a value indicating that the boundary position is in the vertical direction.

Branch 4: If (judgment formula 1) is true and (judgment formula 2) is true, the pixel x at the occlusion boundary is a value indicating that the boundary position is in the horizontal direction and the vertical direction.

The tensor derivation unit 103 diffuses the pixel values at the boundary position according to the boundary direction based on the occlusion boundary representing the boundary position and the boundary direction for the complementary depth map derived by the boundary derivation unit 102. Derivation of the tensor.

Specifically, the tensor derivation unit 103 is an array for spreading the pixel values in the horizontal and vertical directions when it is not the boundary position for each pixel of the complementary depth map, and the pixel value when it is the boundary position in the horizontal direction. An array for spreading the pixels in the vertical direction, an array for spreading the pixel values in the horizontal direction when the boundary position is in the vertical direction, and not spreading the pixel values when the boundary position is in the horizontal direction and the vertical direction. A diffusion tensor having any of the sequences for is derived.

For example, the diffusion tensor is an array of image size × 2 × 2, and the values of the array take real values. Assuming that the image size is vertical H and horizontal W, the diffusion tensor has a 2 × 2 array for each of the H × W pixels as an element. The diffusion tensor is represented by T, an integer of 0 ≦ h <H, 0 ≦ w <W, and the value of the diffusion tensor of the pixel (h, w) is represented by T (h, w). The occlusion boundary derived by the boundary derivation unit 102 is represented by B, and the value of the pixel (h, w) is represented by B (h, w). As described above, T (h, w) is a 2 × 2 array. At this time, the diffusion tensor is derived according to the following.

First, when B (h, w) is a value indicating that it is not a boundary position, T (h, w) is derived according to the following equation.

Further, when B (h, w) is a value indicating that the boundary position is in the horizontal direction, T (h, w) is derived according to the following equation.

Further, when B (h, w) is a value indicating that the boundary position is in the vertical direction, T (h, w) is derived according to the following equation.

Further, when B (h, w) is a value indicating that the boundary position is in the horizontal direction and the vertical direction, T (h, w) is derived according to the following equation.

The smoothing unit 104 generates a super-resolution depth map in which the complementary depth map is super-resolution based on the complementary depth map and the diffusion tensor (see FIG. 7). FIG. 7 shows an example of a super-resolution depth map in which the complementary depth map shown in FIG. 6 is super-resolutiond.

Specifically, the smoothing unit 104 minimizes the energy function expressed by using the diffusion tensor, the super-resolution depth map differentiated on the image plane, and the difference between the complementary depth map and the super-resolution depth map. Generate a super-resolution depth map so that it becomes.

For example, a super-resolution depth map is generated by Huber norm smoothing that minimizes the energy function represented by the following equation (2).

（２）

(2)

However, the super-resolution depth map is represented by U, the complementary depth map is represented by D, and the diffusion tensor is represented by T, and the values at each pixel (h, w) are U (h, w), D (h, w), T ( It is expressed as h, w). That is, the energy function of the above equation (2) is expressed using the product of the diffusion tensor and the super-resolution depth map differentiated on the image plane, and the difference between the complementary depth map and the super-resolution depth map. There is.

Also, || || _ε is the Huber norm, and ∇ means to take the derivative in the image plane. That is, ∇U (h, w) is a two-dimensional vector. λ _u and λ _d are weights that determine the contribution of each term to energy and are positive real numbers. W (h, w) is an array of the size of H × W consisting of positive real numbers to be introduced when weighting is performed for each element. If no weighting is applied, all elements may be set to 1.

The U that minimizes the energy of the above equation (2) can be obtained by the updated equations of equations (3) to (5) by introducing the dual variable P by the First Order Primal Dual algorithm. The initial value of U is D. The initial value of P is arbitrary, but all elements may be zero.

（３）

（４）

（５）

(3)

(4)

(5)

である。τ_Ｐ、τ_ＵはそれぞれＰ、Ｕの更新のための正定数である。ｄｉｖは、発散を表す演算子である。収束するまで、上記式（３）〜式（５）の更新式を繰り返し適用し、最終的なＵ_ｎを超解像デプスマップとする。 here,

Is. τ _P and τ _U are positive constants for updating P and U, respectively. div is an operator that represents divergence. Until convergence, the equation (3) is applied repeatedly updating expressions to (5), the final U _n and super-resolution depth map.

<Operation of depth map super-resolution device according to this embodiment>
Next, the operation of the depth map super-resolution device 10 will be described.

FIG. 8 is a flowchart showing the flow of the depth map super-resolution processing by the depth map super-resolution device 10. The depth map super-resolution processing is performed by the CPU 11 reading the depth map super-resolution program from the ROM 12 or the storage 14, deploying the program in the RAM 13 and executing the program. Further, a sparse depth map and an image of a region corresponding to the depth map are input to the depth map super-resolution device 10.

In step S100, the CPU 11 uses the image representing the area corresponding to the sparse depth map received by the input unit 15 as the upsampling unit 101 to set the pixel value corresponding to the depth value in the sparse depth map. Generates a complementary depth map that complements the pixel values of pixels that do not have it.

In step S102, the CPU 11 derives an occlusion boundary representing the boundary position and the boundary direction from the complementary depth map output from the upsampling unit 101 as the boundary derivation unit 102.

In step S104, the CPU 11, as the tensor derivation unit 103, determines the pixels at the boundary position according to the boundary direction based on the occlusion boundary representing the boundary position and the boundary direction for the complementary depth map derived by the boundary derivation unit 102. Derivation of a diffusion tensor to diffuse the value.

In step S106, the CPU 11 generates a super-resolution depth map in which the complementary depth map is super-resolution based on the complementary depth map and the diffusion tensor as the smoothing unit 104, displays it on the display unit 16, and displays the depth. End the map super-resolution processing routine.

As described above, in the depth map super-resolution device according to the present embodiment, the boundary direction is based on the boundary position and the boundary direction of the depth map in which each pixel has a pixel value corresponding to the depth value of the object. A diffusion tensor for diffusing the pixel value at the boundary position is derived according to the above. The depth map super-resolution device generates a super-resolution depth map in which the depth map is super-resolution based on the depth map and the diffusion tensor. As a result, an appropriate super-resolution depth map having discrete boundaries can be obtained by a simple process.

<Modification example>
The present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the gist of the present invention.

For example, the processing of the boundary derivation unit 102 may be omitted. In this case, the occlusion boundary in the depth map may be accepted by the input unit 15.

Further, the processing of the upsampling unit 101 may be omitted. In this case, the input unit 15 may receive a depth map in which each pixel has a value corresponding to the depth value.

Further, the upsampling unit 101 may generate a complementary depth map by using the minimum image gradient path method. Specifically, a complementary depth map is generated by infiltrating the depth value with the pixel having the depth value as the starting point. That is, when the depth value is infiltrated, the gradient on the image is accumulated. At this time, if the infiltration destination pixel has a value smaller than the cumulative value of the gradient on the image, the depth value is not updated. On the other hand, when the infiltration destination pixel has a value equal to or larger than the cumulative value of the gradient on the image, the depth value of the pixel as the starting point is used for updating. This is repeated until it converges. As a result, the pixels having no depth value are complemented by the depth value of the pixels in the vicinity where the cumulative value of the gradient on the image is the minimum value.

For example, as in the following pseudo code, the direction on the image may be taken into consideration when accumulating the gradients on the image.

In the 11th to 16th lines, the gradients on the image are accumulated in the horizontal direction, and when the infiltrated pixel has a value equal to or greater than the cumulative value of the gradient on the image, the depth value of the pixel that is the starting point. It is updated with. In the 17th to 22nd lines, the gradients on the image are accumulated in the vertical direction, and when the infiltration destination pixel has a value equal to or greater than the cumulative value of the gradient on the image, the depth value of the pixel that is the starting point. It is updated with.

Further, as in the following pseudo code, it is not necessary to consider the direction on the image when accumulating the gradients on the image.

In the 11th to 16th lines, the gradients on the image are accumulated in each of the horizontal direction and the vertical direction, and when the infiltrated pixel has a value equal to or more than the cumulative value of the gradient on the image, it becomes the starting point. It is updated with the depth value of the pixel.

Further, in the above pseudo code, n is a constant of zero or more, which is a cost for increasing the path length. Norm is an absolute norm || or a squared norm || || ² or any norm such as the Huber norm.

Further, the tensor derivation unit 103 may derive the diffusion tensor from the occlusion boundary and the image. Specifically, the anisotropic diffusion tensor Ta is derived by the same method as in Non-Patent Document 2, and the diffusion tensor (referred to as Tb) is derived by the same method as in the above embodiment. The diffusion tensor for each pixel (h, w) is derived by the following equation (6).

（６）

(6)

In the above equation (6), Tb (h, w) and Ta (h, w) are both 2 × 2 matrices, and T (h, w) is the product of the two matrices.

Further, the smoothing unit 104 may generate a super-resolution depth map by the Total Generalized Variation smoothing that minimizes the energy function represented by the following equation (7).

（７）

(7)

However, the complementary depth map is D, the diffusion tensor is T, and the super-resolution depth map is U. A new array V having a size of H × W × 2 is defined. Further, λ _U , λ _V, and λ _d are weights that determine the contribution of each term to energy, and are positive real numbers.

U and V that minimize the energy of the above equation (7) introduce the dual variables P and Q by the First Order Primary Dual algorithm, and the equations (8), (9), (10), and (11) are introduced. ), Equation (12), and the updated equation of equation (13). The initial value of U is D. The initial values of V, P, and Q are arbitrary, but all elements may be set to zero.

（８）

（９）

（１０）

（１１）

（１２）

（１３）

(8)

(9)

(10)

(11)

(12)

(13)

であり、

である。τ_Ｐ、τ_Ｑ、τ_Ｕ、τ_ＶはそれぞれＰ、Ｑ、Ｕ、Ｖの更新のための正定数である。収束するまで更新式を繰り返し適用し、最終的なＵ_ｎを超解像デプスマップとする。 here,

And

Is. τ _P , τ _Q , τ _U , and τ _V are positive constants for updating P, Q, U, and V, respectively. Repeatedly applying the update equation until convergence, the final U _n and super-resolution depth map.

Further, the smoothing unit 104 may generate a super-resolution depth map by the extended Total Generalized Variation smoothing that minimizes the energy function represented by the following equation (14).

（１４）

(14)

A new array V having a size of H × W × 2 is defined. Further, λ _U , λ _V, and λ _d are weights that determine the contribution of each term to energy, and are positive real numbers.

U and V that minimize the energy of the above equation (14) can be obtained by the update equations of the equations (15) to (20) by introducing the dual variables P and Q by the First Older Primary Dual algorithm. The initial value of U is D. The initial values of V, P, and Q are arbitrary, but all elements may be set to zero.

（１５）

（１６）

（１７）

（１８）

（１９）

（２０）

(15)

(16)

(17)

(18)

(19)

(20)

であり、

である。τ_Ｐ、τ_Ｑ、τ_Ｕ、τ_ＶはそれぞれＰ、Ｑ、Ｕ、Ｖの更新のための正定数である。収束するまで更新式を繰り返し適用し、最終的なＵｎを超解像デプスマップとする。 here,

And

Is. τ _P , τ _Q , τ _U , and τ _V are positive constants for updating P, Q, U, and V, respectively. The update formula is repeatedly applied until it converges, and the final Un is used as a super-resolution depth map.

Further, the sparse depth map may have the reciprocal of the depth (inverse depth) as the value instead of the depth value, and the reciprocal of the depth may be set to zero where the depth is not measured. In that case, the output complementary depth map and super-resolution depth map may also have the reciprocal of the depth as the value.

Further, various processors other than the CPU may execute various processes in which the CPU reads the software (program) and executes the software (program) in each of the above embodiments. In this case, the processor includes a PLD (Programmable Logic Device) whose circuit configuration can be changed after manufacturing an FPGA (Field-Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), and the like. An example is a dedicated electric circuit or the like, which is a processor having a circuit configuration designed exclusively for the purpose. In addition, the depth map super-resolution processing may be executed by one of these various processors, or a combination of two or more processors of the same type or different types (for example, a plurality of FPGAs, and a CPU and an FPGA). It may be executed in combination with). Further, the hardware structure of these various processors is, more specifically, an electric circuit in which circuit elements such as semiconductor elements are combined.

Further, in each of the above embodiments, the mode in which the depth map super-resolution program is stored (installed) in the storage 14 in advance has been described, but the present invention is not limited to this. The program is stored in a non-temporary medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versaille Disk Online Memory), and a USB (Universal Serial Bus) memory. It may be provided in the form. Further, the program may be downloaded from an external device via a network.

Regarding the above embodiments, the following additional notes will be further disclosed.

(Appendix 1)
Depth map super-resolution device
Memory and
With at least one processor connected to the memory
Including
The processor
Based on the boundary position and boundary direction of the depth map in which each pixel has a pixel value corresponding to the depth value of the object, a diffusion tensor for diffusing the pixel value at the boundary position according to the boundary direction is derived. death,
A depth map super-resolution device configured to generate a super-resolution depth map in which the depth map is super-resolution based on the depth map and the diffusion tensor.

(Appendix 2)
A non-temporary storage medium that stores a program that can be executed by a computer to perform depth map super-resolution processing.
The depth map super-resolution processing is
Based on the boundary position and boundary direction of the depth map in which each pixel has a pixel value corresponding to the depth value of the object, a diffusion tensor for diffusing the pixel value at the boundary position according to the boundary direction is derived. death,
A non-temporary storage medium that generates a super-resolution depth map in which the depth map is super-resolution based on the depth map and the diffusion tensor.

10 Depth map super-resolution device 15 Input unit 16 Display unit 101 Upsampling unit 102 Boundary derivation unit 103 Tensor derivation unit 104 Smoothing unit

Claims (7)

Based on the boundary position and boundary direction of the depth map in which each pixel has a pixel value corresponding to the depth value of the object, a diffusion tensor for diffusing the pixel value at the boundary position according to the boundary direction is derived. Tensor derivation part and
A smoothing unit that generates a super-resolution depth map in which the depth map is super-resolution based on the depth map and the diffusion tensor.
Depth map super-resolution device with. The tensor derivation unit arranges each pixel of the depth map to spread the pixel values in the horizontal direction and the vertical direction when it is not the boundary position, and sets the pixel value in the vertical direction when it is the boundary position in the horizontal direction. An array for spreading, an array for spreading pixel values horizontally when the boundary position is in the vertical direction, and an array for not spreading pixel values when the boundary position is in the horizontal and vertical directions. The depth map super-resolution device according to claim 1, wherein the diffusion tensor having any one of them as an element is derived. A boundary derivation unit for deriving the boundary position and the boundary direction from the depth map is further included.
The depth map super-resolution device according to claim 1 or 2, wherein the tensor derivation unit derives the diffusion tensor based on the derivation result of the boundary position and the boundary direction. An upsampling unit that generates a complementary depth map that complements the pixel values of pixels that do not have pixel values corresponding to the depth value in the depth map by using an image representing a region corresponding to the depth map is further included. ,
The tensor derivation unit derives the diffusion tensor based on the boundary position and the boundary direction with respect to the complementary depth map.
The smoothing section according to any one of claims 1 to 3, wherein the smoothing unit generates the super-resolution depth map in which the complementary depth map is super-resolution based on the complementary depth map and the diffusion tensor. Depth map super-resolution device. The smoothing unit minimizes the energy function expressed by using the diffusion tensor, the super-resolution depth map differentiated in the image plane, and the difference between the depth map and the super-resolution depth map. The depth map super-resolution apparatus according to any one of claims 1 to 4, wherein the super-resolution depth map is generated. The tensor derivation unit diffuses the pixel value at the boundary position according to the boundary direction based on the boundary position and the boundary direction for the depth map in which each pixel has the pixel value corresponding to the depth value of the object. Derivation of the diffusion tensor of
A depth map super-resolution method in which a smoothing unit generates a super-resolution depth map in which the depth map is super-resolution based on the depth map and the diffusion tensor. A depth map super-resolution program for operating a computer as the depth map super-resolution device according to any one of claims 1 to 5.

Images