CN114758165A - Depth map up-sampling method and device based on hierarchical clustering and boundary enhancement - Google Patents

Abstract

The invention discloses a depth map upsampling method and device based on hierarchical clustering and boundary enhancement, and relates to the technical field of depth map upsampling. The invention not only preprocesses the low-resolution depth map and divides the low-resolution depth map into a depth continuous area and a depth discontinuous area, the seed pixels of the depth continuous area are directly interpolated, and the target pixels of the depth discontinuous area are needed to be processed by the invention, thereby reducing the target number and improving the calculation efficiency. The invention also utilizes an initial sketch map algorithm to preprocess the high-resolution color image, utilizes an anisotropic filter to obtain a smooth uniform area and an enhanced image boundary, and utilizes the boundary information of the sketch map to process for a depth discontinuous area with similar color, thereby being beneficial to subsequent fine image segmentation. When the depth map is subjected to upsampling, a hierarchical classification upsampling method is provided, the target pixels can be subjected to classification interpolation, and the depth aliasing artifact of a depth discontinuous region is effectively prevented.

Description

Depth map up-sampling method and device based on hierarchical clustering and boundary enhancement

Technical Field

The invention relates to the technical field of depth map upsampling, in particular to a depth map upsampling method and device based on hierarchical clustering and boundary enhancement.

Background

In recent years, in various three-dimensional applications, such as intelligent human-computer interaction, three-dimensional scene reconstruction, robot autonomous navigation, and the like, the acquisition of three-dimensional depth information plays an important role. However, due to the high manufacturing cost, the resolution of the depth sensor is slow to develop, far behind the demand. The laser scanning method is not suitable for real-time application and the equipment is very expensive, while the passive stereo vision method can neither eliminate occlusion nor solve the problem of no texture in the rendering process. A fused camera system consisting of a color camera and a depth sensor is widely used for depth acquisition due to its low cost. Typical depth cameras are Time-of-flight (tof) cameras and structured light cameras (such as Kinect), but the depth maps obtained by the cameras have the problems of low resolution, depth information loss, noise interference and the like, so that the application and development of the depth cameras are limited. The depth map upsampling method can reconstruct a high-quality depth map from a low-quality depth map acquired by a depth camera by utilizing algorithm processing under the condition that hardware cannot make breakthrough progress, and has high research value and practical significance. The existing up-sampling method usually reconstructs a high-resolution depth map by using a low-resolution depth map acquired by a depth camera and a high-resolution color image of the same scene. Because the depth upsampling can use the high-resolution color image after the registration as the auxiliary guide information, compared with the super-resolution reconstruction of the common image, the depth upsampling has higher sampling rate and more accurate upsampling depth.

Commonly used depth map upsampling methods are global optimization-based and filter-based methods. The method based on global optimization mainly comprises a method based on a Markov random field and a general generalized variation method, a high-resolution depth map is obtained by optimizing an energy function, the methods solve the ultra-smooth problem, error propagation is brought in the optimization process, sawtooth artifacts appear in a boundary region, and the method cannot be applied in real time due to complex iterative calculation.

The mainstream of the filtering-based method is the joint double-edge sampling JBU method proposed by Kopf et al (DOI: 10.1145/1239451.1239547), which interpolates low-resolution depth into a high-resolution grid by multiplication of two gaussian kernels (spatial kernel, which represents spatial distance in the depth map, and color kernel, which represents chromatic aberration in the optical color image. This method is very fast but creates critical artifacts in certain boundary regions, resulting in the generation of a high resolution depth map with blurred edges. Much research work has been directed around this approach. For example, y.s.ho et al (DOI: 10.1049/el.2013.3956) propose a Joint Bilateral Local Minimum (JBLM) filtering method, mainly proposing the idea of boundary processing, specifically, a depth map is divided into two parts: homogeneous regions and depth discontinuity regions. The JBU method is used in homogeneous regions and a local minimum filter is used in boundary regions, i.e., depth discontinuity regions. The method reduces the deep blending phenomenon of the boundary area on the premise of not increasing the calculation amount and the storage cost. Ming-Yu Liu et al (DOI: 10.1109/CVPR.2013.29) propose a classical joint geodetic sampling JGU method, giving a definition of the geodetic distance of the target pixel to the seed pixel. In a high resolution grid map, the seed pixels are known points in the low resolution depth map, and the target pixels are points whose depths are unknown and are to be interpolated. Firstly, geodesic distances from a current target to all seed pixels are calculated, a few seeds closest to the current target are found, and the values of the seeds are transferred to the current target by Gaussian filtering. As a global algorithm, it can find seeds closer to the current target pixel and use these seeds to further provide an accurate depth boundary to complete the interpolation. However, when the image color across the depth boundary is very rich, severe depth blending artifacts occur.

In 2020, the depth image upsampling method and system based on the depth edge point and the color image, which are disclosed in publication No. CN111489383A, first mark and correct unreliable points in the low resolution depth map according to different situations, and then perform bicubic interpolation to obtain an initialized upsampled depth map. And then, extracting edge points by using a Sobel operator to obtain a low-resolution depth edge map, mapping the low-resolution depth edge map to a high-resolution grid, classifying pixel points of the initialized up-sampling depth map based on structural consistency, mapping real edge pixel points to the edge-enhanced low-resolution depth map, and correcting a depth-reliable pixel region by setting an influence factor to obtain the high-resolution depth map. The method can effectively enhance the detail edge area and complete the reconstruction of the depth missing part. However, the processing process is complicated, the preprocessing process only aims at the low-resolution depth map, the efficiency is reduced, and the effect is reduced when the up-sampling factor is increased.

In 2021, military forces (DOI: 10.3788/IRLA20200081) used bicubic interpolation to initially up-sample the low resolution depth map and used it as the input image for guided filtering. Extracting a reliable edge region by utilizing multi-scale Canny edge detection, extracting a public edge region according to the edge similarity with a high-resolution gray image, classifying pixel points belonging to the public edge region in an initialized and up-sampled low-resolution depth image according to the edge position relation with the high-resolution gray image, weighting coefficients of a guiding filter again, and improving the quality of the up-sampled depth image. The method has high calculation efficiency, can provide reliable data for real-time application, but has poor reconstruction effect on complex texture information.

In 2021, a layered joint bilateral filtering depth map repairing algorithm based on depth confidence is proposed by a Wanqin (DOI: 10.3778/j.issn.1002-8331.1912-0463), depth confidence measurement is proposed on the basis of a depth degradation model, left-right consistency among single pixels is expanded into neighborhood calculation, a threshold value is set, and the pixels are divided into high confidence, low confidence and shielded pixels. And dynamically estimating the pixel weight based on the depth confidence, selecting the size of a corresponding filter, and performing depth restoration by using layered combined bilateral filtering to obtain a high-quality depth map. The layering of the method refers to redefining the weight of the combined bilateral filter according to the depth pixel classification, thereby simplifying the algorithm and improving the efficiency. However, this method has a poor effect of repairing a depth map having a similar color.

Disclosure of Invention

The invention aims to solve the technical problem of providing a depth map upsampling method and device based on hierarchical clustering and boundary enhancement.

In a first aspect, the present invention provides a depth map upsampling method based on hierarchical clustering and boundary enhancement, including: an image preprocessing process and a layered depth map upsampling process;

the image preprocessing process comprises:

preprocessing the low-resolution depth map, dividing the low-resolution depth map into a depth continuous region and a depth discontinuous region to obtain a preprocessed depth map D_I(ii) a Taking the pixels positioned in the depth continuous area as seed pixels for interpolation; taking the pixel positioned in the depth discontinuous area as a target pixel, and performing target pixel interpolation through a seed pixel;

processing the high-resolution color image by using an initial sketch model to obtain an original sketch and a sketch-capable image, and smoothing the high-resolution color image by using the sketch-capable image to realize rapid and effective clustering; adding the original sketch into the smooth high-resolution color image, enhancing the image boundary and obtaining a high-resolution gray scale image I_R；

The hierarchical depth map upsampling process comprises:

for the preprocessed depth map D_ICarrying out layered interpolation, wherein each layer firstly utilizes a k-means clustering method to carry out the high-resolution gray level image I_RIs divided into k _nClasses, each class comprising a plurality ofDiscontinuous areas, and interpolating the target pixel in each area by using a nearest neighbor joint bilateral NJB interpolation method, wherein k is_nThe number of categories of the n-th-layer cluster; and after the last layer of interpolation is finished, almost all target pixels are interpolated, and the residual target pixels which are not interpolated are deleted by using the post-smoothing operation.

Further, in the image preprocessing process, the low-resolution depth map is preprocessed, and the bilinear filtering and the threshold function are adopted for region division.

Further, in the image preprocessing process, the initial sketch model is specifically as follows:

p(I)＝p(I_Φ)p(Φ_sk)p(Φ_nsk)

wherein, p (phi)_sk) Representing a sketch part, p (phi)_nsk) Denotes the non-sketch part, p (I)_Φ) Representing the getalt field model.

Further, the category number k of the k-means cluster_nThe segmentation is gradually decreased layer by layer, and the segmentation from thin to thick of the image is realized.

Further, in the process of upsampling the hierarchical depth map, interpolating the target pixel for each region by using a nearest neighbor joint bilateral NJB interpolation method, specifically:

assuming that the current interpolation region contains T target pixels and S seed pixels, for each target pixel, firstly finding several nearest seeds around the target pixel, ensuring that the selected seeds are in the S seed pixels, and then interpolating the current target pixel by joint bilateral filtering by using the selected seeds:

Wherein L represents the number of selected seeds, g_d(t,s_l) And g_c(t,s_l) The gaussian kernels for spatial distance and chromatic aberration are respectively expressed as follows:

wherein d (t, s)_l) For the current target pixel t and the selected seed pixel s_lGeometric distance between, d_σAnd c_σGiven the scale parameters.

In a second aspect, the present invention provides a depth map upsampling apparatus based on hierarchical clustering and boundary enhancement, including: the device comprises an image preprocessing module and a layered depth map upsampling module;

the image preprocessing module is used for preprocessing the low-resolution depth map, dividing the low-resolution depth map into a depth continuous area and a depth discontinuous area and obtaining a preprocessed depth map D_I(ii) a Taking the pixels positioned in the depth continuous area as seed pixels for interpolation; taking the pixel positioned in the depth discontinuous area as a target pixel, and performing target pixel interpolation through a seed pixel; processing the high-resolution color image by using an initial sketch model to obtain an original sketch and a sketch-capable image, and smoothing the high-resolution color image by using the sketch-capable image to realize rapid and effective clustering; adding the original sketch into the smooth high-resolution color image, enhancing the image boundary and obtaining a high-resolution gray scale image I _R；

The layered depth map upsampling module is used for performing upsampling on the preprocessed depth map D_ICarrying out layered interpolation, wherein each layer firstly utilizes a k-means clustering method to carry out the high-resolution gray level image I_RIs divided into k_nClass, each class comprises a plurality of discontinuous areas, and each area is interpolated by a nearest neighbor joint bilateral NJB interpolation method to carry out interpolation on target pixels, wherein k_nThe number of the categories of the n-th layer of clusters; and after the last layer of interpolation is finished, almost all target pixels are interpolated, and the residual target pixels which are not interpolated are deleted by using the post-smoothing operation.

Further, in the image preprocessing module, the low-resolution depth map is preprocessed, and a bilinear filter and a threshold function are adopted for region division.

Further, in the image preprocessing module, the initial sketch model is specifically as follows:

p(I)＝p(I_Φ)p(Φ_sk)p(Φ_nsk)

wherein, p (phi)_sk) Representing a sketch part, p (phi)_nsk) Denotes the non-sketch part, p (I)_Φ) Representing the getalt field model.

Further, the category number k of the k-means cluster_nThe segmentation is gradually decreased layer by layer, and the segmentation from thin to thick of the image is realized.

Further, in the hierarchical depth map upsampling module, interpolating a target pixel for each region by using a nearest neighbor joint bilateral NJB interpolation method, specifically:

Assuming that the current interpolation region contains T target pixels and S seed pixels, for each target pixel, firstly finding several nearest seeds around the target pixel, ensuring that the selected seeds are in the S seed pixels, and then interpolating the current target pixel by joint bilateral filtering by using the selected seeds:

wherein L represents the number of selected seeds, g_d(t,s_l) And g_c(t,s_l) The gaussian kernels for spatial distance and chromatic aberration are respectively expressed as follows:

wherein d (t, s)_l) For the current target pixel t and the selected seed pixel s_lGeometric distance between, d_σAnd c_σGiven the scale parameters.

The embodiment of the invention at least has the following technical effects or advantages:

1. when image preprocessing is carried out, the method is different from the existing method, not only is the low-resolution depth image preprocessed, but also the bilinear filtering and the threshold function are utilized to carry out region division, the region division is carried out, the region division is divided into a depth continuous region and a depth discontinuous region (namely a boundary region), the seed pixels of the continuous region are directly interpolated, the target pixels of the discontinuous region need to be processed, the target number is reduced, and the calculation efficiency is improved. Meanwhile, the method also utilizes an initial sketch algorithm to preprocess the high-resolution color image, divides the high-resolution color image into a sketch image and a Prime image, utilizes an anisotropic filter to obtain a smooth uniform area and an enhanced image boundary, and utilizes the boundary information of the sketch image to process for depth discontinuous areas with similar colors, thereby being beneficial to subsequent fine image segmentation.

2. When the depth map is subjected to upsampling, the patent provides an upsampling method for hierarchical classification, an image is divided into a plurality of unconnected areas, and the target pixels are interpolated by using seed pixels in the same area. The method comprises the steps of firstly, carrying out hierarchical classification on a high-resolution color image by using a k-means clustering algorithm, wherein the first layer needs to divide the image into more categories, all depth discontinuous points are ensured to be divided into different areas, and the number of the categories is gradually decreased layer by layer. Because the high-resolution color image is smoothed in the homogeneous region and the boundary region is enhanced in the preprocessing process, the target pixels can be classified and interpolated, and the depth confusion artifact of the depth discontinuous region is effectively prevented.

3. When the target pixels are interpolated, a nearest neighbor combined bilateral interpolation method is provided, for each target pixel, several nearest neighbor seed pixels are selected to carry out combined bilateral filtering interpolation, the process of selecting the seed pixels is to search by taking the target pixels as the center, and the size of the maximum search radius is reduced along with the increase of the layer number. And some bad pixels can not be filled in the layered interpolation, and a method similar to bilateral filtering is also utilized for processing, so that the efficiency and the accuracy of the sampling on the depth map are effectively improved.

The foregoing description is only an overview of the technical solutions of the present invention, and the embodiments of the present invention are described below in order to make the technical means of the present invention more clearly understood and to make the above and other objects, features, and advantages of the present invention more clearly understandable.

Drawings

The invention will be further described with reference to the following examples and figures.

FIG. 1 is a flow chart of a method according to an embodiment of the present invention

FIG. 2 is a schematic diagram of the technical framework of the overall solution of the present invention;

FIG. 3 is a flow chart of a hierarchical depth map upsampling process of the present invention;

FIG. 4 is an example of searching for a seed and a target pixel of the present invention;

FIG. 5 is a schematic diagram of the present invention searching L seed pixels for interpolation of a current target;

fig. 6 is a schematic structural diagram of a device according to a second embodiment of the present invention.

Detailed Description

The embodiment of the application provides a depth map upsampling method and device based on hierarchical clustering and boundary enhancement, and the method and device effectively improve the efficiency and accuracy of depth map upsampling and prevent depth aliasing artifacts of depth discontinuous regions by respectively preprocessing a low-resolution depth map and a high-resolution color image and then performing hierarchical interpolation on the depth map by using a clustering algorithm.

The technical scheme in the embodiment of the application has the following general idea:

(1) the low-resolution depth map is preprocessed and divided into a depth continuous area and a depth discontinuous area, and different methods are adopted for respectively carrying out upsampling, so that the processing time is saved, and the efficiency is improved.

(2) Processing the high-resolution color image by using an initial sketch algorithm (Primal sketch) to obtain an original sketch and a sketch-capable image, and on one hand, smoothing the high-resolution color image by using the sketch-capable image to realize rapid and effective clustering; on the other hand, adding the original sketch to the smooth high-resolution color image, enhancing the image boundary, can help to maintain the depth boundary and make it overlap the boundary of the high-resolution color image as much as possible.

(3) The depth map is interpolated hierarchically, each layer comprising two steps:

1) dividing the preprocessed high-resolution color image into k classes by using a k-means clustering method, wherein each class comprises a plurality of discontinuous areas;

2) and for each region, interpolating the target pixel by using a nearest neighbor joint bilateral NJB interpolation method.

In order to find the most accurate seed for interpolation, the image needs to be segmented from thin to thick, i.e. the number k of classes of k-means cluster is decreased.

(4) When the last layer of interpolation is completed, almost all target pixels are interpolated, and for some residual targets, post-smoothing operation is used for deleting.

The technical block diagram of the overall scheme is shown in fig. 2, and the overall scheme is divided into two processes, namely an image preprocessing process and a layered depth map upsampling process.

First, preprocessing process of image

The method mainly comprises two image preprocessing processes which are respectively applied to a low-resolution depth image and a high-resolution color image. The pre-processing a separates the depth discontinuity areas, i.e. the boundary areas, from the continuous areas. The pixels located in the continuous area are called seed pixels and used for interpolation; the pixels located in the discontinuous region are called target pixels, and interpolation is performed by the seed pixels. And the preprocessing B introduces an initial sketch algorithm to carry out image smoothing and boundary enhancement of a high-resolution color image.

1) Preprocessing of low-resolution depth maps A

Aiming at the characteristics of simple texture and less color information of the depth image, bilinear filtering and a threshold function are adopted for carrying out region division, so thatThe number of the seed pixels is increased, the number of the target pixels is reduced, and the calculation time is greatly saved. First, the low resolution depth map is enlarged to the same size as the high resolution color image by bilinear interpolation, called the intermediate up-sampling depth map

The depth is divided into discontinuous areas and continuous areas by a sliding window function shown in a formula (1),

wherein N is_d(i) Represents a rectangular frame centered on pixel i; th_DIs a threshold value for estimating N_d(i) Whether it is a continuous region or a discontinuous region. If N is present_d(i) The maximum depth difference of the middle pixel is larger than the threshold value, then N_d(i) Belonging to a depth discontinuity area, wherein the pixel in the area is a target pixel, otherwise, N_d(i) Belonging to a deep continuum, N_d(i) The pixel in (D) is a seed pixel, and interpolation is directly carried out, so that a preprocessed depth map D is obtained_I。

2) Preprocessing of high-resolution color images B

Since the image of the YCbCr color space is more superior in describing human vision, the gray scale image of a high-resolution color image (high-resolution gray scale image) is finely divided by using an initial sketch algorithm. The initial sketch model can be described simply as:

p(I)＝p(I_Φ)p(Φ_sk)p(Φ_nsk) (2)

wherein, p (phi)_sk) Representing a sketch part, p (phi)_nsk) Representing non-sketch parts, i.e. original sketches, p (I)_Φ) Representing the getalt field model.

Obtaining an original sketch S through an initial sketch model_gAnd sketch S_i. Original sketch S_gProviding a clear image boundary, sketch S_iDescribing the structure of an image, simultaneously for the structure Blurring is performed internally. Therefore, using them to smooth the high resolution grayscale image through an anisotropic filter, enhanced image boundaries and smooth homogeneous regions are obtained:

wherein I is a high-resolution gray scale image, N (I) represents 8 neighborhoods of a pixel I, f_ijFor the anisotropic kernel function:

where pixel i represents the current point to be smoothed, pixel j represents one of the neighborhood pixels of i, j_pRepresenting one of the omega neighbourhood pixels, sigma, in one direction along pixel i_A、σ_BAnd C_tThree given parameters.

The first case indicates that the current pixel and its neighbours are not at the object boundary, i.e. they are in the same homogeneous region. Smoothing the current pixel i, and in order to increase the weight of the pixel j, the parameter σ may be set_AIs arranged relatively large.

The second case only limits the current pixel i located at the boundary of the object, and therefore it covers the case if the neighboring pixel j is located on the boundary. At the same time, to strengthen the boundary, the parameter σ_BShould not be set too large.

The third case represents that the current pixel i is not located at the target boundary but its neighbor pixel j is located at the boundary. In this case, C_tShould be large enough to reduce the impact of the boundary pixel j on the current pixel i.

Thus, a filtered high resolution image I can be obtained_FThe uniform region is smoothed and the boundary is effectively enhanced.

The above process facilitates subsequent k-means based image segmentation. But in some depth discontinuity areas, the colors are very similar and the pixels will be erroneously clustered intoThe same category. This is because k-means clustering is pixel-based, i.e., dividing pixels of similar values into the same class. That is, when regions of similar colors belong to different depth fields, erroneous segmentation may result. To solve this problem, a sketch S_gWill be added to the filtered high resolution image I_FIn the step (1), the first step,

thus, regions that cross a depth boundary will be partitioned into unconnected regions. Finally, a preprocessed image I is obtained_RFor subsequent image segmentation.

Two, hierarchical depth map upsampling process

Providing a layered depth map upsampling method, and utilizing the seed pixel pair to preprocess the depth map D_IThe target pixel in (1) is interpolated. To avoid depth blending artifacts, accurate depth boundaries are obtained by first segmenting the image into multiple unconnected regions and then interpolating the target in each region with seeds in the same region.

1) High-resolution gray scale image segmentation based on k-means

Firstly, based on a k-means clustering algorithm, a high-resolution gray level image I is obtained_RIs divided into k_nClass i, wherein k_nFrom a given set of class numbers. Fig. 2 shows a flow chart of the method, which is performed layer by layer, with N representing the total number of layers. At the first level, the image needs to be segmented into relatively many classes to ensure that all depth discontinuities are segmented into different regions. However, there is also a problem that an object not including the seed pixel region cannot be interpolated. Thus, the number of classes k_nAre progressively decreased layer by layer.

Suppose the training set is { x }₁,x₂,…,x_yThe k-means clustering is to classify the observed value y into k classes C ═ C_1,C_2,…,C_kY to minimize the sum of squares within a class, with the goal of finding C, such that

In the formula, mu_pIs represented by C_pAverage of all points in (1). It can be seen that the k-means algorithm needs to continuously adjust sample classification and calculate a new clustering center, and when a training set contains a large amount of data, the convergence rate of the algorithm is very slow. Thus, the input pre-processed high resolution image I can be pre-processed_RRandom sampling is performed to improve efficiency. Since the high-resolution color image is subjected to the smoothing and boundary enhancement of the homogeneous region in advance, the k-means segmentation can obtain a more definite boundary.

2) Searching corresponding seeds and targets

Assuming that at the nth layer, a high resolution gray scale image I_RIs divided into k_nClass, and a cluster image Ω is obtained as shown in fig. 4 (a). In the cluster image Ω, each gray level represents a different category Ω_i＝{Ω|Ω(j)＝i}，i＝1～k_n. Then each omega is added_iDivided into M discrete regions. The method shown in FIG. 3, using a signal at Ω_imThe seed pixel of (M-1, …, M) interpolates the target of the same region. The black-gray pixel in fig. 4(b) represents Ω_iWhite pixels represent the corresponding object. By the methods, corresponding seeds and targets can be effectively searched. In order to save interpolation time, some very small regions containing only a few tens of pixels will be ignored, which are easily filled in the final smoothing process.

3) Interpolation method based on nearest neighbor joint bilateral

Assume that the current interpolation region contains T target pixels and S seed pixels. As shown in fig. 3, for each target pixel

First find several nearest seeds around it and make sure that the selected seed is in the previous S seed pixels. Then, the selected seeds are used for carrying out interpolation on the current target pixel through joint bilateral filtering:

in the formula, L represents the number of selected seeds. g _d(t,s_l) And g_c(t,s_l) Gaussian kernels for spatial distance and chromatic aberration, respectively, are represented:

in the formula, d (t, s)_l) For the current target pixel t and the selected seed pixel s_lGeometric distance between, d_σAnd c_σGiven the scale parameters. It is worth mentioning s_lIn the same region as t, the key problem is how to determine the current region omega_imAnd L corresponding seed pixels.

For example, enlarging the gray box area in fig. 4(b) is shown in fig. 5, and assuming that the black cross point in fig. 5 is the current target pixel

To obtain the L nearest seed pixels

A square search window is generated for the center. Starting from the search radius omega being 1, the pixels meeting the search are seeds and the pixels are located in the current region omega as the current target pixel_imPixels of these two conditions. If no pixel satisfying the condition can be found, the search radius ω is increased by a step size of 1. Once one such pixel is sought, the search is stopped. All pixels that satisfy the above condition and are located on the current search window (blue box in fig. 4) are used for interpolation. The radius ω of the search window should be decreasing, andnot greater than a given maximum search radius W_nAnd n represents an nth layer. This means that W_nAnd k is_nAre in one-to-one correspondence. W _nDecreasing as the number of layers increases. This is because the seeds are relatively few at the beginning, and are far from the target. But as the amount of interpolation increases, the distance to the target gradually decreases.

Image segmentation based on k-means can be divided into at least two broad categories, namely k_n2, this will result in some bad pixels not being filled in during the hierarchical interpolation. In order to remove these defective pixels, a method similar to bilateral filtering is proposed to process the defective pixels. It is assumed that u is one of the defective pixels,

in the formula, N_s(u) is a rectangular search window centered at u, the size being determined empirically. d (u, v) represents the geometric distance, and I is the original high resolution color image. In this way, a sharply bounded up-sampled depth map can be obtained.

Example one

The embodiment provides a depth map upsampling method based on hierarchical clustering and boundary enhancement, as shown in fig. 1, including: an image preprocessing process and a layered depth map upsampling process;

the image preprocessing process comprises:

preprocessing the low-resolution depth map, dividing the low-resolution depth map into a depth continuous region and a depth discontinuous region to obtain a preprocessed depth map D_I(ii) a Taking the pixels positioned in the depth continuous area as seed pixels for interpolation; taking the pixel positioned in the depth discontinuous area as a target pixel, and performing target pixel interpolation through a seed pixel;

Processing the high-resolution color image by using an initial sketch model to obtain an original sketch and a sketch-capable image, and smoothing the high-resolution color image by using the sketch-capable image to realize quick and effective clustering; at a smooth high scoreAdding the original sketch into the color image, enhancing the image boundary, and obtaining a high-resolution gray level image I_R；

The hierarchical depth map upsampling process comprises:

for the preprocessed depth map D_ICarrying out layered interpolation, wherein each layer firstly utilizes a k-means clustering method to carry out the high-resolution gray level image I_RIs divided into k_nEach class comprises a plurality of discontinuous areas, and each area is interpolated by using a nearest neighbor joint bilateral NJB interpolation method to interpolate a target pixel, wherein k_nThe number of categories of the n-th-layer cluster; and after the last layer of interpolation is finished, almost all target pixels are interpolated, and the residual target pixels which are not interpolated are deleted by using the post-smoothing operation.

In a possible implementation manner, in the image preprocessing process, the low-resolution depth map is preprocessed, and a bilinear filter and a threshold function are used for region division.

Specifically, in the image preprocessing process, the initial sketch model is specifically as follows:

p(I)＝p(I_Φ)p(Φ_sk)p(Φ_nsk)

Wherein, p (phi)_sk) Representing the sketch part, p (phi)_nsk) Denotes the non-sketch part, p (I)_Φ) Representing the getalt field model.

Preferably, the number of categories k of the k-means cluster_nThe segmentation is gradually decreased layer by layer, and the segmentation from thin to thick of the image is realized.

In a possible implementation manner, in the process of upsampling the hierarchical depth map, interpolating a target pixel for each region by using a nearest neighbor joint bilateral NJB interpolation method specifically includes:

assuming that the current interpolation region contains T target pixels and S seed pixels, for each target pixel, firstly finding several nearest seeds around the target pixel, ensuring that the selected seeds are in the S seed pixels, and then interpolating the current target pixel by joint bilateral filtering by using the selected seeds:

wherein L represents the number of selected seeds, g_d(t,s_l) And g_c(t,s_l) The gaussian kernels for spatial distance and chromatic aberration are represented separately as follows:

wherein d (t, s)_l) For the current target pixel t and the selected seed pixel s_lGeometric distance between, d_σAnd c_σGiven the scale parameters.

Based on the same inventive concept, the application also provides a device corresponding to the method in the first embodiment, which is detailed in the second embodiment.

Example two

In this embodiment, there is provided a depth map upsampling apparatus based on hierarchical clustering and boundary enhancement, as shown in fig. 6, including: the device comprises an image preprocessing module and a layered depth map upsampling module;

the image preprocessing module is used for preprocessing the low-resolution depth map, dividing the low-resolution depth map into a depth continuous area and a depth discontinuous area and obtaining a preprocessed depth map D_I(ii) a Taking the pixels positioned in the depth continuous area as seed pixels for interpolation; taking the pixel positioned in the depth discontinuous area as a target pixel, and interpolating the target pixel through a seed pixel; processing the high-resolution color image by using an initial sketch model to obtain an original sketch and a sketch-capable image, and smoothing the high-resolution color image by using the sketch-capable image to realize rapid and effective clustering; adding said original grass to a smooth high resolution color imageEnhancing image boundary to obtain high-resolution gray scale image I_R；

The layered depth map up-sampling module is used for pre-processing the depth map D_ICarrying out layered interpolation, wherein each layer firstly utilizes a k-means clustering method to carry out the high-resolution gray level image I _RIs divided into k_nEach class comprises a plurality of discontinuous areas, and each area is interpolated by using a nearest neighbor joint bilateral NJB interpolation method to interpolate a target pixel, wherein k_nThe number of categories of the n-th-layer cluster; and after the last layer of interpolation is finished, almost all target pixels are interpolated, and the residual target pixels which are not interpolated are deleted by using the post-smoothing operation.

In a possible implementation manner, in the image preprocessing module, the low-resolution depth map is preprocessed, and the bilinear filtering and the threshold function are used for region division.

Specifically, in the image preprocessing module, the initial sketch model is specifically as follows:

p(I)＝p(I_Φ)p(Φ_sk)p(Φ_nsk)

wherein, p (phi)_sk) Representing a sketch part, p (phi)_nsk) Denotes the non-sketch part, p (I)_Φ) Representing the getalt field model.

Preferably, the number of categories k of the k-means cluster_nThe segmentation is gradually decreased layer by layer, and the segmentation from thin to thick of the image is realized.

In a possible implementation manner, in the hierarchical depth map upsampling module, for each region, interpolating a target pixel by using a nearest neighbor joint bilateral NJB interpolation method specifically includes:

assuming that the current interpolation region contains T target pixels and S seed pixels, for each target pixel, firstly finding several nearest seeds around the target pixel, ensuring that the selected seeds are in the S seed pixels, and then interpolating the current target pixel by joint bilateral filtering by using the selected seeds:

Wherein L represents the number of selected seeds, g_d(t,s_l) And g_c(t,s_l) The gaussian kernels for spatial distance and chromatic aberration are represented separately as follows:

wherein d (t, s)_l) For the current target pixel t and the selected seed pixel s_lGeometric distance between, d_σAnd c_σGiven the scale parameters.

Since the apparatus described in the second embodiment of the present invention is an apparatus used for implementing the method of the first embodiment of the present invention, based on the method described in the first embodiment of the present invention, a person skilled in the art can understand the specific structure and the deformation of the apparatus, and thus the details are not described herein. All the devices adopted in the method of the first embodiment of the present invention belong to the protection scope of the present invention.

The technical scheme provided in the embodiment of the application at least has the following technical effects or advantages:

1. when image preprocessing is carried out, the method is different from the existing method, not only is the low-resolution depth image preprocessed, but also the bilinear filtering and the threshold function are utilized to carry out region division, the region division is carried out, the region division is divided into a depth continuous region and a depth discontinuous region (namely a boundary region), the seed pixels of the continuous region are directly interpolated, the target pixels of the discontinuous region need to be processed, the target number is reduced, and the calculation efficiency is improved. Meanwhile, the method also utilizes an initial sketch algorithm to preprocess the high-resolution color image, divides the high-resolution color image into a sketch image and a Prime image, utilizes an anisotropic filter to obtain a smooth uniform area and an enhanced image boundary, and utilizes the boundary information of the sketch image to process for depth discontinuous areas with similar colors, thereby being beneficial to subsequent fine image segmentation.

2. When the depth map is subjected to upsampling, the patent provides an upsampling method for hierarchical classification, an image is divided into a plurality of unconnected areas, and the target pixels are interpolated by using seed pixels in the same area. The method comprises the steps of firstly, carrying out hierarchical classification on a high-resolution color image by using a k-means clustering algorithm, wherein the first layer needs to divide the image into more categories, all depth discontinuous points are ensured to be divided into different areas, and the number of the categories is gradually decreased layer by layer. Because the high-resolution color image is smoothed in the homogeneous region and the boundary region is enhanced in the preprocessing process, the target pixels can be classified and interpolated, and the depth confusion artifact of the depth discontinuous region is effectively prevented.

3. When the interpolation of the target pixels is carried out, a nearest neighbor joint bilateral interpolation method is provided, for each target pixel, several nearest neighbor seed pixels are selected to carry out joint bilateral filtering interpolation, the process of selecting the seed pixels is to search by taking the target pixels as the center, and the size of the maximum search radius is reduced along with the increase of the number of layers. And some bad pixels can not be filled in the layered interpolation, and a method similar to bilateral filtering is also utilized for processing, so that the efficiency and the accuracy of the sampling on the depth map are effectively improved.

As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, apparatus, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present invention has been described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

Although specific embodiments of the invention have been described above, it will be understood by those skilled in the art that the specific embodiments described are illustrative only and are not limiting upon the scope of the invention, and that equivalent modifications and variations can be made by those skilled in the art without departing from the spirit of the invention, which is to be limited only by the appended claims.

Claims (10)

1. A depth map upsampling method based on hierarchical clustering and boundary enhancement is characterized by comprising the following steps: an image preprocessing process and a layered depth map upsampling process;

the image preprocessing process includes:

preprocessing the low-resolution depth map, dividing the low-resolution depth map into a depth continuous area and a depth discontinuous area to obtain a preprocessed depth map D_I(ii) a Taking the pixels positioned in the depth continuous area as seed pixels for interpolation; taking the pixel positioned in the depth discontinuous area as a target pixel, and performing target pixel interpolation through a seed pixel;

processing the high-resolution color image by using an initial sketch model to obtain an original sketch and a sketch-capable image, and smoothing the high-resolution color image by using the sketch-capable image to realize rapid and effective clustering; adding the original sketch into the smooth high-resolution color image, enhancing the image boundary and obtaining a high-resolution gray scale image I_R；

The hierarchical depth map upsampling process comprises:

for the preprocessed depth map D_ICarrying out layered interpolation, wherein each layer firstly utilizes a k-means clustering method to carry out the high-resolution gray level image I_RIs divided into k_nClass, each class comprises a plurality of discontinuous areas, and each area is interpolated by a nearest neighbor joint bilateral NJB interpolation method to carry out interpolation on target pixels, wherein k _nThe number of categories of the n-th-layer cluster; and after the last layer of interpolation is finished, almost all target pixels are interpolated, and the residual target pixels which are not interpolated are deleted by using post-smoothing operation.

2. The method of claim 1, wherein: in the image preprocessing process, the low-resolution depth image is preprocessed, and the bilinear filtering and the threshold function are adopted for carrying out region division.

3. The method of claim 1, wherein: in the image preprocessing process, the initial sketch model specifically comprises the following steps:

p(I)＝p(I_Φ)p(Φ_sk)p(Φ_nsk)

wherein, p (phi)_sk) Representing a sketch part, p (phi)_nsk) Watch (A)Non-sketch portion, p (I)_Φ) Representing the getalt field model.

4. The method of claim 1, wherein: number of classes k of the k-means cluster_nThe segmentation is gradually decreased layer by layer, and the segmentation from thin to thick of the image is realized.

5. The method of claim 1, wherein: in the process of up-sampling the hierarchical depth map, interpolating a target pixel for each region by using a nearest neighbor joint bilateral NJB interpolation method, which specifically comprises the following steps:

assuming that the current interpolation region contains T target pixels and S seed pixels, for each target pixel, firstly finding several nearest seeds around the target pixel, ensuring that the selected seeds are in the S seed pixels, and then interpolating the current target pixel by joint bilateral filtering by using the selected seeds:

Wherein L represents the number of selected seeds, g_d(t,s_l) And g_c(t,s_l) The gaussian kernels for spatial distance and chromatic aberration are respectively expressed as follows:

wherein d (t, s)_l) For the current target pixel t and the selected seed pixel s_lGeometric distance between, d_σAnd c_σGiven scale parameters.

6. An apparatus for depth map upsampling based on hierarchical clustering and boundary enhancement, comprising: the device comprises an image preprocessing module and a layered depth map upsampling module;

the image preprocessing module is used for preprocessing the low-resolution depth map, dividing the low-resolution depth map into a depth continuous area and a depth discontinuous area and obtaining a preprocessed depth map D_I(ii) a Taking the pixels positioned in the depth continuous area as seed pixels for interpolation; taking the pixel positioned in the depth discontinuous area as a target pixel, and performing target pixel interpolation through a seed pixel; processing the high-resolution color image by using an initial sketch model to obtain an original sketch and a sketch-capable image, and smoothing the high-resolution color image by using the sketch-capable image to realize rapid and effective clustering; adding the original sketch into the smooth high-resolution color image, enhancing the image boundary and obtaining a high-resolution gray scale image I _R；

The layered depth map upsampling module is used for performing upsampling on the preprocessed depth map D_ICarrying out layered interpolation, wherein each layer firstly utilizes a k-means clustering method to carry out the high-resolution gray level image I_RIs divided into k_nClass, each class comprises a plurality of discontinuous areas, and each area is interpolated by a nearest neighbor joint bilateral NJB interpolation method to carry out interpolation on target pixels, wherein k_nThe number of the categories of the n-th layer of clusters; and after the last layer of interpolation is finished, almost all target pixels are interpolated, and the residual target pixels which are not interpolated are deleted by using the post-smoothing operation.

7. The apparatus of claim 6, wherein: and in the image preprocessing module, preprocessing is carried out on the low-resolution depth image, and region division is carried out by adopting bilinear filtering and a threshold function.

8. The apparatus of claim 6, wherein: in the image preprocessing module, the initial sketch model is specifically as follows:

p(I)＝p(I_Φ)p(Φ_sk)p(Φ_nsk)

wherein, p (phi)_sk) Representing a sketch part, p (phi)_nsk) Denotes the non-sketch part, p (I)_Φ) Representing the getalt field model.

9. The apparatus of claim 6, wherein: number of classes k of the k-means cluster_nThe segmentation is gradually decreased layer by layer, and the segmentation from thin to thick of the image is realized.

10. The apparatus of claim 6, wherein: in the hierarchical depth map upsampling module, for each region, interpolating a target pixel by using a nearest neighbor joint bilateral NJB interpolation method, which specifically comprises the following steps:

assuming that the current interpolation region contains T target pixels and S seed pixels, for each target pixel, firstly finding several nearest seeds around the target pixel, ensuring that the selected seeds are in the S seed pixels, and then interpolating the current target pixel by joint bilateral filtering by using the selected seeds:

wherein L represents the number of selected seeds, g_d(t,s_l) And g_c(t,s_l) The gaussian kernels for spatial distance and chromatic aberration are respectively expressed as follows:

wherein d (t, s)_l) For the current target pixel t and the selected seed pixel s_lGeometric distance between, d_σAnd c_σGiven the scale parameters.

Images