RU2580466C1 - Device for recovery of depth map of scene - Google Patents

Abstract

FIELD: computer engineering.

SUBSTANCE: invention relates to computer engineering and specifically to image processing systems. Device for recovering depth of a scene is disclosed. Device comprises an input realisation storage unit, a colour image storage unit, a unit for determining quasi-stationarity areas, a unit for generating a mask-element multiplier, a unit for calculating a median estimate, a useful component estimate storage unit, a counter current line, a current column counter, four delay units, a depth map storage unit, depth map mask forming unit, unit for recovering depth map, a clock pulse generator.

EFFECT: reduced error in determining distance from scene objects to sensor camera.

1 cl, 9 dwg

Description

The present invention relates to information-measuring devices and can be used in computer technology, in control systems, receiving and processing stereo images and depth maps.

вида:In general, a simplified mathematical model of a two-dimensional signal (image) is a two-dimensional discrete sequence

type:

where S _{i, j} is a useful component; η _{i, j} is the additive noise component distributed according to the Gaussian law with zero mathematical expectation and constant dispersion; N is the number of rows; M is the number of columns of a two-dimensional image array.

The depth map is a two-dimensional single-channel image containing information about the distance from the sensor plane to the scene objects. It is created using active 3D scanners, for example, the Microsoft Kinect® system, which consists of an infrared projector, an RGB camera and a depth sensor. The distance to various points in the scene is estimated by measuring the relative displacement of the points projected by the projector on the image from the infrared camera [Franz V.A. Initial processing of the image depth map. “Successes of modern radio electronics”, No. 6, 2013]. In some cases, the projected points may not be found, which leads to defects — these can be lost and distorted depths, random noise, uneven edges and surfaces of objects, as well as incorrectly measured depths for some materials with mirror or fine-grained surfaces [ Shen S. Layer Depth Denoising and Completion for Structured-Light RGB-D Cameras. Computer Vision and Pattern Recognition (CVPR), 2013].

There are many methods for processing a depth map, which can be divided into two groups: preliminary and post-processing of a depth map.

The main disadvantages of the preprocessing methods include inaccurate restoration of information about the depth and insufficient improvement of the boundaries of objects, which affects the further use of this information. Therefore, it is important that the depth information received from various sensors is correct for subsequent processing. This improves the quality and performance of various applications.

The main task to be solved is to reduce the error in determining the distance from scene objects to the sensor camera by processing the image of the depth map with noise suppression, improving the boundaries of objects and restoring lost areas of the depth map.

A similar problem may arise: 1) in systems for obtaining 3D models; 2) in radio engineering when processing signals; 3) in digital image analysis processing systems; 4) in robotics when positioning and determining the distance to objects.

Known filtering methods can be applied to images of depth maps to reduce the effect of noise.

,

исходного сигнала. The known method of the moving average [Gonzalez, R. Digital image processing / R. Gonzalez, R. Woods. - M .: Technosphere. - 2005. - 1072 s]. To use it, one implementation of Y _{i, j, is} enough

,

source signal.

,

, определение размера маски фильтра m (ширины «скользящего окна»), для которого производится вычисление среднего арифметического,

, замену центрального из значений Y_i,j,

,

найденным средним

, сдвиг «скользящего окна» на одно значение вправо, вычисление среднего арифметического выбранных значений реализации и так до тех пор, пока маска фильтра не переместится по всему изображению.For the original image, the size of the smoothing filter mask m is determined, i.e. natural number m <N. The moving average method involves storing the original image Y _{i, j,}

,

, determining the size of the filter mask m (the width of the "sliding window") for which the arithmetic mean is calculated,

, replacing the central of the values of Y _{i, j,}

,

found average

, shifting the “sliding window” by one value to the right, calculating the arithmetic average of the selected implementation values, and so on, until the filter mask moves across the entire image.

The width of the “window” is chosen odd, because the smoothed value is calculated for the central value.

The features of the analog device, which coincide with the features of the claimed technical solution, are as follows: storing a discrete signal, highlighting time intervals, finding the arithmetic mean of the signal falling into the selected time intervals, replacing the original two-dimensional discrete implementation of the measurement results with smoothed values.

The disadvantages of the known device are:

- inability to process rows or columns of the image located at the borders of the mask in case the center of the filter approaches the borders of the image;

- the moving average method causes autocorrelation of residues, even if it was absent in the original useful component (Slutsky-Yul effect).

The reasons that impede the achievement of the required technical result are as follows:

- if the width of the “window” of smoothing is 2p + 1, then the first p and the last p of the initial values of the measurement results are not processed;

- since the central value of the “window” of smoothing is calculated as the arithmetic average of the neighboring ones, the values of the estimation of the useful component become dependent.

As non-linear filters, filters based on ordinal statistics are used [Gonzalez, R. Digital Image Processing / R. Gonzalez, R. Woods. - M .: Technosphere. - 2005. - 1072 s]. The response of such a filter is determined by pre-ordering (ranking) the pixel values covered by the filter mask, and then selecting a value located at a certain position in the ordered sequence (i.e., having a certain rank). Filtering is reduced to replacing the original value (in the center of the mask) with the received filter response value. The median filter is best known, which replaces the central value of the filter mask with the median value of the distribution of all values of the measurement results belonging to the filter mask area. To perform median filtering for an image element, you must first sort the increasing pixel values inside the mask, then find the median value and assign the resulting value to the element being processed.

The features of the analog device, which coincide with the features of the claimed technical solution, are as follows: storing the input two-dimensional signal, selecting time periods, replacing the input implementation of the measurement results with smoothed values.

The disadvantages of the known device are:

- inability to process rows or columns of the image located at the borders of the mask in case the center of the filter approaches the borders of the image;

- due to the nonlinearity of the processing method, it is impossible to strictly distinguish between the effects of median filtering on signal and noise;

- median smoothing can only be considered as an effective way to pre-process the input implementation of the measurement results in the case of impulse noise.

The reasons that impede the achievement of the required technical result are as follows:

- median filtering is a non-linear processing method;

- the dependence of the smoothing efficiency of the measurement results on the shape of the useful and noise component.

The block diagram of a device that implements the considered method contains a pulse generator, a switch, a control unit, a storage register, a ranking unit, an average value selection unit, an output register, where an estimate of the initial discrete implementation of the measurement results is stored.

A known method and device for processing a depth map [Method and device for processing a Depth-map, US 20100215251 A1]. The present invention is used to process a depth map, the method including reconstructing a depth map based on a lossy compressed depth map, wherein the depth map includes detailed scene information from the point of view of the scene containing the object. The device allows you to restore the depth map when objects overlap part of the background.

The features of the analog device that match the features of the claimed technical solution are as follows: storing the input two-dimensional signal, processing the depth map, saving the processing result.

The disadvantages of the known device are:

- a large error in the restoration of the contours and boundaries of objects on the depth map;

- a priori information when choosing method parameters while eliminating compression artifacts.

The structural diagram of a device for processing a depth map contains a block for eliminating compression artifacts, a block for restoring a depth map when objects overlap, and an evaluation storage unit.

A device for generating a depth map from two images from different points of view is known [Generating a depth map from a two-dimensional source image for stereoscopic and multiview imaging, US 20130009952 A1]. The depth map is restored from monoscopic source images and is asymmetrically smoothed. The depth map contains depth values focused on the edges of local areas in the original image. Each edge is determined on a given image, a parameter having a calculated value exceeds a predetermined threshold. The depth of this value is based on the corresponding calculated values of the image parameters. The depth map is used to process the original image by image depth based on the rendering algorithm to create a single image that forms a set of monoscopic images with the original image. At least one pair of stereoscopic images is selected from such a set for use in creating various points of view, including for still and moving images.

The features of the analog device that match the features of the claimed technical solution are as follows: storing the input two-dimensional signal, smoothing the depth map, saving the processing result.

The disadvantages of the known device are:

- a priori information when choosing parameters when calculating the boundaries of objects on images;

- restoration of the depth map only for the boundaries of objects in ¹ image.

The structural diagram of a device for processing a depth map contains an image storage unit, a border analysis unit, a depth map synthesis unit, a border smoothing unit, a depth map smoothing unit, a 3D image rendering unit, a 3D image display unit.

Closest to the invention is a device for preprocessing a stereo depth map (Fig. 8) [US Pat. No. 2535183 Russian Federation; IPC G06F 17/00, G06T 15/00. 2013135016/08 (052462); Application 07/25/2013; Publ. 12/10/2014 Bull. No. 34].

Consider a prototype device involves:

,

;- recording the values of the input two-dimensional signal Y _{i, j,}

,

;

- a dilatation operation is applied to each frame of the video sequence to remove areas with lost and distorted pixels, which is a morphological operation and enlarges the image area, expanding its pixels and thereby facilitating the union of image areas that were separated by noise;

- to reduce the influence of noise, an approach is implemented, implemented by an adaptive two-dimensional method of multiplying estimates. In this case, the image of the depth map is divided into local regions of uniform brightness values, which are further approximated by a first-order surface by the least-squares method.

на изображении для одномерных реализаций, полученных из значений пикселей по вертикальным и диагональным направлениям от центрального пикселя исходного изображения, определяются интервалы квазистационарности с помощью способа инверсий. Данные границы позволяют получить интервалы с монотонным изменением яркости сигнала. Далее, все восемь полученных секторов объединяются в одну область Ω. Таким образом, для каждого пикселя формируется окрестность пикселей, близких по значению яркости.For directions

in the image for one-dimensional realizations obtained from pixel values in the vertical and diagonal directions from the central pixel of the original image, the intervals of quasistationarity are determined using the inversion method. These boundaries make it possible to obtain intervals with a monotonic change in the brightness of the signal. Further, all eight obtained sectors are combined into one domain Ω. Thus, for each pixel, a neighborhood of pixels similar in brightness value is formed.

, где ω_i,j,

,

значения бинарной маски, которая принимает значения, равные:The pixel values that fall between all directions and interpolating lines passing through the boundaries of the quasistationary intervals are combined into one domain Ω and approximated by a first-order surface of the form

where ω _{i, j} ,

,

the values of the binary mask, which takes values equal to:

.

.

The values of the coefficients A, B and C are determined using the two-dimensional least squares method [Patent No. 2362207 Russian Federation, C2, IPC G06F 17/17. 2007127727/09; Application 07/19/2007; Publ. 07/20/2009 Bull. No. 20], for finding which the objective function of the form is minimized:

.

.

повторяется для каждого значения пикселя Y_i,j,

,

, при этом формируется весовая функция W_i,j,

,

, значения которой равны количеству размноженных оценок для каждого пикселя.The procedure for obtaining the domain Ω and calculating the estimate

repeated for each pixel value Y _{i, j} ,

,

, the weight function W _{i, j}

,

whose values are equal to the number of multiplied estimates for each pixel.

определяется как среднее арифметическое размноженных адаптивных оценок:

.Resulting Image Rating

defined as the arithmetic mean of the multiplied adaptive estimates:

.

- at the last stage, several smoothed estimates of depth maps are used for the sequence of frames for which the median is calculated. This operation allows you to adjust the estimate of the contours of objects and make them smoother.

The disadvantages of the known prototype device are:

- the inability to detect and restore large areas with lost pixel values;

- a large error in the restoration of the depth map.

The reasons that impede the achievement of the required technical result are as follows:

- this device is intended only to reduce the influence of the additive noise component.

The pre-processing device for the stereo depth map (Fig. 8) contains an input implementation storage unit, a dilatation unit, a quasistationary section determination unit, a mask forming unit, an element-wise multiplier, an approximation unit, an estimate storage unit, an averaging unit, a useful component estimate storage unit, a current counter lines, averaging signal generator, current column counter, first delay block, second delay block, third delay block, fourth delay block, fifth delay block, sixth delay block ki, a seventh delay unit, the eighth delay block, obtaining the median estimate of clock pulse generator.

The essence of the proposed device for reconstructing a map of the depth of the scene is as follows.

A simplified mathematical model of the input two-dimensional signal is presented in accordance with expression (1). The algorithm that implements the operation of the device (Fig. 1) consists of the following steps:

1) A color image and a depth map of one scene are input to the device;

2) Damaged areas are detected using threshold detection, thus creating a mask for damaged areas;

3) The reconstruction of damaged areas occurs by image restoration [Patent No. 2450342 Russian Federation, C1, IPC G06F 17/17. 2011132449/08; Application 08/08/2011; Publ. 05/10/2012 Bull. No. 13];

4) Filtering depth images using an adaptive two-dimensional method of multiplying estimates using information about color and depth;

5) Record the result of recovery.

The image is processed in two channels. In the first, reconstruction and filling of the areas of distorted values of the pixels of the depth map is performed, and in the second, the formation of areas of close gradation of brightness using the brightness channel is performed.

To detect damaged areas, a priori information is used that defects on the depth map are in the black color range (values in RGB [000]).

To reconstruct the pixels of the depth map, the priority value P (δS _{i, j} ) is calculated for each value of the border pixel, which consists of two factors (Fig. 2):

,

,

- квадратный блок пикселей с центром в пикселе δS_i,j;

- количество пикселей квадратного блока,

вектор, ортогональный градиенту в точке δS_i,j;

- вектор, ортогональный границе δS в точке δS_i,j; α - нормированный множитель, который для восьмибитных изображений равен 255.where: δS _{i, j} is the current pixel at the border of available pixels; C (δS _{i, j} ) is the confidence coefficient; D (δS _{i, j} ) is the gradient coefficient;

- a square block of pixels centered on the pixel δS _{i, j} ;

- the number of pixels in a square block,

a vector orthogonal to the gradient at the point δS _{i, j} ;

is the vector orthogonal to the boundary of δS at the point δS _{i, j} ; α is the normalized factor, which is equal to 255 for eight-bit images.

,

равно 1, а для области η равно 0.First, it is assumed that the value of the confidence coefficient C for pixels from the region S _{i, j} ,

,

is equal to 1, and for the region η it is equal to 0.

,

.Calculation of priority using expression (2) allows you to give more weight to pixels that are at brightness differences (borders), thus restoring them in the first place. Taking into account the confidence coefficient C (δS _{i, j} ) allows you to assign less weight to the restored pixels with increasing distance from available pixels from the area S _{i, j} ,

,

.

In the second step, for the pixel p∈ (i, j) with the maximum priority value max (P (δS _{i, j} )) at the boundary δS, the shape of the region for similarity search is adaptively determined using the inversions method, which allows one to correctly take into account the shape of the recovery region and Do not capture extra borders that may lead to incorrect image reconstruction.

, также определяется адаптивно область с помощью способа инверсий. Каждая из полученных областей является квазистационарной, и они находятся по разные стороны от перепада яркости. Данные области объединяются в одну (фиг. 3), таким образом, определяется область Ψ_р с адаптивными размерами и перепадом яркости.For a border pixel adjacent to a pixel p∈ (i, j) having a larger value

, the region is also determined adaptively using the inversion method. Each of the obtained regions is quasistationary, and they are located on different sides of the brightness difference. These areas are combined into one (Fig. 3), thus, the region Ψ _p with adaptive sizes and a difference in brightness is determined.

The pixel p∈ (i, j) with the maximum priority value max (P (δS _{i, j} )) at the boundary δS is determined and the adaptive region Ψ _p belonging to this pixel is selected. Use, which allows you to correctly take into account the shape of the recovery area and not capture extra boundaries that can lead to incorrect image reconstruction. Further, the number of blocks obtained from the original image with available pixels increases by rotating them by 90, 180, 270 degrees. This approach allows to reduce the error of image restoration by increasing the number of blocks and increasing the likelihood of finding a more similar block in the Euclidean metric (Fig. 4)

,

в области доступных пикселей S_i,j, для которых евклидова метрика минимальна (фиг. 5):In the third step, there are blocks

,

in the range of available pixels S _{i, j} for which the Euclidean metric is minimal (Fig. 5):

,

,

in this case, h denotes the sequence numbers of similar blocks, ranked by the Euclidean metric.

The number of similarity blocks R is determined using the confidence interval:

,

,

; α - уровень значимости.Where:

; α is the significance level.

из области доступных пикселей S_i,j:The pixel values in the region η adjacent to the pixel with the maximum priority p are restored by averaging the corresponding pixels of the found regions

from the range of available pixels S _{i, j} :

.

.

The confidence coefficient C for the restored pixels is set equal to the current value of C (p). After that, the procedure for recalculating priority and searching for similar areas with subsequent replacement is repeated.

(фиг. 6), в которых определяются интервалы квазистационарности. Условие квазистационарности проверяется с помощью вычисления случайной величины τ, равной сумме числа инверсий значений пикселей в каждом их восьми направлений двумерного сигнала Y_i,j,

,

.To form adaptive regions of a two-dimensional signal, eight directions are set for each pixel value

(Fig. 6), in which the intervals of quasistationarity are determined. The quasistationary condition is checked by calculating a random variable τ equal to the sum of the number of inversions of pixel values in each of their eight directions of the two-dimensional signal Y _{i, j} ,

,

.

For example, the sum of the number of inversions for direction 5 is:

,

,

where i = const, j = const; Y _{i, j} - the current value of the image pixel with coordinates (i, j); Y _{i + l, j} , l = i + 1 ... d - the subsequent values of the image pixels along the j-th column (movement in direction 5), d≤R, R - the maximum length of the quasistationary interval.

The number of combinations for which the sum of inversions is calculated is:

.

.

The first alternative (decreasing signal) is accepted if:

I.

I.

The rule for adopting the second alternative (increasing signal) has the form:

.

.

The hypothesis of stationarity of the signal is accepted if c ₂ <τ _d <c ₁ , where α is the a priori set value of the error of the first kind.

According to the obtained interval boundaries, for each of the eight sectors formed by the directions 1-2, 2-3, 3-4, 5-6, 7-8, 8-1, quasistationary regions are formed. For this, linear interpolation of the boundaries of adjacent intervals is used by the equation of a straight line passing through two points:

,

,

where (i ₁ , j ₁ ) are the coordinates of the boundary of the direction h, (i ₂ , j ₂ ) are the coordinates of the boundary of the direction h + 1.

на изображении для одномерных реализаций, полученных из значений пикселей по вертикальным и диагональным направлениям от центрального пикселя исходного изображения определяются интервалы квазистационарности с помощью способа инверсий. Данные границы позволяют получить интервалы с монотонным изменением яркости сигнала. Далее, все восемь полученных секторов объединяются в одну область Ω. Таким образом, для каждого пикселя формируется окрестность пикселей, близких по значению яркости.For directions

in the image for one-dimensional implementations obtained from pixel values in the vertical and diagonal directions from the central pixel of the original image, the intervals of quasistationarity are determined using the inversion method. These boundaries make it possible to obtain intervals with a monotonic change in the brightness of the signal. Further, all eight obtained sectors are combined into one domain Ω. Thus, for each pixel, a neighborhood of pixels similar in brightness value is formed.

To restore the boundaries of damaged areas, reduce noise and improve the quality of the depth image, median filtering of the depth map after the reconstruction stage is used in adaptive window combinations based on the color image.

Median filtering is applied in each combined adaptive region Ω _{i, j} for all pixels of the depth map according to the formula:

By performing median filtering in the combined adapted areas (Fig. 7), smoothing and alignment of the edges, as well as noise attenuation are achieved.

The scene depth map recovery device (Fig. 9) contains an input implementation 1 storage unit, the input of which is an information input of the device, the first output of which is connected to the input of the color image storage unit 2, the output of which is connected to the first input of the element-wise multiplier 5 and the first input of the determination unit sections of quasi-stationary 3, the output of which is connected to the input of the mask forming unit 4, the output of which is connected to the second input of the element-wise multiplier 5, the output of which is connected to the first input of the block calculating the median estimation 10, the output of which is connected to the input of the evaluation unit storing a useful component 11 whose output is the data output device; the output of the counter of the current line 6.1 is connected to the second input of the block for determining quasi-stationarity sections 3 and the input of the delay unit 6.2, the output of which is connected to the fourth input of the element-wise multiplier 5 and the input of the delay unit 6.3; the output of the counter of the current column 6.4 is connected to the third input of the block determining sections of quasi-stationary 3 and the input of the delay unit 6.5, the output of which is connected to the third input of the element-wise multiplier 5 and the input of the delay unit 6.6; the second output of the input implementation unit 1 is connected to the input of the depth card 7 storage unit, the output of which is connected to the input of the mask forming unit for the depth card 8, the output of which is connected to the input of the depth card 9 recovery unit, the output of which is connected to the second input of the median estimation unit 10 ; the synchronization of the device is provided by the clock generator 12.

A device for reconstructing a depth map of a scene is implemented as follows. A depth map and a color image of one scene are received at the input of the device, which are further processed in two channels. In the first, reconstruction and filling of the areas of distorted values of the pixels of the depth map is performed, and in the second, the formation of areas of close gradation of brightness using the brightness channel is performed.

At each moment in time, one of the points of the original color image is selected. For it, there is an area with pixels of similar brightness using the inversion method.

Further, to restore damaged areas, reduce noise and improve image quality, depth median filtering is used in adaptive window combinations based on the rule of crossing confidence intervals compiled using a color image. Median filtering is applied in each combined adaptive region for all pixels of the depth map.

EFFECT: reducing the error in determining the distance from scene objects to the sensor camera using image processing of the depth map with noise suppression and restoration of the lost sections of the depth map.

Claims (1)

A scene depth map recovery device comprising an input implementation storage unit, the input of which is the information input of the device; a block for determining quasistationary plots, the output of which is connected to the input of the mask forming unit, to the output of which the second input of the element-wise multiplier is connected; the output of the median estimation calculation unit is connected to the input of the useful component evaluation storage unit, whose output is the information output of the device; the output of the current line counter is connected to the second input of the block determining quasistationary sections and the input of the delay unit, the fourth input of the element-wise multiplier and the input of the delay unit are connected to its output; the counter output of the current column is connected to the third input of the block for determining quasistationary sections and the input of the delay unit, to the output of which the third input of the element-wise multiplier and the input of the delay unit are connected; synchronization of the operation of the device is ensured by a clock generator, characterized in that the first output of the input implementation storage unit is connected to the input of the color image storage unit, the output of which is connected to the first input of the quasi-stationarity detection unit and the first input of the element-wise multiplier, the output of which is connected to the first input of the calculation unit median score; the second output of the input implementation storage unit is connected to the input of the depth map storage unit, the output of which is connected to the input of the depth map mask forming unit, the output of which is connected to the input of the depth map recovery unit, the output of which is connected to the second input of the median estimation calculation unit.

Images