RU2360287C1 - Method of space-time anisotropic bilateral filtration of video signals - Google Patents

Abstract

FIELD: information technology.

SUBSTANCE: present invention relates to methods and devices for processing video signals and images, and particularly to methods and devices for suppressing noise in video signals and images. The method involves selecting the first region around the current pixel; selecting two lines (horizontal and vertical) through the current pixel and pixels in the region of the current pixel; determining the horizontal gradient; determining the vertical gradient; determining the correlation matrix for the horizontal and vertical gradients, as well as eigenvectors and eigen values of the correlation matrix; determining the number of contours; determining direction of the object contour into the image; calculating weighted coefficients of the time filter; calculating adaptation coefficients of the time and bilateral filters to the power of contours, carrying out time filtering; determining the second region; dividing the second region into two sub-regions along the direction of the contour; carrying out bilateral filtration for the first and second sub-regions; calculating the result of bilateral filtration as a weighted sum of results of filtering the first and second sub-ranges; carrying out approximation of the current pixel using pixels located along the direction of the contour; calculating the final result of filtration as a weighted sum of results of bilateral filtration and approximation along the contour.

EFFECT: wider functionality and improved quality of suppressing different types of noises.

5 cl, 2 dwg

Description

The invention relates to methods and devices for processing video signals and images, and in particular to methods and devices for suppressing noise in video signals and images.

The problem of noise suppression in video signals and images is one of the most important tasks in the field of processing video signals and images. Noise reduction can be used both to improve video signals and images, and for specific pre-processing of video signals and images. There are many methods to solve the noise reduction problem. Many good noise reduction solutions are based on complex iterative procedures, but they are not suitable for processing video signals due to excessive computations. Among non-iteration methods, the bilateral filtration method shows good results, while the method has moderate complexity. A bilateral filter is a non-linear filter in which the weight of each pixel is calculated by multiplying the core of a low-pass filter, such as a Gaussian, in the spatial domain by the influence function in the brightness domain, which reduces the weight of the pixel with a large difference in brightness. This filter preserves details very well compared to filters operating only in the spatial domain.

One effective solution for bilateral filtration is the method proposed in US patent No. 7146059 [1]. In the described method perform fast bilateral filtering and use it to display images with a large dynamic range. Separate the image into a base layer, which encodes large-scale changes, and a detail layer. In the base layer, image contrast is reduced while maintaining image details. The base layer is obtained using a bilateral filter that preserves faces. Acceleration of bilateral filtering is obtained using a piecewise linear approximation in the luminance region and corresponding sampling. The disadvantages of the method include the fact that the method is designed to filter images and does not take into account the temporal nature of video signals. Applying this method to video signals will cause a flicker effect. Also, the method does not suppress certain types of noise, such as compression artifacts and camera sensor noise.

U.S. Patent Application Laid-Open No. 2005/0025378 [2], which is closest in concept to the present invention, describes a bilateral filtering method in which a pixel signal is corrected by adding a bilateral correction term to the original pixel signal. The bilateral correction term is a function of the local differences of the signal between the central pixel of the filter core and its neighbors. The bilateral correction term is calculated using an efficient, from the point of view of calculations, approximation based on the expansion in the Taylor series of the normalization expression. The disadvantages of the method include the fact that the method is designed to filter images and does not take into account the temporal nature of video signals. Applying this method to video signals will cause a flicker effect. Also, the method does not suppress certain types of noise, such as compression artifacts and camera sensor noise.

The problem to which the claimed invention is directed is to develop a method of adaptive spatial-temporal anisotropic bilateral filtering of video signals with expanded functionality, namely, with the possibility of processing video frames in accordance with their structure, calculated by calculating the direction of the contours and their strength, with the possibility of adaptation filtering level depending on the image structure, with the ability to suppress various kinds of noise, with the possibility of noise reduction as in ranstvennoy domain and time domain, to implement a pipelined in parallel computing structures with higher quality and noise reduction, which does not occur artifacts in the final filtered image.

The problem is solved by creating a new method of adaptive spatio-temporal anisotropic bilateral filtering of video signals, providing for the following operations for each current image pixel:

- select around the current pixel the first region consisting of the current pixel and several nearest neighboring pixels of the image;

- choose two lines, one of which runs horizontally (horizontal line), and the other vertically (vertical line) through the current pixel and pixels of the current pixel region;

- determine the horizontal gradient along the horizontal line of the region of the current pixel as the difference between the pixel located to the left of the current pixel and the pixel located to the right of the current pixel;

- determine the vertical gradient along the vertical line of the region of the current pixel as the difference between the pixel located above the current pixel and the pixel located below the current pixel;

- determine the correlation matrix for horizontal gradients and vertical gradients of the current pixel region, as well as eigenvectors

θ _l , θ _s and eigenvalues λ _l , λ _{s of the} correlation matrix;

- determine the strength of Edge_Power contours using the formula

where n is a number that specifies the sensitivity to the contour of the object in the image;

- determine the direction Ω of the contour of the object in the image as the angle between the eigenvector θ _s , the correlation matrix with a smaller value of the eigenvalue and the horizontal line;

- for each frame from the selected number of neighboring frames, including the current frame, select around the pixel corresponding to the current pixel, the search area consisting of the current pixel and the nearest neighboring pixels of the image;

- calculate the weighting coefficients of the time filter by comparing the first region (window with a size of K × K) in the current frame with areas of the same size in the search area (window with a size of N × N) in the current and neighboring frames;

- calculate the adaptation coefficients of the temporary and bilateral filters to the strength of the circuits. The adaptation coefficient σ _{t of the} temporary filter to the loop energy is calculated by the formula

,

,

where k _t is an adjustable parameter. For the adaptation coefficient σ _{t of the} time filter to the loop energy, any other concave function can also be used. The adaptation coefficient σ _{r of the} bilateral filter to the loop energy is calculated by the formula

,

,

where k _b is an adjustable parameter, and for the value of σ _r instead of the square root function it seems possible to use any concave function;

- perform filtering in time by finding the weighted sum of pixels that are in the search area in the current and neighboring frames. The filtering result is stored in the out_pixelT variable;

- define the second region as the region of the current pixel in which the current pixel is assigned the value out_pixelT;

- divide the second region into two subregions along the direction Ω - the first and second subregions (Figure 2);

- perform bilateral filtering for the first and second subdomains according to the following formula

where I _i is the intensity of the point with index i,

I _p is the intensity of the point for which filtering is performed (the center point of the mask). As h (spatial), any low-pass filter core can be used. We used Gaussian as such a core;

- calculate the result of bilateral filtering as a weighted summation of the filtering results of the first and second subregions;

- perform the approximation of the current pixel by pixels located along the direction of the path, while the approximation is the arithmetic average of the pixels adjacent to the current pixel located along the path;

- calculate the final filtering result as a weighted sum of the results of bilateral filtering and approximation along the contour.

The technical result of the claimed invention is to expand the functionality and improve the quality of noise suppression of various kinds, such as compression artifacts, camera sensor noise due to image processing both in the spatial domain and in the time domain, and filter adaptation to the image structure by calculating the direction of the contour and dividing the filter mask into two parts along the contour, followed by filtering both parts separately.

The main idea of the claimed method is to replace the value of the initial intensity of the current pixel with a function of the intensity of the surrounding pixels located in the current frame and adjacent frames. In the method, a weighted sum of the values of the initial intensities of pixels of adjacent frames and the current frame is determined. In this case, the weight is determined as a function of differences in the intensities of the pixels that make up the region around the current pixel and the pixel for which the weight is calculated. Then, the result of the weighted summation is again weightedly summed with the initial intensities of the surrounding pixels located in the current frame. The weight is determined through the function of the low-pass filter in the spatial and brightness region. The degree of filtration of the low-pass filter is adapted to the strength of the circuits.

The claimed method is performed in real time. This means that the time it takes to process a single image is less than the time it takes to change images in a video sequence of images.

The method does not require pixel data of the entire image in order to filter the current pixel - simultaneously process image areas of a few pixels.

For a better understanding of the claimed invention the following is a detailed description with the corresponding drawings.

Figure 1 - diagram of a step-by-step implementation of the method of adaptive spatial-temporal anisotropic bilateral filtering of video signals, made according to the invention.

Figure 2 - separation mask bilateral filter along the contour.

Consider the step-by-step implementation of the claimed method (Figure 1).

First, the first region consisting of the current pixel and the nearest neighboring pixels of the image is selected around the current pixel (Step 1). Then, for each (current) image pixel, the following operations are performed.

.The orientation parameters and contour strengths are calculated (Step 2). These parameters include the direction of the contour and 2 values of λ _s and λ _l , where λ _s is the derivative in the α direction and λ _{l is the} derivative in the direction

.

The horizontal and vertical derivatives for the current pixel are calculated as

and

respectively.

Derivatives are considered as random variables. Then, the following values are calculated for the region of the current pixel

These values make up the covariance matrix of a random variable. The smaller eigenvalue of the covariance matrix is calculated as

and the larger eigenvalue of the covariance matrix as

и

соответственно. Определяют силу контура Edge_Power по формулеThe projections of the direction vector of the contour on the horizontal and vertical directions are calculated as

and

respectively. The Edge_Power contour strength is determined by the formula

where n is a number specifying the sensitivity to the contour of the object in the image.

Then, for a selected number of neighboring frames, including the current frame, for each frame, a region consisting of several nearest neighboring image pixels, the so-called search area, is selected around the pixel corresponding to the current pixel (Step 3).

Next, weights for the time filter are calculated (Step 4). Weights are calculated by comparing the first region (window with a size of K × K) in the current frame with regions of the same size in the search area (window with a size of N × N) in the current and neighboring frames. To compare the areas using measure D

where inp (x _n , y _n , t _n ) denotes a pixel from the current frame t _n with coordinates x _n and y _n ,

inp ₂ (p, s, t _n + r) denotes a pixel from the frame t _n + r with coordinates p and s,

до

в горизонтальном направлении,index i varies from

before

in the horizontal direction

до

в вертикальном направлении.index j varies from

before

in the vertical direction.

,Then, the adaptation coefficients of the temporary and bilateral filters to the loop strength are calculated (Step 5). The adaptation coefficient σ _{t of the} temporary filter to the loop energy is calculated by the formula

,

,where k _t is an adjustable parameter. For the adaptation coefficient σ _{t of the} time filter to the loop energy, any other concave function can also be used. The adaptation coefficient σ _{r of the} bilateral filter to the loop energy is calculated by the formula

,

where k _b is an adjustable parameter. For the quantity σ _r, instead of the square root function, any concave function can be used. Moreover, the behavior of the coefficient σ _r can be described as follows. For image areas that do not contain pronounced contours, the degree of bilateral filtering increases. The Edge_Power contour strength is close to 1. If the Edge_Power contour strength is close to 0, the area with the current pixel contains a pronounced contour and the degree of bilateral filtering is reduced.

Next, filtering in time is performed (Step 6). Filtering consists in calculating the weighted sum of the pixels that are in the search area in the current and neighboring frames. The filtering result depends on the number of neighboring frames used for processing. The more frames you use, the better the filter result. The result of temporary filtering is obtained as follows:

обозначает пиксель из кадра t_n+r с координатами x_n+p и y_n+s,Where

denotes a pixel from the frame t _n + r with coordinates x _n + p and y _n + s,

до

в горизонтальном направлении,the index p varies from

before

in the horizontal direction

до

в вертикальном направлении,the index s varies from

before

in the vertical direction

до

во временной области,the index r varies from

before

in the time domain,

T is the number of neighboring frames used for temporal filtering,

w (p, s, r) - pixel weight inp ₂ (x _n + p, y _n + s, t _n + r), which is calculated as

σ _t is the coefficient of adaptation of the temporary filter to the energy of the circuit.

The normalization factor is calculated by the formula

The second region is determined as the region of the current pixel in which the current pixel is set to out _pixelT (Step 7).

Divide the second region into two subregions along the direction Ω - the first and second subregions (Figure 2) (Step 8).

Bilateral filtration of the first and second subregions is performed according to the following formula (Step 9 and 10)

where I _i is the intensity of the point with index i,

I _p is the intensity of the point for which filtering is performed (the center point of the mask).

As h (spatial), any low-pass filter core can be used. We used Gaussian as such a core.

The result of bilateral filtering is calculated through the weighted sum of the filtering results of the first and second subregions (Step 11).

where bil_pixel denotes the result of adaptive bilateral

spatial filter

filt_pixel1 denotes the result of filtering the first subdomain,

filt_pixel2 indicates the result of filtering the second subarea,

w ₁ - the weight of the filtering result of the first subdomain is calculated as follows

w ₂ - the weight of the filtering result of the second subdomain is calculated similarly as w ₁

where sum_grad ₁ is the sum of the gradients calculated along the contour from the side of the first subregion,

sum_grad ₂ - the sum of the gradients calculated along the contour from the side of the second subregion.

Next, the current pixel is approximated by pixels located along the direction of the path (Step 12). Approximation is a combination

int_pixel = (Idir _- + Idir ₊ ) / 2,

where Idir _- and Idir ₊ neighboring pixels located along the path.

Ultimately, the final filtering result is calculated as

filt_pixel = wbil_pixel + (1-w) int_pixel (Step 13), where

Th denotes the threshold, which is determined as a result of learning the algorithm,

int_pixel denotes the result of approximating the current pixel along the path,

filt_pixel indicates the final filter result.

The claimed method can be implemented in a system consisting of functional elements.

The invention can be applied in industrially manufactured devices, such as video cameras, television devices, displays of mobile devices, etc., working with video signals.

Although the above embodiment of the invention has been set forth to illustrate the present invention, it is clear to those skilled in the art that various modifications, additions and substitutions are possible without departing from the scope and meaning of the present invention disclosed in the attached claims.

Claims (5)

1. The method of adaptive spatio-temporal anisotropic bilateral filtering of video signals, in which for each (current) image pixel the following operations are performed:
around the current pixel, select a first region consisting of the current pixel and several nearest neighboring pixels of the image;
two lines are selected, one of which extends horizontally (horizontal line) and the other vertically (vertical line) through the current pixel and the pixels of the current pixel region;
determining a horizontal gradient along the horizontal line of the region of the current pixel as the difference between the pixel located to the left of the current pixel and the pixel located to the right of the current pixel;
determining a vertical gradient along the vertical line of the region of the current pixel as the difference between the pixel located above the current pixel and the pixel located below the current pixel;
determining a correlation matrix for horizontal gradients and vertical gradients of the current pixel region, as well as eigenvectors θ ₁ , θ _s and eigenvalues λ ₁ , λ _{s of the} correlation matrix;
determine the strength of Edge-Power circuits using the formula

where n is a number that specifies the sensitivity to the contour of the object in the image;
determine the direction Ω of the contour of the object in the image as the angle between the eigenvector θ _{s of the} correlation matrix with a smaller value of the eigenvalue and the horizontal line;
for each frame from a selected number of neighboring frames, including the current frame, select around the pixel corresponding to the current pixel, a search area consisting of the current pixel and the nearest neighboring pixels of the image;
calculating the weight coefficients of the time filter by comparing the first region (window with a size of K × K) in the current frame with regions of the same size in the search area (window with a size of N × N) in the current and neighboring frames;
the coefficients of adaptation of the temporary and bilateral filters to the loop power are calculated, and the adaptation coefficient σ _{t of the} temporary filter to the loop energy is calculated by the formula

,
where k _t is an adjustable parameter, and the adaptation coefficient σ _{r of the} bilateral filter to the loop energy is calculated by the formula

,
where k _b is an adjustable parameter;
filtering in time by finding the weighted sum of pixels that are in the search area in the current and neighboring frames, and the filtering result is stored in the variable out_pixelT;
determining a second region as the region of the current pixel in which the current pixel is assigned the value out_pixelT;
divide the second region into two subdomains along the direction Ω — the first and second subdomains;
- perform bilateral filtering for the first and second subdomains according to the following formula:

where I _i is the intensity of the point with index i, I _p is the intensity of the point for which filtering is performed, and the core of the low-pass filter is used as h (spatial);
calculate the result of bilateral filtering, as a weighted summation of the filtering results of the first and second subregions;
approximating the current pixel by pixels located along the direction of the contour, the approximation being the arithmetic average of the pixels adjacent to the current pixel located along the contour;
calculate the final filtering result as a weighted sum of the results of bilateral filtering and approximation along the contour. 2. The method according to claim 1, characterized in that a measure is used to compare the areas of pixels

where inp ₁ (x _n , y _n , t _n ) denotes a pixel from the current frame t _n with coordinates x _n and y _n ,
inp ₂ (p, s, t _n + r) denotes a pixel from the frame t _n + r with coordinates p and s,
index i varies from

before

in the horizontal direction, the index j varies from

before

in the vertical direction. 3. The method according to claim 1, characterized in that the temporary filtering is performed as follows:

where inp ₂ (x _n + p, y _n + s, t _n + r) denotes a pixel from the frame t _n + r with coordinates x _n + p and y _n + s,
p index varies from

before

in the horizontal direction
the index s varies from

before

in the vertical direction
the index r varies from

before

in the time domain,
T is the number of neighboring frames used for temporal filtering,
w (p, s, r) - pixel weight inp ₂ (x _n + p, y _n + s, t _n + r), which is calculated as

σ _t is the coefficient of adaptation of the temporary filter to the energy of the circuit, while the normalization factor is calculated by the formula

4. The method according to claim 1, characterized in that the result of bilateral filtering is calculated through a weighted summation of the filtering results of the first and second subregions, as
bil_pixel = w ₁ filtrationpixel ₁ + w ₂ filtrationpixel ₂ ,
where bil-pixel denotes the result of an adaptive bilateral spatial filter,
flit-pixel1 indicates the result of filtering the first subarea,
filt-pixel2 indicates the result of filtering the second sub-area,
w ₁ - the weight of the filtering result of the first subdomain is calculated as follows:

w ₂ - the weight of the filtering result of the second subdomain is calculated similarly as w ₁ ,

where sum_grad ₁ is the sum of the gradients calculated along the contour from the side of the first subdomain, sum_grad ₂ is the sum of the gradients calculated along the contour from the side of the second subdomain. 5. The method according to claim 1, characterized in that the final filtering result is calculated as
filt_pixel = w bil_pixel + (1-w) int_pixel,
Where

Th denotes the threshold that is determined as a result of learning the algorithm, bil_pixel denotes the result of bilateral filtering, int_pixel denotes the result of approximating the current pixel along the contour, filt_pixel denotes the final filtering result.

Images