RU2393540C2 - Method for increased resolution of digital video sequence - Google Patents

Abstract

FIELD: information technologies.

SUBSTANCE: method for increasing dimension of digital image provides for frame-by-frame processing of video sequence. Besides each pixel of frame is processed taking into account information on the pixel itself and on local area around it, performing the following operations simultaneously: directions of borders in this pixel are identified. Frequency characteristics of processed section are identified. Type of interpolation applied to this pixel is identified. Interpolated value of pixel is calculated on the basis of data on frequency characteristics of processed section and directions of borders. Operation of borders improvement is executed by results of pixel value interpolation. Effect of multiplication is removed by means of final processing with application of function that adapts to proportions of interpolation.

EFFECT: increased dimension of image four and more times without considerable distortions.

8 cl, 3 dwg, 1 tbl

Description

The invention relates to methods for processing video data, and more particularly to methods for increasing the dimensionality of video sequences.

The problem of increasing the dimensionality of images is especially relevant in high definition television (HDTV). There are several effective ways to solve the problem of increasing the resolution of images, but there are difficulties with their application and adaptation to video sequences, when it is necessary to take into account the temporal characteristics of the video sequence. In known methods, interpolation is most often used, based only on the spatial characteristics of the video sequence.

One of these interpolation approaches uses the wavelet analysis method, which is the basis for a large number of video coding and decoding algorithms. In the process of increasing the dimension, wavelets are used as follows. Image decompose using wavelet transform. Then, each set of coefficients is refined to double the size of the incoming image and the inverse wavelet transform is applied.

One of the successful solutions is described in US patent No. 7116836 [1]. In the described technical solution, image data identifying a plurality of image pixels is obtained, and based on these data, a hierarchical structure is formed based on the degree of sharpness (resolution). Then, the output (outgoing) values for the pixels belonging to the image are calculated by extending the pixel values calculated at the low resolution levels of such a hierarchy to higher resolution levels, followed by the refinement of the pixel values thus distributed at the higher resolution levels. The main disadvantage of this method is its high computational complexity and difficulties with hardware implementation.

Closest to the claimed invention is a technical solution described in the patent of the Russian Federation No. 2310911 [2], where a method for increasing the dimension of the image, based on the construction of a map of the directions of the borders on the image. This map of the directions of the edges is processed using morphological operations to remove false directions and reduce the number of possible distortions. Interpolation of each image pixel is carried out in three stages: the first two are carried out along the coordinate axes of the image, and the third forms a certain angle with the coordinate axes of the image, and this angle depends on the value on the map of the direction of the boundaries. This method allows you to suppress jaggies, in contrast to conventional bicubic interpolation. A subsequent sharpening is to improve local contrast based on the brightness conversion curve, with the resulting sharpening depending on the degree of smoothness of the curve. The main disadvantage of this method is the manifestation of the effect of the animation on the interpolated image. Another problem is the appearance of false textures in high-frequency areas.

The problem to which the invention is directed is to develop such a method for interpolating a video sequence that would provide an increase in image size four or more times without noticeable distortion.

The problem is solved by developing a new method, which involves the use of frame-by-frame processing of a video sequence, and each pixel in the frame is processed taking into account information about the pixel itself and the local area around it, performing the following operations:

- determine the direction of the borders in this pixel;

- determine the frequency characteristics of the treated area;

- determine the type of interpolation applicable to the given pixel;

- calculate the interpolated pixel value based on the data on the frequency characteristics of the processed area and the directions of the borders;

- perform an operation to improve the boundaries based on the results of interpolation of the pixel value;

- eliminate the effect of multiplication by additional processing using a function that adapts to the proportions of the interpolation.

When implementing the proposed method, it is important that the frequency characteristics of the processed area was determined using wavelets, and wavelet 5-3 is used to calculate the frequency response of the area, after which the type of area is determined based on threshold values.

When implementing the proposed method, it is important that one of the two types of interpolation is used, namely, if the processed section is not high-frequency, then interpolation based on triangulation with preservation of boundaries is used, if the section is high-frequency, then bicubic interpolation is used. In the intermediate region between the low-frequency and high-frequency sections, the method of the weighted sum of triangulation interpolation and bicubic convolution is used.

When implementing the proposed method, it is advisable to apply additional processing to prevent the effect of blurring (smoothing) of the boundaries, namely, that they use a function that adapts to the proportion of interpolation. This eliminates the effect of animation and ensures the naturalness of the video sequence with increased dimension, even at high magnification (four to five times).

Essentially, the inventive method provides for the use of a combination of a wavelet and a triangulation interpolation method with preserving borders to increase the dimensionality of video sequences. In this case, the main advantage of the proposed method is to obtain a good visual quality of the processed video sequence, especially in the application to broadcasting and amateur videos. In addition, the inventive method is characterized by low requirements for computing resources, ensures the preservation of borders and the clarity of small image details, prevents the appearance of distortion and flickering (flickering) and the effect of animation.

Further, the claimed method is illustrated with the use of graphic materials.

Figure 1 - Block diagram of the main steps of the algorithm according to the invention.

Figure 2 - Scheme for determining the type of pixel interpolation.

Figure 3 - Diagram for determining the type of direction.

The claimed invention provides an increase in the dimension of the video sequence without noticeable distortion due to the use of wavelet analysis and image interpolation by the method of triangulation with preserving borders.

The video sequence is assumed to be in the YCbCr format. This video sequence is processed frame by frame (frame-by-frame processing), wherein each frame is considered as a set of pixels, and each pixel is processed using information about neighboring pixels, i.e. about pixels located in the area surrounding the processed pixel.

The main steps of the proposed method are shown in figure 1.

In step 101, the frequency type of the region is determined using wavelets and determining the direction of the boundaries. Wavelet 5-3 is used in the horizontal and vertical directions to calculate the number corresponding to the frequency. Then apply the threshold principle to determine the type of site. Also at this step, the direction of the border in a given pixel is determined using the value of the derivatives in the directions. Also, the pixel belongs to the "chess" area. Then, according to these values, the type of interpolation is selected.

After that, the interpolated pixel value is calculated based on the information about the frequency of the image section and the boundaries in this pixel (step 102). To increase the dimension, two types of interpolation are used, namely, the triangulation method with preserving the boundaries is used for the low-frequency section, bicubic interpolation (convolution) is used for high-frequency sections. In the intermediate region between the low-frequency and high-frequency sections, the method of the weighted sum of triangulation interpolation and bicubic convolution is used.

To prevent blurring of the borders after interpolation, an additional operation to improve the boundaries is performed (step 103). This additional operation is performed based on a function that adapts to the proportion of interpolation to prevent manifestations of the multiplier effect.

In more detail, step 101 is shown in FIG. In step 201, the horizontal derivative for the pixels of the original image is calculated. This derivative is calculated based on the relation (1) below, as the average value of the differences between pixels that are adjacent in the vertical direction:

where I _Y (i, j) is the value of channel Y of the current pixel.

In step 202, the vertical derivative for the pixels of the original image is calculated. This derivative is calculated based on the relation (2) below, as the average value of the differences between pixels that are adjacent in the horizontal direction:

where I _Y (i, j) is the current pixel.

In step 203, the derivative in the secondary diagonal direction for the pixels of the original image is calculated. This derivative is calculated based on the relation (3) below as the average value of the differences between pixels that are adjacent in the direction of the secondary diagonal:

where I _Y (i, j) is the current pixel.

At step 204, the derivative in the direction of the main diagonal for the pixels of the original image is calculated. This derivative is calculated based on the relation (4) below, as the average value of the differences between the pixels that are adjacent in the direction of the main diagonal:

where I _Y (i, j) is the current pixel.

In step 205, a sub-band of the LV wavelet 5-3 is calculated for the current pixel. The wavelet value is calculated based on the following relationship:

This value characterizes the frequency type of the region to which the pixel being processed belongs: if this value is large, then this pixel belongs to the high frequency region. At step 206, it is additionally checked for each of the color channels whether the given pixel belongs to the “chessboard” area:

where T is a channel, T∈ {Y, Cb, Cr}.

where T is a channel, T∈ {Y, Cb, Cr}.

CHESS_UP_TR, то данный пиксель считается принадлежащим области «шахматная доска», т.е. интерполируется как пиксель высокочастотной области.Then, if A _T <CHESS_TR and W <CHESS_TR, where T is at least one of the channels, and CHESS_TR is a threshold value for all frequency domains, and

CHESS_UP_TR, then this pixel is considered to belong to the “chessboard” area, i.e. interpolated as a pixel in the high frequency region.

At step 207, the most probable direction of this pixel is calculated. The definition of direction consists of two parts: the definition of direction (direction) and the definition of sub-direction. Step 207 is presented in detail in FIG. 3. At step 301, a minimum of derivatives is found to determine the main direction M = min {D _x , D _y , D _d1 , D _d2 }. Then, at step 302, depending on the found minimum, the corresponding direction is established (for example, with M == D _x dir = 0). At step 303, four conditions are calculated:

&&

&&

&&

&&

&&

&&

&&

&&

These conditions are associated with the specification of the direction. SUB_DIR_TR is a predetermined (i.e., given) threshold value used to calculate the difference between two derivatives. Then, in step 304, it is checked whether any of the conditions C _{i is} fulfilled. If so, then one of the directions is changed to a subdirection. At step 305, it is checked whether the number of possible sub-directions exceeds one, i.e. more than one condition is met. If so, then choose a sub-direction, where the difference in the absolute values of the two derivatives is less. After this step, one of the eight possible border directions is assigned to the pixel being processed.

At step 208, it is determined which type of interpolation should be applied to the given pixel. At the input of this step, three values are presented: the possible direction of the boundary, the wavelet value and the “chess” value. The pixel interpolation type is assigned index 8 if the wavelet is less than the predetermined threshold values, or if the pixel belongs to the checkerboard region. Otherwise, the interpolation type is assigned an index equal to the direction number of the pixel border.

After determining the type of pixel interpolation, the pixel value is interpolated based on the triangulation and the detected pixel interpolation type, step 102. If the interpolation type is 8, then the pixel is interpolated using a simple bicubic convolution kernel.

If the type of interpolation is less than eight, then the interpolation is performed using the following mathematical expressions:

where Δ _{s, 1} and Δ _{2 are} determined using the following formulas.

Let δ _x be the distance from the current pixel to the nearest left row (column) of the original image, while the distance between the two nearest rows (columns) of the original image is taken as unity. Let δ _y be the distance from the current pixel to the nearest top line of the original image, while the distance between the two nearest lines of the original image is taken as unity:

where y _{s, k} is determined using the following formula, where I (i, j) is the left-top pixel of the source image closest to the interpolated pixel:

where the parameters (parameter) t _{s, k} and p _{s, k} are taken using the following table:

Tire / parameter one 2 3 (δ _y <δ _x ) 3 (δ _y ≥δ _x ) 4 (δ _y + δ _x ≤1) 4 (δ _y + δ _x > 1) 5 6 7 8 t _1,1 0 -one -one 0 -one one 0 0 -one -one t _1,2 0 0 0 0 0 one 0 0 0 0 t _1,3 0 one one 0 one one 0 0 one one t _1.4 0 2 2 0 2 one 0 0 2 2 t _2,1 one -one one -one 0 -one one one -one -one t _2.2 one 0 one 0 0 0 one one 0 0 t _2,3 one one one one 0 one one one one one t _2,4 one 2 one 2 0 2 one one 2 2 p _1,1 -one 0 0 -one 0 -one -one -one 0 0 p _1,2 0 0 0 0 0 0 0 0 0 0 p _1,3 one 0 0 one 0 one one one 0 0 p _1,4 2 0 0 2 0 2 2 2 0 0 p _2,1 -one one -one one -one one -one -one one one p _2.2 0 one 0 one 0 one 0 0 one one p _2,3 one one one one one one one one one one p _2,4 2 one 2 one 2 one 2 2 one one

Out - value of the current interpolated pixel of the resulting image.

Upon completion of the interpolation, additional processing is performed - step 103. At the first stage, the value of the animation parameter is determined to prevent such an effect from appearing on the image. This parameter depends on the proportion (degree) R of the increase in dimension in the following ratio:

At this stage of the additional processing, only the Y channel pixels are subjected to it. Find the minimum and maximum values of pixels located in the vicinity of the interpolated pixel Oyt _Y. At the same time, the interpolated pixel Oyt _{Y is} normalized to the range [0,1]:

находят следующим образом:Then processed value

found as follows:

where f is the following function:

Instead of f (x), other sharpening functions are also possible.

рассматривают как значение интерполированного пикселя, прошедшего дополнительную обработку и являющегося исходящим пикселем в заявляемом способе увеличения размерности.In this way,

considered as the value of the interpolated pixel that has undergone additional processing and is an outgoing pixel in the claimed method of increasing the dimension.

The claimed invention can find application in industrial installations, the input of which receives images with low resolution. First of all, it is digital television, including high-definition television (HDTV), the input of which receives video sequences in standard resolution, requiring a qualitative increase in resolution. Another application could be display mobile devices using mpeg compression systems. Moreover, the claimed method allows to successfully increase the dimension of color components when restoring the YCbCr 4: 4: 4 format from the YCbCr 4: 2: 0 or 4: 2: 2 formats.

Claims (8)

1. A method of increasing the dimension of a digital image, providing for frame-by-frame processing of a video sequence, each pixel of the frame being processed taking into account information about the pixel itself and the local area around it, performing the following operations:
determine the direction of the borders in this pixel;
determine the frequency characteristics of the treated area;
determine the type of interpolation applicable to a given pixel;
calculating the interpolated pixel value based on data on the frequency characteristics of the treated area and the directions of the borders;
performing an operation to improve boundaries based on the results of interpolation of the pixel value;
eliminate the effect of multiplication by final processing using a function that adapts to the proportions of the interpolation. 2. The method according to claim 1, characterized in that the type of applicable interpolation is determined using wavelets, determining the direction of the boundaries in a given pixel, determining the type of region and determining belonging to the "chess" region, and first of all, calculate the number corresponding to the frequency characteristics of the processed plot using wavelet 5-3, which is used in horizontal and vertical directions; determine the direction of the boundaries in a given pixel using the value of the derivatives in the directions; determine the type of the treated area based on the threshold principle. 3. The method according to claim 2, characterized in that to determine the direction of the boundaries in the current pixel, the values of derivatives calculated using the following formulas are used:
the horizontal derivative for the pixels of the original image is calculated based on the relation below, as the average value of the differences between pixels that are adjacent in the vertical direction:

is the value of channel Y of the current pixel;
the vertical derivative for the pixels of the original image is calculated based on the relation below, as the average value of the differences between pixels that are adjacent in the horizontal direction:

is the current pixel;
the derivative in the direction of the secondary diagonal for the pixels of the original image is calculated, based on the ratio below, as the average value of the differences between pixels that are adjacent in the direction of the secondary diagonal:

is the current pixel;
the derivative in the direction of the main diagonal for the pixels of the original image is calculated based on the relation below, as the average value of the differences between the pixels that are adjacent in the direction of the main diagonal:

is the current pixel. 4. The method according to claim 3, characterized in that the determination of one of eight possible directions of the border in the current pixel is performed using the following operations:
find a minimum of derivatives to determine the main direction,
depending on the found minimum, set the corresponding value of the variable dir, which indicates the direction of the border,
- calculate four conditions:

&&

;

&&

;

&&

;

&&

,
where SUB_DIR_TR is a predetermined threshold value used to calculate the difference between the two derivatives, D _x , D _d1 and D _d2 are derivatives in the horizontal direction and in the direction of the main and secondary diagonals, respectively, the value of the variable dir is calculated in the previous step,
changing one of the directions to a sub-direction, if any of the conditions C _{i is} fulfilled;
if more than one condition is satisfied, then choose a sub-direction, where the difference in the absolute values of the two derivatives is less. 5. The method according to claim 2, characterized in that to determine the frequency response of the processed area using wavelet transform, performed by the formula

where T is a channel, T∈ {Y, Cb, Cr}, where Y, Cb and Cr are the color channels of the image in the color space representation of YCbCr. 6. The method according to claim 2, characterized in that the interpolation is performed using the following mathematical expressions:

,
where Δ _{s, 1} and Δ _{2 are} determined using the following formulas in which δ _x is the distance from the current pixel to the nearest left row (column) of the original image, while the distance between the two nearest rows (columns) of the original image is taken as unity, and δ _у is the distance from the current pixel to the nearest top line of the source image, while the distance between the two nearest lines of the source image is taken as unity:

where dir is the direction of the border, which is determined in paragraph 4;
and y _{s, k} is determined using the following formula, where I (i, j) is the left-top pixel of the original image closest to the interpolated pixel:
y _{s, k} = I (i + t _{s, k} + p _{s, k} ), s = 1,2, k = 1, ... 4,
where the parameters (parameter) t _{s, k} and p _{s, k} take values using the following table:

Type - indicates the type of interpolation applied to the image pixels;
Out - value of the current interpolated pixel of the resulting image. 7. The method according to claim 1, characterized in that on the basis of the frequency characteristics of the processed pixel sections, one of two types of interpolation is used, namely, if the processed section is defined as low-frequency, then interpolation based on triangulation with preserving borders is applied, and if the section is high-frequency, then bicubic interpolation is used, while in the intermediate region between the low-frequency and high-frequency sections, the method of the weighted sum of triangulation interpolation and bicubic is used Coy convolution. 8. The method according to claim 1, characterized in that the additional operation to improve the boundaries includes the following steps:
determine the value of the animation parameter, which depends on the proportion R of the increase in dimension in the following ratio:

moreover, at this stage, only the channel Y pixels are subjected to processing,
find the minimum and maximum values of pixels located in the vicinity of the interpolated pixel Outy,
Outy interpolated pixel normalize to the range [0,1]:

where local_min and lokal_max are the minimum and maximum allowable values for the image pixel.
- find the processed value

in the following way:

where f is the following function:

Images