EP1157558A1

EP1157558A1 - Blocking artifact reduction in the dct domain

Info

Publication number: EP1157558A1
Application number: EP00981258A
Authority: EP
Inventors: Santhana Krishnamachari; Mohamed Abdel-Mottaleb
Original assignee: Koninklijke Philips Electronics NV
Current assignee: Koninklijke Philips NV
Priority date: 1999-11-30
Filing date: 2000-11-10
Publication date: 2001-11-28
Also published as: CN1338182A; KR20010101882A; JP2003516054A; WO2001041453A1; CN1191717C

Abstract

A method and device for removing blocking artifacts in the frequency domain. Two frequency domain blocks (A & B) are received in a video stream. A third frequency domain block is computed directly in the transform domain which overlaps the boundary between the first and second frequency domain blocks. The coefficients of the third block are modified in the frequency domain to remove blocking artifacts and then this modification is translated into a modification of the two frequency domain blocks (A and B) in the frequency domain.

Description

Blocking artifact reduction in the DCT domain.

The invention relates in general to the reduction of blocking artifacts in image data and in particular to a method and device for reducing blocking artifacts in image data using only frequency domain operations.

The compression of an image allows an image to be transmitted in a coded form over a communication channel using fewer bits of data than if it were transmitted in its uncompressed form. Discrete Cosine Transform (DCT) coding is one well-known compression scheme. The image is divided into small rectangular regions or "blocks". Each block is transform coded and transmitted through a communications channel. At the receiver, the blocks are decoded and reassembled into the original image. Typically these blocks are formed by an array of 8X8 pixels. The DCT is applied to each block and then each block is quantized.

The DCT is a linear transformation that creates a new block of pixels; each new pixel is a linear combination of all the incoming pixels of the original block. The DCT based block image coding technique causes degradation of the received image in the form of blocking artifacts. When an image is block coded the reconstructed blocks can be visible in the reconstructed image resulting in the viewer seeing the block boundaries, which in general are due to uncorrelated quantization noise. Quantization noise is independent between blocks yielding a jump or step at the block boundaries. Typically the greater the compression the greater the blocking artifacts. The blocking effect is quite visible to the viewer and can become quite annoying since the eye is so sensitive to the "step" at the block boundary.

This problem has been observed before and attempts to solve this problem have been proposed. Since the blocking artifact is caused by abrupt discontinuity between neighboring blocks, removing these discontinuities can reduce it. Many of the previously proposed solutions operate in the spatial (pixel) domain. One technique involves using a spatially-variant low-pass filter in the pixel domain at the block boundaries. The problem with such an approach is that it involves decompression of the image before the filter can be applied. Typically, however, it is useful to store the image in some compressed form after reduction of the blocking artifacts and therefore this technique requires that the image be recompressed.

A method and device for reducing blocking artifacts is provided which analyzes the DCT characteristics of the boundary between two blocks and reduces the blocking artifacts by smoothing out the abrupt discontinuities at the block boundaries in the frequency domain. A first frequency domain block (A) and a second frequency domain block (B) are received in the video stream. A third frequency domain block (C) is then computed which overlaps the first and second frequency domain blocks at the boundary between the first and second block. The third block is used to smooth the discontinuity at blocks A and B's boundary by adjusting the DCT coefficients of the third block. The computed change in the third block's coefficients is then translated into a change in coefficients of the first and second blocks (A, B). It is an object of the invention to perform this smoothing in the frequency domain only, by using the third block C.

It is an object of the invention that the change in the first and second blocks should satisfy the equation δC = δAM, + δBM₂ where δC is the change in the third block and δA and δB are the changes in the first and second blocks respectively and Mi and M₂ are matrices each having known values.

Yet another object of the invention is to choose, δA and δB to minimize the change to the first and second blocks.

Yet a further object of the invention is to make δA substantially opposite to δB.

It is yet another object of the invention to determine whether the first and second blocks have relatively uniform pixel values within each block with a jump in pixel value across the boundary between the two blocks.

Another object of the invention concerns blocks A and B when they are not relatively uniform. This object is achieved by reducing large DCT coefficients of high frequency components in the third block C.

Yet a further object of the invention is to change the coefficients of the third block C by reducing large high frequency coefficients in the third block C if their respective coefficients in the first and second blocks are substantially zero. Still other objects and advantages of the invention will in part be obvious and will in part be apparent from the specification.

The invention accordingly comprises the several steps and the relation of one or more of such steps with respect to each of the others, and the apparatus embodying features of construction, combinations of elements and arrangement of parts which are adapted to effect such steps, all as exemplified in the following detailed disclosure, and the scope of the invention will be indicated in the claims.

In general the embodiments of the invention pertain to reducing blocking artifacts by analyzing the DCT characteristics of the boundary between two blocks and smoothing the abrupt discontinuities at the block boundaries in the frequency domain. Two frequency domain blocks A and B are received in the video stream and a third block C is computed which overlaps the boundary between the two blocks A and B. The third block is used to smooth the discontinuity at block A and B's boundary by changing the DCT coefficients of the third block and translating this change into a change in coefficients of blocks A and B. This blocking artifact reduction is explained in a first instance with reference to two blocks A and B which each have uniform pixels values in the spatial domain but a step or jump in pixel value occurs across the blocks' boundary and in a second instance with reference to two blocks A and B which do not necessarily have uniform pixels in the spatial domain.

Figure la shows two 8X8 pixel blocks 'a' and 'b' and a third pixel block 'c' which overlaps 'a' and 'b' and covers the boundary 'e'. Figure lb shows an example of the elements of blocks 'a', ay and 'b', b in a first embodiment of the invention. In this case a is the same for all (i,j) and such cases will be denoted as θ where ay = θ for i = 0* 7 and j =

0- 7 (similarly by is the same for all (i,j) and in such cases will be denoted as _^ where by = -l for i = 0* 7 and j = 0^» 7) and ay and by will each denote a single pixel value hereinafter. At high compression, the block boundary 'e' between 'a' and 'b' will exhibit the step, i.e. all the pixel values ay, by are constant to the left and right of the boundary, respectively, with a large discontinuity at the boundary 'e'. Figure 2a shows how the pixel values in block 'c' change across the block. As can be seen from the graph in Figure 2a there is a step edge at column 4. This step edge causes the blocking artifact. To remove this blocking artifact, the step edge must be smoothed to a ramp edge such as the one shown in Figure 2b. The object of the present invention is to smooth the DCT of block 'c', C, in the frequency domain thereby removing the step in the middle of the block 'c' and translate this smoothing into a change in coefficients of DCT blocks A and B.

When receiving a bitstream that has been compressed using either JPEG or

MPEG compression, the DCT blocks A and B are readily available from the compressed bitstream after performing variable length decoding, but the DCT block C is not. Therefore, to smooth the DCT block C it is first necessary to compute the DCT coefficients of block C.

This computation does not merely involve merging the DCT values of the right hand portion of block A and the DCT values of the left hand portion of Block B because each DCT block is a linear combination of all the incoming pixels of a block, making the computation of the DCT block C quite complex. One way to find the DCT coefficients of block C is to convert blocks A and B into the spatial domain and compute block C by linearly combining all of the pixels of block 'c'. This is a cumbersome technique and requires decompressing the blocks.

In the present invention the DCT block C is obtained from DCT blocks A and B without performing inverse DCT. Since blocks 'a', 'b' and 'c' can simply be considered as matrices having elements consisting of pixel values ay, by, respectively, the matrix 'c' can be written in terms of matrices 'a' and 'b' as follows:

c= aK, + bK₂ (1)

where Ki and K₂ are 8X8 matrices. The matrix Ki is given by:

00

K, = 10

where 0 is a 4X4 matrix of all zeros and I is a 4X4 matrix with all diagonal entries equal to 1 and the non-diagonal entries equal to 0. K₂=Kι^T (i.e., K₂ is a transpose of Ki). The DCT of 'c', C, is obtained by the following multiplication using DCT matrices. It is known that to obtain the DCT of a matrix you must multiply as follows: A = DaD^T B = DbD^T

C = DcD^T (2)

(where D is the DCT matrix shown in Fig. 3 and D^τ is the transpose of D and shown in Fig. 8). The original matrices ('a', 'b', 'c') can be obtained from the DCT matrices using the following equations: a = D^TAD (3) b = D^TBD (4) c = D^TCD substituting c in equation 2 with equation 1 :

C = D(aKι +bK₂)D^T (5) and substituting equations 3 and 4 for 'a' and 'b' respectively in equation 5 yields C = D(D^τADKι + D^TBDK₂)D^T

= (DD^TADK] + DD^TBDK₂)D^T

= DD^TADK,D^T + DD^TBDK₂D^T (6)

Knowing that DD^T=D^TD=I C = ADKιD^τ + BDK₂D^T (7)

This equation can be written as:

C = AMi + BM₂ (8)

As can be seen from equation 7, Mi and M₂ are fixed and do not depend on either matrices 'a', 'b', or 'c'.

Figure 4a and Figure 4b show the matrices Mi and M₂ respectively. Looking at these matrices it is seen that in the odd rows the 1st, 3rd, 5th and 7th elements of M] are the same as that of M₂, and the 2nd, 4th, 6th and 8th elements of Mi are the negative of M₂. In the even rows the converse is true. Therefore Mi and M₂ can be written as follows:

M_! = Com + Dif (9)

M₂ = Com - Dif (10)

where Com contains only the common elements of both Mi and M₂ and all other elements are zero and Dif contains only the elements that are different (all other elements being zero), as shown in figures 5a and 5b respectively. Substituting equations 9 and 10 for Mi and M₂ respectively in equation 8, the DCT matrix C can be found from the DCT matrices A and B as follows (without converting back into the spatial domain): C = A(Com+Dif) + B(Com-Dif) = (A + B)Com + (A - B) Dif (11)

Since matrix Com contains only the common elements it has less than 32 non-zero elements. Similarly the matrix Dif contains only 32 non-zero entries. Computing C using equation 11 instead of equation 8 saves on computation since although equation 8 requires two matrix multiplications and so does equation 11, equation 11 performs a multiplication with matrix elements that are mostly zero.

Equation 11 therefore has provided the DCT coefficients of the DCT transform of block c without having to convert blocks A and B back into the pixel domain.

Once the DCT coefficients of block C have been calculated, this block must be "smoothed" to remove the step edge that exists in the middle of the block in the spatial domain. To smooth block C in the DCT domain, without conversion back into the pixel domain, an analysis was performed to see how the coefficients in block C change when the block 'c' in the spatial domain is changed from having a step edge to one which exhibits a ramp type edge.

Figure 6a shows a pixel matrix 'c' which has a step edge at columns 4 and 5. Figure 6b shows a matrix newc that contains a linear interpolation of the values of pixel matrix c in Figure 6a. This newc is the smoothed block in the pixel domain that exhibits a ramp type edge across the block. In Figure 6b the first column contains the pixel values ay of block 'a' and the last column contains the pixel values by of block 'b'. The pixel values in columns 2-7 vary in accordance with a linear interpolation from left to right. As can be seen from Figure 6b there is no longer a step edge between columns 4 and 5. Obviously a linear interpolation is not the only method of removing this step edge but it is the one chosen here for ease of description.

The DCT of newc is NEWC as shown in Figure 7c. The DCT of ^*c' is C as shown in Fig. 7a. The only difference between NEWC and C is that the first row of coefficients has been modified. Since the DCT of a block can by found by the formula DcD^T, where the matrices D and D are shown in Figures 3 and 8 respectively, and the matrix 'c' is shown in Figure 6a, the first row of C can be written using the following equation. (Note all remaining elements of C are zero where ay = a for all values of i=0» 7 and j=0^« 7 and by= _' for all values of i=0« 7 j=0» 7)

(The above equations for calculating Coo-Co can be used for any two blocks having uniform pixel values for all pixels a,, and all pixel values b_y respectively.) Substituting newc shown in Fig 4b for c in equation 2 DnewcD^T= NEWC the first row of NEWC can be wπtten as (again all remaining elements of NEWC are zero): NEWCoo=4(θ+^), NEWCnι=2^1/2 1.841(9-4), NEWC₀₃=2^1/2 0.1918(9-4),

Formula 12 requires the spatial domain values θ,_j and i_ψ however these are easily obtained from the coefficients of A and B without conversion into the spatial domain. In the present case a,_j = θ = 40 and b_y = I = 70 for all values of ι=0 • 7 and j=0 • 7, but the value 9 and I will depend on the frames of video received in the video stream. Figures 9A and 9B show the blocks A and B (the DCT transforms of blocks 'a' and 'b' shown in Fig lb). DCT blocks A and B have only one non-zero coefficient which when divided by 8 yields the pixel values 9 and - of their respective spatial domain blocks 'a' and 'b', e.g. a_υ = 40 = 9 for ι= 0» 7 and j=0« 7 and -l = 70 = b_y for I = 0» 7 and j=0» 7 respectively in this example. Hence the spatial domain pixel values a,_j and b_υ have been obtained directly from blocks A and B simply by dividing the DC coefficient of A and B respectively by eight. To change C into NEWC the first row of coefficients of C must be changed using equation 12. This smoothes the step edge into a ramp edge by operating on block C in the frequency domain only. As stated above, block C does not exist in the video stream and has been computed for the purpose of removing blocking artifacts. Therefore the smoothing of block C must be translated into changes in the coefficients of blocks A and B which blocks are found in the video stream. From the equation C=AMι + BM₂ it is found that C + δC = (A + δA) Mi + (B + δB) M₂

δC =δAM, + δBM₂

taking δC (where δC is the change which must be made to block C to convert it into NEWC) as NEWC-C then

NEWC-C =δAMι + δBM₂

Therefore any values that solve this equation for δA and δB will smooth the boundary between blocks A and B and remove the blocking artifacts There are many ways to solve this equation, but to achieve some of the best results in image quality, one should try to minimize the amount of change to each block and make the change in one block opposite in direction of the change in the other block so that a smooth ramp edge occurs between blocks. A preferred embodiment of the invention sets δA = -δB. The equations then follow:

(NEWC-C) (M₁-M₂)^"1=δA δA is shown in Fig. 11

The change to the coefficients of blocks A and B (A+ δA, B +δB) are shown in Figs. 12 and 13 respectively. By modifying blocks A and B in this manner the edge between blocks A and B is smoothed without conversion into the spatial domain.

A system that uses the method and/or device of the current invention is shown in Fig. 14. A variable length decoder 10 decodes the incoming video stream and inverse quantizer 20 computes the DCT blocks that have the blocking artifacts. The blocking artifact removal system 30 removes the blocking artifacts from the DCT blocks and provides the DCT blocks for display or storage.

The above description pertains to reduction of blocking artifacts when the blocks in the spatial domain contain uniform pixels. When the blocks in the spatial domain contain many high frequency components a different method is used as explained below.

A first method of reducing the blocking artifacts when the blocks do not contain uniform pixels is to reduce the high frequency components in block C which are the result of the block boundary. When computing block C using the method described herein there will be a large number of high frequency DCT components in block C that are due to the block boundary.

Smoothing can be obtained by setting the large high-frequency coefficient values in block C to zero or reducing their values. Once this modification to C has been performed the change to C is translated into a change in blocks A and B as described above. This method can be used for uniform pixel blocks but a more accurate result for uniform blocks is obtained from the method above which uses linear interpolation.

In another embodiment of the invention block C is smoothed by changing the individual coefficient values in C to zero if their corresponding value in block A and B are also zero. The reason this provides smoothing is that it assumes that blocks A and B belong to the same image region and therefore the frequency characteristics of A and B should be similar to each other. Since block C also belongs to the same image region, the frequency characteristics of block C should be similar to the frequency characteristics of blocks A and B. However, since there is a boundary introduced into block C there will be some high frequency characteristics of block C which represent the boundary and these will not be found in blocks A and B and therefore should be set to zero. Accordingly the high frequency components that are zero in both A and B are set to zero in block C. This change in block C is then translated to a change in blocks A and B as described above. Although the discussions above have been restricted to a vertical boundary between blocks A and B, the extension to a horizontal boundary is straightforward. In addition, the above mentioned DCT coefficient modifications can be performed iteratively on the blocks of the image until a desired smoothness is achieve B.

It will thus be seen that the objects set forth above, among those made apparent from the preceding description, are efficiently attained and, since certain changes may be made in carrying out the above method and in the construction set forth without departing from the spirit and scope of the invention, it is intended that all matter contained in the above description and shown in the accompanying drawings shall be interpreted as illustrative and not in a limiting sense.

Claims

CLAIMS:

1. A method of reducing blocking artifacts, comprising: receiving a first frequency domain block (Fig.9a); receiving a second frequency domain block (Fig.9B); computing (30) a third frequency domain block (Fig. 7a) which overlaps a boundary (e) between the first (Fig. 9a) and second (Fig. 9b) frequency domain blocks; computing (30) a change in coefficients of the third frequency domain block (δC) (Fig. 10) to reduce the blocking artifact at the boundary (e) between the first (Fig. 9a) and second frequency domain blocks' (Fig. 9b) respective spatial domain blocks (Fig. lb); and computing (30) the corresponding changes in the first and second frequency domain blocks (δA, δB) which will result in the change in coefficients of the third frequency domain block (δC).

2. The method in accordance with Claim 1 wherein the step of computing the corresponding changes in the first and second frequency domain blocks satisfies the following condition δC = δAM, + δBM₂ where Mj and M₂ are constants.

3. The method as claimed in Claim 2 wherein δA and δB are chosen to minimize the change in the first and second frequency domain blocks.

4. The method as claimed in Claim 2 wherein δA is substantially opposite to δB.

5. The method as claimed in Claim 1 wherein the step of computing the change in coefficients of the third frequency domain block (δC) is performed by reducing large coefficients of high frequency components of the third frequency domain block and subtracting the third frequency domain block.

6. The method as claimed in Claim 1 wherein the step of computing the change in coefficients of the third block is performed by modifying the third frequency domain blocks' coefficients to reduce high frequency coefficients if their respective coefficients in the first and second frequency domain blocks are substantially zero and then subtracting the third frequency domain block.

7. A device for reducing blocking artifacts, comprising: a receiver (Fig. 14) which receives a first and second frequency domain block; a processor (30) which a.) computes a third frequency domain block (Fig. 7a) which overlaps a boundary (e) between the first (Fig. 9a) and second (Fig. 9b) frequency domain blocks, b.) computes a change in coefficients of the third frequency domain block (δC) to reduce the blocking artifact at the boundary between the first and second frequency domain blocks' respective spatial domain blocks, and c.)computes the corresponding changes in the first and second frequency domain blocks (δA, δB) which will result in the change in coefficients of the third frequency domain block (δC).

8. The device in accordance with Claim 7 wherein the processor computes the corresponding changes in the first and second frequency domain blocks such that it satisfies the following condition where Mi and M₂ are constants.

9. The device as claimed in Claim 8 wherein the processor computes δA and δB to minimize the change in the first and second frequency domain blocks.

10. The method as claimed in Claim 8 wherein the processor computes δA to be substantially opposite to δB.

11. The device as claimed in Claim 7 wherein the processor computes the change in coefficients of the third frequency domain block (δC) by reducing large coefficients of high frequency components of the third frequency domain block and subtracting the third frequency domain block.

12. The method as claimed in Claim 7 wherein the step of computing the change in coefficients of the third block is performed by modifying the third frequency domain blocks' coefficients to reduce high frequency coefficients if their respective coefficients in the first and second frequency domain blocks are substantially zero and then subtracting the third frequency domain block.