JP2010193515A - Image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing method for performing processing such as edge enhancement, which includes smoothing processing and edge enhancement processing and improves image quality. <P>SOLUTION: The method includes: a step of calculating a value which indicates a degree of flatness or a degree of steepness per pixel or per pixels of an input image; a step of obtaining a filter coefficient based on the value indicating a degree of flatness or a degree of steepness; and a step of performing filtering processing for pixels using the obtained filter coefficient. The method also includes a step of obtaining the value indicating the degree of flatness or the degree of steepness as a coefficient for performing smoothing filter processing when the value indicates a flat part, and as a coefficient for performing edge enhancement filter processing when the value indicates a steep part, obtaining an intermediate value between the flat part and the steep part as a coefficient continuously changing from the coefficient of the smoothing filter to that of the edge enhancement filter, extracting a value of a quantizing step for each decoded macro block, and controlling so as to reduce intensity of edge enhancement as the value of the quantizing step increases. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image processing method including a filter process for beautifying an image such as a still image or a moving image.

  For general images including people and landscapes, the characteristics of the edge (boundary) part are maintained as they are, and the flat part is further flattened. Means have been proposed for beautifying the entire image by removing wrinkles, spots and the like while preserving the sharpness of the image.

  In moving picture encoding transmission and the like, application of a filter process for removing block noise and mosquito noise present in a decoded picture has been proposed. A deblocking filter is known for the purpose of eliminating this block noise, but this filter cannot eliminate mosquito noise.

  In addition, if the image is adaptively filtered for each pixel using the surrounding image information to improve the image quality, the output of the low-pass filter (smoothing filter), the original image, and the high-pass filter (edge enhancement filter) It has been proposed to switch. In this case, since threshold setting means for switching cannot be easily realized, there is a disadvantage that unnecessary blurring or image disturbance occurs in an edge region or the like.

Conventionally, an image processing means using an epsilon filter shown in FIG. 8 has been proposed for the purpose of correcting, for example, wrinkles, spots, rough skin, etc. in an image of a person's face (see, for example, Patent Document 1). In the figure, reference numeral 111 denotes a difference detection unit, 112 denotes a threshold value determination unit, 113 denotes a multiplication unit, and 114 denotes an addition unit. The difference detection unit 111 causes the pixel of interest X _{m, n at} the coordinates m and n of the input pixel to be displayed. And the threshold value determination unit 112 compares the absolute value of each difference value with the threshold value TH, and the absolute value of the difference value is smaller than the threshold value TH, for example, Δm _{−1, n−1} , Δm _{−1, n} , Δm _{, n + 1} , Δm _{+ 1, n} , Δm _{+ 1, n + 1} are shown. In the multiplication unit 113, predetermined coefficients A _{i, j} (i = 1, 0, −1, j = 1, 0, −1) are respectively multiplied, and the adder 114 adds the respective multiplied values and the target pixel to form an output pixel. Therefore, by maintaining the edge state of the original image for the edge portion of the image and further smoothing the flat portion, it is possible to obtain an image from which undesirable portions such as wrinkles and spots are removed.

  Further, there has been proposed a means for reducing unnecessary components such as wrinkles in a human face of a television image using a filter called RTF (Ripple Trimmed Filter) shown in FIG. 9 (for example, Patent Document 2). reference). In the figure, reference numerals 121 to 123 denote delay circuits for one horizontal scanning line, 124 denotes a weighted addition circuit, 125 and 127 denote subtraction circuits, and 126 denotes a memory (ROM) storing coefficients.

The input image signal _{Xk, i} is input to the first delay circuit 121 and sequentially delayed by the delay circuits 122 and 123, and the respective delay output signals surrounded by the dotted lines are input to the weighted addition circuit 124 to be used as delay output signals. Multiply by coefficient and add. The output signal of the weighted addition circuit 124 corresponds to a signal obtained by subjecting the input image signal to low-pass filtering. This output signal and the output signal of the delay circuit 122 are input to the subtraction circuit 125 to obtain a difference, and the coefficient is read from the memory 126 using this difference as an address and input to the subtraction circuit 127. By subtracting, a desired image signal can be obtained.

  Also, an image processing apparatus shown in FIG. 10 is known for the purpose of obtaining an easy-to-see image by independently processing the image outline blurring problem and noise problem (see, for example, Patent Document 3). In the figure, 131 is an image input unit, 132 is an image processing unit, 133 is a pixel extraction unit, 134 is an image selection switch (SW1, SW2, SW3), 135 is an image integration unit, and 136 is an image output unit. .

  The image processing unit 132 includes a smoothing unit 221, an edge enhancement unit 222, and delay adjustment units 223, 224, and 225. The pixel extraction unit 133 includes an edge pixel extraction unit 231 and a contour extraction unit 232. A noise pixel extraction unit 233, a non-edge pixel extraction unit 234, a contour area calculation unit 235, and delay adjustment units 236 to 239. The image integration unit 135 includes selectors 251 to 253, And an image composition unit 254.

  The image processing unit 132 inputs the noise-removed image signal from which noise has been removed by the smoothing unit 231, the edge-enhanced image signal in which the edge is enhanced by the edge enhancement unit 232, and the unprocessed image signal to the image integration unit 135. . The pixel extraction unit 133 also adds the contour pixel extracted by the contour pixel extraction unit 232, the noise pixel extracted by the noise pixel extraction unit 233, and the non-edge pixel extracted by the non-edge pixel extraction unit 234 to the image integration unit 135. input.

  To the selectors 251 to 253 of the image integration unit 135, the unprocessed image signal, the noise-removed image signal, and the edge-enhanced image signal from the image processing unit 132 are input, respectively, and the contour pixels from the pixel extraction unit 133 are received. The selector 251, the noise pixel is input to the selector 252, and the non-edge pixel is input to the selector 253. These pixels are “1”, and the unprocessed image and the noise-removed image by the image selection switches SW 1 to SW 3 are displayed. Any one of the input unprocessed image signal, noise-removed image signal, and edge-enhanced image signal is input to the image composition unit 254 by performing selection control of any of the edge-enhanced images and image synthesis is performed. . As a result, the edge enhancement process is not performed on the noise pixel, or the noise removal process is not performed on the edge pixel, and the noise removal process and the edge enhancement process can be performed independently. .

JP 2000-105815 A JP 2000-295497 A JP-A-7-152908

  The conventional image processing means using the epsilon filter shown in FIG. 8, for example, can remove unnecessary components in a fine face such as wrinkles and spots, but the edge portion of the current image There is a problem that only the state can be saved and the edge enhancement processing cannot be realized. In addition, since the product-sum operation includes a determination process, there is a problem that the amount of calculation becomes excessive. Furthermore, since it includes the process of comparing the difference between the pixel of interest and its surrounding pixels with a threshold value, there is a problem that an image area with gradation has less filtering effect and becomes unstable.

  Further, when the RTF shown in FIG. 9 is used, the amount of calculation can be reduced as compared with the case where the above-described epsilon filter is used. In this respect, there is a problem inferior to the case where the above-mentioned epsilon filter is used.

  In the conventional example shown in FIG. 10 described above, smoothing processing is performed on the flat portion of the image, and edge enhancement processing is performed on the edge portion, thereby flattening the fine noise components and generating an easy-to-view image. However, it is necessary to appropriately set a threshold value for selection processing by the selectors 251 to 253, and when this threshold value setting is not appropriate, nonuniform pixels are output in an edge portion or a fine area in the image. It will be. As a result, for example, smoothed pixels are mixed with high-frequency emphasized pixels, and conversely, edge-enhanced pixels are mixed with smoothed pixels, and image quality cannot be improved. . In addition, since the amount of calculation is large, there is a problem that high-speed processing is difficult.

  The image processing method of the present invention calculates a filter coefficient based on a process of calculating a flatness or a steepness value for each pixel or a plurality of pixels of an input image, and the flatness or steepness value. And a process of performing filtering processing on the pixel by the filter coefficient obtained by the process, and in the process of obtaining the filter coefficient, the value indicating the flatness or steepness indicates a flat portion. Sometimes obtained as a coefficient for performing smoothing filter processing, obtained as a coefficient for performing edge enhancement filter processing when showing a steep portion, and when showing an intermediate portion between the flat portion and the steep portion, the coefficient of the smoothing filter The process of obtaining the coefficients of the edge enhancement filter as coefficients that change continuously, and for each decoded macroblock The value of the step extracted, the value of the extracted said amount Coca step has a process of limiting to be smaller the intensity of the edge enhancement in accordance with increase.

  A value indicating flatness or steepness, for example, edge strength, is obtained for each pixel or a plurality of pixels of the input image, and the characteristics of the smoothing filter, all-pass filter, and edge enhancement filter change continuously according to this value. By obtaining the filter coefficient, it is possible to execute the processing of image smoothing and edge enhancement without including discontinuous points, and thus a beautified image can be obtained. In addition, the normal two-dimensional linear filtering process, the edge amount calculation process, and the filter coefficient calculation process are sufficient, and the filter coefficient calculation process can table the filter coefficient, so the calculation is mostly based on the two-dimensional filtering process. The amount of calculation can be reduced as compared with the conventional epsilon filter or image enhancement means by filter switching. Also, by referring to the quantization step for each macroblock and controlling the filtering characteristics, the block emphasis intensity is controlled so that block noise and mosquito noise are reduced as the possibility that block noise is generated increases. The image quality can be improved by performing a smoothing process.

It is principal part explanatory drawing of the image processing apparatus of the Example of this invention. It is principal part explanatory drawing of the filter structure of the Example of this invention. It is arrangement | positioning explanatory drawing of a filter coefficient. It is explanatory drawing of the change of a filter coefficient. It is a relation explanatory view of a filter coefficient and a filter characteristic. It is explanatory drawing of an image transmission system. It is explanatory drawing of an image transmission system. It is explanatory drawing of a prior art example. It is explanatory drawing of a prior art example. It is explanatory drawing of a prior art example.

  According to the image processing method of the present invention, a filter coefficient is obtained based on a process of calculating a value indicating a flatness or steepness for each pixel or a plurality of pixels of an input image and a value indicating the flatness or steepness. And a process of filtering the pixel by the filter coefficient obtained in this process. In the process of obtaining the filter coefficient, smoothing is performed when the value indicating the flatness or steepness indicates a flat portion. Obtained as a coefficient for filter processing, obtained as a coefficient for edge enhancement filter processing when showing a steep portion, and changed from a smoothing filter coefficient to a coefficient for an edge enhancement filter when showing an intermediate portion between a flat portion and a steep portion A process for obtaining continuously changing coefficients, and a quantization step value is extracted for each decoded macroblock. The value of the quantization step is intended to include process of limiting to be smaller the intensity of the edge enhancement in accordance with increases.

FIG. 1 is an explanatory diagram of a main part of an embodiment of the present invention, where 1 is a mask scanning unit, 2 is an edge amount calculation unit, 3 is a filter coefficient calculation unit, and 4 is a two-dimensional filter calculation unit. The input image X _{i, j} to the mask scanning unit 1 is filtered to obtain an output image Y _{i, j} , and the mask scanning unit 1 has, for example, a 3 × 3 mask scanning configuration corresponding to the filter configuration. it can. The edge amount calculation unit 2 calculates a flatness or a steepness for each pixel or a plurality of pixels. Hereinafter, a case where an edge amount (edge strength) as a steepness is calculated will be described. That is, the edge amount of the input image X _{i, j} is calculated for each pixel or a plurality of pixels from the mask scanning unit 1, input to the filter coefficient calculation unit 3, and the filter coefficient is obtained to obtain the two-dimensional filter calculation unit 4. And the image signal from the mask scanning unit 1 is filtered to obtain an output image Y _{i, j} .

If the input image is X _{m, n} , the output image is Y _{m, n} , and the filter coefficient is _{Wi, j} , a two-dimensional filter that can obtain a general normalized output is
Y _{m, n} = (ΣW _{i, j} * X _{m + i, n + j} ) / ΣW _{i, j} (1)
It can be expressed as.

As a condition for obtaining a normalized output, by setting the filter bank in advance so that the value of ΣW _{i, j} is constant, division is not necessary, so that the amount of calculation can be reduced. Under these conditions, a low-pass filter, an all-pass filter, and a high-frequency emphasis filter can be formed as the absolute value of the weighting coefficient.

As means for changing a coefficient for each pixel from among a plurality of filter coefficients, it is general to apply means for switching according to the peripheral edge amount for each pixel. For example, as a calculation for obtaining the edge strength,
Laplacian EP _{i, j} = | X _{i, j-} (X _{i-1, j-1} + X _{i-1, j} + X _{i-1, j + 1} + X _{i, j-1} + X _{i, j + 1} + X _{i + 1, j-1} + X _{i + 1, j} + X _{i, j + 1} ) / 8 | (2)
Roberts operator EP _{i, j} = | X _{i, j} −X _{i + 1, j + 1} | + | X _{i, j + 1} −X _{i + 1, j} | (3)
Edge strength EP _{i, j} = | X _{i−1, j} −X _{i + 1, j} | + | X _{i, j−1} −X _{i, j + 1} | (4)
Can be used.

Examples of the smoothing filter, the all-pass filter, and the edge emphasis filter in the case where ΣW _{m, n} = 4 in the 3 × 3 mask two-dimensional filter as the constraint condition of the coefficient are shown below.

All-pass filter W _{i, j} = {0,0,0,0,4,0,0,0,0}
Smoothing filter W _{i, j} = {0, 1, 0, 1, 0, 1, 0, 1, 0}
W _{i, j} = {1,0,1,0,1,0,1,0,1}
High-frequency emphasis filter Wi, _j = {0, -1, 0, -1, 8, -1, 0, -1, 0}
W _{i, j} = {− 1, −1, −1, −1,12, −1, −1, −1, −1}

Here, paying attention to the coefficients of the all-pass filter, the smoothing filter, and the high-frequency emphasis filter,
W _{i, j} = {0,0,0,0,4,0,0,0,0}
W _{i, j} = {0,1,0,1,0,1,0,1,0}
W _{i, j} = {0, −1, 0, −1, 8, −1, 0, −1, 0}
In the case of the combination of (i, j) = (1, 0), (0, 1), (2, 1), (1, 2) are the same coefficient, and the sum of all coefficients C = ΣW _The condition is that _{i and j} .

If the non-zero coefficient other than the center is B and the center coefficient is A, the constraint condition for the coefficient is
C = ΣW _{i, j} = A + 4B
(Where C is a coefficient for normalizing the output at a frequency of 0 Hz).

In order to perform smoothing on the flat part and edge enhancement on the boundary part, when the value of EP _{m, n} is large, the value of B is negative and the value of EP _{m, n} is close to 0 It is desirable that the value of B is a positive value, and B = 0 in the middle range. To calculate B as a value with these conditions, for example:
B = max (min ([alpha] (L-EP (m, n)), E), F) (5)
Can be used. E is the upper limit value of the coefficient B, F is the lower limit value of the coefficient B, and α is the degree of switching between the smoothing filter and the edge enhancement filter. L indicates a region where the entire region is desired to pass.

  FIG. 2 shows a main part of the filter configuration of the image processing apparatus when the mask scanning unit 1 in FIG. 1 has a 3 × 3 mask configuration, and the same reference numerals as those in FIG. Further, L-D of the mask scanning unit 1 is a line buffer, D is a pixel storage unit, and 3 × 3 = 9 output signals are input to the two-dimensional filter calculation unit 4. In the edge amount calculation unit 2, SUBABS indicates a difference absolute value calculation unit, and + indicates an addition unit. Further, the filter coefficient calculation unit 3 shows a case where it is configured by a filter coefficient ROM in which filter coefficients are tabulated. In the two-dimensional filter calculation unit 4, M indicates a multiplication unit and + indicates an addition unit.

  The edge amount calculation unit 2 obtains an absolute value of the difference between the output pixels from the mask scanning unit 1, obtains an edge amount indicating a flatness or a steepness, and inputs it to the filter coefficient computation unit 3. The filter coefficient calculation unit 3 shows a case where a filter coefficient ROM is used to read out the filter coefficient using the edge amount as an address. The read filter coefficient is composed of the above-described coefficients A and B, and is sent to the two-dimensional filter calculation unit 4. input. The two-dimensional filter operation unit 4 performs a product-sum operation on the output signal from the mask scanning unit 1 using a filter coefficient, and a smoothing filter, an all-pass filter, and a high-pass filter (edge enhancement filter) according to the filter coefficient. The characteristic is continuously changed, the flat part is smoothed to make a smooth image, the steep part is made clear by edge enhancement, the other is made as it is, and the state transitions continuously between them. This is an image.

  FIG. 3 is an explanatory diagram of the arrangement of filter coefficients. (A) and (B) of FIG. 3 show examples of coefficients of a 3 × 3 edge detection filter, and (A) is a Roberts edge detection filter. The coefficient arrangement of is shown. (B) shows the coefficient arrangement of the edge detection filters in the horizontal and vertical directions. Also, (C) and (D) in the figure are explanatory diagrams of the 3 × 3 adaptive filter coefficient arrangement, (C) shows the coefficient arrangement in the case of filtering from four adjacent points, and (D) The coefficient arrangement | positioning in the case of performing filtering from adjacent 8 points is shown.

FIG. 4 is an explanatory diagram of the change of the filter coefficient, showing the change of the coefficient B with respect to the absolute value | EP | of the edge strength, where E is the upper limit value of the coefficient B, F is the lower limit value of the coefficient B, and L is all-pass Indicates the area. (A) in the figure shows that when | EP | is small, the coefficient B is an upper limit value E, and when | EP | is large, the coefficient B is a lower limit value F, and | EP | In some cases, the coefficient B is controlled so as to continuously change through 0 corresponding to the value of | EP |. At this time, the coefficient A depends on the condition of C = ΣW _{i, j} .
A = C-4B
Can be obtained as

A filter having such coefficients A and B is
W _{i, j} = {0, B, 0, B, A, B, 0, B, 0}
Thus, it is possible to perform image processing by operating as a smoothing filter in the range of EP <L, an all-pass filter in the vicinity of EP = L, and a high-frequency emphasis (edge emphasis) filter in the range of EP> L. The coefficient B continuously changes from the upper limit value E to the lower limit value F as shown in the figure.

  4B, when operating as a high frequency emphasis filter, the coefficient B is gradually changed from the lower limit value E to 0 as the absolute value of the edge strength increases, and the edge enhancement is changed from large edge enhancement to small edge enhancement. The case where it is set as the filter characteristic to perform is shown. In other words, it is possible to improve the image quality by reducing the emphasis degree of the steep contour region where the absolute value of the edge intensity is high and processing the steep region to be a steep region.

  FIG. 4C shows a case where the coefficient B is limited to values (a), (b), (c), and (d) that are larger than the lower limit F when the absolute value of the edge strength is large. The smoothing process is the same as that shown in (A) and (B), but the degree of the edge enhancement process can be limited. This limitation can be selected stepwise or continuously according to the nature of the input image, and can be applied as a post filter in an image transmission system.

For the one-dimensional filter W _{i, j} = {B, A, B}, A = C−2B and C = 1 are set as constraints on the coefficients A and B, and the coefficient B is set to 1, 0.5, If it is changed to four types of 0 and -1, they become a smoothing filter, an all-pass filter, an edge enhancement (weak) filter, and an edge enhancement (strong) filter, respectively. Regarding the change in the characteristics, when _{Wi, j} = {B, A, B} is Fourier-transformed and expressed by the frequency characteristics, it is as shown in FIG. That is, when the coefficients A = 3 and B = −1, an edge enhancement filter whose output increases as the frequency increases, and when the coefficients A = 2 and B = −0.5, an edge enhancement (weak) filter, When the coefficients A = 1 and B = 0, the output is constant all-pass filter even if the frequency changes. When the coefficients A = −1 and B = 1, the higher the frequency, the smoother the output becomes smaller. It becomes the characteristic of the filter. Note that the coefficients are normalized, so the output at a frequency of 0 Hz is 1.0. As described above, the characteristics of the smoothing filter (low-pass filter) and the high-frequency emphasis filter (high-pass filter) can be smoothly switched by controlling the coefficients.

  As described above, the edge strength around the target pixel is obtained, the filter coefficient is obtained from the edge strength, and the two-dimensional filter processing is performed using the filter coefficient, thereby smoothing a flat portion in the image. The edge emphasis process is performed, and the edge emphasis process is performed on the boundary portion having a large edge strength. That is, it is possible to continuously change the characteristics of the smoothing filter, the all-pass filter, and the edge enhancement filter according to the edge strength around the target pixel, and the software processing is relatively simple. In this case, high-speed computation is possible, and the hardware scale can be reduced as compared with the conventional example.

  The filtering process in the image processing of the present invention includes a normal two-dimensional linear filtering process, an edge amount calculation process, and a filter coefficient calculation process from the edge amount. As shown in FIG. 2, since it can be tabulated by a filter coefficient ROM or the like, it is possible to perform high-speed computation by reducing the amount of computation when using a conventional epsilon filter or by means of filter switching. Become.

  FIG. 6 is an explanatory diagram of an image transmission system, in which 21 is a television camera, 22 is an encoder (ENCODER), 23 is a decoder (DECODER), 24 is a post filter (P to ST-FILTER), and 25 is a display device (DISP). ). The encoded image signal encoded by the encoding method according to the image transmission method is transmitted from the transmission side by the television camera 21 and the encoder 22. On the reception side, the encoded image signal is decoded by the decoder 23, the post-filter 24 performs the filtering process according to the above-described embodiment of the present invention, and the display device 25 displays the image. Therefore, the flat part of the image is smoothed and the boundary part is edge-enhanced and displayed on the display device 25.

  FIG. 7 is an explanatory diagram of an image transmission system, in which 31 is a television camera, 32 is a pre-filter, 33 is a decoder, 34 is an encoder, 35 is a post filter, and a display device on the receiving side is not shown. . The image signal from the television camera 31 is subjected to edge enhancement processing or the like according to the above-described embodiment by the prefilter 32 on the transmission side, and is encoded by the decoder 33 and transmitted. On the reception side, decoding is performed by the encoder 34, and block noise and mosquito noise are removed by the post filter 35. That is, the post filter 35 has a characteristic of performing image processing using a coefficient B that varies depending on edge strength as shown in FIG. 4C, for example, block noise or emphasis that is likely to occur in low-rate transmission. It is possible to remove mosquito noise that tends to occur around the edge and display an easy-to-view image.

  Further, the value of the quantization step determined for each macroblock of the image coding unit is large, and when transmitting at a low bit rate, there is a high possibility that block noise occurs between adjacent macroblocks. When the filtering process including the edge enhancement process described above is performed on the area where the block noise is generated, the edge component existing in the block noise part is enhanced, and the image quality deteriorates. Therefore, the edge enhancement characteristic is changed in real time based on the value of the quantization step for each decoded macroblock. That is, as shown in FIG. 4C, the negative value (lower limit F side) of the coefficient B is set to (a), (b), (c) as the quantization step value increases. Finally, as shown in (d), the edge enhancement is not performed, and even when there is a high possibility of occurrence of block noise, decoding processing is performed as a block noise removal filter characteristic. Can do.

DESCRIPTION OF SYMBOLS 1 Mask scanning part 2 Edge amount calculating part 3 Filter coefficient calculating part 4 Two-dimensional filter calculating part

Claims (1)

A process of calculating a flatness or steepness value for each pixel or pixels of the input image;
Obtaining a filter coefficient based on the value indicating the flatness or steepness;
Filtering the pixel with the filter coefficient obtained by the process,
In the process of obtaining the filter coefficient, the value indicating the flatness or the steepness is obtained as a coefficient for performing the smoothing filter process when indicating a flat part, and the coefficient for performing the edge enhancement filter when indicating the steep part A decoded macro having a process of obtaining a coefficient that continuously changes from a coefficient of the smoothing filter to a coefficient of the edge enhancement filter when an intermediate portion between the flat portion and the steep portion is indicated. An image processing method comprising: extracting a quantization step value for each block, and limiting the intensity of edge enhancement so as to decrease as the extracted quantization step value increases.

Images