JP2009077240A - Image processing apparatus and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a post-filter which performs smoothing processing adaptively in both a temporal direction and a spatial direction. <P>SOLUTION: A motion estimating section 103 estimates a corresponding pixel between frames proximate to an image inputted. An non-isotropic diffusion filter 104 performs weak smoothing in the case where a difference of signal strength is great between pixels corresponding to a temporal direction estimated by the motion estimating section 103, and performs strong smoothing in the case where the difference of signal strength is small therebetween. The non-isotropic diffusion filter 104 then performs strong smoothing in a direction parallel with an edge, and performs weak smoothing in a direction vertical to an edge in intra-frame proximate pixels. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image processing apparatus and an image processing method for realizing high image quality by removing noise using a filter or the like.

  In general, when video / audio information is digitized for recording / transmission, digital data is compression-encoded to reduce the amount of data, and when this is received / reproduced, compression encoding technology that decodes the compressed data is used. It is done. For example, Non-Patent Document 1 discloses H.264. A high compression technique for high-definition images such as H.264 / AVC format is disclosed. Noise generated by compression called block noise and mosquito noise is added to an image compressed by such a compression coding technique. In addition, various noises are present in the image captured by the camera, and it is necessary to effectively remove them.

  In order to remove these noises, the following post filter technique has been proposed.

  For example, Patent Document 1 discloses an image compression apparatus that includes a Laplacian filter that extracts a high-frequency component of image data and a median filter that stores only a high-frequency component that represents a contour portion of the extracted high-frequency component.

  Further, in Patent Document 2, in order to reduce noise of an image with a large motion, an image component of a motion estimation unit that identifies a target block and a target block weighted by a current block and a weighting filtering coefficient is averaged. A system comprising motion compensated noise reduction means is disclosed.

JP 05-344346 A Japanese Patent Application Laid-Open No. 07-274044 Joint Video Team (JVT) of ISO / IEC MPEG & ITU-T VCEG: “Text of International Standard of Joint Video Specification”, ITU-T Rec. H. H.264 | ISO / IEC 14496-10 Advanced Video Coding, (December, 2003).

  According to the technique of the above-mentioned patent document 1, there is a case where the outline (edge) originally included in the image is destroyed by the amount of noise suppression, and high image quality cannot always be realized. In Patent Document 2, the magnitude of motion must be detected over a plurality of frames, and parameters must be determined according to threshold values. It is difficult to easily realize a high-quality filter. In order to realize a high-quality post filter, it is desirable to reflect the continuity of pixels in both the time direction and the spatial direction, but this has not been taken into consideration.

  A conventional filter that performs processing in the spatial direction (two-dimensional plane) has a problem that the edge of the subject is broken by the smoothing and the sharpness of the image is lowered. At that time, although it is possible to perform relatively high-definition smoothing by using an anisotropic diffusion filter, sufficient accuracy cannot be realized only by processing in the spatial direction. On the other hand, a filter that performs processing in the time direction can maintain relatively sharpness, but has a problem that the smoothing ability is not sufficient and noise cannot be sufficiently removed.

  The present invention has been made in view of the above problems, and an object thereof is to provide a post filter that performs adaptive smoothing processing in both the time direction and the spatial direction.

  In order to achieve the above object, an image processing apparatus of the present invention includes a motion estimation unit that estimates corresponding pixels between adjacent frames to an input image, and a pixel that corresponds to a time direction estimated by the motion estimation unit. A filter that performs different smoothing processing depending on the signal strength between them, and the filter performs weak smoothing when the signal strength difference between the pixels is large, and strong smoothing when the signal strength difference is small. To do.

  Further, the filter performs different smoothing processing depending on the direction of the edge between adjacent pixels in the frame, and performs strong smoothing in the direction parallel to the edge, and the direction perpendicular to the edge. Is weakly smoothed.

  The image processing method of the present invention includes a step of estimating corresponding pixels between adjacent frames with respect to an input image, and a smoothing process that differs according to the signal strength between pixels corresponding to the estimated time direction. In the smoothing processing step, weak smoothing is performed when the signal intensity difference is large between pixels, and strong smoothing is performed when the signal intensity difference is small.

  According to the present invention, it is possible to suitably remove noise included in an image and obtain a high-quality image.

Embodiments of the present invention will be described below with reference to the drawings.
FIG. 1 is a block diagram showing an embodiment of an image processing apparatus according to the present invention. The image processing apparatus 100 includes an image input unit 101 that inputs image data, a frame buffer 102 that holds the input image data for each frame, a motion estimation unit 103 that performs motion estimation of corresponding pixels from a plurality of frames, A spatio-temporal anisotropic diffusion filter 104 that performs smoothing in the temporal direction and the spatial direction, and an image output unit 105 that outputs processed image data are provided.

  The image input unit 101 passes the input image data to the frame buffer 102. The frame buffer 102 holds the image data for each frame and passes them to the motion estimation unit 103. The motion estimation unit 103 estimates a corresponding point of motion for an adjacent frame for each pixel of the processing target frame by performing block-unit matching or pixel-unit matching from a plurality of frames. For this, a commonly used method of motion estimation may be used. Although error and estimation failure occur in motion estimation, there is no problem with the method in this embodiment because the spatio-temporal anisotropic diffusion filter 104 acts to avoid the influence of error and estimation failure.

  Next, the spatio-temporal anisotropic diffusion filter 104 performs an adaptive smoothing process in the time direction and the spatial direction. The processing target pixel is subjected to image signal smoothing processing by reflecting an edge or a motion gap between a pixel corresponding to the time direction estimated by the motion estimation unit 103 and a pixel adjacent in the spatial direction. That is, weak smoothing is performed when the signal intensity difference between the pixels is large, and strong smoothing is performed when the signal intensity difference is small. This smoothing process is preferably repeated a predetermined number of times by connecting the switches 106 and 107 to the terminal b side. Repeating increases the accuracy of smoothing and improves the image quality. The number of repetitions is appropriately set according to the image quality of the input image. The smoothing process can improve the image quality most by simultaneously performing the process in both the time direction and the spatial direction. However, depending on the input image, it is effective to perform processing only in the time direction or only in the spatial direction, and the processing pattern can be switched in accordance with the feature of the frame or the feature of the pixel. The image data that has undergone the smoothing process of the image signal is output from the image output unit 105.

  As described above, the image processing apparatus 100 according to the present embodiment includes the spatio-temporal anisotropic diffusion filter 104 that performs smoothing processing in the temporal direction and the spatial direction using the motion information from the motion estimation unit 103. There is.

  Note that if the image data input to the image input unit 101 includes motion information at the time of compression encoding, the motion estimation unit 103 can also perform motion estimation using the motion information. Thereby, the circuit configuration can be simplified.

FIG. 2 is a flowchart showing an example of an image processing method according to the present invention.
In step 201, image data is input to the frame buffer 102 by the image input unit 101. In step 202, the motion estimation unit 103 performs motion estimation processing. The motion estimation process is a process for estimating a corresponding point of motion for an adjacent frame for each pixel of a processing target frame by performing block-unit matching or pixel-unit matching from a plurality of frames.

  In step 203, the spatio-temporal anisotropic diffusion filter 104 is used to perform a smoothing process in the spatio-temporal region including the estimated motion corresponding points. In step 204, the number of executions of the smoothing process is determined. If the predetermined number has not been reached, the smoothing process in step 203 is repeated. If the predetermined number of times has been reached, the image output unit 105 outputs the smoothed image in step 205.

  In the image processing apparatus and the image processing method of the present embodiment, motion estimation is performed between a plurality of frames, and smoothing by a spatio-temporal anisotropic diffusion filter is performed by using corresponding point information in the time direction obtained thereby. I do. As a result, block noise and mosquito noise caused by compression can be reduced while preserving edges such as the shape of the object originally held in the image and the boundary between the object and the background. An output image can be obtained.

  Here, the space-time anisotropic diffusion filter 104 will be described. The space-time anisotropic diffusion filter 104 is a filter that performs anisotropic diffusion filter processing in the time direction and the space direction. An anisotropic diffusion filter is a filter that applies processing represented by a partial differential equation based on an anisotropic thermal diffusion equation to a target pixel, and its definition is described in Reference 1. An anisotropic diffusion filter for a two-dimensional plane performs smoothing with different weights in the tangential direction of an edge and the vertical direction of the edge in an image including a linear edge. That is, strong smoothing is performed in the direction parallel to the edge, and weak smoothing is performed in the direction perpendicular to the edge.

  As an example of the anisotropic diffusion filter, for example, there is a total variation filter (Total Variation Filter, hereinafter abbreviated as TVF) shown in Reference Document 2. The TVF is a filter that performs smoothing adapted to an edge with reference to an absolute value difference (L1 norm) between a pixel to be processed and an adjacent pixel. As another example of the anisotropic diffusion filter, there is an anisotropic diffusion method (Perona-Malik Diffusion (PMD)) proposed by Perona-Malik.

  These anisotropic diffusion filters were originally used for image restoration or the like, and were mainly applied to smoothing pixels in a two-dimensional plane. In the present embodiment, the anisotropic diffusion filter is extended not only to neighboring pixels in the two-dimensional plane but also to neighboring pixels in the time direction in the same manner, thereby extending to a spatiotemporal anisotropic diffusion filter applicable to spatiotemporal. Is.

[Reference 1] Tsuji, Tokumasa, Takahashi, Nakajima: “Basic study on construction of spatial smoothing filter based on anisotropic diffusion method”, IEICE Transactions D-II, Vol. J88-D-II, no. 6, pp. 1024-1034 (2005).
[Reference Document 2] Vese, S.M. Osher: “Modeling Textures with Total Variation Minimization and Oscillating Patterns in Image Processing”, Journal of Scientific Computing, 19, pp. 197 553-572 (2003).

  Hereinafter, the smoothing operation by the space-time anisotropic diffusion filter will be described. Here, the case of TVF will be described, and the filter operation is expressed by Equation 1.

  Here, α indicates a target pixel to be processed, and β and γ indicate pixels in the vicinity of α. N (α) is a set of pixels in the vicinity of α. u represents the signal intensity (for example, luminance value) of the pixel, and w (u) is a function inversely proportional to the difference in signal intensity (luminance) between the pixels. Therefore, w (u) is small when the difference in signal intensity u is large between pixels, and w (u) is large when the difference in signal intensity is small. λ is a fitting parameter for adjusting the strength of filtering. In TVF, filtering is performed by repeatedly applying the filter of Formula 1 to all the pixels of the processing target frame. The luminance u of the pixel is that of the image that has been repeatedly processed, but the pixel of the input image is used for the second term of Equation 1.

FIG. 3 is a diagram illustrating an example of a space-time smoothing process using a TVF.
In FIG. 3, reference numeral 301 denotes a processing target pixel α in the current frame, reference numeral 302 denotes neighboring pixels β and γ in the spatial direction included in the current frame, and reference numeral 303 denotes a temporal direction included in the previous and subsequent frames based on motion estimation. Neighboring pixels β ′ and γ ′. Conventionally, the processing is performed on the neighboring pixels β and γ in the spatial direction indicated by reference numeral 302. However, in this embodiment, processing on neighboring pixels β ′ and γ ′ in the temporal direction indicated by reference numeral 303 is added. Here, with the processing target pixel α as the center, four neighboring pixels in the spatial direction and two neighboring pixels in the preceding and following frames in the time direction are included in the processing range.

  In Equation 1, the weighting factor h for smoothing is determined by a function w (u) that is inversely proportional to the luminance difference between pixels. Therefore, the target pixel α is compared with six neighboring pixels (β, γ, β ′, γ ′), and in the direction where there is a large difference in luminance between them, that is, in the direction perpendicular to the edge in the plane, in the time direction For a direction with a motion gap, the smoothing weight is decreased (weak smoothing process), and for a direction with a small luminance difference, the smoothing weight is increased (strong smoothing process). By repeating this process, filtering reflecting an edge and a gap of motion is realized over a wide range of the screen.

  In this way, by expanding the neighboring pixels for smoothing not only in the spatial direction but also in the temporal direction, not only spatial noise such as block noise and mosquito noise, but also temporal noise such as screen flicker and drift noise. Can be removed.

  According to the smoothing processing of the present embodiment, block noise and mosquito noise generated by compression while preserving the correspondence such as the shape of the object originally possessed by the image, the boundary between the object and the background, and the movement of the object. Flicker and drift noise can be reduced, and a high-quality output image can be obtained.

  FIG. 4 illustrates examples of several processing patterns using a spatio-temporal anisotropic diffusion filter. Again, TVF is used as the anisotropic diffusion filter.

  (A) is the case of the in-screen TVF 401 using spatial neighboring pixels (spatial processing pattern), and (b) is the case of the motion compensation time TVF 402 using the temporal motion compensation position (motion estimation position) as a neighboring pixel ( (Time processing pattern), (c) is a case of a motion compensated spatiotemporal TVF 403 (spatiotemporal processing pattern) using both a spatial neighborhood and a motion compensation position.

  In the above embodiment, it has been described that high-precision smoothing is performed in space-time by the motion-compensated space-time TVF 403 in (c). It goes without saying that image quality can be improved most by performing processing in both the time direction and the spatial direction simultaneously. On the other hand, even in the case of the motion compensation time TVF 402 in (b), a new processing function becomes possible by applying the anisotropic diffusion filter in the time direction. In that case, for example, as shown in (b), it is possible to perform processing in a total of 5 frames including 2 frames before and after, and smoothing in the time direction is performed by repeating the processing in the vicinity of 2 in the time direction.

  Each of the above processing patterns, (a) only in the spatial direction, (b) only in the temporal direction, and (c) both in the temporal direction and the spatial direction may be switched according to the characteristics of the frame and the characteristics of the pixels. Thereby, the image quality can be improved more efficiently by selecting a processing pattern suitable for each frame or pixel.

  Each of the embodiments described above can be effectively implemented either alone or in combination. In addition, the specific configuration of the spatio-temporal anisotropic diffusion filter used for the smoothing process in the spatio-temporal region is not limited to that described in the embodiment, and it goes without saying that any filter having a similar function can be used in the same manner. Yes.

  According to the image processing apparatus and the image processing method of the present embodiment, for example, a decoded image with low image quality encoded at a high compression rate can be output with high image quality on the decoding side.

  Furthermore, the image processing apparatus and the image processing method of the present embodiment are applied to various video processing apparatuses such as an image recording apparatus, a player, a mobile phone, a portable terminal, a digital camera, a TV, a projector, various displays, and a game machine. Therefore, it is possible to provide a video processing apparatus with high accuracy.

The block diagram which shows one Example of the image processing apparatus by this invention. 5 is a flowchart illustrating an example of an image processing method according to the present invention. The figure which shows an example of the smoothing process of the time and space using TVF. An example of a processing pattern using a space-time anisotropic diffusion filter.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 100 ... Image processing apparatus 101 ... Image input part 102 ... Frame buffer 103 ... Motion estimation part 104 ... Spatio-temporal anisotropic diffusion filter 105 ... Image output part

Claims (10)

A motion estimation unit that estimates corresponding pixels between adjacent frames for an input image;
A filter that performs different smoothing processing according to the signal strength between pixels corresponding to the time direction estimated by the motion estimation unit,
An image processing apparatus characterized in that the filter performs weak smoothing when the signal intensity difference between pixels is large and performs strong smoothing when the signal intensity difference is small. The image processing apparatus according to claim 1,
Further, the filter performs different smoothing processing depending on the direction of the edge between adjacent pixels in the frame, and performs strong smoothing in the direction parallel to the edge, and the direction perpendicular to the edge. An image processing apparatus that performs weak smoothing on the image. The image processing apparatus according to claim 1, wherein:
The image processing apparatus, wherein the filter repeatedly performs a smoothing process on the same image a predetermined number of times. The image processing apparatus according to any one of claims 1 to 3,
The image processing apparatus, wherein the filter is an anisotropic diffusion filter. The image processing apparatus according to claim 4,
The image processing apparatus, wherein the anisotropic diffusion filter is a total variation filter. Estimating a corresponding pixel between adjacent frames for an input image;
Performing different smoothing processes depending on their signal strength between pixels corresponding to the estimated time direction,
An image processing method characterized in that, in the smoothing processing step, weak smoothing is performed when the signal intensity difference between pixels is large, and strong smoothing is performed when the signal intensity difference is small. The image processing method according to claim 6,
Further, the method includes a step of performing different smoothing processing depending on the direction of the edge between adjacent pixels in the frame,
In the smoothing processing step, strong smoothing is performed in a direction parallel to the edge, and weak smoothing is performed in a direction perpendicular to the edge. The image processing method according to claim 6 or 7,
In the smoothing process step, the smoothing process for the same image is repeatedly performed a predetermined number of times. The image processing method according to any one of claims 6 to 8, comprising:
An image processing method using an anisotropic diffusion filter in the smoothing step. The image processing method according to claim 9, comprising:
An image processing method using a total variation filter as the anisotropic diffusion filter.

Images