WO2010101292A1

WO2010101292A1 - Method of and apparatus for processing a video image

Info

Publication number: WO2010101292A1
Application number: PCT/JP2010/053925
Authority: WO
Inventors: Andrew Kay; Allan Evans; Graham Jones; Marc Paul Servais; Matti Pentti Taavetti Juvonen; Kenji Maeda
Original assignee: Sharp Kabushiki Kaisha
Priority date: 2009-03-03
Filing date: 2010-03-03
Publication date: 2010-09-10
Also published as: GB0903594D0; GB2468304A

Abstract

A video image is processed (501, 502, 509. 51) to determine an edge strength in a region associated with each image pixel to be processed. For large edge strengths, the pixel is filtered by an edge-preserving filter, such as an epsilon filter (504). For very large edge strengths, no edge-preserving filtering is applied. For small edge strengths, the edge-preserving filtering is replaced by a smoothing filter (505, 506).

Description

DESCRIPTION

TITLE OF INVENTION: METHOD OF AND APPARATUS FOR PROCESSING A VIDEO IMAGE

TECHNICAL FIELD The present invention relates to a method of an apparatus for processing a video image, for example for use with or in a display system, or a television, mobile phone, advertising hoarding, digital photograph display device, computer display, projector or other public or personal devices including such a display system. The method may be used in image processing for improving the perceived quality of digital TV or video without changing what is broadcast or otherwise distributed.

BACKGROUND ART

In this disclosure we consider that a display is a device capable of processing and ' showing static or moving images. Moving images are generally represented as video sequences of instantaneous frames or fields. The difference is that a frame represents an entire image at an instant, whereas a field represents only a portion of an image at an instant, such as every other line of an image .

The advantage of digital video transmission or storage is that techniques of lossy digital compression can be used to reduce the number of bits required to describe the images. However, as the number of bits is decreased the perceived quality of the images can be seriously degraded by the introduction of compression artefacts, which are perceived as noise. In addition, when scaling video content for a larger display size these artefacts tend to become more visible and annoying to the viewer. Current compression techniques such as MPEG2 and H .264 introduce in particular block noise (caused by the division of the scene into blocks, which may not have smooth j oins when decompressed) and mosquito noise caused by not using enough bits to record all the frequencies present in an image, resulting in spatial ringing near strong image edges. Examples can be seen in figure .2 of the accompanying drawings, which shows a detail of a compressed image . Mosquito noise 21 and block noise 22 can be clearly seen.

In "An Efficient Approach to the Reduction of Mosquito Noise for the JPEG/ JPEG2000 Decoded Image by Using Epsilon Filter" (2002 International Conference on Digital

Printing Technologies) Shohdoji et al present a method for reducing mosquito noise using an ε (epsilon) filter. Patent application US20051 17807 (Shohdoji) by the same author proposes an optimisation to the method, reducing the need to calculate ε at every pixel position. WO2007072301 (Philips, 28 June 2007) and US7203234 (Sharp, SLA, 10 April 2007) try to reduce ringing using directional low pass filtering, using information from the decompression (decoder) unit. Often, due to the modular design of display systems, such decompression information is not available to other parts of the system, so this method would not work in such a case .

US6996184 (Sony, 7 February 2006) uses more than one frame of data to reduce noise. This requires extra memory to record the history of the processing.

JP2007174403 (Toshiba, 5 July 2007) improves an image by emphasising edges, but being careful not to emphasize areas potentially containing compression block edges . WO2004084123 (Qualcomm, 30 September 2004) reduces block noise by filtering specifically at positions on block edges.

The ε-filter is described in "An Efficient Approach to the

Reduction of Mosquito Noise for the JPEG/ JPEG2000 Decoded

Image by Using Epsilon Filter" (2002 International Conference on Digital Printing Technologies) by Shohdoji et al and also in patent application US20051 17807 (2 June 2005) . Here we describe it with slightly different notation. There are three main parameters M, N and escale. The luminance value Y(x,y) is either known or calculated for each pixel P(x, y) in an input image . Next the algorithm calculates a measure of variation V(x,y) of Y over rectangular NxN blocks of the image centred at each pixel location (x,y). Let |x| denote, as usual, the absolute value of a number x. Let Λ(x,y,N) denote the set of pixel positions (x',y') such that | jc-jc' | < N/ 2 and \y-y'\ ≤ N/2, that is roughly the rectangle of size JV^" centred on (x,y). One measure of variation to be used for V suggested in US2005117807 is standard deviation:

₍₁) V(x,y) = {∑_(x,_y,_)ffΛ(xyN) [Y(X¹, y')

2 -.1/2 -dw)_£Λ(x,y,N) Y(x",y")/ 1 Λ(x, y,N) I)]²/ 1 Λ(x,y,N) | ,

Next E(x,y) is calculated for each pixel (x,y), where B(x,y) is the maximum of V(x',y') taken over a block Λ(x,y,M) of size M centred at (x,y):

(2) E{x,y) = Max(_X>,y>)εΛ(x,y,M)V(x',y')

Next E(x,y) is multiplied by a constant factor as an adjusting coefficient, escale, to obtain the epsilon matrix

(3) ε(x,y) = B(x,y) * escale

Intuitively the value ε(x,y) represents the "size" of nearby edges, so that it has a larger value for pixels near to (within M of) strong edges. Finally the epsilon matrix is used as the filter strength for an ε-filter, which we will call eps, which is applied to each pixel of the original luminance image to obtain a new luminance image Y'(x,y) = eps(ε, Mε, Y, x, y). To assist the definition of the ε-filter we first define (following Shohdoji)

T_ε[x] = 1, if \ x \ <ε Te[x] = 0, if \ x \ >ε

Then the ε-filter is given by

(4) _{eOs(ε M γ ∑} ,_{Λ .} { ∑(χ_W(,,M) T_s(x,_y)[Y(x'_; y') -Y(x,y)] ,Y(x', y') }

{ ∑(x_',y),Λ(χ,y,M) τ_ε(x,_y) [Y(X¹, y¹) -Y(^χ,y)] }

(Note that US2005117807 has an obvious error here in paragraph 0016, equation ( 1 ) , as it uses its equivalent of Y(x,y) as the multiplier in the numerator rather than Y(x',y')) .

The parameters are the field of epsilon values, ε; a measure, Me, of the size of the area over which the filter operates for each pixel; the original image, Y; and the location of the pixel to be calculated, (x,y). Intuitively this equation defines Y'(x,y) to be the average

Y of those neighbours of (x,y) within M pixels and value within ε(x,y) of Y(x,y). Thus small variations of size less than ε tend to be smoothed out, without compromising the visually important edge. This is illustrated in Figure 1 of the accompanying drawings which depicts a 1-D slice through a luminance 15 image near to two edges 1 1 and 12. The result 14 shows a reduction in mosquito noise 13.

For large values of M or N it becomes expensive in terms of computing power to calculate the result of an ε-filter. It is possible to compensate to some extent for small values of M and N by increasing escale. However, there are disadvantages to increasing escale, as we explain.

We observe that ε-filtering performs well at reducing noise near edges, especially when the correction parameter, escale, is large. However, it tends to create jaggy edges, by removing the softening intermediate values which allow diagonal edges to look smooth. In addition, it does not perform at all well at reducing noise not close to strong image edges, for example block noise. Figure 3 of the accompanying drawings illustrates this. The image 31 before processing exhibits mosquito noise 32, which has been reduced 34 to a much lower level in the image 33 after processing. However the smooth diagonal 35 has been made more jaggy 36.

Figure 7 of the accompanying drawings explains in more detail why this is. In each graph 7a, 7b the horizontal axis represents a 1 -dimensional section through an image, and the vertical axis represents brightness. On the left 7a the graph represents the original image, and on the right 7a is the same image processed by an ε-filter with a large value for escale. In each graph the section shows what happens at an edge, with dark values on the left of the edge and bright ones on the right. Since escale is large, the calculated values of ε are also large compared with the size of the edge. The effect is then that the pixels nearest the edge (filled in black) are drawn outward, nearer to their neighbours within ε. If the edge happens to be a diagonal one as diagrammed in 7c and 7d, then the effect is that the smooth transition from top to bottom of the edge in original image 7c has now become too sharp, resulting in jaggy edges as in processed image 7d. The pixels just on and below the main diagonal (such as those marked 71 ) in 7c correspond to the pixels marked with black circles in 7a.

SUMMARY OF INVENTION A first aspect of the invention provides an apparatus for processing a video image, comprising first means for determining an edge strength in a region of the image associated with an image pixel to be processed, and second means for applying an edge-preserving filter and for substantially not applying a smoothing filter at the pixel in response to a first edge strength and for applying the smoothing filter and for substantially not applying the edge- preserving filter at the pixel in response to a second edge strength less than the first edge strength. The smoothing filter may comprise an image-blurring filter followed by an image-sharpening filter.

The second means may be arranged to apply a combination of the smoothing filter and the edge-preserving filter weighted according to the edge strength for edge strengths between the first and second edge strengths.

The region may contain the pixel to be processed. The first means may be arranged to determine the edge strength as a function of variations in pixel values in a plurality of pixel blocks in the neighbourhood of the pixel to be processed.

The blocks may comprise sets of contiguous pixels in a single pixel row.

The first means may be arranged to determine the edge strength as a function of a maximum of the variations. The first and second means may be arranged to repeat their operations for each of a plurality of pixels of the image .

The plurality of pixels may comprise all image pixels excluding those pixels in a border region of the image .

The first and second means may be arranged to repeat their operations for each image of an image sequence .

The edge-preserving filter may comprise an epsilon filter. Additionally or alternatively the edge-preserving filter may comprise a directional filter.

A second aspect of the invention provides a display including an apparatus of the first aspect. A third aspect of the invention provides a method of processing a video image, comprising determining an edge strength in a region of the image associated with an image pixel to be processed, applying an edge-preserving filter and substantially not applying a smoothing filter at the pixel in response to a first edge strength, and applying the smoothing filter and substantially not applying the edge-preserving filter at the pixel in response to a second edge strength.

A fourth aspect of the invention provides a program for programming a computer to perform a method of the third aspect.

A fifth aspect of the invention provides a computer- readable medium containing a program of the fourth aspect.

A sixth aspect of the invention provides a computer programmed by a program of the fourth aspect.

The invention also provides an apparatus for processing a video image, comprising first means for determining an edge strength in a region of the image associated with an image pixel to be processed, and second means for applying an edge- preserving filter at the pixel in response to a first edge strength and for substantially not applying the edge- preserving filter at the pixel in response to a second edge strength greater than the first edge strength.

The second means may be arranged substantially not to apply the edge-preserving filter when the edge strength is at or adjacent a maximum value thereof. It may be arranged substantially not to apply the edge preserving filter when the edge strength is within 20% of the maximum valve. The second means may be arranged to apply a directional filter at the pixel in response to the second edge strength. It may be arranged substantially not to apply a smoothing filter at the pixel in response to the first edge strength and is arranged to apply the smoothing filter and substantially not to apply the edge-preserving filter at the pixel in response to a second edge strength less than the first edge strength.

The invention also provides a method of processing a video image, comprising determining an edge strength in a region of the image associated with an image pixel to be processed, applying an edge-preserving filter at the pixel in response to a first edge strength and substantially not applying the edge-preserving filter at the pixel in response to a second edge strength greater than the first edge strength.

The ε-filtering method may be modified by using E as a measure of edge strength, using filtering designed not to create jaggy edges when E is nearly maximal. We can determine the near-maximality from the fields V and M which are calculated anyway during the ε-filter method. Optionally, when E is small we use a generic smoothing filter (low pass filter) instead of ε-filtering. For intermediate values (non- small and non-maximal) we perform ε-filtering as usual. Optionally we blend filtering between these three regions to prevent sudden changes in processing which might be perceptible .

It is not necessary to use special information from a compression decoder other than the succession of fields or frames of video data^' itself. This information includes items such as frame rate, motion vectors, amount of quantisation and more. Such information from the decoder can optionally be used to tune the parameters for better performance. This method works well applied to each field or frame without necessarily requiring information from earlier fields or frames . However, information from earlier fields or frames can be used to tune the parameters for better performance.

Data from earlier frames can be used to adjust the filters and filter parameters adaptively, depending on the video content. For example, if the type of the video in the shot is known by analysis or otherwise to be one of cartoon, sport, interview, slow-moving scenery, computer generated, then different parameters can be selected. While in the prior art, square (NxN or MxM) rectangles have generally been used as the domain over which the measure of variation is calculated, the domain over which selective averaging is performed, and the domain of the optional low-pass filter, the present invention is not restricted to square domains . Square domains are included in some embodiments of the invention, but other types of domain may also be used. In particular, a domain which contains only one pixel row ( IxN or IxM) may be advantageous when memory or processing resources are restricted. Such techniques allow the suppression of mosquito noise without introducing more jaggy edges into the image, resulting in a more pleasing image .

Mosquito noise and block noise filters may be integrated with little extra cost, and without the introduction of more jaggy edges.

The implementation of these techniques may be cheap, as they make use of information, V and M which is readily available.

The escale parameter (strength of filter) may be increased to much higher values than the prior art without adding jagginess. This allows smaller values of M and N to be used, which in turn reduces cost of implementation. It allows processing of video which has undergone a larger amount of compression and consequent degradation. Such techniques may provide an image ready for enlarging and/ or sharpening and/ or other processing for display. In particular, sharpening tends to increase jagginess, so an advantage of the present technique is that j aggy edges are not increased before sharpening. With the prior art ε- filter jaggy edges would be increased both by the ε-filter and by subsequent sharpening.

In the case of embodiments using filtering domains containing a single row, the integration of a system for the removal of noise with a very small requirement for computing and memory resources is possible .

The foregoing and other objectives, features, and advantages of the invention will be more readily understood upon consideration of the following detailed description of the invention, taken in conjunction with the accompanying drawings.

BRIEF DESCRIPTION OF DRAWINGS

The invention will be further described, by way of example, with reference to the accompanying drawings, in which:

Figure 1 shows the operation of the standard epsilon filter;

Figure 2 illustrates some types of compression noise; Figure 3 illustrates processing with and without the new method;

Figure 4 illustrates processing with and without the new method;

Figure 5A illustrates an embodiment of the invention; Figure 5B illustrates a modified embodiment of the invention; Figure 6 illustrates the principal components of the system;

Figure 7 illustrates the reason for jaggy edges after epsilon filtering; Figure 8 illustrates the principle of operation of embodiments of the invention where filtering domains include only one row of video data; and

Figure 9 shows a number of different ways in which the invention may be incorporated into devices or products.

DESCRIPTION OF EMBODIMENTS

Figure 6 illustrates the principal components of the system. Video from a compressed source 61 (such as a digital satellite, cable or terrestrial broadcast receiver, or from a device such as a PVR, DVD player or Blu-ray disc player, or from an internet video service, or from a video conferencing service) is passed to a decompressor 62 to create an uncompressed video . An image processing unit 63 applies algorithms to reduce compression artefacts (and optionally applies other algorithms for other purposes, such as image scaling for the particular display unit. ) The cleaned video is then displayed on a panel or other display unit (such as a CRT) 64.

Figure 5A illustrates a preferred embodiment of the invention, typically forming a part of the image processing unit 63. An incoming image 500 that has been previously decompressed is processed to create an output image 508 with fewer visible compression artefacts . The Y component of the input image is passed as a parameter 53 to several parts of the process:

Figure 5A illustrates principal components of this embodiment of the invention, and also shows principal interactions between components.

The embodiment of figure 5A comprises first means for determining an edge strength in a region of the image associated with an image pixel to be processed. In this embodiment E is used as a measure of edge strength. As explained above, E is determined by determining a function V of variations in pixel values in a plurality of pixel blocks in the neighbourhood of the pixel to be processed, and determining E as a function of a maximum of the variations . The pixel blocks may be NxN rectangular blocks, or alternatively the blocks may comprise sets of contiguous pixels in a single pixel row. The embodiment further comprises second means for applying an edge-preserving filter and for substantially not applying a smoothing filter at the pixel in response to a first edge strength, and for applying the smoothing filter and for substantially not applying the edge-preserving filter at the pixel in response to a second edge strength less than the first edge strength.

In this embodiment the edge-preserving filter is an ε- filter 504, but the invention is not limited to this specific edge-preserving filter. In this embodiment the smoothing filter is a block filter that comprises an image-blurring filter

505 followed by an image-sharpening filter 506, but the invention is not limited to this specific smoothing filter.

One part 501 computes V(x, y) according to equation ( 1 ) above . Alternatively other measures can be used, for example variance, or some other measure of edge strength such as a

Canny edge filter. The result is passed to a unit 502 which computes E(x,y) according to equation (2) above . In this embodiment, therefore, the unit 501 and the unit 502 together constitute a "first means for determining an edge strength" since, as explained above , E forms a measure of the edge strength in this embodiment. This result is passed to a unit 503 which computes ε(x, y) by multiplying E by parameter escale 509, according to equation (3) . The result is passed to the ε-filter 504, which also receives the image 500 as a parameter, and computes according to equation (4) , sending the result 52 to the blend unit 507. The results V(x,y) 501 and E(x,y) 502 and escale 509 are also passed to a decision unit 51 which identifies a blend mode for each pixel position (x,y) so that the invention may be applied to each of a plurality of pixels of the image . The Y input image is also passed 53 to a blur unit (image-blurring filter) 505, and the blurred result passed to a sharpening unit (image-sharpening filter) 506, and the result of that 54 is passed to the blend unit 507. The purpose of the blur unit is to remove block noise, so a small gaussian blur is a possible implementation. The sharpening unit is used to optionally recover details of texture which are over-smoothed by the blur units. Any filter designed to remove block noise may optionally replace units 505 and 506. The blend unit 507 also receives the Y input image. Its purpose is to combine the three kinds of input image (original 53, block-filtered 54 and ε-filtered 52) using the result from the blend mode unit 51. The blend unit 507, or the combination of the blend unit 507 and the blend mode unit 51 , may thus be considered as a means that determines whether the epsilon filter 504 is applied or is substantially not applied and that determines whether the smoothing filter 505, 506 is applied or is substantially not applied. The blend unit may combine the inputs by weighting the three inputs and adding them, and the weights (WY, Wb and W_ε respectively) may be received from the blend mode unit 51. The blend unit may optionally scale and sharpen the resulting image for best viewing. Finally the resulting image may be recombined with the unprocessed colour components of the original image to produce a colour image ready for display. The above operations may be repeated for each image of an image sequence.

Calculation of the blend mode is important for this method. The basic idea is that when E(x, y) is large, then there is a nearby hard edge, so the ε-filter is applied but the block filter is not applied (or is substantially is not applied) .

When E(x,y) is small there are no hard edges nearby, and therefore the block filter is applied but the ε-filter filter is not applied (or is substantially is not applied) . That is, for a first (relatively high) edge strength the edge-preserving filter (ε- filter) is applied and the smoothing filter (block filter) is substantially not applied, whereas for a second edge strength less than the first edge strength the smoothing filter is applied and the edge-preserving filter (ε-filter) is substantially not applied.

The exception is when V(x, y) is roughly maximal, that is close in value to its local peak value E(x,y); then, in an optional embodiment, to avoid j aggedness the ε-filter is not applied (and so no filtering occurs) . That is, for a second edge strength higher than the first edge strength the edge- preserving filter (ε-filter in this embodiment) is not applied. One way to test for rough maximality of V(x, y) is to check if V(x,y) > E(x,y) - ti and E(x,y) > £2 for suitable threshold parameters ti and £2. Another method is to test if (E(x,y) - V(x,y)) > t'ι *E(x,y) for a different threshold parameter t'i . In the embodiment of figure 5A the blend unit determines whether or not a particular filter is applied by- setting the weight (WY, Wb and W_ε) of that filter appropriately. Preferably each filter runs continuously, even when processing a pixel for which that filter is not being applied.

In principle however, it would be possible for a filter to be turned off when processing a pixel for which that filter is not being applied (ie, when processing a pixel for which the filter has its weight set to zero) . Good results have been obtained using values M - 3 , N =

3 , escale = 0.6 , ti = 0.02 , £2 = 0.1 (here assuming data values vary from 0.0 black to 1 .0 white) , though -naturally values must be tuned for each particular display type or application, and expected viewing conditions . To avoid sudden transitions between regions it may be advantageous to blend between the block and ε-filter regions, so that over a small range of values of E(x,y) a portion of each is used. One possible way to achieve this is to adjust the weighting factors linearly in this region. For example, given a threshold £3 and a gradient parameter m, calculate a blending parameter r(x, y) as follows:

r(x, y) = max(0, min(l, m * E(x, y) - t₃)))

For example, m=9 and £3 = 0.2 are reasonable values . Then use r(x,y) to control the blend weighting parameters as follows:

if V(x,y) > E(x, y) - U and E(x,y) > t₂ Wγ(x,y) = 1; W_ε(x,y) = 0; W_b(x,y) = 0 else

Wγ(x, y) = 0; W_ε(x,y) = r(x,y);W_b (x,y) = 1 - r(x,y)

The blending operation 507 may be as simple as

Y'(x,y) = Wγ(x,y)*Y+ W_ε(x,y) * eps(ε, M_ε, Y, x,y) + W_b(x,y)*b(x,y)

where b(x,y) is the output 54 of the block filter 505, 506, and Y'(x,y) is the resulting monochrome image 508. For later reference we give the name A to the region with

Wγ(x,y) = 1, B to the region with W_ε(x,y) = 1, and C to the region W_b(x,y) = 1. We call D the blend region, in which 0 < W_ε(x,y) < 1 and 0 < W_b(x,y) < 1.

This blending technique uses the value E(x,y) to determine the blend control parameter r(x,y). This may lead to overly sudden blending in the presence of very sharp features (such as thin horizontal or vertical lines.) It may be preferable to provide a more gradual blend, which may be achieved by many methods including the following, used singly or in combination. For these variations we change the definition of r, and instead write r(x, y) = max(0, TnIn(I, m * B'(x, y) - t3))), where the primed E indicates a different, smoother, edge weighting function as given by example here. We note that ε(x,y) may be calculated (as in equation 4) using either E or E', since there may be an efficiency advantage in not calculating both of them.

For the first example we define E' to be the result of applying a smoothing filter (such as a gaussian kernel in one or two dimensions) to E. For the second example we obtain E' from E by modifying its definition. Whereas E is defined in terms of variance V (in equation 1) we may define E' in terms of V, where

V¹ (X, y) = {Σ_(x-_,yVA(x,y,N) [Y(X¹ , y>(x'-x, y'-y)

- (∑_(x"sy")SΛ(χ_;y,N) Y(x", y").w(x"-x,y"-y)/| Λ(x, y,N) | )]² / | Λ(x, y, N) |)]²}^1/2

and w(dx, dy) is a smoothing kernel, such as a gaussian.

For the third example we may obtain E' from E by modifying instead the equation 2 , so that

where again w '(dx, dy) is a smoothing kernel, such as a gaussian. It will be well understood by those skilled in the art that the various computations and units described above are primarily used for describing the method, and that any implementation may combine or separate calculations to create a different architecture that achieves the same effect, namely the blending of the different types of processing.

Implementation may be in software or in dedicated hardware or some combination. It may work on whole frames or fields at a time, or may work in a streamed mode with a limited amount of buffering, as is common in video processing.

For efficiency it may be advantageous to calculate the blend mode 51 for any particular pixel early on in the processing order, and then only calculate those inputs to the blend unit 507 which have a non-zero weighting. Near the borders of the image the algorithm is less likely to perform well. It may be advantageous to do no processing, or optionally to do other processing, within a few pixels of the borders .

Figure 3 illustrates the results of the processing. The image detail 31 without processing exhibits mosquito noise 32 , which has been reduced 34 to a much lower level in the image after ε-filter processing 33. However the smooth diagonal 35 has been made more j aggy 36. The image 37 created with this embodiment shows the same reduction 38 in mosquito noise but the smooth diagonal 39 is not significantly jaggier than the original. Figure 4 shows the same thing at higher resolution, using shading to represent grey levels. The scale of grey levels 41 is given for reference, from lightest 45 to darkest 46 pixels. 42 is the original image, 43 the image processed by the algorithm in the prior art and 44 the image processed according to the present embodiment. Note that 43 and 44 differ only near and along the sharp diagonal edge . The mosquito noise reduction well above or below the edge is identical. Operation of the noise reduction apparatus of figure 5A is not limited to the mode described above. For example, in another mode an edge-preserving filter may be applied at the pixel in response to one edge strength and the edge- preserving filter may be substantially not applied at the pixel in response to another, greater edge strength. Optionally in this mode, the edge-preserving filter may be substantially not applied when the edge strength is at or adj acent a maximum value thereof, for example the edge-preserving filter may be substantially not applied when the edge strength is within 20% of the maximum value . Optionally in this mode a smoothing filter may be substantially not applied at the pixel in response to the one edge strength, and the smoothing filter may be applied and the edge-preserving filter may be substantially not applied at the pixel in response to a further edge strength that is less than the first edge strength. In a further embodiment of the invention, the epsilon filter 504 of figure 5A may be replaced by a directional filter. A "directional filter" is a filter that enhances edges which run along one or more selected directions, while having no effect on edges that run along directions that are not one of the selected direction(s) . A directional filter is another example of an edge-preserving filter, and this embodiment may operate in any one of the modes described above (so, as an example , the directional filter may be applied and the smoothing filter may substantially not be applied at the pixel in response to a first edge strength, and the smoothing filter may be applied and the directional filter may substantially not be applied at the pixel in response to a second edge strength less than the first edge strength) . In a further embodiment of the invention, shown in figure 5B , the epsilon filter 504 of figure 5 A may be supplemented by a directional filter. The directional filter may precede the epsilon filter 504 (as indicated at 510 in figure 5B) or the directional filter may succeed the epsilon filter 504 (as indicated at 51 1 in figure 5B) ; of these two possibilities it may be preferred for the directional filter to succeed the epsilon filter 504 as indicated at 51 1 in figure 5A. Other features of the embodiment of figure 5B correspond to the embodiment of figure 5A, and their description will not be repeated. In the embodiment of figure 5B, the threshold of the directional filter may be the same as the threshold of the epsilon filter, so that the epsilon filter and the directional filter are applied and the smoothing filter is substantially not applied at a pixel in response to a first edge strength and the smoothing filter is applied and the epsilon filter and the directional filter are substantially not applied in response to a second edge strength less than the first edge strength. Alternatively, the threshold of the directional filter may be different from the threshold of the epsilon filter.

In an alternative embodiment, the unprocessed image is not passed directly to the blend unit for use in blend region A. Instead a directional filter is applied, according to edge direction. The aim is to reduce existing jaggy edges in the image, not simply to avoid adding more jaggy edges. The edge direction information can be obtained from the image directly, or using the results V(x,y) and E(x,y) calculated elsewhere 501 , 502.

In an alternative embodiment the choice of blending of the different filters is performed using morphological operations (that is, considering pixel adjacency) . In this embodiment regions A and B are calculated as before. Region D is calculated to be those pixels which border within a set distance on pixels of region A or B but which lie in neither A nor B . Region C is then calculated to be those remaining pixels not in A, B or D . In region D the blending weights are chosen to give values between B and C. For example, Wγ(x, y) = 0; W_ε(x, y) = ¹A; W_b(x,y) = Va.

In an alternative embodiment, the shapes of the filter regions Λ(x, y, M) need not be squares. Note that there are three distinct uses of Λ(x,y, ...) - for the variance calculation ( 1) , the maximisation of variance (2) and for the epsilon filter (4) - and each one could use a different filter shape. In one example, a rectangle with differing height and width are used. In another example, a region using a different metric, such as

| x'-x | + \ y '-y \ < M is used. The size or shape of the filter may be made to depend on the input image type, or on local properties of the image. For example, the parameter M may^¬ be chosen to vary at each pixel in accordance with the magnitude of V(x,y).

An example of an embodiment with non-square filter regions is one where all calculations are done using pixels from a single video row. In this case, as pixel data are supplied to the system serially row-by-row, the system need store only a few (probably less than 100 per colour channel) pixel values in order to perform the noise reduction function. This allows an implementation which runs at high speed and with low cost.

Figure 8 shows the principle of operation of this type of embodiment. An image 72 is shown, where element 53 is the pixel in the 5th row and 3rd column. Image data arrive serially as a stream 73 of pixel values. The noise reduction system 74 need only take into account nearby pixels in the data stream in order to calculate the output pixel stream 76 if all filtering domains have one row. Other operations 75 may also be applied before or after (as shown) the noise reduction system. In particular, 75 may be a sharpening operation which again operates only on a single pixel row. For example, the noise reduction system 74 may be of the type shown in Figure 5A.

Figure 8 shows operations on a single set of pixel data, as might be used to describe a monochrome image . Colour images may be processed in the same way, either by (a) ^•processing the red, green and blue channels separately and independently, (b) calculating a luminance or luma value or some approximation to luminance and processing only that luminance before recombining it with colour information, or (c) by using information derived from one channel of image information to control the blend function for all three channels .

In an alternative embodiment not all the colour channels need be processed. For example, since the eye is not particularly sensitive to blue light it is relatively unimportant to process the blue channel. The advantage of this embodiment is that less processing power or processing hardware is required.

In an alternative embodiment escale may be varied according to the style of image. For example, if it is known that an animated cartoon sequence is in progress it may be beneficial to increase escale to allow more reduction of mosquito noise . Here the flattening effect of the epsilon filter may be considered an advantage.

In an alternative embodiment extra information is taken from the video decoder, such as motion vectors or estimates of video quality (for example from the number of bits used for coding each block) , to control the strength and shapes of the various filters.

In an alternative embodiment processing is not restricted to the Y (luma) channel. If the image is in RGB colour space, or some other colour space, each channel may be processed independently by the filters.

In an alternative embodiment, the information used in the blend mode decision 51 is calculated solely from one of the RGB channels, and the same blending is applied to all three colours (red, green and blue) . Preferably the G channel is the one used in the blend mode decision, since the green brightness has the strongest influence on luminance .

In an alternative embodiment the image is already in, or converted to, a colour space with one luminance channel Y, and two colour channels which we will call C l and C2. Y is processed as before. However, for processing C l and C2 the system uses E(Y) as a substitute for E(C l ) and E(C2) respectively. This is for efficiency, as it requires only one E calculation to serve for processing all three channels, as compared with the previous embodiment.

In an alternative embodiment the E or ε fields are calculated more approximately, perhaps on a coarser grid. For example, a single ε value might be used for a 3x3 or 5x5 region of pixels. This kind of optimisation is discussed in US20051 17807 Shohdoji et al.

In the embodiment of figure 5A the edge-preserving filter is an epsilon filter. The invention is not however limited to this specific edge-preserving filter. In an alternative embodiment the invention is applied to a bilateral filter instead of an epsilon filter. The well-known bilateral filter is similar to an epsilon filter except that the weight of a pixel depends not only on the value of that pixel relative to a central pixel, but also on the distance of that pixel relative to a central pixel. Bilateral filtering may give rise to j aggy edges in the same way as epsilon filtering, so it is advantageous to use no filtering or a different filter at those pixels which lie close to edges.

Indeed, epsilon filters and bilateral filters are examples of edge-preserving filters. An edge-preserving filter is one which blurs features in an image but which tends to leave stronger edges unaffected. The present invention may be used with other edge-preserving filters, examples of which include : median filter and coring algorithms (both described by C . Poynton, "Digital video and HDTV algorithms and interfaces", Elsevier 2003 , p.33 1 ) ; a method based on

"bandlets" described by E. Le Pennec and S . Mallat, "Sparse geometric image representations with bandelets" , IEEE transactions on image processing, vol 14 (4) , pp423-438 (April 2005) ; and methods based on partial differential equations such as those described in D . Tschmperle . "Fast Anisotropic

Smoothing of Multi-Valued Images using Curvature-Preserving PDEs", International Journal of Computer Vision, IJCV (68) , No .1 , June 2006, pp.65-82.

In the embodiment of figure 5A the smoothing filter comprises an image-blurring filter followed by an image- sharpening filter. The invention is not however limited to this, and the invention may be applied in an analogous way to other kinds of smoothing filters.

The noise reduction method may be incorporated into a display system or an image manipulation system in many different ways. Some examples are shown in Figure 9 which describe, but are not intended to limit, the possibilities .

In Figure 9 (a) , the noise reduction system 78 is included (possibly together with other operations such as sharpening or scaling) in a separate device which may receive a video signal from a source 77 and feed video output to a display 79. Source 77 may for example be an optical disk player, a television tuner (analogue or digital) , a hard disk recorder or a computer. In Figure 9(b) , the noise reduction system 78 is included in the source device 77 and processes video before it is sent to the display device 79.

In Figure 9 (c) , the noise reduction system 78 is included in the display device (which may be a monitor, a television or another device which includes a display such as a mobile phone or personal digital assistant or movie player) .

Figures 9 (d) , 9 (e) and 9 (f) show more specific examples of how a noise reduction system may be included in a device which can be used for viewing television, such as a television set, mobile phone or personal media centre . Such devices typically include at least three processing stages. The TV signal entering the device at the tuner is first demodulated and decoded in a stage 80. The resulting signal is then adapted to the resolution and possibly to the frame rate of the display in a second stage 81 . This stage may include scaling and may include frame-rate conversion. The final stage 82 is to adapt the video signal to the specific characteristics of the display device 83 (for instance, CRT or flat panel or proj ector) . The stage 82 typically takes into account the voltage range and the luminance-data relationship of the display device 83 , and includes such elements as look-up tables and timing controllers.

Figure 9 (d) shows the noise reduction system 78 incorporated in the same hardware as the demodulating/ decoding step. This may be advantageous if parameters derived from decoding are used to control the blending or filtering steps in the algorithm.

Figure 9 (e) shows the noise reduction system 78 in the same hardware as the scaling step 81 . This may have the advantage that when the signal displayed comes from a source other than the built-in tuner/ decoder (for example an external feed from a DVD player) , the noise reduction may still be applied.

Figure 9 (f) shows the noise reduction system 78 incorporated into the part of the system 83 which adapts the signal to the display panel. Again, this may have the advantage that the noise reduction may be applied to data from a number of different sources inside or outside the television set. In this part of the system, memory and processing resources are typically limited, so the one-row version of the system described in Figure 8 may be used.

A noise reduction system that embodies the present invention may be implemented in hardware, for example using dedicated hardware circuits . As an example , the noise reduction system 78 of any one of figures 9 (a) - 9 (f) may be implemented using dedicated hardware circuits .

Alternatively, a noise reduction system that embodies the present invention may be implemented in software under the control of a suitable program, for example a program that controls the noise reduction system 78 of any one of figures

9(a) - 9(f) to carry out a method of the invention. The program may be contained on a suitable computer-readable medium. The noise reduction system 78 of any one figures

9 (a) - 9 (f) may for example be implemented as a computer programmed with a suitable program.

The invention being thus described, it will be obvious that the same way may be varied in many ways. Such variations are not to be regarded as a departure from the spirit and scope of the invention, and all such modifications as would be obvious to one skilled in the art are intended to be included within the scope of the following claims.

INDUSTRIAL APPLICABILITY

As will be understood from the above description, the invention is clearly industrially applicable.

Claims

1. An apparatus for processing a video image, comprising first means for determining an edge strength in a region of the image associated with an image pixel to be processed, and second means for applying an edge-preserving filter and for substantially not applying a smoothing filter at the pixel in response to a first edge strength and for applying the smoothing filter and for substantially not applying the edge-preserving filter at the pixel in response to a second edge strength less than the first edge strength.

2. An apparatus as claimed in claim 1 , in which the smoothing filter comprises an image-blurring filter followed by an image-sharpening filter.

3. An apparatus as claimed in claim 1 or 2 , in which the second means is arranged to apply a combination of the smoothing filter and the edge-preserving filter weighted according to the edge strength for edge strengths between the first and second edge strengths .

4. An apparatus as claimed in any one of the preceding claims, in which the region contains the pixel to be processed.

5. An apparatus as claimed in any one of the preceding claims, in which the first means is arranged to determine the edge strength as a function of variations in pixel values in a plurality of pixel blocks in the neighbourhood of the pixel to be processed.

6. An apparatus as claimed in claim 5, in which the blocks comprises sets of contiguous pixels in a single pixel row.

7. An apparatus as claimed in claim 5 or 6, in which the first means is arranged to determine the edge strength as a function of a maximum of the variations.

8. An apparatus as claimed in any one of the preceding claims, in which the first and second means are arranged to repeat their operations for each of a plurality of pixels of the image.

9. An apparatus as claimed in claim 8, in which the plurality of pixels comprises all image pixels excluding those pixels in a border region of the image .

10. An apparatus as claimed in claim 8 or 9, in which the first and second means are arranged to repeat their operations for each image of an image sequence.

1 1 . An apparatus as claimed in any one of the preceding claims, in which the edge-preserving filter comprises an epsilon filter.

12. An apparatus as claimed in any one of the preceding claims, in which the edge-preserving filter comprises a directional epsilon filter.

13. A display including an apparatus as claimed in any one of the preceding claims.

14. A method of processing a video image, comprising determining an edge strength in a region of the image associated with an image pixel to be processed, applying an edge-preserving filter and substantially not applying a smoothing filter at the pixel in response to a first edge strength, and applying the smoothing filter and substantially not applying the edge-preserving filter at the pixel in response to a second edge strength less than the first edge strength.

15. A program for programming a computer to perform a method as claimed in claim 14.

16. A computer-readable medium containing a program as claimed in claim 15.

17. A computer programmed by a program as claimed in claim 15.