WO2010035700A1

WO2010035700A1 - Method and device for image sharpening technique

Info

Publication number: WO2010035700A1
Application number: PCT/JP2009/066315
Authority: WO
Inventors: Hao Pan; Yeping Su
Original assignee: Sharp Corp
Current assignee: Sharp Corp
Priority date: 2008-09-23
Filing date: 2009-09-09
Publication date: 2010-04-01
Anticipated expiration: 2011-03-23
Also published as: US20100074548A1; US8351725B2

Abstract

A sharpening technique for digital images for presentation on a display. In the technique, an image having a first sharpness at a first scale is received. The image is filtered with an image filter to result in the creation of a first channel and a second channel, where the first channel includes primarily texture information and the second channel includes primarily edge and flat region information. The first channel is filtered to attenuate higher frequency content of the image. The second channel is filtered to sharpen the image. The image is upscaled from the first scale to a second scale, where the upscaling, the filtering of both channels are based upon a plurality of the same pixels of said image from a same buffer. Finally, the first channel, the second channel, and the upscaled image are combined to form the modified image having the different scale.

Description

DESCRIPTION

TITLE OF INVENTION:

METHOD AND DEVICE FOR IMAGE SHARPENING TECHNIQUE

TECHNICAL FIELD

The present invention relates to sharpening of digital images and video for presentation on a display.

BACKGROUND ART

The resolution of a typical liquid crystal display is approximately 1920 x 1080 , which is on the order of standard definition television. The resolution of a typical very high resolution diagonal display may be 4096 x 2 160 or 3840 x 2160 , and typically has viewing distances that are less than 2 picture heights . In order to display an image on a display in an aesthetically pleasing manner sharpening of the image is typically performed. The lack of sharpness typically results from imperfections in the capture process and/ or loss of details due to image compression. However, sharpening of the image tends to result in annoying artifacts .

The traditional approaches to sharpening, such as using an unsharp mask, generally result in the introduction of visible degradation into the image being sharpened. The visual degradation is primarily due to enhancing the high frequency components and noise that often dominates the high frequency bands of an image. To moderate the enhancement of the visible degradation of the image, the amount of sharpening performed is controlled. It is desirable to increase the sharpness of the image content while at the same time suppressing the noise in the resulting image.

SUMMARY OF INVENTION One embodiment of the present invention may be directed towards a method for modifying an image from a first scale to a modified image having a different scale. The method comprises the steps of:

(a) receiving said image having a first sharpness at said first scale;

(b) filtering said image with an image filter to result in the creation of a first channel and a second channel, wherein said first channel includes primarily texture information and said second channel includes primarily edge and flat region information;

(c) filtering said first channel to attenuate higher frequency content of said image in said first channel;

(d) filtering said second channel to sharpen said image in said second channel; (e) upscaling said image from said first scale to said second scale, wherein said upscaling, said filtering of said second channel, and said filtering of said first channel are based upon a plurality of the same pixels of said image from a same buffer used to store said image at said first sharpness and at said first scale;

(f) combining said first channel, said second channel, and said image upscaled to said second scale to form said modified image having said different scale.

Another embodiment of may comprise of a method for modifying an image from a first scale to a modified image having a different scale. The method having the steps of:

(a) receiving said image having a first sharpness at said first scale;

(b) upscaling said image from said first scale to said second scale;

(c) filtering said image with an image filter to result in the creation of a first channel and a second channel further based on said upscaled image and said image, wherein said first channel includes primarily texture information and said second channel includes primarily edge and flat region information;

(d) combining said first channel and said second channel to form said modified image having said different scale. Furthermore a device may be provided for modifying an image from a first scale to a modified image having a second scale. The device comprising:

(a) a buffer for storing said image;

(b) an image filter for receiving said image having a first sharpness at said first scale and filtering said image to result in the creation of a first channel and a second channel, wherein said first channel includes primarily texture information and said second channel includes primarily edge and flat region information; (c) a first filter to filter said first channel to attenuate higher frequency content of said image in said first channel;

(d) a second filter to filter said second channel to sharpen said image in said second channel;

(e) an upscaling section to upscale said image from said first scale to said second scale, wherein said upscale section, said first filter and said second filter use a plurality of the same pixels of said image from said buffer; and

(f) a combining section to combine said first channel, said second channel, and said image upscaled to said second scale to form said modified image having said second scale.

The foregoing and other objectives, features, and advantages of the invention will be more readily understood upon consideration of the following detailed description of the invention, taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 illustrates image sharpening without resolution change. FIG. 2 illustrates image sharpening with resolution change.

FIG. 3 illustrates another image sharpening with resolution change .

FIG. 4 illustrates another image sharpening with resolution change .

FIG. 5 illustrates another image sharpening with resolution change .

FIG. 6 illustrates another image sharpening with resolution change . FIG. 7 illustrates bilateral filter with pre-filtering.

FIG. 8 illustrates comfort noise or texture equalization for noise suppression.

FIG. 9 illustrates another image sharpening with resolution change . FIG. 10 illustrates a combined upscaling, noise reduction, and edge sharpening technique .

FIG. 1 1 illustrates another combined upscaling, noise reduction, and edge sharpening technique .

FIG. 12 illustrates a further combined upscaling, noise reduction, and edge sharpening technique. FIG. 13 illustrates a modified combined upscaling, noise reduction, and edge sharpening technique with reduced memory requirements.

FIGS . 14A and 14B illustrate a pair of different window configurations .

FIG. 15 illustrates an edge sharpening and noise control module suitable for use with FIGS . 14A and 14B .

FIG. 16 illustrates the sharpening technique of FIG. 15.

FIG. 17 illustrates the noise control technique of FIG. 15. FIG. 18 illustrates the result of sharpening and noise control of FIG. 15.

FIG. 19 illustrates an example of a combined upscaling, noise reduction, and edge sharpening technique.

DESCRIPTION OF EMBODIMENTS

Image sharpening techniques are designed to enhance the high frequency aspects of the image. High frequency aspects, such as edges around major features of the image, are particularly desirable to sharpen in order to improve the visual appearance of the image to the viewer. However sharpening high frequency aspects, such as noise in the texture regions of the image, are not desirable because it degrades the visual appearance of the image to the viewer. These tradeoffs are generally contradictory to one another, while on the one hand there is a desire to enhance the high frequency aspects of portions of the image and on the other hand there is a desire not to enhance the high frequency- aspects of other portions of the image.

Attempting to develop a universal sharpening filter has proved to be elusive. It has been determined that sharpening an image is best done using multiple different filters or otherwise a filter applied to only a portion of the image . Each of the different filters should filter the image in a different manner so that the beneficial aspects of each of the filters may be taken advantage of. It is to be understood that any filter may be used, such as those that sharpen the image .

Referring to FIG. 1 , it was determined that the system (device) may preprocess the incoming image with a non-linear edge-adaptive smoothing filter 100 (image filter) that is used to separate the image into multiple "channels" . It is noted that these "channels" need not be frequency bands, or directly related to the frequency domain. The filter 100 may be referred to as an edge-adaptive smoothing filter or any other suitable filter. One channel 102 (second channel) may contain the edge content and slowly varying "flat" regions of the image where edges (which generally contain high spatial frequencies) are retained and remain generally sharp, but textures (which also contain high spatial frequencies) are generally filtered out. The other channel 104 (first channel) may contain the textures of the image minus edges. A linear filter is not especially suitable for such selective filtering.

These two channels, namely, texture information (T) 104 and edge + flat regions 102 (E) , can be used in a number of ways to produce a high quality sharpened image. One approach is to use a sharpening technique 106 (second filter) on the edge + flat regions 102 to enhance the image, such that the image is at a first sharpness but is modified to a second sharpness. The edge + flat regions 102 is generally noise free so the enhancement of that channel does not substantially increase the resulting noise in the final image.

A noise suppression technique 108 (first filter) on the texture channel 104 reduces the noise that is contained in that channel. Thereafter, the two channels T 104 (texture channel) and E 102 (cartoon channel) are recombined 1 10 (combining section) into the final output 1 12. Accordingly, those edge aspects of the image that are desirable to sharpen are suitably enhanced without substantial enhancement of the noise, while the other aspects of the image that are not desirable to sharpen are suitably attenuated in a manner to reduce the noise.

When the input resolution and the desired output resolutions are different, one or more scaling operations may be included to re-scale the image . Referring to FIG. 2, one technique for scaling the image is to include an image scaling operation 200 (upscaling section) at the output thereof. In this manner, the image processing pipeline may process the image at a first input resolution (first scale) then use a high quality scaling operation 200 to achieve the desired output results, for example a modified image at a different resolution (second scale) . This manner of processing maintains the image processing pipeline with relatively low complexity

Referring to FIG. 3 , another technique for scaling the image is to include an image scaling operation 2 10 at the input thereof. In this manner, the image processing pipeline may be designed to process images at a single image resolution that is the same as the scale provided at the output of the input image scaling operation 210 , which decreases the complexity of the image processing pipeline . The image processing pipeline may be optimized for a single image scale.

Referring to FIG. 4, another technique for scaling the image is to include an image scaling operation 220 after the noise suppression 108 in the texture channel (T) 104. The texture channel 104 has particular image characteristics and the scaling filter 220 may be designed to accommodate the particular image characteristics. The combination of the scaling operation and noise suppression may be such that improved image characteristics are achieved. An image scaling operation 240 after the sharpening 106 in the flat images + edges channel (E) 102 , may also be provided. The channel E 102 has particular image characteristics and the scaling filter 240 may be designed to accommodate the particular image characteristics. The combination of the scaling operation and sharpening may be such that improved image characteristics are achieved.

Referring to FIG. 5, another technique for scaling the image is to include an image scaling operation 230 before the noise suppression 108 in the texture channel (T) 104. The scaling 230 may be designed to accommodate the added noise in the channel 104 that is thereafter suppressed by the noise suppression 108. The combination of the scaling operation and noise suppression may be such that improved image characteristics are achieved. An image scaling operation 250 may be provided before the sharpening 106 in the flat images + edge channel (E) 102. The scaling 250 may be designed to accommodate the reduced noise in the channel 102. The combination of the scaling operation and sharpening may be such that improved image characteristics are achieved.

Referring to FIG. 6, depending on the particular implementation requirements, one or more of the scaling operations may be included together with the system. The combination of the different scaling operations achieves a robust design that is readily adaptable to different images and system requirements. It is to be understood that each of the scaling operations may be the same or different. By way of example, the scaling 230 may be a bilinear upscaling operation whereas scaling operation 250 may be a non-linear scaling operation.

More specifically, the edge adaptive smoothing filter 100 applies signal dependent processing to separate strong object edges from fine texture and noises. As one specific example, the system may use bi-lateral filters. One such bilateral filter may be formulated as

which consists of a weight calculation step and a weighted averaging step. For notational simplicity one may assume a one-dimensional filter but there is no loss of generality in doing that. Here ^x is the input signal and * is the filtered signal and the filter kernel ^ consists of a spatial kernel ^s and a range kernel ^w* : W[k,n] = W_s[k,n] . W_R\k,n\ Boxcar, gaussian and triangular kernels may likewise be used for kernels. Other techniques, such as anisotropic diffusion and adaptive smoothing may likewise be used. In general, any filter that tends to preserve edge features while reducing noise may be used. The kernel calculation in bilateral filters is signal dependent and serves to differentiate strong object edges from texture / noise / artifact. This is achieved by assigning lower weights to pixels in the filter window whose intensity values are far from the center pixel intensity. Preferably, a pre- filtering step * W 300 before the weight calculation 302 and weighted average 304 is incorporated to achieve better noise and artifact robustness. By reducing the noises and artifacts, the system may improve the weight calculation, which in turn improves the filtered outcome . This is illustrated as a block diagram in FIG. 7. Several choices of * 300 may be used, including for example, linear low pass filters and median filters. Other nonlinear smoothing filters can also be applied.

It is desirable to suppress noise in the texture channel

104. Since the texture channel 104 primarily contains all the noise and artifacts, attenuating the texture channel 104 will suppress the amount of noise and artifacts in the final output. This can achieved by applying a weight ^ω to the texture channel 104, and the weighting can be either spatially invariant or spatially variant. In the case of spatially variant texture channel weighting, the information from the bilateral filter can be leveraged to determine the amount of suppression. More specifically an edge measure

W[k,0]

E[k] =- N

∑W[k,n] can be derived from the bilateral filter, with higher E values indicating that the current pixel is closer to an edge . Subsequently the texture channel weights can be derived by ω[k] = {l-E[k])* a +β _ _{H ere} a,β _{are introduce}d to control the range and mean for the texture channel weighing.

It is desirable to suppress coding artifacts in the texture channel 104. Most of the displayed contents are compressed one way or another, and inevitably contain various compression artifacts . Among those the most visually objectionable is the blocking artifacts due to the non- overlapping block transforms used in most existing compression algorithms. The blocking artifact manifest itself as small square blocks all over the decompressed images, and most noticeably in flat regions. A deblocking filter may be applied on the texture channel 104 to suppress blocking artifacts. The main difference between the texture channel deblocking and a conventional deblocking is that the texture channel 104 is very noisy and the real obj ect edges are not present or not as strong. One may apply the following steps for the deblocking of the texture channel 104 :

First, calculate an edge measure for each pixel by using

W[k,0]

E[Jc] = - N

∑W[k,n] n=-N and an edge measure for a block by summing up the per-pixel measure .

Second, in the texture channel deblocking process, control the amount of filtering per block edge using the edge measure calculated in the previous step. The smaller the edge measure, the stronger the filter is.

It is desirable to modify the image to provide a sufficient amount of high frequency content so as to have a pleasing appearance. Attenuating the texture channel 104 reduces the noise at the expense of also removing fine detail. This may reduce the perceived sharpness to a human observer, as the resulting output image has less high frequency information than expected. To overcome this issue, texture synthesis may be added to the image to mask structured, coding noise artifacts without reducing high frequency information. A block diagram for the comfort noise is shown in FIG. 8. To be clear, FIG. 8 provides a more detailed description of the noise suppression block 108 shown in FIG. 1.

As can be seen from FIG. 8 , the noise suppression process takes the texture channel 104 as input. The texture data is then analyzed by an image analysis process using an image analyzer 400. In one embodiment, the image analyzer 400 extracts the variance and correlation of image regions within the texture channel 104. Of course, other statistics may be extracted. For example, the location of edges may be identified. By way of example, the mean value of the image region may be extracted.

Outputs from the image analyer block 400 are used to control filtering 402 and multiplication operations 404. For example, the correlation estimate controls the linear filter 402 , which receives input from a random noise generator 410 shown in FIG 8. Additionally, the noise power estimate controls the multiplication operator 404 that follows the linear filter output. Furthermore, the mean and edge information is used to control both the multiplication 404 following the linear filter as well as the multiplication of the input data 406. Finally, the scaled input texture channel 104 and the modified random noise are added 408 and provided as output of the noise suppression component.

In many cases, the non-linear splitting filter will not exactly separate the edges from the textures. Accordingly, some information will leak into each channel from the other channel. Consequently, the adding back together of the two channels may be based upon sensed image characteristics to achieve the final image . Referring to FIG. 9 , a more generalized framework may be used for an improved scaling technique. The technique includes a multi-channel spatial decomposition, which may decompose the image into any suitable set of channels. The decomposition may be linear or non-linear. In addition, the spatial decomposition may be more than two channels, if desired. The channels may include characteristics, such as for example, graphics versus natural, text versus non-text, face versus non-face, texture, and edges + flat region, scrolling text, film grain noise. Depending on the particular implementation, the input image for one or more channels may not be filtered but rather scaled in any manner. The channel specific scaling may use any suitable set of filters or be omitted. The channel specific noise reduction may use any suitable set of filters or be omitted. If desired, the filters may also include temporal information of a video when scaling a particular image. Blending together of the separate channels may be done using any suitable technique .

The upscaling, sharpening, and noise reduction are typically determined based on a two-dimensional moving window whose center moves across every pixel of the input image . To form a 5x5 moving window in an image, the memory used includes buffers having at least 4 consecutive horizontal lines of the image plus 5 pixels. If each horizontal line of the images contains 1920 pixels (full HD) , forming a 5x5 window needs to buffer at least 1920*4+5 pixels. A 5x5 window after 2x upscaling needs at least 1920*8 + 5 pixels, which is about twice as much memory as before upscaling, because each line of the upscaled image now contains 1920*2 pixels after 2x upscaling. Unfortunately, the line buffers are a dominant factor in the overall memory cost of implementing the moving window technique. Accordingly, it is desirable to reduce the line buffers in any particular implementation.

FIG. 10 illustrates a block-diagram that includes a bilateral filter module 500 (image filter) that decomposes the signal into a baseline channel 502 (second channel) and a texture channel 504 (first channel) . As previously described, the baseline channel 502 has the low pass and edge components of the signal, and the texture channel 504 has the texture and noise / artifact components of the signal. The bilateral filter module preferably uses a 9x9 window that requires 8 line buffers in memory 506. The two upscaling modules 508 and 5 10 (upscaling section) upscale the baseline and texture channels by a factor of 2 , respectively, and each preferably uses a 5x5 window. The respective buffers 512 and 514 (memory) each requires 4 line buffers . The unsharp masking (USM) module 516 (second filter) applies the sharpening function to the baseline channel 502 and preferably uses a 1 1x 1 1 window that includes 10 line buffers at the 2x resolution which is effectively 20 line buffers at memory 518. An additional memory 520, which has 5 line buffers at the 2x resolution which is effectively 10 line buffers, is used to synchronize the pixels from the baseline and the texture channels before the final adder 522 (combining section) . In total, the memory requirement of this technique counted by pixels is 8* 1920+4* 1920+4* 1920+ 10* 1920*2+5* 1920*2=36* 1920. This technique has significant memory requirements.

Referring to FIG. 1 1 , an alternative technique to upscaling + noise removal + sharpening is illustrated. In FIG. 1 1 , the upscaling 530 (upscaling section) is moved to the last module, and the sharpening (USM, second filter) 532 and noise control (bilateral filter, first filter) 534 become preprocessing of the upscaling 530. To have the noise control and sharpening performance close to the technique shown in FIG. 10, the bilateral filter 534 may use the same-size 9x9 window that uses 8 line buffers at Ix resolution and the USM 532 may use a much smaller 5x5 window that only uses 4 line buffers at Ix resolution, rather than generally twice as many line buffers at 2x resolution. In addition, there is only one upscaling module 530 in this technique which preferably uses a 5x5 window and associated 4 line buffer. In total, the memory requirement of this technique counted by pixels is 8* 1920+4* 1920+4* 1920= 16* 1920. The memory requirement of this technique is considerably lower than the requirement of the technique illustrated in FIG. 10.

Referring to FIG. 12 , another technique utilizing upscaling + noise removal + sharpening is illustrated. In this technique, the upscaling 540 (upscaling section) is the first module, and sharpening 542 (second filter) and noise control (first filter) 544 are post-processing of the upscaling 540. To 66315

19

have the noise control 544 and sharpening performance close to the technique shown in FIG. 10, the bilateral filter 544 has a much larger 17x17 window that uses 16 line buffers at 2x resolution; the USM 542 uses a same-size 1 1x1 1 window that uses 10 line buffers at 2x resolution; and the upscaling module 540 uses a 5x5 window that uses 4 line buffers at Ix resolution. In total, the memory requirement of this technique counted by pixels is

16*2* 1920+ 10*2* 1920+4* 1920 = 52* 1920. The memory requirement of this technique is considerably greater than the memory requirement of that illustrated in FIG. 10 and FIG. 1 1.

While the previously described memory management systems are sufficient, it is preferable to further reduce the memory requirements to reduce the associated expense. Memory reduction may be achieved by performing the upscaling and sharpening using the same memory and/ or the same window. To achieve this use of shared memory for upscaling and sharpening, these functions are preferably performed in a generally parallel fashion, rather than in a cascading fashion each of which using its own memory buffer.

Further, the sharpening and noise control may be postprocessing to the upscaling, but once again share the same memory buffer with the upscaling, rather than each having its own memory buffer. Thus, by reuse of all or portions of the joint memory buffer for different operations, the memory requirements may be substantially reduced.

Referring to FIG. 13 , a block-diagram of one implementation of a memory efficient technique is illustrated. The technique includes a memory 600 (buffer) that buffers a selected set of scan lines of the image, such as an 8 line buffer. A window within the memory 600 is used as an input to an upscaling module 602 (upscaling section) , which may upscale by any amount, and preferably be a scale of 2. Preferably, the memory used by the upscaling module 602 is a 5x5 window 612. The same memory 600 may likewise be used for edge sharpening (second filter) and noise control (first filter) modules 604, 606, 608, 610. Preferably, the edge sharpening and noise control uses an 8x8 window 614. The edge sharpening and noise control window 614 is preferably larger than the upscaling window 612. Likewise, the edge sharpening and noise control window 614 is preferably overlapping with the upscaling window 612.

Without loss of generality and for the convenience of discussion, the description includes an input having two- dimensional (2-D) raster scanned image that is upscaled by a factor of 2 both horizontally and vertically. The input pixel first passes through a memory 600 that is preferably first-in- first-out.

The upscaling module 602 interpolates four new pixels from an nx n window moving across the input 2-D image. The four new pixels are indexed by up-left {ul) 620, down-left {dl) 622, up-right (ur) 624, down-right (dr) 626, respectively. Each of the four "edge sharpening and noise control" modules 604, 606, 608, 610 sharpens and de-noises each one of these four new pixels from the upscaling module 602, respectively.

Accordingly, it may be observed that the pre-upsampled image data is used for upscaling, edge sharpening, and noise control.

The four new pixels, ul 620, dl 622, ur 624, and dr 626, may have at least two different positions relative to the input pixels, which are illustrated in FIG. 14. As illustrated in FIG.

14A position 1 for pixel ul is co-sited with the central pixel of the window. As illustrated in FIG. 14B for pixel ul, is ¹A pixel off from the central pixel horizontally and vertically. The different relative positions of the pixels result in different filter coefficients of the bilateral and the lowpass filters in the modules.

The pixels in a window may be indexed by the coordinate system with origin at the central pixel of the window, as illustrated in FIGS. 14A and 14B. The four new pixels ul, dl, ur, dr, generated by the upscaling module 602 may have different indexes for the two positions, namely, position 1: ul: (0, 0), dl: (0, -0.5), ur: (0.5, 0), and dr: (0.5, -0.5) position 2: ul: (-1/4, 1/4), dl: (-1/4, -1/4), ur: (1/4, 1/4), and dr: (1/4, -1/4) It is to be understood that the calculated pixels may be one or more pixels, which likewise may be located at any position.

The upscaling module 602 may interpolate new pixels that are missing in the low-resolution images. The upscaling technique may be any linear or non-linear upscaling technique that uses a selected set of pixels, such as a window with a size less than or equal to 9x9. For example, bi-cubic linear interpolation or edge-directed nonlinear interpolation techniques may be used.

The four edge sharpening and noise control modules 604, 606, 608, 610 sharpen the four new pixels from the upscaling module 602 and likewise reduce their noise level. To perform these functions, the modules may use a window of pixels. In contrast to existing sharpening/ noise control techniques that use the pixels after upscaling to form the window, the preferred technique uses the pixels before upscaling to form the window for sharpening and noise reduction after upscaling. Using pixels before upscaling to form the window for edge sharpening and noise control has advantages over using pixels after upscaling. By way of example, it not only reduces the size of window to % of the size of the window using upscaled pixels, but also facilitates sharing the window that the upscaling module 602 uses. It is appropriate to form this window using pixels before upscaling. Upscaling either does not add any new information or just adds a small amount of new information that makes little contribution to sharpening and noise control and can be effectively ignored. For example, many linear upscaling approaches can be seen as a different approximation of the sine function-based upscaling approach. In the frequency domain, the image after the sine function- based linear upscaling has the same frequency components as the image before upscaling. The new pixels interpolated by the sine function-based linear upscaling approach do not provide any new information. Experiments also confirm that forming the window from pixels after upscaling and forming the window from pixels before upscaling have minimal effect on the sharpening and noise control performance . There are preferably four "edge sharpening and noise control" modules 604, 606, 608 , 610. The module responsible for pixel ul, is illustrated in detail in FIG. 15, and will be discussed in detail in the following paragraphs. The remaining three modules may have the same structure as the ul module but different filter coefficients due to the different positions of new pixels.

The "edge sharpening and noise control" module is based on linear lowpass filter and nonlinear bilateral filter. Mathematically, the input and output relationship of this module may be described as f = <^<y^^yζ⁾ +^{β * (}*^w/ -J&)+ ti_*. edge sharpening »°ύe control Eq . ( 1 )

where x^ul is the input new pixel generated by the upscaling module

602 , yJ"_v2 is the output of a bilateral filter 2 , y"_v'_l is the output of a bilateral filter 1 , and yl'_p' is the output of the linear lowpass filter. y"' is the output of this module.

The two parallel sub-modules of the ul module, noise control are utilized by the first term in equation ( 1 ) , and edge sharpening is utilized by the second term in equation 1. The two sub-modules are parallel, sharing the same 9x9 window. The first term in equation ( 1) is the scaled difference between the linear lowpass filtered result y"_p' and the nonlinear bilateral filtered result yJ"_v] . The scaling factor determines the sharpening effect. The linear lowpass filter smoothes both noise and edges . The nonlinear bilateral filter, on the other hand, only smoothes noise but preserves edges.

Therefore, the difference between their results contains primarily edge information. Scaling up this difference has the effect of sharpening edges.

The lowpass filtering, the bilateral edge-preserving filtering, and their difference are conceptually illustrated by a simple I D signal in FIG. 16. FIG. 16(a) illustrates a I D signal that contains a soft edge and random Gaussian noise (5% of the signal strength) . FIG. 16(b) shows the signal that has been processed by an 1 1 -tap bilateral filter, where the noise is removed but the sharpness of the edge is preserved. FIG.

16(c) shows the signal that has been processed by an 1 1 -tap linear filter, where the noise is removed and the edge is softer. FIG. 16(d) shows the difference between (b) and (c) , in which noise is cancelled out with each other and two strong spikes survive .

The preferred linear lowpass filter and bilateral filter 1 are below. The coefficients of the filters are different in the two cases where the pixels interpolated from the upscaling module 602 have two different positions relative to the input pixels.

Position 1 (new pixel ul has the coordinates of (0 , 0) , FIG. 14A:

The new pixel ul does not need to be generated, i. e . , x^ul=xo,o. The linear lowpass filter is

& yv A N _/ ,., _\ Eq. (2)

^Δlp i=-Nj=-N

where '^ are the filter coefficients, and ^iJ are the pixels in the window. The window is defined by JV and the window size is 2N+ 1 by 2N+ 1. The normalization factor is

The filter coefficients w ^IuJI in equation (2) are derived from a 2-D Gaussian kernel (σ) centered at the pixel ul, which has the coordinates of (0 ,0) :

w* = exp(-(z² /2σf +/ /2σf)) Eq. 3

Note that one may drop the normalization factor in equation (3) that usually exists in a Gaussian kernel.

The bilateral filter 1 is

where the normalization factor is ^¹ -" ^{IJ h}} ul

In equation (4) , '-■> are same as the coefficients in equation (2) , which are defined by equation (3) , and are called spatial coefficients in the bilateral filter.

In equation (4) , ^1J are called range coefficients, and are defined on amplitude difference with pixel ul: vu ^{= ex}p(-<Λj -^χo,o)^{2 l2σ}l) _Eq> (₅)

Equation (5) is a 1 -D Gaussian kernel without the normalization factor. This Gaussian kernel may be referred to as a range kernel, and the Gaussian kernel in equation (3) may be referred to as a spatial kernel.

Position 2 (new pixel ul has the coordinates of (-1/4,1/4), FIG. 14B:

The new pixel ul is generated.

The linear lowpass filter is

Z"' = W^M/ -4- ^^-1 w^ul where the normalization factor is "' ^ ^lJ .

The filter coefficients in equation (6) are derived from a 2-D Gaussian kernel centered at position ul, which is the same as the previous case. However, because position ul is at (-1/4,1/4), not (0, 0), the kernel is different from equation 3:

<^/ ₇=exp(-((/ + l/4)²/2σ₁ ²+(j-l/4)²/2σ₁ ²)) Eq.7

and "*"¹.

The bilateral filter 1 is

S vs^υv^ll - -

where the normalization factor of bilateral filter is

ul In equation (8), spatial coefficients ^W'-^J are the same coefficients in equation (6) and defined by equation (7) .

In equation (8) , range coefficients v ^I,^J are defined as

y_ttJ = exp(~(x_u ~ x^ul)² / 2σ_R ² ) Eq. 9

The sharpening effect is controlled by a user-adjustable a, which is multiplied to the difference . To obtain a sharpening effect, a should be bigger than 1.

Noise control may be performed by the second term in equation ( 1 ) , in which yj^u _v2is the output of the bilateral filter 2 in the block-diagram in FIG. 15. The second term in equation ( 1 ) shows that y"_v'₂ is used as the baseline, and β controls how much detail and/ or noise are added back, β is between 0 and 1. When β is 0, all the difference is not added back; when β is 1 , the difference is fully added back.

Noise control is illustrated in FIG. 17. FIG. 17(a) illustrates the input I D signal. FIG. 17(b) illustrates y"_v ^l ₂ which due to edge-preserving property of the bilateral filter 2, >^₂ only contains the edge information with the noise and detail totally removed. FIG. 17 (c) is jc^w/ -j/₂ . x^{ul '} -y*₂ which contains the noise that y"_v'₂ does not contain. FIG. 17(d) is the result with noise suppressed by β(x^υl -yj^u _v2) + y_s ^u _v'₂ with /?=0.5.

The bilateral filter 2 , mathematically described in the following paragraphs, has a different spatial kernel from the bilateral filter 1. Again, the bilateral filter 2 has different coefficients in the two cases where the two different positions of the new pixels relative to the input pixels.

Position 1 (new pixel ul has the coordinates of (0, 0), FIG. 14A: The new pixel ul does not need to be generated, i.e., x^ul=xo,o.

The bilateral filter 2 is

Eq. 10 ώ=^rΣΣ(<>,A,)

where the normalization factor

^~ ZJ^U'.J'I.J

The spatial coefficients '•■> are derived from a 2-D Gaussian kernel centered at the pixel ul, with coordinates (0,0):

«5 = exp(-(z² /2σ₂ ² + f /2σ₂ ²)) Eq.11

The range coefficients '-^J are same as the bilateral filter 1, defined by equation (5). While the range kernel of the bilateral filter 2 is the same as the range kernel of the bilateral filter 1, the spatial kernel in equation (11) is different from the spatial kernel of the bilateral filter 1.

Position 2 (new pixel ul has the coordinates of (-1/4,1/4), FIG. 14B: The new pixel ul needs to be generated. The bilateral filter 2 is

u"' The spatial coefficients ¹>^J are derived from a 2 -D

Gaussian kernel centered at the pixel ul, with coordinates (- 1 / 4, 1 / 4) : w; = exp(-((z + l/4)² /2^² + (7-l/4)² /2^²))

The range coefficients '-^J are same as the bilateral filter 1 , defined by equation (9) . To further simplify the implementation, the bilateral filter 2 may be replaced by a linear filter with the same spatial coefficients as in the bilateral filter 2.

Overall, the "sharpening and noise control" module is conceptually illustrated by the simple I D signal in FIG. 18.

The previous description relates to the "sharpening and noise control" module for pixel ul. The modules for the other three pixels ur, dl, and dr have the same structure . The difference is the spatial kernels that define the spatial coefficients of the three filters (lowpass filter, bilateral filter 1 and 2) are centered at ur, dl, and dr, respectively. In summary, there are total four linear filters, four bilateral filters 1 and four bilateral filters 2 in the four modules. Their spatial coefficients are based on the Gaussian kernels with the same σi / σ2 but different centers. The range coefficients are based on the same Gaussian kernel, as a consequence the two kernels can share the same look-up table in an implementation.

In the above preferred embodiment of the edge sharpening and noise control module, the three filters take two kinds of inputs: low-resolution pixels in the local window and the co-located hi-resolution pixel. In the particular case shown in FIG. 15 , the new pixel x^ul as well as the pixels in the low-resolution 9x9 window are involved in the generation of y^ul . Those input pixels are treated as irregularly spaced samples on the high-resolution grid in the filtering, for both bilateral filters 1 and 2 and the low-pass filter. Based on the same design principle, many variations and extensions may^¬ be done .

By way of example, only low-resolution pixels may be used for filtering. In the special case that the high-resolution pixels are interpolated by a linear upscaler, low-resolution input pixels are sufficient for equivalent filtering. Multiple high-resolution pixels may be used for filtering. For example, neighboring non-co-located high-resolution pixels { x"^v , x^dl , x^dr), as well as x"' , may be used in the generation of y"¹ .

The terms and expressions which have been employed in the foregoing specification are used therein as terms of description and not of limitation, and there is no intention, in the use of such terms and expressions, of excluding equivalents of the features shown and described or portions thereof, it being recognized that the scope of the invention is defined and limited only by the claims which follow.

An example of a system and method can be seen in FIG. 19. An image at a first sharpness and at a first scale (resolution) is inputted and stored in buffer 700. Image data from the buffer 700 is then inputted into an image filter 702 , which filters to create a first channel and a second channel.

The image filter 702 may be a non-linear, edge adaptive filter. The first channel includes primarily texture information and the second channel includes primarily edge and flat region information. A first filter 704 receives the first channel and attenuates higher frequency content from the image . The second channel is sent to a second filter 706 to sharpen the image resulting in a second sharpness. Furthermore, the image data is sent from the buffer 700 to an upscaling section 708, where the image is upscaled from the first scale to a different second scale . The outputs from the upscaling section 708, the first filter 704 and the second filter 706 are then combined at a combining section 701 to result in a modified image having a different scale and sharpness. Clearly, the buffer 700 can be used to provide the same plurality of pixels to each of the image filter 702 , the first filter 704, the second filter 706 and the upscaling section 708. Additionally, there is no order to the upscaling and the filtering into different channels and it is not necessary to have one done before the other. Different pixel windows, having different sizes, to access the image data within the buffer 700 can be used to feed image data to the upscaling section 708 and the image filter 702. The size of the window used to feed the upscaling section 708 may be less than the size of the window used for at least one of the first filter 704 and the second filter 706.

Furthermore, the upscaling section 708 and the image filter 702 can be based on at least some of the same pixels in the buffer 700 or another buffer.

One possible configuration is to have the components shown in FIG. 13 placed into a similar arrangement as that of

FIG. 19 , with the buffer 700 corresponding to memory 600, the first and second filters 704, 706 corresponding to the noise control section and edge sharpening section of a respective edge sharpening and noise control module 604 , 606, 608, 610 and the upscaling section 708 corresponding to the upscaling module 602. With this arrangement, the first and second channels may be fed into a noise control section and an edge sharpening section of each module 604, 606, 608 , 610, respectively. The invention being thus described, it will be obvious that the same way may be varied in many ways. Such variations are not to be regarded as a departure from the spirit and scope of the invention, and all such modifications as would be obvious to one skilled in the art are intended to be included within the scope of the following claims.

Claims

1. A method for modif3άng an image from a first scale to a modified image having a second scale: (a) receiving said image having a first sharpness at said first scale;

(d) filtering said second channel to sharpen said image in said second channel;

(e) upscaling said image from said first scale to said second scale, wherein said upscaling, said filtering of said second channel, and said filtering of said first channel are based upon a plurality of the same pixels of said image from a same buffer used to store said image at said first sharpness and at said first scale;

(f) combining said first channel, said second channel, and said image upscaled to said second scale to form said modified image having said second scale.

2. The method of claim 1 wherein said image filter is a non-linear filter.

3. The method of claim 2 wherein said non-linear filter is an adaptive filter.

4. The method of claim 3 wherein said adaptive filter is edge adaptive.

5. The method of claim 1 wherein the size of a window used for said upscaling is less than the size of a window used for at least one of said filtering of said first channel and said filtering of said second channel.

6. The method of claim 5 wherein said upscaling and said image filter are based upon at least some of the same pixels in the buffer.

7. A method for modifying an image from a first scale to a modified image having a second scale:

(a) receiving said image having a first sharpness at said first scale;

(b) upscaling said image from said first scale to said second scale; (c) filtering said image with an image filter to result in the creation of a first channel and a second channel further based on said upscaled image and said image, wherein said first channel includes primarily texture information and said second channel includes primarily edge and flat region information;

(d) combining said first channel and said second channel to form said modified image having said second scale .

8. The method of claim 7 further comprising filtering said first channel to attenuate higher frequency content of said image in said first channel.

9. The method of claim 7 further comprising filtering said second channel to sharpen said image in said second channel.

10. The method of claim 7 wherein said upscaling, said filtering of said second channel, and said filtering of said first channel are based upon a plurality of the same pixels of said image from a same buffer used to store said image at said first sharpness and at said first scale.

1 1. The method of claim 7 wherein the size of a window used for said upscaling is less than the size of a window used for said filtering.

12. The method of claim 1 1 wherein said upscaling and said image filter are based upon the same pixels in a buffer used to store said image at said first sharpness and at said first scale.

13. The method of claim 9 further comprising said modified image having a second sharpness resulting from the filtering of said second channel, where said second sharpness is different from said first sharpness.

14. A device for modifying an image from a first scale to a modified image having a second scale, said device comprising: (a) a buffer for storing said image;

(b) an image filter for receiving said image having a first sharpness at said first scale and filtering said image to result in the creation of a first channel and a second channel, wherein said first channel includes primarily texture information and said second channel includes primarily edge and flat region information;

(c) a first filter to filter said first channel to attenuate higher frequency content of said image in said first channel;

(d) a second filter to filter said second channel to sharpen said image in said second channel; (e) an upscaling section to upscale said image from said first scale to said second scale, wherein said upscale section, said first filter and said second filter use a plurality of the same pixels of said image from said buffer; and