WO2012109528A1 - Edge-based video interpolation for video and image upsampling - Google Patents

Abstract

Edge-based interpolation for upsampling. One method may include determining an edge characteristic associated with an interpolation point, the edge characteristic having an edge magnitude and an edge angle; selecting an interpolation filter in response to the edge angle; and determining a pixel value at the interpolation point using the selected interpolation filter. Other embodiments include edge-based interpolation followed by an adaptive sharpening filter. The sharpening filter is controlled by the edge-based interpolation parameters that determine the pixels to be sharpened and the sharpening strength.

Description

EDGE-BASED VIDEO INTERPOLATION

FOR VIDEO AND IMAGE UPSAMPLING

CROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to (i) US Provisional Patent Application Serial No. 61/442,069 filed February 11, 2011 entitled "Edge-Based Video Interpolation For Video and Image Upsampling", and (ii) US Provisional Patent Application Serial No. 61/535,353 filed September 15, 2011, entitled "Combined Image Interpolation and Sampling", the contents of both of which are hereby incorporated by reference herein.

BACKGROUND

[0002] In image and video processing, upsampling or zoom refers to the process of increasing the resolution of a digitized image or video. Upsampling is universally found in most video players like VLC player, Windows Media player, etc. Online video hosting websites like YouTube, Hulu, Daily Motion, etc. provide users with the option to upsample the video to full screen resolution. Many current TVs and DVD players come equipped with an inbuilt upsampling module.

[0003] Upsampling generally involves generating new image pixels from existing pixels. In a video frame, a pixel is often coherent to its neighboring pixels. This property is used by most upsampling methods to generate new pixels. In a video or image transcoder pipeline, upsampling can be challenging because the input is a low resolution video having coding artifacts.

[0004] The simplest approach to upsampling is pixel replication, which simply involves replicating pixels in both rows and columns. However, this approach results in severe blockiness. The blockiness and aliasing artifacts seen in pixel replication can be mitigated by applying a low pass filter to the upsampled frame. Some techniques of reducing blockiness include applying a low pass filter. Other techniques generally referred to as interpolation may include averaging of neighboring pixels to determine the new pixel values. One technique of interpolation utilizes "zero stuffing" followed by low-pass filtering. This can be described as enlarging the dimensions of an image by placing zero-valued pixels at intermediate locations, and then applying a lowpass filter to the enlarged image.

[0005] Furthermore, videos commonly found on video hosting websites are sometimes heavily compressed to a low bitrate, and may have low resolution. A commonly noticed artifact in these videos is blurriness. An upsam ling process usually enhances the visibility of the blurriness in the video. Therefore, sharpening filters may be used to restore some of the details back to the video.

[0006] Traditionally, sharpening involves high pass filtering the blurred image and adding the weighted high pass image to the original image. This is referred to as unsharp masking. There are several implementations of high-pass filter, including Laplacian filter and Laplacian of Gaussian filter.

SUMMARY

[0007] One embodiment described herein is a method for interpolating an image comprising: determining an edge characteristic associated with an interpolation point, the edge characteristic having an edge magnitude and an edge angle; selecting an interpolation filter in response to the edge angle; and determining a pixel value at the interpolation point using the selected interpolation filter. The edge characteristic may be based on determining horizontal gradients and vertical gradients of pixel values in neighboring regions associated with the interpolation point. The neighboring regions associated with the interpolation point may be horizontal rectangular regions, vertical rectangular regions or square regions. In one embodiment, the neighboring regions are determined in response to the interpolation point being a row interpolation point, a column interpolation point or a center interpolation point. In some embodiments, the edge characteristic is determined using a first order gradient filter (which may be referred to as a "mask", since a convolution operation is not being

performed). One such first order gradient filter is a modified Sobel operator.

[0008] The method may also include selecting an interpolation filter in response to the edge characteristic. The selection of the interpolation filter may be made in response to the edge magnitude. That is, image points having edge magnitudes (or estimated

magnitudes) greater than a threshold may use one set of interpolation filters (e.g., implemented as hardware circuits or software running on a microprocessor, or a combination thereof), while those below the threshold may use another set of interpolation filters. In some embodiments, the threshold may also be adapted depending on characteristics of the image and/or the edge characteristics.

[0009] The method may utilize interpolation filters that apply greater weighting to the pixels located along the direction of the edge angle, and less or no weight to the pixels located along the direction orthogonal to the edge angle. In one embodiment, the interpolation filter applies greater weighting to nearest-neighbor pixels, an intermediate weighting to the pixels located along the direction of the edge angle, and the least or no weight to the pixels located along the direction orthogonal to the edge angle. In an embodiment, the nearest-neighbor pixels are either pixels in the same row or in the same column as the interpolation point.

[0010] The methods described herein may be implemented on a processor. As such, a computer readable medium made be used for storing instructions that when executed by the processor will cause the processor to: obtain a plurality of edge characteristics each associated with a respective one of a plurality of interpolation points, each of the edge characteristics having an edge magnitude and an edge angle; select an interpolation filter for each one of the plurality of interpolation points in response to the respective edge angle; determine a pixel value for each of the plurality of interpolation points using the

corresponding selected interpolation filter; and output the pixel values.

[0011] In some embodiments the method may be implemented in dedicated hardware or using hardware to accelerate a subset of the calculations. In one embodiment, the interpolation device comprises: an edge characteristic calculator configured to determine edge characteristics for each of a plurality of interpolation points; an interpolation filter selector configured to operate on the edge characteristics and to responsively generate interpolation filter identifiers for each of the plurality of interpolation points; and, an interpolation filter circuit configured to apply one of a plurality of interpolation filters in response to the interpolation filter identifiers and to output interpolated values for the plurality of interpolation points. One or more of these components may be implemented by a processor executing code stored in a computer readable storage medium.

[0012] In further embodiments, an edge-based interpolation for upsampling followed by an adaptive sharpening filter is described herein. In one embodiment, the sharpening filter is controlled by the edge-based interpolation parameters that determine the pixels to be sharpened and the sharpening strength. In one embodiment the method comprises:

determining gradient data for image pixels to be interpolated; determining interpolated pixel values using neighboring pixels according to the gradient data; identifying image pixels to be sharpened based on gradient data; and, sharpening image pixels using a filter strength determined in part based on the gradient data. [0013] In alternative embodiments, the edge-based interpolation filter operation is combined with an adaptive sharpening filter into a single filter called a joint filter. In one embodiment the method comprises: determining gradient data for image pixels to be interpolated; selectively sharpening neighboring original pixels; selectively identifying neighboring pixels that have yet to be interpolated in response to the pixel category of center row or column pixel; and, determining interpolated and sharpened pixel values using neighboring pixels according to a joint sharpening and interpolation filter.

BRIEF DESCRIPTION OF THE DRAWINGS

[0014] A more detailed understanding may be had from the following description, given by way of example in conjunction with the accompanying drawings wherein:

[0015] FIG. 1 is a graphical depiction of pixels in an image;

[0016] FIGS. 2A-2C are graphical depictions of center, column, and row interpolation points, respectively;

[0017] FIG. 3 is a Frame level PSNR comparison between edge-based interpolation and five tap filter approach for vid01_2X sequence; and

[0018] FIG. 4 A is a block diagram of an edge-based interpolation and adaptive sharpening system using two separate filters;

[0019] FIG. 4B is a more detailed block diagram of an edge-based interpolation and adaptive sharpening system using two separate filters;

[0020] FIG. 5 is a flow chart of the interpolation process;

[0021] FIG. 6 is a flow chart of the sharpening process;

[0022] FIG. 7 is a block diagram of joint edge-based interpolation and adaptive sharpening filter.

[0023] FIG. 8 is a pixel diagram of flagged pixels;

[0024] FIG. 9 is a pixel diagram of center, row, and column pixels;

[0025] FIG. 10 illustrates the span of pixels over which the filter coefficients are applied;

[0026] FIG. 11 is a flow chart for one embodiment of a combined filtering method;

[0027] FIG. 12 is a pixel map showing the assignment of sharpening parameters for neighboring pixels of pixels that are (a) center, (b) row, and (c) column pixels; [0028] FIGS. 13-16 are pixel diagrams identifying original pixels used to derive the interpolated pixels in the bounding box;

[0029] FIG. 17 identifies original pixels and estimated interpolated pixels for sharpening;

[0030] FIG. 18A is a system diagram of an example communications system in which one or more disclosed embodiments may be implemented;

[0031] FIG. 18B is a system diagram of an example wireless transmit/receive unit (WTRU) that may be used within the communications system illustrated in FIG. 18A; and,

[0032] FIGS. 18C-18E are system diagrams of example radio access network and example core networks that may be used within the communications system illustrated in FIG. 18A.

DETAILED DESCRIPTION

[0033] The interpolation methods and devices described herein may be used in wired or wireless networks. Devices including handheld devices, desktop, laptop or other computers may be used to perform the methods. This includes cell phones, PDAs, tablet computers and/or displays, as well as cable TV set top boxes, televisions, and the like.

[0034] The interpolation, or a video upsampling scheme, described herein retains fidelity of edges and a computational complexity less than typical FIR filtering techniques. The system may implement any upsampling factor, N and M, applied to width and height of the video, respectively. Described herein for simplicity is upsampling an image or video frame by a factor of two in both dimensions (N=2 and M=2).

[0035] FIG. 1 illustrates a pixel grid containing both original pixels (designated by squares) and estimated pixels, or pixel values to be estimated at interpolation points

(designated by circles). Pixel A has diagonal original pixels; pixel B has two neighboring original pixels along the same column; and pixel C has two neighboring original pixels along the same row. Thus, the estimated pixels can be categorized into three groups: (a) pixels having diagonal neighboring original pixels (labeled as 'A' in FIG. 1), and the estimated pixels at these interpolation points may be referred to as center pixels; (b) pixels having neighboring original column pixels (labeled as 'B' in FIG. 1) as the nearest-neighbor pixels. The estimated pixels may be referred to as column pixels; and (c) pixels having neighboring original row pixels (labeled as 'C in FIG. 1) as the nearest-neighbor pixels. The estimated pixels may be referred to as row pixels.

[0036] In one embodiment, the method includes the following aspects: edge detection; edge angle determination; and pixel estimation. In another embodiment, the method may include determining an edge characteristic associated with an interpolation point, where the edge characteristic includes an edge magnitude and an edge angle; selecting an interpolation filter in response to the edge angle; and determining a pixel value at the interpolation point using the selected interpolation filter.

[0037] Any software or hardware (collectively referred to as an operator) that computes the horizontal and vertical gradients of pixels may be used for edge detection. In one embodiment a modified Sobel operator is used due to its low computational complexity. A Sobel operator includes two square masks that compute the horizontal and vertical gradients. These gradients may then be used to compute or otherwise obtain the angle of the gradient and the angle of the edge, or estimates thereof. The standard Sobel operator is applied to a square grid of pixels, while the modified Sobel operator may be applied to a rectangular grid of pixels.

[0038] When determining an edge characteristic, the estimated pixel category, or interpolation point categories, may be used to select a different modified Sobel mask G_x and G_y. As illustrated in FIGS. 2A-2C, G_x and G_y are masks that may be used to pointwise multiply with the image pixels and then sum the products to computing horizontal (Δχ) and vertical (Ay) gradients (or estimates thereof), respectively.

[0039] In one embodiment, the magnitude G of the gradient (or of the edge) is determined by calculating G = |Δχ| + |Ay|. If G is greater than a threshold T_ecj_ge, then an edge may be present. In the event that G exceeds the threshold, then the angle of the edge gradient may be determined as described below. The value of the threshold T_edge may depend on the resolution of the video, the pixel category, and other factors. Threshold values may be determined empirically, and some that have been found to perform well are set forth below in Table 1. The choice of threshold values can differ: in some embodiments a constant threshold may be used, while in other embodiments the threshold may be adapted on a frame- by-frame or on a block of pixel basis. An adaptive threshold can be computed automatically based on the pixel characteristics within a block. At lower resolution, neighboring pixels are less likely to be coherent, resulting in larger gradient. Use of smaller threshold would result in many pixels being classified as edges. Therefore, to reduce incorrect classification, larger thresholds may be used for smaller resolution videos.

Table 1. List of thresholds corresponding to different frame sizes and pixel categories.

[0040] In various embodiments, the interpolation filter may be selected in response to the edge (or gradient) magnitude, edge (or gradient) angle, or both. In one embodiment, one interpolation filter may be selected for interpolation points where the corresponding edge magnitude G is less than T_ed_ge. In other embodiments, one interpolation filter may be selected for interpolation points where the corresponding edge has either Δχ or Ay equal to zero. In each of the scenarios, an edge angle need not be determined or provided, and the

interpolation filter used to interpolate the new pixel at the interpolation point is as follows: a. For center pixel: (aO + al + bO + bl)/4

b. For column pixel: (al+bl)/2

c. For row pixel: (b0+bl)/2

where aO, al, bO, and bl correspond to pixels labeled in FIGS 2A-2C. Note that Ay=0 and Δχ=0 correspond to the gradient angles 0 and 90 degrees, respectively.

[0041] The gradient angle μ may be calculated as μ = tan^_1(Ay/Ax). Since the edge angle is perpendicular to the gradient angle, the edge angle is Θ = μ + 90°. Note that because the edge and the edge gradient are related by a simple 90 degree rotation, either may be used interchangeably if this relationship is accounted for. Note also that the range of the edge (or gradient) angles may be reduced to the range of 0-180 degrees.

[0042] In some embodiments the interpolation filter to use at respective interpolation points is determined in response to the pixel category and edge angle. In one embodiment, the interpolation assumes that the edges are linear. For curved edges, additional angles may be checked during pixel estimation. Alternatively, edge pixels belonging to a curve may be detected using a Hough transform-based method, and these pixels can be used for interpolating new pixels along a curved edge.

[0043] The pixel estimation process for interpolation points where the edge magnitude exceeds a threshold will now be described. In one embodiment, for center pixels, an interpolation filter is selected based on whether the edge angle is approximately 45 degrees. One embodiment uses one center-interpolation filter for edge angles in the range of 45 to 135 degrees. Alternative embodiments may utilize other ranges of angles.

[0044] For center pixel estimation, the interpolation may be performed as follows:

If Θ is between 35 and 55 degrees: new pixel = (al+b0)/2.

If Θ is between 125 and 145 degrees: new pixel = (a0+bl)/2.

Else new pixel = (a0+al+b0+bl)/4

[0045] For column pixel estimation,

If Θ is between 0 and 90 degrees: new pixel = (bO + 2*(al+bl) + a2)/6

Else new pixel = (aO + 2 * (a 1 +b 1 ) + b2)/6

[0046] For row pixel estimation

If Θ is between 0 and 90 degrees: new pixel = (cO + 2 *(b0+bl) + al)/6

Else new pixel = (aO + 2 *(b0+bl) + cl)/6

[0047] The method may therefore utilize interpolation filters that apply greater weighting to the pixels located along the direction of the edge angle, and less or no weight to the pixels located along the direction orthogonal to the edge angle. In one embodiment, the interpolation filter applies greater weighting to nearest-neighbor pixels, an intermediate weighting to the pixels located along the direction of the edge angle, and the least or no weight to the pixels located along the direction orthogonal to the edge angle. In an embodiment, the nearest-neighbor pixels are either pixels in the same row or in the same column as the interpolation point.

[0048] In one embodiment, the gradient measurements Δχ and Ay may be used in conjunction with a look up table, or LUT, to determine an appropriate interpolation filter. The LUT may store a desired filter impulse response, or may simply provide an interpolation filter identifier that may be used to determine and apply the appropriate interpolation filter.

[0049] One embodiment described herein is a method for interpolating an image comprising: determining an edge characteristic associated with an interpolation point, the edge characteristic having an edge magnitude and an edge angle; selecting an interpolation filter in response to the edge angle; and determining a pixel value at the interpolation point using the selected interpolation filter. The edge characteristic may be based on determining horizontal gradients and vertical gradients of pixel values in neighboring regions associated with the interpolation point. The neighboring regions associated with the interpolation point may be horizontal rectangular regions as shown in FIG. 2B, vertical rectangular regions as shown in FIG. 2C or square regions as shown in FIG. 2A. In one embodiment, the neighboring regions are determined in response to the interpolation point being a row interpolation point, a column interpolation point or a center interpolation point. In some embodiments, the edge characteristic is determined using a first order gradient filter (which may be referred to as a "mask", when a convolution operation is not being performed). One such first order gradient filter is a modified Sobel operator.

[0050] Described below is a comparison of the performance of edge-based interpolation with the five -tap filter technique using videos listed in Table 2. (In the comparison, only a five-tap filter is used without a sharpening filter.) Both the methods are tested by applying them to the a transcoded video. The quality of the upsampled video is measured using peak-signal-to-noise ratio (PSNR) of the luma component, the video quality metric (VQM) score, and subjective comparisons. The input video of the transcoder is used as the reference for our quality measurements.

Table 2. List of test videos. Bitrates of the original (or pre-transcoded) videos are listed

[0051] VQM scores close to zero indicate no artifacts/impairments, while scores close to one are heavily impaired. VQM only takes 15 seconds of videos for comparison. The upsampling schemes are tested on different videos and transcoder bitrates and list the results in Table 3. The term '2X' and '3X' in Table 3 refers to the transcoded video being encoded at half and one third the original bitrate, respectively. For the 'vid' videos, both upsampling methods yield low impairments, with the edge-based interpolation performing slightly better than the 5-tap filter. It should be noted that when looking at the two processed videos, the edge-based interpolation seems to appear significantly better. This is expected as most objective measurement schemes have limitations. The 3X videos often results in a slightly higher VQM score compared to the 2X videos, since lowering the bitrate increases coding distortions.

[0052] For the Book of Eli video, the edge-based interpolation has small perceptual improvement over the 5-tap method. The VQM score for Book of Eli is greater than that for 'vid' videos, since the downsampled video was encoded at 50 times lower bitrate compared to the original video.

[0053] For the 'vid' videos the PSNR is computed for a segment of videos that do not have scene change and report the results in Table 4. For the Book of Eli video, all frames are considered whose PSNR < 50 dB, since all frames with higher PSNR are likely to be blank frames. For vidl5 and vidl6, a maximum PSNR improvement of 0.7 dB over the 5-tap filter is obtained. FIG. 3 illustrates a snapshot of PSNR vs. frame for vid01_2X sequence. It is clear that even on a frame-by-frame basis there is PSNR improvement when using the edge- based interpolation approach.

Table 3. VQM scores. 2X and 3X indicate that the videos have been encoded at half and one third the original bitrate. Filename Approach Frames PSNR-Y (dB) APSNR (dB)

Edge-based interp 1400:2400 36.86 vidOl 2X 5 -tap filter 1400:2400 36.36 0.5

Edge-based interp 2100:2700 33.79 vidl5 2X 5 -tap filter 2100:2700 33.09 0.7

Edge-based interp 800:2000 39.45 vidl6 2X 5 -tap filter 800:2000 38.75 0.7

Edge-based interp 1400:2400 36.2428

vidOl 3X 5 -tap filter 1400:2400 35.8958 0.347

Edge-based interp 2100:2700 33.0581

vidl5 3X 5 -tap filter 2100:2700 32.56 0.4981

Edge-based interp 800:2000 38.93 vidl6 3X 5 -tap filter 800:2000 38.5773 0.3527

Book of Eli 200 Edge-based interp 37.53

kbps 5 -tap filter 37.48 0.05

Table 4. PSNR results. For Book of Eli we use all frames whose PSNR < 50 dB.

[0054] In addition to VQM scores, the two upsampling methods are compared by looking at the upsampled videos. Experimental results obtained by subjecting video frames to the two upsampling schemes indicate that the edge-based interpolation method has sharper edges and more details compared to the five-tap filter approach.

[0055] Complexity comparisons are made by running the upsampling algorithms on 1000 frames of each test sequence on a 2.53 GHz PC, and measure the average encoding time per frame. The relative speed improvement is computed by taking the ratio of the processing time for the 5 -tap filter and the edge-based interpolation filter. The results are reported in

Table 5. We find that the edge-based interpolation is up to 2.5 times faster than the 5 -tap filter.

Table 5. Speed performance results. [0056] Also described herein are apparatuses and methods for interpolation and sharpening a video signal. In one embodiment, the method is performed by a two-step filter approach, where an interpolation filter upsamples the input video to a higher spatial resolution, followed by an sharpening filter that enhances the details in the video. In another embodiment, the process is performed by a joint filter approach where one filter is applied to the input video signal to upsample it to a higher resolution and to enhance the signal details simultaneously. The joint filter performs both interpolation and sharpening together, and may have associated processing gains. In some embodiments the memory access may also be reduced.

[0057] As shown in FIG. 4A, two separate filters - an edge-based interpolation filter and an adaptive sharpening filter are combined to generate upsampled and sharpened video frames from a raw video frame source. In FIG. 4B, one embodiment is depicted where a YUV frame is first upsampled by two in both dimensions using an edge-based interpolator. The edge based interpolator described above may be used. Edge information captured during the interpolation of the luminance component (Y) of the frame may be used for determining pixels to be sharpened, and for controlling the strength of the adaptive sharpening filter. As summarized in FIG. 9, the edge information may be obtained from edge detection filters selected in part on the location of the interpolated pixel as either a center pixel A, a row pixel B or a column pixel C. Sharpening may be performed on the luma component because it usually contains more details or edges. The sharpened luma component is then combined with the chroma components to result in an upsampled sharpened frame. Edge information may also be extracted from one or more of the chroma components, and a weighted combination of the edge information from luma and from one or more of the chroma components may be used. Further, one or more of the chroma components may also go through adaptive sharpening.

[0058] Block diagrams in FIG. 5 and FIG. 6 illustrate one algorithm for edge-based interpolation and sharpening stages, respectively. Two buffers - map[N][M] and

strength[N][M] - store sharpness map and edge strength, respectively, where N and M are the width and height of the upsampled frame. Both the buffers are initialized to zero before interpolating a frame. An entry of Ί ' in the map[i][j] indicates that a pixel located at (i,j) is to be sharpened and '0' otherwise. For each pixel (i, j) to be interpolated, a gradient (grad ;) is computed using the modified Sobel operator. If gradi_j > sharp thresh, where sharp thres is a sharpness threshold, then map[i][j] element for the interpolated pixel and its neighboring pixels is set to one (1). Similarly, strength[i][j] for the interpolated pixel and its neighboring pixels are set to gradjj. For example, if the interpolated pixel at location (i, j) has a gradient gradjj≥ sharp_thresh, then:

map[/c] [p] = 1; i - 1≤ k≤ i + l,j - 1≤ p≤j ^' + 1, and (1) strength[/c] [p] = gradi_j,- i— 1≤ k≤ i + l,j— 1≤ p≤ j + 1.

[0059] This is illustrated in FIG. 8 where symbol 'x' represents an original pixel and Ό' indicates the interpolated pixel. The black dot is an interpolated edge pixel which is flagged for sharpening, and the other pixels within the bounding box are also flagged for sharpening due to their proximity to an edge pixel. In one embodiment a sharpness threshold of 100 is suitable, but the choice of threshold can differ. In other embodiments, sharpness threshold may also be determined from the local image characteristics and adapted to different values throughout the image.

[0060] After obtaining the interpolated frame together with strength[i][j] and map[i][j], the Laplacian of Gaussian (LoG) filter is applied to pixels at (i,j) whose map[i][j]=l . Sharpening is applied to the neighboring pixels of an edge pixel to remove artifacts that might appear as speckles due to non-uniform variation in luminance. In the sharpening stage, the map[i][j] and strength[i][j] information from the interpolation stage is reused. This may lead to lower computational complexity.

[0061] The definition of LoG filter in continuous form is given by

where x

Gaussian. The discrete LoG filter of size KxL is defined as

\ (K-l) / 2 (L-\) I 2

LoG_c ;KxL [x][y] = h(x,y,a) -— ∑

χ=-(Κ-\) Ι 2 y=-(L-\) / 2

(x² + y² - 2a²)h_g (x,y,a)

where h(x,y,a) = (K-l) / 2 (L-l) / 2 , and :=-(ί:-1) / 2 y=-(L-l) / 2

(3)

[0062] The parameter σ determines the filtering strength. Larger σ (> 1) may be used for weak sharpening, while smaller σ (< 1) may used for stronger sharpening. The sharpening filters may further be of various sizes win size. For example, for input videos with small resolution( < 640x360), a win size = 5x5 LoG filter may be chosen; while for higher resolution videos (> 640x360), a win size = 9x9 LoG filter may be chosen. The filter size determines the number of neighboring pixels considered during sharpening. In other embodiments, different filter sizes can be chosen based on both video resolution and content. Further, filter sizes cmay be adapted throughout an image frame. As an example, the 5x5 LoG filter operation may be expressed as:

C_hU][j]

∑C[i + k][j + n] >G_aiii5x5[k][n] (4) where C[i][j] and C_h[i] [j] are the upsampled pixel at (i,j) and its corresponding high frequency component, respectively. FIG. 10 illustrates the span of pixels over which the filter coefficients are applied, where the black dot indicates the interpolated pixel (called the center pixel) that is to be sharpened using a 5x5 LoG filter. The LoG filter coefficients are applied to the pixels within the bounding box.

[0063] In one embodiment, σ¾ is selected based on the edge strength as follows:

a_l, p_l < strength[i][j] < _yo₂

a₂ , p₂ < strength[i][j] < p₃ a_s , strength[i][j] > _yo_s

where p_l ..,ps are edge strength thresholds, and a_ls ...,as are S different ay values associated with different levels of sharpening. In one embodiment, σ¾ is defined as ∞(no sharpening), strength[i][j] < 100

1, 100≤ strength[i][j] < 200

σ* ^~ 1.2, 200≤ strength[i] j] < 300 · (5)

1.4, strength[i] j] > 300

[0064] In addition to the edge gradient, local image characteristics may also be used for determining σ¾. For example, over-sharpening of strong edges (that is, use of small σ¾ values on strong edges) may result in ringing artifact. To avoid this, larger values for σ¾ may be used as edge strengths strength[i][j] increases.

[0065] Finally, the sharpened pixel is C_oui [/][ ] derived as

o_u =

_{. (6)}

[0066] One embodiment uses λ=1, but in other embodiments different λ values can be used. In other embodiments, the sharpening filter can also be made sensitive to noise, by detecting noise before the interpolation stage, and using this information to choose a LoG filter with larger σ (σ > 1). This will ensure that noise detected as edges are not amplified. Similarly, in other embodiments, the sharpening process can be made sensitive to false edges, such as blocking artifacts. A blocking artifact detection algorithm can be used prior to interpolation. During the sharpening process, pixels classified as false edges can be smoothened instead of being sharpened.

[0067] FIG. 7 depicts a joint edge-based interpolation and adaptive sharpening filter embodiment. In the previous two-step filter approach, interpolated pixels are computed within a filter window before the sharpening filter is applied. In the joint filter approach, the dependency between interpolation and sharpening is reduced. In this embodiment, the edge- based interpolation is combined with sharpening filter to provide a joint filter. A flow chart for the combined filtering approach is given in FIG. 11. The gradient grad_y is computed for pixel located at (i,j) using a modified Sobel operator. Two buffers are used - buffers sharp flag and future grad - to flag the sharpening of neighboring pixels located at (k,n) that have not been interpolated yet. If the gradient grad_y is greater than or equal to a predefined sharpness threshold sharp hresh, then the following two operations are performed on the neighboring 3x3 pixels, as illustrated in FIG. 12:

1. Sharpen the neighboring original pixels using a LoG filter whose σ is selected based on gradi_j using Equation (5). In FIG. 12, these pixels are indicated by 'x'. 2. Set the sharp flag and future grad of the neighboring pixels that have not yet been interpolated as follows:

If pixel (i,j) is a center pixel, then set

sharp_flag[i + 1 ] [j ] = sharp_flag[i] [j + 1 ] = 1 , and

future_grad[i + 1 ] [j ] = future_grad[i] [j + 1 ] = grad_y .

If pixel (i,j) is a row pixel, then set

sharp flagTi + 1 ] [j - 1 ] = sharp flagTi + 1 ] [j] = sharp[i+ 1 ] [j + 1 ] = 1 , and

future_grad[i + 1 ] [j - 1 ] = future_grad[i + 1 ] [j j ] = future_grad[i + 1 ] [j j + 1 ] = grady .

If pixel (i,j) is a column pixel, then set

sharp_flag[i + 1 ] [j - 1 ] = sharp_flag[i + 1 ] [j + 1 ] = 1 , and

future_grad[i + 1 ] [j - 1 ] = future_grad[i + 1 ] [j + 1 ] = grad_y .

[0068] In one embodiment, sharp thresh = 100, and pixel sharpening candidates are the neighboring 3x3 pixels. In other embodiments, a larger neighborhood is considered, and a different sharp thresh may be used; for example, the value of sharp thres may be determined based on local image characteristics. If grad_y is less than sharpjhresh, then check if the associated sharp_flag[i][j] is equal to 1. If sharp_flag[i][j] is 1 , then use its associated future_grad[i][j] value to choose a suitable joint filter, otherwise continue to use grad_y for selecting the joint filter. By using future_grad[i][j], the systems uses the gradient of a previously interpolated neighboring pixel at (i-a,j-b) whose gradi__aj_b > sharp thresh, and 0< a,b < 1.

[0069] As depicted in FIG. 12, the black dot is a pixel being interpolated and sharpened, and has a gradient greater than the sharpening threshold. The 'x' are original pixels to be sharpened, and gray dots are pixels yet to be interpolated and are flagged for sharpening. Three possible cases exist: (a) center, (b) row, and (c) column pixel interpolation and sharpening.

[0070] Following are two embodiments of the joint filter. In other embodiments, the joint filter operates partially on pixels that have already been interpolated in addition to the original pixels.

[0071] In the two-step filter approach described above, pixels within win size are used for sharpening as illustrated by the box in FIG. 13. In this embodiment, all original pixels that were used to derive the interpolated pixels in the bounding box are identified, as illustrated by the bold X's in FIG. 13. The sharpening filter operates these original pixels. Because these pixels were used in deriving the interpolated pixels in the bounding box, the high frequency component computed from these original pixels would be close to the high frequency component computed from all pixels in the bounding box.

[0072] The joint filter can be written as the sum of the two filters given by

h(grad ,edge angle, win _ size, category) = h_EI (grad ,edge angle, filter _ size(win_size),category) + λ■ LoG ( (grad), filter _ size(win _ size), category ),

(?)

[0073] where LoG is the sharpening filter applied only to the original pixels, and h_Ei is the edge-based interpolator. In this embodiment, λ=1, but in other embodiments different λ values can be chosen. The LoG parameter σ is a function of the edge strength, such as the one defined in Equation (5). For one embodiment, where the input video resolutions is less than 640 x 360, set win size = 5x5, else set it to 9x9.

[0074] The edge-based interpolator hEI is a function of the gradient, edge angle, filter size and pixel category. As described above with respect to FIG. 9, for edge-based

interpolation, interpolated pixels are categorized as center, row, and column pixels, based on their position in the pixel grid. Therefore, including the original pixels, the upsampled frame has four pixel categories. Each interpolated pixel category (e.g., center, row, or column) has three filters, each of which corresponds to one of the three edge directions. For the row and column pixels, the three edge directions are no edge, edge angles between 0 and 90 degrees, and edge angles between 90 and 180 degrees. For center pixels, the three edge directions are no edge, edge angles between 35 and 55 degrees, and edge angles between 125 and 145 degrees. During sharpening, consider original pixels that were used to derive the interpolated pixels within win size. For example, for sharpening the center pixel in FIG. 13, filtering is performed on the 4x4 original pixels. Therefore, a 4x4 LoG filter is considered, resulting in a 4x4 joint filter in Equation (7). The size of the joint filter is denoted as filter size(win size), and therefore for FIG. 13, filter size(win size) = filter_size(5x5) = 4x4. Similarly, the filter_size(5x5) for row, column, and original pixels are 5x4, 4x5, and 5x5, respectively, as illustrated in FIGS. 14, 15, and 16.

[0075] In FIG. 13, interpolation and sharpening of the center pixel indicated by the black dot is shown. The interpolated pixels within win size are derived from the original pixels represented as bold 'x'. A LoG filter of 4x4 is applied to these original pixels. In FIG. 14, interpolation and sharpening of row pixel indicated by the black dot is shown. A LoG filter of 5x4 is applied to the original pixels indicated by bold 'x'. In FIG. 15, interpolation and sharpening of column pixel indicated by the black dot is shown. A LoG filter of 4x5 is applied to the original pixels indicated by bold 'x'. In FIG. 16, sharpening of original pixel labeled 'A' is depicted. A LoG filter of 5x5 is applied to the original pixels indicated by bold

[0076] Following is one embodiment of the joint filter for center pixels for small screen video resolution (e.g. 480x204). The procedure remains similar for row, column, and original pixels. Small modifications may be made to arrive at joint filter design for larger video resolutions (e.g., 1080p, 720p). For small screen video resolution where for example win_size = 5x5 is used, the resulting joint filter size filter_size(win_size) = 4x4.

[0077] Following are the interpolation filters used to generate the center pixels based on the edge direction:

filter 1 ; edge in first direction, and gradient > edge threshold r (gradient, edge angle, filter_size(win_size)) ^: filter 2; edge in second direction, and gradient > edge threshold filter 3; else, where the edge threshold determines whether an interpolated pixel is an edge pixel or not.

[0078] An embodiment of center pixel interpolation filters is given below.

0 0 0 0

0 0 0.5 0

< edge angle < 55° and gradient > edge threshold

0 0.5 0 0

0 0 0 0

0 0 0 0

0 0.5 0 0

r (gradient, edge angle, 4 x 4) = , 125° < edge angle < 145° and gradient > edge threshold

0 0 0.5 0

0 0 0 0

[0079] In one embodiment, the edge threshold > sharp thresh. The LoG filter of filter_size(win_size) = KxL, with standard deviation σ is defined in Equation (3).

[0080] The LoG filters for the four σ values as in Equation (5) with αι=∞, a₂=l, a₃=1.2, and a₄=1.4 are as follows: 0 0 0 0

0 0 0 0

LoG,

0 0 0 0

0 0 0 0

0.0717 0.0511 0.0511 0.0717^"

0.0511 - 0.1739 - 0.1739 0.0511

LoG

0.0511 - 0.1739 - 0.1739 0.0511

0.0717 0.0511 0.0511 0.0717

^"0.0532 0.0214 0.0214 0.0532;

0.0214 - 0.0961 - 0.0961 0.0214;

LoG

0.0214 - 0.0961 - 0.0961 0.0214;

0.0532 0.0214 0.0214 0.0532;

^"0.0365 0.0096 0.0096 0.0365 ^"

0.0096 - 0.0557 - 0.0557 0.0096

LoG

0.0096 (9)

- 0.0557 - 0.0557 0.0096

0.0365 0.0096 0.0096 0.0365

[0081] The joint filter for the center pixels is then given by

{gradient, edge angle, win _size) = h_{cmter 1} {gradient, edge angle, filter _size{win _ size))

-- LoG_{1 4x4}+h_{E r} {gradient, edge angle,A x 4)

0.0717 0.0511 0.0511 0.0717

0.0511 - 0.1739 0.3261 0.0511

, 35° < edge angle< 55° and gradients edge threshold

0.0511 0.3261 - 0.1739 0.0511

0.0717 0.0511 0.0511 0.0717

0.0717 0.0511 0.0511 0.0717^"

0.0511 0.3261 - 0.1739 0.0511

, 125° < edge angle< 145° and gradient > edge threshold

0.0511 - 0.1739 0.3261 0.0511

0.0717 0.0511 0.0511 0.0717

[0082] Similarly,

₂ {gradient, edge angle, win _ size) = LoG {gradient, edge angle,4 x 4)

{gradient, edge angle, win _ size) = LoG_{l i} {gradient, edge angle,4 x 4)

h_center∞{gradient,edge angle, win _ size) = LoG, _r {gradient, edge angle,4 x 4) = h_E , {gradient, edge angle

[0083] The filter h_center,_∞ is used when only interpolation is applied and sharpening is not. The joint filters for all other pixel categories may be derived in a similar manner. In one embodiment, four different σ values as listed in Equation (5), and two different win size values are used. Therefore, the number of joint filters for interpolated pixels is:

N_j0m- _t__fllters__m- _terp__pixels = Number of pixel categories x Number of interpolation directions x Number of σ x Number of win_size

= 3 x3x 4x 2

= 72.

[0084] The original pixels need sharpening only, and therefore during its filter design edge-based interpolation is not included. The number of joint filters for original pixels is:

N_{jo irA}_ _filters __{orig ≠xeh} = Number of σ x Number of win_size

= 4 x 2 = 8.

[0085] The total number of filters required is

^"^Total ^ jo mt_ filters _mt erp _ pixels ^ jo int_ filters _orig _ pixels ^ 80. ^13)

[0086] These filters may be stored in look-up-tables, and may be chosen based on pixel category, gradient, edge angle, and win_size.

[0087] In an alternative embodiment, the joint filter is the same as Equation (7), but it uses a different sharpening component. One example of the alternative joint filter is described for center pixels for small screen video resolution (e.g, 480x204). The same design approach extends to row, column, and original pixels as well. For this example, win size = 5x5. Original pixels and estimated interpolated pixels for sharpening are as illustrated in FIG. 17, where all pixels within win size are used for sharpening, with the interpolated pixels (gray dots) estimated from the original pixels (indicated by bold 'x').

[0088] In one embodiment, the interpolated pixels (gray dots in FIG. 17) are estimated using edge-based interpolation for the case when gradients are smaller than edge threshold. This corresponds to the following estimation filters for center, row, and column pixels: 0.25 0.25

h. center est

0.25 0.25

0 0

h row est 0.5 0.5

0 0

0 0.5 0

h. col est

0 0.5 0

(14)

[0089] FIG. 17 can be represented in a table form in Table 6 below, where Q_j are pixels. The array in gray represent pixels located in win size = 5x5, and the entries in bold represent the original pixels. Therefore, non-bold pixels in the gray region are the estimated interpolated pixels.

Ta e

[0090] The 5x5 sharpening filter is represented below.

[0091] The sharpening operation is:

[0092] The sharpening filter g that operates only on original pixels, and yet provides the same high frequency component Ch in Equation (16) may be used according to:

3 3 i=0j=0 _ _{( 17)}

[0093] To illustrate the design of g, the coefficients goo and goi are derived. Each

product term in Equation (16) is expressed in terms of the original pixels as follows.

Q _!/¾o = 0.25 * (C₀₀ + C₀₂ + C₂₀ + C₂₂) * /¾o = 0.25/¾₀C₀₀ + 0.25/_¾0C₀₂ + 0.25/¾₀C₂₀ + 0.25/¾₀C₂₂

C₁₂½ = 0.5 * (Co2 0.5A₀₁C₀₂ + 0.5/¾ ₂

(18)

<¾½ = 0.25 Q₂ + C₀₄ + C₂₂ + C₂₄) *ho₂ = 0.25/¾₂C₀₂ + 0.25 /¾₂C₂₂ + 0.25½C₀₄ + 0.25/¾₂C

[0094] From Equation (17), the sum of coefficients corresponding to Coo is goo, and sum of coefficients corresponding to C₀₂ is goi- Combining the terms in Equation (18)

provides:

goo = 0.25/!₀₀

g₀₁ = 0.25*00 + 0.5*0! + 0.25*02 ^'

[0095] The filter coefficients of g_center for the center pixels are given by: center(o(grad), filter _size(win_size)) = gcemer(o(grad),4x4) =

0.25*(hoo+h₀₂) + 0.5*hoi 0.25*(ho2+ho4)+0.5*h₀₃ 0.25*h₀₄

0.25*(hoo+ho2+h₂₀+h₂₂)+ 0.25*(h₀₂+h₀₄+h₂₂+h₂₄)+ 0.25*(h₀4+h₂₄)+

0.5*(hoi+hio+hi2+h₂i)+hi i 0.5*(h₃₀+h₁₂+h₁₄+h₂₃)+h₁₃ 0.5*h₁₄ 0.25*(h₂₀+h₂₂+h₄₀+h₄₂)+ 0.25*(h₂₂+ h₂₄+h₄₂+h₄₄)+ 0.25*(h₂₄+h₄₄)+

0.5*(h₂i+h₃₀+h ₂+h₄i)+h₃i 0.5*(h₂₃+h₃₂+h ₄+h₄₃)+h₃₃ 0.5*h₃₄ 0.25*h4o+0.5*h4i+0.25*h42 0.25*h₄₂+0.5*h₄ +0.25*h₄₄ 0.25*h₄₄

[0096] Filter coefficients of g_row ,g_coiumn , and g_orig, which are used for the

column and original pixels, respectively, can be derived in a similar manner.

[0097] The joint filter design 2 is defined as: h(grad, edge angle, win_size, category) = h_EI (grad, edge angle,filter _ size(win _ size), category) +

g(a(grad), filter _ size(win _ size), category),

where g(.) is the sharpening component. Continuing, the joint filter for the center pixels is: hcenter(grad, edge angle, 5x5) = hEI_center(grad, edge angle, 4x4)+ gcenter(a(grad), 4x4), where hEI center is as defined in Equation (8).

[0098] The total number of filters used in the two embodiments (design 1 and design 2) of joint filters is the same, as given in Equation (13). The filter sizes of both methods are listed in Table 7. Design 2 uses smaller filters resulting in lower complexity. Design 2 may also yield better subjective quality than design 1 and it has similar quality as the two-step filter approach. This is expected because design 2 uses estimated interpolated pixels during sharpening, and hence is a close approximation of the two-step filter approach

mathematically. It has been observed that sometimes design 1 yields better subjective quality than design 2. Therefore, an adaptive approach can be used to switch between the two joint filter designs; for example, the adaptive model may be based on local image characteristics.

Table 7. Size of the joint filters for win_size=5x5.

[0099] The interpolation and sharpening filters described herein may be incorporated into any of a wide variety of terminals, such as, without limitation, digital televisions, wireless communication devices, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, tablet computers, digital cameras, digital recording devices, video gaming devices, video game consoles, cellular or satellite radio telephones, digital media players, and the like.

[00100] FIG. 18A is a diagram of an example communications system 100 in which one or more disclosed embodiments may be implemented. The communications system 100 may be a multiple access system that provides content, such as voice, data, video, messaging, broadcast, etc., to multiple wireless users. The communications system 100 may enable multiple wireless users to access such content through the sharing of system resources, including wireless bandwidth. For example, the communications systems 100 may employ one or more channel access methods, such as code division multiple access (CDMA), time division multiple access (TDMA), frequency division multiple access (FDMA), orthogonal FDMA (OFDM A), single-carrier FDMA (SC-FDMA), and the like.

[00101] As shown in FIG. 18A, the communications system 100 may include wireless transmit/receive units (WTRUs) 102a, 102b, 102c, 102d, a radio access network (RAN) 104, a core network 106, a public switched telephone network (PSTN) 108, the Internet 110, and other networks 112, though it will be appreciated that the disclosed embodiments contemplate any number of WTRUs, base stations, networks, and/or network elements. Each of the WTRUs 102a, 102b, 102c, 102d may be any type of device configured to operate and/or communicate in a wireless environment. By way of example, the WTRUs 102a, 102b, 102c, 102d may be configured to transmit and/or receive wireless signals and may include user equipment (UE), a mobile station, a fixed or mobile subscriber unit, a pager, a cellular telephone, a personal digital assistant (PDA), a smartphone, a laptop, a netbook, a personal computer, a wireless sensor, consumer electronics, and the like.

[00102] The communications systems 100 may also include a base station 114a and a base station 114b. Each of the base stations 114a, 114b may be any type of device configured to wirelessly interface with at least one of the WTRUs 102a, 102b, 102c, 102d to facilitate access to one or more communication networks, such as the core network 106, the Internet 110, and/or the networks 112. By way of example, the base stations 114a, 114b may be a base transceiver station (BTS), a Node-B, an eNode B, a Home Node B, a Home eNode B, a site controller, an access point (AP), a wireless router, and the like. While the base stations 114a, 114b are each depicted as a single element, it will be appreciated that the base stations 114a, 114b may include any number of interconnected base stations and/or network elements.

[00103] The base station 114a may be part of the RAN 104, which may also include other base stations and/or network elements (not shown), such as a base station controller (BSC), a radio network controller (RNC), relay nodes, etc. The base station 114a and/or the base station 114b may be configured to transmit and/or receive wireless signals within a particular geographic region, which may be referred to as a cell (not shown). The cell may further be divided into cell sectors. For example, the cell associated with the base station 114a may be divided into three sectors. Thus, in one embodiment, the base station 114a may include three transceivers, i.e., one for each sector of the cell. In another embodiment, the base station 114a may employ multiple -input multiple output (MIMO) technology and, therefore, may utilize multiple transceivers for each sector of the cell.

[00104] The base stations 114a, 114b may communicate with one or more of the WTRUs 102a, 102b, 102c, 102d over an air interface 116, which may be any suitable wireless communication link (e.g., radio frequency (RF), microwave, infrared (IR), ultraviolet (UV), visible light, etc.). The air interface 116 may be established using any suitable radio access technology (RAT).

[00105] More specifically, as noted above, the communications system 100 may be a multiple access system and may employ one or more channel access schemes, such as CDMA, TDMA, FDMA, OFDMA, SC-FDMA, and the like. For example, the base station 114a in the RAN 104 and the WTRUs 102a, 102b, 102c may implement a radio technology such as Universal Mobile Telecommunications System (UMTS) Terrestrial Radio Access (UTRA), which may establish the air interface 116 using wideband CDMA (WCDMA). WCDMA may include communication protocols such as High-Speed Packet Access (HSPA) and/or Evolved HSPA (HSPA+). HSPA may include High-Speed Downlink Packet Access (HSDPA) and/or High-Speed Uplink Packet Access (HSUPA).

[00106] In another embodiment, the base station 114a and the WTRUs 102a,

102b, 102c may implement a radio technology such as Evolved UMTS Terrestrial Radio Access (E-UTRA), which may establish the air interface 116 using Long Term Evolution (LTE) and/or LTE- Advanced (LTE-A).

[00107] In other embodiments, the base station 114a and the WTRUs 102a,

102b, 102c may implement radio technologies such as IEEE 802.16 (i.e., Worldwide Interoperability for Microwave Access (WiMAX)), CDMA2000, CDMA2000 IX,

CDMA2000 EV-DO, Interim Standard 2000 (IS-2000), Interim Standard 95 (IS-95), Interim Standard 856 (IS-856), Global System for Mobile communications (GSM), Enhanced Data rates for GSM Evolution (EDGE), GSM EDGE (GERAN), and the like.

[00108] The base station 114b in FIG. 18A may be a wireless router, Home

Node B, Home eNode B, or access point, for example, and may utilize any suitable RAT for facilitating wireless connectivity in a localized area, such as a place of business, a home, a vehicle, a campus, and the like. In one embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as IEEE 802.11 to establish a wireless local area network (WLAN). In another embodiment, the base station 114b and the WTRUs 102c, 102d may implement a radio technology such as IEEE 802.15 to establish a wireless personal area network (WPAN). In yet another embodiment, the base station 114b and the WTRUs 102c, 102d may utilize a cellular-based RAT (e.g., WCDMA, CDMA2000, GSM, LTE, LTE-A, etc.) to establish a picocell or femtocell. As shown in FIG. 18A, the base station 114b may have a direct connection to the Internet 110. Thus, the base station 114b may not be required to access the Internet 110 via the core network 106.

[00109] The RAN 104 may be in communication with the core network 106, which may be any type of network configured to provide voice, data, applications, and/or voice over internet protocol (VoIP) services to one or more of the WTRUs 102a, 102b, 102c, 102d. For example, the core network 106 may provide call control, billing services, mobile location-based services, pre-paid calling, Internet connectivity, video distribution, etc., and/or perform high-level security functions, such as user authentication. Although not shown in FIG. 18A, it will be appreciated that the RAN 104 and/or the core network 106 may be in direct or indirect communication with other RANs that employ the same RAT as the RAN 104 or a different RAT. For example, in addition to being connected to the RAN 104, which may be utilizing an E-UTRA radio technology, the core network 106 may also be in communication with another RAN (not shown) employing a GSM radio technology.

[00110] The core network 106 may also serve as a gateway for the WTRUs

102a, 102b, 102c, 102d to access the PSTN 108, the Internet 110, and/or other networks 112. The PSTN 108 may include circuit-switched telephone networks that provide plain old telephone service (POTS). The Internet 110 may include a global system of interconnected computer networks and devices that use common communication protocols, such as the transmission control protocol (TCP), user datagram protocol (UDP) and the internet protocol (IP) in the TCP/IP internet protocol suite. The networks 112 may include wired or wireless communications networks owned and/or operated by other service providers. For example, the networks 112 may include another core network connected to one or more RANs, which may employ the same RAT as the RAN 104 or a different RAT.

[00111] Some or all of the WTRUs 102a, 102b, 102c, 102d in the

communications system 100 may include multi-mode capabilities, i.e., the WTRUs 102a, 102b, 102c, 102d may include multiple transceivers for communicating with different wireless networks over different wireless links. For example, the WTRU 102c shown in FIG. 18A may be configured to communicate with the base station 114a, which may employ a cellular-based radio technology, and with the base station 114b, which may employ an IEEE 802 radio technology.

[00112] FIG. 18B is a system diagram of an example WTRU 102. As shown in

FIG. 18B, the WTRU 102 may include a processor 118, a transceiver 120, a transmit/receive element 122, a speaker/microphone 124, a keypad 126, a display/touchpad 128, nonremovable memory 130, removable memory 132, a power source 134, a global positioning system (GPS) chipset 136, and other peripherals 138. It will be appreciated that the WTRU 102 may include any sub-combination of the foregoing elements while remaining consistent with an embodiment.

[00113] The processor 118 may be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), a plurality of microprocessors, one or more microprocessors in association with a DSP core, a controller, a microcontroller, Application Specific Integrated Circuits (ASICs), Field Programmable Gate Array (FPGAs) circuits, any other type of integrated circuit (IC), a state machine, and the like. The processor 118 may perform signal coding, data processing, power control, input/output processing, and/or any other functionality that enables the WTRU 102 to operate in a wireless environment. The processor 118 may be coupled to the transceiver 120, which may be coupled to the transmit/receive element 122. While FIG. 18B depicts the processor 118 and the transceiver 120 as separate components, it will be appreciated that the processor 118 and the transceiver 120 may be integrated together in an electronic package or chip.

[00114] The transmit/receive element 122 may be configured to transmit signals to, or receive signals from, a base station (e.g., the base station 114a) over the air interface 116. For example, in one embodiment, the transmit/receive element 122 may be an antenna configured to transmit and/or receive RF signals. In another embodiment, the transmit/receive element 122 may be an emitter/detector configured to transmit and/or receive IR, UV, or visible light signals, for example. In yet another embodiment, the transmit/receive element 122 may be configured to transmit and receive both RF and light signals. It will be appreciated that the transmit/receive element 122 may be configured to transmit and/or receive any combination of wireless signals.

[00115] In addition, although the transmit/receive element 122 is depicted in

FIG. 18B as a single element, the WTRU 102 may include any number of transmit/receive elements 122. More specifically, the WTRU 102 may employ MIMO technology. Thus, in one embodiment, the WTRU 102 may include two or more transmit/receive elements 122 (e.g., multiple antennas) for transmitting and receiving wireless signals over the air interface 116.

[00116] The transceiver 120 may be configured to modulate the signals that are to be transmitted by the transmit/receive element 122 and to demodulate the signals that are received by the transmit/receive element 122. As noted above, the WTRU 102 may have multi-mode capabilities. Thus, the transceiver 120 may include multiple transceivers for enabling the WTRU 102 to communicate via multiple RATs, such as UTRA and IEEE 802.11 , for example.

[00117] The processor 118 of the WTRU 102 may be coupled to, and may receive user input data from, the speaker/microphone 124, the keypad 126, and/or the display/touchpad 128 (e.g., a liquid crystal display (LCD) display unit or organic light- emitting diode (OLED) display unit). The processor 118 may also output user data to the speaker/microphone 124, the keypad 126, and/or the display/touchpad 128. In addition, the processor 118 may access information from, and store data in, any type of suitable memory, such as the non-removable memory 130 and/or the removable memory 132. The nonremovable memory 130 may include random-access memory (RAM), read-only memory (ROM), a hard disk, or any other type of memory storage device. The removable memory 132 may include a subscriber identity module (SIM) card, a memory stick, a secure digital (SD) memory card, and the like. In other embodiments, the processor 118 may access information from, and store data in, memory that is not physically located on the WTRU 102, such as on a server or a home computer (not shown).

[00118] The processor 118 may receive power from the power source 134, and may be configured to distribute and/or control the power to the other components in the WTRU 102. The power source 134 may be any suitable device for powering the WTRU 102. For example, the power source 134 may include one or more dry cell batteries (e.g., nickel- cadmium (NiCd), nickel-zinc (NiZn), nickel metal hydride (NiMH), lithium-ion (Li-ion), etc.), solar cells, fuel cells, and the like.

[00119] The processor 118 may also be coupled to the GPS chipset 136, which may be configured to provide location information (e.g., longitude and latitude) regarding the current location of the WTRU 102. In addition to, or in lieu of, the information from the GPS chipset 136, the WTRU 102 may receive location information over the air interface 116 from a base station (e.g., base stations 114a, 114b) and/or determine its location based on the timing of the signals being received from two or more nearby base stations. It will be appreciated that the WTRU 102 may acquire location information by way of any suitable location-determination method while remaining consistent with an embodiment.

[00120] The processor 118 may further be coupled to other peripherals 138, which may include one or more software and/or hardware modules that provide additional features, functionality and/or wired or wireless connectivity. For example, the peripherals 138 may include an accelerometer, an e-compass, a satellite transceiver, a digital camera (for photographs or video), a universal serial bus (USB) port, a vibration device, a television transceiver, a hands free headset, a Bluetooth® module, a frequency modulated (FM) radio unit, a digital music player, a media player, a video game player module, an Internet browser, and the like.

[00121] FIG. 18C is a system diagram of the RAN 104 and the core network

106 according to an embodiment. As noted above, the RAN 104 may employ a UTRA radio technology to communicate with the WTRUs 102a, 102b, 102c over the air interface 116. The RAN 104 may also be in communication with the core network 106. As shown in FIG. 18C, the RAN 104 may include Node-Bs 140a, 140b, 140c, which may each include one or more transceivers for communicating with the WTRUs 102a, 102b, 102c over the air interface 116. The Node-Bs 140a, 140b, 140c may each be associated with a particular cell (not shown) within the RAN 104. The RAN 104 may also include RNCs 142a, 142b. It will be appreciated that the RAN 104 may include any number of Node-Bs and RNCs while remaining consistent with an embodiment.

[00122] As shown in FIG. 18C, the Node-Bs 140a, 140b may be in

communication with the RNC 142a. Additionally, the Node-B 140c may be in

communication with the RNC142b. The Node-Bs 140a, 140b, 140c may communicate with the respective RNCs 142a, 142b via an Iub interface. The RNCs 142a, 142b may be in communication with one another via an Iur interface. Each of the RNCs 142a, 142b may be configured to control the respective Node-Bs 140a, 140b, 140c to which it is connected. In addition, each of the RNCs 142a, 142b may be configured to carry out or support other functionality, such as outer loop power control, load control, admission control, packet scheduling, handover control, macrodiversity, security functions, data encryption, and the like. [00123] The core network 106 shown in FIG. 18C may include a media gateway (MGW) 144, a mobile switching center (MSC) 146, a serving GPRS support node (SGSN) 148, and/or a gateway GPRS support node (GGSN) 150. While each of the foregoing elements are depicted as part of the core network 106, it will be appreciated that any one of these elements may be owned and/or operated by an entity other than the core network operator.

[00124] The RNC 142a in the RAN 104 may be connected to the MSC 146 in the core network 106 via an IuCS interface. The MSC 146 may be connected to the MGW 144. The MSC 146 and the MGW 144 may provide the WTRUs 102a, 102b, 102c with access to circuit-switched networks, such as the PSTN 108, to facilitate communications between the WTRUs 102a, 102b, 102c and traditional land-line communications devices.

[00125] The RNC 142a in the RAN 104 may also be connected to the SGSN

148 in the core network 106 via an IuPS interface. The SGSN 148 may be connected to the GGSN 150. The SGSN 148 and the GGSN 150 may provide the WTRUs 102a, 102b, 102c with access to packet-switched networks, such as the Internet 110, to facilitate

communications between and the WTRUs 102a, 102b, 102c and IP-enabled devices.

[00126] As noted above, the core network 106 may also be connected to the networks 112, which may include other wired or wireless networks that are owned and/or operated by other service providers.

[00127] FIG. 18D is a system diagram of the RAN 104 and the core network

106 according to an embodiment. As noted above, the RAN 104 may employ an E-UTRA radio technology to communicate with the WTRUs 102a, 102b, 102c over the air interface 116. The RAN 104 may also be in communication with the core network 106.

[00128] The RAN 104 may include eNode-Bs 160a, 160b, 160c, though it will be appreciated that the RAN 104 may include any number of eNode-Bs while remaining consistent with an embodiment. The eNode-Bs 160a, 160b, 160c may each include one or more transceivers for communicating with the WTRUs 102a, 102b, 102c over the air interface 116. In one embodiment, the eNode-Bs may implement MIMO technology. Thus, the eNode-B 160a, for example, may use multiple antennas to transmit wireless signals to, and receive wireless signals from, the WTRU 102a.

[00129] Each of the eNode-Bs may be associated with a particular cell (not shown) and may be configured to handle radio resource management decisions, handover decisions, scheduling of users in the uplink and/or downlink, and the like. As shown in FIG. 18D, the eNode-Bs may communicate with one another over an X2 interface.

[00130] The core network 106 shown in FIG. 18D may include a mobility management gateway (MME) 162, a serving gateway 164, and a packet data network (PDN) gateway 166. While each of the foregoing elements are depicted as part of the core network 106, it will be appreciated that any one of these elements may be owned and/or operated by an entity other than the core network operator.

[00131] The MME 162 may be connected to each of the eNode-Bs 160a, 160b, 160c in the RAN 104 via an SI interface and may serve as a control node. For example, the MME 162 may be responsible for authenticating users of the WTRUs 102a, 102b, 102c, bearer activation/deactivation, selecting a particular serving gateway during an initial attach of the WTRUs 102a, 102b, 102c, and the like. The MME 162 may also provide a control plane function for switching between the RAN 104 and other RANs (not shown) that employ other radio technologies, such as GSM or WCDMA.

[00132] The serving gateway 164 may be connected to each of the eNode Bs in the RAN 104 via the SI interface. The serving gateway 164 may generally route and forward user data packets to/from the WTRUs 102a, 102b, 102c. The serving gateway 164 may also perform other functions, such as anchoring user planes during inter-eNode B handovers, triggering paging when downlink data is available for the WTRUs 102a, 102b, 102c, managing and storing contexts of the WTRUs 102a, 102b, 102c, and the like.

[00133] The serving gateway 164 may also be connected to the PDN gateway

166, which may provide the WTRUs 102a, 102b, 102c with access to packet-switched networks, such as the Internet 110, to facilitate communications between the WTRUs 102a, 102b, 102c and IP-enabled devices.

[00134] The core network 106 may facilitate communications with other networks. For example, the core network 106 may provide the WTRUs 102a, 102b, 102c with access to circuit-switched networks, such as the PSTN 108, to facilitate communications between the WTRUs 102a, 102b, 102c and traditional land- line communications devices. For example, the core network 106 may include, or may communicate with, an IP gateway (e.g., an IP multimedia subsystem (IMS) server) that serves as an interface between the core network 106 and the PSTN 108. In addition, the core network 106 may provide the WTRUs 102a, 102b, 102c with access to the networks 112, which may include other wired or wireless networks that are owned and/or operated by other service providers.

[00135] FIG. 18E is a system diagram of the RAN 104 and the core network

106 according to an embodiment. The RAN 104 may be an access service network (ASN) that employs IEEE 802.16 radio technology to communicate with the WTRUs 102a, 102b, 102c over the air interface 116. As will be further discussed below, the communication links between the different functional entities of the WTRUs 102a, 102b, 102c, the RAN 104, and the core network 106 may be defined as reference points.

[00136] As shown in FIG. 18E, the RAN 104 may include base stations 170a,

170b, 170c, and an ASN gateway 172, though it will be appreciated that the RAN 104 may include any number of base stations and ASN gateways while remaining consistent with an embodiment. The base stations 170a, 170b, 170c may each be associated with a particular cell (not shown) in the RAN 104 and may each include one or more transceivers for communicating with the WTRUs 102a, 102b, 102c over the air interface 116. In one embodiment, the base stations 170a, 170b, 170c may implement MIMO technology. Thus, the base station 170a, for example, may use multiple antennas to transmit wireless signals to, and receive wireless signals from, the WTRU 102a. The base stations 170a, 170b, 170c may also provide mobility management functions, such as handoff triggering, tunnel

establishment, radio resource management, traffic classification, quality of service (QoS) policy enforcement, and the like. The ASN gateway 172 may serve as a traffic aggregation point and may be responsible for paging, caching of subscriber profiles, routing to the core network 106, and the like.

[00137] The air interface 116 between the WTRUs 102a, 102b, 102c and the

RAN 104 may be defined as an Rl reference point that implements the IEEE 802.16 specification. In addition, each of the WTRUs 102a, 102b, 102c may establish a logical interface (not shown) with the core network 106. The logical interface between the WTRUs 102a, 102b, 102c and the core network 106 may be defined as an R2 reference point, which may be used for authentication, authorization, IP host configuration management, and/or mobility management.

[00138] The communication link between each of the base stations 140a, 140b,

140c may be defined as an R8 reference point that includes protocols for facilitating WTRU handovers and the transfer of data between base stations. The communication link between the base stations and the ASN gateway 172 may be defined as an R6 reference point. The R6 reference point may include protocols for facilitating mobility management based on mobility events associated with each of the WTRUs 102a, 102b, 100c.

[00139] As shown in FIG. 18C, the RAN 104 may be connected to the core network 106. The communication link between the RAN 104 and the core network 106 may defined as an R3 reference point that includes protocols for facilitating data transfer and mobility management capabilities, for example. The core network 106 may include a mobile IP home agent (MIP-HA) 144, an authentication, authorization, accounting (AAA) server 176, and a gateway 178. While each of the foregoing elements are depicted as part of the core network 106, it will be appreciated that any one of these elements may be owned and/or operated by an entity other than the core network operator.

[00140] The MIP-HA may be responsible for IP address management, and may enable the WTRUs 102a, 102b, 102c to roam between different ASNs and/or different core networks. The MIP-HA 174 may provide the WTRUs 102a, 102b, 102c with access to packet-switched networks, such as the Internet 110, to facilitate communications between the WTRUs 102a, 102b, 102c and IP-enabled devices. The AAA server 146 may be responsible for user authentication and for supporting user services. The gateway 178 may facilitate interworking with other networks. For example, the gateway 178 may provide the WTRUs 102a, 102b, 102c with access to circuit-switched networks, such as the PSTN 108, to facilitate communications between the WTRUs 102a, 102b, 102c and traditional land-line communications devices. In addition, the gateway 178 may provide the WTRUs 102a, 102b, 102c with access to the networks 112, which may include other wired or wireless networks that are owned and/or operated by other service providers.

[00141] Although not shown in FIG. 18E, it will be appreciated that the RAN

104 may be connected to other ASNs and the core network 106 may be connected to other core networks. The communication link between the RAN 104 the other ASNs may be defined as an R4 reference point, which may include protocols for coordinating the mobility of the WTRUs 102a, 102b, 102c between the RAN 104 and the other ASNs. The

communication link between the core network 106 and the other core networks may be defined as an R5 reference, which may include protocols for facilitating interworking between home core networks and visited core networks. [00142] Although features and elements are described above in particular combinations, one of ordinary skill in the art will appreciate that each feature or element can be used alone or in any combination with the other features and elements. In addition, the methods described herein may be implemented in a computer program, software, or firmware incorporated in a computer-readable medium for execution by a computer or processor. Examples of computer-readable media computer-readable storage media. Examples of computer-readable storage media include, but are not limited to, a read only memory (ROM), a random access memory (RAM), a register, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disks, magneto-optical media, and optical media such as CD-ROM disks, and digital versatile disks (DVDs). A processor in association with software may be used to implement a radio frequency transceiver for use in a WTRU, UE, terminal, base station, RNC, or any host computer.

[00143] Variations of the method, apparatus and system described above are possible without departing from the scope of the invention. In view of the wide variety of embodiments that can be applied, it should be understood that the illustrated embodiments are exemplary only, and should not be taken as limiting the scope of the following claims.

[00144] Moreover, in the embodiments described above, processing platforms, computing systems, controllers, and other devices containing processors are noted. These devices may contain at least one Central Processing Unit ("CPU") and memory. In accordance with the practices of persons skilled in the art of computer programming, reference to acts and symbolic representations of operations or instructions may be performed by the various CPUs and memories. Such acts and operations or instructions may be referred to as being "executed," "computer executed" or "CPU executed."

[00145] One of ordinary skill in the art will appreciate that the acts and symbolically represented operations or instructions include the manipulation of electrical signals by the CPU. An electrical system represents data bits that can cause a resulting transformation or reduction of the electrical signals and the maintenance of data bits at memory locations in a memory system to thereby reconfigure or otherwise alter the CPU's operation, as well as other processing of signals. The memory locations where data bits are maintained are physical locations that have particular electrical, magnetic, optical, or organic properties corresponding to or representative of the data bits. It should be understood that the exemplary embodiments are not limited to the above-mentioned platforms or CPUs and that other platforms and CPUs may support the described methods.

[00146] The data bits may also be maintained on a computer readable medium including magnetic disks, optical disks, and any other volatile (e.g., Random Access Memory ("RAM")) or non-volatile (e.g., Read-Only Memory ("ROM")) mass storage system readable by the CPU. The computer readable medium may include cooperating or interconnected computer readable medium, which exist exclusively on the processing system or are distributed among multiple interconnected processing systems that may be local or remote to the processing system. It should be understood that the exemplary embodiments are not limited to the above-mentioned memories and that other platforms and memories may support the described methods.

[00147] No element, act, or instruction used in the description of the present application should be construed as critical or essential to the invention unless explicitly described as such. Also, as used herein, the article "a" is intended to include one or more items. Where only one item is intended, the term "one" or similar language is used. Further, the terms "any of followed by a listing of a plurality of items and/or a plurality of categories of items, as used herein, are intended to include "any of," "any combination of," "any multiple of," and/or "any combination of multiples of the items and/or the categories of items, individually or in conjunction with other items and/or other categories of items.

Further, as used herein, the term "set" is intended to include any number of items, including zero. Further, as used herein, the term "number" is intended to include any number, including zero.

[00148] Moreover, the claims should not be read as limited to the described order or elements unless stated to that effect. In addition, use of the term "means" in any claim is intended to invoke 35 U.S.C. § 112, 6, and any claim without the word "means" is not so intended.

Claims

What is claimed:

1. A method for interpolating an image comprising:

determining an edge characteristic associated with an interpolation point, the edge characteristic having an edge magnitude and an edge angle;

selecting an interpolation filter in response to the edge angle; and

determining a pixel value at the interpolation point using the selected interpolation filter.

2. The method of claim 1 wherein the edge characteristic is based on determining horizontal gradients and vertical gradients of pixel values in neighboring regions associated with the interpolation point.

3. The method of claim 2 wherein the neighboring regions associated with the interpolation point are one of a horizontal rectangular region, a vertical rectangular region or a square region.

4. The method of claim 2 wherein the neighboring regions are determined in response to the interpolation point being a row interpolation point, a column interpolation point or a center interpolation point.

5. The method of claim 1 wherein the edge characteristic is determined using a first order gradient filter.

6. The method of claim 1 wherein the edge characteristic is determined using a modified Sobel operator.

7. The method of claim 1 further comprising selecting an interpolation filter in response to the edge magnitude.

8. The method of claim 1 further comprising selecting an interpolation filter in response to an edge gradient.

9. The method of claim 1 wherein the interpolation filter applies greater weighting to the pixels located along the direction of the edge angle, and less or no weight to the pixels located along the edge gradient angle.

10. The method of claim 1 wherein the interpolation filter applies greater weighting to nearest-neighbor pixels and to the pixels located along the direction of the edge angle, and less or no weight to the pixels located along the direction of the edge gradient angle.

11. The method of claim 10 wherein the nearest-neighbor pixels are either pixels in the same row or in the same column as the interpolation point.

12. A computer readable medium for storing instructions that when executed by a processor will cause the processor to:

obtain a plurality of edge characteristics each associated with a respective one of a plurality of interpolation points, each of the edge characteristics having an edge magnitude and an edge angle;

select an interpolation filter for each one of the plurality of interpolation points in response to the respective edge angle;

determine a pixel value for each of the plurality of interpolation points using the corresponding selected interpolation filter; and

output the pixel values.

13. An interpolation device comprising:

an edge characteristic calculator configured to determine edge characteristics for each of a plurality of interpolation points;

an interpolation filter selector configured to operate on the edge characteristics and to responsively generate interpolation filter identifiers for each of the plurality of interpolation points; and,

an interpolation filter circuit configured to apply one of a plurality of interpolation filters in response to the interpolation filter identifiers and to output interpolated values for the plurality of interpolation points.

14. A method comprising :

interpolating pixels in an image frame in response to gradient data; and

sharpening pixels in the image frame in response to the gradient data.

15. The method of claim 14 wherein the interpolating includes weighting neighbor pixels according to the gradient data.

16. The method of claim 14 wherein sharpening includes determining high frequency information associated with individual pixels.

17. The method of claim 14 wherein the sharpening of pixels is performed for pixels having a gradient greater than a threshold.

18. The method of claim 14 wherein the sharpening of pixels is performed on pixels having a gradient greater than a threshold and on corresponding neighboring pixels.

19. The method of claim 14 wherein the sharpening of pixels is performed on pixels according to a sharpening map.

20. The method of claim 14 wherein the sharpening of pixels is performed using a strength parameter.

21. The method of claim 20 wherein the strength parameter is determined based on a gradient magnitude.

22. The method of claim 14 wherein interpolating pixels and sharpening pixels is done simultaneously with a combined filter.

23. An apparatus comprising:

an interpolation module configured to interpolate pixels in an image frame in response to gradient data; and a sharpening filter module configured to sharpen pixels in the image frame in response to the gradient data.

24. The apparatus of claim 23, wherein the interpolation module and sharpening filter module comprise a processor and computer readable memory device for storing instructions.

25. The apparatus of claim 23 comprising one or more digital signal processing filters.

26. A tangible computer readable storage medium having stored thereon instructions that when executed, cause a processing device to execute the method of claim 14.

27. A method comprising:

determining gradient data for image pixels to be interpolated;

determining interpolated pixel values using neighboring pixels according to the gradient data;

identifying image pixels to be sharpened based on gradient data; and,

sharpening image pixels using a filter strength determined in part based on the gradient data.

28. The method of claim 27 wherein determining interpolated pixel values is performed based in part on a pixel category being one of center, row or column.

29. A method comprising:

determining gradient data for image pixels to be interpolated;

selectively sharpening neighboring original pixels;

selectively identifying neighboring pixels that have yet to be interpolated in response to the pixel category of center row or column pixel; and,

determining interpolated and sharpened pixel values using neighboring pixels according to a joint sharpening and interpolation filter.

30. The method of claim 29 wherein the joint sharpening and interpolation filter performs interpolation based on an edge angle and a sharpening based on original pixels within a window.

31. The method of claim 30 wherein the joint sharpening and interpolation filter performs a sharpening also based on estimated interpolation points within the window.