WO2014008329A1

WO2014008329A1 - System and method to enhance and process a digital image

Info

Publication number: WO2014008329A1
Application number: PCT/US2013/049200
Authority: WO
Inventors: Eberhard Herbert FISCH; Junlan Yang
Original assignee: Marseille Networks, Inc.
Priority date: 2012-07-03
Filing date: 2013-07-03
Publication date: 2014-01-09
Also published as: CN104756151A

Abstract

A Digital Video Processing System is configured to process an input digital image with a certain level of noise in its video data information, including a first type of noise that is undesirable and a second type of noise that is desirable. The Digital Video Processing System is further configured to selectively process the input digital image to remove the first type of noise while maintaining the second type of noise by independently processing the second type of noise to a desired output resolution and by combining the video data information of the processed second type of noise with video data information of a processed and noise free input image to produce output video data information corresponding to a processed output digital image at a desired resolution and/or format. The Digital Video Processing System transmits the video data information of the processed output digital image to an electronic display device, or a region of an electronic display device at a desired output resolution.

Description

SYSTEM AND METHOD TO ENHANCE AND PROCESS A DIGITAL IMAGE

PRIORITY CLAIM

[0001] This patent application claims the benefit of and priority to U.S. Provisional Patent Application Number 61/690,804 filed on July 3^rd, 2012, which is incorporated herein by reference in its entirety, and to U.S. Utility Patent Application Number 13/694,089 filed October 29^th, 2012, which is incorporated herein by reference in its entirety.

TECHNICAL FIELD

[0002] The subject of this application relates generally to two-dimensional and three- dimensional video and digital image processing and more particularly to the processing of video data information of a digital image.

BACKGROUND

[0003] Video enhancement techniques are designed to enhance the visual quality of videos and images. Due to an ever increasing resolution of display devices, video enhancement is becoming an indispensable component for most of the high quality video processing systems and products. Video enhancements usually include a range of features such as focus enhancement, motion stability/blur removal, contrast enhancement, color adjustment, signal filtering, flicker reduction, and frame rate conversion. In general, there are various types of enhancement for digital images of certain application such as medical scan, MRI imaging which have been studied and analyzed, and certain techniques are useful in enhancing the contrast or sharpness given the monotonic nature of such digital images. However, normal motion video images or typical images or are not monotonic in nature. One would expect a typical digital image to include different regions comprising different levels of details. Some regions may include very high amount of fine details (texture), other regions may include edges or contours (jaggy), yet other regions may include low or hardly any details at all (flat region). Given a certain region size, e.g. a block of pixels having a number of lines and each line having a number of pixels, a region may include multiple levels of details corresponding to different portions of the region, e.g. a region may include a combination of texture, jaggy, or flat portions. [0004] In general, there are various noise reduction techniques that may be used to process an input digital image in order to reduce noise or certain other artifacts of the input image or to enhance a certain characteristic of the input image such as color or contrast. However, these various techniques when applied systematically to the video data information of the input digital image may generate good output results for one type of images while a detrimental result may be generated for other types of images. For example, applying noise reduction techniques for a flat or monotonic image may greatly reduce noise resulting in an enhanced viewing quality, but if the input image is not monotonic, e.g. a typical image, then the noise reduction technique would greatly reduce the amount of viewable details and hence lowering the viewing quality. The noise reduction process would remove the high level of details as well as any existing noise from the input image, and thus cannot distinguish different types of noises such as high frequency details, comfort noise such as film grain, compression noise, or other types of noise and artifacts. Thus, such noise reduction process may result in an un-natural appearance of the processed digital image and/or reduced sharpness for regions that include high level of details or edges, while flat regions may appear smoother or cleaner. Therefore, there is a need for an improved system and method to process and enhance digital images.

SUMMARY

[0005] The present disclosure provides a video and image processing and enhancement system particularly suited for providing detail enhancement, sharpness enhancement, noise reduction, and edge enhancement while taking into account various characteristics and features of a region of the image being processed, a group of pixels within the region of the image being processed, and/or the image as a whole. Thus, using the video data information content of a local neighborhood or a group of pixels, the effect of processing video data information to perform one type of enhancement would not detrimentally affect the results of another desired enhancement.

[0006] The present disclosure further provides an intelligent video and image processing and enhancement system that can distinguish the various types of enhancements needed or being performed concurrently of a respective pixel such that the enhanced video data information of each output pixel is generated while taking into consideration the video data information content of a local neighborhood or a group of pixels surrounding the respective pixel. [0007] Moreover, in certain embodiments, the enhancement processing for an output pixel is optimized by controlling the type and amount of processing being performed, for example by including an intelligent (or weighted) blending of two or more types of enhancement processing using a feedback mechanism from using contextual video data information surrounding the pixel being processed. The present disclosure further provides an intelligent video and image processing and enhancement system that can process and generate video data information of each output pixel which is optimized for an enhanced viewing quality while processing artifacts and input image noise are greatly reduced or eliminated.

[0008] The present disclosure further provides a system and method to enhance and process digital images using customized processing for different regions or portions of the digital image such that the overall viewing quality of the digital image is enhanced. In particular, a variety of techniques for processing - e.g. sharpening - may be applied to a first region within the digital image, while different techniques for processing - e.g. noise reduction - maybe applied to a second region within the digital image such that the overall viewing quality of the digital image is enhanced. In one embodiment, certain characteristics of local regions within an input digital image are used to determine what type of video processing such as sharpening or noise reduction should be applied to a pixel within a local region.

[0009] In order to process video data information for one pixel within a region of the input digital image, a set of one or more parameters are extracted using video data information of each pixel within the group of pixels, where the region of the input image correspond to a portion of the input image, and the group of pixels surrounds the current pixel being currently processed and is determined using a window of pixels having a first number of lines and each line having a first number of pixels or an N x N block of pixels. The set of one or more parameters are used to control and to dynamically adapt the amount of filtering or processing to be performed to generate video data information for one or more output pixels using video data information of the one pixel to be processed and using video data information of the group of pixels. Thus, each pixel within a region of the digital image would be optimally processed while taking into consideration the characteristics of the local region,

neighborhood, window, and/or block of pixels.

[0010] In accordance with one embodiment, a Digital Video Processing System is described to process an input digital image, where the video data information of the input digital image includes an undesired noise and a comfort noise. The Digital Video Processing System extracts the input noise that corresponds to both the undesired noise and the comfort noise out of the input image, processes the remaining noise free image, scaling the noise free image to a desired output resolution, extracts the comfort noise from the input noise, processes the comfort noise and regenerates the comfort noise at the desired output resolution, adds the processed comfort noise to the processed noise free input image to produce the output digital image at the desired resolution, and displays the processed output digital image using an electronic display device, or a region of an electronic display device at the desired output resolution.

[0011] In accordance with certain embodiments, a method for processing video data information of a first pixel of an input image is disclosed. The method comprises receiving video data information of a first portion of the input image, determining a first group of pixels within the first portion of the input image, wherein the first group of pixels includes the first pixel, estimating at least one value of a first indicator parameter using video data information of the first group of pixels, generating a noise reduction parameter value using the at least one value of the first indicator parameter, generating video data information of an output pixel by using the noise reduction parameter value to process video data information of the first group of pixels, wherein the output pixel corresponds to a processed version of the first pixel of the input image, and storing the video data information of the output pixel using an output buffer.

[0012] In accordance with certain embodiments, a method for processing video data information of a pixel of an input image, the method comprises receiving video data information of a first portion of the input image, estimating at least one value of a first indicator parameter using video data information of a first group of pixels, each pixel of the first group of pixels is located within the first portion of the input image, wherein the first group of pixels includes the pixel, generating a noise reduction parameter value using the at least one value of the first indicator parameter, processing video data information of each pixel of the first group of pixels using any one of the noise reduction parameter value and the at least one value of the first indicator parameter, and generating video data information of an output pixel using the processed video data information of each pixel of the first group of pixels, wherein the output pixel corresponds to a processed version of the pixel of the input image. The video data information of the output pixel can be stored using an output buffer, streamed to an electronic display system, or transmitted to a secondary system for further processing, conversion, blending, or formatting for one or more different types of video data information formats and standards.

[0013] In accordance with one embodiment, the at least one value of the first indicator parameter indicates that the first group of pixels corresponds to any one or more of a flat area, a detail area and an edge.

[0014] In accordance with one embodiment, the video data information of the first group of pixels used to estimate the at least one value of the first indicator parameter includes only one of a color and luminance format of the video data information of each pixel within the first group of pixels.

[0015] In accordance with one embodiment, the video data information of each pixel of the first portion of the input image is stored using an input buffer.

[0016] In accordance with one embodiment, the first group of pixels includes at least three lines of pixels, and each line of pixels having at least three pixels.

[0017] In accordance with one embodiment, at least one value of a first indicator parameter is estimated using video data information of the first group of pixels by determining one or more distinctive features of the first group of pixels. The one or more distinctive features of the first group of pixels correspond to any one of a flat area, a detail area and an edge, and is determined by using a linear or a non-linear function to process video data information of each pixel of the first group of pixels. An estimate of the at least one value of a first indicator parameter is indicative of a distinctive feature of the first group of pixels.

[0018] In accordance with one embodiment, at least one value of a first indicator parameter is estimated using video data information of the first group of pixels by determining one or more distinctive features of the first group of pixels by using a non-linear function to process video data information of each pixel of the first group of pixels, wherein the one or more distinctive features of the first group of pixels correspond to any one of a flat area, a detail area and an edge, and estimating at least one value of a first indicator parameter indicative of a distinctive feature of the first group of pixels.

[0019] In accordance with one embodiment, the generating a noise reduction parameter value using the at least one value of the first indicator parameter comprises generating a noise reduction parameter value based on (i) the at least one value of the first indicator parameter, and (ii) the location of the pixel within the first group of pixels.

[0020] In accordance with one embodiment, the generating a noise reduction parameter value using the at least one value of the first indicator parameter comprises generating a noise reduction parameter value for each pixel of the first group of pixels based on (i) the at least one value of the first indicator parameter, and (ii) the location of each pixel of the first group of pixels within the first group of pixels.

[0021] In accordance with one embodiment, the noise reduction parameter value is to effect a high level of smoothing for the pixel when the first group of pixels correspond to a flat area, and the noise reduction parameter value is to effect a low level of smoothing for the pixel when the first group of pixels correspond to a detail area or an edge.

[0022] In accordance with certain embodiment, a system for processing video data information of a pixel of an input image is described. The system comprises a first processing subsystem operable (i) to receive video data information of a first portion of the input image, (ii) to generate at least one value of a first indicator parameter using video data information of a first group of pixels, and (iii) to generate a noise reduction parameter value using the at least one value of the first indicator parameter, wherein the first group of pixels (i) includes the pixel, and (ii) is within the first portion of the input image; and wherein a second processing subsystem is operable (i) to generate video data information of an output pixel in response to the noise reduction parameter value and video data information of the first group of pixels, and (ii) to store the video data information of the output pixel into an output buffer, wherein the output pixel corresponds to a processed version of the pixel of the input image.

[0023] In accordance with one embodiment, the first group of pixels includes at least three lines of pixels and each line of pixels includes at least three pixels.

[0024] In accordance with one embodiment, the at least one value of the first indicator parameter indicates that the first group of pixels corresponds to any one or more of a flat area, a detail area and an edge.

[0025] In accordance with one embodiment, the first processing subsystem stores video data information of each pixel of the first portion of the input image into an input buffer. [0026] In accordance with one embodiment, the first processing subsystem is operable to process video data information of the first group of pixels, and to generate the at least one value of the first indicator parameter based at least in part on the processed video data information of the first group of pixels indicating that the video data information of the first group of pixels correspond to one or more of a flat area, a detail area, and an edge.

[0027] In accordance with one embodiment, the first processing subsystem is operable to generate the at least one value of the first indicator parameter using a non-linear function to process video data information of each pixel of the first group of pixels.

[0028] In accordance with one embodiment, the first processing subsystem is operable to generate the noise reduction parameter value based at least in part on (i) the at least one value of the first indicator parameter, and (ii) the location of the pixel within the first group of pixels.

[0029] In accordance with one embodiment, the first processing subsystem is operable to generate a noise reduction parameter value for each pixel of the first group of pixels based at least in part on (i) the at least one value of the first indicator parameter, and (ii) the location of each pixel of the first group of pixels within the first group of pixels.

[0030] In accordance with one embodiment, the second processing subsystem is operable to generate video data information of the output pixel by applying a high level of smoothing to the video data information of the pixel based at least in part (i) on the noise reduction parameter value, and (ii) in response to the video data information of the first group of pixels corresponding to a flat area.

[0031] In accordance with one embodiment, the second processing subsystem is operable to generate video data information of the output pixel by applying a low level of smoothing to the video data information of the pixel based at least in part (i) on the noise reduction parameter value, and (ii) in response to the video data information of the first group of pixels corresponding to a detail area or an edge.

[0032] In accordance with certain embodiments, a method for processing video data information of an input image comprises: selecting a first group of pixels within the input image; generating at least one value of a first indicator parameter, wherein the at least one value of the first indicator parameter indicates that video data information of the first group of pixels correspond to one or more of a flat area, a detail area and an edge; generating a noise reduction smoothing parameter using the at least one value of the first indicator parameter; generating video data information of an output pixel based at least in part on (i) video data information of each pixel within the first group of pixels, and (ii) the noise reduction smoothing parameter, wherein the output pixel correspond to a processed version of a first pixel within the first group of pixels.

[0033] In accordance with one embodiment, the video data information of the output pixel is generated using a high level of noise reduction processing when the video data information of the first group of pixels correspond to a flat area.

[0034] In accordance with one embodiment, the video data information of the output pixel is generated using a low level of noise reduction processing when the video data information of the first group of pixels correspond to a detail area or an edge.

[0035] In accordance with one embodiment, generating a noise reduction smoothing parameter using the at least one value of the first indicator parameter comprises generating a noise reduction smoothing parameter value for each pixel of the first group of pixels based on (i) the at least one value of the first indicator parameter, and (ii) the location of each pixel of the first group of pixels within the first group of pixels.

[0036] In accordance with one embodiment, the generating a noise reduction smoothing parameter using the at least one value of the first indicator parameter comprises generating a noise reduction smoothing parameter value for the first pixel within the first group of pixels based at least in part on (i) the at least one value of the first indicator parameter, and (ii) the location of the first pixel within the first group of pixels.

[0037] In accordance with one embodiment, the video data information of the first group of pixels to be used for processing includes only one of a color and luminance format of the video data information of each pixel within the first group of pixels.

[0038] In accordance with one embodiment, a method for processing a digital image comprises adding or blending a small amount of a processed comfort noise to a noise- free processed video data information. The noise-free processed video data information is the results of applying a noise reduction processing step to the video data information of the input digital image. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Fig. 1 is an example of a digital image processing system in accordance with one embodiment.

[0040] Fig. 2 illustrates an example of using a Euclidean coordinate space to translate a block of pixels B₂ (0,0) to B₂ (x₀, y₀), in accordance with one embodiment.

[0041] Fig. 3 illustrates an example of a digital image or a portion of a digital image in accordance with one embodiment.

[0042] Fig. 4 is an example of a digital image processing system in accordance with one embodiment.

[0043] Fig. 5 is an example of a digital image processing system in accordance with one embodiment.

DETAILED DESCRIPTION

[0044] A Digital Video Processing System is provided to process an input digital image, where video data information of the input digital image includes an undesired noise and a desired comfort noise. The basic concept of maintaining the same natural look and comfort noise for digital images after being digitally processed is particularly important especially when the processing steps can include up-scaling the resolution of the input image to a desired output resolution.

[0045] The following paragraphs present a summary of some of the issues related to the processing and upscaling of a digital image to include a controlled and adaptive noise removal and reintroduction of a certain level of comfort noise. In addition, based on the contextual content of the video data information of a region, the processing and/or upscaling of the digital image may alter, adapt, or include specific processing steps for a certain location or a region within the digital image that is different from the processing steps used for another region within the digital image.

[0046] The familiar film look is preferred by many viewers rather than the CCD sensory captured digital video. Sometimes, in digital video post-production a synthesized film grain noise is introduced as comfort noise or to achieve specific or desired effects. For example, a common stylistic artistic technique is to introduce high levels of film grain noise and simulated scratches for flashbacks to historical events. Therefore, applying noise reduction techniques would necessarily removes this "noise" - which was introduced as a part of artistic technique - and thus viewing the resultant relatively-noise-free processed video data information would defeat the original intended artistic expression.

[0047] Furthermore, film grain can be correlated with the intensity of local flat regions of an image. For example, a very little noise may be added to very bright or dark flat areas, or to edges of an image. Although we can synthesize film grain we don't know what the "correct" noise level in different regions within the image or even the type of noise that exist, whether it is a desired or undesired type of noise. After applying a noise reduction processing step, all or majority of noise is filtered out and it is very difficult for currently known noise reduction algorithms to distinguish between different types of noise, let alone a desired type of noise such as fine grain noise.

[0048] For example, if an image with certain type of noise is processed and its resolution is scaled up, then consequently the fine grain noise is scaled up as well which results in large blob-like random patterns rather than fine grained comfort noise. It may be possible to synthesize fine grained film grain with a fixed average density and dynamic range and add it back to the larger up-scaled image but this does not replicate the original image region dependent variations in fine grain density and dynamic range of the original smaller resolution image or video frame. The processed or up-scaled video would have an artificial look after being processed and scaled using current methods. Therefore, there exist a need to process an image or a video frame at a high resolution and enhance the overall viewing quality of the processed digital image using multiple techniques for noise reduction and sharpening while maintaining a similar comfort noise or film grain viewing experience of the original image.

[0049] The following paragraphs describe video data information processing systems and methods that can achieve a noise reduction while retaining the original intended artistic expression of the input image. In addition, the present disclosure presents techniques to process video data information of captured digital images received at a given resolution, process and output the video data information at higher resolution to be displayed using a high-resolution electronic display panel. Fine grain noise should be separated from the image at the lower native resolution. In addition, other un-desirable effects such as false contours, jaggies, compression block, and mosquito noise should be eliminated as well. The natural picture region-dependent comfort noise should then be re- introduced with the look of the original digital image at the higher resolution after up-scaling.

[0050] The disclosure and the various features and advantageous details are explained more fully with reference to the nonlimiting embodiments that are illustrated in the accompanying drawings and detailed in the following description. Furthermore, description of well-known processing techniques, components, and equipment are omitted so as not to unnecessarily obscure the invention in detail. It should be understood, however, that the detailed description and the specific examples, while indicating embodiments of the invention, are given by way of illustration and not by way of limitation. Various substitutions, modifications, additions, and/or rearrangements within the spirit and/or scope of the underlying inventive concepts will become apparent to those of ordinary skill in the art from this disclosure. Other features and associated advantages will become apparent with reference to the following detailed description of specific embodiments in connection with the accompanying mathematical representations, computational models, and drawings. Various implementation techniques may be used to implement a given mathematical representations or a computational model so as to properly process video data information in accordance with one or more of the detailed embodiments below.

[0051] A region of a digital image would include a neighborhood of pixels. When the region of the digital image corresponds to a high level of details, then the video data information of each pixel within the region of the digital image is expected to have a high degree of variance with respect to video data information of the other pixels within the region. Such region would correspond to having a high level of texture or details content, for example a portion of digital image of a colorful butterfly.

[0052] In contrast, a region of a digital image with a low level of details would be considered a flat region and would include a neighborhood of pixels where the video data information of each pixel of the region is expected to have a low variance with respect to video data information of the other pixels of this flat region, which is indicative of low level details or a flat region, for example a portion of digital image of blue sky.

[0053] Furthermore, a region of a digital image that contains edges or contours may be referred to as a Jaggy region and thus would include multiple regions or neighborhoods of pixels that varies between flat and detail regions. For example, the video data information of some of the pixels at the boundary of an edge or a contour are expected to have a high variance with respect to video data information of other pixels corresponding to the edge boundary neighborhood. For example, the variance between pixels that exist within edges of a text within a digital image is low, while a high variance is expected between a pixel within the edge and a pixel just outside the boundary of an edge. Thus, the video data information of each of the pixels comprising the edge itself is expected to have a low variance with respect to the other pixels forming the edge itself. Similarly, the video data information of each of the pixels located within the neighborhood of the edge, e.g. few pixels away from the edge, may have a low variance with respect to other pixels forming the region or neighborhood of pixels close to the edge or contour. However, a high variance is expected between two adjacent pixels where one pixel is part of an edge while the other pixel is not part of the edge.

[0054] Moreover, a portion of a digital image may be classified using additional criteria based on a given relationship between the video data information content of each of the pixels within a neighborhood of pixels of the region within the portion of the digital image. Additionally, each region within the portion of the digital image may include one or more types of details. In other words, a region may include multiple local regions (or neighborhood of pixels) each of which contains different level of details, edges or contours, and flat areas.

[0055] Consequently, applying a certain type of enhancement technique used for digital images of certain type or application - such as medical scans or finger print images - will not be suited to enhance the viewing experience of typical digital images containing many different types of image details. For example, applying a noise reduction filtering technique or sharpening algorithm to an MRI scan may produce a better and enhanced viewing experience of that image. However, applying the same technique to a typical image can actually reduce noise or sharpen the image in certain regions having one type, e.g. a flat area, while having a devastating effect on other regions of the image having different type, e.g. detail area or an edge. Such processing may cause blurring or other severe artifacts. Thus, an unsuccessful enhancement and a decrease of the overall viewing quality of digital images or a moving picture. The following detailed description of systems and methods to enhance the viewing quality of digital images while taking into consideration multiple and distinct parameters that can affect and alter, modify or dynamically adjust the processing of the input digital according to desired enhancement and based on video data information content of global and local regions within the input digital image.

[0056] Specific processing of a features of digital image such as detail enhancement, edge enhancement and noise reduction will be used as an example. Detail enhancement refers to the process of enhancing the details portions in the image such that the details are more visible and an increase in contrast is achieved. Edge enhancement mainly concentrates on jaggy removal, where the term "jaggy" or "jaggies" are referring to the jagged, stairs-like or toothed like appearance which is usually seen on straight edges in digital images such as a scene boundary or edges of text within the image. Edge enhancement not only removes the jaggies but also removes noise around the edges thus improves the consistency, appearance, and viewing or readability of the edges. Noise reduction is used to remove random noise, film grain noise, as well as mosquito noise and block noise generated with video compression process. A processing step to reintroduce a processed version of extracted film grain noise to the output enhanced image is also described. These enhancements are obtained using separate processing steps and logically applied to generate the final video data information of the output image to create an enhanced viewing experience of the input image.

[0057] Various attempts have been made to provide certain enhancement for video and digital images. However, the resultant processing artifacts is a major source of problem that worsens the viewing experience. Taking each enhancement steps separately would introduce undesired processing artifacts. For example, a noise reduction processing steps can easily cause loss of details in textured regions and blur around edges while improving flat region or areas. Edge enhancement processing steps can introduce smeared texture in texture regions. Detail enhancement processing steps when applied to noisy regions or images can increase the noise level while at the same time improving sharpness in other regions. These processing artifacts are quite annoying for normal viewers. Therefore, using an intelligent system to control and dynamically alter or adjust such processing steps according to a region or local neighborhood content would result in a major improvement and achieve the generation of output video data information providing an enhanced viewing experience.

[0058] As described earlier, a fundamental reason for the artifacts is due to the fact that the processing techniques are not adaptive to images or regions within the images to be processed, either in terms of local features or global features. In other words, the type of enhancement processing and the strengths or extent of processing to be applied should be different for different types of content of regions within the images as well as for different types of images. For instance, we should only apply a light noise reduction around edges and yet an even smaller magnitude of noise reduction processing for a micro-textured region; so that edges and textures are better preserved. In contrast, we should apply a large magnitude or a strong noise reduction processing to a flat region as noise is most visible there; and at the same time there is no edges or detail area that can suffer from the noise reduction processing steps.

[0059] The term "coupled" is defined as connected, although not necessarily directly, and not necessarily mechanically. The terms "a" and "an" are defined as one or more unless this disclosure explicitly requires otherwise. The terms "comprise" (and any form of comprise, such as "comprises" and "comprising"), "have" (and any form of have, such as "has" and "having"), "include" (and any form of include, such as "includes" and "including") and "contain" (and any form of contain, such as "contains" and "containing") are open-ended linking verbs. As a result, a method or device that "comprises," "has," "includes" or

"contains" one or more steps or elements possesses those one or more steps or elements, but is not limited to possessing only those one or more elements. Likewise, a step of a method or an element of a device that "comprises," "has," "includes" or "contains" one or more features possesses those one or more features, but is not limited to possessing only those one or more features. Furthermore, a device or structure that is configured in a certain way is configured in at least that way, but may also be configured in ways that are not listed.

[0060] Various processing techniques can be used to process a digital image or to scale a digital image from a first specified resolution to a second specified resolution. In the following paragraphs certain definitions and mathematical representations of a digital image model will be presented, and the processing of a digital image is discussed in accordance with certain embodiments.

[0061] Let z(x, y) by the original noisy video frame with width and height W x H before up-scaling. The noisy video can be thought of as being composed of the sum of three separate additive frames:

^{— V}WxH 9WXH (x, y) + ^uWxH (x, y) eqf(l) where x = 0,1,^■■■ , W— 1; and y = 0,1, ··· , H— 1 wherein, the clean noiseless video is v_{W H}(x, y), the good fine grained comfort noise is g_{W H} {^x, y)_> ^and the undesirable bad noise is b_WxH(x, y).

[0062] A processing step is performed to separate v_WxH(x, y) and g_W H i^x _> y) fr°^{m me} noisy input frame z_{W H}(x, y) as separate channels or as separate frames of the digital image. Examples of such processing step are described below. An T N x N up-scaling algorithm is now applied to the clean video frame v_WxH(x, y) to produce scaled clean image

V_NWxNH(x, y) of size NW x NH, where N is a scaling factor that can be either an integer or a fraction. For the fine grain comfort noise frame gwx.H ^^x _> y) ^we first increase the frame size by zero padding by a factor of T N x N. This produces a fine grain noise image of the size NW x NH denoted by gNwxNH (^x _> y which matches the size of the clean up-scaled video frame V_{NW NH}(x, y). However, the region dependent density of the larger comfort noise image gNwxNH (^x _> V) is of the original density in the smaller sized noise frame 9wxH (.^x _> y) - F°^r a square block B_NxN(x₀, y₀) of JV x W pixel coordinates centered at image coordinates (x₀, y₀) in the smaller sized comfort noise image 9WXH (^x _> y) ^me local fine grain density is d(x_Q, y₀) = (number of noise grains in B_NxN(x₀, y₀) ) / N x N. The average number of film grains or film grain density d(x₀, y₀) is a local average measured at each image coordinates (x₀, y₀) .

[0063] In accordance with one embodiment, an example is used to illustrate the above description using a scaling factor N = 2, a 4x4 window W = 4, and H = 4.

The video source image with only the good fine grained comfort noise can be written as:

^Vll + 911 ^V12 ^V13 ^V14 + 914

^V21 ^V22 + g22 ^V23 ^V24

v_4x4(x, y + g_4x4(x, y = eq{2)

^ l + fl^ l ^V32 ^V33 + g33 ^V34

.^V41 + #41 ^V42 + #42 ^V43 + #43 ^V44 + #44

Separating the good fine grained comfort noise from the clean video image:

^'V-ii v₁₂ v₁₃ v₁₄ #11 0 0 #14

^V21 ^V22 ^V23 ^V24 0 #22 0 0

eq(3

^V31 ^V32 ^V33 ^V34 #31 0 #33 0

_^V41 ^V42 ^V43 ^V44. .#41 #42 #43 #44 Scaling of the clean video using N=2: v₁₂ v₁₃ v₁₄ v₁₅ v₁₆ V₁₇ v₁₈i

V21 V₂₂ V22 V24 v₂₅ v₂₆ v₂₇ v₂₈

!¾! Vl3 v₁₄ V31 V32 V33 V34 v₃₅ V36 v₃₇ V38

¾1 V22 V₂3 V41 V42 V43 V44 v₄₅ V46 V₄₇ V48 eq(

V₃1 V32 V33 V34 v₅₁ v₅₂ V₅3 V₅4 v₅₅ v₅₆ v₅₇ v₅₈ v₄₁ V42 V43 V44. v₆₂ V₆3 V₆4 Vee v₆₇ v₆₈

V₇₁ v₇₂ V₇3 v₇₄ v₇₅ v₇₆ v₇₇ v₇₈

v₈₂ V₈3 V₈4 v₈₅ V₈₆ v₈₇ v₈₈\

[0064] Zero padding of the good fine grained comfort noise component would results in fine grain image of correct size but it is too sparse. The sampling density needs to be increased by a factor of 4, and the film grain density increases by factor of 4.

[0065] There are a number of approaches for accomplishing the increase of grain density without introducing any period patterns. Only for the purpose of illustration we work through this example using a method based on applying non-overlapping random coordinate shifts on the original sparse grain image #₈x8(^x, y)-

[0066] We want to avoid any overlap of sample positions since we don't intend to average any samples of the film grain noise values. Thus we may use a method of applying random shifts to the sparse grain by drawing the shift vector at random from coordinate shift vector set. For example, as shown in eq(7).

S₀ = {(0,1), (0, -1), (1,0), (-1,0), (1,1), (-1, -1)} eq{7) [0067] The shift using this set of lattice points will insure no overlap between the original sparse matrix and the new shifted one at coordinates with non-zero values.

Let s be a coordinate shift vector drawn at random from the set

O, t) G S₀ = {(0,1), (0, -1), (1,0), (-1,0), (1,1), (-1, -1)} eq(8

[0068] The shift vector is drawn without replacement. That way the shifts will be applied in a random order with the same shift never appearing more than once.

Random selection of shift s_t

_Sl = (¾, = (1,1) eq(10) S₁ <- S₀ - _Sl = {(0,1), (0, -1), (1,0), (-1,0), (-1, -1)}

9i(x, y) = #₀O + i, y + l) eq(12

Random selection of shift ;

s₂ = (s₂, t₂) eq(15)

S₂ <" S_t - s₂ = {(0, -1), (1,0), (-1,0), (-1, -1)} eq(16)

#2 fe ) = #oO< y + 1) eq(17)

Random shift selection of shift s₃

s₃ = (s₃, t₃) = (1,0)

S₂ <- S₂ - s₃ = {(0, -1), (-1,0), (-1, -1)}

Hence,

G x,y) = g₀ x,y) + #iO,y) + g₂(*,y + #₃0,y) eq(25

-

-

= 9o 0< ) + 9i 0< y) + 92 (x, y) + 5s (*, y) eq (29)

[0069] The final good fine grained comfort noise scaled matrix is presented in eq(31) as presented below.

Additional randomization may be added as part of the algorithm of the example shown above.

[0070] An example of a Digital Video Processing System 100 for processing a digital image is shown in Fig. 1. The Digital Video Processing System 100 provides for the processing of an input digital image, where the video data information of the input digital image includes an input noise component comprising (i) an undesired noise and (ii) a comfort noise. In accordance with one embodiment, the Digital Video Processing System 100 extracts both the undesired noise and the comfort noise from the input image, processes the noise free image, including scaling of the noise free image to a desired output resolution, processes the comfort noise to match the desired output resolution, adds the processed comfort noise to the processed noise free input image to produce the output digital image at the desired resolution which is to be displayed using an electronic display device or a region of an electronic display device.

[0071] The input to the Digital Video Processing System 100 is shown as z_WxH(x, y) 101, which corresponds to a digital image or a portion of a digital image with a width W and height H in pixels. The video data information of the input digital image received by the Digital Video Processing System 100 includes a comfort noise component, an undesired noise component, and a relatively noise free input image. A mathematical representation for the video data information of the input image maybe written as follows:

(x, y) 101 (x, y) + g_WxH(x, y) + (x, y) eq(32)

Where: v_{W H}(x, y) corresponds to video data information of a relatively noise free input image (this accomplished by processing the input image and removing noise components as will be discussed below). 9WXH (^X _> ) corresponds to video data information of a comfort noise component of the input image. b_WxH(x, y) corresponds to video data information of an undesired noise component of the input image.

[0072] A system for noise removal, Noise Removal 110, is configured to receive and process the input image z_WxH(x, y) 101 by extracting or removing all what may be considered as noise to obtain the relatively noise free component v_WxH(x, y) 111.

[0073] An inverting system S 1 15 is used to produce Output 1 16 which is the inverse of the relatively noise free component v_WxH(x, y) 111. Using an adder system S 120, the input image z_{W H}(x, y) 101 is added to Output 116 (the inverse of v_{W H} ). Thus, in effect, the input image z_WxH (x, y) 101 is subtracted from the inverse of v_WxH (x, y) 111 to obtain video data information values corresponding to the extracted noise - both of the comfort noise component and the undesired noise component. The adder system produces Output 121 which is modeled mathematically in equation (33) as:

Output 121 = z_WxH(x, y) - v_WxH(x, y) = g_WxH(x, y) + b_WxH(x, y) eq(33

Effectively, the Output 121 of the adder system S 120 includes both of the desired comfort noise or film grain g_{W H}{x, y) and the unwanted noise b_WxH(x, y).

[0074] There are various algorithms which can be employed for the good fine grained comfort noise separation, e.g. a scalar block or using a randomized grain density increase method. A Film Grain Extraction 140 receives the Output 121 and removes the unwanted noise component b_{W H}(x, y) and produces an Output 141 that effectively equals or corresponds to the comfort noise component of the input image, and is modeled equation (34) below.

Output 141 = g_WxH(x, y) eq(34)

[0075] The Output 141 corresponding to the desired comfort noise or film grain g_{W H}(x, y) is provided to an N x N Zero Padding 150 system, where zero padding are added to g_WxH (^x _> y) matrix and produces an Output 151 corresponding to the comfort noise at a target resolution, g_NWxNH(x, y), for example as described above the size of the film grain matrix is increased to a target resolution having an NW pixels in width, and NH pixels in height. Various processing means for padding and scaling maybe used to generate output video data information (Output 151) corresponding to the comfort noise at a desired target resolution as modeled in equation (35) below.

Output 151 = g_NWxNH(x, y) eq (35)

[0076] A Grain Density 160 system receives at its input Output 151 which is the comfort noise padded and processed to the target resolution, and generate an increased density comfort noise at the target or desired resolution Output 161 based on and using

(S x_Ntfi y))- Thus, effectively Output 161 corresponds to video data information of processed comfort noise or film grain matrix at the desired output resolution, modeled in equation (36) below.

Output 161 = G_{NW NH}{x, y) eq (36)

[0077] For example, using a method based on applying non-overlapping random coordinate shifts, as described above, on the original sparse grain image Output 151, the Grain Density 160 system produces an Output 161 which corresponds to a scaled-up comfort noise or film grain matrix G_NWxNH(x, y) using the zero-padded film grain matrix 9_NWxNH(x, y)-

[0078] The relatively noise free v_WxH is received and processed to the target resolution using an T N x N Up-Scalar 130 system to produce a scaled up Noise Free Output 131 that can be modeled as shown in equation (37).

Output 131 = V_NWxNH(x, y) eq(37)

[0079] An adder system S 170, receives and adds Output 161 to Output 13 land produces processed Output 171 modeled in equation (38) below.

Output 171 = Z_NWxNH(x, y) eq(3S)

Output 171 corresponds to a processed digital image that is a scaled-up version of the input digital image or a portion of the input digital image. The Output 171 corresponds to a resolution with a width of NW and a height of NH in pixels. The Output 171 is used to transmit or stream the video data information of each pixel of the Output 171 to be displayed using an electronic display device or a region of an electronic display device. [0080] Therefore, the Digital Video Processing System 100 as described above and shown in Fig. 1 provide the capability to process an input digital image having a low resolution, z_{W H}(x, y), that includes a noise free component v_{W H}(x, y), an undesired noise component b_WxH(x, y) to be removed, and a desired comfort noise component g_WxH (x, y) to be kept in order to maintain a natural look at high resolution.

Input 101 = z_{W H} x, y) = v_{W H} x, y) + g_{W H} x, y) + b_{W H} x, y) eq(39)

The Output 171 of the adder system S 170 can thus be modeled in equation (40) below:

Output 171 = Z, NWXNH (x, y)— G_NWxNH(x, y) + V_r NWXNH O, y) eq(40

[0081] The Digital Video Processing System 100 provides the capability to process an input image z_WxH(x, y) and produces an output having a natural look of comfort noise. In accordance with one embodiment, the Digital Video Processing System 100 comprises a system to measure the flatness of the image. In the following paragraphs, a method for implementing the system for measuring the flatness is described in accordance with one embodiment. In addition, the Noise Removal 1 10 block may include such system or other systems to implement the noise removal function from the input digital image.

[0082] A system to measure the flatness of an area of a digital image comprises the following steps:

[0083] Step 1: Estimate gradient of the image. In order to remove film grain where it is most noticeable we compute a gradient estimate and the following approximation for a flat area versus an edge or details of an area:

Flat areas: V/(x, y) « 0 eq(41

Edge or detail areas: V/(x, y) » 0 eq(42

There are many algorithms for estimating the derivatives in the horizontal and vertical direction. Here we used a simple Gaussian Sobel difference operator, which is commonly known, but also one can use other methods with varying accuracy. [0084] Step 2: Threshold low gradient values (Other methods would not require selection of threshold value). We choose a threshold of T = 5 for this exam le.

[0085] Step 3: Calculate gradient norm

[0086] Step 4: Calculate a non-linear function g(\\ V/(x, y) ||) of the gradient norm to enhance flatness measure and get an alpha image. We experimented with both

<7i(liv/0,y)ll) -,y « 0.5 eq(46

i + y||v/(x,y)||²

<7₂(llV/0,y)||) = exp{-i||V/(x,y)||²/a²}, « 3 eq(47

[0087] We obtained similar results for the gradient map flatness measures above and used

for this example using various digital images.

For 0≤#(||V/(x,y)||)≤ l eq(48)

A strong indicator of a flat area corresponds to a value of approximately one:

<7(llv/0,y)ll) « l eq(49)

A strong indicator of an edge or detail area corresponds to a value of approximately zero:

<7(llv/0,y)ll)«o eqr(50)

[0088] We vary a noise reduction smoothing parameter 5 according to the formula shown in equation (51). The smoothing parameter may change for each pixel location (x,y) depending on g(IIVf(x,y)ll). 5 = s x, y) = s_min + g(\\Vf(x, y \\ [s_max - s_min] eq(51)

[0089] The smoothing level s(x, y) varies linearly between a minimum and maximum smoothing value. For this example, we chose the following parameters:

For #(ll /0, y) ll) « 1

s(x, y) = s_max « 3.5 eq(52)

For #(||V/(x, y)||) « 0 s(x, y) = s_max « 1.0 eq(53 [0090] Flatness Alpha Channel Separation of Image, and the alpha channel for flatness is:

/0, y)#(l|V/0, y) ||) eq (54)

The lightest regions receive a high level of smoothing. The large flat areas are where noise is most visible.

[0091] The complementary alpha image channel is computed by the equation:

/(x, y)(l - #(||V/(x, y) ||)) eq(SS)

The bright areas of the complementary alpha image of the digital image z_WxH(x, y) correspond to edges and detail areas of various objects within the digital image that receive almost no smoothing. Noise is most visible where a large flat area exists within the digital image.

[0092] Furthermore, the Digital Video Processing System 100 comprises one or more systems that provides a local neighborhood coordinate analysis which will be described in the following paragraphs. While discussing image processing algorithms, confusion often arises when discussing the processing of and/or computing a new sample value. It is natural to think of an image as a B bit precision pixel level value z = z(x, y) in the range z G {-2^s"1, -2^s"1 + 1, -2, -1,0,1,2, - , 2^s"1 - 1} eq S6)

[0093] The spatial image locations or coordinates (x, y) are locations on some discrete rectangular grid, £l_WxH of width W and height H : i = {(*.y) ^: x = 0,1,2,- ,W - l,y = 0,1,2,- ,H - 1} eq(57

[0094] For digital images the y coordinate axis is thought of as a line number increasing from the top to the bottom of a frame. Let N = 2r + 1 where r = 1,2, ··· be the radius of a local square neighborhood or analysis window centered at the origin 0 = (°) defined as

B_r(0,0) ={s = Qe2xl: u,v = 0,±l,±2,— ,±r} eq(58

[0095] Define the metric d_∞ : M^2xl→ [0,∞) by the rule d_∞(w,v) = maxflw-L— v^, \w₂— v₂\] Vw, v G M^2xl eq(59)

[0096] This metric is topologically equivalent to the standard Euclidean distance d₂: M^2X1→ [O.co) eq(60)

d₂(w,v) = (Iwi v₁1² + |w₂ - v₂\^{2 1/2} Vw,v G »2X1 eq(61

[0097] Define a parameterization or scanning order s(n of the coordinates in the analysis window B_r(0,0): s:{l,2,-,N²}→ B_r(0,0), s(n) = n = 1,2,-, N^< eq(62)

[0098] For square neighborhoods of radius r = 1,2, ··· the infinity metric is more convenient since it matches the square block geometry. The infinity metric is defined as

= max{|x₀— u(ri)\, \y₀— v(n \] eq(63

[0099] We will choose column wise scanning order for the parameterization s(n) because MATLAB and many other linear algebra oriented high level languages use the LAPACK library, which was originally written in FORTRAN. Unlike C, FORTRAN compilers store a rectangular matrix as stacked columns rather than rows. We will define linear operators for N x N matrices by first mapping them to N² x 1 column vectors in the isomorphic Euclidean vector spaceM^w2xl. Since the N x N data matrices are already stored as stacked columns the conversion ofaWxJV matrix does not require data in memory to be re-shuffled.

[0100] For the block neighborhood is defined as:

/U

fl_r(0,0) = {s = ( )εΖχΖ: u,v = 0,±l, ±2, ··· , ±r}

= {SGZXZ: llpo -s(n)||_∞ < r) eq(64

[0101] The column wise scanning parameterization s^« = (¾)'^{n = 1}'²'-' eq(65 given by the following equations:

u(n) = [(n - 1) % N]— r eq(69)

[0102] As a simple concrete example for radius r = 2 of a neighborhood with respect to the infinity metric can be visualized as a matrix of coordinate vectors.

For example, in the example of N = 2r + 1 = 5, the parameterization of a local coordinate system centered at the origin is: fl₂(0,0) = { s(n) = I I : 1 < n≤ N²

v{n) (_°₂)

¾(0,0) eq(71

r₂ ²) (^"2¹) (2) © [0103] The local coordinate system can be translated to a coordinate of interest p₀ in the image rectangle by translation of the local coordinate block: B₂ (0,0) to B₂ (x_0< Yo) , f°^r example using the Euclidean Coordinate Space 200, as shown in Fig. 2.

B₂ (x₀, y₀ =

T ^■ Ω Ω WXH eq(73

T_po (s(ⁿ» = Po + s(n) = Q + (^) ; n = 1,2, ··· , N² _eq(74)

[0104] In accordance with one embodiment, an example Film Grain Extraction 140 system including a spatial distance penalty weight mask will be described in the following paragraphs. Since objects in natural images don't favor any particular orientation we use a circular two dimensional Gaussian weight mask for weight pixel level values as the

Euclidean coordinate distance on the plane increases from the center pixel location (x, y) of the analysis coordinate block as modeled in equation (75) below.

'X + u

y) : u, v = 0, ±1, ±2, ··· , +r eq(75 ^y + v.

[0105] As an example and in accordance with one embodiment:

Set: r = 3 eq {76)

Thus, the block size is:

N = 2r + 1 = 7 eq{77)

The Gaussian weight mask is defined as: w_s(u, v) = w_s(x + u, y + v) = exp j—— [u² + i7²]/s²j eq(7S) [0106] In step 3 above, we gave a flatness adaptive smoothing parameter formula

5 = s x, y) = s_min + g(\\Vf(x, y \\ [s_max - s_min] eq(79

'min and eq(S0)

[0107] The smoothing parameter 5 controls the radius of the Gaussian function w_s(u, v) = exp j—— [u² + i7²]/s² j eq(81)

[0108] For radial distance from the center:

r(x + u, y + v) = TJU² + v²≥ V s ; w_s(u, v) « 0 eq(82)

Pixel level values at that radial distance are set to zero.

[0109] Step 1: Calculate a more robust estimation of flatness g_{x y} near (x, y)

First, set 5 = s_min = 1 and calculate the weight mask using for example an 8 to 1 accuracy, other bit accuracies may be used.

At 8 bits the weight mask with r = 3 is:

eqr(83)

[0110] The maximum weight at the center is 256 which should correspond to no spatial distance penalty for the center location. The normalization constant for the above matrix is:

c(x, y) = 256 eq(S4)

9x,y g(\\Vf(x + u, y + v \\ _Wl(u, v eq(85) c(x, y)

u=—r v=-r

[0111] Step 2: Calculate local window image data value estimate f_{x y} for a pixel at center (x, y) using the same radial distance penalty mask as for the robust flatness estimate at (x, y). The pixel level values are weighted or penalized according to the distance as well. Y Y g(\\Vf(x + u,y + v)\\)f(x + u,y + v) eq(86 c(x,y)

[0112] Step 3: Calculate local window image data flat model deviation residual block r x + u, y + v) = fix + u, y + v)— f_x eq(87

[0113] Step 4: Calculate local window image data flat model deviation penalty weight mask and multiply by previous radial distance penalty mask. This time the values for minimum and maximum pixel deviation are: t_min = 1.5 eq(88) tmax = 5.0 eqr(89) r(x + u, y + v) = f{x + u, y + v)— f_xy eq (90) w₂(u,v) = exp j—— [r(x + u,y + v)²]/t²| eq(91) w(u, v) = w₂(u, v)w₁(u, v) eq(92)

The model penalty mask h_{t t} u, v values can be characterized as follows: Consider pixel level values in the block deviate from the local flat model f x + u, y + v) = f_{x y} by significantly more than the allowed tolerance of +t_min. We could be centered at flat region on one side of an edge.

[0114] Suppose (x— 1, y— 1) is located on the other side of an edge relative to the center of the block(x,y). The pixel level value is provided by:

/(z-l.y-l) eq(93)

The pixel model deviation becomes: r(x - l,y) = f{x - l.y) - f_xy = 100, f(x,y) = 5 eq(94) [0115] Spatially the coordinate (x— 1, y) is very close to (x, y) so the spatial weight would be very high. In this case using the computed spatial weight matrix the spatial weight for fix— 1, y) is presented in equation (95) below.

Μ¾ (-1,0) = ^ * 0.6 eq(95)

[0116] The spatially weighted contribution of the pixel would be as shown in equation (96) below.

0-6 ^■ (f_xy + 100) = 0.6^■ 105 = 63 eq(96)

[0117] The actual value is f x, y) = 5 which gets a spatial weight of 1.0 since it is at the window center. Edge blurring would be severe if we only used the spatial penalty weight mask. The situation is remedied by the second flat model deviation weight mask w₂ (u, v).

w₂ (-l,0) = exp j- - [100²]/1.5²j « 0 eq(98)

Consequently, the weight for the outlier pixel is the combined model and the spatial penalty mask: w(-l,0) = w₂ (-l,0) w^-l.O) = 0.6 x 0 = 0 eq(99)

Thus insuring that the outlier pixel contributes nothing to the final estimate and edge blurring is avoided. The following paragraphs provides additional disclosure of digital image processing systems in accordance with various embodiments described above. A person skilled in the art will appreciate that various embodiments may be used in combinations or as a standalone systems to achieve a desired outcome. Furthermore, one or more embodiments may be used in conjunction or combined with currently existing digital image processing systems to produce an enhanced digital image processing systems without departing from the spirit of each of the described embodiments.

[0118] The digital representation of the image can greatly vary in quality depending on the sampling of the original image. Each sample of the original image represents a small portion of the overall image. Therefore, more samples are required in order to have a better or more accurate representation of the original image. A pixel represents one sample of the original image. Normally, an image is sampled using a two-dimensional grid having a width, X, and a height, Y, that are specified in unit of pixel, where the digital image resolution corresponds to X time Y, and each pixel corresponds to the smallest single component of the original image. For example, a first camera with a resolution of 640 x 480 would have 640 pixels in the horizontal direction and 480 pixels in the vertical direction. The digital image resolution, total number of pixels, is 307,200 pixels. Higher resolution requires more pixels to be generated when capturing an image, and the closer the digital image produced is to the original image. Hence, a second digital camera with a resolution of 1280 x 960 would have a total number of pixels of 1,228,800 pixels or four times the resolution of the first camera.

[0119] Each pixel of a digital image corresponds to data information that is expressed as a number of bits that is used to describe each pixel ( or sample) of the original image. This data information is normally expressed as number of bits per pixel (bpp). A broader range of distinct colors can be represented with a higher number of bits per pixel. Nowadays, There are many different formats that are in use to capture and/or display color information, e.g. the RGB. For example, a 24-bit color model uses 8 bits to represent red, 8 bits to represent blue and 8 bits to represent green. Under this model, each of these three colors possesses a 2⁸ or 256 levels. Therefore, they can be combined (256 x 256 x 256) to give a possible 16,777,216 colors.

[0120] A video camera captures a scene for a specific duration of time, and produces many sequential digital images. Normally, each digital image is referred to as a frame, having a frame size defined as number of horizontal pixels x number of vertical pixels. A frame rate is also specified that represents the number of frames being captured per second. In addition, a scanning system is identified as progressive or interlaced to indicate how the video frames are generated and thus how they should be processed and displayed so that the original scene is faithfully reproduced when these digital images are played back in sequence, e.g. using an electronic display panel or a digital television set .

[0121] In order to reproduce the original scene timing, each digital image, or frame within the scene, must be reproduced and displayed in a given amount of time. Hence, the time required to process and display one pixel is limited and finite. Electronic display devices resolution is specified, in a similar way as explained above for a digital camera, as having X by Y pixels. Again, the higher the resolution of the electronic display device is, the better the image that is being reproduced. As the electronic display panel technology advances to an ever-higher resolution, a bigger challenge to the device electronics is to be able to process data information for each pixel within an ever-smaller amount of time.

[0122] The processing demands on electronic circuits for High-Definition television (HD TV) , e.g. 1,920 pixels wide and 1,080 pixels high, is much greater than a Standard- Definition television (SD TV), , e.g. 720 pixels wide and 480 pixels high. The next generation of digital TVs and projectors, recently developed, will be able to display four times the high definition resolution of current HD TV sets. This Quad-HD set is capable of displaying 3,840 pixels wide and 2, 160 pixels high. This presents a big challenge to the processing circuitry, where each pixel must be processed and faithfully reproduced regardless of the resolution of the input image resolution. Various Quad-HD electronic display devices may be able to receive image content from standardized interfaces such as HDMI or

DisplayPort and/or other lower resolution connectors for HD such as 1,920 x 1,080 or lower yet such as SD TV.

[0123] In general, various processing techniques can be used to process or scale digital images from one specified resolution to another specified resolution. Electronic display device such as Quad-HD using four electronic display panels, which simply can be referred to as monitors or electronic display devices, can display images using a resolution of 3840 x 2160 pixels, or 3840 pixels in width and 2160 pixels in height, for a total of 8,294,400 pixels. Each region of the four regions can display a HD digital image. A high definition digital image may be specified using 1920 pixels in width and 1080 pixels in height, or in other words a resolution of 1,920 x 1,080 for a total of 2,073,600 pixels. As you can see The Quad- HD monitor is capable of displaying four times the number of pixels that define the digital image using four regions each of which is capable of displaying a full 1,920 x 1,080 pixels. Various embodiments provide the capability to process a low resolution digital image by splitting the image at its native resolution to two or more low resolution regions, process and scale each of the low resolution regions to produce and output a scaled and processed region that matches the desired region of the electronic display device. For example, a high resolution electronic display device receives and directs each of the scaled and processed regions to be displayed using a corresponding region of the electronic display device.

Therefore, the overall displayed image resolution is the combination of the scaled-up resolution of each of the processed regions, e.g. 3840 x 2160. [0124] For example, a Quad HD electronic display panel comprises four identical display regions each with a predefined resolution, any of the display regions may be specified using a different resolution and a different width and height for each region as desired. Therefore, each display region can be driven using independently processed digital images in accordance with the requirement of the display region. In accordance with one embodiment, the output of such digital image processing systems is programmable to provide compatibility with various video formats, mixing with other digital image input, or the various standards to transmit the video data information of the processed images to each region of the electronic display device.

[0125] In accordance with one embodiment, various types of electronic display devices or monitors with multiple defined regions may be coupled to any of the Digital Image

Processing Systems, described in this disclosure, to receive and to display the processed video data information produced by output modules of the Digital Image Processing System, each module output video data information that matches a desired resolution of a display region within the electronic display device. Furthermore, an electronic display device or monitor may comprise an arbitrary number of display regions as may be specified by M rows and N columns where the total number of regions can be represented by M rows times N columns. Hence, in accordance with one embodiment, there are M x N output modules coupled to the electronic display device using a data path which may comprise a single dedicated channel for each of the display regions of the electronic display panel.

Alternatively, a common channel may be used where processed video data information is tagged with the destination address of any one of the regions of the electronic display device.

[0126] An example Digital Image 300 is shown in Fig. 3. In accordance with one embodiment, the Digital Image 300 comprises, as an example, a group of 6x8 pixels representing the Digital Image 300 or a portion of the Digital Image 300 to be processed. Each pixel is symbolically represented using a blackened circle and comprises video data information corresponding to the pixel's color, brightness, and/or other characteristic that will be used to process or display the pixel. An electronic display device is used to display the processed video data information corresponding to each pixel. A letter "P" within each circle is used to denote a pixel.

[0127] An example of a neighborhood block of pixels Quadrangle 310 comprising a group of 3x3 pixels is shown in Fig. 3. Quadrangle 310 may comprise any other group size, for example a group of 5x5 pixels. The letter "A" corresponds to a Center Pixel 311 that is substantially at the center of the Quadrangle 310. The letter "F" is used to denote a Face Pixel 312 that is directly above, below, to the right, or to the left of the Center Pixel 311 "A" of the Quadrangle 310. The letter "C" is used to denote a Corner Pixel 313 that is located substantially in a diagonal direction from the Center Pixel 311 "A" of the Quadrangle 310.

[0128] The following paragraphs describe various embodiments using the neighborhood pixel block for the processing of video data information of an input image. Throughout the specification, a reference to the word "image" should also be interpreted as a reference to a digital image, video image, a portion of a given digital image, a region of a digital image, a portion of a given video image, a region of a video image, or any other combinations or means to represent an image using data information. Various topologies and implementation techniques may be used for the digital image processing systems being described that may include one or more embodiments. Various techniques can also be used to receive the digital image from a memory or a storage device, or the digital image processing system may receive in a streaming fashion the video data information of a digital image (or a portion of a digital image or a region of a digital image).

[0129] Furthermore, throughout the specification, a reference to "video data information" may comprise the complete video data information corresponding to a pixel or a portion of the complete video data information. For example using 24-bit RGB, a reference to video data information may correspond to all 24-bit, or just an 8-bit portion, a 16-bit portion, or a portion with any bit size that matches a specific implementation of a Digital Image

Processing System. Another example, a reference to "video data information" using color and luminance format, may be a reference to any one of the color and luminance video data information or both.

[0130] A Digital Image Processing System 400, as shown in Fig. 4, is configured to receive video data information of at least two digital images. For example, Digital Image 401 and Digital Image 402, as shown in Fig. 4, correspond to an example of portions of the two digital images. The Digital Image Processing System 400 may retrieve the video data information by issuing commands to a storage system. In accordance with one embodiment, the video data information of Digital Image 401 and Digital Image 402 may be streamed to the input of the Digital Image Processing System 400 sequentially or in parallel using multiple data paths. Furthermore, each data path is capable of transporting the video data information of Digital Image 401 and Digital Image 402 one pixel at a time, a group of pixels at a time, a line or a region at a time. Streaming video data information may also be accomplished in accordance with available techniques or standards, for example HDMI.

[0131] In accordance with one embodiment, the Digital Image Processing System 400 receives a first digital image Digital Image 401 and a second digital image Digital Image 402 sequentially, i.e. all of the video data information of Digital Image 401 is received followed by all or a portion of the video data information of Digital Image 402, and the Digital Image Processing System 400 appropriately stores the video data information locally to be retrieved or accessed for the processing of the video data information of a first pixel from the first image using at least video data information of multiple pixels in the neighborhood of the first pixel, and at least the video data information of one pixel from the second image.

[0132] In accordance with one embodiment, the Digital Image Processing System 400 retrieves a portion of the video data information of a second image Digital Image 402 while processing a portion of the video data information of a first image Digital Image 401 being streamed to the input of the Digital Image Processing System 400. Various combinations and techniques may be used to provide, retrieve, or stream the video data information necessary to be processed by the Digital Image Processing System 400. In accordance with one embodiment, the Digital Image 401 comprising video data information for PixelS_(X;y) corresponding to a first point in time, e.g. at time = (t-z), and the Digital Image 402 comprising video data information for PixelS_(X;y) corresponding to a second point in time, e.g. at time = (t). In this example, Digital Image 402 occurs at a later point in time than does Digital Image 401. In accordance with one embodiment, the Digital Image 401 occurs at a later point in time than does Digital Image 402.

[0133] In accordance with one embodiment, the Digital Image Processing System 400 determines a first location of a first quadrangle, Quadrangle 405, within the first image Digital Image 401. Digital Image Processing System 400 keeps track of the current pixel or group of pixels being processed using the sequential nature video format of pixel location within each line of pixels. Quadrangle 405 corresponds to a first part of a first portion of the first image Digital Image 401 and comprises a first number of lines and each line comprises a first number of pixels. In this example, the first number of line is three and the first number of pixels is three. Quadrangle 405 comprises a first pixel "A" that is located substantially at the center of Quadrangle 405. [0134] In accordance with one embodiment, the Digital Image Processing System 400 is configured or programmed to process the first pixel "A" (or a first group of pixels, where the first group of pixels is located at the center of Quadrangle 405) using at least two of four neighboring pixels denoted by the letter "F" that are directly above, below, to the right and to the left of the first pixel "A", and at least two of four neighboring pixels denoted by the letter "C" that are located in a diagonal direction from the first pixel "A". The video data information corresponding to the first pixel "A" or the first group of pixels is shown in Fig. 4 as 0-415.

[0135] In accordance with one embodiment, the Digital Image Processing System 400 determines at least three groups of pixels, each pixel of the three groups of pixels is located within the Quadrangle 405, wherein (i) a first group of the at least three groups of pixels comprising at least one pixel, namely the first pixel "A", (ii) a second group of the at least three groups of pixels comprising at least two pixels, example of pixels in this group are shown in Fig. 4 and are denoted by letter "F", each pixel of the second group is located along one of four directions directly above, below, to the right, and to the left of the first pixel "A" or the first group, and (iii) a third group of the at least three groups of pixels comprising at least two pixels, example of pixels in this group are shown in Fig. 4 and are denoted by letter "C", each pixel of the third group is located along one of four diagonal directions with respect to the first group, wherein the first group of pixels includes the first pixel.

[0136] In accordance with one embodiment, the Digital Image Processing System 400 uses the video data information corresponding to the first pixel "A" (or the first group of pixels), 0-415, to generate a first video data information value 0-426 for the first pixel "A" by choosing a preprogrammed or predetermined first multiplier value ¾ and multiplying the video data information value (A) of the first pixel "A" time the first multiplier:

First video data information eq(100)

Similarly, if the first group comprises more than one pixel, then the first video data information value for the first group is generated using the sum of the multiplication values of the first multiplier value times the video data information value of each pixel of the first group:

First video data information eq(101) [0137] The Digital Image Processing System 400 uses the received video data information corresponding to the second group of pixels, 0-416, to generate a second video data information, 0-427, for the second group of pixels denoted by the letter "F" in Quadrangle 405 by choosing a preprogrammed or predetermined second multiplier value, K₂, and multiplying the video data information value of each pixel of the second group times the second multiplier and summing the results:

Second video data information eq(102)

[0138] The Digital Image Processing System 400 uses the received video data information corresponding to the third group of pixels, 0-417, to generate a third video data information, 0-428, for the third group of pixels denoted by the letter "C" in Quadrangle 405 by choosing a preprogrammed or predetermined third multiplier value, K₃, and multiplying the video data information value of each pixel of the third group times the third multiplier and summing the results:

Third video data information eq(103)

[0139] In accordance with one embodiment Digital Image Processing System 400, utilize a subsystem block, having multiplier functions and an adder function, namely M421, M422, and M423, to generate the first, second, and third video data information values (0-426, O- 427, 0-428) respectively. Various implementations may be used including software or hardware to perform the multiplication or addition functions or to accomplish the generation of the first, second and third video data information substantially using similar relationship as indicated above. Various techniques in hardware implementation may also be used to implement the multiplication function or the adder function in accordance with a desired performance level of the Digital Image Processing System 400.

[0140] The Digital Image Processing System 400 generates a first mask value, 0-438, corresponding to the first pixel "A" or the first group by using an adder function block S435 to sum the first video data information value 0-426, second video data information value O- 427, and third video data information value 0-428.

[0141] In accordance with one embodiment, the Digital Image Processing System 400 generates a fourth video data information value 0-418 for a second pixel "A" within Quadrangle 406. The second pixel location within the second video image Digital Image 402 corresponds substantially to the first pixel location within the first video image Digital Image 401. For example, if the first pixel "A" location is at pixel number 100 of line number 150 within the first video image Digital Image 401, then the second pixel "A" location is located at pixel number 100 of line number 150 within the second video image Digital Image 402, or possibly in the neighborhood of pixel number 100 of line number 150 within the second video image Digital Image 402. Similarly, the location of the Quadrangle 406 within the second video image Digital Image 402 corresponds substantially to the location of

Quadrangle 405 within the first video image Digital Image 401.

[0142] In accordance with one embodiment, the Digital Image Processing System 400 generates a fourth video data information value 0-418 for a fourth group of pixels located within Quadrangle 406 within Digital Image 402. The fourth group of pixels comprising the second pixel "A", as defined in the previous paragraph, and each pixel of the fourth group is located within a first part of the received first portion of the second image Digital Image 402. The second pixel location within the second video image Digital Image 402 corresponds substantially to the first pixel location within the first video image Digital Image 401, as described above. Similarly, the location of the Quadrangle 406 within the second video image Digital Image 402 corresponds substantially to the location of Quadrangle 405 within the first video image Digital Image 401.

[0143] In accordance with one embodiment, the Digital Image Processing System 400 generates a fifth video data information value 0-488 by performing the following steps. Determining a second mask multiplier value 1-486 that is used as a first input to a multiplier function block M485. The second input of the multiplier M485 receives a second mask value 0-478 corresponding to the first pixel and is generated using an adder function block S475. The inverse of the fourth video data information 0-471 is generated by inverting the fourth video data information 0-418 using an inverse function block Inv-470. The adder S475 generates the second mask value 0-478 by summing the inverse of the fourth video data information 0-471 and a seventh video data information 0-468. The seventh video data information 0-468 corresponding to the first pixel and is derived based on the first mask value as will be described in the following paragraphs.

[0144] In accordance with one embodiment, the Digital Image Processing System 400 generates a sixth video data information value 0-498 using the sum of the fourth video data information value 0-418 and the fifth video data information value 0-488 which is the output of the multiplier M485. The sixth video data information value 0-498 corresponding to the first pixel and representing video data information of a first processed pixel, Temporal Filtered Output Pixel (A) 499, of a first processed image Digital Image 491.

[0145] In accordance with one embodiment, the Digital Image Processing System 400 outputs the video data information of the first processed pixel 0-498 of the first processed image Digital Image 491 to an electronic display device 490. The electronic display device 490 displays the first processed pixel using a pixel display location that corresponds to the first pixel "A" location within the first image Digital Image 401. The pixel display location may be located within a predefined display region of an electronic display device comprising at least two display regions.

[0146] In accordance with one embodiment, the Digital Image Processing System 400 generates the seventh video data information value 0-468 by performing the following steps. Determining a third mask multiplier value 1-456 that is used as a first input to a multiplier function block M455. The second input of the multiplier M455 receives a third mask value 0-448 corresponding to the first pixel and is generated using an adder function block S445. An eighth video data information 0-41 1 corresponding to the inverse of the first pixel "A" video data information is generated by inverting the received video data information 0-415 using an inverse function block Inv 410. The adder S445 generates the third mask value O- 448 by summing the eighth video data information 0-411 and the first mask value 0-438. The multiplier M455 generates a ninth video data information 0-458 at its output corresponding to the multiplication of the third mask value 0-448 times the third mask multiplier 1-456. The seventh video data information value 0-468 is generated using an adder function block S465 to sum the ninth video data information 0-458 and the received video data information 0-415.

[0147] In accordance with one embodiment, the Digital Image Processing System 400 further determines a first location of a second quadrangle, not shown in Fig. 4, by shifting the first location of the Quadrangle 405 by at least one pixel within Digital Image 401. The second quadrangle comprises (i) pixels corresponding to a second part of the first portion of the first video image Digital Image 401, (ii) a third pixel that is located substantially at the center of the second quadrangle, and (iii) at least one pixel of the Quadrangle 405. [0148] In accordance with one embodiment, the Digital Image Processing System 400 further determines a fifth group of pixels, the fifth group comprising at least one pixel including a fourth pixel, each pixel of the fifth group is located within a second part of the first portion of the second video image Digital Image 402, the fourth pixel is located substantially at the center of the fifth group. The fourth pixel location within the second video image Digital Image 402 corresponds substantially to the third pixel location within the first video image Digital Image 401.

[0149] In accordance with one embodiment, the Digital Image Processing System 400 further generates a tenth video data information value corresponding to the third pixel by repeating the processing steps, as described above, appropriately using the video data information of the pixels of the second quadrangle and the fifth group of pixels instead of the first quadrangle and the fourth group, respectively. This is similar to a pipeline processing of the video data information of each pixel of each line of each digital image that is processed by the Digital Image Processing System 400. Therefore, the Digital Image Processing System 400 generates one processed pixel after another for each of the input pixels. The tenth video data information value representing a video data information of a second processed pixel of the first processed image, and the Digital Image Processing System 400 outputs the video data information of the second processed pixel of the first processed image to the electronic display device 490.

[0150] In accordance with one embodiment, the Digital Image Processing System 400 determines value of each of the first mask multiplier, the second mask multiplier, the first, second, and third multipliers using at least one of a fixed value, a programmable value, a computed value, on-the-fly generated value, and a dynamically computed value. In addition, dynamically computed value comprises manipulation of one or more values in response to variables that change value based on at least one characteristic of the video data information of the input image Digital Image 401, and/or based on at least one user defined value.

[0151] In accordance with one embodiment, the Digital Image Processing System 400 determines the three groups of pixels within Quadrangle 405 using the relative location of each pixel of the at least three groups of pixels with respect to the first pixel "A". [0152] In accordance with one embodiment, the Digital Image Processing System 400 comprises a programmable register to determine the size of Quadrangle 405. The minimum size Quadrangle 405 corresponds to at least one line, each line having at least three pixels.

[0153] A Digital Image Processing System 500, as shown in Fig. 5, is a simplified version of the Digital Image Processing System 400, the input stages are similar to those described above for Fig. 4. The processing of the video data information of the Digital Image 501 using at least video data information of one pixel from Digital Image 502 is slightly different and uses one less multiplier function block. The Digital Image Processing System 500 is configured to receive video data information of at least two digital images. For example Digital Image 501 and Digital Image 502, as shown, correspond to an example of portions of the two digital images. The Digital Image Processing System 500 may retrieve the video data information by issuing commands to a storage system. In accordance with one embodiment, the video data information of Digital Image 501 and Digital Image 502 may be streamed to the input of the Digital Image Processing System 500 sequentially or in parallel using multiple data paths. Furthermore, each data path is capable of transporting the video data information of Digital Image 501 and Digital Image 502 one pixel at a time, a group of pixels at a time, a line or a region at a time. Streaming video data information may also be accomplished in accordance with available techniques or standards, for example HDMI.

[0154] In accordance with one embodiment, the Digital Image Processing System 500 receives a first digital image Digital Image 501 and a second digital image Digital Image 502 sequentially, i.e. all of the video data information of Digital Image 501 is received followed by all or a portion of the video data information of Digital Image 502, and the Digital Image Processing System 500 appropriately stores the video data information locally to be retrieved or accessed for the processing of the video data information of a first pixel from the first image using at least video data information of multiple pixels in the neighborhood of the first pixel, and at least the video data information of one pixel from the second image.

[0155] In accordance with one embodiment, the Digital Image Processing System 500 retrieves a portion of the video data information of a second image Digital Image 502 while processing a portion of the video data information of a first image Digital Image 501 being streamed to the input of the Digital Image Processing System 500. Various combinations and techniques may be used to provide, retrieve, or stream the video data information necessary to be processed by the Digital Image Processing System 500. In accordance with one embodiment, the Digital Image 501 comprising video data information for PixelS_(X;y) corresponding to a first point in time, e.g. at time = (t-z), and the Digital Image 502 comprising video data information for PixelS_(X;y) corresponding to a second point in time, e.g. at time = (t). In this example, Digital Image 502 occurs at a later point in time than does Digital Image 401. In accordance with one embodiment, the Digital Image 501 occurs at a later point in time than does Digital Image 502.

[0156] In accordance with one embodiment, the Digital Image Processing System 500 determines a first location of a first quadrangle, Quadrangle 505, within the first image Digital Image 501. Digital Image Processing System 500 keeps track of the current pixel or group of pixels being processed using the sequential nature video format of pixel location within each line of pixels. Quadrangle 505 corresponds to a first part of a first portion of the first image Digital Image 501 and comprises a first number of lines and each line comprises a first number of pixels. In this example, the first number of line is three and the first number of pixels is three. Quadrangle 505 comprises a first pixel "A" that is located substantially at the center of Quadrangle 505.

[0157] In accordance with one embodiment, the Digital Image Processing System 500 is configured or programmed to process the first pixel "A" (or a first group of pixels, where the first group of pixels is located at the center of Quadrangle 505) using at least two of four neighboring pixels denoted by the letter "F" that are directly above, below, to the right and to the left of the first pixel "A", and at least two of four neighboring pixels denoted by the letter "C" that are located in a diagonal direction from the first pixel "A". The video data information corresponding to the first pixel "A" or the first group of pixels is shown in Fig. 4 as 0-515.

[0158] In accordance with one embodiment, the Digital Image Processing System 500 determines at least three groups of pixels, each pixel of the three groups of pixels is located within the Quadrangle 505, wherein (i) a first group of the at least three groups of pixels comprising at least one pixel, namely the first pixel "A", (ii) a second group of the at least three groups of pixels comprising at least two pixels, example of pixels in this group are shown in Fig. 5 and are denoted by letter "F", each pixel of the second group is located along one of four directions directly above, below, to the right, and to the left of the first pixel "A" or the first group, and (iii) a third group of the at least three groups of pixels comprising at least two pixels, example of pixels in this group are shown in Fig. 5 and are denoted by letter "C", each pixel of the third group is located along one of four diagonal directions with respect to the first group, wherein the first group of pixels includes the first pixel.

[0159] In accordance with one embodiment, the Digital Image Processing System 500 uses the video data information corresponding to the first pixel "A" (or the first group of pixels), 0-515, to generate a first video data information value 0-526 for the first pixel "A" by choosing a preprogrammed or predetermined first multiplier value ¾ and multiplying the video data information value (A) of the first pixel "A" time the first multiplier:

First video data information = K^A eq(\04)

First video data information = Ki_*∑ A eq(\05)

[0160] The Digital Image Processing System 500 uses the received video data information corresponding to the second group of pixels, 0-516, to generate a second video data information, 0-527, for the second group of pixels denoted by the letter "F" in Quadrangle 505 by choosing a preprogrammed or predetermined second multiplier value, K₂, and multiplying the video data information value of each pixel of the second group times the second multiplier and summing the results:

Second video data information = K_2*∑ F eq(\06)

[0161] The Digital Image Processing System 500 uses the received video data information corresponding to the third group of pixels, 0-517, to generate a third video data information, 0-528, for the third group of pixels denoted by the letter "C" in Quadrangle 505 by choosing a preprogrammed or predetermined third multiplier value, K₃, and multiplying the video data information value of each pixel of the third group times the third multiplier and summing the results:

Third video data information eq(l07) [0162] In accordance with one embodiment Digital Image Processing System 500, utilize a subsystem block, having multiplier functions and an adder function, namely M521, M522, and M523, to generate the first, second, and third video data information values (0-526, O- 527, 0-528) respectively. Various implementations may be used including software or hardware to perform the multiplication or addition functions or to accomplish the generation of the first, second and third video data information substantially using similar relationship as indicated above. Various techniques in hardware implementation may also be used to implement the multiplication function or the adder function in accordance with a desired performance level of the Digital Image Processing System 500.

[0163] The Digital Image Processing System 500 generates a first mask value, 0-538, corresponding to the first pixel "A" or the first group by using an adder function block S535 to sum the first video data information value 0-526, second video data information value O- 527, and third video data information value 0-528.

[0164] In accordance with one embodiment, the Digital Image Processing System 500 generates a fourth video data information value 0-518 for a second pixel "A" within

Quadrangle 506. The second pixel location within the second video image Digital Image 502 corresponds substantially to the first pixel location within the first video image Digital Image 401. For example, if the first pixel "A" location is at pixel number 100 of line number 150 within the first video image Digital Image 501, then the second pixel "A" location is located at pixel number 100 of line number 150 within the second video image Digital Image 502, or possibly in the neighborhood of pixel number 100 of line number 150 within the second video image Digital Image 502. Similarly, the location of the Quadrangle 506 within the second video image Digital Image 502 corresponds substantially to the location of

Quadrangle 505 within the first video image Digital Image 501.

[0165] In accordance with one embodiment, the Digital Image Processing System 500 generates a fourth video data information value 0-518 for a fourth group of pixels located within Quadrangle 506 within Digital Image 502. The fourth group of pixels comprising the second pixel "A", as defined in the previous paragraph, and each pixel of the fourth group is located within a first part of the received first portion of the second image Digital Image 502. The second pixel location within the second video image Digital Image 502 corresponds substantially to the first pixel location within the first video image Digital Image 501, as described above. Similarly, the location of the Quadrangle 506 within the second video image Digital Image 502 corresponds substantially to the location of Quadrangle 505 within the first video image Digital Image 501.

[0166] In accordance with one embodiment, the Digital Image Processing System 500 generates a fifth video data information value 0-558 by performing the following steps. Determining a second mask multiplier value 1-586 that is used as a first input to a multiplier function block M585. The second input of the multiplier M585 receives a second mask value 0-548 corresponding to the first pixel and is generated using an adder function block S545. The inverse of the fourth video data information 0-511 is generated by inverting the fourth video data information 0-518 using an inverse function block Inv-510. The adder S545 generates the second mask value 0-548 by summing the inverse of the fourth video data information 0-51 1 and the first mask value 0-538.

[0167] In accordance with one embodiment, the Digital Image Processing System 500 generates a sixth video data information value 0-598 using the sum of the fourth video data information value 0-518 and the fifth video data information value 0-558 which is the output of the multiplier M485. The sixth video data information value 0-598 corresponding to the first pixel and representing video data information of a first processed pixel, Temporal Filtered Output Pixel (A) 599, of a first processed image Digital Image 591.

[0168] In accordance with one embodiment, the Digital Image Processing System 500 outputs the video data information of the first processed pixel 0-598 of the first processed image Digital Image 591 to an electronic display device 590. The electronic display device 590 displays the first processed pixel using a pixel display location that corresponds to the first pixel "A" location within the first image Digital Image 501. The pixel display location may be located within a predefined display region of an electronic display device comprising at least two display regions.

[0169] In accordance with one embodiment, the Digital Image Processing System 500 further determines a first location of a second quadrangle, not shown in Fig. 5, by shifting the first location of the Quadrangle 505 by at least one pixel within Digital Image 501. The second quadrangle comprises (i) pixels corresponding to a second part of the first portion of the first video image Digital Image 501, (ii) a third pixel that is located substantially at the center of the second quadrangle, and (iii) at least one pixel of the Quadrangle 505. [0170] In accordance with one embodiment, the Digital Image Processing System 500 further determines a fifth group of pixels, the fifth group comprising at least one pixel including a fourth pixel, each pixel of the fifth group is located within a second part of the first portion of the second video image Digital Image 502, the fourth pixel is located substantially at the center of the fifth group. The fourth pixel location within the second video image Digital Image 502 corresponds substantially to the third pixel location within the first video image Digital Image 501.

[0171] In accordance with one embodiment, the Digital Image Processing System 500 further generates a seventh video data information value corresponding to the third pixel by repeating the processing steps, as described above, appropriately using the video data information of the pixels of the second quadrangle and the fifth group of pixels instead of the first quadrangle and the fourth group, respectively. This is similar to a pipeline processing of the video data information of each pixel of each line of each digital image that is processed by the Digital Image Processing System 500. Therefore, the Digital Image Processing System 500 generates one processed pixel after another for each of the input pixels. The seventh video data information value representing a video data information of a second processed pixel of the first processed image, and the Digital Image Processing System 500 outputs the video data information of the second processed pixel of the first processed image to the electronic display device 590.

[0172] In accordance with one embodiment, the Digital Image Processing System 500 determines the value of each of the first mask multiplier, the first, second, and third multipliers using at least one of a fixed value, a programmable value, a computed value, on- the-fly generated value, and a dynamically computed value. In addition, dynamically computed value comprises manipulation of one or more values in response to variables that change value based on at least one characteristic of the video data information of the input image Digital Image 401, and/or based on at least one user defined value.

[0173] In accordance with one embodiment, the Digital Image Processing System 500 determines the three groups of pixels within Quadrangle 505 using the relative location of each pixel of the at least three groups of pixels with respect to the first pixel "A". In accordance with one embodiment, the Digital Image Processing System 500 comprises a programmable register to determine the size of Quadrangle 505. The minimum size

Quadrangle 505 corresponds to at least one line, each line having at least three pixels. [0174] The embodiments as described in this disclosure are used interchangeably to develop a certain and unique Digital Image Processing System that meets a widely varying input digital image processing requirements to produce processed images with varying desired effects and or system performance, including various techniques to process and split an input digital image, processing one or more regions within each split input image by generating video data information for one or more output pixels using video data information of a group of pixels within the one or more regions within each split input image, and display the processed split input image using a display region of a multi-region electronic display. Furthermore, various embodiments provide the ability to customize the Digital Image Processing System to meet an ever-changing electronic display panels' size and specification such as having programmable registers, or non-volatile memory to store certain operating parameters which can be updated, e.g. over a network, or directly in order to change the performance of the Digital Image Processing System.

Claims

WE CLAIM:

1. A method for processing video data information of a respective pixel of an input image, the method comprising:

receiving video data information of a first portion of the input image, the first portion including a first group of pixels, the first group of pixels including the respective pixel;

estimating at least one value of a first indicator parameter using video data information of the first group of pixels;

generating a noise reduction parameter value using the at least one value of the first indicator parameter;

generating video data information of an output pixel by processing video data information of each pixel of the first group of pixels using one or more of the noise reduction parameter value and the at least one value of the first indicator parameter, wherein the output pixel corresponds to a processed version of the respective pixel of the input image; and

storing the video data information of the output pixel using an output buffer.

2. The method of Claim 1, wherein the at least one value of the first indicator parameter indicates that the first group of pixels corresponds to any one or more of a flat area, a detail area and an edge.

3. The method of Claim 1, wherein the video data information of the first group of pixels used to estimate the at least one value of the first indicator parameter includes only one of a color and luminance format of the video data information of each pixel within the first group of pixels.

4. The method of Claim 1, further comprising:

storing video data information of each pixel of the first portion of the input image using an input buffer.

5. The method of Claim 1, wherein the first group of pixels includes at least three lines of pixels, and each line of pixels having at least three pixels.

6. The method of Claim 1, wherein the estimating at least one value of a first indicator parameter using video data information of the first group of pixels comprises:

determining one or more distinctive features of the first group of pixels by using a non-linear function to process video data information of each pixel of the first group of pixels, wherein the one or more distinctive features of the first group of pixels correspond to any one of a flat area, a detail area and an edge; and

estimating at least one value of a first indicator parameter corresponding to a distinctive feature of the first group of pixels.

7. The method of Claim 1, wherein the generating a noise reduction parameter value using the at least one value of the first indicator parameter comprises:

generating a noise reduction parameter value based on (i) the at least one value of the first indicator parameter, and (ii) the location of the respective pixel within the first group of pixels.

8. The method of Claim 1, wherein the generating a noise reduction parameter value using the at least one value of the first indicator parameter comprises:

generating a noise reduction parameter value for each pixel of the first group of pixels based on (i) the at least one value of the first indicator parameter, and (ii) the location of each pixel of the first group of pixels within the first group of pixels.

9. The method of Claim 1, wherein (i) the noise reduction parameter value is to effect a high level of smoothing for the respective pixel when the first group of pixels correspond to a flat area, and (ii) the noise reduction parameter value is to effect a low level of smoothing for the respective pixel when the first group of pixels correspond to a detail area or an edge.

10. A system for processing video data information of a pixel of an input image, the system comprising:

a first processing subsystem operable (i) to receive video data information of a first portion of the input image, (ii) to generate at least one value of a first indicator parameter using video data information of a first group of pixels, and (iii) to generate a noise reduction parameter value using the at least one value of the first indicator parameter, wherein the first portion includes the first group of pixels, and the first group of pixels include the pixel; and a second processing subsystem operable (i) to generate video data information of an output pixel in response to the noise reduction parameter value and video data information of the first group of pixels, the second processing system stores the video data information of the output pixel into an output buffer, wherein the output pixel corresponds to a processed version of the pixel of the input image.

1 1. The system of Claim 10, wherein the first group of pixels includes at least three lines of pixels and each line of pixels includes at least three pixels.

12. The system of Claim 10, wherein the at least one value of the first indicator parameter indicates that the first group of pixels corresponds to any one or more of a flat area, a detail area and an edge.

13. The system of Claim 10, wherein the first processing subsystem storing video data information of each pixel of the first portion of the input image into an input buffer.

14. The system of Claim 10, wherein the first processing subsystem is operable to process video data information of the first group of pixels, and to generate the at least one value of the first indicator parameter based at least in part on the processed video data information of the first group of pixels indicating that the video data information of the first group of pixels correspond to one or more of a flat area, a detail area, and an edge.

15. The system of Claim 14, wherein the first processing subsystem is operable to generate the at least one value of the first indicator parameter using a non-linear function to process video data information of each pixel of the first group of pixels.

16. The system of Claim 10, wherein the first processing subsystem is operable to generate the noise reduction parameter value based at least in part on (i) the at least one value of the first indicator parameter, and (ii) the location of the pixel within the first group of pixels.

17. The system of Claim 10, wherein the first processing subsystem is operable to generate a noise reduction parameter value for each pixel of the first group of pixels based at least in part on (i) the at least one value of the first indicator parameter, and (ii) the location of each pixel of the first group of pixels within the first group of pixels.

18. The system of Claim 10, wherein the second processing subsystem is operable to generate video data information of the output pixel by applying a high level of smoothing to the video data information of the pixel based at least in part (i) on the noise reduction parameter value, and (ii) in response to the video data information of the first group of pixels corresponding to a flat area.

19. The system of Claim 10, wherein the second processing subsystem is operable to generate video data information of the output pixel by applying a low level of smoothing to the video data information of the pixel based at least in part (i) on the noise reduction parameter value, and (ii) in response to the video data information of the first group of pixels corresponding to a detail area or an edge.