WO2009094618A1

WO2009094618A1 - System and method for cross-talk correction

Info

Publication number: WO2009094618A1
Application number: PCT/US2009/031955
Authority: WO
Inventors: Keigo Hirakawa
Original assignee: President And Fellows Of Harvard College
Priority date: 2008-01-25
Filing date: 2009-01-26
Publication date: 2009-07-30

Abstract

According to one aspect provided are systems and methods for image processing to improve color fidelity of captured image data. One embodiment includes the acts of removing effects of attenuation resulting from characteristics of an image capture device, wherein removing effects of attenuation resulting from characteristics of the image capture device further includes determining the attenuation for at least part of a Fourier domain representation of a captured image, correcting the attenuation in the at least part of a Fourier domain representation of the captured image, and storing a corrected representation of an original image. According to another aspect, improvements in color fidelity can be performed with or after demosaicking. In some embodiments, improvements in color fidelity can be achieved using a stored demosaicking kernel to simplify calculations of correction values. According to another aspect, desaturation artifacts are characterized as the attenuation of the modulated signals by the frequency response of the cross-talk kernel at the carrier frequencies. In one embodiment, the method to correct the cross-talk contaminations may be derived from the interplay between cross-talk and demosaicking- which can be reduced to a pixel-wise matrix operation. In another embodiment, the brilliance of the color (saturation) is restored after an inverse matrix operation.

Description

SYSTEM AND METHOD FOR CROSS-TALK CORRECTION

BACKGROUND

Field of Invention

The present invention relates to image processing, and more particularly to correcting cross-talk attenuation in spectral sampled color filter array data.

Discussion of Related Art In digital imaging applications, data are typically obtained via a spatial sub sampling procedure implemented as a color filter array (CFA), a physical construction whereby each pixel location measures only a single color. The most well known of these schemes involve the canonical primary colors of light: red, green, and blue. In particular, the Bayer pattern CFA attempts to complement humans' spatial color sensitivity via a quincunx sampling of the green component that is twice as dense as that of red and blue. Specifically, let

index pixel locations and define

to be the corresponding color triple. Then the Bayer pattern CFA image, ^

is given by:

where

Some existing alternatives to the Bayer pattern include Fuji's octagonal sampling, Sony's four color sampling, Polaroid's striped sampling, CMY sampling, hexagonal sampling, and irregular patterns. The Bayer pattern is illustrated in Fig. 32A, four color sampling is illustrated in Fig. 32B, striped sampling is illustrated in Fig. 32C, and hexagonal sampling is illustrated in Fig. 32D. The terms "Demosaicing" or "demosaicking" refer to the inverse problem of reconstructing a spatially undersampled vector field whose components correspond to particular colors. Use of the Bayer sampling pattern is ubiquitous in today's still and video digital cameras; it can be fairly said to dominate the market. Consequently, much attention has been given to the problem of demosaicing color images acquired under the Bayer pattern sampling scheme.

While a number of methods have been proposed for reconstruction of subsampled data patterns, it is well known that the optimal solution (in the sense of minimal norm) to this ill- posed inverse problem, corresponding to bandlimited interpolation of each spatially subsampled color channel separately, produces perceptually significant artifacts. The perceptual artifacts produced using demosaicing algorithms on sub-sampled data obtained using a Bayer-pattern is caused both by the spatial undersampling inherent in the Bayer pattern and the observation that values of the color triple exhibit significant correlation, particularly at high spatial frequencies: such content often signifies the presence of edges, whereas low- frequency information contributes to distinctly perceived color content. As such, most demosaicing algorithms described in the literature attempt to make use (either implicitly or explicitly) of this correlation structure in the spatial frequency domain. Most work in this area focuses on the interplay between the acquisitions stages and subsequent digital processing. Assuming a full-color image (i.e., a full set of color triples), and consequently, a key reconstruction task of demosaicing is first necessary.

Cost effectiveness has helped secure the dominance of single-sensor solution over the alternative color image acquisition configurations in consumer electronics. The popularity of this design paradigm is also marked by the sustained progress in the research of color filter array (CFA): discussed in B.E. Bayer, "Color imaging array," US Patent 3,971,065, 1976, S. Yamanaka, "Solid state color camera," US Patent 4,054,906, 1977, R. Lukac and K.N.

Plataniotis, "Color filter arrays: Design and performance analysis," IEEE Transactions on Consumer Electronics, vol. 51, pp. 1260-1267, 2005, K. Hirakawa and PJ. Wolfe, "Spatio- spectral color filter array design for enhanced image fidelity," in Proceedings of the IEEE International Conference on Image Processing, 2007, vol. 2, pp. 81-84, Extended version submitted to IEEE Transactions on Image Processing, October 2007, M. Paramar and SJ. Reeves, "A perceptually based design methodology for color filter arrays," in Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, 2004, vol. 3, pp. 473-476. Also as evidenced by research in interpolation discussed in D. Alleysson, S. Susstrunk, and J. Herault, "Linear demosaicing inspired by the human visual system," IEEE Transactions on Image Processing, vol. 14, pp. 439-449, 2005, E. Dubois, "Filter design for adaptive frequency-domain Bayer demosaicking," in Proceedings of the IEEE International Conference on Image Processing, 2006, pp. 2705-2708, B.K. Guntruk, J. Glotzbach, Y. Altunbasak, R.W. Schafer, and R.M. Mersereau, "Demosaicking: Color filter array interpolation in single chip digital cameras," IEEE Signal Processing Magazine, vol. 22, pp. 44-54, 2005, K. Hirakawa and T.W. Parks, "Adaptive homogeneity-directed demosaicing algorithm," IEEE Transactions on Image Processing, vol. 14, pp. 360-369, 2005, R. Lukac and K.N. Plataniotis, "Single- sensor camera image processing," in Color Image Processing: Methods and Applications, R. Lukac and K.N. Plataniotis, Eds., pp. 363-392. CRC Press, Boca Raton, FL, 2006.

As well as other research directed to other aspects of similar models: denoising research as discussed in K. Hirakawa and T.W. Parks, "Joint demosaicing and denoising," IEEE Transactions on Image Processing, vol. 15, pp. 2146-2157, 2006, compression research of N. Zhang and X. Wu, "Lossless compression of color mosaic images," IEEE Transactions on Image Processing, vol. 15, no. 6, pp. 1379-1388, June 2006, and research on quantum efficiency of pixel sensors in T. Kijima, H. Nakamura, J. Compton, and J. Hamilton, "Image sensor with improved light sensitivity," US Patent Application 2007/0177236.

In recent years, however, the trend in the industry is to increase the image resolution as a response— at least in part— to the heightened consumer awareness and expectation of digital color image contents. The challenges posed by densely populating pixel sensors are not limited to the manufacturability of integrated circuits alone, as there are implications for image processing as well— great care must be taken in analyzing the image sensor because acquisition typically represents the first step in the digital camera pipeline and largely determines the image quality achievable by subsequent processing schemes.

A well-known drawback to shrinking the geometry of the pixel sensors is that it increases noise. Studies suggest that the number of photons encountered during a spatio- temporal integration is a Poisson process— the "noise" variance scales linearly with the intensity of the light, integration time, and the surface area of the sensor and lens. Consequently, the signal-to-noise ratio is poor when the surface of the sensor chip is subdivided into smaller pixels.

Photon and electron leakages caused by optical refraction and minority carriers are the sources of an interaction between neighboring pixels— the so-called "cross-talk" phenomenon- — and it is another artifact of the decreased distance between neighboring pixels that complicates the reconstruction of the desired image signal. "Desaturation" of color is the most noticeable aftermath of cross-talk in single-sensor color image acquisition devices as a result of combining neighboring pixel measurements representing different portions of the visible spectrum. Conventional research in cross-talk has so far focused on the physical aspects - Agranov et al., "Crosstalk and microlens study in a color cmos image sensor," IEEE Trans. Electron Devices, vol. 50, no. 1, 2003, Rhodes et al. "Cmos imager technology shrinks and image performance," in IEEE Microelectronics and Electron Devices, 2004, 1. Shcherback et al., "A comprehensive cmos aps crosstalk study: Photoresponse model, technology, and design trends," IEEE Trans. Electron Devices, vol. 51, no. 21, 2004. In sum, cross-talk is an important problem that still deserves attention.

Let x : Z² — > R³ , x = [x_r,x_g ,x_b]^τ be the color image of interest, where x(n) is the

RGB tristimulus value at pixel location n e Z². Once the light enters into the camera through the lens, it is focused onto the surface of sensor array. Ideally, value digitally recorded by the pixel sensor y : Z² -> R is proportional to the intensity of the light that penetrates through the color filter at the corresponding pixel location: y(n) = c(n)^T x(n), where c : Z² — > R³ , c = [c_r , c_g , c_b ]^τ is the color filter array. This is effectively a spatio-spectral subsampling procedure implemented as a color filter array, whereby each pixel location measures only a portion of the visible spectrum selected from amongst a chosen "color partition" of that spectrum.

Two predominant causes for cross-talk contamination are optical diffraction and minority carriers. Optical diffraction occurs when a high incidence angle of the light entering the substrate causes the photons to stray away from the center of the pixel; microlenses can help to reduce this risk [Agranov et al., "Crosstalk and microlens study in a color cmos image sensor"]. The diffusion is stochastic but mostly linear with respect to the intensity of the light.

The incident angle is typically wider for the pixel sensors far from the lens axis, and thus the light that reaches photosensitive material can be modeled as spatially-variant convolution:

where /z₀ : Z² xZ² — > R₊ is the location-dependent impulse response. The precise modeling of /z_o(n,m) as a function of sensor geometry is an active area of research involving sophisticated simulation [Rhodes et al., "Cmos imager technology shrinks and image performance", and Shcherback et al., "A comprehensive cmos aps crosstalk study: Photoresponse model, technology, and design trends"].

Minority carrier deteriorates the signal when electrons stray from the target after the charge is collected [Agranov et al., "Crosstalk and microlens study in a color cmos image sensor"]. This carrier is typically deterministic and mostly linear with respect to the signal strength, and it can be modeled as spatially-invariant convolution:

where It₁ : Z² — > R₊ is the convolution kernel. Motivated by physics, the characteristics of this diffusion process are crudely modeled as It₁ (m) <χ _e ^~nτm-^niκ _{^ w}here K is the diffusion constant, τ is the sample interval, and Il - Il is the Euclidean distance. [Shcherback et al., "A comprehensive cmos aps crosstalk study: Photoresponse model, technology, and design trends"] Combining the effects of optical diffusion and minority carrier, the overall acquisition process is:

where h : Z² xZ² — > R₊ represents the combined effect of the convolution filters h₀ and It₁ . h can be known a priori as it is not data-dependent and it can be parameterized via calibration experiments.

SUMMARY

Spatio-spectral sampling theory provides insight into a method for the analysis of the mechanism underlying the cross-talk contaminations. This analysis models the attenuation of color information not only based on the sensor and leakage characteristics but also as a function of the color image content and the demosaicking method. Provided is a simple and effective color correction scheme and a comparison of the sensitivity of various color filter array patterns as characterized by the interplay between aliasing, cross-talk, and demosaicking. Image sensor measurements are subject to degradation caused by the photon and electron leakage— and the color image data acquired via a spatial subsampling procedure implemented as a color filter array is especially vulnerable due to the ambiguation between neighboring pixels that measure different portions of the visible spectrum. This "cross-talk" phenomenon is expected to become more severe as the electronics industry's trend to shrink the device footprint continues, because pixel sensors are more densely packed together. Moreover, the problem of leakage cannot simply be "un-done" by existing de-convolution or image de- blurring methods owing to the complexity of the sensor data. Analysis of the mechanism underlying the cross-talk problem is surprisingly straightforward. Comprehensive analysis provides a simple and effective correction scheme for cross-talk given a choice of color filter array in a digital camera. According to one aspect of the present invention, a computer implemented method for image processing to improve color fidelity of captured image data is provided. The method comprises the acts of removing effects of attenuation resulting from characteristics of an image capture device, wherein the act of removing effects of attenuation resulting from characteristics of the image capture device further comprises determining the attenuation for at least part of a Fourier domain representation of a captured image, correcting the attenuation in the at least part of a Fourier domain representation of the captured image, and storing a corrected representation of an original image. According to one embodiment of the present invention, the act of determining the attenuation further comprises an act of approximating the attenuation of chrominance and luminance signals resulting from cross-talk. According to another embodiment of the invention, the method further comprises an act of reducing computational complexity of correcting the attenuation by reducing computation of cross-talk correction to a matrix inversion. According to another embodiment of the invention, the matrix inversion is performed pixel-wise. According to another embodiment of the invention, the act of determining the attenuation for the at least part of a representation of a captured image further comprises calculating a cross-talk kernel.

According to one embodiment of the present invention, the at least part of a representation of a captured image comprises the cross-talk kernel. According to another embodiment of the invention, the act of determining the attenuation for at least part of a representation of captured image further comprises determining the attenuation based, at least in part, on at least one carrier frequency of at least one signal in a Fourier domain representation of the captured image. According to another embodiment of the invention, the cross-talk kernel is time-invariant. According to another embodiment of the invention, the at least part of a Fourier domain representation comprises at least part of at least one difference image signal and at least one baseband signal. According to another embodiment of the invention, the characteristics of the image capture device further comprise at least one of optical diffraction and minority carrier interference.

According to one embodiment of the invention, the method further comprises an act of displaying a corrected representation of an original image. According to another embodiment of the invention, the method further comprises an act of demosaicking captured image data in conjunction with the act of removing effects of the attenuation resulting from properties of the image capture device. According to another embodiment of the invention, the method further comprises an act of demosaicking captured image data, and wherein the act of removing effects of attenuation resulting from properties of the image capture device occurs after the act of demosaicking. According to another embodiment of the invention, the at least one baseband signal represents the luminance component of the representation of the image and the at least one difference image signal represents the chrominance component of the representation of the image. According to another embodiment of the invention, the method further comprises an act of reconstructing a color image, wherein the act of reconstructing the color image further comprises an act of approximating the attenuation of chrominance and luminance signals resulting from cross-talk.

According to one embodiment of the present invention, the at least part of a Fourier domain representation of a captured image comprises at least part of a Fourier domain representation of color filter array. According to another embodiment of the invention, at least part of a Fourier domain representation of a captured image comprises at least part of color filter array. According to another embodiment of the invention, the act of determining further comprising an act of generating a correction value by approximating the attenuation as spatially- invariant in a Fourier domain representation of the captured image data. According to another embodiment of the invention, the act of removing effects of attenuation resulting from characteristics of the image capture device further comprises an act of accessing a demosaicking kernel generated from demosaicking of the captured image data.

According to one aspect of the present invention, a system for capturing image data is provided. The system comprises a color filter array comprising a plurality of color filters adapted to filter light, a plurality of photosensitive elements, each photosensitive element configured to measure light received through the plurality of color filters and output data values, a processing component coupled to output of the plurality of photosensitive elements and adapted to remove effects of attenuation resulting from measuring the filtered light, wherein the processing component is further adapted to remove effects of attenuation by determining attenuation for at least part of a Fourier domain representation of the captured light, and correcting the attenuation to at least one difference image signal and at least one baseband signal, and a storage component for storing a corrected representation of an original image. According to one embodiment of the present invention, the processing component is further adapted to access a demosaicking kernel generated from demosaicking of the captured image data to determine a correction value. According to another embodiment of the invention, the processing component is further adapted to determine the attenuation by approximating the attenuation of chrominance and luminance signals resulting from cross-talk. According to another embodiment of the invention, the processing component is further adapted to reduce computational complexity of correcting the attenuation by reducing computation of cross -talk correction to a matrix inversion.

According to one aspect of the present invention, a method for correcting attenuation of captured image data resulting from cross-talk contamination of captured image data in association with image demosaicking is provided. The method comprises transforming a representation of a captured image into at least one baseband signal and at least one difference image signal in a Fourier domain, determining attenuation for at least a portion of the transformed image signals in the Fourier domain, correcting at least one attenuated image signal, and storing the at least one corrected image signal. According to one embodiment of the present invention, the act of determining attenuation for at least a portion of the transformed image signals in the Fourier domain further comprises an act of accessing a demosaicking kernel generated from demosaicking of the captured image data. According to another embodiment of the invention, the act of determining the attenuation further comprises an act of approximating the attenuation of chrominance and luminance signals resulting from cross-talk. According to another embodiment of the invention, the method further comprises an act of reducing computational complexity of correcting the at least one attenuated image signal by reducing computation of cross-talk correction to a matrix inversion. According to another embodiment of the invention, the act of determining the attenuation further comprises calculating a cross-talk kernel. According to one aspect of the present invention, a method for performing cross-talk correction is provided. The method comprises the acts of determining a correction kernel from at least a portion of a transformed image signal, wherein the at least a portion of the transformed image signal is at least part of a Fourier domain representation of captured image data, determining attenuation based, at least in part, on a cross-talk kernel, correcting the attenuation of at least one image signal using the correction kernel, and storing the corrected at least one image signal. According to one embodiment of the present invention, the act of determining attenuation further comprises an act of accessing a demosaicking kernel generated from demosaicking of the captured image data. According to another embodiment of the invention, the act of determining the attenuation further comprises an act of approximating the attenuation of chrominance and luminance signals resulting from cross-talk. According to another embodiment of the invention, the method further comprises an act of reducing computational complexity of correcting the attenuation by reducing computation of cross-talk correction to a matrix inversion. According to another embodiment of the invention, the method further comprises an act of generating a corrected representation of an original image from the corrected at least one signal. According to another embodiment of the invention, the method further comprises an act of accounting for the combined effects of optical diffraction and minority carrier interference using the cross-talk kernel. According to one aspect of the present invention, a method for digital image processing is provided. The method comprises capturing image data representing an original image, determining a correction value based, at least in part, on attenuation of captured image data resulting from an image capture device, correcting data values for at least a portion of the captured image data by the correction value, and storing corrected image data. According to one embodiment of the present invention, the act of determining a correction value further comprises an act of accessing a demosaicking kernel generated from demosaicking of the captured image data. According to another embodiment of the invention, the act of determining the attenuation further comprises an act of approximating the attenuation of chrominance and luminance signals resulting from cross-talk. According to another embodiment of the invention, the method further comprises an act of reducing computational complexity of correcting the attenuation by reducing computation of cross-talk correction to a matrix inversion. According to another embodiment of the invention, the act of correcting data values for at least a portion of the captured image data by the correction value is performed in conjunction with an act of demosaicking the captured image data. According to another embodiment of the invention, the act of correcting data values for at least a portion of the captured image data by the correction value includes an act of storing a demosaicking kernel generated from demosaicking of the captured image data. According to another embodiment of the invention, the method further comprises an act of generating a reconstructed image from the corrected image data. According to another embodiment of the invention, the method further comprises an act of generating the correction value by approximating the attenuation as spatially- invariant in a Fourier domain representation of the captured image data.

According to one aspect of the present invention, a system for correcting attenuation of captured image data resulting from cross-talk contamination of captured image data in association with image demosaicking is provided. The system comprises an image capture component for measuring incident light on a plurality of photosensitive elements and outputting data values, a processing component coupled to output of the image capture component and for transforming image data into at least one baseband signal and at least one difference image signal in a Fourier domain, wherein the processing component is further adapted to demosaic the transformed image data, a correcting component adapted to determine attenuation for at least a portion of the transformed image signals in the Fourier domain, wherein the correction component is further adapted to correct the attenuation of at least one image signal, and a storage component for storing the at least one corrected image signal. According to one embodiment of the present invention, the processing component is further adapted to store a demosaicking kernel generated from demosaicking of the captured image data and wherein the correcting component is further adapted to access the demosaicking kernel as part of determining the attenuation. According to another embodiment of the invention, the correcting component is further adapted to determine the attenuation by approximating the attenuation of chrominance and luminance signals resulting from cross-talk. According to another embodiment of the invention, a correcting component is further adapted to reduce computational complexity of correcting the attenuation by reducing computation of cross-talk correction to a matrix inversion.

According to one aspect of the present invention, a system for performing cross-talk correction is provided. The system comprises a correction component adapted to determine a correction kernel from at least a portion of a transformed image signal, wherein the at least a portion of the transformed image signal is at least part of a Fourier domain representation of captured image data, a calculation component adapted to calculate attenuation based, at least in part, on a cross-talk kernel, wherein the calculation component is further adapted to correct at least one image signal using the correction kernel, and a storage component for storing the corrected at least one image signal.

According to one aspect of the present invention, a image capture device is provided. The image capture device comprises a color filter array comprising a plurality of color filters adapted to filter light, a plurality of photosensitive elements, each photosensitive element configured to measure light received through the plurality of color filters and output data values, a processing component adapted to receive image data representing an original image, wherein the processing component is further adapted to boost at least a portion of the captured image data to correct for attenuation of at least a portion of the captured image data, and a storage component adapted to store corrected image data.

According to one aspect of the present invention, a system for performing cross-talk correction is provided. The system comprises a correction component adapted to determine a correction kernel from at least a portion of a transformed image signal, wherein the at least a portion of the transformed image signal is at least part of a Fourier domain representation of color filter array, a calculation component adapted to calculate attenuation based, at least in part, on a Fourier domain representation of color filter array, wherein the calculation component is further adapted to correct at least one image signal using the correction kernel, and a storage component for storing the corrected at least one image signal. Further embodiments include the above described methods implemented as computer readable instructions stored on computer readable medium that instruct a computer to perform the above described methods. According to one embodiment, storage typically includes a computer readable and writeable nonvolatile recording medium in which signals are stored that define a program to be executed by a computer processor or information stored on or in the medium to be processed by the program. The medium may, for example, be a disk, flash memory, CD or DVD-ROM.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings are not intended to be drawn to scale. In the drawings, each identical or nearly identical component that is illustrated in various figures is represented by a like numeral. For purposes of clarity, not every component may be labeled in every drawing. The drawing are presented by way of illustration only and are not intended to be limiting. In the drawings,

Fig. 1 illustrates a Bayer CFA pattern; Fig. 2 illustrates an example image capturing device according to one embodiment of the present invention;

Figs. 3A-3C illustrate examples of the log-magnitude spectra of color channels of a "bike" image; Figs. 4A-4D illustrate examples of the "bike" image for different CFA patterns;

Figs. 5A-5D illustrate examples of the "bike" image after correction;

Fig. 6 illustrates an example process flow for performing cross-talk correction according to some aspects of the invention; Fig. 7 illustrates an example frequency domain representation of color components of an image captured using a CFA in accordance with one example spatio-spectral sampling system;

Fig. 8 illustrates a table indicating color components of example CFAs;

Figs. 9A-E illustrate example representations of CFAs; Figs. lOA-C illustrate sample images that may be captured using a CFA;

Figs. 11A-H illustrate sensor data of a sample image captured using four different CFAs and demosaiced versions of this sensor data;

Figs. 12A-H illustrate sensor data of a sample image captured using four different CFAs and demosaiced versions of this sensor data; Figs. 13A-H illustrate sensor data of a sample image captured using four different

CFAs and demosaiced versions of this sensor data;

Figs. 14A-D illustrate frequency domain representations of a sample image capturing using four difference CFAs;

Fig. 15 illustrates a table indicating evaluations of various demosaiced versions of images capture using four different CFAs;

Fig. 16 illustrates a table indicating an arrangement of output elements;

Fig. 17(a)-(k) illustrate different examples of color filter arrays;

Fig. 18(a)-(k) illustrate the log-magnitude spectra of respective sensor data representing the "lighthouse" image; Fig. 19(a)-(f) illustrate log-magnitude spectra of color channels of the "lighthouse" image showing contrast in bandwidth between color channels and a chrominance/luminance representation;

Fig. 20(a) illustrates an example of frequency domain representation of an image having radially symmetric characteristics acquired under a Bayer pattern; Fig. 21(a)-(h) illustrate examples of CFA patterns and corresponding log-magnitude spectra for a "lighthouse" image;

Fig. 22 illustrates a table indicating evaluations of aliasing of CFA patterns; Fig. 23(a)-(l) illustrate detail of an original image and examples of sensor images captured with examples of CFAs and the corresponding reconstructions;

Fig. 24(a)-(l) illustrate an original image, sensors images examples, and their corresponding reconstructions; Fig. 25(a)-(l) illustrate an original image, sensors images examples, and their corresponding reconstructions;

Fig. 26 illustrates a table evaluating mean-square reconstruction error for examples of CFA patterns;

Fig. 27(a)-(f) illustrate examples of reconstruction of sensor data without denoising methods applied;

Figs. 28(a)-(e) illustrate examples of color filter array design;

Figs. 29 (a)-(f) illustrate examples of the log-magnitude of co-efficients displayed as pixel intensity;

Figs. 30(a)-(d) show examples of stimuli and display of a widely available color image; Fig. 31 illustrates an example of an induced CFA pattern;

Figs. 32A-D illustrates a Bayer pattern, hexagonal pattern, four color pattern, and striped pattern CFA;

Fig. 33 illustrates an example image capturing device according to one example spatio- spectral sampling system; Figs. 34A-D illustrate example frequency domain representations of color components of an image captured using a Bayer pattern CFA;

Figs. 35 A-B illustrate examples of frequency domain representations of images having dominant horizontal or vertical components, respectively;

Figs. 36A-C illustrate frequency domain representations of different CFA patterns; and Fig. 37 illustrates an example process that may be used to determine a CFA.

DETAILED DESCRIPTION

This invention is not limited in its application to the details of construction and the arrangement of components set forth in the following description or illustrated in the drawings. The invention is capable of other embodiments and of being practiced or of being carried out in various ways. Also, the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. The use of "including," "comprising," "having," "containing," "involving," and variations thereof herein, is meant to encompass the items listed thereafter and equivalents thereof as well as additional items.

Various embodiments of the present invention may include one or more cameras or other image capturing devices. In some embodiments, a camera 200, as illustrated in Fig. 2, includes a plurality of light sensitive elements 201. Each light sensitive element is configured to measure a magnitude of light 203 at a location within an image being captured. The measurements of light may later be combined to create a representation of the image being captured in a process referred to as demosaicking. According to one embodiment, a process of cross-talk correction is performed in conjunction with a process of demosaicking. In one embodiment, the plurality of light sensitive elements 201 includes a plurality of photo sensitive capacitors of a charge-coupled device (CCD). In one embodiment, the plurality of light sensitive elements 201 include one or more complementary metal-oxide- semiconductor (CMOS). During image capture, each photosensitive capacitor is exposed to light 203 for a desired period of time, thereby generating an electric charge proportional to a magnitude of the light at a corresponding image location. After the desired period of time, the electric charges of each of the photosensitive capacitors is then be measured to determine the corresponding magnitudes of light at each image location.

In some embodiments, in order to capture a color image, one or more color filters 205 are disposed on one or more of the light sensitive elements 201. In some embodiments, a color filter 205 cuts out or filters electromagnetic radiations of specified wavelengths. In some implementations, a color filter 205 is generated by placing a layer of coloring materials (e.g., ink, dye) of a desired color or colors on at least a portion of a clear substrate. In traditional image capturing applications, the color filters are arranged into a color filter array (CFA) 207. A CFA generally refers to a spatially varying pattern of color filters. The majority of traditional CFA' s have filters arranged in the Bayer pattern, an example of a Bayer pattern is shown in Figure 1.

In some embodiments, an indication of the magnitudes of light measured by each light sensitive element is transmitted to at least one processor 209. In one embodiment in which a plurality of photosensitive capacitors of a CCD are used as light sensitive elements 201, the current in each photosensitive capacitor is measured and converted into a signal that is transmitted from the CCD to the processor 209. In some embodiments, the processor 209 includes a general purpose microprocessor and/or an application specific integrated circuit (ASIC). In some embodiments, the processor includes memory elements (e.g., registers, RAM, ROM) configured to store data (e.g., measured magnitudes of light, processing instructions, demosaicked representations of the original image, cross-talk corrected representations of the original image, etc.). In some embodiments, the processor 209 is part of the image capturing device (e.g., camera 200). In other embodiments, the processor 209 is part of a general purpose computer or other computing device.

In some embodiments, the processor 209 may be coupled to a communication network 211 (e.g., a bus, the Internet, a LAN). In some embodiments, one or more storage components 213, a display component 215, a network interface component (not shown), a user interface component 217, and/or any other desired component may be coupled to the communication network 211 and communicate with the processor 209. In some implementations, the storage components 213 may include nonvolatile storage components (e.g., memory cards, hard drives, ROM) and/or volatile memory (e.g., RAM). In some implementations, the storage components 213 may be used to store mosaicked and/or demosaicked representations of images captured using the light sensitive elements 201, as well as representations of images prior to cross-talk correction, and/or representations of images after cross-talk correction.

In some embodiments, the processor 209 may be configured to perform a plurality of processing functions, such as responding to user input, processing image data from the photosensitive elements 201, and/or controlling the storage and display elements 213, 215. In some embodiments, one or more such processors may be configured to perform demosaicking and/or denoising and/or cross-talk correction functions on image data captured by the light sensitive elements 201.

In some embodiments, the image capturing device (e.g., camera 200) and/or processor 209 may be configured to store or transmit at least one representation of an image (e.g., on an internal, portable, and/or external storage device, to a communication network). In some implementations, the representation may include demosaicked and/or denoised and/or crosstalk corrected representation of the image. In some implementations the representation may include a representation of the magnitudes of light measured by the light sensitive elements 201 in accordance with embodiments of the present invention. In some embodiments, the representation may be stored or transferred in a machine readable format, such as a JPEG or any other electronic file format.

In some embodiments, the image capturing device may include a video camera configured to capture representations of a series of images. In addition to or as an alternative to capturing a representation of a single image, as described above, such a video camera may capture a plurality of representations of a plurality of images over time. The plurality of representations may comprise a video. The video may be stored on a machine readable medium in any format, such as a MPEG or any other electronic file format.

Analysis of Cross-Talk

With the aid of spatio-spectral sampling analysis [K. Hirakawa and PJ. Wolfe, "Spatio- spectral color filter array design for enhanced image fidelity," and Alleysson et al., "Linear demosaicing inspired by the human visual system"] the coding of color information "embedded" in the sensor data y as a function of cross-talk kernel h becomes an issue for analysis. In one embodiment, assume c_r + c_g + c_b = γ for some constant γ (this assumption is true by default for pure-color CFAs). Then,

where the difference images x_a = x_r - x_g and x_β = x_b - x_g can be taken as a proxy for chrominance component of x while x_g is similar to the luminance of x . An advantage to { x_a, x_g , x_β } representation is that x_a and x_β enjoy rapid spectral decay, while x_g embodies image features such as edges and textures. These characteristics are observable in in Figures 3A-3C. Shown in Figures 3A-3C are the log magnitude spectra of the color channels of the "bike" image (Figure 3A illustrates F-χ_g , Figure 3B illustrates F x_a , and Figure 3c illustrates

Spatio-Spectal Sampling Processing and Examples

International Application PCT/US/85946 to Hirakawa, et al. entitled "A NEW SPATIO- SPECTRAL SAMPLING PARADIGM FOR IMAGING AND A NOVEL COLOR FILTER ARRAY DESIGN" and filed on November 29, 2007, describes spatio-spectral analysis of image data and embodiments of CFAs that may be used in conjunction with some aspects of the present invention. Examples of spatio spectal sampling may include one or more cameras or other image capturing devices. In some examples, a camera 3300, as illustrated in Fig. 33, may include a plurality of light sensitive elements 3301. Each light sensitive element may be configured to measure a magnitude of light 3303 at a location within an image being captured. The measurements of light may later be combined to create a representation of the image being captured in a process referred to as demosaicing.

In one example, the plurality of light sensitive elements 3301 may include a plurality of photo sensitive capacitors of a charge-coupled device (CCD). In one example, the plurality of light sensitive elements 3301 may include one or more complementary metal-oxide- semiconductor (CMOS). During image capture, each photosensitive capacitor may be exposed to light 3303 for a desired period of time, thereby generating an electric charge proportional to a magnitude of the light at a corresponding image location. After the desired period of time, the electric charges of each of the photosensitive capacitors may then be measured to determine the corresponding magnitudes of light at each image location.

In some examples, in order to capture a color image, one or more color filters 3305 may be disposed on one or more of the light sensitive elements 3301. In some examples, a color filter 3305 may cut out or filter electromagnetic radiations of specified wavelengths. In some implementations, a color filter 3305 may be generated by placing a layer of coloring materials (e.g., ink, dye) of a desired color or colors on at least a portion of a clear substrate. In traditional image capturing applications, the color filters are arranged into a color filter array (CFA) 3307. A CFA generally refers to a spatially varying pattern of color filters. The majority of traditional CFA' s have filters arranged in the Bayer pattern, as described above. In some examples, an indication of the magnitudes of light measured by each light sensitive element may be transmitted to at least one processor 3309. In one example in which a plurality of photosensitive capacitors of a CCD are used as light sensitive elements 3301, the current in each photosensitive capacitor may be measured and converted into a signal that may be transmitted from the CCD to the processor 3309. In some examples, the processor 3309 may include a general purpose microprocessor and/or an application specific integrated circuit (ASIC). In some examples, the processor may include memory elements (e.g., registers, RAM, ROM) configured to store data (e.g., measured magnitudes of light, processing instructions, demosaiced representations of the original image). In some examples, the processor 3309 may be part of the image capturing device (e.g., camera 3300). In other examples, the processor 3309 may be part of a general purpose computer or other computing device.

In some examples, the processor 3309 may be coupled to a communication network 3311 (e.g., a bus, the Internet, a LAN). In some examples, one or more storage components 3313, a display component 3315, a network interface component (not shown), a user interface component 217, and/or any other desired component may be coupled to the communication network 3311 and communicate with the processor 3309. In some implementations, the storage components 3313 may include nonvolatile storage components (e.g., memory cards, hard drives, ROM) and/or volatile memory (e.g., RAM). In some implementations, the storage components 3313 may be used to store mosaiced and/or demosaiced representations of images captured using the light sensitive elements 3301.

In some examples, the processor 3309 may be configured to perform a plurality of processing functions, such as responding to user input, processing image data from the photosensitive elements 3301, and/or controlling the storage and display elements 3313, 3315. In some examples, one or more such processors may be configured to perform demosaicing and/or denoising functions on image data captured by the light sensitive elements 3301.

In some examples, the image capturing device (e.g., camera 3300) and/or processor 3309 may be configured to store or transmit at least one representation of an image (e.g., on an internal, portable, and/or external storage device, to a communication network). In some implementations, the representation may include demosaiced and/or denoised representation of the image. In some examples the representation may include a representation of the magnitudes of light measured by the light sensitive elements 3301. In some examples, the representation may be stored or transferred in a machine readable format, such as a JPEG or any other electronic file format. In some examples, the image capturing device may include a video camera configured to capture representations of a series of images. In addition to or as an alternative to capturing a representation of a single image, as described above, such a video camera may capture a plurality of representations of a plurality of images over time. The plurality of representations may comprise a video. The video may be stored on a machine readable medium in any format, such as a MPEG or any other electronic file format.

Color Filter Patterns It is recognized that there are a number of limitations associated with existing demosaicing algorithms for Bayer-patterned data and other existing CFA data including:

• Simple interpolation schemes often lead to undesirable (and commercially unacceptable) visual distortions and artifacts; • More sophisticated interpolation schemes are highly nonlinear and complex/costly to implement in hardware; and

• Nonlinear techniques (e.g., directional filtering) further exacerbate the problem of characterizing (and hence removing) the noise associated with the image acquisition process. These complications may be at least partially attributed to the spectral content of Bayer pattern and other traditional CFAs that exhibit a complicated form of aliasing due to sampling, making it difficult or impossible to recover the true image.

In one aspect, it is recognized that despite its widespread use, the Bayer pattern and other traditional CFA patterns impose inherent limitations upon the design of interpolation and denoising methods. A popular interpretation of the Bayer pattern in the recent literature involves decomposing the sensor image into

and the spatially subsampled difference images

and

Because the high frequency contents of r, g, and b may be highly redundant, it is often accepted that a_{M are bandlimited In particular;}

consider the spectrum associated with Bayer-patterned data, as shown in Figs. 34A-D. Figs. 3A-D illustrate a log-magnitude spectra of a typical color image known as "Clown." Fig. 34A illustrates the

component in the frequency domain; Fig. 34B illustrates a difference

signal given by

in the frequency domain; Fig. 34C illustrates a difference signal given

by ' ^' in the frequency domain; and Fig. 34D illustrates the combination of these signals showing the spectral periodization associated with the Bayer pattern described by

_where

(4)

and

indexes pixel locations in the Bayer pattern.

As can be seen from Fig. 34D, the rectangular subsampling lattice induces spectral copies of the difference signals of Figs. 34B and 34C centered about the set of frequencies ^ _eacj_[i indicated by 3401. Note that one spectral copy of the difference

signals not indicated by a numeral, referred to as a baseband, may appear at the origin. The copies of the difference signals and the central signals 3403 interfere with one another, thereby reducing allowable bandwidth of both signals. Hence, by reducing allowable bandwidth, the Bayer spectral periodization "penalizes" the very horizontal and vertical features which are most likely to appear in typical "real- world" images. Such image features may roughly be categorized as predominantly horizontal as is seen in Fig. 35A or vertical as is seen in Fig 35B. ) In these examples, the spectral aliases of

and centered around either the vertical

or horizontal axes, respectively, are non-overlapping with 9 C^ , and therefore introducing nonlinearity (such as directional filtering) may resolve the aliasing problems. However, the design of nonlinear methods may prove nontrivial and may be subject to zippering artifacts. In one aspect, it is recognized that Four-color sampling, which is similar to Bayer pattern sampling, may have even more severe restrictions. For example, let ^_{where e}{«) _{is the fourth color such as} emerald, and ^r = ^T{^ where

^•J^* indicates a 2-dimensional Fourier transform. Assuming that ^ is likewise bandlimited, a Fourier analysis of the sensor image yields spectral interpretation as illustrated in Fig. 36A. While Fig. 36A resembles Fig. 34D, the composition of the spectral copies of ⁰^ Pi ^anc* T at the horizontal and vertical axes are different, and thus it may not allow directional filtering as a viable option.

Analysis of Color Filter Arrays According to another aspect, the spatio- spectral sampling induced by a typical CFA pattern was studied. Model of CFAs as a spatial array of pixel sensors were used. As discussed above, a physical device termed a color filter rests over the photosensitive element at each pixel location; it passes a certain portion of the visible spectrum of light according to its chemical composition. The resultant measurement may be considered as an inner product resulting from a spatiotemporal integration of the incident light over each pixel's physical area and exposure time, taken with respect to the color filter's spectral response. While this spectral response is an intrinsic property of the filter, its perceived color is in fact a function of the environmental ilruminant; however, here we adopt the standard convention and identify the filters typically used in practice by their "colors" as red, green, and blue. Discussed below, our later analysis will rely on convex combinations of these components as "pure-color" building blocks. Other issues such as dye chemistry, quality control, prior art, etc. arise in an industry setting, and may also be taken into account in some examples. Issues associated with "color science" need not be addressed in the analysis of color filter arrays, rather optimization of relevant objective metrics, rather than subjective metrics related to perception, will be reviewed. In one typical acquisition scenario, a regular, repeating CFA pattern comprises a tiling of the image plane formed by the union of interleaved sets of sampling, which in turn may be understood in terms of lattices; the spectral periodicity of the resultant sampled color image is determined by a so-called dual lattice, thereby enabling us to characterize the effect on individual color channels of the spatio- spectral sampling induced by various CFA patterns. By explicitly considering this spatio-spectral representation, analysis quantifies both the fundamental limitations of existing CFA designs as evidenced through their associated demosaicking algorithms, and to explicitly demonstrate the sub-optimality of a class of designs that includes most CFA patterns currently in wide use.

The known Bayer pattern CFA, illustrated in Figure 32 and in an alternative example in Figure 17(a), attempts to complement humans' spatial color sensitivity via a quincunx sampling of the green component that is twice as dense as that of red and blue. Though the Bayer pattern remains the industry standard, a variety of alternative color schemes and geometries have been considered over the years, shown in Figure 17(a)-(f) as pure-color designs and Figure 17(g)-(k) as panchromatic designs. Analyzing the geometric structure of color filter arrays as well as the algebraic structure of the sampling patterns they induce, permits identification of some shortcomings in state of the art color filter array design and demosaicing - examples of the structures are shown in Figure 18(a)-(f), illustrating log- magnitude spectra of respective sensor data representing the "lighthouse" image, corresponding to CFA designs shown in Figure 17(a)-(f). The requisite tools and vocabulary for this task will be provided by the notion of point lattices.

Point Lattices as Sampling Sets: We will employ the notion of lattices to describe the spatial subsampling induced by

CFA patterns. Formally, a uniform lattice

comprises a discrete subgroup of n- dimensional Euclidean space whose quotient is compact. We say a nonsingular matrix M having real entries generates a point lattice Λ_M if Λ_M = Λ / Z" , in which case columns of M are said to form a basis for the lattice. In the engineering literature M is often called a sampling matrix, as it may generate a periodic measurement pattern indexed by ^-tuples of integers precisely according to the lattice Λ_M. We associate with each lattice Λ_M a quantity VOI(Λ_M) := I det(M)l that may generalize the notion of a sampling period and is independent of the lattice basis. Under one aspect of the analysis, we focus on the unit- volume square lattice ^^*- as the setting for our image sensor — though our subsequent results apply equally well to other settings and different implementations, such as octagonal or hexagonal sensor geometries, for example. The associated color sampling patterns may then be represented by less dense lattices, said to be sublattices of ^^{^} if each of their elements also belongs to ^* . Example

sublattices of ^^" include the square sublattice generated by and the quincunx

sublattice generated by

The volume of a sublattice -^f **-^" is consequently integer-valued, and no less than wl(£^" i ------. 1

^^" can be written as a disjoint union of distinct translates of a given sublattice A_M — whereupon such translates can be associated with a red, green, or blue purecolor filter. The set of distinct translates of Λ_M by vectors in Ξ:^J is said to form a set of cosets of Λ_M in 2>', with the corresponding vectors termed coset vectors. The number of distinct coset vectors (including the zero vector) — and hence disjoint translates — of such a sublattice is given by VOI(Λ_M). Note that if we specify a regular, repeating pure-color CFA pattern based on the sublattice ^ΛN ^■-^'■^' ^^", then the number of different colors it admits should not exceed the number of distinct cosets in VOI(Λ_M). TO this end, we call a sampling matrix M admissible if it generates a sublattice with VOI(Λ_M) ≥ 3, and henceforth consider admissible generators.

A Lattice Model for Color Filter Arrays:

Our goal is first to analyze pure-color CFAs comprising disjoint sampling patterns of red, green, and blue. To this end, recall that in a single-sensor camera, a scalar measurement is made at each spatial location leading to an idealized, noise-free model in which the

sensor measurement y(n) can be expressed as the inner product of the true color image triple

and the action of the CFA c(n) :=

as follows:

Recall that by our description, a pure-color _x ^{t {} n ^{\ , fi ■■} _^*,- } _comp_ri_ses a regular, repeating pattern that measures only a single component of the color triple at each spatial location, and hence its elements are represented by the standard basis.

Our example of a model for the geometry of a pure-color CFA will thus be a vector- valued indicator function expressed in the language of lattices. We wish to partition ^ into three structured sampling sets, each of which will be written as a union of selected cosets of a given sublattice ^Λ* ^{^} This notion accounts for the color triple associated with each pixel. To this end, in one example, let M denote an admissible generator for sublattice

^^{l %} ^ ^■ and let Ψ_nΨ_g,Ψ^ represent mutually exclusive subsets of coset vectors associated respectively with the spatial sampling locations of colors red, green, and blue. If -^"can be written as the disjoint union of the three interleaved sampling structures Ψ_r + A_M, Ψ_g + A_M, and Ψ_b + A_M, each one comprising a union of selected cosets as {Ψ + A_MJ ⁼ v Η < ' -l- -W^| _men we call the result a 3-partition of -^- ^~ . Note that, since under this scenario every sensor measures exactly one color, it suffices to specify (M,Ψ_nΨ^), from which we can obtain the coset vectors Ψ_g accordingly and can be used in luminance/chrominance parameterization. Using a 3-partition of -^^" to be one model for pure-color sampling, in contrast to the new panchromatic CFA designs we introduce later. The sampling structure of the purecolor CFA associated with 3-partition (M,Ψ_nΨ^) is then defined pointwise as follows:

in this example we let δ{-) denote the Dirac or Kronecker delta function, as indicated by sampling context. Pure-color CFAs hence perform a spatio-chromatic subsampling that effectively multiplexes red, blue, and green components in the spatial frequency domain — leading to the image recovery task known as demosaicking. However, because the spatio- spectral content of these color channels tends to be correlated at high spatial frequencies, representations that avoid this correlation are often employed in contemporary demosaicking algorithms. In this example we exploit a luminance/chrominance representation common in the literature to simplify our lattice model for pure-color CFAs. Following the measurement model of (A), each pixel sensor measurement may equivalently be represented in terms of a green

channel x_g and difference channels x_a := x_r -x_g and X_β := X_b~x_g: '^ ^{u ! ~~} ^ ^{A;! Λ '} ** '

One feature of this particular representation is that these difference channels enjoy rapid spectral decay and can serve as a proxy for chrominance, whereas the green channel can be taken to represent luminance. As our eventual image recovery task will be to approximate the true color image triple x(n) from acquired sensor data y{n), note that recovering either representation of x(n) is equivalent. Moreover, the representation of (C) allows us to re-cast the pure-color sampling structure of (B) in terms of sampling structures associated with the difference channels xa and xβ.

Fourier Analysis of Pure-Color CFAs:

It is known how to compute the Fourier representation induced by the spatial subsampling of certain CFA patterns, however, we provide for a Fourier representation of all rectangular, periodic, pure-color CFAs in terms of the sublattice A_M associated with a given 3- partition (M,Ψ_r,ψb) of ^ . Owing to their Abelian structure, lattices admit the notion of a Fourier transform as specified by a dual or reciprocal lattice. The spectral periodicity properties of a color image sampled along a lattice A_M are determined by its dual lattice ^^Λ2 (the Poisson summation formula being a simple univariate example), thereby enabling characterization of the effect on individual color channels of the spatio- spectral sampling induced by various CFA patterns.

According to the discrete and periodic nature of repeating pure-color CFA sampling patterns, in one example, the dual lattice -*^■■&* defines a unit cell about the origin in B^ with associated volume ^v copies of which in turn form a tessellation

of the spatial frequency plane. Under our normalization of the Fourier transform, the dual lattice -W = ~^{n Λ 1 1}^ associated with an admissible sampling matrix M will in turn admit ⁱ>^{t ώ}_ ^..-^{■ i;ι iLi} _{as a} sublattice. As a model for sensor geometry is the lattice &^~ , it thus suffices to restrict attention to dual lattice points contained in the unit cell [-π, π) x [-π, π) in the spatial frequency plane.

A result that follows is the characterization of the spatio-spectral properties of images acquired under pure-color CFAs. Let "' -*^' denote the two-dimensional Fourier transform operator applied to x(n), with X(ω) := -^ ^J parameterized by angular frequency ^ ^•■■ !&''. Then from (B2) and (B3), we have that the Fourier transform of sensor data y(n) over dual lattice points contained in the unit cell [-π, π) x [-π, π) is given by:

Thus we see that the lattice structure of the chosen 3-partition induces spectral copies of the difference channels x_a and X_β centered about the set of carrier frequencies ' ^{' M :} I-^"" " - ^{;ι ■ f •} As restrictions of lattices, these sets will always include the origin — corresponding to "baseband" copies of the respective difference channels. In this manner (D) may be interpreted

as specifying a baseband "luminance" signal ^VJU^A _A/.^{) \ <Λ[Λ} _M ^! with the remainder of its terms comprising lower-bandwidth "chrominance" information modulated away from the origin. Recalling the interpretation of VOI(Λ_M) as the maximum number of distinct colors supported by a given CFA, we see that the ratios of color components comprising luminance information depend directly on \Ψ_r\ and IΨ^I, the number of coset vectors associated with difference channels x_a and X_β.

An example of the baseband signal - * ⁵^ ' corresponding to a typical color image is shown in Figure 19(e); the locations of spectral replicates modulated away from the origin are shown in Figure 18 for several 3-partitions corresponding to pure-color CFA patterns in wide use. From these examples, it may be seen that aliasing occurs whenever there is pairwise overlap of the spectral supports of _and

tor

In the absence of aliasing, chrominance information can be successfully "demodulated" and then used to recover X_g(co) from the baseband luminance channel ^f '■"^*' ' via standard filtering techniques; however, for each instance depicted in Figure 18, the placement of chrominance information in the spatial frequency plane is seen to result in aliasing.

Figure 19(a)-(f) shows the log-magnitude spectra of color channels of the "lighthouse" image, showing the contrast in bandwidth between the (x_r, x_g, X_b) representations and a chrominance/luminance representation in terms of (X_a, Xi, X_β). Further inspection confirms each of these patterns is sub-optimal from an aliasing perspective — a notion discussed in more detail below. By making precise the minimal assumptions on color channel bandwidths necessary, the sub-optimality may be further demonstrated in another example:

Assumption 1 (Bounded Bandwidth): Assume the supports of (X_g,X_α,X_β) to be bounded, such that the associated luminance and chrominance channels comprising (D) are contained in balls of radii ^r'^{: c}- '^{τ tlilk}' ' ^■ '- ^<τ' respectively.

Assumption 2 (Total Bandwidth): Assume ^r> + ^?V > ^"• This implies that the physical resolution of one example of an image sensor is such that aliasing may in fact occur, depending on the placement of chrominance information in the Fourier domain. Assumption 3 (Relative Bandwidth): Assume ^r<- ^ ' < ^■ This is consistent with known empirically reported results, and follows the known correlation of color channels.

These assumptions imply that in order to maximize the allowable spectral radii ?v and ^?γ subject to the zero-aliasing constraint required for perfect reconstruction, spectral replicates representing chrominance information should be placed along the perimeter of the unit cell [-π, π) x [-π, π) in the spatial frequency plane. The CFA patterns of Figures 2(d-f) and 2(i-k) may be seen by inspection to violate this condition.

Moreover, it follows from Assumption 3 that the least optimal placement of spectral replicates along this perimeter is about the points (-π, 0) and (0,-π) — as these points minimize the maximum allowable spectral radii. As may be seen in Figures 18(a-c) and 18(g,h,j,k), the popular Bayer pattern and many others are sub-optimal from this perspective. Moreover, by reducing allowable channel bandwidth along the coordinate axes, these patterns are sensitive to the very horizontal and vertical features which frequently dominate typical known images. Sub-optimality of Pure-Color CFAs:

Neither pure-color nor panchromatic CFA designs currently in use are optimal from the perspective of spatial aliasing. No periodic, pure-color CFA design can ever attain optimality — a fact reflected by the failure in practice of simple linear reconstruction methods when used with such patterns. In essence, these patterns determine a lattice that packs luminance and chrominance information into the unit cell [-π, π) x [-π, π). The above spatiospectral analysis confirms — that a sphere-packing strategy is required, rather than the sphere-filling approach of current patterns. Accordingly, in one approach we first exhibit the spatio- spectral requirements necessary to maximize the allowable spectral radii ^y '^{■ d l}" ^' °" , subject to the constraint of zero aliasing. To this end, in one example apply Assumptions 1-3, to a proper sublattice '^lM ''^■- *-' along with a set Ψ of associated coset vectors.

Proposition 1 (Bandwidth Maximization): Amongst all sets {Ψ+Λ_M}, those that maximize

subject to the constraint of zero aliasing take the following form: for every _{such mat we h}ave that

and

In this example, (D) shows that dual lattice points

associated with nonzero weights

represent "carrier frequencies" for chrominance information. The proposition thus specifies that in order to simultaneously maximize the allowable bandwidth of luminance and chrominance, all such carrier frequencies contained in the Fourier-domain unit cell [-π, π)² must be placed maximally far from the origin.

Proof: Consider radii ^r* ^arκ' ^r<: as rays in the Cartesian plane which define balls representing the respective maximal directional bandwidths of luminance and chrominance information, as per Assumption 1. A sole ball representing luminance is centered about the origin, and each chrominance ball is taken to be centered around a candidate lattice point W_e then seek the set of λ yielding

an arrangement that admits, with no intersection of spheres, the maximal

Assumption 3 *^• ' *^: '- ' ^■ in turn implies, for this example, that we need only consider the case of chrominance-luminance aliasing, rather than chrominance-chrominance aliasing. Noting that , with equality if and only if *^'*^{' atκl} '^'•■ are collinear, we

may therefore take ^r£ to be a ray emanating from the origin, and ° to be a collinear ray emanating from any other candidate point λ. For any angle taken by rays \^Ti- i ^im^ !'^■ khe maximum of is attained by Amongst all

members of this set, Assumption 2 *-'^v + ^■'^'-' ^{;> π )} excludes the set of points {(-π, 0), (0,-π)}, and the proposition is proved.

According to another example, the following proposition in turn provides an upper bound on the volume of any sublattice ^{Λ M '"}- ^- satisfying the condition for bandwidth maximization specified by Proposition 1:

Proposition 2 (Volume Limitation): Let (M, Ψ) determine a sampling set {Ψ+Λ_M} taking the bandwidth-maximizing form of Proposition 1. Then VOI(Λ_M) ≤ 2IΨI.

The proof of this proposition follows:

Define , according to our definition of

a pure-color CFA. The bandwidth maximization requirement set forth in Proposition 1 implies a unique Fourier reconstruction in ω = ((D₁, ω₂) as

where Proposition 1 implies that the allowable domain of A in the spatial frequency plane is described by the following four mutually exclusive sets:

Because the set {Ωi Pl [-π, π) } consists of either a point or two disjoint line segments for all i, the resultant inverse Fourier transform simplifies as follows:

where ∫i denotes ∅ in each respective term.

We now enumerate all scenarios concerning

, and show that vol(Λ_M) ≤ 2IΨI in every possible case. First, suppose

i aind

to be non-zero functions. Then it is easy to verify, using

that there exist indices ⁿ

such that

and

without loss of generality, then

Otherwise put, it is shown that:

The 4 x 4 matrix above is rank 3 and its column space is orthogonal to (-1, 1, 1, -1). However, the inner product of (-1, 1, 1, -1) and (1, 0, 0, l)-f0-f3 is non-zero regardless of the values of f0 and f3:

We conclude that the equality above cannot hold, and thus obtain a direct contradiction to the hypothesis that * ^{] Jlk J}'^: are both non-zero functions.

We next consider all remaining scenarios. Suppose first that ^*' is a non-zero function and ^ is zero (or equivalently /- non-zero and /ι zero). Then:

Because /"-K'lt 'V -H-^{l 1} \ut = ^t* ^\ hn -H-I ' ^α h ;- _we conclude that C(Λ) = 1 - c(m, «2 + 1), or equivalently, VOI(Λ_M) = 2IΨI. If instead, we have that/ϊ, /2 are zero and /3 non-zero, (M, Ψ) determines quincunx sampling, whereupon vol(Λ_M) = 2IΨI. Lastly, if J₁, /₂, /₃ are zero, then Λ_M is an integer lattice, and we have that vol(Λ_M) = IΨI. Hence we see that in all possible cases, vol(Λ_M) ≤ 2IΨI.

Together, Propositions 1 and 2 imply the sub-optimality of periodic, pure-color CFA designs with respect to aliasing considerations. Indeed, a surprising result follows from these propositions, that any such design that seeks to maximize the allowable spectral radii of luminance and chrominance cannot simultaneously admit three distinct colors. To illustrate, suppose in one example that a 3-partition (M,Ψ_r,Ψ_b) is designed such that mutually exclusive sampling sets {Ψ_r +Λ_M} and {Ψ/, +Λ_M} both satisfy the conditions of Proposition 1. Then by Proposition 2 we have that vol(Λ_M) ≤ IΨ_rl + IΨ_bl, but it follows from our earlier definition of a 3-partition that vol(Λ_M) = IΨ_rl+IΨ_gl+IΨ_bl. Thus the set of coset vectors Ψg indexing the sampling of the third color component must be empty.

Aliased Sensor Data and Demosaicking

The preceding discussion confirms the sub-optimality of all periodic, pure-color CFA patterns with respect to the metric of spatial aliasing. However, owing to the prevalence of the Bayer pattern in currently manufactured still and video digital cameras, much attention in this field has been given to the problem of demosaicking color images acquired in this manner. As is known, an ideal demosaicking solution should exhibit two main traits: low computational complexity for efficient hardware implementation, and amenability to analysis for accurate color fidelity and noise suppression. For instance, in the absence of further assumptions on the relationships between tristimulus values, the optimal linear reconstruction is indicated by an orthogonal projection onto the space of bandlimited functions, applied separately to each subsampled color channel. However, it is well known that this solution produces unacceptable artifacts, as aliasing prevents perfect reconstruction (see Figures 4(a), (b) and Figure 20(a)). Recalling Figures 17 and 18, this aliasing problem follows from the sub-optimality of purecolor CFAs with respect to aliasing.

As such, most known demosaicing algorithms described in the literature make use (either implicitly or explicitly) of correlation structure in the spatial frequency domain, often in the form of local sparsity or directional filtering. As noted in our earlier discussion, the set of carrier frequencies induced by the Bayer pattern includes (-π, 0) and (0,-π), locations that are particularly susceptible to aliasing by horizontal and vertical edges. Figures 35(a) and 35(b) indicate these scenarios, respectively; it may be seen that in contrast to the radially symmetric baseband spectrum of Figure 20(a), representing the idealized spectral support of the channels (x_g, JCa, *b) °f ^a color image acquired under the Bayer pattern, chrominance-luminance aliasing occurs along one of either the horizontal or vertical axes. However, successful reconstruction can still occur if a non-corrupted copy of this chrominance information is recovered, thereby explaining the popularity of known (nonlinear) directional filtering steps. CFA design can be view as a problem of spatial frequency multiplexing, and the CFA demosaicking problem as one of demultiplexing to recover subcarriers, with spectral aliasing given the interpretation of "cross talk."

As discussed above, it is fair to conclude that existing linear interpolation schemes often lead to undesirable (and commercially unacceptable) distortion and visual artifacts. However, more sophisticated schemes are typically highly nonlinear and can be costly to implement in typical ASIC and DSP hardware environments. Moreover, nonlinear techniques, such as those requiring local edge detection, further exacerbate the problem of characterizing (and hence mitigating) the various sources of noise associated with the image acquisition process. Robustness of the detection variable, sensitivity to noise, and overall model accuracy all affect the quality of reconstruction. As is known, recent works have demonstrated the inadequacies of treating the denoising and interpolation tasks separately and has led to a number of methods designed to treat these problems jointly.

In one aspect, it is recognized that octagonal and hexagonal sampling using Bayer patterns or other traditional CFA patterns may also have similar problems. Motivated by the density of pixel sensors on a CMOS/CCD sensor chip, octagonal and hexagonal sampling have been suggested as improvements to traditional CFAs. Octagonal sampling is often implemented as a forty-five degree rotation of Bayer patterned sampling scheme. While the horizontal and vertical axes in the frequency domain will be less subject to aliasing, we expect similar aliasing problems arising overall, as illustrated in Fig. 36B. Hexagonal sampling, which also densely packs pixel sensors, has the advantage of sampling an equal number of red, green, and blue pixel components that are uniformly distributed over the surface of the sensor chip, but a disadvantage that the sampling rates in the horizontal and vertical directions differ. Its Fourier representation is found in Fig. 36C.

In one aspect, it is recognized that by considering the spectral wavelength sampling requirements as well as the spatial sampling requirements associated with the image acquisition process, we may conceive of a new paradigm for designing color filter array patterns. In some examples, the CFA pattern may be used as means to modulate the image signals in the frequency domain and may thereby aid in preventing the frequency components needed for the reconstruction of the image from aliasing (e.g., overlapping). Examples may offer the potential to significantly reduce hardware complexity in a wide variety of applications, while at the same time improving output color image quality.

According to one aspect, the sub-optimality of periodic, pure-color CFA designs discussed above, yields the optimal periodic designs of CFA patterns as necessarily panchromatic. In one example, assuming a regular, repeating rectangular pattern, and putting aside issues of white-balancing and sensor noise, the analysis discussed above motivates consideration of linear combinations of prototype pure-color filters, rather than restricting the values c_r(n), c_g(n), C_b(n) to the set {0, 1 } implied by pure-color designs. Hence, we let 0 < c_r(n), C_g(n), C_b(n) < 1 indicate the array, with each value now representing a mixture of colors. Though panchromaticity implies that the notion of a 3-partition (M,Ψ_r,Ψ_b) and its associated lattice structure no longer applies, use of the Fourier-domain principles introduced above permit direct specification of the chrominance carrier frequencies A _λ of interest. In this manner the optimality condition of Proposition 1 may be fulfilled, whereupon in certain examples, the risk of aliasing is reduced and hence overall image integrity is better preserved by the sensor data. Image data acquired in this manner can be shown as easily manipulated, enjoying simple reconstruction schemes, and admitting favorable computation quality trade-offs with respect to subsequent processing in the imaging pipeline.

Examples of Optimal Panchromatic CFA Design Methodologies In one example, outlined is a method of spatio- spectral CFA design that satisfies the bandwidth maximization property of Proposition 1. First, define c_α(n) := (c_r(n)-μ_α)/p, and C_β(n) := (c_b(n)-μ_β)/p, where μ_α and μ_β are the DC components of c_r and C_b, respectively, and p and γ are constants whose role as design parameters will be discussed in greater detail below. By imposing the convexity constraint c_r + c_g + C_b = γ, (C) becomes:

where i_{s now} the baseband signal that can be taken to

represent luminance. Recalling the lowpass properties of ^x

formulation of the CFA design problem enables modulation of these terms via multiplication with c_a{n) and c^n) such that the Fourier transforms of the frequency-modulated difference images are maximally separated from the baseband spectrum -^c = -^FlC'-- In this example, assume that the Fourier transforms of c_a and C_β, respectively, take the form:

where " denotes complex conjugation, ^ ^ ^ the carrier frequencies, and -V^ e C _me corresponding weights (with conjugate symmetry ensuring that the resultant c_a and C_β are real- valued). It follows that the observed sensor data # is the sum of ^- <' and the modulated versions of X_a and X_β\

This example enables the specification of CFA design parameters directly in the

Fourier domain, by way of carrier frequencies * - ^f and weights {s^, U}. In keeping with Proposition 1, the restriction

_S enforced. Determination of the resultant color filters as a function of parameters μ_a, μ_β, p, γ then follows from inverse

Fourier transforms

In certain examples, it is necessary to ensure physical realizability of the resultant CFA. (Requiring -■ ^«-V^t ».^{,> , (}-_fl(π^{. t,} *^⁽ n > ._:.~ 1. ^ To accomplish this, in one example, first define v_α := min_π c_a(n), V_β := min_π C_β{n), and K := max_π(c_α(«) + Cffji)). By assigning

it follows that a resultant CFA design may be expressed as

In this example, the offsets μ_a and μ_β ensure nonnegativity of c_r and Q,, because yθc_α > -//_α and pc^ > -^. The constant γ guarantees nonnegativity of c_g, because c_g = γ - c_r— C_b and pK > p{c_a + C_β). Finally, the maximum value of c_r(n), c_g(n), C_b{n) is equal to 1, owing to the multiplier p.

According to one aspect, robustness to aliasing is achieved via ensuring that spectral replicates lie along the perimeter of the Fourier-domain region B = [-π,π)x[-π,π) while avoiding the values [-π,0]^τ and [0,-π]^τ along the horizontal and vertical axes, our spatio- spectral CFA design aims, in some examples, to preserve the integrity of the color image signal in the sensor data. Image data acquired this way are easily accessible, enjoy simple reconstruction schemes, and admit favorable computation-quality trade-offs that have the potential to ease subsequent processing in the imaging pipeline.

For one example, let 0 < c_r(n),c_g(n),c_b(n) ≤ 1 indicate the CFA projection values at a particular spatial location, where c_r(n),c_g (n),c_b(n) now assume continuous values and hence represent a mixture of prototype channels. With the additional constraint that c_r + c_g + c_b = γ , it follows that

and we may determine the modulation frequencies of difference channels x_a(n) and X_β(n) by our choice of c_r(n) and c_b(n) . Some examples may be designed such that Fourier transforms of the frequency-modulated difference images X_a(ω -λ_r), X _β (ω - λ_b ) are maximally separated from the baseband spectrum X _g (ω) .

In the steps outlined below of an example process, we first specify the carrier frequencies {τ,} and corresponding weights S₁J₁ G C in color filters c_r , c_b . Recalling that for constants V, K we have that

and noting that the support of the nonzero frequency components in c_r,c_b is unchanged by translation and rescaling, we then manipulate our candidate color filter values until the realizability condition

_g is met:

Algorithm 1 : CFA Design Example

• Specify initial values {τ_; , ^ , t_; } . Set modulation frequencies:

• Add a constant v_r = mincj:⁰⁾(n),v₆ = minc^⁰⁾(n) (non-negativity):

• Scale by r= (max_/f^ + c^n))^"1 (convex combination): .

• Find green:

cf 1 cf cf .

• Scale by

^® f f ¹

Figure 28(a)-(e) illustrates each step of the algorithm. In particular Figure 28(a)-(e) illustrates examples of color filter array design visualized in Cartesian coordinates (c_r , c_b , c ), with the dotted cube representing the space of physically realizable color filters (0 < c_r (n),c (n),c₆(n) < 1). In figure 28 Algorithm 1, steps 1-5 are shown as (a)-(e), respectively.

According to the example process, in the first step, the carrier frequencies are determined by taking the inverse Fourier transform of δ(ω ± T₁ ) . The symmetry in this step guarantees real-valued color filter array (where ^τ denotes complex conjugation). However, this color filter, in general, may not be physically realizable (some points in Figure 28(a) fall outside of the first quadrant, for example). In the second step, constants V_r,V_b are subtracted to guarantee non-negativity of color filters (Figure 28(b)). The scaling by K and the computed values of green in the next two steps place the color filters on the simplex plane to ensure convexity and

(Figure 28(c-d)). Finally, the multiplication by γ in the last step maximizes the quantum efficiency of the color filters (Figure 28(e)). And the resultant color filter array is physically realizable.

It follows from simple algebra that the observed sensor data y is the sum of X _g and the modulated versions of X_a and X _β : This

example approach enables the specification of CFA design parameters directly in the Fourier domain, by the way of carrier frequencies {τ,} and weights [S₁J₁ ] . In some emboidment the carrier frequencies satisfy

The assumption made in some emboidments of the proposed CFA patterns is the bandlimitedness of the difference images x_a and x_β , and therefore the robustness of performance may hinge on how well this claim holds (e.g. invariance to the influences of illuminant). In emboidments where the bandlimitedness assumption fails to keep, the increased distance between the baseline signal and the modulated difference images reduces the risk of aliasing significantly, effectively increasing the spatial resolution of the imaging sensor. Consequently, the interpolation is less sensitive to the directionality of the image features and a linear demosaicking method suffices for most applications. Linearization of the demosaicking step is attractive because it can be coded more efficiently in DSP chips, it eliminates the temporal toggling pixel problems in video sequences, it yields more favorable setup for deblurring, and it yields more tractable noise and distortion characterizations.

One important consequence of Algorithm 1 is that the color filter at each pixel location is a mixture of red, green, and blue colors rather than a single color— because color filters are commonly realized by pigment layers of cyan, magenta, and yellow (subtractive colors) dyes over an array of pixel sensors, γ > 1 suggests a further improvement in terms of quantum efficiency for other examples. Furthermore, in some examples it is easier to control for sensor saturation because the relative quantum efficiency of the color filter at each pixel location is approximately uniform (c_r + c + c_b = y). The space of feasible initialization parameters {τ,, $,,?, } are underconstrained, offering flexibility in optimizing the CFA design according to other desirable characteristics including: demosaicking performance or linear reconstructability, periodicity of CFA pattern, resilience to illuminant spectrum, numerical stability, and quantum efficiency, and some examples may incorporate one, some, all or various combinations of such optimizations.

In one example, the conjugate carrier frequencies are C₀. (n) = cf (n)^"1 and

C_β(n) = cf (n)^"1. When the carrier frequencies are orthogonal, the modulated signal can be recovered by a multiplication by the conjugate carrier frequency followed by a low-pass filter. With a perfect partitioning of X _g , X_a , and X _β in the frequency domain, an exact reconstruction uses the following scheme:

where "*" is a discret convolution operator and the pass-bands of the low-pass filters h_a,h_g, h_β match the respective bandwidths of the signals x_a,x_g,x_β .

In one example the camera pipeline can be exploited to reduce the complexity of the reconstruction method greatly. Given the mutual exclusivity of the signals in the Fourier domain, use c^^{ h_a +h_g + c_b ⁽⁰⁾h_β = δ , where <5(n) is a Kronecker delta function. Then using the linearity and modulation properties of convolution,

Then the demodulation in (III) simplifies to:

The first term in (IV) is a 3x3 matrix multiplication, which is a completely "pixel- wise" operation, whereas the spatial processing component is contained in the second term. In the usual layout of the digital camera architecture, a color conversion module follows immediately, converting the tristimulus output from demosaicking to a standard color space representation through another 3x3 matrix multiplication on a per pixel basis. The two cascading matrix multiplication steps can therefore be performed together in tandem, where the combined matrix is computed offline and preloaded into the camera system.

With sufficient separation of the modulated signals in the frequency domain, crudely designed low-pass filters are sufficient for the reconstruction task, in some emboidments. One example is implements (IV) using a separable two-dimensional odd-length triangle filter, which is a linear phase filter with a modest cutoff in the frequency domain. A length "2q — l " filter can be implemented by four cascading boxcar filters, which has the following Z - transform:

where Z₁ and Z₂ correspond to delay line in horizontal and vertical directions, respectively. The computational complexity of the above system is 8 adders for h_a and h_β each.

Moreover, in examples of 4x4 repeating square patterns for the CFA, the carrier frequencies cf^] and cf^] are often proportional to sequences of ± l's (and by extension, c_a and c_β also). In this case, the multiplication by " -1 " before addition in (V) simply replaces adders with subtracters, which is trivial to implement. The overall per-pixel complexity of the demodulation demosaicking in examples implementing (IV) is therefore comparable to that of bilinear interpolation (16 add/subtract operations per full pixel), despite its state-of-the-art image quality performance.

Other Features of Optimal Panchromatic Design Examples

A feature of the framework is that carrier frequencies [X₁] and their weights [S₁, t_t] are specified directly, with (E) and (G) ensuring that the resultant CFA is physically realizable. Patterns designed in this manner are panchromatic by definition, as they satisfy the conditions of Proposition 1 yet support three color components; however, as discussed above with respect to Assumptions 1-3, at least some examples avoid at least some of the shortcomings of previously proposed panchromatic CFA designs. In one example, the convexity constraint γ = c_r+c_g+C_b helps to ensure uniform quantum efficiency of the pixel sensors across the image plane, one consideration in avoiding under- and over-saturated sensor measurements within a single image. Moreover, CFAs are often implemented as a combination of so-called subtractive colors in practice, in which case the condition maxn(cr(n), cg(n), cb(n)) = 1, used in at least some examples, ensures that as many photons as possible will penetrate the physical filters.

While certain examples do not explicitly take into account the effects of different illuminants, Assumption 3 states that the bandwidth of luminance exceeds that of chrominance. Thus, robustness of the resultant patterns to changes in illuminant therefore may hinge on how well this relative bandwidth assumption holds under various lighting conditions. It is expect that designs generated by our approach will be no more sensitive to varying illuminants than existing schemes. Even where certain examples violate Assumption 3 - causing an increase in aliasing due to the larger spectral support of chrominance information - it is still the case that spectral replicates induced by examples of the panchromatic designs are farther from the baseband luminance channel, thereby reducing the risk of chrominance-luminance aliasing effects for the sake of chrominance-chrominance ones.

Specification of CFA patterns satisfying the requirements of Proposition 1 is not unique, as the problem of choosing the parameters {λi, si, ti} in (E) is under-constrained. Based on (G), other examples may be generated with a parameter search to determine panchromatic patterns that satisfy (E) and possess other desirable characteristics. Alternative example optimize other features and include:

Periodicity of the CFA pattern:

In one example, constraining components of X₁ to be rational multiples of π ensures periodicity of the resultant CFA. For example, letting components of X₁ be equal to multiples of π/2 induces a 4 X 4 pattern.

Numerical stability of CFA design:

In another example, owing to the modulation weights Js₁, tj, the observed sensor data at frequency X₁ corresponds to a mixture of difference channels x_a and X_β. Large singular values for this linear combination ensure their separability via a requisite matrix inversion, while "equal treatment" of x_a and X_β is made explicit by setting Is₁I = It₁I.

Resilience to illuminant spectrum: In yet another example, the mixture of color components that appears in the baseband luminance signal ^Λ is fixed for any given CFA pattern. Implicitly, therefore, patterns such as Bayer assume a change in illuminant to be a perturbation from the 1 : 2 : 1 proportion of red, green, and blue. Following this logic, this baseband luminance can be adjusted to yield a mixture complementing the "average" illuminant color, in order to minimize deviation from it.

Pixel sensor quantum efficiency:

In a further example, as noted above, γ is a proxy for the quantum efficiency of the pixel sensors. As a result, CFA designs with large γ and p\ s_; + t_;l values tolerate more noise, and hence are favorable for low-light sensing. Amenability to linear reconstruction:

In one example, a linear reconstruction method based on demodulation is sensitive to image features oriented orthogonally to carrier frequency vectors λ; (though this sensitivity is reduced relative to pure-color CFA sampling, due to the increased separation of luminance and chrominance information). Decreasing the total number of carriers, and placing them as far from the origin as possible, subject to the avoidance of chrominance-chrominance aliasing, may serve to further mitigate aliasing.

Demosaicking performance:

In another example, using a diverse set of test images and demosaicking methods, color image acquisition and reconstruction can be simulated. A numerical evaluation of the resultant error yields an empirical measure of reconstructability that may be used to refine the CFA design.

The various examples implementing different optimizations need not be considered separately, and some examples may optimize over a variety of the conditions discussed, various combinations are contemplated, as pairs as well as greater multiples.

Optimal Linear Reconstruction Methodology Examples

A completely linear reconstruction methodology accompanies some examples of new panchromatic CFA patterns, in which the sensor data are subjected to bandpass filtering in order to recover modulated chrominance information and effect a full-color linear reconstruction. It is known that the choice of reconstruction method greatly influences the quality of the output image. Only certain methods have been tested against, however, the CFA design maximizes recoverability by mitigating aliasing effects. In one example, the optimal linear reconstruction method is presented as a reference, in order to compare sensitivity and robustness of various existing and new CFA patterns. The design and optimization of nonlinear demosaicking methods, which have the potential to further improve output image quality, are not addressed.

In one example conjugate modulation sequences are defined:

_{when these sequ}ences are orthogonal, the chrominance information can be recovered via a multiplication by the corresponding conjugate carrier frequency followed by lowpass filtering. Assuming no overlap amongst the supports of Λ^'fύv .^i, and X^{> (}«?^}, obtained is the exact reconstruction of the full-color image -?ⁱⁿ ) as:

where * denotes the discrete convolution operator, and the passbands of the lowpass filters h_^Jι_t. hø _are assumed to match the respective bandwidths of the signals ^:{<>--^ffN-^!>

Given sufficient separation of the chrominance information in the frequency domain, even simple lowpass filters designed for efficiency will suffice for the reconstruction task. In one example, a separable 2-dimensional odd-length triangle filter - a linear-phase filter with modest frequency roll-off - is easily implemented in existing ASIC or DSP architectures. In some examples, a new CFA pattern may be determined in a process similar to process 3700, which is illustrated in Fig. 37. Process 3700 may begin at Block 3701 as indicated in Fig. 37. As indicated in Block 3703, process 3700 may include a step of placing copies of difference signal components in a frequency domain representations of an image captured using the new CFA pattern. Fig. 7, for example, illustrates a frequency domain representation of an image captured using a CFA pattern in which copies 701 of difference signals (e.g., ^a^ⁿ^ and β«\ⁿ)) are positioned away from the horizontal and vertical axes. In some examples, light sensitive elements or a processor coupled to a light sensitive element may be configured to increase a measured magnitude of light to decrease noise, for example, by multiplying a representation of the magnitude by a number larger than one. The positioning of copies 701 may be changed after such multiplication, for example if the positions were arranged without accounting for such multiplication. In some examples, the copies 701 may be regained by returning the magnitude values to their original values. In some examples, if a multiplication factor or other increasing factor is known, a CFA may be designed to account for this factor so that the copies 701 are positioned at desired locations after increasing the magnitudes.

In some examples, the copies 701 may be positioned so that difference signal copies are positioned at symmetrical locations around the horizontal axes 703, vertical 705 axes and diagonal axes. In some implementations, the copies 701 may be positioned in locations that may not interfere with an expected dominant orientation of the central signal 707. For example, in some implementations, the copies 701 may be positioned away from the horizontal 703 and vertical 703 so they do not interfere with horizontal and vertical image features. In some implementations, the number of copies 701 placed in the representation may correspond to a number of different color signals that may be captured. For example, although the present example is given using only two difference signals (e.g., one corresponding to red and one corresponding to blue), the same representation may be used with a CFA having up to four separate difference signals.

As indicated in Block 3705, after determining the location of the difference signal copies 701, the frequency domain representations may be converted to a spatial representation using an inverse Fourier transform. The inverse Fourier transform may transform the frequency domain representation into a spatial representation that indicates a number (e.g., three in the case of a red, green, and blue array) of color component values at each location of a color filter array. Because some of these values may be negative values (i.e., not physically realizable spectral values), in some examples, as indicated in Block 3707, in some examples, an additional step of enabling physically realizable values may be performed. For example, in one implementation, color component values may be shifted so that all color component values are positive or zero values. For example, in some implementations, the most negative color component value of all the color component values from every spatial location may be added to all the color component values.

As indicated in Block 3709, in some examples, after enabling physically realizable color component values, the color component values may be scaled so that the sum of all color components at each location is less than or equal to 1. This scaling of the color components (e.g., Red + Blue + Green) may include summing the color components at each location and determining which summed value or values is largest. Color components at each of the locations may then be divided by this largest summed value, if it is greater than one, so that each set of summed color components ranges between zero (lowest) and one (highest). As indicated in Block 3711, the main color component of the original frequency domain representation may be added to some of the color components so that the sum of each of the color component values at each location equals one. In the example in which green is the central color and red and blue are represented by the difference signals, a green component is added to the color components. The process 3700 may end at Block 3713. It should be recognized that process 3700 or any other process used to determine a CFA pattern is not limited to the present example steps. Additional or alternative steps may be included in a process in accordance with examples.

The final spatial representation of the new CFA generated through process 3700 may be used to generate a CFA according to examples. Some color filter arrays arranged in accordance with some examples may be far less sensitive to the directionality of the image features, and thus offers the potential to significantly reduce hardware complexity in a wide variety of applications, while at the same time improving output color image quality.

In one example CFA, let ^Cr ^¹J indicate a percentage of red light allowed through a filter at location n, ^C9 \ⁿ) indicate a percentage of green light allowed through a filter at location n, and ^ ^; indicate a percentage of blue light allowed through a filter at location n. In one example implementation, let

_g for each location of the color filter array. That is, at the pixel location n the image sensor measures )

_an(j _may _ai_so I₃₆ constrai .ned by

^C ₇-[^Ti) + c_g[n) +c_b(ή) = 1 Q _{e ^} convex combination) then:

Assuming that ^{a = r ~} 8 and β ⁼ b — g are bandlimited signals, and recalling that a multiplication by ^cr^{( n}) and ^cHⁿ) in the spatial domain may implicate a convolution in the

Ot (ri) a t, 1

Fourier domain, it is recognized that in some examples, and ^{■ ■ }} may be modulated via the multiplication with ^Cr ^ⁿ' and ^Cfc \ ⁿJ such that the 2D Fourier transform of the frequency-modulated difference images occupy the regions in the frequency domain not used by # (^'■«-) .

In some examples, to accomplish this task, assume that the color filter i has 2D Fourier transform of the form: _h( ) _a ( ) + , { + _%) + β{ ), (6)

where

denotes complex conjugate of

is a Dirac delta function, and

l _{are the Fourier transforms of ^ respectively}

In some examples, ^l may be restricted such that its horizontal and/or vertical components are set to ^. Then, in such examples, the Fourier transforms of

_{and are}

the sums of frequency modulated difference images:

where ^ is a 2D Fourier transform, and ∫ (α ) = A, and ∫ (β) = . Overall, in such examples, the Fourier transform of the sensor image may be described by:

In some examples, the design parameters of the color filter consist of carrier

frequencies

and their weights , and it may be advantageous to choose them such

that can be partitioned into the unmodulated signal G + So A + toB and the modulated signals s_i A + tiB and s_iA + tiB in the frequency domain.

In some examples, a CFA may be designed according to one or more of the following criteria:

• In some examples, for ^

to avoid aliasing (e.g., overlap) with G, the carrier frequencies may be chosen far from the origin. In particular, the horizontal and vertical axes in the 2D Fourier transforms may be devoted exclusively for G.

• The carrier frequencies may spread out such that and

for ^l T J are mutually exclusive in the Fourier domain. • In some examples, when carrier frequencies are rational multiples of ^a , the inverse Fourier

transforms of ^Cr \ ⁿJ , ^CH J _? and * ^s" ' may assume periodicity. For example, for * lying

on multiples of

may result in a 4 x 4 CFA pattern.

•In some examples, ^c'r \ⁿ) and ^& ^ ^'* may be nonnegative to ease creation of a physically realizable filter. In some examples, the DC coefficients ^s° and ° may therefore be large enough to outweigh the negative values (in the pixel domain) introduced by *-^Sii **/> " r .

• In some examples,

One implementation may include a 4 x 4 CFA pattern. In particular, one CFA pattern in which spectral copies of red and blue difference signals are positioned away from verticals and horizontals, thereby increasing useful horizontal and vertical frequency information, is represented below:

where C₁ , C₂ , C₃ , C₄ , C₅ , C₆ , C₁ , and c₈ correspond to filter colors within the CFA. The filter colors are given by a combination of red, green, and blue color components of light that the filter allows to pass. It should be understood that the examples are not limited to red, green, and blue color components. Rather examples may comprise a CFA comprising any color and any number of colors. The percentages of each color component allowed to pass a filter in the above example is given by:

where the components of each of the red, green, and blue components of each filter are described as (red component, green component, blue component).

Fig. 8 illustrates a table indicating five example CFA patterns (A-E). Figs. 9A-E illustrate representations of the CFA patterns described by the table of Fig. 8, respectively.

Some examples may have the following qualities and advantages over prior-art CFA patterns:

• In some implementations, by translating the spectral periodization to non- vertical and non-horizontal positions, it is less sensitive to the directionality of dominant image features;

• In some implementations, by admitting a restriction of the color filter array values to convex combinations of primary colors, it is ensured to be realizable in practice;

• In some implementations, by ensuring the linearity of corresponding demosaicing methods, it eases the denoising problem and also makes a natural choice for video cameras, where nonlinear processing methods often introduce temporal artifacts;

• In some implementations, by extending this idea it is possible to enable an increased number of distinct color filters overall, thereby offering the potential for improved color fidelity and white-point estimation.

Some examples may include interesting characteristics. First, a consequence of some CFA patterns may include that the color filter at each pixel location is a mixture of red, green, and blue colors rather than a single color. Because color filters are commonly realized by printing cyan, magenta, and yellow (subtractive colors) dyes or inks over an array of pixel sensors, this mixture suggests a further improvement in terms of noise and luminous efficiency. Red, green, or blue single color filters often translates to a thicker dye over each pixel sensor, reducing the number of photons that penetrate through the color filters and thereby increasing the level of noise in the system. On the other hand, a convex combination of the red, green, and blue colors in the CFA may use less subtractive colors. In fact, because

in some examples

, the color filters can be scaled up. For example, _{can be codec}ι _{into C}FA instead as to allow

more light to enter, and the measured sensor value can be scaled down by

after an A/D converter.

Second, some designs of a CFA may have 2n x 2n (i.e. even number) pattern sizes.

One motivation for such designs may stem from Fig. 33, which motivates assigning ^- to high frequencies. In some examples, it may be favorable, therefore, to set either the horizontal and/or vertical components of the carrier frequency to ^. The effective spatial resolution of the imaging sensor may also be increased because the size of the areas in G that do not overlap with the modulated difference images may be far larger than that of the existing CFA patterns. Third, the difference signal copies may be fixed away from the horizontal and vertical axes in some examples. CFA designed in this manner may consequently be less sensitive to the directionality of the image features, and the corresponding demosaicing algorithms do not necessarily need nonlinear elements to control the directionality of the interpolation filters. Linearization of demosaicing step is attractive because it can be coded more efficiently in DSP chips, it eliminates the temporal toggling pixel problems in video sequences, it yields more favorable setup for deblurring, and the noise and distortion characterizations are more tractable. Linearization of demosaicing without the loss of image quality may position the proposed CFA scheme well with the current market trend in digital imaging of rapidly growing spatial resolutions and frame rate.

Fourth, some examples extend naturally to choices of colors other than red, green, and blue. For example, replacing green with a luminance component may be advantageous in some applications. Examples also enable an increased number of distinct color filters overall, thereby offering the potential for improved color fidelity and white-point estimation. To see this, the CFA model in equation 5 above may extend to a linear combination of g(n) plus a desired number of difference images, ^ak⁽n) modulated by the corresponding filter mixture, ^ck{ⁿ) . The desired number of colors that the CFA represents in the spatio-spectral sampling sense may be made clearer by studying the number of carrier frequencies we may introduce before overcrowding the Fourier domain. Finally, restricting color filters so they may be physically realizable — that is,

may be a problem when the support of ^w' exceeds assumptions of some examples described above. While large values for I ^vs ^l- ^f t ^t \^J may overcome this leakage, contamination may be apparent when they are small. Therefore, it may be advantageous to

( a . f . l maximize ¹^ ^{l ■} "^■> within the constraint of physical realizability in some examples.

Data sampled using a CFA arranged in accordance with various examples may be used to generate a demosaiced representation of the original image. One method of demosaicing an image is described in U.S. patent application 60/861,669, to Hirakawa, entitled A FRAMEWORK FOR WAVELET-BASED ANALYSIS AND PROCESSING OF COLOR FILTER ARRAY IMAGES WITH APPLICATIONS TO DENOISING AND

DEMOSAICING, having attorney docket number 2838, and filed on November 29, 2006, which is hereby incorporated herein by reference. In this method, the ordinary Bayer pattern was assumed. If we consider the normalized image spectrum to be supported on ^^~'¹ ' "J , it is easy to see that this demosaicing method may require the constraint that 9&Ϊ ^₆ supported in a ball of radius ^r.? ^{= ;ΪF}/^'4 about the origin, and that the difference signals ^Qf«∑ = ^ri'ⁿ> - &ⁿ) and &

be supported with radius no greater than ^{y<s = K}''^-L .

In contrast, using the example 4 x 4 CFA pattern above (i.e., equation 9), it may be seen that for the same constraint on the difference signals' Fourier support, * ^{d ~ N} ' % we have r_s = i-^\fa ^{■ ■} i ^^/-^ _an increase of about 20% in the allowable spectral radius, or about 45% in the recoverable spectral area. Even more important, the possibility exists to recover the entire green spectrum at exactly 0, 45, and 90 degrees, corresponding to horizontally-, diagonally-, and vertically-oriented image features, respectively. In some implementations, for fixed periodicity of the difference- signal spectral copies, the proposed orientation may be seen to admit a substantial radius ' ^s for fixed ' " . Some examples may include a corresponding demosaicing method that is completely linear, but yet yields image quality comparable or better than the existing methods for Bayer pattern. For example, assuming that the carrier frequencies are far enough from each other such that the modulated difference images can be isolated from each other using a convolution filter, the following steps summarizes an example algorithm: 1) Design a convolution filter '^ι<Αⁿ) that rejects the frequency components occupied by the modulated difference images while passing the rest so

2) Design convolution filters '^li[ⁿ Λyhose passbands are centered around *. Then

_{can be mo}dulated to solve for ^(ω) and ■^(^w) using standard amplitude modulator designs.

3) Take linear combinations of Ho * and H;Y to solve for

. 4) Let

It should be recognized that this is an example algorithm only. Examples may also include any demosaicing algorithm whether linear or nonlinear and include any set of steps.

Example Images

Figs. lOA-C illustrate three example source images. Figs. 11A-D show the sensor data from four filters, ^-ⁿ--% represented using the orthogonal projection onto the subspace spanned by the color coding of each respective color filter:

It can be seen from the figures that the contiguity of image features such as edges are preserved better in the proposed sensor image shown in Figs. 1OB using pattern A from Fig. 8, Fig. HC using pattern B from Fig. 8, and Fig HC using pattern C from Fig. 8, compared to the Bayer sensor image shown in Fig. 1 IA. For example, it is difficult to tell the object boundaries in Fig. HA, while they are less ambiguous in Figs 11B-D. The appearance of color differs greatly between all sensor images. In particular, pattern B, which is horizontally and vertically symmetric, has a washed-out appearance to the color. Indeed, the non-symmetric patterns enjoy coefficients ^"- °^! ° with larger values, hence increasing the color accuracy.

Figs. 11 E-H, show the reconstructed images corresponding to each sensor image from Figs.1 IA-D, respectively. The proposed sensor images use the completely linear demosaicing algorithm outlined above.

Similar to Figs. 11A-H, Figs. 12A-H display the respective sensor image and reconstructed image of the original image from Fig. 1OB using the Bayer pattern, and patterns A, B, and C from Fig. 8. Also, similar to Figs. 11A-H, Figs. 13A-H display the respective sensor image and reconstructed image of the original image from Fig. 1OC using the Bayer pattern, and patterns A, B, and C from Fig. 8.

We see that the reconstruction from the proposed CFA suffers significantly less from zippering artifacts, but has a slightly softer appearance. Other noticeable differences are that the diagonal edge in Fig. 1 IF-H are less jagged and that textured regions, such as the one in Fig. 1OB, do not suffer from artifacts.

Next, we compare the spectrums of Figs. 14A-D, the two dimensional Fourier transforms of sensor images ^ -'representing the "lighthouse" image assuming different CFA patterns. Fig 14 A, which is the sensor image for Bayer pattern, suggests aliasing on both horizontal and vertical axes due to spectral copies of the difference images. On the other hand, in Figs. 14B-D which represent pattern A, B, and C from the table of Fig. 8, respectively, the frequency-modulated difference images appear on the outer perimeters of the Fourier domain away from the horizontal and vertical axes. In particular, Fig. 14B is similar to the pictoral representation of the sensor image in Fig. 8. As a measure of comparison, numerical evaluation using a means square test of the demosaiced images are listed in the table of Fig. 15, including two different Bayer pattern demosaicing algorithms as reference. However, this is not a measure of the integrity of image features preserved by the CFA patterns, as they are demosaicing algorithm-dependent as well as CFA-dependent. The physical realizability of some examples may depend on the chemistry of the dye used in the CCD/CMOS fabrication. Suppose that the difference signals' Fourier support is defined in terms of the radius around the origin, ^Q and "£. Then for a given CFA size, the proposed CFA coding scheme, along with the corresponding demosaicing method, may give us a way to optimally maximize d_g, the allowable frequency support for the green image in terms of radius around the origin. As with any modeling problems, however, the assumption of bandlimitedness can be violated in real life and leakage of frequency components may corrupt the quality of the output image. The issues with noise associated with poor lighting conditions can complicate the problem further. On the other hand, in some examples, optical blurs and oversampling can easily compensate for the incompatibility of the assumed bandwidth.

Evaluation Certain examples demonstrably perform better than existing methods and patterns in terms of mean-square reconstruction error and in terms of aliasing minimization.

Examples of Optimal Panchromatic CFA Patterns

Discussed are examples of N x N CFA patterns generated by conducting an exhaustive search for the optimal ' ^{* (} > over a parameter space restricted to satisfy the following rules:

. (p_roposition X)

•

(induces N x N periodicity)

• Number of distinct carrier frequencies is limited to two

• Red-green-blue ratio in ' < is 1 : 1 : 1 or 1 : 2 : 1

Each set of parameters satisfying the above constraints represent different examples, however, analyzed are the example for which the combination yielded the largest singular values for the choice of weights {s_t, t,}. (Choices of weights discussed above).

Table I PANCHROMATIC CFA PATTERNS SPECIFIED IN TERMS OF *^{λ * • * ■ h }}

The resultant optimal parameters are described in Table I for N = 4, 6, 8; and the corresponding patterns are shown in Figure 21(a)-(h). Figure 21(a)-(d) illustrate examples of CFA patterns generated using the parameters recited in Table I, and Figures 21(e)-(h) illustrate their corresponding "lighthouse" log-magnitude spectra. In these examples, each of these new patterns is panchromatic, and may be asymmetric in the horizontal and vertical directions; all use fewer than N xN color filters in practice. Moreover, there are equal numbers of "neighboring" colors for each color used in the CFA pattern — (some conventional approaches would indicate that this feature simplifies the characterization of cross-talk noise (photon and electron leakages)).

Reduction of Aliasing Effects Using Examples of New CFA Patterns To investigate the potential of these example patterns to reduce aliasing effects, two sets of test images were used to provide full-color proxy data: a standard set of twenty known Kodak images originally acquired on film, and a set of six images measured at multiple wavelengths, available at http://spectral.joensuu.fi, University of Joensuu Color Group, "Spectral database." In keeping with standard practice, simulated data y(n) were obtained for each CFA pattern by "sensing" these images according to (C). Describing in more detail issues inherent in our image test sets, and in particular the latter set of images measured at multiple wavelengths: while the known standard test set of Kodak images provide a widely accepted means of comparing various algorithms, it is known that numerical simulations using this test set are subject to uncertainties about how the digital image data were acquired (e.g., resolution, illumination) and whether they have undergone any additional processing (e.g., white-balancing, gamma correction). To reduce these uncertainties to yield better-controlled experiments, adopted in evaluations are directly measured multi- wavelength image data (identified above) as an additional form of full-color proxy. In this example, the quantum efficiencies of pure-color CFA values were taken directly from the data sheet of a popular Sony sensor (Sony Corporation, "Diagonal 6mm (type 1/3) progressive scan ccd image sensor with square pixel for color cameras," available at http://products.sel.sony.com/semi/PDF/ICX204AK.pdf, 2004); those of some examples of the proposed panchromatic designs are assumed to be linear combinations of these prototype pure-color responses. To generate the reduction in aliasing effects yielded by examples of the new CFA designs, recall from discussion above of (D) and (F) that sensor data Y (ω) can be interpreted as a superposition of baseband luminance channel ^γ> and chrominance information in the form of frequency-shifted versions of X_a and X_β. The mean square error of a linear filter ' '*^* acting on # to estimate ^;r^ has the form Σw iKw*⁾ M*'«J ^~ ui«^jir Barring additional assumptions, the optimal filter in this example is given by Wiener- Hopf, whose expected squared error is described in greater detail below. The Wiener- Hopf filter in this example may be defined in the Fourier domain as

where S_y is the power spectral density of M and -¹^_{* *} ^• is the cross-

spectral density of ' ^f and ^ . If

are in turn assumed mutually independent, it follows from (G) that:

The expected squared error may be computed from:

The distortion in ^ with respect to ^{α ■ <} and -' ^ is likewise:

Differences in the number of carrier frequencies and the relative sizes of the weights {s_t,

t_t) render less useful for comparing the performance of two distinct CFA patterns

directly. However, the central role of

in both

and (I) implies that both are useful analytical tools to understand aliasing effects associated with a particular CFA. For the case of mutually independent

to follow from (G) as :

where 5 denotes a (cross-) power spectral density. In this example, the quantity

corresponds to the inner product between the expected squared magnitudes of ^f- and the modulated versions of ^Λ>< and ^{Λ j}; it evaluates to zero in the absence of aliasing, but is large when aliasing is severe. Thus the integrand in (I) can be taken as a measure of aliasing relative to the magnitude of the sensor data, and is useful for comparing the performance of different CFA patterns. In Table II, shown in Figure 22, aliasing measurements '''« are reported for both sets of test images. In general, there is a significant decrease in aliasing severity when using proposed CFA designs V and Y (patterns with a 1 : 1 : 1 ratio of red-green-blue) rather than pure-color CFA patterns — a trend which is consistent across both sets of test images. The degree of aliasing associated can be visualized with various CFAs by comparing the spectral content of the sensor data, as shown in Figure 18 and the bottom row of Figure 21. In one example, owing to its higher modulation frequencies, the chrominance information illustrated in Figure 21 is disjoint from the baseband luminance information. This reduces the overall risk of aliasing, relative to existing patterns — though any particular aliasing effects will depend on image content. For example, when presented with strong vertical image features, Pattern Z, of Table I, is at a lower risk of aliasing than the Bayer pattern, but at a higher risk than Patterns V-Y, of Table I.

Improved Linear Reconstruction Using New Examples of CFA Patterns

To evaluate the potential of Patterns V-Z, of Table I, for improved demosaicking performance, we employed two contemporary nonlinear demosaicking algorithms the first authored by B. K. Gunturk, Y. Altunbasak, and R. M. Mersereau, in "Color plane interpolation using alternating projections," IEEE Trans. Image Process., vol. 11, no. 9, pp. 997-1013, Sept. 2002 (hereinafter "EVALl"), the second authored by K. Hirakawa and T. W. Parks, in

"Adaptive homogeneity-directed demosaicing algorithm," IEEE Trans. Image Process., vol. 14, no. 3, pp. 360-369, Mar. 2005 (hereinafter "EVAL2"), to serve as a baseline in conjunction with the Bayer CFA pattern. We then employed the simple linear demosaicking scheme discussed above for new panchromatic Patterns V-Z, in which the sensor data y(n) were subjected to bandpass filtering in order to reconstruct the full-color image " ' according to (H).

Figures 23-25(b-f) show simulated sensor data y(n), acquired respectively via the Bayer pattern and example Patterns V-Z, and are represented using an orthogonal projection of the full-color test images onto the subspace spanned by each respective color filter. Figure 23(a)-(f) illustrates (a) detail of original "structure 11"; (b)-(f) illustrate sensor images using a Bayer pattern, Pattern V, Pattern X, Pattern Y, and Pattern Z, respectively. Figure 23(g)-(l) shows (g) state of the art nonlinear reconstruction of (b) according to EVALl; (h)-(l) show optimal linear reconstruction of (b)-(f), respectively. Figure 24(a)-(f) describes examples of: (a) Detail of original "house" image; (b)-(f) Sensor images using Bayer pattern, pattern V, pattern X, pattern Y, and pattern Z, respectively. Figure 24(g)-(l) shows: (g) State-of-the-art nonlinear reconstruction of (b) according to EVALl; (h-1) optimal linear reconstruction of (b- f), respectively. By visual inspection, the contiguity of the image features such as edges are better preserved better in the proposed sensor images relative to the Bayer sensor image. For example, it is more difficult to discern object boundaries in Figure 24(b) than in Figures 24(c- f). Although, it is known that any color image reconstruction depends on the choice of algorithm as well as the choice of CFA pattern, demosaicking experiments provide some sense of the performance gains and trends that we may expect from the new class of spatio-spectral CFA designs introduced.

To this end, Figures 23-25(g-l) show examples of reconstructions corresponding to each sensor image. In comparison to the iterative, nonlinear demosaicking method of EVALl, shown is the reconstructions corresponding to examples of new panchromatic CFAs are significantly less prone to severe aliasing effects; the examples suffer much less from zippering artifacts yet preserve the sharpness of image features.

Figure 25(a)-(f) illustrates: (a) detail of original "lighthouse" image; (b)-(f) Sensor images using Bayer pattern, pattern V, pattern X, pattern Y, and pattern Z, respectively. Figures 25(g)-(l) show: (g) state-of-the-art nonlinear reconstruction of (b) according to EVALl; (h)-(l) optimal linear reconstruction of (b-f), respectively.

Other noticeable differences are the less-jagged diagonal edges in Figures 24(i-l), and that certain textured regions, such as the ones in Figures 25(i-l), do not suffer from artifacts.

These observations are reflected in the measurements of mean-square reconstruction error listed in Table III, Figure 26, whereupon it may be seen that entirely linear reconstructions under Patterns V and X consistently outperform even the state-of-the-art demosaicking algorithms applied to test data acquired under the Bayer pattern. Compared to a linear demosaicking of the Bayer sensor image for the same computational complexity, the proposed CFAs show significant and consistent improvements in both visual quality and mean- square reconstruction error. Overall, the differences for multi-wavelength data are that the demosaicking algorithm of EVAL2 now outperforms the method in EVALl, and is comparable to the optimal linear reconstruction based on the color filter array of Pattern V.

The enhanced quantum efficiencies of the panchromatic color filters afforded by these new patterns also yield increased robustness to sensor noise. As an example, Figure 27 shows reconstructions corresponding to those of Figures 23-25, but for simulated sensor data subjected to Poisson noise, with no denoising applied. In particular Figures 27(a) - (f) illustrates: reconstruction in noise (no denoising methods applied) (a) state-of-the-art reconstruction of Bayer sensor data in Figures 23-25(b) under the influence of noise according to EVALl; (d)-(f) optimal linear reconstruction of Pattern V sensor data in Figures 23-25(c) under the influence of noise.

The corresponding noise contributions are seen by inspection to be less severe in reconstructions obtained from Pattern X than from the Bayer pattern, suggesting the potential for new reconstruction methods to lead to more accurate means of joint denoising and demosaicking.

The image formation of display in human vision is often modeled by the contrast sensitivity function (CSF), which has a low-pass effect to filter out the granularity of individual elements, and the use of the CFA patterns is ubiquitous in today's cathode -ray tube, plasma, and liquid crystal display media. The most well known of the color schemes involve the canonical primary colors of light: red, green, and blue. In some cases, inclusion of a fourth color such as white is considered.

Subpixel rendering and anti-aliasing are some well-known techniques to take advantage of the pixel geometry and arrangement to increase the perceived resolution of the display. The maximal resolution of the display, however, is ultimately limited fundamentally by the choice of the pre-determined CFA pattern. More specifically, the bandwidths of the stimuli that drive the intensity level of the subpixel is typically lower than that of the image features we would like to represent. In this paper, we explicitly quantify the information loss associated with the

CFA-based image representation in terms of spectral wavelength as well as the spatial sampling requirements. We then propose a framework for designing and analyzing alternatives to the existing CFA patterns for color image display devices, which overcomes the shortcomings of the previous patterns.

Let ⁿ be location index and x(n) = (Jc₁ (n), x₂(n), x₃(n))^τ be the corresponding red,

green, blue (RGB) color triple. Given a two-dimensional signal, terminologies such as frequency and spectral are to be interpreted in the context of two dimensional Fourier transforms, which is also denoted by -^r O. This is not to be confused with spectral wavelength, the wavelengths of the light in the sense of color science. This paper is concerned with the design of CFA, which is a spatially varying pattern. Although the image data that drive the light emitting element are sometimes referred to as CFA image in the literature, we use stimuli to avoid confusion with CFA, the coding of color which is not dependent on the image signal content. Analysis of Display Stimuli

Let ^ * " P-*^{' -} such that '/⁽ * >^{l 1} = N r>>> K K³ is the linear transformation decomposing the color defined in RGB color space into luminance component J₁ (n) and two chrominance components J₂(Ii), J₃(Ii) (e.g. YUV, YIQ, YCbCr). Because the high frequency contents of x_vx₂,x₃ are highly redundant, it is often accepted that J₂ and J₃ are bandlimited.

See Figure 29(a-c). This is strongly supported by the fact that the Pearson product-moment correlation coefficient of a color image measured between the color channels is typically larger than 0.9. It is then not surprising that CSF has far wider passband on J₁ when compared to that of J₂ and J₃ . In one example, let c(n) = (C_j(n),c₂(n),c₃(n))^r e {e_j,e₂,e₃} be the red, green, and blue indicator for CFA color at pixel location n , respectively, where *'^■■ *-- R denotes the standard basis. The stimuli « ( >>' J -.- P- and displayed image >"{ »^'* ^ R are:

where c = c typically and φ ≡ M^~rc . Most luminance-chromi-nance decompositions are designed such that J₂ = J₃ = 0 when X₁ = X₂ = x₃. Consequently, it can readily be shown that

2π φ₂(ή) and φ₃ (n) are pure sinusoids, where the corresponding frequencies are ( — ,0) and

for vertical and diagonal stripe sampling, respectively. Thus, the Fourier transform

. . . .. . r i i • • _{1 J 1 J} , , . of the stimuli is a sum of the chrominance signals modulated by and the

luminance signal. Figures 29(d-e) illustrate examples of spectra, revealing severe aliasing between the luminance and the chrominance components near the modulation frequencies.

Figure 29(f) illustrates the log-magnitude of co-effeciencts displayed as pixel intesity, with DC found the center, for a proposed CFA.

In one example, let /z,(n) be the CSF for j,(n) and '*' denotes convolution. The low- level human visual response to a color image x(n) is W u - ; = ^{i ♦ 1I} i ^{ * ^{} ,} H a ^{J' i} ^ i t ^J1 > ⁾ where U'. i ^j- ^■ i n i = h. I n < * a \jt "^■ _ Recall that multiplication in the spatial domain implicates a convolution in the Fourier domain, and in one example, the passband of h_t is wide enough such that h_t (n) * y_t (n) = y,-(n) . In another example, choose {c, c} such that W(^χ) ■-- >Vn-). To do so, examine the properties of VW'):

where ψ = Mc . Then our condition can be rewritten as:

Let $(n) = kψ^n) for some constant k_t . Then (AA) can be viewed as a classical amplitude modulation problem, where multiple signals of interest are transmitted over the same media by modulating first via the multiplication in the time domain. Thus the carrier frequencies ψ_i = φ_i must be chosen such that:

In one example, when i ≠ j we would like the frequency contents of ^.(n)^.(n) to be sufficiently high such that ψ_i (n)^.(n)y .(n) is outside the passband for ^(n) . Since the bandwidth of J₁(Ii) is the largest, we set φ_x (n) = ^^(n) = 1,Vn . One possible strategy is to modulate y₂ and y₃ via the multiplication with φ₂ and φ₃ such that the 2D Fourier transform of the frequency-modulated chrominance signals occupy the regions in the frequency domain not used by y_λ . In some emboidments, an example process includes designing φ and ψ in the frequency domain, recovering their inverse Fourier transforms to find the equivalent representation in the spatial domain, and computing c = M^rφ and c = M^ψ . In one example, {φ₂,φ₃} have two- dimensional Fourier transform of the following from:

where ^ S &"^*, δ(-) is the Dirac delta function, and I₁ denotes complex conjugate of s_t .

Note that the symmetry properties in (AB) guarantees that the inverse Fourier transform is real. The overall Fourier transform of the stimuli is a sum of luminance and frequency-modulated chrominances:

Note that the design parameters of the color filter consist of carrier frequencies {τ, } and their weights { S₁ , t_t } . In one example, an optimal CFA pattern achieves partitioning of

J₁(Ii), y₂(ή), J₃(Ii) in the frequency domain representation of the stimuli. By choosing a carrier T₁ sufficiently far from the baseband (i.e., high-frequency), the chances of ^₂J₂ and ^₃J₃ overlapping with φ_λy_λ in the frequency domain is minimized while ψ_λφ₂ , ψ_λφ₃ , ψ₂Φ_\ ■, ^and ψ₃φ_λ fall outside of the passband for Zz₁ , Zz₁ , h₂ , and Zz₃ , respectively. As a result, the effective spatial resolution of image display is increased because the size of the areas in ^Ci >"ι ^■ that do not overlap with the modulated chrominance signals are far larger than that of the existing CFA patterns. Furthermore, according to another aspect, a CFA pattern with the carrier frequencies fixed away from DC is consequently less sensitive to the directionality of the image features. The requirement that u be non-negative can be met by setting [S₁J₁ ) relatively small compared to the DC value (compensated for by k₂ and k₃ ). Furthermore, the choice of carrier frequencies may be restricted to rational multiples of π so that the inverse Fourier transform of π (AB) is periodic. For example, T₁ lying on multiples of — induces a 4x4 CFA pattern. One

unique aspect of the proposed CFA scheme is that the resulting pattern does not consist of pure red, green, and blue samples, but rather of a mixture at every subpixel position. A design of CFA pattern satisfying the proposed design criteria set forth above is not unique. Thus further evalutation may be beneficial. Consider an example of a CFA pattern induced with the following parameters:

The proposed example of a pattern implies a periodic structure of size 2x4 , and though as it appears on Figure 31 that it consists of the usual red, green, and blue in addition to light blue, every color in this figure is actually a mixture of all primaries (it is easy to verify that arranging red, green, blue plus a fourth color in similar 2x4 lattice pattern would result in a sub-optimal performance). Figure 31 illustrates an example of an induced CFA pattern. Figure 30(a)-(d) shows examples of stimuli and display of a widely available color image, "Barbara," using the above pattern. As apparent from the figure, the vertical stripe CFA is subject to severe aliasing in high-frequency regions such as the scarf and the pants of ""Barbara" image (shown in Figure 30(a)). Although the diagonal stripe CFA is a clear improvement over the vertical, it is unable to suppress the aliasing completely in the textured regions (Figure 30(b)). The proposed CFA, however, is able to resolve the high-frequency content of the image (Figure 30(c)). In particular, the textures on the scarf and the pants, which are oriented in many directions, are recognizable without major artifacts. The improvements in Figure 30 can be explained via the Fourier transform of the stimuli u in Figure 29(f). The distance between DC and the frequency-modulated chrominance signals in the two- dimensional Fourier domain are far greater, and aliasing is much less likely to occur.

Additionally according to another aspect, the stimuli u(n) (Figure 30(d)) can be regarded as a grayscale representation of the color image x(n) with additional information about its chromaticity content embedded inside low- visibility, high-frequency textures. Such mapping gives rise to a reversible coding of color images in black-and-white prints, a technique proven useful in scanning and printing applications.

According to another aspect, the display device CFA is evaluated in terms of throughput of stimuli as limited by aliasing. It is shown the spectral replicas of the chrominance signals induced by existing CFA patterns are centered around frequencies that are not sufficiently far from the DC, consequently overlapping with the luminance signal spectrum and reducing the throughput of the stimuli. By reinterpreting the interactions between the stimuli, display CFA, and CSF in terms of amplitude modulation, an alternative CFA coding scheme that modulates the chrominance signals to a higher frequency relative to common schemes is provided in some embodiments. Other Input and Output Devices

Although the above examples were described in terms of a color filter array, it should be understood that the examples are not so limited. Rather examples may include any input device, including any input device configured to measure one or more magnitudes of electromagnetic radiation. Some examples may include any electromagnetic spectral filter rather than being limited to the color filters described above. In some examples in which electromagnetic radiation outside of the visible spectrum is included in a spectral filter and measured by a electromagnetic radiation sensitive element, an arrangement of filters may be referred to as a multi- spectral filter array rather than a color filter array. Furthermore, examples are not limited to input devices. Some examples may include arrangements of electromagnetic output elements, such as LEDs, or LCD pixels of a computer monitor that are configured to output one wavelength or a band of wavelengths of electromagnetic radiation.

In typical display output applications, images are typically realized via a spatial subsampling procedure implemented as a high-resolution color display pattern, a physical construction whereby each pixel location displays only a single color (e.g., a single wavelength or a single set of wavelengths).

By considering the spectral wavelength sampling requirements as well as the spatial sampling requirements associated with the image display process, we may create a new paradigm for designing display schemes. Specifically, in some examples, the pattern may be used as means to modulate output image signals in the frequency domain, thereby positioning at least some aliased components outside the passbands of a respective contrast sensitivity function, chrominance and/or luminance, of the human eye.

One example method of arranging electromagnetic output elements of predetermined wavelengths for the purpose of displaying color images is described below. Previously, we considered the triple ' ^Cr ' ⁿ ■' ^{: C}9 (ⁿJ ^{■■ c}ι> (ⁿ) > to represent color filter array (CFA) values. In some examples of an output device, consider v^cHⁿ/- ^cg{ⁿ>- ^{cM n;} J to represent color components of display elements instead (e.g., color phosphors, LED's, and the like). In one example, a design choice of a linear combination of the display elements " 2-i ^cr { ⁿ) • ^cg { ⁿ ) • ^cb { ⁿ ) ≥: 1 may yield a pattern in the frequency domain, such that no spectral copy appears on the horizontal or vertical axes, thereby reducing or eliminating aliasing effects in displayed color images. In other examples, as described above with respect to CFA patterns, output element patterns may be arranged in any fashion to position spectral copies at desired positions in a frequency domain representation.

A specific example of this display configuration, designed in the Fourier transform domain, is given in the table of Fig. 16. It should be understood that the table is given as an example only. In the example illustrated by the table, two dimensional Fourier transforms are

denoted by is a Dirac delta function. In some examples, as in the example of the

table in Fig. 16, may be constant for all locations.

It is also recognized that humans' eyes observe in luminance and chrominance components, where the luminance contrast sensitivity function (CSF) is nearly all-pass, while the chrominance CSF has much narrower bandwidth. The spatio- spectral sampling design scheme for display devices proposed above allows some examples to exploit these biomimetic properties by modulating the difference images outside of the passbands of each respective CSF.

Cross Talk Analysis Continued

Recalling that point- wise multiplication in space is equivalent to convolution in the Fourier domain: in one embodiment, c_r and c_b are finite sums of pure sinusoids (true by default for periodic CFAs):

Where J represents a two-dimensional Fourier transform, ω e (R/2;r)² is the radial frequency index, and δ is a Dirac function. Here, the carrier frequencies λ_; e (R/2/r)² and weights S_j ,t_j G C are the parameters determined fully by the choice of CFA pattern.

Examples of such may be found in Hirakawa and Wolfe, "Spatio-spectral color filter array design for enhanced image fidelity". In one example of a process, 600, for image processing, step 602 is performed either by calculating the carrier frequencies λ_} e (R/2/r)² and weights

S_j ,t_j £ C used by the image capture device or by having the values provided. In one example, when an image is captured using an image capture device, 604, the relevant values are stored in memory of the image capture device. In an alternative embodiment, the relevant values are stored in the memory of a different processing system used to perform cross-talk correction. Thus Fourier transform of c^rx is the linear combination of the baseline signal ( §x_g ) and the "modulated" versions of difference images (&x_a and & x_β ):

Let /z(n,m) = h(m) —that is the cross-talk effect is spatially- invariant— and H = Jh . Then,

J where the approximation in the last step is justified owing to the smoothness of H and the bandlimitedness of β - x_a and & ^'x_β .

The first term in (1) denotes the blurring that occurs as a result of spatial averaging in h - —the consequence of this low-pass filtering is usually negligible compared to the optical blurring (i.e. out of focus lens). However, the more noticeable artifact is "desaturation"— this is evidenced by the attenuation of the modulated difference images β{ s_}x_a +t_}x_β} by the factor

of H(λ_j) . As a result, a reconstructed image often appears less colorful. One can observe this

characteristic in Figures 4A-4D. According to one embodiment, due to the fact that h is a lowpass filter, higher modulation terms suffer from increased level of desaturation. As the spatially-variant impulse response h varies very smoothly over n , there is no need to to distinguish between h and h in (1) other than that the attenuation term

is a function of pixel location n .

Cross-Talk Color Correction

The role of the correction scheme needed to improve the color fidelity is to cancel the effects of the attenuation H_n (λ_;) . One naϊve approach to accomplishing this is ^vvde- convolution"— rescaling 3- -y by H_n ^-1 — naϊve because of its numerical instabilities (especially in the presence of noise) and high computational costs. Moreover, the regularization terms employed in existing ^vvde-blurring" methods are tuned to enhance image features and are not intended for the subsampled color image data dealt with in a single-sensor color imaging device.

Alternatively, one may focus on the color fidelity after or in conjunction with demosaicking—a process of reconstructing a spatially undersampled vector field whose components correspond to particular color tristimulus values. Re- write (1) in the spatial domain as follows:

where is a slightly blurred green image, and— without loss

of generality— let λ₀ = [0,0]^r and H_n (λ₀) = 1. Though overwhelming majority of the state-of- the-art demosaicking methods are nonlinear, they are often conditionally linear (examples include directional filtering, data-driven kernels, etc.) and can be re-written as a spatially- variant convolution. Let f : Z² x Z² — > R³ be the impulse response of the demosaicking kernel where F_n : (RJ2π)² → R³ and

Then, the reconstructed color image (according to the chosen demosaicking method) is:

where the approximation in the last step is two-fold:

The approximation in (3) stems from the fact that one key goal of demosaicking is precisely to achieve According to some embodiments, the

existence of blurring due to h in fact makes this task even easier). Referring again to Figure 6, in an example process 600, demosaicing occurs at 606. The approximation in (4)— which applies to x_β as well— is justified owing to the smoothness of H_n and the bandlimitedness of the difference images.

Besides the negligible blur introduced by h , the matrix M_n e R^3x3 as indicated in (2) represents the desaturation as manifested by the cross-talk contamination— it is full-rank (unless H_n (λ_;) = 0 for some j ) and off-line computable. Thus, the cross-talk color correction scheme is a simple pixel-wise matrix inversion:

which can be performed after 608(NO) (by retaining F(λ_; ) after demosaicking) or in conjunction with demosaicking step 608(YES) (by combining M^"1 with f ). At 610, using the value of f,obtained during demosaicing in combination with M^"1 cross-talk corrected image data is obtained. At 612 a corrected representation is reconstructed. If cross-talk correction is not performed in conjunction with demosaicing 608(NO) F(λ_; ) is stored for later processing at 614. Later process may involved tranfer of image data to a processing system, or may take place in the image capture device itself.

In many existing demosaicking methods, the kernel f is predetermined and thus the matrix M_n is easy to (pre-)compute. Take, for example, directional filtering strategy— f is limited to a few choices (e.g. "vertical" and "horizontal" interpolation kernels), considerably reducing the complexity of the proposed cross-talk color correction for real-time processing for some embodiments.

Examples of Verification Results

For some examples of verification results, a time-invariant cross-talk kernel is used: /z(n,m) = (τl 2rc) exp(- Il Tm Il Ire) , τ = 5 , K = 2. Work was performed with CFA patterns discussed in Bayer U.S. Pat. No. 3,971065, Yamanaka U.S. Pat. No. 4,054,906, Lukac and Plataniotis, "Color filter arrays: Design and performance analysis," and Hirakawa and Wolfe, "Spatio- spectral color filter array design for enhanced image fidelity" and demosaicking methods in [Hirakawa and Wolfe, "Spatio-spectral color filter array design for enhanced image fidelity" ] (linear) and [Hirakawa and Parks, "Adaptive homogeneity-directed demosaicing algorithm"] (nonlinear) for Bayer pattern case.

The zoomed portions of example processed images are shown in Figures 4A-5D and the mean square errors of reconstructions are reported in Table 1. Though the smoothing in the output images of demosaicking step (shown in Figures 4A-4D) is hardly noticeable, the colors appear desaturated to varying degrees— [4] is most severe and [1] is least affected— but in all cases color desaturation dominates the reconstruction error. The proposed color correction scheme restores the desired color, and the output from [4] now outperforms [I]. See Figures 5A-5D.

Performance comparison for various CFA patterns and reconstruction methods tested on the 20 Kodak test set images of Guntruk et al. "Demosaicking: Color filter array interpolation in single chip digital cameras" using nonlinear and linear demosaicking methods discussed in - Hirakawa and Parks, "Adaptive homogeneity-directed demosaicing algorithm," - nonlinear and - Hirakawa and Wolfe, "Spatio-spectral color filter array design for enhanced image fidelity," - linear.

Summary MSE statistics are expressed as ^vvbefore/after" color correction. [1] is taken from Bayer U.S. Pat. No. 3,971065; [2] is taken from Yamanaka U.S. Pat. No. 4,054,906; [3] is taken from Lukac and Plataniotis, "Color filter arrays: Design and performance analysis"; and [4] is taken from Hirakawa and Wolfe, "Spatio-spectral color filter array design for enhanced image fidelity".

In another aspect, equations (1) and (2) provide insights into numerical stability of color filter arrays. The color filter array with higher carrier frequencies (greater Il λ_} Il )— though more robust for demosaicking— are likely to suffer cross-talk phenomenon (smaller H_n(λ_}) ), as evidenced by Figures 4A-5D and Table 1. Accoridng to one embodiment, the severity of the desaturation artifact has far less bearing on the effectiveness of the proposed color correction scheme, however. Indeed, the CFA sampling can be viewed as spatial- frequency multiplexing, where the CFA demosaicking is then a demultiplexing problem to recover subcarriers, with spectral overlap given the interpretation of aliasing as discussed in [Dubois, "Filter design for adaptive frequency-domain Bayer demosaicking"]. The cross-talk contamination attenuate both ^x_g (ω) and β^'{s_Jx_a + t_JX_β}(ω -λ_j ) by an equal amount

( H_n(K_j ) )— and their ratio remains fixed even after the proposed color correction scheme, which has the effect of boosting the baseline and the subcarriers by H^"1 . In net, the contribution of aliasing to the overall image quality is independent of cross-talk contamination, and the color correction scheme remains numerically stable even if the L² matrix norm of M_n is small, as shown by the last column of Table 1.

The interactions between cross-talk and noise is quite complex, and further analysis may require consideration of optical diffraction and minority carrier diffusion separately. Recall that the number of photons encountered during a spatio-temporal integration is a

Poisson process. Although the optical diffraction takes place before the charge collection, the Poisson process in the sensor measurements is no longer spatially independent owing to the minority carrier diffusion which couples the neighboring pixel sensor values after the photoncurrent is generated as discussed in Shcherback et al., "A comprehensive cmos aps crosstalk study: Photoresponse model, technology, and design trends."

According to another aspect, analysis of cross-talk clarifies a common misunderstanding. Putting aside the manufacturing variabilities, it is often claimed that a color filter array with a fixed number of neighbors corresponding to each color filter type is more robust to cross-talk contamination as discussed in Shcherback et al. It is apparent from (1), however, that there is no evidence to support the advantages to this arrangement.

Aided by spatio-spectral sampling theories, the cross-talk phenomenon may be analyzed as the coding of chrominance data embedded in the sensor measurements. Due to the bandlimitedness of the chrominance images, the desaturation artifacts are characterized as the attenuation of the modulated signals by the frequency response of the cross-talk kernel at the carrier frequencies. In one embodiment, the method to correct the cross-talk contaminations may be dervied from the interplay between cross-talk and demosaicking— which can be reduced to a pixel- wise matrix operation. In another embodiment, the brilliance of the color (saturation) is restored after an inverse matrix operation, as confirmed by our numerical examples of evalution resutls. Having thus described several aspects of at least one embodiment of this invention, it is to be appreciated various alterations, modifications, and improvements will readily occur to those skilled in the art. Such alterations, modifications, and improvements are intended to be part of this disclosure, and are intended to be within the spirit and scope of the invention. Accordingly, the foregoing description and drawings are by way of example only. What is claimed is:

Claims

1. A computer implemented method for image processing to improve color fidelity of captured image data, the method comprising the acts of: removing effects of attenuation resulting from characteristics of an image capture device, wherein the act of removing effects of attenuation resulting from characteristics of the image capture device further comprises: determining the attenuation for at least part of a Fourier domain representation of a captured image; correcting the attenuation in the at least part of a Fourier domain representation of the captured image; and storing a corrected representation of an original image.

2. The method of claim 1, wherein the act of determining the attenuation further comprises an act of approximating the attenuation of chrominance and luminance signals resulting from cross-talk.

3. The method of claim 1, further comprising an act of reducing computational complexity of correcting the attenuation by reducing computation of cross-talk correction to a matrix inversion.

4. The method of claim 3, wherein the matrix inversion is performed pixel- wise.

5. The method of claim 1, wherein the act of determining the attenuation for the at least part of a representation of a captured image further comprises calculating a cross-talk kernel.

6. The method of claim 5, wherein the at least part of a representation of a captured image comprises the cross-talk kernel.

7. The method of claim 5, wherein the act of determining the attenuation for at least part of a representation of captured image further comprises determining the attenuation based, at least in part, on at least one carrier frequency of at least one signal in a Fourier domain representation of the captured image.

8. The method of claim 5, wherein the cross-talk kernel is time-invariant.

9. The method of claim 1, 2, 3, or 5, wherein the at least part of a Fourier domain representation comprises at least part of at least one difference image signal and at least one baseband signal.

10. The method of claim 1, 2, 3, or 5, wherein the characteristics of the image capture device further comprise at least one of optical diffraction and minority carrier interference.

11. The method of claim 1 , 2, 3, or 5, further comprising an act of displaying a corrected representation of an original image.

12. The method according to claim 1, 2, 3, or 5, further comprising an act of demosaicking captured image data in conjunction with the act of removing effects of the attenuation resulting from properties of the image capture device.

13. The method according to claim 1, 2, 3, or 5, further comprising an act of demosaicking captured image data, and wherein the act of removing effects of attenuation resulting from properties of the image capture device occurs after the act of demosaicking.

14. The method of claim 9, wherein the at least one baseband signal represents the luminance component of the representation of the image and the at least one difference image signal represents the chrominance component of the representation of the image.

15. The method of claim 14, further comprising an act of reconstructing a color image, wherein the act of reconstructing the color image further comprises an act of approximating the attenuation of chrominance and luminance signals resulting from cross-talk.

16. The method of claim 1, 2, 3, or 5, wherein the at least part of a Fourier domain representation of a captured image comprises at least part of a Fourier domain representation of color filter array.

17. The method of claim 1, 2, 3, or 5, wherein at least part of a Fourier domain representation of a captured image comprises at least part of color filter array.

18. The method of claim 1, wherein the act of determining further comprising an act of generating a correction value by approximating the attenuation as spatially-invariant in a

Fourier domain representation of the captured image data.

19. The method of claim 1, 2, 3, or 5, wherein the act of removing effects of attenuation resulting from characteristics of the image capture device further comprises an act of accessing a demosaicking kernel generated from demosaicking of the captured image data.

20. A system for capturing image data, the system comprising: a color filter array comprising a plurality of color filters adapted to filter light; a plurality of photosensitive elements, each photosensitive element configured to measure light received through the plurality of color filters and output data values; a processing component coupled to output of the plurality of photosensitive elements and adapted to remove effects of attenuation resulting from measuring the filtered light, wherein the processing component is further adapted to remove effects of attenuation by determining attenuation for at least part of a Fourier domain representation of the captured light, and correcting the attenuation to at least one difference image signal and at least one baseband signal; and a storage component for storing a corrected representation of an original image.

21. The system of claim 20, wherein the processing component is further adapted to access a demosaicking kernel generated from demosaicking of the captured image data to determine a correction value.

22. The system of claim 20, wherein the processing component is further adapted to determine the attenuation by approximating the attenuation of chrominance and luminance signals resulting from cross-talk.

23. The system of claim 20, wherein the processing component is further adapted to reduce computational complexity of correcting the attenuation by reducing computation of cross -talk correction to a matrix inversion.

24. A method for correcting attenuation of captured image data resulting from cross-talk contamination of captured image data in association with image demosaicking, the method comprising: transforming a representation of a captured image into at least one baseband signal and at least one difference image signal in a Fourier domain; determining attenuation for at least a portion of the transformed image signals in the

Fourier domain; correcting at least one attenuated image signal; and storing the at least one corrected image signal.

25. The method of claim 24, wherein the act of determining attenuation for at least a portion of the transformed image signals in the Fourier domain further comprises an act of accessing a demosaicking kernel generated from demosaicking of the captured image data.

26. The method of claim 24, wherein the act of determining the attenuation further comprises an act of approximating the attenuation of chrominance and luminance signals resulting from cross-talk.

27. The method of claim 26, further comprising an act of reducing computational complexity of correcting the at least one attenuated image signal by reducing computation of cross-talk correction to a matrix inversion.

28. The method of claim 24, wherein the act of determining the attenuation further comprises calculating a cross-talk kernel.

29. A method for performing cross-talk correction, the method comprising the acts of determining a correction kernel from at least a portion of a transformed image signal, wherein the at least a portion of the transformed image signal is at least part of a Fourier domain representation of captured image data; determining attenuation based, at least in part, on a cross-talk kernel; correcting the attenuation of at least one image signal using the correction kernel; and storing the corrected at least one image signal.

30. The method of claim 29, wherein the act of determining attenuation further comprises an act of accessing a demosaicking kernel generated from demosaicking of the captured image data.

31. The method of claim 29, wherein the act of determining the attenuation further comprises an act of approximating the attenuation of chrominance and luminance signals resulting from cross-talk.

32. The method of claim 31, further comprising an act of reducing computational complexity of correcting the attenuation by reducing computation of cross-talk correction to a matrix inversion.

33. The method of claim 29, 30, 31, or 32, further comprising an act of generating a corrected representation of an original image from the corrected at least one signal.

34. The method of claim 29, 30, 31, or 32, further comprising an act of accounting for the combined effects of optical diffraction and minority carrier interference using the cross-talk kernel.

35. A method for digital image processing, the method comprising: capturing image data representing an original image; determining a correction value based, at least in part, on attenuation of captured image data resulting from an image capture device; correcting data values for at least a portion of the captured image data by the correction value; and storing corrected image data.

36. The method of claim 35, wherein the act of determining a correction value further comprises an act of accessing a demosaicking kernel generated from demosaicking of the captured image data.

37. The method of claim 36, wherein the act of determining the attenuation further comprises an act of approximating the attenuation of chrominance and luminance signals resulting from cross-talk.

38. The method of claim 37, further comprising an act of reducing computational complexity of correcting the attenuation by reducing computation of cross-talk correction to a matrix inversion.

39. The method of claim 35, 36, 37, or 38, wherein the act of correcting data values for at least a portion of the captured image data by the correction value is performed in conjunction with an act of demosaicking the captured image data.

40. The method of claim 39, wherein the act of correcting data values for at least a portion of the captured image data by the correction value includes an act of storing a demosaicking kernel generated from demosaicking of the captured image data.

41. The method of claim 40, further comprising an act of generating a reconstructed image from the corrected image data.

42. The method of claim 35, 36, 37, or 38, further comprising an act of generating the correction value by approximating the attenuation as spatially-invariant in a Fourier domain representation of the captured image data.

43. A system for correcting attenuation of captured image data resulting from cross-talk contamination of captured image data in association with image demosaicking, the system comprising: an image capture component for measuring incident light on a plurality of photosensitive elements and outputting data values; a processing component coupled to output of the image capture component and for transforming image data into at least one baseband signal and at least one difference image signal in a Fourier domain, wherein the processing component is further adapted to demosaic the transformed image data; a correcting component adapted to determine attenuation for at least a portion of the transformed image signals in the Fourier domain, wherein the correction component is further adapted to correct the attenuation of at least one image signal; and a storage component for storing the at least one corrected image signal.

44. The system of claim 43, wherein the processing component is further adapted to store a demosaicking kernel generated from demosaicking of the captured image data and wherein the correcting component is further adapted to access the demosaicking kernel as part of determining the attenuation.

45. The system of claim 43, wherein the a correcting component is further adapted to determine the attenuation by approximating the attenuation of chrominance and luminance signals resulting from cross-talk.

46. The system of claim 43, wherein a correcting component is further adapted to reduce computational complexity of correcting the attenuation by reducing computation of cross -talk correction to a matrix inversion.

47. A system for performing cross-talk correction, the system comprising: a correction component adapted to determine a correction kernel from at least a portion of a transformed image signal, wherein the at least a portion of the transformed image signal is at least part of a Fourier domain representation of captured image data; a calculation component adapted to calculate attenuation based, at least in part, on a cross-talk kernel, wherein the calculation component is further adapted to correct at least one image signal using the correction kernel; and a storage component for storing the corrected at least one image signal.

48. A image capture device, the image capture device comprising a color filter array comprising a plurality of color filters adapted to filter light; a plurality of photosensitive elements, each photosensitive element configured to measure light received through the plurality of color filters and output data values; a processing component adapted to receive image data representing an original image, wherein the processing component is further adapted to boost at least a portion of the captured image data to correct for attenuation of at least a portion of the captured image data; and a storage component adapted to store corrected image data.

49. A system for performing cross-talk correction, the system comprising: a correction component adapted to determine a correction kernel from at least a portion of a transformed image signal, wherein the at least a portion of the transformed image signal is at least part of a Fourier domain representation of color filter array; a calculation component adapted to calculate attenuation based, at least in part, on a Fourier domain representation of color filter array, wherein the calculation component is further adapted to correct at least one image signal using the correction kernel; and a storage component for storing the corrected at least one image signal.