WO2001099431A2

WO2001099431A2 - Method and apparatus for increasing resolution in digital imaging system by offsetting pixels

Info

Publication number: WO2001099431A2
Application number: PCT/US2001/018954
Authority: WO
Inventors: Wade Thomas Cathey, Jr.; Hans Brandon Wach
Original assignee: University Technology Corporation
Priority date: 2000-06-16
Filing date: 2001-06-13
Publication date: 2001-12-27
Also published as: WO2001099431A3; AU2001266878A1

Abstract

Methods and apparatus for increasing resolution in digital imaging systems reposition the fixed detector arrays (or the red, green, and blue images on the detector arrays) so that the resolution of the composite three color image is increased. For example, the green CCD image is shifted horizontally and vertically 1/3 of a pixel with respect to the red CCD image, and the blue CCD image is further shifted horizontally and vertically 1/3 of a pixel with respect to the green CCD image. Image alignment may be accomplished with some combination of CCD movement and/or image movement. The color recovery process, also called reconstruction, may be aided by use of one of several interpolation methods. Color filter arrays may be used over the CCDs.

Description

METHODS AND APPARATUS FOR INCREASING RESOLUTION IN DIGITAL IMAGING SYSTEMS BY OFFSETTING PIXELS

This invention was made with Government support awarded by the national science foundation under Grant Number NSF MIP-9613754 and the Army Research Office under Grant Number DOD DAAG 55-97-1 -

0348. The Government has certain rights in this invention.

BACKGROUND OF THE INVENTION

FIELD OF THE INVENTION

The present invention relates to methods and apparatus for increasing resolution in digital imaging systems by permanently offsetting pixels in overlapped red, green and blue images and reconstructing the full color image.

DESCRIPTION OF THE PRIOR ART

In digital cameras, there are two generally accepted methods for acquiring color images using a light detector array. The first method is to put a color filter array (CFA) over the detector array, such that certain pixels detect certain colors. For example, one third of the pixels would detect red light, one third of the pixels would detect green light, and one third of the pixels would detect blue light. The result of using such a color filter is that each pixel in the array only sees one color, so that to obtain a full color image, interpolation must be used to estimate the other two colors at each pixel. Figures 1A and 1 B (prior art) show how this method is used. The image obtained by the detector array through the color filter array has an arrangement of red pixels, blue pixels, and green pixels, for example in the Bayer CFA pattern, as shown in Figure 1A. The intensity of each pixel indicates how much of the selected color of light was detected. In Figure 1 B all of the pixels have all three colors of light, because the green light is interpolated in some manner, for example between pixels G1 , G4, G5, and G7 to determine the intensity of green light in the upper left hand pixel , and so on. Thus, each three color pixel is really accurate only with respect to one of the three colors.

The second method is more expensive, and more accurate with respect to color. Three separate detector arrays are used, one for each color. Thus three images, one for each color, are formed. Then the three images are overlapped to give a composite full color image. The grey scale resolution is no better than the first method, but the accuracy of the colors is greater. Figure 2 (prior art) indicates how the three images are overlapped to generate an accurate three color image. In addition, each array has a 100% fill factor (ratio of pixel size to pixel spacing), meaning there is little or no dead space between pixels. This reduces aliasing. Figure 3 (prior art) shows how this occurs. The main lobe of the image spectrum curve is as narrow as possible (the first zero is at the sampling frequency) when the fill factor is 100%.

Aliasing is an important factor in generating color images. Aliasing occurs when the lens presents more spatial detail to the detector array than it can record. The spatial detail that is left over appears as incorrect, less detailed information. That is, the image has errors caused by the detailed information that is masquerading, or aliasing, as less detailed information.

It is known in the art that the resolution of a detector array can be increased by making multiple exposures of the image and moving the image with respect to the detector array between exposures. For example, if the detector array is shifted by 1/2 pixel, the image resolution recorded by the combination of the two images is doubled in the direction of the shift. This gives erroneous images if the object moves between exposures. For example, a black on white edge would appear as a colored edge.

A need remains in the art for methods and apparatus for increasing resolution and reducing aliasing in digital imaging systems by permanently offsetting pixels in overlapped red, green and blue images and properly reconstructing the full color image.

SUMMARY OF THE INVENTION

It is an object of the present invention to provide methods and apparatus for increasing resolution in digital imaging systems by permanently offsetting pixels in overlapped red, green and blue images and properly reconstructing the full color image.

This is accomplished by repositioning the fixed detector arrays (or the red, green, and blue images on the detector arrays) so that the resolution of the composite three color image is increased. For example, the green CCD image is shifted horizontally and vertically 1/3 of a pixel with respect to the red CCD image, and the blue CCD image is further shifted horizontally and vertically 1/3 of a pixel with respect to the green CCD image. The detectors themselves need not be moved, but instead the three images can be shifted by the appropriate subpixel amount. The color representation of the resulting image can be as good as the image acquired with the detector arrays aligned, and the resolution can be nearly tripled (the fill factor must be decreased to triple the resolution).

In the general case, n arrays are used. Each array has pixels of height p and width σ, and each one is shifted by p/n vertically and/ or σ/n horizontally.

Another embodiment uses three detector arrays, each with a color filter array on it, and displaces the images by subpixel amounts as described above. One advantage of this approach is that some color filter arrays allot proportionally more pixels to green, because it is more important to human viewers. An example of a color filter array is the Bayer pattern color filter array, shown in Figure 1 (prior art).

The color recovery process, also called reconstruction, is aided by use of one of several interpolation methods. The simplest method entails replicating the color value of each pixel nine times (in a 3 by 3 array) and using the raw red, green, and blue values at each effective pixel location.

A second reconstruction method entails using simple linear interpolation of each color plane. A third reconstruction method interpolates the green color plane by means of linear interpolation, and then interpolates the red and blue color planes with help from the knowledge of the green color plane.

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 A (prior art) shows an image obtained by a detector array through the color filter array.

Figure 1 B (prior art) shows the reconstructed three color image with one method for interpolating the pixels of the image of Figure 1A.

Figure 2 (prior art) indicates how the three one color images are overlapped to reconstruct a three color image.

Figure 3 (prior art) shows the image spatial frequency spectrum curve obtained using the method of Figure 2.

Figure 4 is a diagram showing red, green, and blue CCD images with offset pixels according to the present invention.

Figure 5 shows the image spectrum curve obtained using the offset pixel method of Figure 4.

Figure 6A is a diagram showing three images obtained by detectors through Bayer color filter arrays with offset pixels. Figure 6B is a diagram showing a similar two detector embodiment.

Figure 7 shows a three by three array of reconstituted pixels obtained by a first reconstruction method according to the present invention.

Figure 8A is a diagram showing a blue array with zero padding and Figure 8B shows a low pass filter for filtering this array illustrating a second reconstruction method according to the present invention. Figure 9 is a flow diagram illustrating a third reconstruction method according to the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Figure 4 is a diagram showing red, green, and blue CCD images with offset pixels according to the present invention. These CCD images are shown as arrays having three by three pixels, for simplicity. In practical use, the CCD images would be on the order of 500 by 500 pixels, or larger.

In the example of Figure 4, red CCD image 402 is overlaid by blue CCD image 404, which in turn is overlaid by green CCD image 406. In one preferred embodiment, blue CCD image 404 is offset by 1/3 of a pixel, both vertically and horizontally, from red CCD image 402. Green CCD image 406 is further offset by 1/3 of a pixel both vertically and horizontally, in the same direction. This orientation results in an effective sample rate that is three times the normal sampling rate, vertically and horizontally, of the individual CCD arrays. The effective sample rate around the perimeter of the overlaid image is not as high, but in most imaging systems, there are surplus pixels around the perimeter of the array detector.

Each sensor, or CCD, has its own inherent resolution cutoff, which is affected by both the pixel spacing and the pixel size. The digital signal processing combines the three images from the CCDs and forms the effective pixels, reconstructing the image. Figure 7 shows the simplest method of reconstruction in the embodiment of Figure 4. In the general case, n arrays could be used. Each array has pixels of height p and width σ, and each one is shifted by p/n vertically and/or σ/n horizontally. The detected image in the horizontal dimension, y_h, can then be written as: n-l y_h = [g(x) *rect ] 3 comb(^) (1) i=0

Where g(x) is the intensity distribution at the detector, ^* indicates the convolution operator,

#1/2 rect(x) = <0, otherwise and comb (x) = 3 δ(x-k) =-4

Taking the Fourier transform of (1) the spectrum of the detected image is n-1 y_h = σ²G(u)sinc(σu) *comb(σu) 3 exp(-j 2πui -J) (2) i=0

The summation in (2) can be rewritten as,

|^xp(-jπu ) 3 exp(jπu — (2i+ 1)), neven

"J σ — i=-n/2

3 exp(-i2πui — ) = έ (3) ι=o ⁿ I (n-l)σ ^(n~1)/2 _σ

I exp(-jπu ) 3 exp(jπu — 2i), nodd

< ⁿ i=-(n-l)/2

Using the Euler relation the summations in (3) can be written as cosines as:

(4)

Two cases of particular interest are when n=2 and n=3. Ignoring the phase term, which is not detected by a CCD, the image spectra of these two cases are:

y_h |_n=2 = σ²G(u)sinc(σu) * 2cos(2πu— )comb(σu) (5a)

|_n=3 = σ²G(u)sinc(σu) * (l +2cos(2πuγ ))comb(σu) (5b)

Curve 506 is the closest replicated spectrum (caused by the sampling) that remains after sampling with a 1/2 pixel shift. For example, the blue image may be moved down 1/2 pixel and the green image may be moved to the side 1/2 pixel.

Curve 508 is the closest replicated spectrum that remains after sampling with a 1/3 pixel shift, and shows that again there is no aliasing. In this case the blue image might be moved down 1/3 pixel and to the right 1/3 pixel, and the green image might be moved down 2/3 pixel and to the right 2/3 pixel. Curve 510 is a sine function related to the width of the pixel, and provides low pass filtering of the image. To increase the resolution in the case of a 1/3 pixel shift, the pixel size (and hence the fill factor) must be reduced. This broadens the sine function and thereby passes higher frequencies. Any replicated spectrum that overlapped the image spectrum would result in aliasing of the image. Comparison with the image spectrum of Figure 3 (prior art) shows how the resolution has increased, and aliasing has been greatly reduced or eliminated.

Figure 6A is a diagram showing three images obtained by detectors through Bayer color filter arrays with offset pixels. Bayer color filter arrays (CFAs) use twice as many green pixels as red or blue pixels, because green is the most important color to human viewers. The three images (for example) are overlaid in an offset manner as described above. There are a multitude of variations on this idea not only in reconstruction methods but also in the CCD CFAs used. These CFAs take on many different forms and are usually designed to facilitate reconstruction through knowledge of the human visual system. The idea of using shifted CCD arrays to increase resolution, antialiasing and color performance can be extended to use any combination of CFAs. Another such example might be the use of one green CCD and two Bayer CFAs.

Figure 6B shows an alternative embodiment, wherein one detector array could be used to detect the green portion of images, and a second array with red and blue filters could be displaced 1/2 pixel both in the x and y directions. This doubles the resolution of the system, reduces aliasing, provides more green information for the color interpolation (or image reconstruction), and reduces the number of CCD arrays needed.

Given the 1/3 pixel shifting described with regard to Figure 6A, a generalized pixel results in nine effective pixels, as shown in Figure 4. Figure 7 shows a three by three array of reconstituted effective pixels obtained by the simplest reconstruction method (for example section 408 of Figure 4, coterminous with blue pixel 22). This method of reconstruction entails replicating the value of each color pixel nine times (in a three by three array), and applying the raw red, green, and blue values at each location.

The present method does not attempt to increase the resolution of each channel, but rather that of the final three-color image. In order to prevent unwanted color effects in the final image, some form of reconstruction is desirable. Figures 8a and 8b illustrate linear interpolation of each channel, or color plane.

Figure 8A is a diagram showing a blue array with zero padding. Each channel or color plane is increased to nine times its original size (e.g. A 100 by 100 pixel array would be transformed into a 300 by 300 value array) by inserting eight zeroes around each pixel value. A simple low pass filter is then applied to each channel in order to replace the inserted zeroes with their estimated values as shown in Figure 8B.

Figure 8B shows a five by five low pass filter for filtering the array of Figure 8A (the filter can also be larger). The coefficients used in the filter are denoted u, w, x, y, z and can be optimized for best performance by those skilled in the art of filter design. The only restriction on the coefficients of the filter shown in Figure 8B are that x+z=1 and 2w+y+u=1. This reconstruction method produces a better image than the method shown in Figure 7, but it ignores the fact that the human eye responds better to green light than to red or blue. Also, it blindly interpolates each channel separately.

Figure 9 is a flow diagram illustrating a third reconstruction method, which interpolates the green channel first, and then uses the results to better interpolate the red and blue channels. This method avoids the blind interpolation of the previous method and also utilizes a major aspect of the human visual system. In the case of the blue channel, the above objective is achieved as illustrated in Figure 9. Again, eight zeros are placed around the known blue values to create a zero-padded version of the blue channel as shown in Figure 8A. The zero-padded blue channel and the interpolated green channel are divided on a pixel by pixel basis. This yields a new channel that has ratios of blue and green at pixel locations where blue was known and zeros elsewhere. It is this new channel that is then lowpass filtered, recalling the same nuances described in the method above, to replace the zero values. Finally, a pixel by pixel multiplication with the resulting channel and the interpolated green channel produces the final interpolated blue channel. An identical process is used to interpolate the red channel. The three interpolated channels now compose a reconstructed image with increased resolution, antialiasing and color performance.

Those skilled in the art will appreciate that many other configurations fall within the spirit and scope of the present invention.

What is claimed is:

Claims

1 . Improved apparatus for increasing resolution in a digital imaging system of the type having at least two fixed detector arrays for imaging an object and reconstruction means for composing a composite image from the images captured by the detector arrays, wherein the improvement comprises:

each detector array is located such that its captured image is shifted by a subpixel amount with respect to the images captured by the other detector arrays; and

the reconstruction means provides color for each portion of the shifted images.

2. The apparatus of claim 1 , having two fixed detector arrays, one configured for capturing a red and blue image, and the other for capturing a green image, wherein the images captured by the detector arrays are shifted 1/2 of a pixel with respect to each other captured image.

3. The apparatus of claim 1 having three fixed detector arrays.

4. The apparatus of claim 3, wherein the three detector arrays are configured for capturing a red, a green, and a blue image, and wherein the images captured by the detector arrays are shifted 1/3 of a pixel with respect to each other captured image.

5. The apparatus of claim 3, wherein the three detector arrays are configured for capturing Bayer pattern images, and wherein the images captured by the detector arrays are shifted 1/3 of a pixel with respect to each other captured image.

6. A method of imaging an object comprising the steps of:

(a) capturing three images of the object;

(b) shifting each captured image a subpixel amount with respect to the other captured images; and

(c) interpolating between the shifted images to form a composite image.

7. A method of imaging an object comprising the steps of:

(a) capturing two images of the object;

(b) shifting each captured image a subpixel amount with respect to the other captured image; and

(c) interpolating between the shifted images to form a composite image.