DEMOSAICING USING WAVELET FILTERING FOR DIGITAL IMAGING DEVICE
BACKGROUND OF THE INVENTION
1. Field of the Invention
The present invention relates generally to digital imaging devices, and more specifically, to digital video cameras and still cameras using a single-channel, solid-state imaging sensor array.
2. Background of the Invention
Solid-state digital video and still cameras are becoming more and more popular. Most of the current digital cameras use a single-channel, solid-state photosensor array, and many of the digital still cameras on the market now are single channel charge coupled device cameras, or 1-
CCD cameras. In such cameras, the solid-state photosensors are CCD's arranged in an array in a matrix fashion with each CCD defining an element in the matrix called a "pixel". By placing a mosaiced color filter over the photosensor array, a single channel image is created where each pixel has a single color component. For example for a red-green-blue (RGB) system, a pixel has a single red (R), green (G), or blue (B) color component of a full color image. This forms a single channel mosaiced color image. The single channel mosaiced color image has to be demosaiced subsequently so that a full RGB (three-channel) image is created where each pixel contains the three RGB color components.
Each single-channel digital camera has its own unique demosaicing method. Although a number of demosaicing methods have been proposed, many do not take into account the filtering
and sampling process performed by the optical lens and the CCD. Thus, they often generate images that are not sharp and/or which have aliasing effects. Others lack proper modeling of the filtering and sampling process and/or use interpolation filters that either do not have a good frequency response, are not mathematically interpolating, or are not smooth. Thus there is a need for a demosaicing method which properly accounts for the sampling and filtering process performed by the optical lens and CCD, and thus which provides a higher resolution image.
SUMMARY OF THE INVENTION
According to one embodiment of the present invention, a method is provided for processing single- channel mosaiced image information from a sensor and then demosaicing the information using a wavelet filter to provide a super-resolution demosaiced image for further image processing. The super-resolution imaging may also be used for any application involving data interpolation.
In accordance with another embodiment of the present invention, a demosaicing method is provided in which a mosaiced image is received in a plurality of colors and separated into a plurality of single color images. The plurality of single color images are interpolated using wavelet filters, and the interpolated single color images are combined into a demosaiced color image. According to one embodiment of the present invention, one-dimensional wavelet filters are used for the interpolation in the demosaicing process. Alternatively, according to a different embodiment, quincunx wavelet filters may be used for such interpolation.
In one aspect of the present invention, the filters are thresholded by setting to zero coefficients which are smaller than a predetermined threshold.
Another embodiment of the present invention provides a digital camera having a lens for focusing light from a scene through a RGB mosaiced filter onto a solid state photosensor array containing a matrix of CCD's, which provide a single channel mosaiced image output. A demosaicing process uses a wavelet filter to provide super-resolution imaging for demosaicing of the mosaiced image to provide an image output.
The present invention further provides for generating high-resolution image/video from a low-resolution one and may be used to interpolate an NTSC or CCIR-601 signal into a HDTV signal.
The above and additional advantages of the present invention will become apparent to those skilled in the art from a reading of the following detailed description when taken in conjunction with the accompanying drawings.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 shows a solid state digital camera incorporating one embodiment of the present invention;
FIG. 2 shows a representation of a mosaiced image in a Bayer pattern; FIG. 3 shows a representation of a full RGB image output after demosaicing; FIG. 4 shows a flow chart of one embodiment of a imaging process including demosaicing used in the solid state digital camera of FIG. 1;
FIG. 5 shows a flow chart of the demosaicing process according to one embodiment of the present invention;
FIG. 6 is a representation of a series of separated mosaiced image planes each of which is output from the separation process according to one embodiment of the present invention; FIG. 7 is a representation of a series of separated demosaiced image planes each of which is input to the combination process according to one embodiment of the present invention;
FIG. 8 shows a flow chart of an alternate embodiment of the filtering process of the imaging process of FIG. 4; and,
FIG. 9 is a representation of a series of separate demosaiced image planes produced by the alternate filtering of FIG. 8.
DETAILED DESCRIPTION OF THE INVENTION
Referring now to FIG. 1, a solid state digital camera 10 is shown having a camera body 12 and an attached lens system 14 with a lens filter 15. Although a single lens reflex camera is shown it will be appreciated by those skilled in the art that the present invention could be implemented for other still digital imaging devices as well as motion digital imaging devices.
The camera 10 has a mirror 16 which pivots on a pivot 18 to initially direct light both from an image as well as ambient light through the lens filter 15 and the lens system 14 up to a prism 20 and out through an eyepiece 22.
When a picture is being taken, the mirror 16 is pivoted up so as to allow light to strike a recording medium 24. In a digital still camera (DSC), the digital recording medium 24 is a matrix of photosensors, such as analog output, single channel, charge-coupled devices (CCD's) 26. Each of the CCD's 26 represents one picture element, or pixel, of the picture and the full array is called
the "mosaic" pattern. Although cyan, magenta, yellow, and green filters can be used, in the embodiment shown in Figure 1, each of the CCD's 26 has a green (G), red (R), or blue (B) filter over it, generally designated as a mosaic filter 28. After processing, the image falling on these CCD's 26 is digitally recorded, either electronically or magnetically, for later playback. A processing unit 30 incorporating an embodiment of the present invention processes the signals from the CCD's 26 though to the image output.
One example of a mosaiced image 50 may be seen with reference to United States Patent No. 3,971,065 entitled "Color Imaging Array", granted to Bryce E. Bayer, such pattern being commonly referred to as a "Bayer pattern". Figure 2 shows a portion of a representation of part of a mosaiced image 50 in a Bayer pattern, wherein columns "i" (52) show only columns one through six, and rows "j" (54) show only rows 1 through 6. As seen in Figure 2, row 1 starts with a green "G" pixel and a red "R" pixel, and then alternates between the two, while row 2 starts with a blue "B" pixel and a green "G" pixel and then alternates between the two. As will be appreciated from Figure 2, this pattern is repeated for the subsequent rows.
In order to reproduce a full RGB image 60 as shown in Figure 3, the RGB colors of a mosaiced image such as the mosaiced image 50 of Figure 2 must be determined. This is effected through a "demosaicing" process.
Figure 4 shows a flow chart 70 of the solid state digital camera 10 including the demosaicing process of the present invention. Light from an image passes through the mosaic filter 28 onto the CCD's 26, which photo-electrically convert the light falling on them into proportional analog electrical signals at step 71. The analog signals are then converted into digital signals by an analog to digital (A/D) conversion step 72. The digital signals are then output for demosaicing at demosaicing step 74. After demosaicing processing at step 74, the output is subjected to image processing 76, which includes a series of conventional image processing steps, including white balancing and compression. The resulting image output 78 may be provided to user devices such as a recording unit or display device. It will be evident to those skilled in the art, that various mechanisms or devices for performing the conventional image processing steps of the aforementioned method are known and the invention is not limited to any such one mechanism or device.
Referring now to FIG. 5, a flow chart of one embodiment of the demosaicing process 74 of the present invention is shown. According to such embodiment, after the analog signals are converted to digital signals, the thus converted signals are provided to a separation unit 80, which processes the signal to provide separated image planes. In a preferred embodiment such planes are G-R-B planes. G-R-B signals representative of the planes are then subjected to G, R, and B
interpolation at interpolation units 82, 84, and 86, respectively. After interpolation, each of the G,
R, and B interpolated signals are then provided to combination unit 88, which provides a demosaiced signal output for image processing 76.
One embodiment of the separation unit 80 wherein the output has G-R-B planes may be seen in Figure 6. As seen therein, the output from separation unit 80 is a series of three mosaiced image planes 100, 102, and 104. In such embodiment, the image plane 100 preferably includes a first row starting with a green (G) pixel and then alternating between zero-value (0) and green (G) pixels, while the second row starts with a zero-value (0) pixel and then alternates between green
(G) and zero-value (0) pixels. Image plane 102 preferably includes a first row starting with a zero-value (0) pixel and then alternating between red (R) and zero-value (0) pixels, while the second row includes all zero-value (0) pixels. Finally, image plane 104 preferably includes a first row starting with all zero-value (0) pixels, while the second row starts with a blue (B) pixel and then alternates between zero-value(O) and blue (B) pixels. In each of the image planes 100, 102, and 104, the first and second row patterns are then preferably repeated in rows three and four, five and six, and so on.
Once the image planes have been created by separation unit 80, as stated previously, they are provided to each of the interpolating units 82, 84, 86. In a preferred embodiment, image planes 100, 102 and 104, are respectively input to the G, R, and B interpolation units 82, 84, and 86. The outputs from such units are shown in Figure 7, and comprise mosaiced image planes shown at 110, 112, and 114, respectively.
Operation of the imaging process including an embodiment of the demosaicing process of the present invention will now be discussed with reference to Figures 1 and 2. When light from an image enters the camera 10 through a lens filter 15 and lens system 14, it is reflected by the mirror 16 which is in the down position as shown in Figure 1. The reflected light travels upwards into prism 20 and then again reflected through prism 20 to exit out eyepiece 22, where a user sees the exact image "seen" by lens filter 15 and the lens system 14.
When a photograph is taken, the mirror 16 is caused to pivot upward out of the way so that instead of being reflected, the light will pass through a mosaic filter 28 to strike a photosensor matrix array 26, which, again, preferably include single-channel solid-state CCD's. After a single- channel image is created from the photosensor array, each pixel has one color component, R, G, or B, and a mosaiced image 50, preferably in the form of the Bayer pattern of Figure 2 is formed. The mosaiced image 50 will then be digitized by the in A/D conversion step 72.
According to the present invention a demosaicing process is subsequently applied to generate the missing color components. For example, referring to Figure 2, for the pixel at the upper-left comer at column 1, row 1 (bold-faced and italicized), only the G component is created from the photo sensor array 26. Thus, the missing R and B components have to be generated using a demosaicing process. Similarly, for the pixel at column 3, row 4 (bold-faced and italicized), only the R component is created, and the missing G and B components must be generated through demosaicing. The output is subsequently subject to demosaicing step 74 where the mosaiced image 50 is demosaiced and the full RGB image 60 is generated. The full RGB image 60 after demosaicing is shown in Fig. 3.
It will be appreciated that the demosaicing step 74 plays an important role in the digital camera 10, as it has a direct impact on image quality. It is important that the filter used in the demosaicing process have a good frequency response, be interpolating, and be smooth. It was determined that "wavelet filters" provided the aforementioned desired characteristics.
While wavelet filters have been used in the fields of signal analysis and synthesis, particularly with respect to sound and image signals, for the most part such filters have been limited to use in image compression. As known to those skilled in the art, as used in such image compression processes, "wavelet transforms" provide representations of any arbitrary signals as a superposition of wavelets. These wavelets are themselves functions which are, in turn, generated from a single function through dilations and translations and then passed through filters to allow decomposition of a signal into different levels (each of which is further decomposed with a resolution adapted to the particular level). (A detailed discussion of the operation and creation of wavelet transforms will not be presented herein as these functions are understood to those skilled in the art.)
One embodiment of the demosaicing process 74 of the present invention is shown in Fig. 5. The mosaiced output from A/D conversion unit 72 (FIG. 2) is then subjected to the demosaicing process 74. Initially, the mosaiced output is separated by the separation unit 80 into the three separated mosaiced image planes 100, 102, and 104, respectively as shown in Fig. 6. (Again, although these planes are shown as R, G, and B planes, it will be understood that cyan, magenta, yellow, and green planes may also be used). The mosaiced image planes 100, 102 and 104 are then provided as inputs to G, R, and B interpolation units 82, 84, and 86, respectively (FIG. 5). The outputs from these interpolation units 82, 84, and 86, respectively, are shown in Fig. 7 as G, R, and B mosaiced planes 110, 112 and 114, respectively. These three mosaiced planes 110, 112 and 114 are subsequently combined in the combination unit 88, the output demosaiced RGB signal of which is subjected to image processing 76. Description of embodiments of the interpolation units 82, 84, and 86 will now be discussed in more detail.
In one embodiment, the G interpolation unit 82 uses a two-dimensional quincunx wavelet filter, although it will be appreciated to those skilled in the art that other quincunx wavelet filters may be used as well. The coefficients of one embodiment of such filter may be seen in Table I below:
Table I
The interpolated data may be given by the following equation: + m,; +«)/2
l5 x V2
where F, represents this quincunx filter, X
g represents the G mosaiced image plane 100 at pixel position (i,j), and Y
g is the demosaiced output 110 as shown in Fig. 7.
The R interpolation unit 84 preferably uses a one-dimensional wavelet filter, although, again, it will be appreciated to those skilled in the art that other quincunx wavelet filters may be used. One such filter for the R interpolation unit may have the following coefficients: ^[-0.045636 -0.028772 0.295636 0.557543 0.295636 -0.028772 -0.045636]
It is necessary to perform two filtering processes, once in the horizontal direction and once in the vertical direction. However, the order in which these processes is performed is arbitrary. In the following example, the vertical filtering is completed first.
When filtering is applied in the vertical direction at pixel position (i,j), the filtered output is given by the equation:
Zr( j) = ∑F2(m) x Xr(i + m,j) x 2 χ [2
where: Xr represents the R mosaiced image plane 102.
After Zr has been computed as above, each row of the image pattern (e.g., the Bayer pattern of
Fig. 2) is flipped. Then the following equation is then used to derive Z where Zr represent this new signal with pixel index i=0, ..., N-l, j=0, ..., N-l
Zr(i,j) = Zr(i,N -\ - j)
Horizontal filtering is then performed on Zr and the resulting output of the R interpolation unit 84 is given by the equation:
Yr(i,j) = ∑F2(n) x Zr(i,j + n) x f2 . n = -3
Each row of Ϋr is flipped to get the output Y'r. Thus: Yr (i, j) = Yr (i, N - \ - j) . This output, Yrι which is the R demosaiced image plane 112, is shown in Fig. 7.
The B interpolation unit 86 preferably uses the same filter F2 as the R interpolation unit 84. The only difference between these two interpolation units is that in the B interpolation unit 86, the two flipping processes are performed during vertical filtering, while no flipping is needed for horizontal filtering.
An alternative embodiment of this invention can be realized by thresholding the filter F i.e., setting those coefficients of F
2 smaller than a selected threshold. This can reduce the computational cost. One embodiment of such a thresholded filter, F
lt is shown in Table II below with the threshold set at 15:
Table II
The G interpolation unit 82 then performs the following summation: γ i > ) = Σ ∑F(m,ή)χ Xg(i + m,j + n)l2'5 χ 2lk nι=-4.ι=-4 with k given by: k = ∑ ∑Ft(m,n) . nι=-4 «=-4
It will be appreciated that other thresholds may be used.
After processing in combination unit 88, the fully demosaiced RGB image 60 output is then provided to the image processing unit 76. In the image processing unit 76, the RGB image will go through the series of conventional image processing steps, including white balancing and compression, and finally be output as the image output 78.
Referring now to FIG. 8, an alternate embodiment of the filtering process included in the imaging process step of FIG. 4 is shown. In such embodiment, the image output from the demosaicing step 74 is then input into low-pass and high pass branches 151 and 153. In the low- pass branch 151, the input is provided for filtering in F3' block 154, the output of which is then provided to a "reduce by 2" block 156 for removal of every other pixel. The output from "reduce
by 2" block 156 is then provided to the Q block 158 to for quantitization. After quantitization, the output is provided to an "increase by 2" block 160 for addition of every other pixel (which is preferably zero in this embodiment). The output of the "increase by 2" block 160 is then provided to F3 block 162 for front-end weighted averaging, the output of which is then provided to a sum block 164 which, in turn, provides the image output 78.
The image output from the demosaicing step 74 is input into the high-pass branch 153 and is filtered in H3' block 170. The output of H3' block 170 is then provided to a "reduce by 2" block 172 for removal of every other pixel, after which it is passed to the Q block 174 to be quantized. After quantization in Q block 174, the output is provided to an "increase by 2" block 176 for addition of every other pixel (which is preferably zero in this embodiment). The output of the "increase by 2" block 176 is provided to H3 block 178 for front-end weighted averaging and then to the sum block 164 to be added with the F3 block 162 output.
Referring now to FIG. 9, therein is shown a series of two separate demosaiced image planes 200 and 202 produced in the alternate filtering of the present invention. In operation in the low-pass branch 151, the image input from demosaicing step 74 is weight averaged by wavelet filtering in F3' block 154. Alternating pixels are removed in "decrease by 2" block 156 and then the real number value of pixels is quantized into one of a discrete number of values in the quantize block 158. The quantized output then has alternating zeros inserted by the "increase by 2" block 160. The image data is then weight averaged by wavelet filtering in F3 block 162 and provided to sum block 164.
In operation in the high-pass branch 153, the image input from demosaicing step 74 is weight averaged by wavelet filtering in H3 λ block 170. Alternating pixels are removed in "decrease by 2" block 172 and then the real number value of pixels is quantized into one of a discrete number of values in the quantize block 174. The quantized output then has alternating zeros inserted by the "increase by 2" block 176. The image data is then weight averaged by wavelet filtering in H3 block 178 and provided to the sum block 164 where it is summed with the output of the F3 block 162 for the image output 78.
When using the compression system shown in the image processing flow chart 150, the F2 filtering in the demosaicing step 74 is replaced with F3 filtering for R, G, and B planes in the R, G, and B interpolations 82, 84, and 86. For R and B, the filtering process is the same as before. For G, the plane is separated into two planes, filtered, summed, and divided by two. That is, the G plane 100 is separated into the two G planes 200 and 202 , and each is filtered using F3 filtering.
No flipping in either the horizontal or vertical direction is required. The two separate G planes 200 and 202 are summed and divided by two to give an image which is used as the G plane 100.
In this alternate embodiment, the compression algorithm used in the image processing step 76 uses known wavelet compression. In such a situation, the wavelet filtering used in the demosaicing step 74 should be chosen such that the high frequency band signals in the wavelet compression will be small and close to zero. With small coefficients, efficient compression will be obtained.
For example, when a pair of analysis wavelet filters, F3' and H3', are used in wavelet compression, there is a corresponding synthesis filter pair, F3 and H3. The compression system 150 (shown only one-dimensional and for one stage of the entire system) indicates that F2 should be replaced by F3. Thus, the R, G, and B planes are demosaiced by using F3. In the demosaicing of the G plane 100, it is separated into the two planes 200 and 202, as shown in FIG. 9. For either plane 200 or 202, the same filtering process is applied as for the R and B planes except that no flipping is needed in either horizontal or vertical direction. The filtering results of the two separate G planes 200 and 202 are then combined by summing in the sum block 164 and dividing the sum by two. This average value is then used as the G plane 110.
As described above, the present invention provides super-resolution imaging for any application involving data interpolation and more particularly for generating high-resolution image/video from a low-resolution image/video. One application is in the interpolation of an NTSC or CCIR-601 signal into a HDTV signal.
While the invention has been described in conjunction with a specific best mode, it is to be understood that many alternatives, modifications, and variations will be apparent to those skilled in the art in light of the aforegoing description. Accordingly, it is intended to embrace all such alternatives, modifications, and variations which fall within the spirit and scope of the included claims. All matters set forth herein or shown in the accompanying drawings are to be interpreted in an illustrative and non-limiting sense.