KR101243285B1 - Apparatus and method for reconstructing color image based on multi-spectrum using Bayer Color Filter Array camera - Google Patents

Abstract

Disclosed are a multispectral based color image generating apparatus and method using a Bayer color filter array camera. The image capturing unit acquires the first image photographing the target object by the imaging device having the Bayer RGB color filter array-based image sensor and the second image capturing the target object by further installing a monochromatic wideband color filter on the image capturing apparatus. do. The channel generator generates red, green, and blue color channels from the first image and the second image, respectively, using the color pattern of the second image determined based on the RGB color filter arrangement and the spectral response of the wideband color filter. The color interpolator interpolates the color values of pixels to which no color value is assigned in the plurality of color channels. The image generator generates a color image corresponding to the target object by combining the plurality of color channels. According to the present invention, when configuring the red (R) and blue (B) channels, twice as much information as the conventional single-sensor CFA image can be obtained, thereby effectively demosaicing with a small amount of computation when applying the demosaicing method. And solve the problem of color superposition. In addition, by using the existing Bayer CFA image sensor, multi-spectral images can be obtained without having to produce a new type of CFA.

Description

Apparatus and method for reconstructing color image based on multi-spectrum using Bayer Color Filter Array camera}

The present invention is a multispectral based color image generating apparatus and method using a Bayer Color Filter Array (CFA) camera, and more specifically, more effective demosaicing using an additional color filter in a camera using Bayer CFA. The present invention relates to a multispectral-based color image generating apparatus and method for performing demosicking.

In principle, three image sensors are required to acquire color images from a digital camera. However, in order to use three image sensors having different spectral channels, an optical unit and an electronic control unit for optically separating the light incident on the lens and transmitting the same to each sensor are additionally required. Such additional systems increase the physical size and cost of the camera.

Therefore, in a typical digital camera, an image obtained using a color filter array (CFA) is sampled to acquire color information using only one image sensor. Currently, most digital cameras use Bayer CFA (Bayer CFA) to obtain color information.

A camera using one image sensor and Bayer CFA extracts only one color information from each pixel. Therefore, Bayer CFA can acquire only one color information among the RGB channels for each pixel, and the color information of the other two channels should be estimated using a method such as image interpolation. A process for estimating such color information is called color filter array interpolation or demosaiking.

Demosaicing samples the image stored by the Bayer pattern into red (R), green (G), and blue (B) pixels. It estimates using the pixel information of a different channel from the pixel information.

In the simplest example of demosaicing, a color image may be generated by performing linear interpolation or cubic interpolation on each color channel. However, there is a problem in that high frequency components of an image cannot be interpolated effectively, and unintended colors are generated due to superposition of information between color channels. In order to solve this problem, various demosaicing techniques such as adaptive interpolation, regularization, and POCS (Projections Onto convex Sets) method have been studied.

Meanwhile, as a method for generating a color image using a single image sensor, various CFA patterns in which different color filters are arranged in a similar form to Bayer CFA may be used. A typical example is the CYGM pattern, which is generally robust to noise due to its high light transmittance compared to the characteristics of RGB color filters, but loss of color and signal-to-noise ratio occurs during conversion back to RGB for display.

Alternatively, instead of using a Bayer CFA image sensor, a high-resolution image can be obtained using a 3CCD camera that separates the color components into RGB with a prism. In addition, a filter wheel including different color filters may be attached to a lens to continuously capture an image to generate a color image or a multispectral image.

As another method, a method of modifying a multiple color-filter aperture camera system that restores the focus of multiple objects by using an optical unit in which an RGB color filter is added to a plurality of apertures has been proposed. These methods can obtain a high resolution image without using demosaicing, but it requires a process of acquiring three or more images in succession.

Korean Laid-Open Patent Publication No. 2008-0083712 discloses a method and apparatus for generating Bayer color mosaic interpolation for an image. Prior patents select color interpolation points and interpolate each color interpolation point using values for each color in the color pattern. This can alleviate the zipper effect that can be caused by variations in interpolation and the generation of random color dots that produce horizontal or vertical lines with one dot at regular intervals. However, there is a problem that the calculation amount of the color image interpolation for demosaicing increases.

SUMMARY OF THE INVENTION The present invention provides a multispectrum-based color image generating apparatus and method capable of more effectively demosaicing by obtaining color information by attaching an additional color filter to a Bayer RGB color filter array-based image sensor. Is in.

Another technical problem to be solved by the present invention is to attach a color filter to a Bayer RGB color filter array-based image sensor to obtain color information and to perform a more efficient demosaicing method on a computer. A computer readable recording medium having recorded thereon a program for execution is provided.

In order to achieve the above technical problem, a multi-spectrum-based color image generating apparatus using a Bayer color filter array camera according to the present invention is obtained by photographing a target object by an imaging device having a Bayer RGB color filter array-based image sensor. An image photographing unit configured to acquire a second image photographing the target object by further installing a single color wideband color filter on the first image and the image pickup device; Generating red, green, and blue color channels from the first and second images, respectively, using the color pattern of the second image determined based on the RGB color filter arrangement and the spectral response of the broadband color filter. A channel generator; A color interpolator configured to interpolate color values of pixels to which color values are not assigned in the plurality of color channels; And an image generator for generating a color image corresponding to the target object by combining the plurality of color channels.

In order to achieve the above technical problem, a multi-spectrum-based color image generation method using a Bayer color filter array camera according to the present invention, the target object is photographed by an imaging device equipped with a Bayer RGB color filter array-based image sensor An image capturing step of obtaining a second image capturing the target object by further installing a single color wideband color filter on the first image and the image capturing apparatus; Generating red, green, and blue color channels from the first and second images, respectively, using the color pattern of the second image determined based on the RGB color filter arrangement and the spectral response of the broadband color filter. Channel generation step; A color interpolation step of interpolating color values of pixels to which color values are not assigned in the plurality of color channels; And an image generating step of generating a color image corresponding to the target object by combining the plurality of color channels.

According to the multi-spectrum based color image generating apparatus and method using the Bayer color filter array camera according to the present invention, when configuring the red (R) and blue (B) channels, twice as much information as the conventional single sensor CFA image can be obtained. When using the demosaicing method, it is possible to perform effective demosaicing with a small amount of computation and solve the problem of color superposition. Color images can be acquired with a relatively small number of images and can be extended to image acquisition systems for multiple spectrums.

In addition, by using the existing Bayer CFA image sensor, multi-spectral images can be obtained without having to produce a new type of CFA. Furthermore, the loss of light passing through the lens color filter and the CFA can be considered, and the exposure of the two images can be set differently to be used for technologies such as high dynamic range (HDR).

1 is a block diagram showing the configuration of a preferred embodiment of a multi-spectrum based color image generating apparatus using a Bayer color filter array camera according to the present invention;
2 is a view showing a relative spectral response according to the wavelength of a color filter in the visible light band,
3 shows the relative spectral response of the Bayer RGB color filter and the spectral response of the wideband green (G) color filter in the visible light band;
4 is a view showing color patterns of a first image and a second image determined based on a spectral response of a Bayer RGB color filter array and a wideband green color filter;
5 is a flowchart illustrating a process of performing a preferred embodiment of the method for generating a multispectral based color image using a Bayer color filter array camera according to the present invention;
FIG. 6 is a diagram illustrating a result of comparing PSNRs for color images generated by the present invention and a conventional demosaicing technique for each RGB channel, and FIG.
FIG. 7 is a diagram illustrating a part of a resultant image of Image 8 of FIG. 6 for a detailed comparison of color images.

Hereinafter, exemplary embodiments of a multispectral-based color image generating apparatus and method using a Bayer color filter array camera according to the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram showing the configuration of a preferred embodiment of a multi-spectrum based color image generating apparatus using a Bayer color filter array camera according to the present invention.

Referring to FIG. 1, a color image generating apparatus according to the present invention includes an image photographing unit 110, a channel generating unit 120, a color interpolation unit 130, and an image generating unit 140.

The image capturing unit 110 is a first image photographing the target object by an image pickup device having a Bayer RGB color filter array-based image sensor, and a single color broadband color filter is further installed on the image capturing device to capture the target object. 2 Acquire the video.

The image capturing unit 110 acquires the first image by the image capturing apparatus, and subsequently acquires the second image by the image capturing apparatus further provided with a monochromatic wideband color filter. The monochromatic wideband color filter is provided in the lens portion of the imaging device. In this case, the monochromatic wideband color filter may be manually installed in the lens unit of the imaging apparatus during continuous shooting, or may be automatically installed in the lens unit of the imaging apparatus continuously after the first image is acquired by the image capturing unit 110.

The channel generator 120 uses the color patterns of the second image determined based on the RGB color filter arrangement and the spectral response of the wideband color filter, respectively, to generate red, green, and blue color channels from the first and second images, respectively. Create

2 is a diagram illustrating a relative spectral response according to the wavelength of a color filter in the visible light band.

Referring to FIG. 2, in the case of the Bayer RGB color filter arrangement, the spectral response for the red component band corresponds to graph 5, the spectral response for the green component band corresponds to graph 3, and the spectral response for the blue component band corresponds to graph 1. do.

3 shows the relative spectral response of the Bayer RGB color filter and the spectral response of the wideband green (G) color filter in the visible light band.

The solid line in the graph of FIG. 3 is a graph showing the relative spectral response of the Bayer RGB color filter. The spectral responses for the red, green and blue channels, respectively, correspond to graph 5, graph 3 and graph 1 shown in FIG. Also, the dotted line is a graph showing the spectral response of the wideband green (G) color filter.

Referring to FIG. 3, the spectral response of the wideband green (G) color filter overlaps with the response of the red and blue channels of the Bayer RGB color filter. As a monochromatic wideband color filter, color components of R and B can be used, but it is effective to use a wideband green color filter to obtain color component information with a small number of images. This is because the G component is most sensitive to the human eye and is similar to the luminance component of the image.

4 is a diagram illustrating color patterns of a first image and a second image that are determined based on a spectral response of a Bayer RGB color filter array and a wideband green color filter.

Referring back to the graph of FIG. 3, when the light passes through the broadband green color filter, some of the R and B color components are also transferred to the CFA, so the light passing through the R pixel of the CFA is colored in the yellow (Y) band. With the same principle, light passing through pixel B of CFA represents the color component of the cyan (C) band. Of course, the light passing through the G pixel represents the color component of the same G band. Thus, two consecutive first and second images obtained with an imaging device using a wideband green color filter may be used for red (R), green (G), blue (B), yellow (Y), and cyan (C). Dog band components can be detected.

The color relationship of the light passing through the color filter of RGB and CMY described above may be summarized as in Equation 1 below.

Here, R is a red light source, G is a green light source, B is a blue light source, C is a cyan light source, M is a red purple light source, Y is a yellow light source, and W is a white light source including all band colors.

Referring back to FIG. 1, the channel generator 120 uses an R pixel sampled in a 1/4 of a first image without using a color filter to configure an R channel, and obtains a second channel obtained by using a wideband G filter. Use C pixels sampled 1/4 of the image. In the third equation of Equation 1, the R component can be simply calculated from the C pixel through the relationship between C and R.

Similarly, the channel generator 120 uses B pixels sampled at 1/4 in the first image without using the color filter to form B channels, and 1/4 at the second image obtained using the wideband G filter. Use sampled Y pixels. In the first equation of Equation 1, the Y component can be simply calculated from the B pixel through the relationship between B and Y. In the case of the G channel, the G component is calculated by averaging the G pixel values of the acquired first and second images.

The color interpolator 130 interpolates the color values of pixels to which color values are not given in the plurality of color channels. Since the other half of the pixels in the R-channel image cannot obtain information, it can be interpolated using simple image interpolation or conventional demosaicing. As a result, the empty pixel is calculated using four adjacent pixels. In this manner, the color interpolator 130 may obtain the interpolated pixels of the other half of the B channel image, and the same applies to the G channel.

As the interpolation method or demosaicing method used by the color interpolation unit 130, various existing methods such as linear interpolation, cubic interpolation, adaptive interpolation technique, regularization method, and POCS method are possible. In addition, it is effective to use a demosaicing method using information between RGB channels to improve the PSNR. Color interpolation can effectively reduce noise in channels that are most sensitive to the human visual system and close to luminance images.

Finally, the image generator 140 generates a color image corresponding to the target object by adding the RGB channels obtained above.

FIG. 5 is a flowchart illustrating an exemplary embodiment of a method of generating a multispectral-based color image using a Bayer color filter array camera according to the present invention.

The image capturing unit 110 is a first image photographing the target object by an image pickup device having a Bayer RGB color filter array-based image sensor, and a single color broadband color filter is further installed on the image capturing device to capture the target object. 2 images are acquired (S510).

As a monochromatic wideband color filter, a wideband G color filter is typically used. This is because the information of the color component can be obtained with a small number of images, and the G component is most sensitive to the human eye and is similar to the luminance component of the image.

The channel generator 120 uses the color patterns of the second image determined based on the RGB color filter arrangement and the spectral response of the wideband color filter, respectively, to generate red, green, and blue color channels from the first and second images, respectively. It generates (S520).

The color pattern of the second image includes Y pixels, G pixels, and C pixels. The channel generator 120 generates an R channel including an R pixel and a C pixel, a G channel including only a G pixel, and a B channel including a Y pixel and a B pixel, and generates an R channel using a color relationship to light. The color value corresponding to the C pixel at and the color value corresponding to the Y pixel at the B channel are obtained. The color value of each pixel of the G channel is determined by an average of the color values of the G pixels of the first image and the second image.

The color interpolator 130 interpolates the color values of pixels to which color values are not assigned in the plurality of color channels (S530). As the interpolation method or demosaicing method used by the color interpolation unit 130, various existing methods such as linear interpolation, cubic interpolation, adaptive interpolation technique, regularization method, and POCS method are possible.

The image generator 140 generates a color image corresponding to the target object by combining the plurality of color channels (S540).

Experiments were conducted to evaluate the performance of the present invention. In order to evaluate the performance of the present invention, it is necessary to know specific information about the CFA spectrum band of an actual commercial camera and to manufacture an external wideband G filter corresponding thereto. Therefore, in this experiment, the image which passed the wideband G filter was generated by simulation, and the color image generated by the existing demosaicing technique was compared with the color image generated by the present invention.

FIG. 6 is a diagram illustrating a result of comparing PSNRs for color images generated by the present invention and a conventional demosaicing technique for each RGB channel. For each method, the results are shown in the order of R channel, G channel, and B channel for each image.

Conventional demosaicing techniques used linear interpolation and Gunturk's method. The PSNR of the color image produced by the present invention is improved over the result using the linear interpolation method, but shows a lower value than the result using the Gunturk method. The reason is that since the conventional edge-directed interpolation method is applied as the interpolation method for each channel of RGB, PSNR value is lower than that of Gunturk's method which uses inter-channel information and repeatedly calculates. Therefore, if the inter-channel information is used for interpolation in the color image generating apparatus according to the present invention, the PSNR may be improved.

FIG. 7 is a diagram illustrating a part of a resultant image of Image 8 of FIG. 6 for a detailed comparison of color images.

(A) Image is an original image, (b) Image is a linear interpolation method, (c) Image is a Gunturk method and (d) is an image to which the method of the present invention is applied Drawing. In the image (d) of FIG. 7, it can be seen that the color overlap problem is solved in the high frequency component as compared with the conventional method.

The present invention can also be embodied as computer-readable codes on a computer-readable recording medium. A computer-readable recording medium includes all kinds of recording apparatuses in which data that can be read by a computer system is stored. Examples of the computer-readable recording medium include a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like, and may be implemented in the form of a carrier wave (for example, transmission via the Internet) . The computer readable recording medium can also be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is clearly understood that the same is by way of illustration and example only and is not to be taken by way of limitation in the embodiment in which said invention is directed. It will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the scope of the appended claims.

Claims (9)

A first image photographing a target object by an image pickup device equipped with a Bayer RGB color filter array-based image sensor and a wideband color filter of a single color are further installed on the image pickup device to acquire a second image photographing the target object. An image photographing unit;
Generating red, green, and blue color channels from the first and second images, respectively, using the color pattern of the second image determined based on the RGB color filter arrangement and the spectral response of the broadband color filter. A channel generator;
A color interpolator configured to interpolate color values of pixels to which color values are not assigned in the plurality of color channels; And
And an image generator configured to combine the plurality of color channels to generate a color image corresponding to the target object.
The channel generator includes a red channel including a red pixel of the first image and a cyan pixel of the second image, a green channel including an average value of color values of the green pixels of the first and second images, and the first channel. A blue channel including a blue pixel of an image and a yellow pixel of the second image is generated, and a color value corresponding to a cyan pixel of the red channel and a yellow pixel of the blue channel are generated using a color relationship to light. Color image generating apparatus characterized by calculating the color value. The method of claim 1,
And the image capturing unit is further provided with a wide band green color filter in the image capturing apparatus to obtain a second image capturing the target object. delete The method of claim 1,
The color interpolation unit interpolates color values of pixels to which color values are not assigned in the plurality of color channels by a demosaicing method using information between the red, green, and blue color channels. . A first image photographing a target object by an image pickup device equipped with a Bayer RGB color filter array-based image sensor and a wideband color filter of a single color are further installed on the image pickup device to acquire a second image photographing the target object. Imaging step;
Generating red, green, and blue color channels from the first and second images, respectively, using the color pattern of the second image determined based on the RGB color filter arrangement and the spectral response of the broadband color filter. Channel generation step;
A color interpolation step of interpolating color values of pixels to which color values are not assigned in the plurality of color channels; And
And an image generating step of generating a color image corresponding to the target object by combining the plurality of color channels.
In the channel generating step, a red channel including a red pixel of the first image and a cyan pixel of the second image, a green channel including an average value of color values of the green pixels of the first and second images, and the first channel. A blue channel including a blue pixel of one image and a yellow pixel of the second image is generated, and a color value corresponding to a cyan pixel of the red channel and a yellow pixel of the blue channel are generated by using a color relationship with light. Color image generation method comprising the step of calculating the color value. 6. The method of claim 5,
In the image capturing step, a wideband green color filter is further installed in the image capturing apparatus to obtain a second image photographing the target object. delete 6. The method of claim 5,
In the color interpolation step, a color image is generated by interpolating color values of pixels to which color values are not assigned in the plurality of color channels by a demosaicing method using information between the red, green, and blue color channels. Way. A computer-readable recording medium having recorded thereon a program for executing the color image generating method according to claim 5 on a computer.

Images