KR20170055112A

KR20170055112A - Image processing apparatus

Info

Publication number: KR20170055112A
Application number: KR1020150157874A
Authority: KR
Inventors: 강세진; 송석범
Original assignee: 엠텍비젼 주식회사
Priority date: 2015-11-11
Filing date: 2015-11-11
Publication date: 2017-05-19

Abstract

An image processing apparatus is disclosed. The apparatus of the present invention is an apparatus that outputs an optical image signal to an electric image signal using a CMYP color filter array in which a pampetic (P) pixel, a cyan (CY) pixel, a magenta (MG) sensor; Interpolating the pixel values of the panchromatic pixels corresponding to the respective color pixels using the pixel values of the neighboring panchromatic (P) pixels, interpolating the pixel values of the interpolated panchromatic (P ') pixels and the corresponding pixel values of the color pixels And extracts infrared (IR) components using the pixel values of the interpolated panchromatic (P ') pixels and the pixel values of the RGB color components.

Description

[0001] IMAGE PROCESSING APPARATUS [0002]

The present invention relates to an image processing apparatus.

2. Description of the Related Art Generally, an image sensor is a semiconductor device that converts an optical image into an electric signal, and includes a charge couple device (CCD) image sensor and a complementary metal oxide semiconductor (CMOS) image sensor Is widely used. These image sensors use a color filter array (CFA) implemented as R, G, B (Red, Green, Blue) for color implementation.

The pixels of the image sensor have spectral characteristics in response to light in the visible and IR regions. 1A is a diagram exemplifying a pattern of a Bayer-type CFA, and FIG. 1B is a diagram exemplifying a pattern of RGB-IR CFA.

A typical color image is composed of three primary colors R, G, and B, but most CCD or CMOS image sensors output only one of the three primary colors at each pixel position. In this image sensor, a color filter for a luminance component is divided into a total photocell (photo (i. E., &Quot; green " cell, and the remaining chrominance components (i.e., red and blue) are arranged in a repeated crossing manner. Therefore, it can be seen that the number of color filter arrays shown in FIG. 1A has a ratio of 1: 2: 1 with respect to R, G,

2 is a graph showing transmission characteristics of each filter of a general RGB CFA.

When an image sensor using RGB CFA as shown in FIG. 1A receives light in a near IR (NIR) region without filtering, it is advantageous to increase the size of an image signal. However, A problem of incorporation into color pixels may occur.

Thus, there is a need for a method that does not allow NIR light to mix with light in the visible light region while also accommodating light in the NIR region.

As a conventional technique for this, there is a method of separating colors from NIR by using an optical dual band pass filter in a panchromatic cell. In this method, the NIR component is separated from each color signal. Since IR can be obtained by interpolating RGB in all the pixels of the image sensor, there is a problem that the IR accuracy is degraded.

As shown in FIG. 1B, a method of outputting a visible image and an IR image, which is composed of R, G, and B, is divided into a part that receives visible light and a part that receives IR, For an empty pixel, an image can be generated by interpolation. However, when pixels are constructed using interpolation, the interpolation performance is degraded if there is no correlation between the pixels used for interpolation, so that a low-resolution image can not be obtained.

SUMMARY OF THE INVENTION The present invention provides a digital image processing apparatus capable of effectively separating a NIR light into an image sensor using a fan-shaped chromatic cell and then separating the NIR light into a visible image and an IR image, thereby improving the sensitivity of the image sensor .

According to an aspect of the present invention, there is provided an image processing apparatus including an image processing unit configured to generate an optical image signal by arranging an optical image signal in a matrix such that a pixel of a pixel is formed of a p-chromatic pixel, a cyan pixel, a magenta pixel, An image sensor outputting an electrical image signal using the CMYP color filter array; Interpolating the pixel values of the panchromatic pixels corresponding to the respective color pixels using the pixel values of the neighboring panchromatic (P) pixels, interpolating the pixel values of the interpolated panchromatic (P ') pixels and the corresponding pixel values of the color pixels And extracts infrared (IR) components by using the pixel values of the interpolated punchmatic (P ') pixels and the pixel values of the RGB color components.

In one embodiment of the present invention, the CMYP color filter array is arranged in the order of P, YE, P and MG in the first line, in the order of CY, P, MG and P in the second line, P, MG, P, and YE, and the fourth line may include a minimum repeating unit arranged in order of MG, P, CY, and P, respectively.

In one embodiment of the present invention, the CMYP color filter array is arranged in the order of P, CY, P and MG in the first line, in the order of CY, P, MG and P in the second line, P, Y, and P in the order of MG, P, YE, and P in the fourth line.

In one embodiment of the present invention, the CMYP color filter array is arranged in the order of CY, P, YE and P in the first line, P, MGP and MG in the second line, YE, P, CY, and P, and the fourth line may include a minimum repeating unit arranged in the order of P, MG, P, and MG.

In one embodiment of the present invention, the CMYP color filter array is arranged in the order of CY, P, YE and P in the first line, P, MGP and MG in the second line, CY, P, YE, and P, and the fourth line may include a minimum repeating unit arranged in the order of P, MG, P, and MG.

In one embodiment of the present invention, the signal processing unit includes: an interpolator that interpolates pixel values of a pixel of a panchromatic pixel corresponding to a color pixel by multiplying pixel values of a plurality of neighboring panchromatic (P) pixels by a weight; An extracting unit for extracting RGB color components by subtracting the pixel value of the corresponding color pixel from the pixel value of the interpolated panchromatic pixel; And an IR detector for detecting an IR component by subtracting the sum of the RGB color components from the pixel value of the interpolated fan-shaped pixel.

The present invention has the effect of improving the sensitivity of the image sensor because the IR signal can be accurately output while simultaneously outputting the IR signal in addition to the color signal.

1A is a diagram exemplifying a pattern of a Bayer-type CFA, and FIG. 1B is a diagram exemplifying a pattern of RGB-IR CFA.
2 is a graph showing transmission characteristics of each filter of a general RGB CFA.
3 is a block diagram schematically illustrating a digital image processing apparatus according to an embodiment of the present invention.
4 is an exemplary view showing spectral characteristics of a dual bandpass filter.
FIGS. 5A to 5D illustrate an example of an arrangement structure of minimum units (minimum repetitive units) repeatedly arranged in an array form in a panchromatic CMYP CFA.
FIG. 6 is an exemplary diagram for explaining a method of interpolating the values of the panchromatic cell in the interpolator of FIG. 3;
FIGS. 7A to 7D are diagrams for explaining a method of removing NIR components and separating R, G, and B color components in the separator of FIG. 3 when using the panchromatic CMYP CFA of FIGS. 5A to 5D, respectively.

While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that the invention is not intended to be limited to the particular embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention.

Hereinafter, a preferred embodiment of the present invention will be described in detail with reference to the accompanying drawings.

3 is a block diagram schematically illustrating a digital image processing apparatus according to an embodiment of the present invention.

As shown in the drawings, the digital image processing apparatus of an embodiment of the present invention may include an image sensor 2 and a signal processing unit 3 that receive light transmitted through the lens 1.

The image sensor 2 can convert a video signal (optical signal) input through a lens into an analog video signal, which is an electrical signal, and output the analog video signal. This image sensor 2 may comprise a dual bandpass filter 21 and a panchromatic CMYP CFA 22.

The dual bandpass filter 2 filters an image signal input through the lens 1 to selectively transmit a video signal in a visible light region and a video signal in a near infrared region (NIR), and to input the image signal into a panchromatic CMYP CFA 22 .

4 is a diagram illustrating spectral characteristics of the dual band pass filter 21. The dual band pass filter 21 selectively outputs a video signal having a wavelength corresponding to a visible light region and a video signal having a wavelength of a near infrared region (NIR) . Accordingly, the analog image signal output from the image sensor 2 may include an infrared (IR) component as well as a color (RGB) component.

The panchromatic CMYP CFA 22 may be configured as a two-dimensional array of a plurality of units. The panchromatic CMYP CFA 22 may receive an image signal transmitted through the dual bandpass filter 21 and convert the analog image signal into a CMYP pattern analog image signal. At this time, the panchromatic CMYP CFA 22 may include a panchromatic (P) pixel, a cyan (CY) pixel, a magenta (MG) pixel and a yellow (YE) pixel arranged in a two-dimensional array. The structure of the panchromatic CMYP CFA 22 will be described later in detail.

The signal processing unit 3 converts the analog image signal output from the image sensor 2 into a digital image signal to interpolate the value P 'of the panchromatic cell corresponding to each color (CY, MG, YE) The RGB color component from which the NIR component has been removed is extracted using the value (P ') of the panchromatic cell and the value of the corresponding color (CY, MG, YE) pixel, and the extracted RGB color component and the interpolated panchromatic cell The IR component can be detected using the value P '. At this time, the signal processing unit 3 can separate the RGB color components from which the NIR component has been removed by obtaining the difference between the value P 'of the interpolated fan-shaped chromatic cell and the value of the corresponding color (CY, MG, YE) And can output the IR signal simultaneously from the difference between the RGB color component and the interpolated pseudo-chromatic cell value P '.

The signal processing unit 3 includes an analog-to-digital converter (ADC) 31 for converting an analog image signal output from the image sensor 2 into a digital image signal, An interpolator 32 interpolating the value P 'of the panchromatic cell corresponding to each color (CY, MG, YE) pixel using the values of the pixels of the panchromatic (P) A separator 33 for separating RGB color components using the difference between the values of the color (CY, MG, YE) and the color (CY, MG, YE) And an IR detector 34 for detecting the IR signal using the difference.

FIGS. 5A to 5D are exemplary views showing an arrangement structure of a minimal repeating unit (hereinafter referred to as 'minimum repeating unit') arranged in an array form in a panchromatic CMYP CFA 22. FIG.

The Panchromatic CMYP CFA 22 may be arranged such that the minimum repeating units of any one of Figs. 5A to 5D are two-dimensionally repeated in the horizontal and vertical directions. The minimal repetitive unit of this panchromatic CMYP CFA 22 may include a panchromatic (P) pixel, a cyan (CY) pixel, a magenta (MG) pixel, and a yellow (YE) pixel.

The panchromatic CMYP CFA 22 has a plurality of pixels comprising a two-dimensional array including pixels of a pantomachrome (P) pixel, a cyan (CY) pixel, a magenta (MG) pixel, and a yellow (YE) The image signal transmitted through the dual band pass filter 21 can be converted into an analog image signal of the corresponding CMYP pattern. At this time, the pixel of the fan-chromatic (P) is a color signal (P = R + G) obtained by transmitting all of red (R), green (G), blue (B) and infrared (IR) (CY = G + B + IR), which has transmitted green (G), blue (B) and infrared (IR) And the magenta (MG) pixel outputs a color signal (MG = G + B + IR) transmitted through red (G), blue (B) and infrared rays IR from an incident image signal, (YE = R + G + IR) in which red (R), green (G) and infrared (IR) are transmitted through the incident image signal.

The minimum repeating units of FIGS. 5A to 5D commonly have a size of 4 × 4 pixels, and may include 8 P pixels, 4 MG pixels, 2 CY pixels, and 2 YE pixels. 5A to 5D, the P pixel and the color (CY, MG, YE) pixels may alternately be arranged along the horizontal direction and the vertical direction. That is, P pixels are arranged adjacent to each other on the right and left and upper and lower sides of each color (CY, MG, YE) pixel, and the same color pixel or another color pixel in the diagonal direction.

5A, the first line L1 is arranged in the order of P, YE, P and MG, the second line L2 is arranged in the order of CY, P, MG and P, The fourth line L3 may be arranged in the order of P, MG, P, and YE, and the fourth line L4 may be arranged in the order of MG, P, CY, 5A, the MG pixels are arranged in each of the lines L1 to L4 so as to be arranged in a diagonal direction, and the YE pixels and the CY pixels are arranged in the diagonally arranged upper left and right lower regions Respectively.

5B, the positions of the CY and YE pixels in the first line L1 and the fourth line L4 are different from those in FIG. 5A. That is, two (CY, YE) pixels of the same color (CY, YE) may be arranged in the upper left and lower right regions of the MG pixels arranged in the diagonal direction, respectively. 5B, the first line L1 is arranged in the order of P, CY, P and MG, the second line L2 is arranged in the order of CY, P, MG and P, The fourth line L3 may be arranged in the order of P, MG, P, and YE, and the fourth line L4 may be arranged in the order of MG, P, YE,

The minimum repeating unit of FIG. 5C is a configuration in which MG pixels are not arranged in a diagonal direction but two MG pixels are arranged in a second line (L2) and a fourth line (L4), and the first line (L1) L3, respectively, one CY pixel and one YE pixel are arranged. At this time, in the first line (L1) and the third line (L3), the CY pixel and the YE pixel may be disposed opposite to each other. 5C, the first line L1 is arranged in the order of CY, P, YE and P, the second line L2 is arranged in the order of P, MGP and MG, and the third line L3 may be arranged in the order of YE, P, CY and P, and the fourth line L4 may be arranged in the order of P, MG, P and MG.

The minimum repeating unit of FIG. 5D can be arranged so that the positions of the CY and YE pixels in the third line L3 are shifted from each other, as compared with the minimum repeating unit of FIG. 5C. That is, in the first line (L1) and the third line (L3), the CY pixel and the YE pixel can be arranged at the same position. 5D, the first line L1 is arranged in the order of CY, P, YE and P, the second line L2 is arranged in the order of P, MG P and MG, and the third line L3 may be arranged in the order of CY, P, YE and P, and the fourth line L4 may be arranged in the order of P, MG, P and MG.

FIG. 6 is an exemplary diagram for explaining a method of interpolating the values of the panchromatic cell in the interpolator of FIG. 3;

The interpolator 32 can interpolate the value P 'of the panchromatic cell corresponding to each color (CY, MG, YE) pixel by using the values of the plurality of neighboring panchromatic (P) pixels. For example, the interpolator 32 may multiply the values of the four panchromatic (P) pixels closest to the panchromatic cell by a weight and assign it to the value of the panchromatic cell P '. In this case, the value P 'of the panchromatic cell can be assigned, for example, as shown in the following equation.

At this time, the weight value can be linearly determined in proportion to the distance from the pixel of the piechromatic (P).

In FIG. 6, only one panchromatic cell P 'is displayed, but it is obvious that interpolation can be performed for all other panchromatic cells P'. The interpolation algorithm may be any one of conventional non-adaptive interpolation algorithms or adaptive interpolation algorithms. However, the present invention is not limited thereto, and various algorithms may be used.

FIGS. 7A to 7D are diagrams for explaining a method for removing NIR components and separating R, G, and B color components in the separating unit of FIG. 3 when using the panchromatic CMYP CFA of FIGS. 5A to 5D, respectively.

A value P 'of the interpolation cell interpolated by the interpolator 32 can be expressed by the following equation.

In addition, the values of the respective colors (CY, MG, YE) can be expressed by the following equations.

Accordingly, if the value of the corresponding color pixel is subtracted from the value (P ') of each fan-shaped chromatic cell interpolated by the interpolation unit 32 as shown in the following equation, only the value of the RGB color component from which the IR component is removed can be obtained have.

In this way, the separation unit 33 subtracts the value of the color pixel corresponding to the corresponding panchromatic cell from the value (P ') of each of the panchromatic cells interpolated by the interpolation unit 32, thereby separating the RGB component from the digital image signal Then the value can be extracted.

Thereafter, the IR detecting unit 34 can detect the IR component by calculating the difference between the sum of the RGB obtained by the separating unit 33 and the value P 'of the interpolated panchromatic cell, and can be expressed by the following equation.

As described above, according to the signal processing unit 3 of the present invention, it is possible to accurately output the IR signal while simultaneously outputting the IR signal in addition to the color signal, thereby improving the sensitivity of the image sensor.

Hereinafter, a method of performing image processing using the extracted RGB signals and IR signals may be performed through the same methods as conventional methods, and a description thereof will be omitted.

While the invention has been shown and described with reference to certain preferred embodiments thereof, it will be understood by those skilled in the art that various changes and modifications may be made without departing from the spirit and scope of the invention as defined by the appended claims. Accordingly, the true scope of the present invention should be determined by the following claims.

1: Lens 2: Image sensor
3: signal processing unit 21: dual bandpass filter
22: CMYP CFA 31: ADC
32: interpolator 33: separator
34: IR detector

Claims

An image sensor for outputting an optical image signal to an electrical image signal using a CMYP color filter array in which a pampetic (P) pixel, a cyan (CY) pixel, a magenta (MG) pixel and an elo (YE) pixel are arranged; And
Interpolating the pixel values of the panchromatic pixels corresponding to the respective color pixels using the pixel values of neighboring panchromatic (P) pixels, and using the pixel values of the interpolated panchromatic (P ') pixels and the pixel values of the corresponding color pixels And a signal processing unit for extracting RGB color components and extracting infrared (IR) components by using pixel values of interpolated panchromatic (P ') pixels and pixel values of RGB color components.

The method of claim 1, wherein the CMYP color filter array comprises:
MG, P and YE in the order of PY, YE, P and MG in the first line, CY, P, MG and P in the order of CY, P, MG and P in the second line, Line includes a minimum repeating unit arranged in the order of MG, P, CY, and P, respectively.

The method of claim 1, wherein the CMYP color filter array comprises:
MG, P and YE in the order of P, CY, P and MG in the first line, CY, P, MG and P in the order of CY, P, MG and P in the second line, Line includes a minimum repeating unit arranged in the order of MG, P, YE, and P, respectively.

The method of claim 1, wherein the CMYP color filter array comprises:
P, Y, and P in the first line, P, MG P, and MG in the second line, YE, P, CY, and P in the third line, And a minimum repeating unit arranged in the order of P, MG, P, and MG.

The method of claim 1, wherein the CMYP color filter array comprises:
P, YE, and P in the first line, P, MG P, and MG in the second line, CY, P, YE, and P in the third line, And a minimum repeating unit arranged in the order of P, MG, P, and MG.

The signal processing apparatus according to claim 1,
A interpolator for interpolating the pixel values of the panchromatic pixels corresponding to the color pixels by multiplying the pixel values of the plurality of neighboring panchromatic (P) pixels by weights;
An extracting unit for extracting RGB color components by subtracting the pixel value of the corresponding color pixel from the pixel value of the interpolated panchromatic pixel; And
And an IR detector for detecting an IR component by subtracting the sum of the RGB color components from the pixel values of the interpolated panchromatic pixels.