WO2017034076A1

WO2017034076A1 - Image sensor, digital image processing apparatus using same, and image signal processing method therefor

Info

Publication number: WO2017034076A1
Application number: PCT/KR2015/011519
Authority: WO
Inventors: 민대성; 강세진; 송석범
Original assignee: (주)픽셀플러스; 에이치기술(주)
Priority date: 2015-08-27
Filing date: 2015-10-29
Publication date: 2017-03-02
Also published as: KR20170025087A

Abstract

The present technology provides a digital image processing apparatus capable of improving sensitivity using a panchromatic CMYP color filter array. The digital image processing apparatus of the present technology may allow light of the near infrared (NIR) region as well as light of a visible light region to enter an image sensor having a panchromatic CMYP color filter array, and simply and effectively separate and then remove an infrared ingredient from a color filtered image signal.

Description

Image sensor, digital image processing device and image signal processing method using same

The present invention relates to a digital image processing apparatus, and more particularly, to the image sensor having a panchromatic pixel, together with the light in the near infrared (NIR) region as well as the light in the visible region, and then infrared rays in the color filtered image signal. The present invention relates to a digital image processing apparatus capable of improving sensitivity by separating and removing components simply and effectively.

In general, an image sensor is a semiconductor device that converts an optical image into an electrical signal, and a charge couple device (CCD) image sensor and a CMOS (Complementaty MOS) image sensor are widely used. The image sensor uses a color filter array (CFA) implemented with R, G, and B (Red, Green, Blue) to realize color.

FIG. 1 is a diagram illustrating a pattern of a color filter array having a Bayer type.

A general color image is composed of three primary colors of R, G, and B, but most CCD or CMOS image sensors output only one component of three primary colors at each pixel position. In such an image sensor, the color filter for the luminance component is selected in the entire photocell in consideration of the characteristic having the highest resolution with respect to the luminance component (ie, green) of the human visual system. 50% of the cells are placed and the remaining chrominance components (ie, red and blue) are arranged in a repetitive cross form. Therefore, it can be seen that the number of color filter arrays shown in FIG. 1 has a ratio of 1: 2: 1 for R, G, and B.

FIG. 2 is a graph showing the spectral response of each filter of a general RGB CFA.

When the NIR light (NIR light) is incident on the image sensor using the RGB CFA as shown in FIG. 1 without filtering, the image signal has an advantage of increasing the size of the image signal, but the NIR light is incorporated into all the color pixels of the CFA. Problems may arise.

Therefore, there is a need for a method of receiving NIR light but not mixing NIR light with light in the visible light region.

An object of the present invention is to provide a digital imaging device capable of improving the sensitivity of an image sensor by injecting NIR light into an image sensor using a panchromatic cell and then separating it simply and effectively.

An image sensor according to an embodiment of the present invention uses a CMYP color filter array in which an image signal, which is an optical signal, is arranged with panchromatic (P) pixels, cyan (CY) pixels, magenta (MG) pixels, and yellow (YE) pixels. The CMYP color filter array includes a panchromatic (P) pixel, a cyan (CY) pixel, a magenta (MG) pixel, and a yellow (YE) pixel. P, YE, P, MG, the second line is CY, P, MG, P, the third line is P, MG, P, YE, the fourth line is MG, It may include a minimum repeating unit arranged in the order of P, CY, P.

The image sensor according to another embodiment of the present invention uses a CMYP color filter array in which an image signal, which is an optical signal, is arranged with panchromatic (P) pixels, cyan (CY) pixels, magenta (MG) pixels, and yellow (YE) pixels. The CMYP color filter array includes a panchromatic (P) pixel, a cyan (CY) pixel, a magenta (MG) pixel, and a yellow (YE) pixel. P, CY, P, MG in order, second line is CY, P, MG, P in order, third line is P, MG, P, YE in order, fourth line MG, It may include a minimum repeating unit arranged in the order of P, YE, P.

According to another embodiment of the present invention, an image sensor includes an image signal, which is an optical signal, and a CMYP color filter array in which panchromatic (P) pixels, cyan (CY) pixels, magenta (MG) pixels, and yellow (YE) pixels are arranged. The CMYP color filter array includes a panchromatic (P) pixel, a cyan (CY) pixel, a magenta (MG) pixel, and a yellow (YE) pixel. It is arranged in the order of CY, P, YE, P, the second line is arranged in the order of P, MG P, MG, the third line is arranged in the order of YE, P, CY, P, the fourth line is P, It may include a minimum repeating unit arranged in the order of MG, P, MG.

According to another embodiment of the present invention, an image sensor uses an image signal, which is an optical signal, as a CMYP color filter array in which panchromatic (P) pixels, cyan (CY) pixels, magenta (MG) pixels, and yellow (YE) pixels are arranged. The CMYP color filter array includes a panchromatic (P) pixel, a cyan (CY) pixel, a magenta (MG) pixel, and a yellow (YE) pixel. CY, P, YE, P arranged in order, second line P, MG P, MG ordered, third line CY, P, YE, P ordered, fourth line P, MG It may include a minimum repeating unit arranged in order, P, MG.

A digital image processing apparatus according to an embodiment of the present invention uses a CMYP color filter array in which an image signal, which is an optical signal, includes a panchromatic (P) pixel, a cyan (CY) pixel, a magenta (MG) pixel, and a yellow (YE) pixel. An image sensor for converting into an electrical analog video signal and outputting the converted video signal; And converts the analog image signal output from the image sensor into a digital image signal and interpolates the value of the panchromatic cell using the values of the panchromatic (P) pixels, and the value of the interpolated panchromatic cell and the corresponding cyan (CY). ) May include an image signal processor that extracts R, G, and B color components by using values of a pixel, a magenta (MG) pixel, or a yellow (YE) pixel.

Preferably, the CMYP color filter array of the image sensor includes panchromatic (P) pixels, cyan (CY) pixels, magenta (MG) pixels and yellow (YE) pixels, with the first line in the order of P, YE, P, MG. Arranged in the order of CY, P, MG, P in the second line, arranged in the order of P, MG, P, YE in the third line, and arranged in the order of MG, P, CY, P in the fourth line. It may include a minimum repeat unit.

Preferably, the CMYP color filter array of the image sensor comprises panchromatic (P) pixels, cyan (CY) pixels, magenta (MG) pixels, and yellow (YE) pixels, with the first line in the order of P, CY, P, MG. Arranged in the order of CY, P, MG, P in the second line, arranged in the order of P, MG, P, YE in the third line, and arranged in the order of MG, P, YE, P in the fourth line. It may include a minimum repeat unit.

Preferably, the CMYP color filter array of the image sensor comprises panchromatic (P) pixels, cyan (CY) pixels, magenta (MG) pixels, and yellow (YE) pixels, with the first line in the order CY, P, YE, P. Arranged in order of P, MG P, MG on the second line, YE, P, CY, P on the third line, and P, MG, P, MG on the fourth line It may include repeating units.

Preferably, the CMYP color filter array of the image sensor comprises panchromatic (P) pixels, cyan (CY) pixels, magenta (MG) pixels, and yellow (YE) pixels, with the first line in the order CY, P, YE, P. Arranged in order of P, MG P, MG on the second line, CY, P, YE, P on the third line, and P, MG, P, MG on the fourth line It may include repeating units.

In an image signal processing method of a digital image processing apparatus according to an embodiment of the present invention, an image signal, which is an optical signal, includes a panchromatic (P) pixel, a cyan (CY) pixel, a magenta (MG) pixel, and a yellow (YE) pixel. Converting into an electrical analog video signal using a CMYP color filter array; Converting the analog video signal into a digital video signal; Interpolating a value of a panchromatic cell using a value of the panchromatic (P) pixels in the digital image signal; And extracting R, G, and B color components by obtaining a difference between the values of the interpolated panchromatic cell corresponding to the values of cyan, magenta, and yellow pixels. have.

According to the present invention, the sensitivity of the image sensor can be improved by injecting NIR light into the image sensor using the panchromatic cell and then effectively separating it.

1 is a diagram illustrating a pattern of a color filter array in Bayer form by way of example.

3 is a block diagram showing the configuration of a digital imaging apparatus according to an embodiment of the present invention.

4 shows spectral characteristics of a dual band pass filter.

5A through 5D are exemplary views illustrating an arrangement structure of a minimum unit repeatedly arranged in an array form in the panchromatic CMYP CFA 120 of FIG. 3.

6 is a diagram illustrating a method of interpolating a panchromatic cell in the panchromatic cell interpolator of FIG. 3.

7A to 7D are diagrams for explaining a method of removing NIR components and separating R, G, and B color components from the color separation unit of FIG. 3 when using the panchromatic CMYP CFA of FIGS. 5A to 5D, respectively.

Hereinafter, some embodiments of the present invention will be described in detail through exemplary drawings. In adding reference numerals to the components of each drawing, it should be noted that the same reference numerals are assigned to the same components as much as possible even though they are shown in different drawings. In addition, in describing the embodiments of the present invention, when it is determined that the detailed description of the related well-known configuration or function interferes with the understanding of the embodiments of the present invention, the detailed description thereof will be omitted.

The digital imaging apparatus according to the present embodiment includes an image sensor 100 and an image signal processor 200.

The image sensor 100 converts an image signal (optical signal) input through a lens into an analog image signal, which is an electrical signal, and outputs the analog signal. This image sensor 100 includes a dual band pass filter 110 and a panchromatic CMYP CFA 120.

The dual band pass filter 110 filters the image signal incident through the lens and selectively transmits the image signal in the visible region and the image signal in the near infrared (NIR) region to enter the panchromatic CMYP CFA 120. 4 is a diagram illustrating spectral characteristics of the dual band pass filter 110. The dual band pass filter 110 selectively selects an image signal having a wavelength corresponding to a visible light region and an image signal having a wavelength of a near infrared (NIR) region. Permeate through. Therefore, the analog image signal output from the image sensor 100 includes infrared (IR) components as well as color (R, G, B) components.

The panchromatic CMYP CFA 120 is composed of a plurality of unit pixels in a two-dimensional array, and receives an image signal transmitted through the dual band pass filter 110 and converts it into an analog image signal having a CMYP pattern. In this case, the panchromatic CMYP CFA 120 includes panchromatic (P) pixels, cyan (CY) pixels, magenta (MG) pixels, and yellow (YE) pixels arranged in a two-dimensional array. The structure of the panchromatic CMYP CFA 120 will be described later in detail.

The image signal processor 200 converts an analog image signal output from the image sensor 100 into a digital image signal to interpolate and interpolate a value P ′ of a panchromatic cell corresponding to each color (CY, MG, YE) pixel. The R, G, and B color components from which the NIR component is removed are extracted by using the value P ′ of the extracted panchromatic cell and values of the corresponding color (CY, MG, YE) pixels. At this time, the image signal processor 200 calculates a difference between the value P ′ of the interpolated panchromatic cell and the value of the color CY, MG, YE pixel corresponding thereto, and removes the N, R, and G colors. Isolate the ingredients.

The image signal processing unit 200 is an analog-digital converter (ADC) 210 and an analog-digital converter 210 for converting an analog image signal output from the image sensor 100 into a digital image signal. The panchromatic cell interpolation unit 220 interpolating the value P ′ of the panchromatic cell corresponding to each color (CY, MG, YE) pixel using the values of the panchromatic (P) pixels in the output digital image signal and the interpolated And a color separator 230 that separates the R, G, and B color components by using a difference between the value P ′ of the panchromatic cell and the value of the color CY, MG, and YE pixels.

5A through 5D are exemplary views illustrating an arrangement structure of a minimum repeating unit (hereinafter, referred to as a 'minimal repeating unit') that is repeatedly arranged in an array form in the panchromatic CMYP CFA 120.

In the panchromatic CMYP CFA 120, the minimum repeating units of any one of FIGS. 5A to 5D are repeatedly arranged in two dimensions in the horizontal and vertical directions. The minimum repeating unit of the panchromatic CMYP CFA 120 includes a panchromatic (P) pixel, a cyan (CY) pixel, a magenta (MG) pixel, and a yellow (YE) pixel.

The panchromatic CMYP CFA 120 is composed of a plurality of pixels including a panchromatic (P) pixel, cyan (CY), magenta (Magenta, MG) pixels, and yellow (Yel, YE) pixels. The image signal transmitted through the dual band pass filter 110 is converted into an analog image signal having a corresponding CMYP pattern. At this time, the panchromatic (P) pixel transmits all the red (R), green (G), blue (B), and infrared (IR) signals from the incident image signal (visible light + NIR light) (P = R + G). + B + IR, and the cyan pixel outputs a color signal (CY = G + B + IR) transmitted through green (G), blue (B), and infrared (IR) from the incident image signal. In addition, the magenta (MG) pixel outputs a color signal (MG = G + B + IR) through which red (G), blue (B), and infrared (IR) are transmitted from the incident image signal, and an YE pixel. Outputs a color signal (YE = R + G + IR) through which red (R), green (G), and infrared (IR) are transmitted from the incident image signal.

The minimum repetition unit of FIGS. 5A-5D has a size of 4 × 4 pixels in common and includes eight P pixels, four MG pixels, two CY pixels and two YE pixels. In the minimum repeating unit of FIGS. 5A to 5D, the P pixels and the colors CY, MG, and YE pixels are alternately arranged along the horizontal direction and the vertical direction. That is, P pixels are arranged adjacent to the left and right and up and down of each color (CY, MG, YE) pixel, and the same color pixel or another color pixel is disposed adjacent to each other in the diagonal direction.

For example, in the minimum repeating unit of FIG. 5A, the first line L1 is arranged in the order of P, YE, P, MG, the second line L2 is arranged in the order of CY, P, MG, P, and the third line. (L3) is arranged in order of P, MG, P, YE, and the fourth line (L4) is arranged in order of MG, P, CY, P. That is, in the minimum repeating unit of FIG. 5A, MG pixels are arranged one by one in each line L1 to L4 to be arranged in a diagonal direction, and YE pixels and CY pixels are arranged in the upper left and lower right corners of the MG pixels arranged diagonally. One each in the area.

The minimum repeating unit of FIG. 5B is different from the positions of the CY pixel and the YE pixel in the first line L1 and the fourth line L4 in comparison with FIG. 5A. That is, two identical color (CY, YE) pixels are disposed in the upper left and lower right regions of the MG pixels arranged diagonally. For example, in the minimum repeating unit of FIG. 5B, the first line L1 is arranged in order of P, CY, P, MG, the second line L2 is arranged in order of CY, P, MG, P, and the third line. (L3) is arranged in the order of P, MG, P, YE, and the fourth line (L4) is arranged in the order of MG, P, YE, P.

In the minimum repeating unit of FIG. 5C, two MG pixels are arranged on the second line L2 and the fourth line L4 without MG pixels arranged in a diagonal direction, and the first line L1 and the third line ( One CY pixel and one YE pixel are disposed in L3). In this case, the CY pixel and the YE pixel are disposed opposite to each other in the first line L1 and the third line L3. For example, in the minimum repeating unit of FIG. 5C, the first line L1 is arranged in the order of CY, P, YE, P, the second line L2 is arranged in the order of P, MG P, MG, and the third line ( L3) is arranged in the order of YE, P, CY, P, and the fourth line L4 is arranged in the order of P, MG, P, MG.

The minimum repeating unit of FIG. 5D is disposed so that the positions of the CY pixel and the YE pixel are interchanged in the third line L3 as compared with the minimum repeating unit of FIG. 5C. That is, the CY pixel and the YE pixel are disposed at the same position in the first line L1 and the third line L3. For example, in the minimum repeating unit of FIG. 5D, the first line L1 is arranged in the order of CY, P, YE, P, the second line L2 is arranged in the order of P, MG P, MG, and the third line ( L3) is arranged in the order of CY, P, YE, P, and the fourth line L4 is arranged in the order of P, MG, P, MG.

FIG. 6 is a diagram illustrating a method of interpolating a value of a panchromatic cell in a panchromatic cell interpolator.

The panchromatic cell interpolator 220 interpolates a value P ′ of a panchromatic cell corresponding to each color CY, MG, and YE pixel by using values of neighboring panchromatic P pixels. For example, the panchromatic cell interpolator 220 may multiply the values of the four panchromatic P pixels closest to the panchromatic cell by a weight and assign them to the value of the panchromatic cell P '. In this case, the value P 'of the panchromatic cell may be allocated, for example, as follows.

P '= (P ₁ + P ₂ + P ₃ + P ₄ ) / 4 = R + G + B + IR

In this case, the weight may be determined linearly in proportion to the distance from the panchromatic P pixel.

In FIG. 6, only one panchromatic cell P 'is displayed, but interpolation is performed on all other panchromatic cells P'. Such an interpolation algorithm may be one of conventional non-adaptive interpolation methods (Nonadaptive Algorithms) or adaptive interpolation method (Adaptive Algorithms).

7A to 7D are diagrams for describing a method of removing NIR components and separating R, G, and B color components from the color separation unit of FIG. 3 when using the panchromatic CMYP CFAs of FIGS. 5A to 5D, respectively.

The value P ′ of the panchromatic cell interpolated by the panchromatic cell interpolator 220 may be expressed as in the following equation.

P '= R + G + B + IR

In addition, the values of the pixels CY, MG, and YE may be represented as follows.

CY = G + B + IR

MG = R + B + IR

YE = R + G + IR

Therefore, when subtracting the value of the corresponding color pixel from the value P ′ of each panchromatic cell interpolated by the panchromatic cell interpolation unit 220 as shown below, the color components of R, G, and B whose IR components are removed Only the value for can be obtained.

P'- CY = (R + G + B + IR)-(G + B + IR) = R

P'- MG = (R + G + B + IR)-(R + B + IR) = G

P'- YE = (R + G + B + IR)-(R + G + IR) = B

As described above, the color separator 230 subtracts the value of the color pixel corresponding to the corresponding panchromatic cell from the value P ′ of each panchromatic cell interpolated by the panchromatic cell interpolator 220 to R, G, and B in the digital image signal. The components are separated and their values extracted.

Thereafter, the method of processing the image signal using the extracted R, G, and B values may be made through the same methods as the conventional methods, and thus description thereof will be omitted.

The above description is merely illustrative of the technical idea of the present invention, and those skilled in the art to which the present invention pertains may make various modifications and changes without departing from the essential characteristics of the present invention.

Therefore, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention but to describe the present invention, and the scope of the technical idea of the present invention is not limited by these embodiments. The protection scope of the present invention should be interpreted by the following claims, and all technical ideas within the equivalent scope should be interpreted as being included in the scope of the present invention.

Claims

An image that converts and outputs an optical signal into an electrical analog video signal using a CMYP color filter array in which panchromatic (P) pixels, cyan (CY) pixels, magenta (MG) pixels, and yellow (YE) pixels are arranged. sensor; And

After converting the analog image signal output from the image sensor into a digital image signal, the value of the panchromatic cell is interpolated using the values of the panchromatic (P) pixels, and the value of the interpolated panchromatic cell and the corresponding cyan (CY) And an image signal processor for extracting R, G, and B color components using values of a pixel, a magenta (MG) pixel, or a yellow (YE) pixel.
The method of claim 1, wherein the CMYP color filter array

Panchromatic (P) pixels, cyan (CY) pixels, magenta (MG) pixels, and yellow (YE) pixels are arranged in order of P, YE, P, MG on the first line, and CY, P, MG on the second line. , P is arranged in order, the third line is arranged in the order of P, MG, P, YE, and the fourth line includes the minimum repeating unit arranged in the order of MG, P, CY, P Processing unit.
The method of claim 1, wherein the CMYP color filter array

Panchromatic (P) pixels, cyan (CY) pixels, magenta (MG) pixels, and yellow (YE) pixels are arranged in the order of P, CY, P, MG on the first line, and CY, P, MG on the second line. , P is arranged in order, the third line is arranged in the order of P, MG, P, YE, the fourth line includes a minimum repeating unit arranged in the order of MG, P, YE, P Processing unit.
The method of claim 1, wherein the CMYP color filter array

Panchromatic (P) pixels, cyan (CY) pixels, magenta (MG) pixels, and yellow (YE) pixels are arranged in the order of CY, P, YE, P on the first line, and P, MG P, Arranged in the order of MG, the third line is arranged in the order of YE, P, CY, P, the fourth line includes the minimum repeating unit arranged in the order of P, MG, P, MG Device.
The method of claim 1, wherein the CMYP color filter array

Panchromatic (P) pixels, cyan (CY) pixels, magenta (MG) pixels, and yellow (YE) pixels are arranged in the order of CY, P, YE, P on the first line, and P, MG P, Arranged in the order of MG, the third line is arranged in the order of CY, P, YE, P, and the fourth line includes the minimum repeating units arranged in the order of P, MG, P, MG Device.
The image signal processor of claim 1, wherein the image signal processor

R, G, and B color components are extracted by using the difference between the values of interpolated panchromatic cells and the values of cyan, magenta, and yellow pixels corresponding to the corresponding panchromatic cells. Digital image processing device.
An image that converts and outputs an optical signal into an electrical analog video signal using a CMYP color filter array in which panchromatic (P) pixels, cyan (CY) pixels, magenta (MG) pixels, and yellow (YE) pixels are arranged. In the sensor,

The CMYP color filter array

Panchromatic (P) pixels, cyan (CY) pixels, magenta (MG) pixels, and yellow (YE) pixels are arranged in order of P, YE, P, MG on the first line, and CY, P, MG on the second line. , P order, the third line is arranged in the order of P, MG, P, YE, and the fourth line includes the minimum repeating units arranged in the order of MG, P, CY, P.
An image that converts and outputs an optical signal into an electrical analog video signal using a CMYP color filter array in which panchromatic (P) pixels, cyan (CY) pixels, magenta (MG) pixels, and yellow (YE) pixels are arranged. In the sensor,

The CMYP color filter array

Panchromatic (P) pixels, cyan (CY) pixels, magenta (MG) pixels, and yellow (YE) pixels are arranged in the order of P, CY, P, MG on the first line, and CY, P, MG on the second line. The image sensor comprises the minimum repeating units arranged in order of P, in the order of P, MG, P, YE in the third line, and in the order of MG, P, YE, P in the fourth line.
An image that converts and outputs an optical signal into an electrical analog video signal using a CMYP color filter array in which panchromatic (P) pixels, cyan (CY) pixels, magenta (MG) pixels, and yellow (YE) pixels are arranged. In the sensor,

The CMYP color filter array

Panchromatic (P) pixels, cyan (CY) pixels, magenta (MG) pixels, and yellow (YE) pixels are arranged in the order of CY, P, YE, P on the first line, and P, MG P, An image sensor including a minimum repeating unit arranged in the order of MG, in the order of YE, P, CY, and P in the third line, and in the order of P, MG, P, and MG in the fourth line.
An image that converts and outputs an optical signal into an electrical analog video signal using a CMYP color filter array in which panchromatic (P) pixels, cyan (CY) pixels, magenta (MG) pixels, and yellow (YE) pixels are arranged. In the sensor,

The CMYP color filter array

Panchromatic (P) pixels, cyan (CY) pixels, magenta (MG) pixels, and yellow (YE) pixels are arranged in the order of CY, P, YE, P on the first line, and P, MG P, An image sensor comprising a minimum repeating unit arranged in the order of MG, the third line in the order of CY, P, YE, P, and the fourth line in the order of P, MG, P, MG.
Converting an image signal, which is an optical signal, into an electrical analog image signal using a CMYP color filter array in which panchromatic (P) pixels, cyan (CY) pixels, magenta (MG) pixels, and yellow (YE) pixels are arranged;

Converting the analog video signal into a digital video signal;

Interpolating a value of a panchromatic cell using a value of the panchromatic (P) pixels in the digital image signal; And

And extracting R, G, and B color components by obtaining a difference between values of the interpolated panchromatic cell corresponding to the values of cyan, magenta, and yellow pixels. Image signal processing method of the processing device.