KR20180024298A - image sensor for visible color and near infrared imaging - Google Patents

Abstract

The present invention relates to a CMOS image sensor for photographing a visible light color image and a near infrared image using visible light and near-infrared light. The CMOS image sensor of the present invention comprises: a first pixel for sensing light in a wavelength region corresponding to a first color of visible light, and near-infrared light; a second pixel for sensing light in a wavelength region corresponding to a second color of the visible light, and the near-infrared light; a third pixel for sensing light of the entire wavelength region of the visible light, and being not responsive to the near-infrared light; and a fourth pixel for sensing the light of the entire wavelength region of the visible light and the near-infrared light. The CMOS image sensor further comprises a multi-exposure adjustment unit for adjusting an exposure time for at least one pixel among the first to fourth pixels independently of other pixels when an image is photographed; a color/IR image extracting unit for calculating output signals of the first to fourth pixels, and extracting a color image signal of visible light and a near-infrared image signal; and a mode adjustment unit for adjusting a mode among a visible light image photographing mode, a near-infrared light image photographing mode, and a mode for simultaneously photographing a visible light image and a near-infrared light image in accordance with an external command.

Description

BACKGROUND OF THE INVENTION Field of the Invention [0001] The present invention relates to an image sensor for visible light color and near-

The present invention relates to an image sensor capable of simultaneously or selectively capturing a visible light color image and a near-infrared image, and more particularly, to a CMOS image sensor.

The CMOS image sensor is an imaging device manufactured using semiconductor CMOS manufacturing technology. A CMOS image sensor includes a two-dimensional array of pixels having the function of receiving light and converting it into a voltage signal. The two-dimensional arrangement of the pixels corresponds to a semiconductor film serving as an optical film commonly used in the past. Each pixel of the two-dimensional array converts a light signal incident on the pixel into photoelectrons using a photodiode, and then outputs a voltage signal proportional to the number of photoelectrons. Since the semiconductor making the CMOS image sensor is silicon (Si), the wavelength? Of the light to which the pixels of the CMOS image sensor reacts is limited to the visible light and near infrared light of less than 1100 nm, the limit of which silicon can absorb light.

In the prior art, three types of color filters of R, G, and B are formed on a pixel array in the form of a Bayer pattern in order to capture a color image using a CMOS image sensor. The unit cell constituting the Bayer pattern is shown in FIG. The unit cell includes two G (green) pixels, R (red) pixels, and B (blue) pixels arranged diagonally. The required number of 2-by-2 unit cells are repeated two-dimensionally to form a desired pixel array. The CMOS image sensor appropriately adjusts the exposure time that the pixel receives light and converts it into signal electrons according to the brightness of incident light.

Most CMOS image sensors include an auto exposure control (AEC) circuit that automatically controls the exposure time of a pixel to capture the best image in a variety of brightness situations. As schematically shown in FIG. 8, the conventional RGB Bayer pattern CMOS image sensor drives the R, G, and B pixels at the same exposure time without distinguishing them. Therefore, such a CMOS image sensor includes only one AEC circuit.

The R, G, and B color filters formed on the current pixel have material characteristics that pass light in the infrared region in addition to the R, G, and B wavelength regions of the target visible light ray. Therefore, in order to obtain an accurate color image using a pixel array of the RGB Bayer pattern, an IR cutoff filter is additionally disposed to block infrared light from entering the pixel array. Of course, in this case, you can not shoot near-infrared images. Currently, a color image camera using a CMOS image sensor inserts an infrared cut filter in a proper place in the optical path of a lens optical system that guides light to enter the sensor.

Many commonly used surveillance cameras have a mechanism for mechanically inserting or removing an IR filter into the optical path. In the daytime, the infrared ray cut filter is inserted into the optical path to capture an accurate visible ray color image while the infrared ray is blocked. At night, the infrared ray cut filter is removed from the optical path while illuminating the object with the near infrared ray light source, have.

The above-described prior art structure has a complicated structure, which increases the cost, increases the probability of occurrence of a failure, and can not simultaneously photograph a visible light color image and a near infrared ray image.

1. Korean Patent Publication No. 10-2011-0114315 2. US registered patent US 3,971,065

The present invention provides an image sensor capable of simultaneously or selectively photographing a visible light color image and a near infrared ray image by using a visible light and a near infrared light simultaneously.

More specifically, the present invention provides an image sensor for both visible and near-infrared rays, which does not require an infrared cutoff filter to obtain a color image in the visible light region.

More specifically, the present invention aims to provide an image sensor having high sensitivity by using light in the entire region of the visible light and near-infrared region together.

According to an aspect of the present invention, there is provided an image sensor for a visible light color image and a near infrared ray image, comprising: a first pixel for sensing light in a wavelength region corresponding to a first color of a visible light and near-infrared light; A second pixel for sensing light and near-infrared light in a wavelength range corresponding to a second color of the visible light; A third pixel that senses light in the entire wavelength region of the visible light and does not react to the near-infrared light; And a fourth pixel for sensing light and near-infrared light in the entire wavelength range of the visible light, and for adjusting the exposure time of at least one of the first to fourth pixels independently of other pixels when capturing the image, And an exposure adjustment unit.

Extracting the near infrared ray image signal from an output signal of the fourth pixel and an output signal of the third pixel; Extracting the image signal of the first color from the difference between the output signal of the first pixel and the near-infrared image signal; Extracting the image signal of the second color from the difference between the output signal of the second pixel and the near-infrared image signal; A color extracting unit that extracts a video signal of a third color different from the first color and the second color from a difference between a sum of the video signal of the first color and the video signal of the second color, / IR image extracting unit.

The image sensor may further include a mode adjusting unit for adjusting the image sensor to a mode for simultaneously photographing the visible light image mode, the near infrared image mode, or the visible light image and the near-infrared image according to an external command .

Here, the first color may be red, and the second color may be blue.

Here, the first color may be red and the second color may be green.

Here, the first color may be blue and the second color may be green.

Here, the multiple exposure adjustment unit adjusts the exposure time of the first pixel and the exposure time of the second pixel equally, and the exposure time of the third pixel is adjusted independently of the exposure time of the first pixel And the exposure time of the fourth pixel can be adjusted independently of the exposure time of the first pixel and the exposure time of the third pixel.

Here, the multiple exposure adjustment unit may include:

Wherein the exposure time of the first pixel and the exposure time of the second pixel are the same and the exposure time of the third pixel and the exposure time of the fourth pixel are the same, It can be adjusted independently.

Here, the image sensor may include an infrared ray blocking structure in a light incident path of the third pixel, and the infrared ray blocking structure may not exist in a light incident path of the first pixel, the second pixel, and the fourth pixel. do.

The image sensor for visible light color image and near infrared ray image of the present invention having the above-described configuration can advantageously simultaneously or selectively capture a visible light color image and a near infrared ray image by using visible light and near infrared light simultaneously.

The image sensor for visible light color image and near infrared ray image of the present invention realizes a technique of not using an infrared ray cutoff filter in the outside in order to obtain a color image in the visible light region, thereby simplifying the image system using the image sensor, And the durability is improved.

The visible light color image sensor and the near infrared ray image sensor of the present invention can advantageously increase the photographing sensitivity by using all the visible light and the near infrared light together.

1 is a block diagram illustrating an image sensor for a visible light color image and a near infrared ray image according to an embodiment of the present invention.
FIG. 2 is a circuit conceptual diagram showing the configuration of each unit cell and multiple exposure adjustment unit of the image sensor of FIG. 1;
3 is a circuit conceptual diagram showing another configuration of each unit cell and a multiple exposure adjustment unit constituting the image sensor of FIG.
FIG. 4 is a conceptual diagram of an M × N pixel array in which RBW _V C unit cells using the three independent exposure times of FIG. 2 are repeated.
FIG. 5 is a conceptual diagram of an M × N pixel array in which RBW _V C unit cells using the two independent exposure periods of FIG. 3 are repeated.
FIG. 6 is a flowchart illustrating a process of calculating a near-infrared ray image and a visible ray color image according to Equations 1 to 16. FIG.
FIG. 7 is a flowchart illustrating a noise filtering process for reducing signal noise before and after demodulating in FIG. 6;
8 is a conceptual diagram combining a unit cell in which three types of R, G, and B pixels are formed in the form of a Bayer pattern and an exposure adjustment circuit in order to capture a color image using an image sensor in the related art.

The present invention relates to a CMOS image sensor capable of simultaneously capturing a color image and a near-infrared image in a visible light region using visible light and near-infrared light. Several types of color filter arrays having unique characteristics for light in the visible and near-infrared wavelength regions are arranged on the two-dimensional pixel array of the image sensor. The sensitivity of the pixel to light may vary greatly depending on the type of color filter formed on the pixel. To overcome such sensitivity differences and to obtain an optimal visible light color image and a near infrared ray image, multiple automatic exposure adjustment (multiple exposure adjustment) is performed to independently adjust the exposure time of the pixel according to the type of color filter formed on the pixel auto exposure control) circuit. A method of designing an array of color filters capable of simultaneously outputting a visible light color image and a near infrared light image by receiving light mixed with a visible light and a near infrared light and extracting a visible light color image and a near infrared light image using each color channel signal present.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

In describing the present invention, the terms first, second, etc. may be used to describe various elements, but the elements may not be limited by terms. Terms are for the sole purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component.

It is to be understood that when an element is referred to as being connected or connected to another element, it may be directly connected or connected to the other element, but it may be understood that other elements may be present in between .

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular expressions may include plural expressions unless the context clearly dictates otherwise.

It is to be understood that the term " comprising, " or " comprising " as used herein is intended to specify the presence of stated features, integers, But do not preclude the presence or addition of steps, operations, elements, components, or combinations thereof.

In addition, the shape and size of the elements in the drawings may be exaggerated in order to clarify the description of the invention.

[Example]

FIG. 1 illustrates an image sensor for a visible light color image and a near infrared ray image according to an embodiment of the present invention. In the illustrated image sensor, four pixels constitute one unit cell, and only one unit cell is shown. However, since a large number of unit cells are arranged two-dimensionally, It is of course possible to form an image.

The image sensor 100 includes a first pixel 111 for detecting light in a wavelength range corresponding to a first color of a visible light and near-infrared light; A second pixel 112 for sensing light in a wavelength region corresponding to a second color of the visible light and near-infrared light; A third pixel 113 that senses light in the entire wavelength range of the visible light and does not react to the near-infrared light; And a fourth pixel 114 for sensing light and near-infrared light in the entire wavelength range of the visible light.

The illustrated image sensor 100 includes a multiple exposure adjustment unit 170 that adjusts the exposure time of at least one of the first to fourth pixels 111 to 114 independently of other pixels at the time of shooting. A color / IR image extracting unit 180 for calculating output signals of the first to fourth pixels to extract a color image signal and a near-infrared image signal of a visible light line; And a mode adjusting unit (190) for adjusting the image sensor to a mode for simultaneously photographing a visible light color image photographing mode, a near-infrared image photographing mode, a visible light color image and a near-infrared image according to an external command can do.

Although the multiple exposure adjusting unit 170, the color / IR image extracting unit 180, and the mode adjusting unit 190 are represented as independent blocks, two or more blocks of three blocks may be integrated into one control unit .

The external command as a reference for adjusting (setting) the mode by the mode adjusting unit 190 may be a direct mode setting command or information that is a reference for mode setting such as time or light amount. In the latter case, the reference information may be a means (e.g., a timer or a light amount sensor) provided in the image sensor. However, in the case where the image sensor is defined only in the configurations other than the means provided therein, The signal output from the means can also be seen as an external command.

For example, according to the timer output signal, the mode adjusting unit 190 may operate the image sensor in the color image photographing mode during the daytime, and operate the near infrared ray image photographing mode when the predetermined reference time elapses.

For example, when the amount of light exceeds a predetermined reference value, the mode adjusting unit 190 operates the image sensor in the color image photographing mode, and if the light amount does not exceed the predetermined reference value, the mode adjusting unit 190 operates in the near- .

FIG. 2 shows the configuration of each unit cell 120 and the multiple exposure adjustment unit 170 constituting the image sensor of FIG. One unit cell is composed of four pixels. In addition, the first color and the second color are two colors arbitrarily selected from three colors of red (R), blue (B), and green (G). For example, the first color may be red (R), and the second color may be blue (B). In this case, the green (G) color is calculated by subtracting the red (R) and blue (B) output signals from the entire visible light output signal. Hereinafter, the first color and the second color will be described as R and B in the description of the present embodiment.

As shown in FIG. 2, the image sensor shown in FIG. 2 is a two-dimensional array of two-dimensional pixels by repeating a 2-by-2 RBW _V C unit cell as shown in FIG.

The R pixel in this embodiment is a pixel that reacts with the R (red) wavelength region of the visible light and the near-infrared light.

A typical R pixel is made by placing an R filter on a pixel made up of a silicon semiconductor device. The R filter bandpasses the R wavelength region in the visible light region and transmits the band-pass filtered light to the pixel. The R filter constituted on the current pixel is the actual situation to transmit the infrared ray due to the characteristic of the filter material. Even though the R filter passes the wavelength of the infrared region, the pixel reacts only to the near infrared region whose wavelength is less than 1100 nm due to the characteristics of the silicon semiconductor material which is the material of the optical element of the pixel.

Therefore, when no external infrared cutoff filter is used, the output signal obtained from the R pixel can be expressed by R + IR _R. Here, R is the output due to the red light of the visible light transmitted through the R filter, and IR _R is the output due to the near-infrared light transmitted through the R filter.

The B pixel in this embodiment is a pixel that responds to the blue wavelength region of the visible light and near-infrared light.

A typical B pixel is made by placing a blue filter on a pixel made up of a silicon semiconductor device. The B filter bandpasses the B wavelength region in the visible light region and transmits it to the pixel. The B filter formed on the current pixel is the actual situation that transmits the infrared rays due to the characteristic of the filter material. Even though the B filter passes the wavelength of the infrared region, the pixel reacts only to the near infrared region whose wavelength is less than 1100 nm due to the characteristic of the silicon semiconductor material which is the material of the optical element of the pixel.

Therefore, when an external infrared cutoff filter is not used, the output signal obtained from the B pixel can be expressed as B + IR _B. Here, B is the output due to the blue light of the visible light transmitted through the B filter, and IR _B is the output due to the near-infrared light transmitted through the B filter. A W _V (visible white) pixel is a pixel that responds to light in the entire wavelength range of the visible light region and does not react to light in the near infrared region. And a W _V filter (a visible white filter) is disposed on the pixel. W _V filter is a filter that passes all light in the visible wavelength range and blocks the light in the infrared range. Visible light is a filter made of transparent infrared blocking material. Therefore, even when an external infrared cutoff filter is not used as in the present invention, the output signal obtained from the W _V pixel is only caused by white visible light without a signal component due to near infrared rays, and can be expressed as W. A C (clear) pixel is a pixel that reacts to light in the visible and infrared full wavelength regions. It is a pixel with a C filter placed on the pixel. C filter is a filter made of a transparent material that transmits all the light in the incident visible light ray and the infrared light region. Even though the transparent C filter passes the wavelength of the infrared region, the pixel reacts only to the near infrared region having a wavelength of less than 1100 nm due to the characteristic of the silicon semiconductor material which is the material of the optical element of the pixel.

Therefore, when an external infrared cutoff filter is not used, the output signal obtained from the C pixel can be expressed as W + IR. Here, W is an output which is a white visible ray and IR is an output due to near-infrared rays. In summary, the signals R _{+ P} , B _{+ P} , W _VP , and C _{+ P} output from the R pixel, the B pixel, the W _V pixel, and the C pixel are expressed by the following equations (1) to (3).

[Equation 1]

_{R + P} = R ₊ IR R

&Quot; (2) "

B _{+ P} = B + IR _B

&Quot; (3) "

W _VP = W

&Quot; (4) "

C _{+ P} = W + IR

In the symbols R _{+ P} , B _{+ P} , W _VP , and C _{+ P} described above, the subscript + indicates that the visible light and the near infrared light signal are mixed, and the subscript V indicates that only the visible light signal exists. The subscript P indicates that the signal has been output from the pixel and has not yet been processed by subsequent signal processing.

In FIG. 2, the R pixel and the B pixel are located diagonally to each other, and the W _V pixel and the C pixel are located diagonally to each other. However, it is needless to say that four different pixels of the unit cell can arbitrarily change their positions as needed.

The R pixel, the B pixel, the W _V pixel, and the C pixel constituting the unit cell each have a large difference in the wavelength region of the light corresponding to the light. For example, it is expected that the average output signal values of W _V and C pixels will be significantly larger than those of R pixels or B pixels in an environment in which objects are illuminated by a light source uniformly covering the entire wavelength range of visible light. Considering the case where the exposure time of all the pixels is adjusted to the same one using an AEC (auto exposure control) circuit as in the conventional technique, the following problems are encountered. If the exposure time is adjusted so that the R pixel or the B pixel is sufficiently output, the output signals of the W _V pixel and the C pixel are saturated and the image information is lost. On the contrary, if the exposure time is adjusted so that the output signal of the W _V pixel and that of the C pixel are adjusted to be appropriate, the R pixel and the B pixel can reduce the number of photoelectrons generated by the light in the light receiving element of the pixel, It becomes a situation that is not great. That is, the signal to noise ratio (SNR) of the output signals of the R pixel or the B pixel is reduced, resulting in deterioration of image quality.

As another example, when the illumination includes more near infrared rays than visible light, the output signal value of the W _V pixel which does not react to the near-infrared rays becomes much smaller than the value of the C pixel. In this case, if the exposure time is appropriately adjusted to the W _V pixel, the C pixel is saturated and the near infrared ray image information is lost. Conversely, if the exposure time is adjusted appropriately for the C pixel, the SNR of the output signal of the W _V pixel is reduced, Image quality deteriorates.

Therefore, when the pixel array is constituted by the unit cells of the RBW _V C pattern proposed by the present invention, independently adjusting the exposure time of the pixels using at least two independent AEC circuits minimizes the loss of image information, It is necessary to acquire a large image signal.

The illustrated multiple exposure adjustment unit 170 is composed of three AEC circuits. In this description, the exposure time of the R pixel is denoted by T1, the exposure time of the B pixel is denoted by T2, the exposure time of the W _V pixel is denoted by T3, and the exposure time of the C pixel is denoted by T4.

In general, when shooting images in most lighting environments, it can be expected that the average output signal of R pixels and the average output signal of B pixels are similar. Therefore, in order to reduce the number of signal lines and the number of AEC circuits for adjusting the exposure time within the two-dimensional pixel array, it is effective to equalize the exposure time T1 of the R pixel and the exposure time T2 of the B pixel. That is, the R pixel and the B pixel use the first AEC circuit (AEC1) to adjust the two exposure times equally to T1 = T2. The exposure time T3 of the W _V pixel is adjusted independently of T1 (= T2) using the second AEC circuit AEC2. The exposure time T4 of the C pixel is adjusted independently of the exposure time T1 (= T2) and the exposure time T3 using the third AEC circuit (AEC3). In this case, three AEC circuits should be provided. 2 shows an RBW _V C unit cell having three AEC circuits and independently adjusting T1 (= T2), T3, and T4.

As another example, the exposure time T3 and the exposure time T4 can be adjusted in the same manner for an image sensor used only in an illumination environment in which near infrared rays are hardly included. That is, two AEC circuits are provided and the exposure time T1 = T2 and the exposure time T3 = T4 are independently adjusted. This simplifies the signal line and external AEC circuit for adjusting the exposure time inside the two-dimensional pixel array. 3 shows an RBW _V C unit cell that adjusts T1 (= T2) using the first AEC circuit AEC1 and independently adjusts T3 (= T4) using the second AEC circuit AEC2 . That is, the multiple exposure adjustment unit 170 'of FIG. 3 is composed of two AEC circuits.

FIG. 4 is a conceptual diagram of an M × N pixel array in which RBW _V C unit cells using the three independent exposure times of FIG. 2 described above are repeatedly formed.

FIG. 5 is a conceptual diagram of an M × N pixel array in which RBW _V C unit cells using the two independent exposure periods of FIG. 3 described above are repeatedly formed.

Next, an image signal processing method for obtaining a near-infrared ray image and a visible ray color image in a two-dimensional pixel array in which RBW _V C unit cells are repeated is described. The image signal processing method described below can be performed by the color / IR image extracting unit 180 of FIG.

In the case where the exposure time of each pixel is different as in the present invention, it is the first step of the image signal processing that the output signal from the pixel is normalized by the ratio of the exposure time. B pixels, and B pixels when standardizing the signals R _{+ P} , B _{+ P} , W _VP , and C _{+ P} output from the R pixel, B pixel, W _V pixel, and C pixel based on the exposure time T1, W _V pixel and C pixel signals are expressed by the following equations (5) to (8).

&Quot; (5) "

&Quot; (6) "

&Quot; (7) "

&Quot; (8) "

The subscript N indicates the normalized value according to the exposure time ratio. In this description, a formula normalized with the exposure time T1 is developed. However, even if the exposure time T1, T2, T3, or T4 is standardized and the equation is developed, an equivalent image is obtained as a result.

When the two-dimensional pixel array is formed as shown in FIG. 4 or 5 by repeating the RBW _V C unit cell, R pixels, B pixels, W _V pixels, and C pixels are arranged one row at a time and one row at a time . According to the implementation, demosaicking may be performed to match all values of R _{+ N} , B _{+ N} , W _VN , and C _{+ N} to all (i, j) positions of the two-dimensional pixel array. In order to achieve the object of the present invention, it is necessary to perform demodulation in the present stage among various processes of the video signal processing. The demodulating method may be selected from several methods of demodulating according to various conventional techniques known in the art and used in combination. Therefore, in the present embodiment, a detailed description of the demodulating method will be omitted. The values of the respective color channels obtained for the (i, j) positions of the array by performing the di-mocking can be expressed by the following equations (9) to (12).

&Quot; (9) "

&Quot; (10) "

&Quot; (11) "

&Quot; (12) "

In the above equations, the subscript D represents a demosaicked signal value.

Obtaining the near infrared ray image and the visible ray color image means to obtain the IR value and the R, G, B values at every position (i, j) of the two-dimensional pixel array. From the equations (9) to (12), the near infrared ray image and the visible ray color image can be calculated as follows.

In the equations (11) and (12), the near IR IR value is calculated by the following Equation (13) as the difference between the C channel value and the W _V channel value.

&Quot; (13) "

Here, the adjustment parameter a _W is a correction parameter for correcting the color error due to the difference in spectral responsivity in the W (white) region where W _V pixel and C pixel are the entire wavelength region of the visible light, Variable. The control variable a _W value can be determined by analyzing the response of the two pixels by wavelength or by color tuning the actual image.

Next, R and B values are calculated according to the following equations (14) and (15) using Equations (9), (10) and (13).

&Quot; (14) "

&Quot; (15) "

In the above equations (14) and (15), the adjustment variables a _RIR and a _BIR are values determined in accordance with wavelength difference of near infrared rays of R pixel, B pixel and C pixel, respectively. The values of the pixels can be determined by analyzing the response of the near-infrared wavelengths or by performing color tuning of the actual image.

Next, the G value is calculated using Equation (11), Equation (14) and Equation (15) as follows.

&Quot; (16) "

In the above equation (16), the adjustment variable a _R is a parameter for correcting the color error due to the difference in the reactivity of the R and W _V pixels in the R (red) wavelength region. The control variable a _B is a parameter for correcting the color error due to the difference in the reactivity of the B pixel and the W _V pixel in the blue wavelength region of the visible light. A control variable _R and a _B may determine that value to the R pixel, B pixel, _V pixel W visible light wavelengths reaction analysis or measurement or color tuning of the real image of the. The IR _D (i, j) values calculated by Equation (13) are the near infrared ray images for all positions (i, j) of the two-dimensional pixel array, and R _D (i, j), B _D (i, j) and G _D (i, j) are directly color images.

6 is a flowchart illustrating a method of calculating a near infrared ray image and a color image according to Equations 1 to 16. Referring to FIG. The near infrared rays image IR _D and the color image R _D are generated starting from the signals R _{+ P} , B _{+ P} , W _VP and C _{+ P} signals output from the R pixel, B pixel, W _V pixel and C pixels of the two- , G _D , and B _D.

The illustrated image calculation method may be performed by the color / IR image extraction unit 180 of FIG.

Showing a video output method, comprising: obtaining an output signal from the two-dimensional pixel arrangement of R pixels, B pixels, the pixel W _V, C pixel (S10); Normalizing the obtained output signals (S20); Demodulating the normalized signals (S30); Calculating a near infrared ray image from the demodulated signals (S40); Calculating an R image and a B image from the demodulated signals (S50); And calculating a G image from the demodulated signals and the R image and the B image (S60).

In the normalizing step S20, the normalization process according to the above-described equations (5) to (8) can be performed. In the demagnetizing step S30, the demodulating process according to Equations (9) to (12) may be performed.

In an application in which the resolution of the image is not an issue, the demosaicing step S30 is omitted, and only the R, B, W, _V , and C four pixel output signals belonging to one unit cell are used, And information of a color image of a visible ray can be extracted. However, in this case, the resolution of the image is reduced to about half in both the horizontal and vertical directions.

FIG. 7 is a flowchart illustrating a noise filtering process for reducing signal noise before and after demodulating in FIG. 6 according to another embodiment of the present invention.

The image calculation method according to the flowchart may further include performing a first filtering step S25 after the normalizing step S20 and performing a second filtering step S35 after the demagnetizing step S30 Is substantially the same as the case of Fig. 6 above.

As described above, in the present embodiment, a CMOS image sensor using a two-dimensional pixel array in which unit cells of an RBW _V C pattern capable of simultaneously receiving visible light and near-infrared light are repeated is proposed. Then, a method of acquiring a near infrared ray image and a visible ray color image using the signals output from the R pixel, B pixel, W _V pixel, and C pixels of the two-dimensional pixel array is presented. In particular, we propose a method to obtain optimal near infrared ray image and color image with high SNR without loss of image information in wide light brightness category by combining RBW _V C pixel pattern and two or more independent automatic exposure time adjustment.

In this embodiment, one W _V pixel and one C pixel which are responsive to the entire wavelength region of the visible light are arranged for each unit cell. Thus, the sensitivity of the sensor to visible light increases as compared to image sensors that use conventional RGB Bayer pixel patterns with two G pixels per unit cell that react to the G wavelength region of the visible light.

In addition, the image sensor of the present invention does not need to use an infrared cut filter to obtain a visible light color image.

That is, in the image sensor of this embodiment, the infrared ray blocking structure exists in the light incidence path of the W _V pixel for visible light sensing except for the near-infrared light, but the R pixel, the B pixel, and the C There is no infrared blocking structure in the light incidence path of the pixel. To this end it is advantageous to employ than a structure, but also to place the infrared cut filter only the light incident path to the W pixels in the _V fixedly, W W _{V V} pixel filter not itself does not respond to the infrared has been deposited.

Conventionally, in the case of a hybrid type image sensor capable of selectively picking up a near infrared ray image and a visible ray color image, in order to switch a near infrared ray image photographing mode and a visible light color image photographing mode, Shall be provided.

On the other hand, if the image sensor of this embodiment is used, a mechanical device for inserting or removing the infrared cut filter into the optical path can be removed from the camera system. Accordingly, the camera system can be simplified, costs can be reduced, and durability can be improved.

In this case, the mode adjusting unit 190 of FIG. 1 does not perform optical adjustment such as insertion / removal of an IR cut filter from outside the image sensor in switching to the visible light color image pickup mode and the near infrared ray image pickup mode, And adjusts the output format of the image signals subjected to the image processing and image processing of the IR image extracting unit.

In the case of the near infrared ray image taking mode, the near infrared rays signal value according to Equation (13) is calculated, and in the case of the visible light color image taking mode, the R, B, and G values according to Equations (14) to (16) are calculated. In other words, in the case of the near infrared ray image capturing mode, the mode adjusting unit 190 performs the steps up to S40 in the flowchart of FIG. 6 or 7, and performs all the steps up to S60 in the case of the color image capturing mode. The process can be controlled.

It should be noted that the above-described embodiments are intended to be illustrative, not limiting. In addition, it will be understood by those of ordinary skill in the art that various embodiments are possible within the scope of the technical idea of the present invention.

100: Image sensor
111: a pixel that detects light in the wavelength region corresponding to the first color of the visible light and near-infrared light
112: a pixel that detects light in the wavelength region corresponding to the second color of the visible light and near-infrared light
113: a pixel that senses light in the entire wavelength region of visible light and does not react to near-infrared light
114: a pixel that detects light and near-infrared light in the entire wavelength range of visible light
170: multiple exposure adjustment section
180: Color / IR image extracting unit
190:

Claims (9)

A first pixel for sensing light in a wavelength region corresponding to a first color of the visible light and near-infrared light;
A second pixel for sensing light and near-infrared light in a wavelength range corresponding to a second color of the visible light;
A third pixel that senses light in the entire wavelength region of the visible light and does not react to the near-infrared light;
And a fourth pixel for sensing light and near-infrared light in the entire wavelength region of the visible light,
Further comprising a multiple exposure adjustment unit that independently adjusts the exposure time for at least one of the first to fourth pixels when the image is captured, independently of other pixels. The method according to claim 1,
Extracting the near-infrared image signal from the difference between the output signal of the fourth pixel and the output signal of the third pixel;
Extracting the image signal of the first color from the difference between the output signal of the first pixel and the near-infrared image signal;
Extracting the image signal of the second color from the difference between the output signal of the second pixel and the near-infrared image signal;
Extracting a video signal of a third color different from the first color and the second color from a difference between the sum of the video signal of the first color and the video signal of the second color and the output signal of the third pixel
And a color / IR image extracting unit. The method according to claim 1,
And a mode adjuster for adjusting the image sensor to a mode for simultaneously photographing a visible ray image photographing mode, a near-infrared image photographing mode, or a visible light image and a near-infrared image according to an external command. The method according to claim 1,
Wherein the first color is red and the second color is blue.
Image sensor. The method according to claim 1,
Wherein the first color is red and the second color is green.
Image sensor. The method according to claim 1,
The first color is blue and the second color is green,
Image sensor. The method according to claim 1,
Wherein the multiple exposure adjustment unit comprises:
Adjusting the exposure time of the first pixel and the exposure time of the second pixel equally, and adjusting the exposure time of the third pixel independently of the exposure time of the first pixel, Wherein the time adjusts independently of the exposure time of the first pixel and the exposure time of the third pixel. The method according to claim 1,
Wherein the multiple exposure adjustment unit comprises:
Wherein the exposure time of the first pixel and the exposure time of the second pixel are the same and the exposure time of the third pixel and the exposure time of the fourth pixel are the same, Independently adjustable image sensor. The method according to claim 1,
Wherein an infrared ray blocking structure is present in a light incident path of the third pixel, and an infrared ray blocking structure is not present in a light incident path of the first pixel, the second pixel, and the fourth pixel.

Images