KR101921995B1 - image sensor for visible color and near infrared imaging and sensing method - Google Patents

Abstract

The present invention provides an image sensor for a visible light color image and a near infrared ray image, comprising: a first pixel for sensing light in a wavelength range corresponding to a first color of a visible light and near-infrared light; A second pixel for sensing light and near-infrared light in a wavelength range corresponding to a second color of the visible light; A third pixel that senses light in the entire wavelength region of the visible light and does not react to the near-infrared light; A fourth pixel for sensing light and near-infrared light in the entire wavelength range of the visible light; An adaptive pixel grouping adjustment unit that divides the first through fourth pixels into two or more groups according to an illumination environment; A multiple exposure adjuster that adjusts the exposure time for each group differently for the divided groups; And a pixel group connecting element for transmitting a signal for adjusting the exposure time of each group output by the multiple exposure adjuster to pixels belonging to each group.

Description

BACKGROUND OF THE INVENTION Field of the Invention [0001] The present invention relates to an image sensor and a sensing method for visible light color and near-

The present invention relates to an image sensor capable of simultaneously or selectively photographing a visible light color image and a near infrared ray image.

Among CMOS image sensors, CMOS image sensor is a CMOS image sensor. A CMOS image sensor includes a two-dimensional array of pixels having the function of receiving light and converting it into a voltage signal. Each pixel of the two-dimensional array converts a light signal incident on the pixel into photoelectrons using a photodiode, and then outputs a voltage signal proportional to the number of photoelectrons. The semiconductor material that makes CMOS image sensors is silicon (Si). The area of the wavelength? Of the light to which the silicon reacts is limited to less than 1100 nm, which is the limit for silicon to absorb light. That is, a CMOS image sensor has a visible light with a wavelength λ of less than 1100 nm and a near- (near infrared).

In the prior art, three types of color filters of R, G, and B are formed on a pixel array in the form of a Bayer pattern in order to capture a color image using a CMOS image sensor. The unit cell constituting the Bayer pattern is composed of two G (green) pixels at a diagonal position and one R (red) pixel and another B (blue) pixel at another diagonal position. The required number of 2x2 unit cells are repeated two-dimensionally to form a desired pixel array.

The CMOS image sensor adjusts the exposure time of a pixel, which receives light and converts it into signal electrons according to the brightness of incident light. Most CMOS image sensors include an auto exposure control (AEC) circuit that automatically controls the exposure time of a pixel to capture the best image in a variety of brightness situations. Conventional RGB Bayer pattern CMOS image sensors drive R, G, and B pixels at the same exposure time without distinguishing them. Therefore, such a CMOS image sensor includes only one AEC circuit.

The R, G, and B color filters deposited on the current pixel have material properties that pass the light in the infrared region in addition to the R, G, and B wavelength regions of the target visible light, respectively. Therefore, in order to obtain an accurate visible light color image using a pixel array of the RGB Bayer pattern, an IR cutoff filter is additionally disposed to block infrared light from entering the pixel array. Of course, in this case, the near infrared ray image can not be photographed.

Many surveillance cameras commonly used have a device for mechanically inserting or removing an IR filter into the optical path. In the daytime, the infrared ray cut filter is inserted into the optical path to capture the correct visible ray color image while the infrared ray is blocked. At night, the infrared ray cut filter is removed from the optical path and the near infrared ray light is used to illuminate the object. have.

The above-described prior art structure has a complicated structure, increases the cost, and increases the probability of failure occurrence due to repeated mechanical operations. Further, it is impossible to simultaneously photograph a visible light color image and a near infrared ray image.

US 3971065 US 20130242148 A1

The present invention provides an image sensor capable of simultaneously or selectively photographing a visible light color image and a near infrared ray image by using a visible light and a near infrared light simultaneously.

More specifically, the present invention provides an image sensor for both visible and near-infrared rays, which does not require an infrared cutoff filter to obtain a color image in the visible light region.

More specifically, the present invention aims to provide an image sensor having high sensitivity by using light in the entire region of the visible light and near-infrared region together.

According to an aspect of the present invention, there is provided an image sensor for a visible light color image and a near infrared ray image, comprising: a first pixel for sensing light in a wavelength region corresponding to a first color of a visible light and near-infrared light; A second pixel for sensing light and near-infrared light in a wavelength range corresponding to a second color of the visible light; A third pixel that senses light in the entire wavelength region of the visible light and does not react to the near-infrared light; A fourth pixel for sensing light and near-infrared light in the entire wavelength range of the visible light; An adaptive pixel grouping adjustment unit that divides the first through fourth pixels into two or more groups according to an illumination environment; A multiple exposure adjustment unit for adjusting the exposure time for each of the divided groups separately from each other; And a pixel group connecting element for transmitting a signal for adjusting the exposure time of each group output by the multiple exposure adjuster to pixels belonging to each group.

Extracting the near infrared ray image signal from a difference between the output signal of the fourth pixel and the output signal of the third pixel; Extracting the image signal of the first color from the difference between the output signal of the first pixel and the near-infrared image signal; Extracting the image signal of the second color from the difference between the output signal of the second pixel and the near-infrared image signal; A color extracting unit that extracts a video signal of a third color different from the first color and the second color from a difference between a sum of the video signal of the first color and the video signal of the second color, / IR image extracting unit.

The apparatus may further include a mode adjusting unit that adjusts the image sensor to a mode for simultaneously photographing a visible light image photographing mode, a near-infrared image photographing mode, a visible light image, and a near-infrared image.

The apparatus may further include an exposure attribute information recording unit for recording information on the group classified by the adaptive pixel grouping coordinator and the exposure time separately adjusted for each group by the multiple exposure coordinator.

According to another aspect of the present invention, there is provided a sensing method for a visible light color image and a near infrared ray image sensing, including: a first pixel sensing light in a wavelength region corresponding to a first color of a visible light and near-infrared light; A second pixel for sensing light and near-infrared light in a wavelength range corresponding to a second color of the visible light; A third pixel that senses light in the entire wavelength region of the visible light and does not react to the near-infrared light; And a fourth pixel for sensing light and near-infrared light in the entire wavelength range of the visible light, the sensing method comprising: adaptively converting the first to fourth pixels into two or more groups according to the illumination environment A step of discriminating; Determining exposure times to be applied to the two or more groups separately from each other; Exposing the first to fourth pixels with the determined exposure times and obtaining a pixel signal; And extracting the visible light color image and the near infrared ray image from the output signals of the first to fourth pixels.

Here, in the step of extracting the visible light color image and the near infrared ray image, the near infrared ray image signal is extracted from the difference between the output signal of the fourth pixel and the output signal of the third pixel; Extracting the image signal of the first color from the difference between the output signal of the first pixel and the near-infrared image signal; Extracting the image signal of the second color from the difference between the output signal of the second pixel and the near-infrared image signal; And extracting a video signal of a third color different from the first color and the second color from the difference between the sum of the video signal of the first color and the video signal of the second color and the output signal of the third pixel have.

The method may further include adjusting the visible light image mode or the near infrared image mode or the mode for simultaneously photographing the visible light image and the near infrared image according to the illumination environment.

The image sensor for visible light color image and near infrared ray image of the present invention having the above-described configuration can advantageously simultaneously or selectively capture a visible light color image and a near infrared ray image by using visible light and near infrared light simultaneously.

The image sensor for visible light color image and near infrared ray image of the present invention realizes a technique of not using an infrared ray cutoff filter in the outside in order to obtain a color image in the visible light region, thereby simplifying the image system using the image sensor, And the durability is improved.

The image sensor for visible light color image and near infrared ray image of the present invention has an advantage of increasing the sensitivity of photographing by using the light of all the visible light region and the near infrared region together.

1 is a block diagram illustrating an image sensor for a visible light color image and a near infrared ray image according to an embodiment of the present invention.
FIG. 2 is a circuit conceptual diagram illustrating an RBW _V C unit cell of a pixel array constituting the image sensor of FIG. 1, and an adaptive pixel grouping adjusting unit, a multiple automatic exposure adjusting unit, and a pixel group connecting device for grouping pixels to adjust an exposure time.
3 is a conceptual diagram of an MNRBW _V C pixel array in which the exposure time is adjusted by the adaptive pixel grouping adjuster, the multiple auto exposure adjuster, and the pixel group connecting element of FIG.
FIG. 4 is a flow chart illustrating an embodiment of a sensing method that can be performed by the image sensor of FIG. 1;
FIG. 5 is a flowchart illustrating a process of calculating a near-infrared ray image and a visible ray color image according to Equations 1 to 16. FIG.
FIG. 6 is a flowchart illustrating a noise filtering process for reducing signal noise before and after demodulating in FIG. 5;

The present invention relates to an image sensor, particularly, a CMOS image sensor capable of simultaneously capturing a color image and a near-infrared image in a visible light region using visible light and near-infrared light. Several types of color filter arrays having unique characteristics for light in the visible and near-infrared wavelength regions are arranged on the two-dimensional pixel array of the image sensor. The sensitivity of the pixel to light may vary greatly depending on the type of color filter formed on the pixel. An adaptive pixel grouping controller (hereinafter referred to as " adaptive pixel grouping ") is an apparatus for adaptively adjusting an exposure time of a pixel in order to overcome sensitivity differences according to the types of pixels and to obtain an optimal visible light color image and a near- We propose an image sensor that includes a part, a multiple auto exposure control part, and a pixel group connecting device. It receives visible light and near infrared rays as input, We design an array of color filters that can output simultaneously and propose a method of extracting visible light color image and near infrared image using each color channel signal.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

In describing the present invention, the terms first, second, etc. may be used to describe various elements, but the elements may not be limited by terms. Terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component.

It is to be understood that when an element is referred to as being connected or connected to another element, it may be directly connected or connected to the other element, but it may be understood that other elements may exist in between.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The singular expressions may include plural expressions unless the context clearly dictates otherwise.

It is to be understood that the term " comprising, " or " comprising " as used herein is intended to specify the presence of stated features, integers, But do not preclude the presence or addition of steps, operations, elements, components, or combinations thereof.

In addition, the shape and size of the elements in the drawings may be exaggerated in order to clarify the description of the invention.

[Example]

FIG. 1 illustrates an image sensor for a visible light color image and a near infrared ray image according to an embodiment of the present invention. The image sensor 100 includes a first pixel 111 for detecting light in a wavelength region corresponding to a first color of a visible light and near-infrared light; A second pixel 112 for sensing light in a wavelength region corresponding to a second color of the visible light and near-infrared light; A third pixel 113 that senses light in the entire wavelength range of the visible light and does not react to the near-infrared light; And a fourth pixel 114 for sensing light and near-infrared light in the entire wavelength range of the visible light. Here, the first color and the second color may be two colors arbitrarily selected from red (R), blue (B), and green (G) colors in the visible light region. For example, the first color may be red (R), and the second color may be blue (B). In this case, the green (G) color is calculated by subtracting the red (R) and blue (B) output signals from the entire visible light output signal. Hereinafter, the first color and the second color will be described as R and B in the description of the present embodiment. The pixel array 130 consists of a 2 × 2 RBW _V C unit cell 120 consisting of R pixels, B pixels, W _V pixels, and C pixels in a required number of two-dimensional repeats.

The image sensor 100 includes an adaptive pixel grouping adjustment unit 160 for dividing the first to fourth pixels into two or more groups according to an illumination environment, A plurality of automatic exposure adjusting units (180) for adjusting the exposure time of each group output from the plurality of automatic exposure adjusting units (160), and a pixel group connecting element 150).

The adaptive pixel grouping control unit 160 adaptively groups the first through fourth pixels 111 through 114 into two or more groups according to changes in the illumination environment . The adaptive pixel grouping adjustment unit 160 may be operated manually according to an instruction (or setting) outside the image sensor 100 or may be actively operated by self determination using image signal information within the image sensor 100 have.

When the adaptive pixel grouping adjustment unit 160 is operated manually, a user who knows information about an external illumination environment can set a grouping of pixels according to a photographing purpose of the image. The information on the illumination environment is information on a wavelength component and intensity of light emitted from the illumination device. The external command includes an instruction by an external illuminometer, a color thermometer, or a signal output from the color analyzer. As another example, in the case of natural sunlight illumination, it may be the time of photographing or date information.

When the adaptive pixel grouping adjuster 160 actively operates, the image signal information of the previous image frames is collected and analyzed according to the change of the illumination environment while the image is being photographed. Thus, at the appropriate moment, You can change the group. That is, the pixel group can be adjusted in real time in accordance with the change of the illumination environment during image capturing. As another example, the relative size of the output signal for each pixel output from the pixel array 130 in a state in which a light source that artificially illuminates for photographing is directly irradiated to the pixel array 130 or indirectly irradiated (reflected light is irradiated) The adaptive pixel grouping adjuster 160 may group the pixels prior to imaging by self analysis.

The multiple auto exposure control part 180 determines the exposure time of a pixel for each group according to the grouping of the pixels by the adaptive pixel grouping adjuster 160 Independently and automatically adjust independently of each other. The adaptive pixel grouping adjustment unit 160 and the multiple automatic exposure adjustment unit 180 exchange necessary information through a signal line or a signal line bundle 161. In FIG. 1, the adaptive pixel grouping adjustment unit 160 and the multiple automatic exposure adjustment unit 180 are expressly divided into two blocks according to their functions, but may be implemented by integrating them into one block to perform each function .

The pixel group connecting device 150 receives the control signal of the adaptive pixel grouping adjustment unit 160 and supplies the exposure time adjustment signal of each group generated by the multiple automatic exposure adjustment unit 180 to a corresponding group And has a function of connecting and transmitting pixels. The pixel group connecting element 150 is controlled by the adaptive pixel grouping adjusting unit 160 through the signal line bundle 170. The exposure time adjustment signals output from the multiple automatic exposure adjustment unit 180 are transmitted to the input terminal of the pixel group connecting element 150 through the signal line bundle 190. As shown, the pixel group connection element 150 may form a partial structure of a row driver 200 for driving the pixel array 130. [

The row driver 200 functions to simultaneously control the operation of pixels belonging to each row along each row in the pixel array 130. [

The illustrated column signal reading circuit 210 reads the signals of the pixels belonging to the row in a parallel manner as the operation of each row proceeds in the pixel array 130 It performs the function of issuing.

The image sensor 100 includes a color / IR image extracting unit 220 for extracting a color image and / or a near-infrared image in a visible light region using output signals of pixels photographed; A mode adjusting unit 230 for adjusting the image sensor 100 to a mode for simultaneously photographing a visible ray image photographing mode or a near infrared ray image photographing mode or a visible light image and a near infrared ray image; An exposure meta data writer for recording information on exposure time adjusted independently for each group by the multiple automatic exposure adjusting unit 180 and a group classified by the adaptive pixel grouping adjusting unit 160, (240). ≪ / RTI >

The color / IR image extraction unit 220 performs a function of extracting a color image signal and a near-infrared image signal of a visible light ray by calculating output signals of the column signal reading circuit 210.

The mode controller 230 selectively adjusts the mode of the image sensor 100 from among a mode for simultaneously photographing a visible light color image mode, a near-infrared image mode, a visible light color image, and a near-infrared image. The mode adjustment may be performed manually according to a command external to the image sensor 100 or may be actively performed based on self-determination using image signal information in the image sensor 100 or the like.

In FIG. 1, the color / IR image extracting unit 220 and the mode adjusting unit 230 are expressly divided into two blocks in accordance with their functions, but may be implemented by integrating them into one block to perform each function.

The image photographed by the image sensor according to an embodiment of the present invention can be regarded as a weighted image by differential exposure time with respect to light of a specific wavelength band. Generally, the image sensor performs a process of normalizing the output signal value of the pixels in consideration of the difference of the exposure time of each pixel group in itself, and proceeds with the subsequent image signal processing. However, when image data photographed by an image sensor according to the present invention is used in a further image analysis or image conversion, an operator may cause an undesired result due to the weight. In order to prevent such a side effect, the exposure attribute information recording unit 240 records attribute information (for example, the above-mentioned attribute information) in the header or the metadata format on the image data photographed by the image sensor 100 according to the present invention, Information about exposure time adjusted differently for each group by the multiple automatic exposure adjusting unit 180 and the groups classified by the adaptive pixel grouping adjusting unit 160).

FIG. 2 illustrates a unit cell 120 that generates a pixel array 130 in the image sensor 100 of FIG. In addition, the signal connection structure of the pixels of the unit cell 120, the pixel group connecting device 150, the adaptive pixel grouping adjusting unit 160, and the multiple automatic exposure adjusting units 180 are explicitly shown.

The unit cell 120 includes four pixels. The R pixel 111 and the B pixel 112 are positioned diagonally with respect to each other and the W _V pixel 113 and the C pixel 114 are positioned diagonally with respect to each other. However, it is needless to say that four different pixels of the unit cell can arbitrarily change their positions as needed.

An R pixel 111 is a pixel that reacts to an R (red) wavelength region of visible light and near-infrared light. A typical R pixel is made by placing an R filter on a pixel made up of a silicon semiconductor device. The R filter bandpasses the R wavelength region in the visible light region and transmits the band-pass filtered light to the pixel. The R filter that is deposited on the current pixel is the actual situation that transmits the infrared rays due to the characteristic of the filter material. Although the R filter passes the wavelength of the infrared region, the pixel reacts up to the near infrared region having a wavelength of less than 1100 nm due to the characteristics of the silicon semiconductor material, which is a material of the optical element of the pixel. Therefore, when no external infrared cutoff filter is used, the output signal obtained from the R pixel can be expressed by R + IR _R. Here, R is the output due to the red light of the visible light transmitted through the R filter, and IR _R is the output due to the near-infrared light transmitted through the R filter.

The blue pixel 112 is a pixel that reacts to the blue wavelength region of the visible light and near-infrared light. A typical B pixel is made by placing a blue filter on a pixel made up of a silicon semiconductor device. The B filter bandpasses the B wavelength region in the visible light region and transmits it to the pixel. The B filter that is deposited on the current pixel is the actual situation to transmit the infrared ray because of the characteristic of the filter material. Although the B filter passes the wavelength in the infrared region, the pixel reacts up to the near infrared region having a wavelength of less than 1100 nm due to the characteristics of the silicon semiconductor material which is the material of the optical element of the pixel. Therefore, when an external infrared cutoff filter is not used, the output signal obtained from the B pixel can be expressed as B + IR _B. Here, B is the output due to the blue light of the visible light transmitted through the B filter, and IR _B is the output due to the near-infrared light transmitted through the B filter.

W _V (visible white) pixel 113 is a pixel that responds to light in the entire wavelength region of the visible light region and does not react to light in the near-infrared region. And a W _V filter (a visible white filter) is disposed on the pixel. W _V filter is a filter that passes all light in the visible wavelength range and blocks the light in the infrared range. It is a filter made of a substance that is transparent to visible light and blocks infrared rays. Therefore, even when an external infrared cutoff filter is not used as in the present invention, the output signal obtained from a W _V pixel is only due to white visible light without a signal component due to near infrared rays, and can be expressed as W.

C (clear) pixel 114 is a pixel that responds to light in the visible and infrared full wavelength regions. It is a pixel with a C filter placed on the pixel. C filter is a filter made of a transparent material that transmits all the light in the incident visible light ray and the infrared light region. C filter passes the wavelength of the infrared ray region, the pixel reacts up to the near infrared ray region having a wavelength of less than 1100 nm due to the characteristic of the silicon semiconductor material which is the material of the optical element of the pixel. Therefore, when an external infrared cutoff filter is not used, the output signal obtained from the C pixel can be expressed as W + IR. Here, W is an output which is a white visible ray and IR is an output due to near-infrared rays.

In summary, the signals R _{+ P} , B _{+ P} , W _VP , and C _{+ P} output from the R pixel, the B pixel, the W _V pixel, and the C pixel are expressed by the following equations (1) to (4).

[Equation 1]

_{R + P} = R ₊ IR R

&Quot; (2) "

B _{+ P} = B + IR _B

&Quot; (3) "

W _VP = W

&Quot; (4) "

C _{+ P} = W + IR

In the symbols R _{+ P} , B _{+ P} , and C _{+ P} described above, the subscript + indicates that the visible light and the near infrared light signal are mixed, and in W _VP , the subscript V indicates that only the visible light signal exists. The subscript P indicates that the signal has been output from the pixel and has not yet been processed by subsequent signal processing.

The R pixel 111, the B pixel 112, the W _V pixel 113, and the C pixel 114 constituting the unit cell each have a large difference in the wavelength region of the light corresponding to the light. For example, in an environment that uniformly covers the entire wavelength range of visible light and illuminates objects with a weak light source, the average output signal values of W _V and C pixels are significantly larger than R and B pixels It is expected. In this case, when the exposure time of all the pixels is adjusted by using a single auto exposure control (AEC) circuit as in the conventional technique, the following problems are encountered. If the exposure time is adjusted so that the R pixel or the B pixel is sufficiently output, the output signals of the W _V pixel and the C pixel are saturated and the image information is lost. On the contrary, if the exposure time is adjusted so that the output signal of the W _V pixel and that of the C pixel are adjusted to be appropriate, the R pixel and the B pixel can reduce the number of photoelectrons generated by the light in the light receiving element of the pixel, It becomes a situation that is not great. That is, the signal to noise ratio (SNR) of the output signals of the R pixel or the B pixel is reduced, resulting in deterioration of image quality. Therefore, in such an illumination environment, it is preferable to adjust the exposure time of the pixel together with the R pixel and the B pixel as one group, and to adjust the exposure time of the pixel together with the W _V pixel and the C pixel as another group Do. At this time, the exposure time of the two groups is adjusted independently.

As another example, let us consider a case where the light contains a small amount of visible light and mainly includes near infrared light. In this case, the output signal value of the W _V pixel which does not react to the near-infrared ray becomes much smaller than the value of the R pixel, the B pixel, and the C pixel. In this case, if the exposure time is appropriately adjusted to the W _V pixel, the R pixel, the B pixel, and the C pixel are saturated, and the near infrared ray image information is lost. On the contrary, if the exposure time is adjusted in accordance with the R pixel, the B pixel, and the C pixel, the SNR of the output signal of the W _V pixel is reduced and the image quality of the visible light region is deteriorated. Therefore, in such an illumination environment, it is preferable to adjust the exposure time of the pixel together with the R pixel, the B pixel, and the C pixel as one group, and the W _V pixel as another group to adjust the exposure time of the pixel. At this time, the exposure time of the two groups is adjusted independently.

As another example, let us consider a case where the illumination mainly includes the near-infrared region and the R (red) wavelength region of the visible light rather than the light of the other wavelength region. It can occur in outdoor surveillance cameras at sunrise and sunset. In this case, the value of R pixel and the value of C pixel will be significantly larger than the value of B pixel or W _V pixel. Therefore, in such an illumination environment, it is preferable to adjust the exposure time of the pixel together with the R pixel and the C pixel as one group, and adjust the exposure time of the pixel with the B pixel and the W _V pixel as another group . At this time, the exposure time of the two groups is adjusted independently.

In the present invention, a method of independently adjusting the exposure time of each group by grouping pixels adaptively according to various illumination conditions is presented.

The adaptive pixel grouping control unit 160 has a function of adaptively grouping the first to fourth pixels 111 to 114 into two or more groups according to changes in the illumination environment . The multiple auto exposure control part 180 automatically adjusts the exposure time independently for each group independently of the grouping of the pixels by the adaptive pixel grouping adjuster 160.

Using the concrete example, the structures shown in Fig. 2 operate in conjunction to adjust the exposure time of the pixel. As a first example, we consider an environment that evenly covers the entire wavelength range of visible light and illuminates objects with a weak light source in the near infrared region. In this case, the adaptive pixel grouping adjustment unit 160 designates the R pixel 111 and the B pixel 112 as the first group, designates the W _V pixel 113 and the C pixel 114 as the second group, Group. The pixel grouping of the adaptive pixel grouping adjuster 160 may actively operate according to an external command, or may be actively operated by using information of a video signal output from the pixels of the image sensor.

The grouping information thus determined is transmitted to a multiple auto exposure control part 180 through a signal line or a signal line bundle 161. The first automatic exposure adjustment circuit AEC1 181 of the multiple auto exposure control part 180 is adapted to receive a plurality of pixel image signals output from the pixel array 130 of FIG. An appropriate value of the first group exposure time T1 is determined using the video signals of the R pixel 111 and the B pixel 112. [ And then outputs the first group exposure adjustment signal to the signal line 191. The second automatic exposure adjustment circuit AEC2 182 includes a plurality of W _V pixels 113 belonging to the second group and pixel signals of C pixels 114 belonging to the second group among the pixel image signals output from the pixel arrangement 130 of FIG. The appropriate value of the second group exposure time T2 is determined. And then outputs the second group exposure adjustment signal to the signal line 192.

On the other hand, the pixel group connecting device 150 receives a control signal from the adaptive pixel grouping adjustment unit 160 through the signal line bundle 170. The pixel group connecting element 150 generates an exposure time adjusting signal for each group which is generated by the multiple automatic exposure adjusting unit 180 and transmitted to the input terminal of the pixel group connecting element 150 through the signal line bundle 190 To the pixels belonging to the group correctly.

In this embodiment, a pixel group connecting device implemented using four mux elements will be described. The output signal line 191 of the first automatic exposure adjusting circuit AEC1 181 belonging to the multiple automatic exposure adjusting unit 180 and the output signal line 192 of the second automatic exposure adjusting circuit AEC2 182 belong to four mux elements MUX1 151, MUX2 152, MUX3 153, and MUX4 154, respectively. The outputs of the MUX1 151, the MUX2 152, the MUX3 153 and the MUX4 154 are sequentially supplied to the R pixel 111 and the B pixel 112 through the signal lines 141, 142, 143 and 144, A W _V pixel 113, and a C pixel 114, respectively. The adaptive pixel grouping adjustment unit 160 adjusts the input and output terminals of the MUX1 151, the MUX2 152, the MUX3 153 and the MUX4 154 through the signal lines 171, 172, 173, And transmits a control signal to each mux for adjusting the connection state. The output terminal of each mux is connected to one of the signal line 191 and the signal line 192 according to the control signal.

In the present example, the R pixel 111 and the B pixel 112 belong to the first group, and the outputs of the MUX1 151 and the MUX2 152 are connected to the signal line 191. [ That is, the exposure time of the R pixel 111 and the B pixel 112 is adjusted by the first automatic exposure adjusting circuit AEC1 181 such that the exposure time is T1. On the other hand, the W _V pixel 113 and the C pixel 114 belong to the second group, and the outputs of the MUX3 153 and the MUX4 154 are connected to the signal line 192. [ That is, the exposure time of the W _V pixel 113 and the C pixel 114 is adjusted by the second automatic exposure adjusting circuit AEC2 182 such that the exposure time is T2.

As another example, let us consider a case where the light contains a small amount of visible light and mainly includes near infrared light. In this case, the adaptive pixel grouping adjustment unit 160 assigns the R pixel 111, the B 112, and the C pixel 114 in which a video signal largely comes out in response to the near-infrared rays as a group and does not react to the near- The W _V pixel 113 in which the video signal is small is designated as another group. That is, the R pixel 111, the B pixel 112 and the C pixel 114 are designated as the first group, and the W _V pixel 113 is designated as the second group. Accordingly, the first automatic exposure adjustment circuit AEC1 181 of the multiple automatic exposure adjustment unit 180 may include a plurality of R pixels 111 belonging to the first group among the pixel image signals output from the pixel arrangement 130 of FIG. 1, The B pixel 112, and the C pixel 114, the appropriate value of the first group exposure time T1 is determined. And then outputs the first group exposure adjustment signal to the signal line 191. The second automatic exposure adjustment circuit AEC2 182 uses the video signals of the plurality of W _V pixels 113 belonging to the second group among the pixel video signals output from the pixel arrangement 130 of FIG. And determines an appropriate value of the exposure time T2. And then outputs the second group exposure adjustment signal to the signal line 192. On the other hand, the adaptive pixel grouping adjustment unit 160 sends control signals through the signal lines 171, 172, 173 and 174, and the outputs of the MUX1 151, the MUX2 152 and the MUX4 154 are connected to the signal line 191, (153) is connected to the signal line (192). The exposure adjustment signals of the signal line 191 are transmitted to the R pixel 111, the B 112 and the C pixel 114 through the signal lines 141, 142 and 144 and the exposure adjustment signals of the signal line 192 are transmitted to the signal line 143, And is connected to the W _V pixel 113 through the W _V pixel 113. That is, R pixels (111), B pixels (112), C exposure time of the pixel 114 is adjusted so that the T1 exposure time by a first automatic exposure adjusting circuit AEC1 (181), W _V pixel 113 Is adjusted by the second automatic exposure adjusting circuit AEC2 182 such that the exposure time is T2.

3 is a conceptual diagram of an MN pixel array in which the RBW _V C unit cell of FIG. 2 described above is repeatedly formed. A row structure 131 is formed by horizontally crossing N 2 × 2 RBW _V C unit cells 120 and a pixel array 130 and includes a signal line 141 connected to R pixels of each unit cell, A signal line 142 connected to pixels, a signal line 143 connected to W _V pixels, and a signal line 144 connected to C pixels. In addition, the row structure 131 includes a pixel group connecting element 150. The output of the four mux elements 151, 152, 153 and 154 constituting the pixel group connecting element 150 and the connection structure of the pixels of the unit cell are the same as in FIG. Also, signal connection structures of the four mux elements 151, 152, 153, and 154, the adaptive pixel grouping adjustment unit 160, and the multiple automatic exposure adjustment unit 180 are the same as in FIG. This row structure 131 is repeated M times to form an image sensor having an M x N pixel array 130.

2 and 3 illustrate an example in which the multiple autoexposure adjuster 180 includes two independent automatic exposure adjustment circuits AEC1 181 and AEC2 182. However, it goes without saying that the multiple automatic exposure unit 180 may be extended to include two or more independent automatic exposure adjustment circuits. In addition, the adaptive pixel grouping adjustment unit 160 can group the pixels belonging to the pixel array 130 into two or more groups.

1, image signal processing for obtaining a near-infrared ray image and a visible ray color image using the pixel signals output from the pixel array 130 by repeating the RBW _V C unit cell through the column signal reading circuit 210 Explain the method. The method of processing a video signal to be described later can be performed by the color / IR image extracting unit 220 of FIG.

In the case where the exposure time of each pixel is different as in the present invention, it is the first step of the image signal processing that the output signal from the pixel is normalized by the ratio of the exposure time. The exposure time of pixels belonging to the first group is denoted by T1, and the exposure time belonging to the second group is denoted by T2. For convenience, description will be made of normalizing the output signal of each pixel based on the exposure time T1 of the first group. However, even if the exposure time T2 of the second group is standardized and the formula is developed, an equivalent image is obtained as a result. However, in order to prevent saturation of the saturated video signals in the calculation process, it is preferable to standardize the exposure time of the group having the longest exposure time based on the exposure time.

The equations are developed by taking the case where the R pixel and the B pixel belong to the first group and the W _V pixel and the C pixel belong to the second group, respectively, without losing the generality. In this case, the exposure time of the R pixel and the B pixel is T1, and the exposure time of the W _V pixel and the C pixel is T2.

The signals R _{+ P} , B _{+ P} , W _VP , and C _{+ P} output from the R pixel, the B pixel, the W _V pixel, and the C pixel are as shown in Equations 1 to 4. When these signals are normalized to the first group exposure time T1, the normalized R pixel, B pixel, W _V pixel, and C pixel signals are expressed by the following equations (5) to (8).

&Quot; (5) "

&Quot; (6) "

&Quot; (7) "

&Quot; (8) "

The subscript N indicates the normalized value according to the exposure time ratio.

When a two-dimensional pixel array is formed as shown in FIG. 3 by repeating RBW _V C unit cells, R pixels, B pixels, W _V pixels, and C pixels are arranged one row at a time and one row at a time. According to the implementation, demosaicking may be performed to match all values of R _{+ N} , B _{+ N} , W _VN , and C _{+ N} to all (i, j) positions of the two-dimensional pixel array. In order to achieve the object of the present invention, it is necessary to perform demodulation in the present stage among various processes of the video signal processing. The demodulating method may be selected from several methods of demodulating according to various conventional techniques known in the art and used in combination. Therefore, in the present embodiment, a detailed description of the demodulating method will be omitted. The values of the respective color channels obtained for the (i, j) positions of the array by performing the di-mocking can be expressed by the following equations (9) to (12).

&Quot; (9) "

&Quot; (10) "

&Quot; (11) "

&Quot; (12) "

In the above equations, the subscript D represents a demosaicked signal value.

Obtaining the near infrared ray image and the visible ray color image means to obtain the IR value and the R, G, B values at every position (i, j) of the two-dimensional pixel array. From the equations (9) to (12), the near infrared ray image and the visible ray color image can be calculated as follows.

In the equations (11) and (12), the near IR IR value is calculated by the following Equation (13) as the difference between the C channel value and the W _V channel value.

&Quot; (13) "

Here, the adjustment parameter a _W is a correction parameter for correcting the color error due to the difference in spectral responsivity in the W (white) region where W _V pixel and C pixel are the entire wavelength region of the visible light, Variable. The control variable a _W value can be determined by analyzing the response of the two pixels by wavelength or by color tuning the actual image.

Next, R and B values are calculated according to the following equations (14) and (15) using Equations (9), (10) and (13).

&Quot; (14) "

&Quot; (15) "

In the above equations (14) and (15), the adjustment variables a _RIR and a _BIR are values determined in accordance with wavelength difference of near infrared rays of R pixel, B pixel and C pixel, respectively. The values of the pixels can be determined by analyzing the response of the near-infrared wavelengths or by performing color tuning of the actual image.

Next, the G value is calculated using Equation (11), Equation (14) and Equation (15) as follows.

&Quot; (16) "

In the expression (16), the adjustment variable a _R is a parameter for correcting the color error due to the difference in reactivity of the R pixel and the W _V pixel in the R (red) wavelength region of the visible light due to the wavelength. The control variable a _B is a parameter for correcting the color error due to the difference in the reactivity of the B pixel and the W _V pixel in the blue wavelength region of the visible light. A control variable _R and a _B may determine that value to the R pixel, B pixel, _V pixel W visible light wavelengths reaction analysis or measurement or color tuning of the real image of the. The IR _D (i, j) values calculated by Equation (13) are the near infrared ray images for all positions (i, j) of the two-dimensional pixel array, and R _D (i, j), B _D (i, j) and G _D (i, j) are directly color images.

FIG. 4 is a flowchart illustrating a sensing method for generating a visible light color image and a near-infrared image that can be performed by an image sensor according to an embodiment of the present invention. Referring to FIG.

The sensing method includes: S1) dividing the first to fourth pixels of FIGS. 1 to 3 into two or more groups adaptively according to an illumination environment; Separately determining exposure times to be applied to the two or more groups (S3); Exposing the first through fourth pixels at the determined exposure times and acquiring a signal of the pixel (S5); Extracting a visible ray color image and / or a near-infrared ray image from the output signals of the first to fourth pixels (S7); And outputting the image signal obtained in the step of extracting the image to the outside (S9).

According to the embodiment, the method may further include a step (S6) of adjusting the visible light ray imaging mode or the near infrared ray imaging mode or the mode for simultaneously photographing the visible light ray image and the near infrared ray image adaptively to the illumination environment. The signal output process of step S5 and the signal output process of step S9 are changed according to the mode selection of step S6.

The step S1 may be performed by the adaptive pixel grouping adjustment unit 160 of FIG. 1, and the step S3 may be performed by the multiple automatic exposure unit 180 of FIG.

The step S5 may be performed using the pixel array 130, the pixel group connecting element 150, the row driver 200, and the column signal reading circuit 210 of FIG. have.

The step S7 may be performed by the color / IR image extraction unit 220 of FIG.

The step S6 may be performed by the mode adjusting unit 230 of FIG.

FIG. 5 is a flowchart illustrating a method of calculating near-infrared images and color images according to Equations 1 to 16. Referring to FIG. The near infrared rays image IR _D and the color image R _D are generated starting from the signals R _{+ P} , B _{+ P} , W _VP and C _{+ P} signals output from the R pixel, B pixel, W _V pixel and C pixels of the two- , G _D , and B _D.

Showing a video output method, comprising: obtaining an output signal from the two-dimensional pixel arrangement of R pixels, B pixels, W _V pixels, the pixel C (S10); Normalizing the obtained output signals (S20); Demodulating the normalized signals (S30); Calculating a near infrared ray image from the demodulated signals (S40); Calculating an R image and a B image from the demodulated signals (S50); And calculating a G image from the demagnetized W _V signals and the R image and the B image (S60).

In the normalizing step S20, the normalization process according to the above-described equations (5) to (8) can be performed. In the demagnetizing step S30, the demodulating process according to Equations (9) to (12) may be performed.

In the application in which the resolution of the image is not an issue, the demosaicing step S30 is omitted and only the R, B, W, _V , and C pixel output signals belonging to one unit cell are used, And information of a color image of a visible ray can be extracted. In this case, however, the resolution of the image is reduced to about half in both the horizontal and vertical directions.

FIG. 6 is a flowchart illustrating a noise filtering process for reducing signal noise before and after demodulating in FIG. 5 according to another embodiment of the present invention.

The image calculation method according to the flowchart may further include performing a first filtering step S25 after the normalizing step S20 and performing a second filtering step S35 after the demagnetizing step S30 Is substantially the same as the case of Fig.

The image calculation method shown in FIG. 6 or FIG. 5 is performed by the color / IR image extraction unit 220 of FIG.

As described above, in the present embodiment, a CMOS image sensor using a two-dimensional pixel array in which unit cells of an RBW _V C pattern capable of simultaneously receiving visible light and near-infrared light are repeated is proposed. Then, a method of acquiring a near-infrared ray image and a visible ray color image using signals output from R pixels, B pixels, W _V pixels and C pixels of a two-dimensional pixel array is presented. In particular, by combining the adaptive pixel grouping adjuster 160, the multiple automatic exposure adjuster 180, and the pixel group connecting device 150 to adaptively group the pixels according to the illumination environment and independently adjust the exposure time of each group, We propose a method to obtain optimal near infrared ray image and color image with high SNR without loss of image information in lighting environment.

In this embodiment, one W _V pixel and one C pixel which are responsive to the entire wavelength region of the visible light are arranged for each unit cell. Thus, the sensitivity of the sensor to visible light increases as compared to image sensors that use conventional RGB Bayer pixel patterns with two G pixels per unit cell that react to the G wavelength region of the visible light.

In addition, the image sensor of the present invention does not need to use an infrared cut filter outside the image sensor in order to obtain a visible light color image.

Conventionally, in the case of an image sensor capable of selectively picking up a near infrared ray image and a visible ray color image, in order to switch a near infrared ray image photographing mode and a visible light color image photographing mode, Equipment.

On the other hand, if the image sensor of this embodiment is used, a mechanical device for inserting or removing the infrared cut filter into the optical path can be removed from the camera system. Accordingly, the camera system can be simplified, costs can be reduced, and durability can be improved.

In this case, when the mode adjusting unit 230 of FIG. 1 switches to the visible light color image photographing mode and the near infrared ray image photographing mode, the mode adjusting unit 230 may adjust the color / IR And adjusts the output format of the image signals that have undergone image processing and image processing of the image extracting unit.

In the case of the near infrared ray image taking mode, the near infrared rays signal value according to Equation (13) is calculated, and in the case of the visible light color image taking mode, the R, B, and G values according to Equations (14) to (16) are calculated. In other words, in the case of the near infrared ray imaging mode, the mode adjusting unit 230 performs the steps up to S40 in the flowchart of FIG. 6 or FIG. 5, and performs all the steps up to S60 in the color image shooting mode. The process can be controlled.

It should be noted that the above-described embodiments are intended to be illustrative, not limiting. In addition, it will be understood by those of ordinary skill in the art that various embodiments are possible within the scope of the technical idea of the present invention.

100: Image sensor
111: a pixel that detects light in the wavelength region corresponding to the first color of the visible light and near-infrared light
112: a pixel that detects light in the wavelength region corresponding to the second color of the visible light and near-infrared light
113: a pixel that senses light in the entire wavelength region of visible light and does not react to near-infrared light
114: a pixel that detects light and near-infrared light in the entire wavelength range of visible light
150: pixel group connecting device
160: an adaptive pixel group connecting device
180: Multiple auto exposure control part
200: row driver
210: a column signal reading circuit
220: color / IR image extraction part
230: Mode control part mode control part)
240: exposure meta data writer

Claims (7)

An R pixel for detecting light in a wavelength range corresponding to the R color of the visible light and near-infrared light;
A B pixel for detecting light and near-infrared light in a wavelength range corresponding to the B color of the visible light;
A Wv pixel that senses light in the entire wavelength range of the visible light and does not react to near-infrared light;
And a C pixel for sensing light in a whole wavelength region of the visible light and near-infrared light,
A configuration for simultaneously performing image capture for visible light and image capture for near infrared light,
An adaptive pixel grouping adjustment unit that divides the four pixels into two or more groups according to an illumination environment;
A multiple exposure adjustment unit for adjusting the exposure time for each of the divided groups separately from each other; And
And a pixel group connecting device for transmitting a signal for adjusting the exposure time of each of the plurality of groups outputted by the multiple exposure adjusting unit to pixels belonging to each group,
, ≪ / RTI &
Wherein the adaptive pixel grouping adjustment unit comprises:
The R pixel and the B pixel are grouped into one group, the Wv pixel and the C pixel are grouped into another group to adjust the exposure time of the pixel,
The R pixel, the B pixel, and the C pixel as one group, the Wv pixel as another group, and adjust the exposure time of the pixel,
The R pixel and the C pixel are grouped into one group, and the B pixel and the Wv pixel are grouped into another group to adjust the exposure time of the pixel. The method according to claim 1,
Extracting a near infrared ray image signal from a difference between the output signal of the C pixel and the output signal of the Wv pixel;
Extracting the R-color image signal from the difference between the output signal of the R pixel and the near-infrared image signal;
Extracting the B color image signal from the difference between the output signal of the B pixel and the near infrared ray image signal;
A color / IR image extracting unit for extracting a G color image signal of a visible ray from a difference between a sum of the R color image signal and the B color image signal and an output signal of the Wv pixel,
And an image sensor. The method according to claim 1,
And a mode adjuster for adjusting the image sensor to a mode for simultaneously photographing a visible light image photographing mode, a near-infrared image photographing mode, or a visible light image and a near-infrared image. The method according to claim 1,
An exposure attribute information recording unit for recording information on an exposure time adjusted by the adaptive pixel grouping adjustment unit and a different exposure adjustment unit for each group,
And an image sensor. delete delete delete

Images