WO2013027340A1

WO2013027340A1 - Imaging device

Info

Publication number: WO2013027340A1
Application number: PCT/JP2012/004917
Authority: WO
Inventors: 藤井　俊哉; 信一寺西; 圭一森
Original assignee: パナソニック株式会社
Priority date: 2011-08-24
Filing date: 2012-08-02
Publication date: 2013-02-28
Also published as: JP2014207493A

Abstract

To form a range image, an infrared laser (202) is provided to create a dot pattern by irradiating infrared light on a subject (200) through a coarse filter (203). Incident light from the subject (200) passes through a 2-band optical filter (205), which transmits the visible light wavelength region and the wavelength region in the vicinity of the irradiated infrared light, and forms an image on a CCD image sensor (210). The CCD image sensor (210) has a first pixel for infrared exposure, a second pixel for visible light exposure, and an overflow drain for simultaneously discharging the signal charges of the first and second pixels, and time-divides light into infrared exposure light of the first pixel and visible exposure light of the second pixel.

Description

Imaging device

The present invention relates to an imaging apparatus capable of realizing visible light imaging and infrared light imaging with a single image sensor.

There is a technique for calculating the distance between the subject and the imaging device based on an image obtained by irradiating the subject with infrared light and receiving the reflected light by the imaging device. The obtained image is called a distance image. As a method for obtaining a distance image, a stereo method, a pattern irradiation method, a TOF (time-of-flight) method, and the like are known.

A conventional image pickup apparatus realizes color image pickup by background light and distance image pickup by infrared light by a single CCD (charge coupled device) image sensor. For this reason, each of the RGB light receiving units that receive red (R), green (G), and blue (B) light, which are the three primary colors of visible light, and accumulate signal charges, A vertical overflow drain (vertical overflow drain) that has an interline transfer CCD image sensor with a planar array of IR (infrared) light receiving parts that receive light and accumulate signal charges, and sweeps out signal charges from all light receiving parts : VOD) and a horizontal overflow drain (LOD) that sweeps out only the signal charge of the IR light receiving section at high speed. The LOD includes a sweep gate connected to the IR light receiving unit and a drain region in which the signal charge of the IR light receiving unit is swept through the sweep gate. The LOD is operated after VOD, and then the signal charge is read from all the light receiving units, thereby independently controlling the visible light exposure time and the infrared light exposure time (see Patent Documents 1 and 2).

The optical filter provided in the RGB light receiving unit has a two-layer structure of each of the three primary color filters and an infrared light cut filter. The optical filter provided in the IR light receiving unit has a two-layer structure of an infrared light transmission / visible light cut filter and a transparent film for flattening the two-layer structure filter of the RGB light receiving unit. (See Patent Documents 1 and 2).

Also, the following driving method in the TOF type imaging apparatus is known. In other words, IR imaging is performed on the imaging target space every other frame scanning period while imaging visible light and infrared light every frame scanning period. A visible light image is generated for each frame scanning period, and the IR image signal obtained by imaging at the time of non-irradiation of IR pulse is subtracted from the IR image signal obtained by imaging at the time of IR pulse irradiation. A distance image excluding the influence of the IR component in the light is generated every one frame scanning period (see Patent Document 2).

JP 2008-5213 A JP 2008-8700 A

According to the above prior art, it is necessary to provide not only the VOD generally seen in the IR light receiving part but also an LOD that presses the area of the photodiode that is the photosensitive part of the image sensor. Therefore, the conventional technology has a problem that basic performance such as sensitivity, saturation, smear, etc. must be sacrificed.

In addition, since the above-described prior art is provided with an optical filter having a two-layer structure in each of the RGB light receiving unit and the IR light receiving unit, not only the device manufacturing difficulty level is increased, but also important for the sensitivity characteristics. There was a problem of obstructing the thinning of the optical layer.

An object of the present invention is to provide an imaging apparatus capable of realizing high-quality color image imaging using background light and high-precision distance image imaging using infrared light with a single inexpensive image sensor.

In order to achieve the above object, an imaging apparatus according to the present invention has irradiation means for irradiating a subject with infrared light, and spectral transmission characteristics for transmitting a visible wavelength region and a wavelength region near the irradiated infrared light. An image pickup apparatus comprising: a two-band optical filter that receives incident light from a subject; and an image sensor that converts an image formed through the two-band optical filter into an electrical signal. A first pixel that receives infrared light that has passed through the two-band optical filter and performs infrared light exposure; a second pixel that receives visible light that has passed through the two-band optical filter and performs visible light exposure; Discharge means for simultaneously discharging the signal charges of the first and second pixels, and performing infrared light exposure of the first pixel and visible light exposure of the second pixel in a time-sharing manner. .

In this specification, receiving light by the image sensor is referred to as “light reception”, and applying light to the image sensor to convert the light into an electrical signal is referred to as “exposure”.

According to the present invention, pixel signal charges can be discharged by a single discharging means for all pixels at once. In addition, since the infrared light exposure and the visible light exposure are performed in a time-sharing manner, the two-band optical filter and each pixel filter having a single layer structure are used in combination, so that the optical of visible light and infrared light can be combined. Mixing can be minimized. Therefore, it is possible to provide an imaging apparatus capable of realizing high-quality color image capturing using background light and high-precision distance image capturing using infrared light with a single inexpensive image sensor.

1 is a block diagram illustrating a configuration of an imaging apparatus according to a first embodiment of the present invention. It is a figure which shows the spectral transmission factor of the 2 band optical filter in FIG. It is a schematic block diagram of the CCD image sensor in FIG. It is a figure which shows the arrangement | sequence of the color filter arrange | positioned on the photodiode in FIG. FIG. 2 is a timing diagram illustrating an operation of the imaging apparatus in FIG. 1. It is a block diagram which shows the structure of the imaging device which concerns on the 2nd Embodiment of this invention. It is a circuit diagram which shows the structure of the pixel cell in the MOS image sensor in FIG. FIG. 8 is a schematic configuration diagram of the MOS image sensor in FIG. 6 in which the pixel cells in FIG. 7 are two-dimensionally arranged. FIG. 7 is a timing diagram illustrating an operation of the imaging apparatus in FIG. 6. It is a block diagram which shows the structure of the imaging device which concerns on the 3rd Embodiment of this invention. It is a schematic block diagram of the CCD image sensor in FIG. FIG. 11 is a timing diagram illustrating an operation of the imaging apparatus in FIG. 10.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

<< First Embodiment >>
FIG. 1 shows a configuration of an imaging apparatus according to the first embodiment of the present invention. In FIG. 1, reference numeral 200 denotes a subject, and 201 denotes background light illumination. 1 includes an infrared laser 202, a rough surface filter 203, an optical lens 204, a two-band optical filter 205, a CCD image sensor 210, an analog front-end (AFE) 220, and the like. , A V driver (VDR) 221 and an image signal processor (ISP) 230. The ISP 230 includes a camera preprocessing unit 240, a color camera processing unit / distance image calculation processing unit 250, a central processing unit (CPU) 260, and a timing generator (TG) 270. The camera preprocessing unit 240 includes a subtraction circuit 241 and an AE (automatic exposure) detection circuit 242.

According to the imaging apparatus of FIG. 1, the infrared laser 202 emits infrared light having a wavelength of 850 nm, and the irradiated infrared light passes through the rough surface filter 203 so that a coherent random image for obtaining a distance image is obtained. The subject 200 is irradiated with infrared light having a number of speckle patterns based on a speckle pattern (coherent random speckle pattern).

The infrared light reflected from the subject 200 forms an image on the CCD image sensor 210 via the optical lens 204 and the two-band optical filter 205. On the other hand, during the period when infrared light irradiation is not performed, only the reflected light from the subject 200 by the background light illumination 201 forms an image on the same CCD image sensor 210 via the same optical lens 204 and the two-band optical filter 205. To do.

FIG. 2 is a diagram showing the spectral transmittance of the two-band optical filter 205 in FIG. The two-band optical filter 205 is an optical filter having spectral transmission characteristics that transmits a visible light wavelength region and a wavelength region near 850 nm.

The CCD image sensor 210 converts the formed image into an electrical signal. A signal output from the CCD image sensor 210 is converted into a digital signal via the AFE 220. The digitized signal is subjected to processing such as flaw correction and noise removal in the camera preprocessing unit 240 in the ISP 230. In the camera preprocessing unit 240, the input signal is given to the subtraction circuit 241 and the AE detection circuit 242.

The color camera processing unit and the distance image calculation processing unit 250 output the result of camera processing such as edge enhancement and color adjustment as a color image, while comparing the coherent random speckle pattern from the infrared imaging result with the reference pattern. The distance image is output so that the distance can be calculated.

The CPU 260 controls the entire system of the imaging apparatus according to the register settings of the

blocks

240, 250, and 270. In particular, the CPU 260 subjects the output from the AE detection circuit 242 to time-axis filtering, etc., and independently calculates the visible light exposure time and the infrared light exposure time of the CCD image sensor 210 to obtain the information. Tell TG270.

The TG 270 generates a horizontal drive pulse to be directly supplied to the CCD image sensor 210, supplies a substrate discharge pulse and a vertical drive pulse to the CCD image sensor 210 via the VDR 221, and generates a drive signal to the infrared laser 202. To do. Among these, the substrate discharge pulse is a pulse for discharging the signal charge of the pixel, and is generated at a timing at which the shutter speed commanded by the CPU 260 is obtained. Thereby, natural AE control can be performed in both visible light exposure and infrared light exposure.

FIG. 3 is a schematic configuration diagram of the CCD image sensor 210 in FIG. The CCD image sensor 210 shown in FIG. 3 is an image sensor of progressive scan compatible and interline transfer system, and includes a photodiode (PD) 21, a vertical transfer unit 22, a horizontal transfer unit 23, and a charge detection. The unit 24 and the VOD 25 are included. The PDs 21 are arranged in a matrix and each constitute a pixel. The vertical transfer unit 22 has, for example, a configuration of four gates V1, V2, V3, and V4 per pixel, and is driven by four-phase pulses φV1, φV2, φV3, and φV4. Of these, φV1 is an all-pixel readout pulse. That is, the signal charge readout gate of the PD 21 is configured to also serve as the V1 gate of the vertical transfer unit 22. The horizontal transfer unit 23 has, for example, a two-phase (H1 and H2) configuration. Further, when the substrate discharge pulse φSub is given, the signal charges of all the pixels are discharged to the substrate all at once via the VOD 25. In FIG. 3, for convenience, the VOD 25 is described as being adjacent to the same surface as the PD 21, but in reality, the VOD 25 is adjacent to the PD 21 in the bulk direction (the depth direction of the semiconductor substrate). .

FIG. 4 shows an arrangement of color filters arranged on the PD 21 in FIG. Here, it is assumed that the pixel unit has a 2 × 2 unit array composed of a total of four pixels of R pixel, G pixel, B pixel, and IR pixel. So-called RGB color filters are arranged in the R pixel, G pixel, and B pixel. A visible light cut filter is disposed in the IR pixel. The color filter array is not limited to this example, and when m and n are each an integer of 2 or more, the pixel portion may have an m × n unit array.

FIG. 5 is a timing chart showing the operation of the imaging apparatus of FIG. Here, it is assumed that the rate of the vertical synchronization pulse VD is 60 frames per second (60 fps), imaging with irradiated infrared light is performed in odd frames, and visible light imaging with background light is performed in even frames. To do.

In the first frame, the signal charges of all the pixels are discharged by applying the substrate discharge pulse φSub, the infrared light exposure time 11 is started by the end of the application of the substrate discharge pulse φSub, and the infrared light irradiation time by the infrared laser 202 at the same time. 13 starts. On the other hand, since the background light illumination 201 is continuously lit, the reflected light from the subject 200 includes both infrared light and visible light. However, only the reflected light in the vicinity of 850 nm is received by the IR pixel by the two-band optical filter 205 and the visible light cut filter of the IR pixel, photoelectric conversion is performed, and the signal charge is applied to the IR pixel during the infrared light exposure time 11. Accumulated.

Next, by applying an H level to the all-pixel readout pulse φV1 at the beginning of the second frame, the signal charges of all the pixels including the IR pixel are read out to the vertical transfer unit 22 and the infrared light exposure time 11 ends. To do. At the same time, the infrared light irradiation time 13 by the infrared laser 202 ends.

Thereafter, an IR pixel signal is output as an infrared light exposure signal at the image sensor output timing 14 shown in FIG. 5 via the vertical transfer unit 22, the horizontal transfer unit 23, and the charge detection unit 24, and is processed in the subsequent stage. Migrate to At this time, RGB pixel signals are also output, but these RGB pixel signals are not processed in the subsequent stage.

After reading the signal charges of all the pixels in the second frame, the signal charges of all the pixels are discharged by applying the substrate discharge pulse φSub so that the necessary exposure time is reached, and the visible light exposure time 12 is set by the end of the application of the substrate discharge pulse φSub. Start. At this time, since the infrared light irradiation by the infrared laser 202 is stopped, the reflected light from the subject 200 does not include the irradiated infrared light component but includes only the background light component. However, the background light may contain some IR components in addition to the visible light component. The RGB color filter in the RGB pixel has a slight transmission characteristic with respect to the IR component. Therefore, in the R pixel, the R component of visible light and a slight IR component are received by the two-band optical filter 205 and the R color filter, and photoelectric conversion is performed, and a signal is transmitted to the R pixel during the visible light exposure time 12. Charge is accumulated. In the G pixel, the 2-band optical filter 205 and the G color filter receive the G component of visible light and a slight amount of IR component to perform photoelectric conversion, and signal charges are applied to the G pixel during the visible light exposure time 12. Accumulated. In the B pixel, the 2-band optical filter 205 and the B color filter receive the B component of visible light and a slight amount of IR component to perform photoelectric conversion, and signal charge is applied to the B pixel during the visible light exposure time 12. Accumulated. In the IR pixel, the two-band optical filter 205 and the visible light cut filter receive only a small amount of IR component in the background light to perform photoelectric conversion, and the signal charge is applied to the IR pixel during the visible light exposure time 12. Is accumulated.

Next, by applying an H level to the all-pixel readout pulse φV1 at the beginning of the third frame, the signal charges of all the pixels are read out to the vertical transfer unit 22 and the visible light exposure time 12 ends.

Thereafter, the signals of all the pixels including the RGB pixels and the IR pixels are subjected to visible light exposure through the vertical transfer unit 22, the horizontal transfer unit 23, and the charge detection unit 24 at the image sensor output timing 15 shown in FIG. Output as a signal.

Further, the operation of the CCD image sensor 210 is repeated, and the infrared light exposure and the visible light exposure are alternately performed in a time division manner so that the infrared light exposure time 11 and the visible light exposure time 12 appear every other frame. Done.

Here, the AE operation based on the output signal of the CCD image sensor 210 will be described. As described above, the pixel signal obtained at the infrared light exposure time 11 of the odd frame is output at the timing 14 in the even frame. At this time, only the output signal of the IR pixel is detected by the AE detection circuit 242, and the output is read by the CPU 260. The CPU 260 determines the shutter speed of the next odd frame through the time filter using only the detection result of the IR pixel signal output from the CCD image sensor 210 for each even frame.

On the other hand, the pixel signal obtained at the visible light exposure time 12 of the even frame is output at the timing 15 in the odd frame. At this time, the output signal of the RGB pixel is detected by the AE detection circuit 242 and the output is read by the CPU 260. The CPU 260 determines the shutter speed of the next even frame through the time filter using only the detection result of the RGB pixel signal output from the CCD image sensor 210 for each odd frame.

As described above, as if two cameras, a 30 fps irradiation infrared light imaging camera and a 30 fps background light color imaging camera, were installed using one 60 fps CCD image sensor 210, respectively. Independent AE control is realized.

Finally, the RGB pixel signals received as reflected light from the subject 200 during the visible light exposure time 12 are subjected to white balance processing, contour enhancement processing, and the like by the color camera processing unit and the distance image calculation processing unit 250. Output as an image. On the other hand, the IR pixel signal received as reflected light of the coherent random speckle pattern from the subject 200 during the infrared light exposure time 11 is collated with the reference pattern of the speckle pattern stored in advance, and from the address shift, the color camera It is calculated by the processing unit and the distance image calculation processing unit 250, converted into a distance image, and output.

As described above, according to the present embodiment, pixel signal charges can be discharged by the VOD 25 for all the pixels at once. In addition, since the infrared light exposure and the visible light exposure are performed in a time-sharing manner, the two-band optical filter 205 and each pixel filter having a single-layer structure on the CCD image sensor 210 can be used together. Optical mixing with infrared light can be minimized. Therefore, high-quality color image capturing using background light and high-accuracy distance image capturing using infrared light can be realized with an inexpensive system.

Further, in order to control the infrared light exposure time 11 and the visible light exposure time 12 independently of each other, the output of the AE detection circuit 242 is divided into infrared light and visible light, and each is independently controlled by AE. Therefore, since the light sources are different from each other, it becomes possible to control the infrared light exposure time 11 of the IR pixel and the visible light exposure time 12 of the RGB pixel, which are greatly different from each other, to the optimum exposure time, as if it were a measurement with the visible camera. It is possible to further improve the image quality of visible image capturing and the accuracy of distance image capturing so that the two cameras, the distance camera, operate simultaneously.

Further, the signal charges of all the pixels are read out so as to determine the end of the visible light exposure time 12 from the application timing of the substrate discharge pulse φSub for discharging the signal charges of all the pixels so as to determine the start of the visible light exposure time 12. In the period up to the application timing of the all-pixel readout pulse φV1, since the infrared light irradiation by the infrared laser 202 is stopped, the irradiation time of the infrared light is shortened to reduce power and heat generation. be able to.

In this embodiment, the infrared light exposure time 11 and the infrared light irradiation time 13 are matched, but the infrared light irradiation time 13 includes the infrared light exposure time 11 and the visible light exposure time 12 is set. If not included, there is no particular problem.

Further, in the case of a bad environment where higher accuracy of the distance image is required, such as when the background light has an overwhelmingly large IR component, the visible light exposure time is determined from the IR pixel signal level of the infrared light exposure time 11. The result of subtracting by the subtracting circuit 241 after correcting the signals of the 12 IR pixels to the level of the same exposure time by taking each exposure time into consideration is given to the color camera processing unit and the distance image calculation processing unit 250. input. As a result, it is possible to suppress the noise of the IR component in the vicinity of 850 nm of the background light, and realize a further highly accurate range image calculation.

Further, when higher color reproducibility is required as an application in a severe environment where the background light has an overwhelmingly large IR component, the visible light exposure time is determined from the signal level of each RGB pixel in the visible light exposure time 12. The signal of 12 IR pixels has the same transmittance level, taking into account the sensitivity due to the infrared light transmittance near 850 nm as the overall characteristics of the 2-band optical filter 205 and the filter arranged in each pixel. As described above, the output of each of the RGB pixels and the IR pixel is corrected, and the result of subtraction by the subtraction circuit 241 is input to the color camera processing unit and the distance image calculation processing unit 250. Thereby, the noise of the IR component near 850 nm of background light can be suppressed, and higher color reproducibility and high image quality can be realized.

<< Second Embodiment >>
FIG. 6 shows a configuration of an imaging apparatus according to the second embodiment of the present invention. The imaging apparatus of FIG. 6 is obtained by replacing the CCD image sensor 210 in FIG. 1 with a progressive scan type MOS (metal-oxide-semiconductor) image sensor 211 equipped with a global shutter function and a global reset function. is there. Hereinafter, the second embodiment will be described with emphasis on the differences from the first embodiment.

FIG. 7 is a circuit diagram showing a configuration of the pixel cell 101 in the MOS image sensor 211 in FIG. Here, n is an integer from 1 to 4. Each pixel cell 101 includes a PD serving as a photosensitive portion, a shutter transistor M5 for resetting (discharging) signal charges accumulated in the PD by setting the shutter signal TXSn to H, and a floating diffusion (floating) having a storage capacity. difusion: FD), a read transistor M1 for reading the signal charge accumulated in the PD to the FD by setting the transfer signal TXn to H, and a signal charge read to the FD by setting the reset signal RSn to H A reset transistor M2 for resetting, a source follower transistor M3 whose gate is connected to FD, and a line selection transistor M4 that connects the source follower transistor M3 to the vertical signal line by setting the selection signal SELn to H. The The drains of the reset transistor M2, the source follower transistor M3, and the shutter transistor M5 are connected to the pixel electrode VDD.

FIG. 8 is a schematic configuration diagram of the MOS image sensor 211 in FIG. 6 in which the pixel cells 101 in FIG. 7 are two-dimensionally arranged. The MOS image sensor 211 in FIG. 8 includes a pixel portion 102 in which pixel cells 101 are arranged in a two-dimensional form of 4 rows and 4 columns, and fixed pattern noise (FPN) generated due to variations in transistor threshold voltage for each column. The FPN removing unit 103 for removing the signal, the horizontal selecting unit 104 for sequentially selecting the output signal of the FPN removing unit 103, and the differential amplifier 105 for amplifying the output signal of the horizontal selecting unit 104. ing. Note that the pixel unit 102 has a small size of 4 rows and 4 columns for convenience of explanation.

The rows of the pixel unit 102 are sequentially selected by a vertical scanning unit (not shown). A current source 109 is connected to the vertical signal line 106 in each column. Each column of the FPN removing unit 103 includes a signal level sample transistor M11 that receives the sample hold signal SHS, a reset level sample transistor M12 that receives the sample hold signal SHN, a signal level capacitor C11, and a reset level capacitor C12. Consists of Each column of the horizontal selection unit 104 includes a first column selection transistor M21 interposed between the signal level capacitor C11 and the first horizontal signal line 107, a reset level capacitor C12, and a second horizontal signal line 108. And a second column selection transistor M22 interposed therebetween. Each column of the horizontal selection unit 104 is sequentially selected by signals H1 to H4 from a horizontal scanning unit (not shown). The differential amplifier 105 amplifies the potential difference between the first horizontal signal line 107 and the second horizontal signal line 108.

Further, the R, G, B, and IR color filters shown in FIG. 4 are arranged in each pixel as in the first embodiment.

The CPU 260 in FIG. 6 controls the entire system of the imaging apparatus according to the register settings of the

blocks

240, 250, and 270. In particular, the CPU 260 subjects the output from the AE detection circuit 242 to time-axis filtering, etc., and independently calculates the visible light exposure time and the infrared light exposure time of the MOS image sensor 211 to obtain the information, The information is directly transmitted to the MOS image sensor 211 by communication means such as serial transfer, and various drive pulses are generated in the MOS image sensor 211. As a result, the pulse timing of the shutter signals TXS1 to TXS4 for discharging the signal charge of each PD is generated so as to achieve the shutter speed commanded by the CPU 260, and natural AE control is performed in both visible light exposure and infrared light exposure. I can do it.

FIG. 9 is a timing chart showing the operation of the imaging apparatus of FIG. Here, it is assumed that the rate of the vertical synchronization pulse VD is 60 frames per second (60 fps), imaging with irradiated infrared light is performed in odd frames, and visible light imaging with background light is performed in even frames. To do.

In the first frame, the signal charges of all the pixels are discharged by applying the pulse of the shutter signal TXSn (n = 1 to 4) so as to turn on the shutter transistor M5 that collectively discharges the signal charges accumulated in all the PDs. When the pulse application of the shutter signal TXSn ends, the infrared light exposure time 131 starts, and at the same time, the infrared light irradiation time 133 by the infrared laser 202 starts. On the other hand, since the background light illumination 201 is continuously lit, the reflected light from the subject 200 includes both infrared light and visible light. However, only the reflected light in the vicinity of 850 nm is received by the IR pixel by the two-band optical filter 205 and the visible light cut filter of the IR pixel, and photoelectric conversion is performed, and the signal charge is transferred to the IR pixel during the infrared light exposure time 131. Accumulated.

Next, by applying a pulse of the transfer signal TXn (n = 1 to 4) so as to turn on the read transistor M1 for collectively reading the signal charges accumulated in all the PDs to the FD at the beginning of the second frame, The signal charge is read to all the FDs, and the infrared light exposure time 131 ends. At the same time, the infrared light irradiation time 133 by the infrared laser 202 ends.

Thereafter, the voltage signal converted into the signal voltage by the FD by the pulse of the selection signal SEL1 is read to the vertical signal line 106 through the source follower transistor M3, and the IR pixel signal is output through the FPN removal unit 103. Then, the process proceeds to the later stage. Subsequently, the lines are sequentially selected by the pulses of the selection signals SEL2, SEL3, and SEL4. As a result, an IR pixel signal for one screen is output as an infrared light exposure signal at the image sensor output timing 134 shown in FIG. At this time, RGB pixel signals are also output, but these RGB pixel signals are not processed in the subsequent stage.

After the signal charges of all the pixels are read to the FD in the second frame, the shutter signal is turned on so as to turn on the shutter transistor M5 that collectively discharges the signal charges accumulated in all the PDs so that the necessary exposure time is reached. The signal charges of all the pixels are discharged by applying a pulse of TXSn (n = 1 to 4), and the visible light exposure time 132 starts when the pulse application of the shutter signal TXSn is completed. At this time, since the infrared light irradiation by the infrared laser 202 is stopped, the reflected light from the subject 200 does not include the irradiated infrared light component but includes only the background light component. However, the background light may contain some IR components in addition to the visible light component. The RGB color filter in the RGB pixel has a slight transmission characteristic with respect to the IR component. Thus, the RGB pixels are sensitive to some IR component derived from background light in addition to each filtered visible light component and accumulate signal charge during the visible light exposure time 132. In the IR pixel, the two-band optical filter 205 and the visible light cut filter receive only some IR components in the background light and perform photoelectric conversion, and signal charges accumulate in the IR pixel during the visible light exposure time 132. Is done.

Next, by applying a pulse of the transfer signal TXn (n = 1 to 4) so as to turn on the read transistor M1 for collectively reading the signal charges accumulated in all PDs to the FD at the beginning of the third frame. The signal charges of all the pixels are read out to all the FDs, and the visible light exposure time 132 ends.

Thereafter, the voltage signal converted into the signal voltage by the FD by the pulse of the selection signal SEL1 is read to the vertical signal line 106 via the source follower transistor M3, and all the pixel signals are output via the FPN removal unit 103. Then, the process proceeds to the later stage. Subsequently, the lines are sequentially selected by the pulses of the selection signals SEL2, SEL3, and SEL4. As a result, at the image sensor output timing 135 shown in FIG. 9, all pixel signals for one screen including RGB pixels and IR pixels are output as visible light exposure signals.

Further, the operation of the MOS image sensor 211 is repeated, and the infrared light exposure and the visible light exposure are alternately performed in a time division manner so that the infrared light exposure time 131 and the visible light exposure time 132 appear every other frame. Done.

Here, the AE operation based on the output signal of the MOS image sensor 211 will be described. As described above, the pixel signal obtained at the infrared light exposure time 131 of the odd frame is output at the timing 134 in the even frame. At this time, only the output signal of the IR pixel is detected by the AE detection circuit 242, and the output is read by the CPU 260. The CPU 260 determines the shutter speed of the next odd frame through the time filter using only the detection result of the IR pixel signal output from the MOS image sensor 211 for each even frame.

On the other hand, the pixel signal obtained at the visible light exposure time 132 of the even frame is output at the timing 135 in the odd frame. At this time, the output signal of the RGB pixel is detected by the AE detection circuit 242 and the output is read by the CPU 260. The CPU 260 determines the shutter speed of the next even frame through the time filter using only the detection result of the RGB pixel signal output from the MOS image sensor 211 for each odd frame.

As described above, using one 60 fps MOS image sensor 211, as if two cameras were installed, a 30 fps irradiation infrared light imaging camera and a 30 fps background light color imaging camera, Independent AE control is realized.

As described above, according to the present embodiment, pixel signal charges can be discharged by the shutter transistor M5 for all the pixels at once. In addition, since the infrared light exposure and the visible light exposure are performed in a time-sharing manner, by using the two-band optical filter 205 and each pixel filter having a single layer structure on the MOS image sensor 211 in combination with the visible light, Optical mixing with infrared light can be minimized. Therefore, high-quality color image capturing using background light and high-accuracy distance image capturing using infrared light can be realized with an inexpensive system.

Further, in order to control the infrared light exposure time 131 and the visible light exposure time 132 independently of each other, the output of the AE detection circuit 242 is divided into infrared light and visible light, and each is independently controlled by AE. As a result, the IR light exposure time 131 of the IR pixel and the visible light exposure time 132 of the RGB pixel, which are greatly different from each other, can be controlled to the optimum exposure time. It is possible to further improve the image quality of visible image capturing and the accuracy of distance image capturing so that the two cameras, the distance camera, operate simultaneously.

Further, the end of the visible light exposure time 132 is started from the pulse application timing of the shutter signal TXSn (n = 1 to 4) for discharging the signal charges accumulated in the PDs of all the pixels so as to determine the start of the visible light exposure time 132. As determined, the infrared light irradiation by the infrared laser 202 is stopped during the period up to the pulse application timing of the transfer signal TXn (n = 1 to 4) for collectively reading the PD signal charges of all pixels to the FD. Therefore, power and heat generation can be reduced by shortening the irradiation time of infrared light.

In this embodiment, the infrared light exposure time 131 and the infrared light irradiation time 133 are matched, but the infrared light irradiation time 133 includes the infrared light exposure time 131 and the visible light exposure time 132 is set. If not included, there is no particular problem.

Further, in the case of a bad environment where higher accuracy of the distance image is required, such as when the background light has an overwhelmingly large IR component, the visible light exposure time is determined from the IR pixel signal level of the infrared light exposure time 131. The result of subtracting by the subtracting circuit 241 after correcting the signal of the 132 IR pixels to the level of the same exposure time with each exposure time taken into consideration is given to the color camera processing unit and the distance image calculation processing unit 250. input. As a result, it is possible to suppress the noise of the IR component in the vicinity of 850 nm of the background light, and realize a further highly accurate range image calculation.

Further, when higher color reproducibility is required as an application in a severe environment where the background light has an overwhelmingly large IR component, the visible light exposure time is determined from the signal level of each RGB pixel in the visible light exposure time 132. The signal from the 132 IR pixels has the same transmittance level in consideration of the sensitivity due to the infrared light transmittance near 850 nm as the overall characteristics of the two-band optical filter 205 and the filter arranged in each pixel. As described above, the output of each of the RGB pixels and the IR pixel is corrected, and the result of subtraction by the subtraction circuit 241 is input to the color camera processing unit and the distance image calculation processing unit 250. Thereby, the noise of the IR component near 850 nm of background light can be suppressed, and higher color reproducibility and high image quality can be realized.

<< Third Embodiment >>
FIG. 10 shows a configuration of an imaging apparatus according to the third embodiment of the present invention. The imaging device of FIG. 10 is obtained by replacing the CCD image sensor 210 in FIG. 1 with a CCD image sensor 212 in which signal charge readout gates of IR pixels and RGB signal charge readout gates are wired independently of each other. Hereinafter, the third embodiment will be described with emphasis on differences from the first embodiment.

FIG. 11 is a schematic configuration diagram of the CCD image sensor 212 in FIG. A CCD image sensor 212 shown in FIG. 11 is an image sensor of progressive scan compatible and interline transfer system, and includes a PD 71, a vertical transfer unit 72, a horizontal transfer unit 73, a charge detection unit 74, and a VOD 75. Composed. The PDs 71 are arranged in a matrix and constitute pixels. Each pixel is provided with R, G, B, and IR color filters as in the first embodiment. The vertical transfer unit 72 has, for example, a configuration of four gates V1, V7, V3, and V4 per pixel, and is driven by four-phase pulses φV1, φV7, φV3, and φV4. Of these, φV1 is an RGB pixel readout pulse. In other words, the RGB pixel has a configuration in which the signal charge readout gate also serves as the V1 gate of the vertical transfer unit 72. However, the gate of the vertical transfer unit 72 that also serves as the IR pixel readout gate is newly provided as a V5 gate independent of the V1 gate. φV5 is an IR pixel readout pulse. The horizontal transfer unit 73 has, for example, a two-phase (H1 and H2) configuration. Further, when the substrate discharge pulse φSub is given, the signal charges of all the pixels are discharged to the substrate all at once via the VOD 75.

FIG. 12 is a timing chart showing the operation of the imaging apparatus of FIG. Here, description will be made assuming that the rate of the vertical synchronization pulse VD is 60 fps, imaging with irradiated infrared light is performed in the first half of one frame, and visible light imaging with background light is performed in the second half. Note that the timing of the IR pixel readout pulse φV5 can be arbitrarily set for each horizontal period.

In the first frame, application of the substrate discharge pulse φSub discharges signal charges of all the pixels, and when the application of the substrate discharge pulse φSub ends, the infrared light exposure time 81 starts, and at the same time, the infrared light irradiation time by the infrared laser 202 83 starts. On the other hand, since the background light illumination 201 is continuously lit, the reflected light from the subject 200 includes both infrared light and visible light. However, only the reflected light in the vicinity of 850 nm is received by the IR pixel by the two-band optical filter 205 and the visible light cut filter of the IR pixel, photoelectric conversion is performed, and the signal charge is applied to the IR pixel during the infrared light exposure time 81. Accumulated.

Next, by applying an H level to the IR pixel readout pulse φV5, the signal charge of the IR pixel is read out to the vertical transfer unit 72, and the infrared light exposure time 81 ends. At the same time, the infrared light irradiation time 83 by the infrared laser 202 ends.

Thereafter, an IR pixel signal is output as an infrared light exposure signal at the image sensor output timing 84 shown in FIG. 12 via the vertical transfer unit 72, the horizontal transfer unit 73, and the charge detection unit 74, and processing in the subsequent stage Migrate to

On the other hand, after reading the signal charge of the IR pixel, the signal charge of all the pixels is discharged by applying the substrate discharge pulse φSub so that the necessary exposure time is reached, and the visible light exposure time 82 starts when the application of the substrate discharge pulse φSub is completed. To do. At this time, since the infrared irradiation by the infrared laser 202 is stopped, the reflected light from the subject 200 does not include the irradiated infrared light component but includes only the background light component. However, the background light may contain some IR components in addition to the visible light component. The RGB color filter in the RGB pixel has a slight transmission characteristic with respect to the IR component. Thus, the RGB pixels are sensitive to some IR component derived from background light in addition to each filtered visible light component and accumulate signal charge during the visible light exposure time 132. In the IR pixel, the two-band optical filter 205 and the visible light cut filter receive only some IR components in the background light and perform photoelectric conversion, and signal charges accumulate in the IR pixel during the visible light exposure time 132. Is done.

Next, by applying an H level to the RGB pixel readout pulse φV1 at the beginning of the second frame, the signal charges of the RGB pixels are read out to the vertical transfer unit 72, and the visible light exposure time 82 ends.

Subsequently, RGB pixel signals are output as visible light exposure signals at the image sensor output timing 85 shown in FIG. 12 via the vertical transfer unit 72, the horizontal transfer unit 73, and the charge detection unit 74. In this case, the IR pixel signal is also being output, but the signal charge of each of the R, G, B, and IR pixels is vertical by performing vertical transfer so that the RGB pixel signal is contained in the empty packet of the vertical transfer unit 72. Without mixing in the transfer unit 72, it is possible to output all pixels independently while causing a shift between the

timings

84 and 85.

Further, the operation of the CCD image sensor 212 is similarly repeated in units of frames, and the infrared light exposure and the visible light exposure are sometimes performed so that the infrared light exposure time 81 and the visible light exposure time 82 appear in one frame. Alternation is performed alternately.

Here, the AE operation based on the output signal of the CCD image sensor 212 will be described. As described above, the pixel signal obtained at the infrared light exposure time 81 is output over a period of about 1 V from the timing in the middle of the frame. At this time, the output signal of the IR pixel is detected by the AE detection circuit 242 and the output is read by the CPU 260. The CPU 260 uses only the detection result of the IR pixel signal to determine the shutter speed of the infrared light exposure time 81 of the next frame through the time filter.

On the other hand, the pixel signal obtained at the visible light exposure time 82 is output from the top of the frame. At this time, the output signal of the RGB pixel is detected by the AE detection circuit 242 and the output is read by the CPU 260. The CPU 260 determines the shutter speed for the next visible light exposure time 82 through the time filter using only the detection result of the RGB pixel signal output from the CCD image sensor 212 for each frame.

In this way, using one 60 fps CCD image sensor 212, as if two cameras were installed, a 60 fps irradiation infrared light imaging camera and a 60 fps background light color imaging camera, each was independent. AE control is realized.

As described above, according to the present embodiment, the pixel signal charge can be discharged by the VOD 75 for all the pixels at once. In addition, since the infrared light exposure and the visible light exposure are performed in a time-sharing manner, the two-band optical filter 205 and each pixel filter having a single layer structure on the CCD image sensor 212 are used in combination. Optical mixing with infrared light can be minimized. Therefore, high-quality color image capturing using background light and high-accuracy range image capturing using infrared light can be realized with an inexpensive system.

Further, in order to control the infrared light exposure time 81 and the visible light exposure time 82 independently of each other, the output of the AE detection circuit 242 is divided into infrared light and visible light, and each is independently controlled by AE. Therefore, since the respective light sources are different, it is possible to control the infrared light exposure time 81 of the IR pixel and the visible light exposure time 82 of the RGB pixel, which are greatly different from each other, to the optimum exposure time, as if they were measured with a visible camera. It is possible to further improve the image quality of visible image capturing and the accuracy of distance image capturing so that the two cameras, the distance camera, operate simultaneously.

In addition, the signal charge readout gate of the IR pixel and the signal charge readout gate of the RGB pixel are wired independently of each other, and the time division between the infrared light exposure and the visible light exposure is completed within one frame. The application of the signal charge readout pulse φV5 ends the infrared light exposure time 81, and the application of the signal charge readout pulse φV1 of the RGB pixels ends the visible light exposure time 82. Therefore, the frame rate of the CCD image sensor 212 Both a visible image and a distance image can be obtained at the same frame rate, and the system can be greatly speeded up.

Further, the signal charges of the RGB pixels are read out so as to determine the end of the visible light exposure time 82 from the application timing of the substrate discharge pulse φSub that discharges the signal charges of all the pixels so as to determine the start of the visible light exposure time 82. In the period up to the application timing of the RGB pixel readout pulse φV1, the infrared light irradiation by the infrared laser 202 is stopped. Therefore, the power and heat generation are reduced by shortening the irradiation time of the infrared light. be able to.

In this embodiment, the infrared light exposure time 81 and the infrared light irradiation time 83 are matched, but the infrared light irradiation time 83 includes the infrared light exposure time 81 and the visible light exposure time 82 is set. If not included, there is no particular problem.

Further, this embodiment can also be applied to a MOS image sensor having a global shutter function and a global reset function by making the row selection wiring of the IR pixel and the RGB pixel independent.

Since the present invention realizes distance image capturing using an infrared light irradiation pattern and color image capturing using background light in one system, it can be applied to an imaging device such as a game machine for gesture authentication or a signage field such as person authentication. Can be used.

11, 81, 131 Infrared

light exposure time

12, 82, 132 Visible

light exposure time

13, 83, 133 Infrared

light irradiation time

14, 84, 134 Infrared light exposure time image

sensor output timing

15, 85, 135 Visible Image sensor output timing of

light exposure time

21, 71 Photodiode (pixel)
22, 72

Vertical transfer units

23, 73

Horizontal transfer units

24, 74

Charge detection units

25, 75 Vertical overflow drain (VOD: discharge means)
DESCRIPTION OF SYMBOLS 101 Pixel cell 102 Pixel part 103 Fixed pattern noise (FPN) removal part 104 Horizontal selection part 105 Differential amplifier 106 Vertical signal line 107,108 Horizontal signal line 109 Current source 200 Subject 201 Background light illumination 202 Infrared laser (irradiation means)
203 Rough surface filter 204 Optical lens 205 Two-band optical filter 210 CCD image sensor 211 MOS image sensor 212 CCD image sensor 220 Analog front end (AFE)
221 V driver (VDR)
230 Image processing device (ISP)
240 Camera pre-processing unit 241 Subtraction circuit 242 AE detection circuit 250 Color camera processing unit and distance image calculation processing unit 260 Central processing unit (CPU)
270 Timing Generator (TG)

Claims

Irradiating means for irradiating the subject with infrared light;
A two-band optical filter having a spectral transmission characteristic that transmits a visible light wavelength region and a wavelength region near the irradiated infrared light and receiving incident light from the subject;
An image pickup apparatus comprising: an image sensor that converts an image formed through the two-band optical filter into an electric signal;
The image sensor is
A first pixel that receives infrared light transmitted through the two-band optical filter and performs infrared light exposure;
A second pixel that receives visible light transmitted through the two-band optical filter and performs visible light exposure;
Discharging means for simultaneously discharging the signal charges of the first and second pixels,
An imaging apparatus, wherein infrared light exposure of the first pixel and visible light exposure of the second pixel are performed in a time-sharing manner.
The imaging device according to claim 1,
The image sensor includes a pixel unit having a unit arrangement of m × n (m and n are each an integer of 2 or more) in which the first pixel and the second pixel are mixed. Imaging device.
The imaging device according to claim 1,
The image sensor includes a pixel unit having a 2 × 2 unit array in which the first pixel and the three second pixels are mixed.
An imaging apparatus according to claim 1, wherein a visible light cut filter is provided in each of the first pixels, and an RGB color filter is provided in each of the three second pixels.
The imaging device according to claim 1,
An imaging apparatus, wherein a time division cycle of infrared light exposure of the first pixel and visible light exposure of the second pixel is synchronized with an output image frame of the image sensor.
The imaging apparatus according to claim 4.
From the output of the first pixel in the frame where infrared light exposure of the first pixel is performed, the output of the first pixel in the frame where visible light exposure of the second pixel is performed is set to each exposure time. An imaging apparatus, further comprising a subtracting circuit that corrects and subtracts in consideration.
The imaging apparatus according to claim 4.
From the output of the second pixel in the frame in which the visible light exposure of the second pixel is performed, the output of the first pixel in the frame is corrected in consideration of the sensitivity due to the infrared light transmittance of each pixel. An image pickup apparatus, further comprising a subtracting circuit for subtracting.
The imaging device according to claim 1,
An AE detection circuit for detecting the output of the image sensor;
The output of the AE detection circuit is for infrared light and visible light so that the time of infrared light exposure of the first pixel and the time of visible light exposure of the second pixel are controlled independently of each other. And an AE control independently for each of them.
The imaging device according to claim 1,
The irradiating means determines at least the end of visible light exposure of the second pixel from the timing of pulse application for discharging the signal charges of all pixels so as to determine the start of visible light exposure of the second pixel. As described above, the infrared light irradiation is stopped during a period until the pulse application timing for reading out the signal charges of all the pixels.
The imaging device according to claim 1,
Wiring the signal charge readout gate of the first pixel and the signal charge readout gate of the second pixel independently of each other;
A time division cycle of infrared light exposure of the first pixel and visible light exposure of the second pixel is completed within one frame of the output image of the image sensor;
Infrared light exposure of the first pixel is terminated at the timing of applying a pulse to the signal charge readout gate of the first pixel, and the first timing is applied to the signal charge readout gate of the second pixel. An imaging apparatus characterized by ending visible light exposure of the second pixel.
The imaging apparatus according to claim 9, wherein
An AE detection circuit for detecting the output of the image sensor;
The output of the AE detection circuit is for infrared light and visible light so that the time of infrared light exposure of the first pixel and the time of visible light exposure of the second pixel are controlled independently of each other. And an AE control independently for each of them.
The imaging apparatus according to claim 9, wherein
The irradiating means determines at least the end of visible light exposure of the second pixel from the timing of pulse application for discharging the signal charges of all pixels so as to determine the start of visible light exposure of the second pixel. As described above, the infrared light irradiation is stopped in a period until the timing of pulse application to the signal charge readout gate of the second pixel.