JP2014207493A - Imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a high performance imaging apparatus capable of taking distance images and color images using infrared light irradiation with a combination of one image sensor and photographic lens at low cost.SOLUTION: An imaging apparatus includes: infrared light pattern irradiation means; a two-band transparency optical filter formed over the entire surface of an image sensor which allows, in addition to the entire of visible range light, the light adjacent to the wavelength of the infrared light pattern irradiation to transmit therethrough; plural pixels for receiving infrared light and plural pixels for receiving visible light formed on the image sensor; and discharge means that discharges signal charges simultaneously from every pixels. The imaging apparatus performs an infrared light irradiation exposure and background light exposure in a time-division manner.

Description

  The present invention relates to an imaging apparatus.

  An imaging apparatus that integrates a conventional visible light imaging and a distance image imaging using infrared light into a single image sensor and an optical lens will be described with reference to FIGS.

  First, FIG. 12 shows light receiving units 90a to 90c that receive red (R), blue (B), and green (G), which are the three primary colors of visible light, and accumulate signal charges, and infrared light (IR). Is a CCD image sensor of an interline transfer system in which an IR light receiving unit 90d that receives light and accumulates signal charges is arranged in a plane, and a general vertical overflow that sweeps out signal charges of all pixels of the light receiving units 90a to 90d Since the distance between the drain (VOD) and the subject of each pixel is a ToF (Time of Flight) method, a horizontal overflow drain (LOD) that can sweep away only the signal charge of the IR light receiving unit 90d at high speed is provided. The LOD is composed of a drain region in which the signal charge of the IR light receiving unit 90d is swept to the drain via a sweep gate 91 connected to the IR light receiving unit 90d. In this configuration, the LOD is operated after the VOD, and then the signal charges are read from the light receiving units 90a to 90d.

  In FIG. 13, each of the three primary color filters 102 that receive visible light installed in each pixel generally transmits infrared light, and therefore, an infrared light cut filter 101 that blocks infrared light. In addition, an optical filter that receives infrared light includes an infrared light transmission filter and a transparent filter for flattening the two-layer color filter that receives visible light. It has a two-layer structure and suppresses optical mixing of infrared light and visible light.

  Further, in FIG. 14, irradiation of IR pulses to the imaging target space is performed every other frame scanning period while imaging of visible light pixels and infrared light pixels is performed every frame scanning period. A visible light image is generated for each frame scanning period, and an IR image signal (S2IR) obtained by imaging at the time of non-irradiation of IR pulse from an IR image signal (S1IR) obtained by imaging at the time of IR pulse irradiation Is subtracted to generate a distance image excluding the influence of the infrared light component in the outside light every one frame scanning period.

JP 2008-8700 A JP 2008-5213 A

  However, the prior art has the following problems.

  First, in addition to installing a general vertical overflow drain (VOD) for IR pixels, it is necessary to install a horizontal overflow drain (LOD) that compresses the photosensitive area (PD) area of the image sensor. However, there is a problem that basic performance such as sensitivity, saturation, and smear must be sacrificed.

  In addition, for visible light pixels, each pixel requires a complex double-layer optical filter configuration of RGB color filters and infrared light cut filters, which not only increases device manufacturing difficulty but also is important for sensitivity characteristics. It has the subject of inhibiting the thin film formation of an optical layer.

  For IR pixels, in addition to a visible light cut filter for each pixel, a complex two-layer structure in which a transparent filter is laminated for flattening must be adopted, which not only increases the manufacturing difficulty of the device but also increases the sensitivity. It has the subject of inhibiting the thinning of the optical layer important for the characteristics.

  In view of the above problems, the present invention provides an imaging apparatus that integrates high-quality visible light imaging and high-accuracy distance image imaging into one system using an inexpensive single image sensor having a general configuration.

  In order to achieve the above object, an imaging apparatus according to an aspect of the present invention transmits an infrared light pattern irradiating unit and a visible wavelength in the vicinity of the infrared light emitted from the infrared light pattern irradiating unit. A two-band transmission type optical filter, a plurality of first pixels that receive infrared light from the optical filter and perform irradiation infrared light exposure, and receive visible light from the optical filter and background light exposure A plurality of second pixels that perform the above and an image sensor that discharges the signal charges of the first and second pixels at the same time, and performs the irradiation infrared light exposure and the background light exposure in a time-sharing manner, It has.

  With this configuration, pixel signal charge can be discharged by a single unit for all the pixels, and the optical filter installed in each pixel can adopt a general simple structure of a single layer. The optical filter minimizes optical mixing of visible light and infrared light, and can realize a good color image and a high-precision distance image.

  According to the present invention, it becomes possible to provide visible light imaging and distance image imaging by infrared light irradiation with a single lens and image sensor in combination with a high image quality and high accuracy at a low cost, At the same time, it is possible to realize an imaging device capable of suppressing power and heat generation.

1 is a configuration diagram of an imaging apparatus according to a first embodiment of the present invention. The figure which shows the spectral transmittance of the 2 band optical filter which concerns on the 1st-3rd embodiment of this invention. 1 is a configuration diagram of a solid-state imaging device according to a first embodiment of the present invention. Color filter arrangement according to first to third embodiments of the present invention Operation timing diagram of imaging device and solid-state imaging device according to first embodiment of the present invention Configuration diagram of an imaging apparatus according to a second embodiment of the present invention The block diagram of the solid-state imaging device concerning the 2nd Embodiment of this invention Operation Timing Diagram of Imaging Device and Solid-State Imaging Device According to Second Embodiment of Present Invention Configuration diagram of an imaging apparatus according to a third embodiment of the present invention Configuration diagram of a solid-state imaging device according to a third embodiment of the present invention Operation timing chart of imaging device and solid-state imaging device of third embodiment of the present invention Conventional image sensor configuration diagram Cross-sectional view of a conventional image sensor pixel Prior art system timing diagram

  Embodiments of the present invention will be described below with reference to the drawings. In the present embodiment, the time that the image sensor (pixel) receives light is called light reception, and the time that the image sensor (pixel) receives light is called exposure or exposure time.

(First embodiment)
FIG. 1 shows the configuration of an imaging apparatus (camera) according to the first embodiment of the present invention.

  As shown in FIG. 1, the imaging apparatus according to the present embodiment is provided with an infrared light irradiation laser of 850 nm, and a rough surface filter is provided on the irradiation light, so that a subject is irradiated with a large number of speckle patterns by a coherent random speckle pattern. Is done.

  In addition, the reflected light passes through an optical lens and a two-band optical filter having a spectral transmittance shown in FIG. 2 that transmits the visible light wavelength region and the near infrared wavelength region near 850 nm. The image formed on is converted into an electrical signal. On the other hand, during the time when infrared irradiation is not performed, an image formed by reflecting the reflected light from the subject by background light illumination on the same image sensor through the same optical lens and the two-band optical filter is converted into an electric signal. 1 is a CCD image sensor (solid-state imaging device).

  The signal output from the image sensor is converted into a digital signal via an analog front end (AFE). The digitized signal is subjected to scratch correction or the like by a camera preprocessing unit in the latter ISP. The camera pre-processing unit performs AE detection and noise removal as necessary.

  The ISP also includes a color camera processing unit that performs camera color processing such as contour enhancement and color adjustment, and a distance image calculation that calculates a distance by comparing a coherent random speckle pattern from an infrared light pixel with a reference pattern. A processing unit is installed. Further, the CPU controls the entire system by register settings of each block.

  In particular, the output from the AE detection circuit is subjected to a filter in the time axis direction, and the visible light exposure time and the infrared light exposure time of the image sensor are independently calculated, and the information is sent to the timing generator (TG). The TG generates the image sensor drive pulse, generates the substrate discharge pulse timing to discharge the pixel signal charge so that the shutter speed commanded by the CPU at this time, visible light exposure, infrared light exposure It is a feature of this embodiment that both can perform natural AE control.

  Next, details of the image sensor provided in the imaging apparatus according to the present embodiment will be described with reference to FIGS. 3 and 4.

  From FIG. 3, it is an interline transfer type image sensor corresponding to progressive scan, and includes a photodiode (pixel) 21, a four-phase drive vertical transfer unit 22 having, for example, four gates per pixel, and a two-phase horizontal transfer unit, for example. 23, a charge detection unit 24, and a vertical overflow drain (VOD) 25. The signal signal readout gate of the pixel is configured to be shared by the V1 gate of the four-phase vertical transfer section (V1 to V4) 22.

  In FIG. 3, for easy understanding of the present invention, the VOD 25 is shown in the horizontal direction of the pixel, but in actuality, it is configured in the bulk direction of the pixel (depth direction of the semiconductor substrate). . When a high voltage is applied to the Sub terminal connected to the VOD substrate, the signal charges of all the pixels are discharged to the substrate all at once. Each pixel is provided with R, G, B, and IR color filters shown in FIG.

  Next, the operation timing of the imaging device and the solid-state imaging device according to the embodiment of the present invention will be described with reference to FIG.

  First, the case where the vertical synchronization pulse VD is 60 frames per second (60 fps), an odd-numbered frame is irradiated infrared imaging, and an even-numbered frame is visible light imaging using background light will be described.

  In the first frame, the signal charges of all the pixels are discharged by applying the substrate discharge pulse (φSub) of the final transfer unit, the irradiation infrared light exposure time is started at the end of the φSub application, and the infrared laser is irradiated at the same time.

  Since the background light continues to be lit, the reflected light from the subject includes both infrared light and visible light, but the reflected light near 850 nm is obtained by the two-band optical filter and the visible light cut filter of the infrared light receiving pixel. Only the infrared light receiving pixel receives light and performs photoelectric conversion, and signal charges are accumulated in the pixel during the infrared light exposure time.

  Next, by applying an H level to the all pixel readout pulse φV1 at the beginning of the second frame, signal charges of all pixels including pixels that receive infrared light are read out to the vertical transfer unit, and irradiation infrared light exposure is performed. The time ends. At the same time, the infrared laser irradiation is stopped.

  In this embodiment, the infrared laser irradiation period and the irradiation infrared light exposure time are matched, but the infrared light irradiation period includes the infrared light exposure time 11, and the visible light exposure time 12 is set. If it is not included, there is no particular problem.

  Thereafter, the infrared light pixel is transmitted at the timing (image sensor output timing of infrared light exposure time 1) 14 shown in FIG. 5 via the vertical transfer unit 22, the horizontal transfer unit 23, and the charge detection unit 24 described in FIG. A signal is output, and the process proceeds to the subsequent stage. At this time, RGB pixel signals are also output, but they are not processed in a later stage.

  On the other hand, after reading the signal charges of all the pixels in the second frame, the signal charges of all the pixels are discharged by applying φSub to all the pixels so that the required exposure time is reached, and the background light exposure time is started by completing the φSub application, At this time, since the laser irradiation is stopped, the reflected light from the subject includes only the background light component.

  Moreover, when each background light contains an infrared light component due to the transmission of the two-band optical filter and the RGB filter in each of the RGB pixels, a part of the infrared light is transferred to each of the R, G, and B pixels. The component and the part of the infrared light component, the G component and the part of the infrared light component, the B component and the part of the infrared light component are received by the R, G, and B pixels. In addition, due to the two-band optical filter and the IR filter transmission, only a part of the infrared light component is received by the IR pixel and subjected to photoelectric conversion, and the visible light exposure time in each of the R, G, B, and IR pixels. Accumulated in.

  Next, by applying an H level to the all-pixel readout pulse φV1 at the beginning of the third frame, all the pixels are read out to the vertical transfer unit, and the background light exposure time ends.

  Thereafter, all the pixels including the IR pixels at the timing (image sensor output timing of visible light exposure time 1) 15 shown in FIG. 5 via the vertical transfer unit 22, the horizontal transfer unit 23, and the charge detection unit 24 described in FIG. Is output.

  Further, the operation of the image sensor is repeated, and the irradiation infrared light exposure time and the background light exposure time are operated in a time division manner every frame.

  Next, the AE operation will be described from the signal output from the image sensor.

  The pixel signal of the odd-frame irradiation infrared light exposure time is output to the even frame, and only the signal of the IR pixel that receives the infrared light is detected by the AE detection circuit of the ISP circuit, and the output is read by the CPU. The shutter speed of the next odd frame is determined by the CPU through the time filter using only the signal detection result of the IR pixel output from the image sensor every time.

  The pixel signal of the background light exposure time of the even frame is output to the odd frame, the RGB pixel signal receiving visible light is detected by the AE detection circuit of the ISP circuit, the output is read by the CPU, and the odd frame is read for each odd frame. Using only the signal detection result of the visible light pixel output from the image sensor, the shutter speed of the next even frame is determined by the CPU through the time filter.

  In this manner, using the 60 fps image sensor, AE control is realized independently as if two cameras of 30 fps irradiation infrared light imaging and 30 fps background light color imaging were installed.

  Next, the IR pixel signal received as the reflected light of the coherent random speckle pattern from the subject during the irradiation infrared light exposure time is compared with the reference pattern of the speckle pattern stored in advance, and the distance image It is calculated by an arithmetic circuit, converted into a distance image and output.

  In the present embodiment, in a bad environment where the distance image accuracy is more required such as the background light having an overwhelmingly large infrared light component, the background is determined from the signal level of the IR pixel during the irradiation infrared light exposure time. The result of subtraction by the subtracting circuit after the IR pixel signal of the light exposure time is corrected so as to have the same exposure time level in consideration of the ratio of each exposure time is input to the distance image calculation processing unit in the subsequent stage. Thereby, it is possible to suppress the noise of the infrared component in the vicinity of 850 nm of the background light, and to realize the distance image calculation with higher accuracy.

  Next, the RGB pixel signal received as reflected light from the subject during the background light exposure time is output as a color image processed by the color camera processing unit, such as white balance and edge enhancement processing.

  As described above with reference to the drawings, the imaging device according to the present embodiment is required to have higher color reproducibility as an application in a severe environment such as an overwhelmingly large infrared light component in the background light. The IR signal of the background light exposure time to the IR pixel signal of the background light exposure time from the signal level of each RGB pixel of the background light exposure time, and the infrared near 850 nm as the overall characteristics of the filter and the two-band optical filter installed in each pixel. In consideration of the light transmittance, the transmittance ratio of each of the RGB pixels and the IR pixel is corrected so that the same transmittance level is obtained. Thereafter, the result of subtraction by the subtraction circuit is input to the subsequent color camera processing unit.

  Thereby, the noise of the infrared light component near 850 nm of background light can be suppressed, and an image pickup apparatus with higher color reproduction and higher image quality can be realized.

(Second Embodiment)
First, the configuration of an imaging apparatus (camera) according to the second embodiment of the present invention will be described with reference to FIG.

  As shown in FIG. 6, the imaging apparatus according to the present embodiment is provided with an infrared light irradiation laser of 850 nm, and a rough surface filter is provided on the irradiation light, so that a subject is irradiated with a large number of speckle patterns by a coherent random speckle pattern. Is done. 6 is a MOS image sensor (solid-state imaging device).

  An image sensor equipped with a global shutter function through an optical lens and a two-band optical filter having the spectral transmittance shown in FIG. 2 that transmits the reflected light through a visible light wavelength region and a near-infrared wavelength region near 850 nm. An image formed on the (solid-state imaging device) is converted into an electrical signal.

  On the other hand, when the infrared light irradiation is not performed, the reflected light from the subject by the background light illumination is converted into an electrical signal from the image formed on the same image sensor through the same optical lens and the two-band optical filter. is there.

  A signal output from the image sensor is converted into a digital signal via an analog front end (AFE). The digitized signal is subjected to scratch correction or the like by a camera preprocessing unit in the latter ISP. The camera pre-processing unit performs AE detection and noise removal as necessary.

  In addition, a color camera processing unit that performs camera color processing such as contour enhancement and color adjustment, and a distance image calculation processing unit that calculates the distance by comparing the coherent random speckle pattern from the infrared light pixel with the reference pattern are installed. Has been.

  The CPU controls the entire system by register settings of each block. In particular, in this embodiment, the output from the AE detection circuit is filtered in the time axis direction, and the visible light exposure time and the infrared light exposure time of the image sensor are independently calculated, and the information is serialized. The information is transmitted directly to the image sensor by communication means such as transfer, and various drive pulses are generated in the image sensor. At this time, the timing of the discharge pulses TXS1 to TXSn for discharging the signal charges of the pixels so as to be the shutter speed commanded by the CPU is generated, and natural AE control can be performed for both visible light exposure and infrared light exposure. It is.

  Next, the details of the image sensor installed in the imaging apparatus according to the present embodiment will be described with reference to FIG.

  From FIG. 7, the image sensor according to the present embodiment is compatible with the progressive scan system equipped with the global shutter & global reset function, and one cell of the photoelectric conversion unit has the following configuration, and the photodiode PD (Pixel), reset transistor M5 that resets signal charges accumulated in the PD collectively, floating diffusion FD, read transistor M1 that collectively reads signal charges accumulated in the PD to the FD, and read to the FD It comprises a reset transistor M2 for resetting the signal charge, a source follower transistor M3, and a line selection transistor M4.

  Hereinafter, the voltage signal read out to the vertical signal line is output as an output signal through an FPN elimination circuit generated by Vt variation of the source follower transistor. Each pixel is provided with color filters of R, G, B, and IR shown in FIG. 4 as in the first embodiment.

  Next, the operation timing of the imaging device and the solid-state imaging device according to the embodiment of the present invention will be described with reference to FIG.

  First, the vertical synchronization pulse VD is 60 frames per second (60 fps), and an example in which an odd-numbered frame is irradiated infrared imaging and an even-numbered frame is visible light imaging using background light will be described. Signal charges accumulated in all PDs in the first frame Applying the gate pulse TXSn to turn on the reset transistor M5 that collectively resets the signal charges of all the pixels is discharged, and when the discharge pulse TXSn is applied, the irradiation infrared light exposure time starts, and at the same time, the infrared laser is irradiated. The

  Since the background light continues to be lit, the reflected light from the subject contains both infrared light and visible light. However, the reflected light near 850 nm is reflected by the two-band optical filter and the visible light cut filter of the infrared light receiving pixel. Only the light is received by the infrared light receiving pixels and subjected to photoelectric conversion, and signal charges are accumulated in the pixels during the infrared light exposure time.

  Next, the signal charges accumulated in all PDs at the beginning of the second frame are collectively read out to the FD. By applying the gate pulse TXn that turns on the readout transistor M1, the signal charges of all the pixels are read out to all the FDs. The exposure time ends. At the same time, the infrared laser irradiation is stopped.

  In the present embodiment, the infrared laser irradiation period and the irradiation infrared light exposure time are matched, but the infrared light irradiation period includes the infrared light exposure time 131, and the visible light exposure time 132 is set to be the same. If it is not included, there is no functional problem.

  Thereafter, the voltage signal converted into the signal voltage by the FD by the signal line selection pulse SEL1 is read to the vertical signal line through the source follower, and the infrared light pixel signal is output through the FPN removal circuit, The process proceeds to the subsequent stage, and the lines SEL2 and SEL3 are sequentially selected and an infrared pixel signal for one screen is output. At this time, RGB pixel signals are also output, but they are not processed in a later stage.

  That is, the infrared light pixel signal is output at the timing (image sensor output timing of infrared light exposure time 1) 134 shown in FIG. 8, and the process proceeds to the subsequent stage.

  On the other hand, after the signal charges of all the pixels in the second frame are read to the FD, the gate pulse TXSn is applied to turn on the reset transistors M5 that collectively reset the signal charges accumulated in all the PDs so that the required exposure time is reached. The signal charges of all the pixels are discharged, and the background light exposure time starts when the application of the gate pulse TXSn ends. At this time, since the laser irradiation is stopped, the reflected light from the subject contains only the background light component. .

  When each of the RGB pixels has a two-band optical filter and an RGB filter, when the background light includes an infrared light component, a part of the infrared light is transferred to each of the R, G, and B pixels. A part of the infrared light component, the G component and the part of the infrared light component, the B component and the part of the infrared light component are received by each of the R, G, and B pixels. Only a part of infrared light components are received by the filter and the IR pixel and subjected to photoelectric conversion, and accumulated in the R, G, B, and IR pixels during the visible light exposure time.

  Next, signal charges accumulated in all PDs at the beginning of the third frame are collectively read to the FD. By applying a gate pulse TXn that turns on the read transistor M1, the signal charges of all the pixels are read to all the FDs, and background light exposure is performed. The time ends.

  Hereinafter, the voltage signal converted into the signal voltage by the FD by the signal line selection pulse SEL2 is read out to the vertical signal line through the source follower, and all the pixels including the IR pixel from the image sensor through the FPN removal circuit. Is output.

  That is, all the pixels including the IR pixels are output at the timing (image sensor output timing of visible light exposure time 1) 135 shown in FIG.

  Thereafter, the operation of the image sensor is repeated, and the irradiation infrared light exposure time and the background light exposure time are operated in a time-sharing manner every frame.

  Next, the AE operation will be described from the signal output from the image sensor.

  First, the pixel signal of the odd-frame irradiation infrared light exposure time is output to the even frame, only the signal of the IR pixel that receives infrared light is detected by the AE detection circuit of the ISP circuit, and the output is read by the CPU. The shutter speed of the next odd frame is determined by the CPU through a time filter using only the signal detection result of the IR pixel output from the image sensor for each even frame.

  The pixel signal of the background light exposure time of the even frame is output to the odd frame, the RGB pixel signal receiving visible light is detected by the AE detection circuit of the ISP circuit, the output is read by the CPU, and the image is imaged every odd frame. Using only the signal detection result of the visible light pixel output from the sensor, the shutter speed of the next even frame is determined by the CPU through the time filter.

  In this way, using an image sensor of 60 fps, AE control is realized independently as if two cameras were installed, that is, 30 fps irradiation infrared light imaging and 30 fps background light color imaging. It is.

  Next, the IR pixel signal received as the reflected light of the coherent random speckle pattern from the subject during the irradiation infrared light exposure time is collated with a pre-stored speckle pattern reference, and the distance image calculation is performed based on the address deviation. It is calculated by a circuit, converted into a distance image and output.

  In the present embodiment, in the case where distance image accuracy is required such that the background light has an overwhelmingly large infrared light component, the background light exposure is performed based on the signal level of the IR pixel during the irradiation infrared light exposure time. By inputting the result of subtraction by the subtracting circuit after correcting the IR pixel signal of time to the same exposure time level in consideration of each exposure time ratio, the background image is input to the background image processing unit. It is possible to suppress the noise of the infrared light component in the vicinity of 850 nm of light, and to realize a highly accurate distance image calculation.

  Next, the RGB pixel signal received as reflected light from the subject during the background light exposure time is output as a color image processed by the color camera processing unit, such as white balance and edge enhancement processing.

  In this embodiment, when high color reproducibility is required as an application under severe conditions such as an overwhelmingly large infrared light component in the background light, each RGB pixel signal of the background light exposure time is required. From the level, the IR pixel signal of the background light exposure time is taken into account by adding the infrared light transmittance near 850 nm as the total characteristics of the filter installed in each pixel and the two-band optical filter, and the same transmittance level Thus, the transmittance ratio of each of the RGB pixels and the IR pixel is corrected.

  After that, the result of subtraction by the subtraction circuit is input to the subsequent color camera processing unit, so that noise of the infrared light component in the vicinity of 850 nm of the background light is suppressed, and an imaging device with higher color reproduction and higher image quality is realized. be able to.

(Third embodiment)
The configuration of an imaging apparatus according to the third embodiment of the present invention is shown in FIG.

  As shown in FIG. 9, the imaging apparatus according to the present embodiment is provided with an infrared light irradiation laser of 850 nm, and a rough surface filter is provided on the irradiation light, so that a subject has a large number of speckle patterns based on a coherent random speckle pattern. Irradiated.

  The reflected light is transmitted through an optical lens, a visible light wavelength region and a near-infrared wavelength region near 850 nm, and passed through the two-band optical filter having the spectral transmittance shown in FIG. The image formed is converted into an electrical signal. 9 is a CCD image sensor (solid-state imaging device).

  On the other hand, during the time when the infrared light irradiation is not performed, an image obtained by forming the reflected light from the subject by the background light illumination on the same image sensor through the same optical lens and the two-band optical filter is converted into an electric signal.

  The signal output from the image sensor is converted into a digital signal via an analog front end (AFE). The digitized signal is subjected to scratch correction or the like by a camera pre-processing unit in the subsequent ISP, and is also subjected to AE detection. In addition, a color camera processing unit that performs camera color processing such as contour enhancement and color adjustment, and a distance image calculation processing unit that calculates the distance by comparing the coherent random speckle pattern from the infrared light pixel with the reference pattern are installed. Has been.

  Further, the CPU controls the entire system by register settings of each block. In particular, in this embodiment, the output from the AE detection circuit is filtered in the time axis direction, and the visible light exposure time and the infrared light exposure time of the image sensor are independently calculated, and the information is timed. The TG generates a drive pulse for the image sensor, generates a timing of the substrate discharge pulse for discharging the signal charge of the pixel so as to achieve the shutter speed commanded by the CPU at this time, and visible light exposure In addition, natural AE control is performed for both infrared light exposure.

  Next, the configuration of the image sensor mounted on the imaging apparatus according to the present embodiment will be described with reference to FIG.

  As shown in FIG. 10, the image sensor according to the present embodiment is an interline transfer system compatible with progressive scan, and includes a photodiode (pixel) 71, four-gate vertical transfer unit 72 with four gates per pixel, and two-phase horizontal transfer. Part 73, charge detection part 74, and vertical overflow drain (VOD) 75.

  The pixel signal charge readout gate is also used as the V1 gate of the four-phase vertical transfer section (V1 to V4). The important thing here is that only the gate of the vertical transfer unit which is also used as the readout gate of the IR pixel is set as an independent wiring and newly installed as V5.

  In FIG. 10, the VOD 75 is shown in the horizontal direction of the pixel of the pixel for easy understanding of the present invention. However, the VOD 75 is actually configured on the deep side of the pixel in the bulk direction (depth direction of the semiconductor substrate). ing. When a high voltage is applied to the Sub terminal connected to the VOD substrate, the signal charges of all the pixels are discharged to the substrate all at once.

  Each pixel is provided with R, G, B, and IR color filters shown in FIG.

  Next, the operation timing of the imaging device and the solid-state imaging device according to the embodiment of the present invention will be described with reference to FIG.

  First, the vertical synchronization pulse VD is 60 FPS, and the case where the first half of one frame is irradiated infrared imaging and the second half is visible light imaging using background light will be described. In the first frame, the substrate discharge pulse φSub of the first transfer unit is applied. The signal charges of all the pixels are discharged, and the irradiation infrared light exposure time starts when the φSub application ends, and at the same time, the infrared laser is irradiated.

  Since the background light continues to be lit, the reflected light from the subject includes both infrared light and visible light, but the reflected light near 850 nm is obtained by the two-band optical filter and the visible light cut filter of the infrared light receiving pixel. Only the infrared light receiving pixel receives light and performs photoelectric conversion, and signal charges are accumulated in the pixel during the infrared light exposure time.

  Next, by applying an H level to the IR pixel readout pulse φV5, the signal charge of the pixel that receives infrared light is read out to the vertical transfer unit, and the irradiation infrared light exposure time ends. At the same time, the infrared laser irradiation is stopped. In this example, the infrared light laser irradiation period and the irradiation infrared light exposure time are matched, but the infrared light irradiation period includes the side exposure time 81, and if it does not include the visible light exposure time 82, it is functional. Is fine.

  Thereafter, the infrared light pixel is transmitted from the image sensor through the vertical transfer unit 72, the horizontal transfer unit 73, and the charge detection unit 74 shown in FIG. 10 at the timing shown in FIG. A signal is output, and the process proceeds to the subsequent stage.

  On the other hand, after reading the signal charge of the IR pixel, the signal charge of all the pixels is discharged by applying the substrate discharge pulse φSub to all the pixels so that the required exposure time is reached, and the background light exposure time is started by the end of the application of the discharge pulse φSub. Since the laser irradiation is stopped at this time, the reflected light from the subject contains only the background light component.

  In addition, by using a two-band optical filter and an RGB filter for each RGB pixel, when the background light includes an infrared light component, a part of the infrared light is converted into an R component for each of the R, G, and B pixels. And the part of the infrared light component, the G component and the part of the infrared light component, the B component and the part of the infrared light component are received by each of the R, G, and B pixels, and are exposed to visible light. Accumulate over time.

  Next, by applying an H level to the all-pixel readout pulse φV1 at the beginning of the second frame, the signal charges of visible light (RGB) pixels are read out to the vertical transfer unit, and the background light exposure time ends.

  Subsequently, visible light is transmitted from the image sensor through the vertical transfer unit 72, horizontal transfer unit 73, and charge detection unit 74 shown in FIG. 10 at the timing (image sensor output timing of visible light exposure time 1) 85 shown in FIG. A pixel signal is output.

  In this case, the IR pixel signal is also being output, but the signal charges of the R, G, B, and IR pixels are transferred within the vertical transfer unit by performing vertical transfer so that the RGB pixels fit in the empty packet of the vertical transfer unit. In this case, all pixels can be output independently while causing a timing shift between timing 84 and timing 85 without mixing.

  After that, the operation of the image sensor is repeated in units of frames, and the irradiation infrared light exposure time and the background light exposure time are operated in a time-sharing manner within one frame.

  Next, the AE operation will be described from the signal output from the image sensor.

  First, the pixel signal of the irradiation infrared light exposure time is output over a period of about 1 V from the timing in the middle of the frame, the signal of the IR pixel that receives the infrared light is detected by the AE detection circuit of the ISP circuit, and the output is obtained. Using only the IR pixel signal detection result read by the CPU, the shutter speed of the irradiation infrared light exposure time of the next frame is determined by the CPU via a time filter.

  The pixel signal of the background light exposure time is output from the head of the frame, the RGB pixel signal receiving visible light is detected by the AE detection circuit of the ISP circuit, the output is read by the CPU, and the image sensor is read from the image sensor every frame. Using only the signal detection result of the output visible light pixel, the shutter speed of the next background light exposure time is determined by the CPU through the time filter.

  In this manner, each of the AEs independently of each other as if two cameras of 60 fps irradiation infrared light imaging and 60 fps background light color imaging were installed in spite of using a 60 fps image sensor. Control is realized.

  Next, the IR pixel signal received as the reflected light of the coherent random speckle pattern from the subject during the irradiation infrared light exposure time is collated with the reference pattern of the speckle pattern stored in advance, and the distance image calculation is performed from the address deviation. It is calculated by a circuit, converted into a distance image and output.

  Next, the RGB pixel signal received as reflected light from the subject during the background light exposure time is output as a color image processed by the color camera processing unit, such as white balance and edge enhancement processing.

  In this way, each of the functions functions independently as if two cameras were installed for 60 fps irradiation infrared light imaging and 60 fps background light color imaging in spite of using a 60 fps image sensor. This is realized by a single camera system that integrates a distance image by visible infrared light and a visible light color image.

  In this embodiment, a MOS image sensor having a global shutter and a global reset function can be used by making the row selection wiring of the IR pixel and the visible light pixel independent wiring.

(Summary)
As described above with reference to the drawings, the imaging apparatus according to the embodiments (first to third embodiments) of the present invention includes the infrared light pattern irradiating means, and the entire visible range is provided on the entire surface of the image sensor. In addition to this, a two-band transmission type optical filter that transmits only the vicinity of the wavelength of the infrared light pattern irradiation is installed, and the image sensor has a plurality of pixels that receive infrared light and a plurality of pixels that receive visible light. The discharge means for discharging the signal charges of all the pixels at the same time is installed, and the discharge means for discharging the signal charges of the pixels is set for all pixels by performing the irradiation infrared light exposure time and the background light exposure time in a time-sharing manner. It can be done by a single unit, and the optical filter installed in each pixel can adopt a general simple structure of a single layer, and optical mixing of visible light and infrared light by a two-band transmission type optical filter. Minimize It can range image of good color images and high accuracy can be realized by an inexpensive system.

  In addition, in order to control the irradiation infrared light exposure time and the visible light exposure time independently, the output of the detection circuit is divided into the infrared light AE and the visible light AE independently, and each is controlled by AE independently. For example, since the light source is different, the pixel signal level for receiving visible light with a background light exposure time with significantly different conditions and the pixel signal level for receiving infrared light with an irradiation infrared light exposure time are controlled to optimum light receiving times, respectively. Thus, as if two visible cameras and a distance measuring camera are operating at the same time, it is possible to further improve the image quality of visible image capturing and the accuracy of distance image capturing.

  Further, the pixel (PD) that discharges the accumulated charge of all the pixels that determines the start of the visible light exposure time, and the pixel that reads the accumulated charges of all the pixels that determine the end of the visible light exposure time from the application timing of the pixel (PD) signal charge discharge pulse ( PD) For the period up to the timing of applying the signal readout pulse, at least if the infrared light pattern irradiation is stopped, the irradiation time of the infrared light is reduced to a minimum, thereby reducing the power and heat generation. Can be done.

  Further, from the signal output of the pixel that receives the infrared light of the irradiation infrared light exposure frame, the signal output of the pixel that receives the infrared light of the background light exposure frame, taking each light reception time into account. If a circuit that corrects and subtracts the light reception time ratio is installed, it is possible to eliminate the minute infrared light component of the background light contained in the pixels that receive the infrared light of the irradiated infrared light exposure frame, It is possible to further improve the accuracy of distance image capturing.

  Further, from the signal output of the pixels that receive the visible light in each of the background light exposure frames, the signal output of the pixels that receive the infrared light in the same unit array of the background light exposure frames is changed to the red of each pixel. If a circuit that corrects and subtracts the ratio of the external light's spectral transmittance is added, even if the background light contains an infrared light component, the red light that is transmitted through the 2-band optical filter The external light component can be excluded from the signal output of the pixels that receive the visible light in each of the background light exposure frames, and a higher color reproduction of the visible image can be achieved.

  In addition, the signal charge readout gate of the pixel that receives the infrared light and the signal charge readout gate of the pixel that receives the visible light are independently wired, and the time division of the irradiation infrared light exposure and the background light exposure is one. If the exposure is completed with the signal charge readout pulse of the pixel receiving the visible light after the infrared light exposure is completed by applying the signal charge readout pulse of the pixel receiving the infrared light, the image is completed. Both a visible image and a distance image can be obtained at the same frame rate as the sensor frame rate, and the speed of the system can be greatly increased.

  The present invention realizes distance image capturing and color image capturing using an infrared light irradiation pattern in a single system, and can be used for an imaging apparatus such as a game machine for gesture authentication or a signage field such as person authentication.

21 Photodiode (pixel)
22 Vertical Transfer Unit 23 Horizontal Transfer Unit 24 Charge Detection Unit 25 Vertical Overflow Drain (VOD)
71 Photodiode (pixel)
72 Vertical transfer section,
73 Horizontal transfer section,
74 Charge detector,
75 Vertical overflow drain (VOD)
131 Infrared light exposure time 132 Visible light exposure time

Claims (11)

Infrared light pattern irradiation means;
A two-band transmission type optical filter that transmits a wavelength in the vicinity of the infrared light irradiated from the infrared light pattern irradiating means together with a visible region;
A plurality of first pixels that receive infrared light from the optical filter and perform irradiation infrared light exposure; a plurality of second pixels that receive visible light from the optical filter and perform background light exposure; An image sensor for discharging the signal charges of the first and second pixels simultaneously, and performing the irradiation infrared light exposure and the background light exposure in a time-sharing manner;
An imaging apparatus comprising:   2. The image pickup apparatus according to claim 1, wherein the pixels receiving the infrared light and the pixels receiving the background light are mixed to form an n × m unit array.   An optical filter that cuts visible light is installed in the pixel that receives infrared light, and an RGB optical color filter is installed in each pixel that receives the background light, and 2 × 2 is used as a unit array. Item 2. The imaging device according to Item 2.   The imaging apparatus according to any one of claims 1, 2, and 3, wherein a time division cycle of the irradiation infrared light exposure and the visible light exposure is synchronized in units of every other frame.   In order to control the irradiation infrared light exposure time and the visible light exposure time independently, the output of the detection circuit is separately divided into the infrared light AE and the visible light AE, and each is controlled independently. The imaging apparatus according to claim 4, characterized in that:   From the timing of applying the PD discharge pulse for discharging the accumulated charge of all the pixels determining the start of the visible light exposure time to the timing of applying the PD readout pulse for reading the accumulated charge of all the pixels determining the end of the visible light exposure time. 6. The imaging apparatus according to claim 4, wherein at least the infrared light pattern irradiation is stopped for a period.   From the signal output of the pixel that receives the infrared light of the irradiation infrared light exposure frame, the signal output of the pixel that receives the infrared light of the background light exposure frame, taking into account the respective light reception times, the light reception time 7. The imaging apparatus according to claim 5, further comprising a circuit for correcting and subtracting the ratio.   From the signal output of the pixel that receives the visible light of each of the background light exposure frames, the signal output of the pixel that receives the infrared light of the background light exposure frame, and the spectral transmittance of the infrared light of each pixel The imaging apparatus according to claim 5, further comprising a circuit for correcting and subtracting the ratio in consideration of the sensitivity according to claim 8.   The signal charge readout gate of the pixel receiving the infrared light and the signal charge readout gate of the pixel receiving the visible light are independently wired, and the time division of the irradiation infrared light exposure and the background light exposure is within one frame. The infrared light exposure is completed by applying the signal charge readout pulse of the pixel receiving the infrared light, and the light reception time is ended by the signal charge readout pulse of the pixel receiving the visible light. The imaging device according to claim 1.   In order to control the irradiation infrared light exposure time and the visible light exposure time independently, the output of the detection circuit is separately divided into the infrared light AE and the visible light AE, and each is controlled independently. The imaging apparatus according to claim 9, characterized in that:   The timing of applying the signal charge readout pulse of the pixel receiving the visible light that determines the end of the visible light exposure time from the timing of applying the PD discharge pulse for discharging the accumulated charge of all the pixels that determines the start of the visible light exposure time 11. The imaging apparatus according to claim 9, wherein at least the infrared light pattern irradiation is stopped for a period up to.