JP2011243862A - Imaging device and imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To realize functions of a distance image sensor acquiring a distance to a target subject utilizing light irradiated from a light source and a biometric detection sensor detecting a living body with light irradiated from a light source, using one imaging device acquiring an image in an RGB system or the like and an image of near-infrared light.SOLUTION: An imaging device according to the present invention includes an optical filter 32, and an output part 40 separately outputting a visible light image and a near-infrared light image by arranging a visible light sensor (visible light sensor constituted of two layers 38 and 39) and a near-infrared light sensor (non-visible light sensor constituted of two layers 36 and 37) along a direction of incoming light. Therefore, functions of distance measurement and/or biometric detection can be realized using one imaging device capable of high resolution imaging without image displacement between the both images and without reduction in resolution.

Description

  The present invention relates to an imaging apparatus that realizes the functions of a distance image sensor and a living body detection sensor using a single imaging device that acquires a captured (visible light) image such as RGB and a near-infrared light image, and the imaging device. .

For example, a distance image camera in which distance information is added to an image has been proposed for use in motion recognition and gesture recognition using the body. As a distance measuring method, a TOF (Time of Flight) method is generally used in which near-infrared light is emitted from a light source, and a distance is measured by measuring a delay time of light reflected by a subject.
In the TOF method, the range image sensor is a sensor that detects near-infrared light, and many TOF-type range image cameras output light and dark images and distance images using near-infrared light.

However, for example, depending on applications such as security applications and image composition, composition with a visible light color image is required, and in this case, a two-camera configuration of a distance image camera and a color image camera is adopted.
In the case of the two-camera configuration, since the viewpoints are different, the two images may be misaligned due to the parallax between the two cameras, and frame synchronization control is required between the two cameras. There is a disadvantage that it requires a synchronous control of the field of view between cameras, resulting in a complicated configuration.
The above-mentioned problem can be solved by capturing the distance image and the color image with an integrated imaging device. Conventionally, a method of capturing a distance image and a color image with an integrated imaging device has been proposed (see, for example, Patent Documents 1 and 2).

  Patent Document 1 discloses a method using two imaging devices for red (R), green (G), and blue (B) visible light detection and near-infrared light (NIR) detection, and RGB and NIR. A method of detecting with one imaging device is mentioned.

  In the method described in Patent Document 2, a color filter of an imaging device uses a red (R), green (G), green (G), or blue (B) array called a Bayer array, and one of G is near red. R, G, B, NIR is detected by replacing with outside (NIR).

On the other hand, a method for detecting only a human skin from an image has been proposed (see, for example, Patent Document 3).
By the method described in Patent Document 3, it is possible to extract an image for detecting a living body (for example, only a human skin).
The obtained biological image is a planar image and becomes a bright and dark image by near infrared light.

Special table 2007-526453 JP 2008-8700 A Japanese Patent No. 2823564

In the method using two imaging devices described in Patent Document 1, the focal length of the imaging lens is shortened because a prism that separates RGB and NIR wavelengths exists between the imaging lens and the imaging device. There are restrictions. As a result, it is difficult to widen the angle of view.
If another method using one image pickup device described in Patent Document 1 is employed, the difficulty of expanding the angle of view is eliminated. The imaging device used in this method is one in which NIR pixels are interspersed between RGB visible light pixels.
Therefore, in the image acquired from this imaging device, sufficient resolution cannot be obtained for each of the RGB image and the distance measurement image, and the distance measurement pixel (NIR pixel) and the RGB pixel are the same for the same portion of the subject. The disadvantage is that location information cannot be obtained.

The method described in Patent Document 2 has a similar disadvantage. In the method described in Patent Document 2, by reducing one green (G) pixel in a plurality of pixels of a repetitive color array, the resolution is higher than that of an RGB image that can be produced by a primary color imaging device having the same number of pixels. Is disadvantageous.
In addition, since the distance measurement pixel cannot acquire information at the same position as the RGB pixel for the same portion of the subject, the distance must be measured at a position of an image different from the RGB image.

If distance information can be added in the method capable of acquiring a biological image described in Patent Document 3, for example, a human face or hand can be captured as a stereoscopic image. Color information can also be added by synthesizing a color image with a biological image.
However, in the method described in Patent Document 3, it is difficult to widen the angle of view, as in Patent Document 1.
Similarly to Patent Documents 1 and 2, even when the angle of view is widened, the ratio of the distance measurement pixels in the imaging device is high to some extent. Therefore, there are disadvantages that a sufficient resolution cannot be obtained for each of the RGB image and the distance measurement image, and that the distance measurement pixel (NIR pixel) cannot acquire information at the same position as the RGB pixel.

A biological image can be extracted using NIR images and RGB color images obtained by separately capturing the same subject at the same angle of view. In this case, the living body is detected from the NIR image, and the RGB image portion having the same pixel address range as the detected living body portion may be extracted.
However, since the two images obtained even when the same subject is imaged at the same angle of view are usually somewhat different, the deviation of the image becomes an error component of the biological image, and the high accuracy of the biological image is obtained. Extraction is difficult.

  The present invention provides an imaging device having a configuration capable of separately outputting a visible light image and a near-infrared light image, each of which has a high resolution and can easily correspond to a position in the image, and can easily measure a distance and detect a living body. It is to provide. In addition, the present invention provides an imaging apparatus (for example, a camera system) capable of performing at least one of distance measurement and living body detection as well as normal RGB image acquisition using the imaging device having such a configuration. is there.

An imaging device according to an aspect of the present invention includes a pixel array in which an optical filter is disposed on a side on which image light is incident, and the pixels of the pixel array include a first filter region, a visible light sensor unit, and invisible light. It has a sensor part.
The first filter region transmits visible light having a wavelength corresponding to at least one of a plurality of colors of primary colors or complementary colors, and invisible light having a wavelength longer than that of visible light.
The visible light sensor unit has a main sensitivity to the visible light from the first filter region.
The invisible light sensor unit is formed at a position farther from the side where the image light enters than the visible light sensor unit, and has a main sensitivity to the invisible light from the first filter region.

In the present invention, the output unit is provided for each pixel of the pixel array, or is shared by a plurality of pixels commonly connected to the output signal line. The output unit separates and outputs a signal of a visible light image generated by photoelectric conversion of the visible light sensor unit and a signal of a non-visible light image generated by photoelectric conversion of the non-visible light sensor unit.
In the present invention, it is preferable that the output unit includes a plurality of charge storage units and a switch that distributes the generated charges of the invisible light sensor unit to the plurality of charge storage units.

  According to such a configuration, the visible light sensor unit and the invisible light sensor unit are provided in the same pixel. For this reason, each pixel data of the visible light image in which the pixel data is generated by the photoelectric conversion of the visible light sensor unit and the invisible light image in which the pixel data is generated by the photoelectric conversion of the invisible light sensor unit correspond in position. . The output unit is provided for each pixel or shared by a plurality of pixels commonly connected to the output signal line. Therefore, it is possible to separate and output a visible light image and a non-visible light image corresponding to pixel data in position. At this time, when the visible light sensor unit is provided for each pixel, the resolution of the visible light image, in other words, the pixel data density included in one frame image is maximized, so that the image quality is not impaired.

  On the other hand, each pixel data of the non-visible light image is usually less in charge amount than the pixel data of the visible light image. In particular, in the case of distance measurement or living body detection, non-visible light is applied to a subject, and reflected light (returned light) scattered by the reflection is received. Therefore, in these applications, the data amount (charge amount) of the non-visible light pixel obtained from the non-visible light sensor unit for one pixel may be orders of magnitude smaller than the data amount of the visible light pixel.

  In a preferred embodiment of the present invention, since a plurality of charge storage units are provided in the output unit and the input of the charges is controlled by a switch, the non-visible light data for a plurality of pixels are combined (combined). )be able to. The non-visible light data after synthesis is a part of the non-visible light image, but since the amount of data increases, it is input to the processing unit in the imaging device without being buried in noise, or externally for processing. It can be discharged.

This desirable configuration of the output unit is also useful for distance measurement and living body detection.
For example, in the case of the distance measurement of the TOF method, the object is pulsed with invisible light for a certain time, the return light of the invisible light is incident on the imaging device, and the visible light sensor unit passes through a fixed light receiving time. Is output as a detection signal. For example, the delay until the detection signal is obtained is detected with reference to the time when the emission pulse is emitted. Specifically, for example, there is a method in which the pulse signal charge is divided into two in the middle of the pulse duration of the detection signal corresponding to the emission pulse length of invisible light, and the delay time is obtained from the ratio of the divided signal charge amount. is there.
The configuration of the output unit having a plurality of charge storage units and switches for distributing them is suitable for implementation of this method.

In the case of living body detection, two non-visible lights having different wavelength regions, one of which is mainly reflected by human skin and the other is mainly absorbed, are applied to the subject. In addition, as for these two invisible lights, it is permitted that a part of the short wavelength side of the invisible light region overlaps the visible light region.
Then, the return light of each invisible light is received by the same imaging device. At this time, in order to separate two detection signals obtained by photoelectrically converting two return lights corresponding to two invisible lights, an output unit having a plurality of charge storage units and a switch for distributing them The configuration can be suitably used.

An imaging apparatus according to another aspect of the present invention includes an imaging device including a pixel array in which an optical filter is disposed on an image light incident side, a light emitting unit, a lens, a control unit, and a signal processing unit.
The light emitting unit emits and projects at least one invisible light having a plurality of wavelength regions that are single or different from each other on the longer wavelength side than the wavelength region of visible light, and a part of the projection light is reflected by the subject. Light is directed to reach the imaging device.
The lens focuses the return light on the imaging device.
The control unit controls a light emission timing of the light emitting unit.
The signal processing unit processes a signal of a visible light image output from the imaging device.

In the present invention, the imaging device in the imaging apparatus is configured similarly to the imaging device according to one aspect of the present invention described above. Therefore, in addition to processing and outputting a visible light image, the above-described processing for distance measurement and living body detection can be performed.
More specifically, the signal processing unit processes the signal of the visible light image. In addition, the signal processing unit and the control unit control the light reception time and near infrared light emission of the imaging device, and measure the distance to the subject based on the obtained signal of the invisible light image. At least one of an arithmetic processing to perform and a biological detection processing to detect an image of the skin of the subject.

  According to the present invention, an imaging device having a configuration capable of separately outputting a visible light image and a near-infrared light image, each of which has a high resolution and can easily correspond to a position in the image, and can easily measure a distance and detect a living body. A device can be provided. In addition, according to the present invention, there is provided an imaging device (for example, a camera system) capable of performing at least one of distance measurement and living body detection while obtaining an image such as normal RGB using the imaging device having such a configuration. be able to.

It is a block diagram of the imaging device containing the structural example in which both distance measurement and biometric detection which concern on the 1st-11th embodiment are possible. It is a figure which shows typically the cross-sectional structure in the pixel array of an imaging device. (A) is a figure which shows the basic color arrangement | sequence of a primary color type | system | group optical filter. FIG. 6B is a diagram illustrating an arrangement of colors and wavelength transmission characteristics of the primary color optical filter in the first embodiment. It is a schematic diagram of the multiple diffused layer structure which implement | achieves the vertical stacking of the RGB three sensor parts from the board | substrate deep part side. It is a graph showing the light absorption length in a silicon | silicone which is a common semiconductor substrate material versus wavelength. It is a more detailed schematic cross-sectional view of the imaging device according to the first embodiment. It is explanatory drawing of the measuring method containing the pulse waveform for showing the timing of distance measurement. It is a figure which shows the basic color arrangement | sequence of a complementary color type | system | group optical filter, and the arrangement | sequence of the color and wavelength transmission characteristic of a complementary color type | system | group optical filter in 2nd Embodiment. It is a schematic cross section in the pixel array of the imaging device concerning 2nd Embodiment. It is a schematic cross section in the pixel array of the imaging device concerning 3rd Embodiment. It is a schematic cross section in the pixel array of the imaging device concerning 4th Embodiment. It is a schematic cross section in the pixel array of the imaging device concerning 5th Embodiment. It is a schematic cross section in the pixel array of the imaging device concerning 6th Embodiment. FIG. 16 is a wiring diagram when a charge storage unit is commonly used for four pixels in the sixth embodiment. It is the pixel arrangement | sequence figure used for description of the data interpolation method. It is explanatory drawing of the interpolation method. It is explanatory drawing of the interpolation method. It is explanatory drawing of the interpolation method. It is explanatory drawing of the interpolation method. FIG. 16 is a wiring diagram when a charge storage unit is commonly used for four pixels in the seventh embodiment. It is a schematic cross section in the pixel array of the imaging device concerning 9th Embodiment. In the ninth embodiment, it is a wiring diagram in the case where a charge storage unit is commonly used for four pixels. It is a schematic cross section in the pixel array of the imaging device concerning 10th Embodiment. In a 10th embodiment, it is a wiring diagram in case a charge storage part is used in common for four pixels. It is a schematic cross section in the pixel array of the imaging device concerning 11th Embodiment. In an 11th embodiment, it is a wiring diagram in case a charge storage part is used in common for four pixels.

Embodiments of the present invention will be described in the following order with reference to the drawings.
1. First Embodiment: An embodiment in which two layers of sensor portions are arranged in the substrate depth direction in pixels of primary color arrangement.
2. Second Embodiment: This is an embodiment in which three layers of sensor portions are arranged in the substrate depth direction with pixels of complementary color arrangement.
3. Third Embodiment: An embodiment having sensor units for all colors in a single pixel.
4). Fourth Embodiment: This is an embodiment in which two layers of sensor units are arranged in the substrate depth direction with pixels of complementary color arrangement.
5). Fifth embodiment: an embodiment in which four layers of sensor units are arranged in the substrate depth direction in pixels of primary color arrangement.
6). Sixth Embodiment: This is an embodiment in which NIR pixel data is mixed by dividing the charge storage section into dedicated and shared.
7). Seventh embodiment: an embodiment in which the imaging device of the sixth embodiment is changed to a complementary color system and the sensor unit is arranged in three layers.
8). Eighth embodiment: an embodiment in which the sensor unit is changed to a two-layer arrangement from the seventh embodiment.
9. Ninth embodiment: an embodiment having an NIR sensor unit common to pixels.
10. Tenth Embodiment: Another embodiment having an NIR sensor unit common to pixels.
11. Eleventh embodiment: Another embodiment having a common NIR sensor unit for pixels.

<1. First Embodiment>
[Configuration of imaging device]
FIG. 1 shows a configuration diagram of an imaging apparatus including a configuration example capable of both distance measurement and living body detection.
The imaging device 1 illustrated in FIG. 1 can be used for consumer electronic devices such as game machines, industrial electronic devices such as in-vehicle sensor devices and measuring machines.
The imaging apparatus 1 includes a first light emitting unit 2A for measuring the distance from the subject 100 and a second light emitting unit 2B for detecting a difference in reflected light amount between the first light emitting unit 2A. Although the details will be described later, the first and second light emitting units 2A and 2B emit non-visible light in different wavelength regions toward the subject 100, and the difference in the amount of reflected light at that time is the first light emitting unit 2A. And at the time of living body detection using both the second light emitting unit 2B. In order to measure the distance from the subject 100, it is only necessary to irradiate invisible light from at least one of the first light emitting unit 2A and the second light emitting unit 2B. Therefore, here, the first light emitting unit 2A is used as an example. Use.
In the specification of the present invention, when it is referred to as invisible light, it is limited to invisible light having a wavelength longer than the wavelength region of visible light. The invisible light may be any of near-infrared (NIR) light, infrared (IR) light close to the long wavelength side with respect to the wavelength range of visible light, and light partially including the IR region, mainly NIR light. Hereinafter, invisible light is referred to as NIR light.

The imaging apparatus 1 includes an imaging device 3 that receives return light when near-infrared light (NIR light) from the first and second light emitting units 2A and 2B is reflected by the subject 100, and the subject 100 and the imaging device 3 And a lens 4 that is disposed between them and forms an image of the return light on the imaging device 3.
The imaging apparatus 1 further includes a signal processing unit 5 and a control unit 6 that process the output signal of the imaging device 3. The control unit 6 performs overall control of the first and second light emitting units 2A and 2B, the imaging device 3, and the signal processing unit 5, and controls the optical system including the lens 4 in some cases.
Specifically, the control unit 6 controls the light emission timing of the first light emitting unit 2A (and the second light emitting unit 2B: in the case of biological detection), the electronic shutter of the imaging device 3, and the timing of the switch 43 (described later). To do. The signal processing unit 5 has a configuration capable of executing processing of image distance measurement and living body detection in addition to processing of visible light images such as RGB (noise removal, color correction, etc.) from output data from the imaging device 3.

The first light emitting unit 2A includes a light emitting member 21 such as a light emitting diode (LED) and a driver circuit 22 for causing it to emit light.
The light emitting member 21 of the first light emitting unit 2A emits invisible light, for example, near infrared light, and the light emitting direction of the near infrared light toward the light emitting member 21 is directed to the subject 100.
Similarly to the first light emitting unit 2A, the second light emitting unit 2B includes a light emitting member 21 and a driver circuit 22 for causing it to emit light. The wavelength of the light emitting member 21 in the second light emitting unit 2B is also mainly near infrared light, and the light emitting direction of the light emitting member 21 is directed to the subject 100. However, the near-infrared light from the first light emitting unit 2A and the near-infrared light from the second light emitting unit 2B are light in different wavelength regions.

  Sensitivity characteristics for the color and brightness of the human eye have sensitivity in the visible light region. Although the visible light region has various definitions, it is generally a wavelength region of electromagnetic waves having an upper limit in the vicinity of 700 to 750 nm (for example, 780 nm) and a lower limit of slightly less than 400 nm (for example, 380 nm). An infrared region adjacent to the visible light region is referred to as a near infrared region, and this region is a wavelength region of electromagnetic waves from a lower limit of 700 to 800 nm to an upper limit of 1.5 to 3 μm (for example, 2.5 μm). However, the human eye has little sensitivity on the longer wavelength side than about 700 nm.

By the way, it is said that the reflectance of a living body (typically human skin) increases when the wavelength of light is around 750 to 800 nm and decreases at 970 nm. As the most preferable wavelength of light, the wavelength of the first light emitting unit 2A is around 800 nm (near the lower limit of near infrared light), and the wavelength of the second light emitting unit 2B is around 970 nm.
However, in order to detect a living body, the difference in the amount of reflected light only needs to be set to a recognizable level. Therefore, if the wavelength difference can be set within that range, the wavelength of light from the first and second light emitting units 2B This is not the case.
Although it is desirable that the wavelength region of the first light emitting unit 2A and the wavelength region of the second light emitting unit 2B do not overlap, the emission wavelength of the LED has a certain width, and thus it is allowed that the two wavelength regions partially overlap. .
The two non-visible lights emitted by the first and second light emitting units 2A and 2B are allowed to overlap a part of the short wavelength side of the non-visible light region with the visible light region depending on the wavelength definition of the visible light. Is done. For example, when the long-wavelength side boundary of visible light is defined as 800 nm, if the wavelength of the first light emitting unit 2A is around 800 nm, a part of the short wavelength side of the wavelength range overlaps the visible light region. On the other hand, when the long-wavelength side boundary of visible light is defined as around 750 nm, the wavelength range of the first light emitting unit 2A does not usually overlap with the visible light region.

  The imaging lens 4 is a general lens used in cameras, camcorders, and the like, and is selected according to specifications such as the size and angle of view of the imaging device 3. The lens 4 is desirably arranged as a part of an optical system having a focus adjustment function according to the distance from the subject 100 under the control of the control unit 6.

[Configuration of imaging device]
FIG. 2 schematically shows a cross-sectional configuration of the pixel array of the imaging device 3.
An imaging device 3 for generating a captured image (referred to as a visible light image in this specification) such as an RGB image at a high resolution includes a microlens array 31 and an optical filter 32 from the light incident side as shown in FIG. The pixel circuit layer 33 and the photosensor are formed on the semiconductor substrate 35.
A part of the photosensor and the pixel circuit are formed using an active region (for example, a plurality of wells) of the semiconductor substrate 35. The pixel circuit layer 33 is an arrangement region other than a circuit element such as a substrate transistor formed on the semiconductor substrate 35. For example, a passive element such as a capacitor is used as a pixel circuit by using a conductive layer stacked via a dielectric film. Arranged in layer 33. Electrical interference between the photosensor and the substrate transistor of the pixel circuit is prevented by element isolation or a channel stopper region. It is possible to form some transistors in the pixel circuit layer 33 from TFTs or the like.

The light condensed by the imaging lens 4 is further condensed for each pixel by the microlens array 31, passes through the optical filter 32, and enters the photosensor. The optical filter 32 is divided into a filter portion having wavelength selectivity through which light collected by the microlens array 31 is transmitted and a portion (so-called black matrix) that blocks light between the filter portions. The light blocking part is not essential, but it is desirable to have it in order to prevent color mixing and external light mixing.
The wavelength-selected light passes through the pixel opening provided so as not to block the light by the wiring or conductive layer of the pixel circuit layer 33, and is incident on the photosensor without loss, and is photoelectrically converted there. Charges generated by photoelectric conversion (referred to as light received charges or photo charges) are accumulated in the photosensor for a certain period of time and then discharged from the pixel array as image data via the output portion of the pixel circuit. In some cases, a processing circuit for output processing (noise suppression and signal conversion) is arranged in a peripheral region other than the pixel array in the imaging device 3, and in this case, the pixel data is processed after the output processing in the peripheral circuit. Is output to the signal processing unit 5.

The microlens array 31 is for improving the light collection efficiency to the photosensor.
The present invention is characterized by the optical filter 32 and the photosensor 34 among the components shown in FIG. 2, which will be described in detail below.

Regarding the basic color arrangement, the optical filter 32 has a color arrangement called a Bayer arrangement which is a primary color filter of the general imaging device 3 as shown in FIG. In the Bayer array, filter portions that transmit each color are arranged in an array of red (R), green (G), green (G), and blue (B).
Each filter section of each color filter section has a characteristic of transmitting a near-infrared (NIR) wavelength in addition to the transmission of one corresponding color. Each filter unit has transmission characteristics of R + NIR, G + NIR, and B + NIR.
The optical filter array shown in FIG. 3B indicates that it has a transmission characteristic of near infrared wavelength in addition to the visible light transmission characteristic of each color by the notation shown by R + NIR, G + NIR, and B + NIR. In the first embodiment, an optical filter 32 having the arrangement shown in FIG. 3B is used.

  As a technique for realizing the characteristics of the optical filter 32, a technique in which a plurality of high-refractive index films made of a metal oxide film or the like are stacked in layers with a low-refractive index light-transmitting film interposed therebetween can be exemplified. By controlling the type of the high refractive index film (for example, the material of the metal oxide film) and the number of layers, the type of the low refractive index film (for example, the material of the light transmitting film), and the thickness, a multilayer having a desired wavelength selection characteristic. A membrane can be realized.

  The photosensor 34 shown in FIG. 2 includes a visible light sensor unit that detects visible light of RGB close to the substrate surface, and an NIR formed at a position farther in the light incident direction (substrate deeper side) than the visible light sensor unit. It has a vertically stacked structure of two sensor parts with a non-visible light sensor part to be detected (hereinafter referred to as an NIR sensor part).

In order to realize a visible light sensor unit located on the deeper side of the substrate as the sensor unit with a longer detection wavelength, the visible light sensor unit has N-type and P-type as a plurality of layers corresponding to the detection wavelengths corresponding to a plurality of colors. The semiconductor layer has a multilayer structure.
In other words, in the visible light sensor unit that photoelectrically converts three or four colors of visible light, three or four sensor units having main sensitivities in different wavelength regions are imaged as the sensor units corresponding to the wavelength regions having longer wavelengths. It is arranged at a position far from the optical filter 32 side through which light enters.

  Note that since the imaging device 3 in this embodiment is an RGB primary color system, it receives three colors of visible light. However, as in other embodiments described later, for example, in the case of a complementary color system, it receives four colors of visible light. To do. Further, even in the primary color system, there is a case of a four-color arrangement having light receiving pixels of white (W) in addition to RGB.

In the case of the color arrangement shown in FIG. 3B, the visible light sensor unit has a vertically stacked structure of three sensor units having main sensitivity for three colors.
FIG. 4 is a schematic diagram of a multiple diffusion layer structure that realizes vertical stacking of three sensor units (photodiodes having a PN junction) corresponding to RGB in order from the deep side of the substrate. FIG. 5 is a graph showing the light absorption length versus wavelength in silicon, which is a general semiconductor substrate material.

The vertically stacked structure of the three sensor units realized by the multiple diffusion layers shown in FIG. 4 utilizes the optical characteristics of silicon in FIG.
As shown in FIG. 5, the light absorption coefficient on the short wavelength side is high on the silicon surface (substrate surface) side, and conversely, the light absorption coefficient on the long wavelength side is high on the deep side of the substrate. This is due to the fact that the greater the wavelength of light incident on the silicon substrate, the deeper the light enters the silicon before being absorbed. Therefore, when the photodiode is formed near the surface inside the semiconductor substrate, the absorption coefficient of short-wavelength light is relatively high with respect to long-wavelength light, and the proportion of short-wavelength light converted into carriers near the surface is higher. Since it is high, sensitivity to short wavelength light is relatively high.

Most of the blue light having a wavelength of 400 to 490 nm is absorbed by the silicon substrate at a depth of about 0.2 to 0.5 μm. For this reason, as shown in FIG. 4, the photodiode (sensor part) having the main sensitivity to the wavelength of blue (B) by the shallowest n-type layer 39 and the next deepest p-type well 38 in the example of FIG. It is formed around a depth of about 0.2 μm from the substrate surface.
On the other hand, most of red light having the longest wavelength of 600 to 750 nm is absorbed by silicon at a depth of about 2.0 to 7.5 μm. Therefore, as shown in FIG. 4, the deepest n-type well 37 and p-type semiconductor substrate 35 allow another photodiode (sensor unit) having a main sensitivity to the red (R) wavelength to be the example of FIG. Then, it is formed around a depth of about 2.8 μm from the substrate surface.
Green light having a wavelength between 500 and 560 nm and between blue and red is mostly absorbed by silicon at a depth of about 0.8 to 1.5 μm. Therefore, in the example of FIG. 4, the p-type well 38 and the n-type well 37 allow another photodiode (sensor part) having a main sensitivity to the green (G) wavelength to reach 0. It is formed around a depth of 8 μm.
The depth of the center of each sensor part (pn junction surface) in FIG. 4 is an example, the blue (B) sensor part is 0.2 to 0.5 μm, the green (G) sensor part is 0.8 to 1.5 μm, The red (R) sensor unit is arbitrarily set within each depth range of 2.0 to 7.5 μm.

  On the other hand, near-infrared light having a wavelength of 800 to 900 nm is absorbed by the silicon substrate at a depth of about 30 μm or less. Therefore, although not shown in FIG. 4, a near-infrared light sensor portion is formed by providing a p-type well deeper than the n-type well 37.

FIG. 6 shows a more detailed schematic cross-sectional view of the imaging device 3 according to the first embodiment. FIG. 6 is a diagram showing the configuration of FIG. 2 in more detail, and configurations common to both diagrams are denoted by the same reference numerals.
The cross section shown in FIG. 6 corresponds to a cross section taken along line AA in the optical filter array shown in FIG.
As shown in FIG. 6, a p-type well 36 common to the pixels is formed in the semiconductor substrate 35 in order to add a near infrared light (NIR) sensor unit to the configuration of FIG. 4.

A photosensor 34R of a pixel that receives light transmitted through the filter unit (R + NIR) is formed inside a p-type well 36 that is common to the pixels.
The deepest n-type well 37 in the first layer is formed at a depth (about 30 μm or less from the surface) that is strongly sensitive to the emission wavelength (near infrared light) of the first light emitting portion 2A (FIG. 1). . The third n-type layer 39R is formed at a depth (about 2.0 to 7.5 μm from the surface) that is highly sensitive to red light. The second p-type well 38R is formed between the first and third layers in the depth direction.
A non-visible light sensor portion for NIR detection is formed at a pn junction position between the p-type well 36 and the first n-type well 37, and the second p-type well 38R and the third n-type layer 39R. A visible light sensor unit for R detection is formed at the pn junction position.

  In this example, since the signal charge (light-receiving charge) is an electron, the n-type layer of each sensor unit is a charge storage layer. Therefore, NIR pixel data constituting an NIR image (non-visible light image) is output from the n-type well 37. In addition, the R pixel component data of the visible light image is output from the n-type layer 39R.

  As described above, the R and NIR lights transmitted through the filter unit (R + NIR) are dispersed by the depth of the photosensor 34R, separated by the output unit formed in the pixel circuit layer 33, and output.

In the photosensor 34G that receives light transmitted through the filter portion (G + NIR), the n-type well 37 in the first layer is the same as the photosensor 34R. However, the p-type well 38G of the second layer and the n-type layer 39G of the third layer are formed so that the boundary (pn junction) between them is located in the sensitivity range of green (G).
Specifically, the third n-type layer 39G is formed at a depth (about 0.8 to 1.5 μm from the surface) that is strongly sensitive to green light, and the second p-type well 38G It is formed at a depth between the first and third layers.
The G pixel component data of the visible light image is output from the n-type layer 39G, and the NIR pixel data of the non-visible light image is output from the n-type well 37.
As described above, the G and NIR lights transmitted through the filter unit (G + NIR) are dispersed by the depth of the photosensor 34G, and are separated and output by the output unit formed in the pixel circuit layer 33.

In the photosensor 34B that receives light transmitted through the filter portion (B + NIR), the n-type well 37 in the first layer is the same as the photosensors 34R and 34G. However, the p-type well 38B of the second layer and the n-type layer 39B of the third layer are formed so that the boundary (pn junction) between them is located within the sensitivity range of blue (B).
Specifically, the third n-type layer 39B is formed at a depth (approximately 0.8 to 1.5 μm from the surface) that is highly sensitive to blue light, and the second p-type well 38B It is formed at a depth between the first and third layers.
Data of the B pixel component of the visible light image is output from the n-type layer 39B, and NIR pixel data of the non-visible light image is output from the n-type well 37.
As described above, the B and NIR lights transmitted through the filter unit (B + NIR) are dispersed by the depth of the photosensor 34B, separated by the output unit formed in the pixel circuit layer 33, and output.

[Output section]
In the case of this example, an output unit 40 is provided for each pixel.
The output unit 40 includes a plurality of charge storage units 41 and a switch 43 that switches the output destination of the received light charges from the n-type well 37 to one of the plurality of charge storage units 41. The output unit 40 includes an output circuit 42. The output circuit 42 has a function of amplification and reset as necessary based on the function of the selector. The visible light pixel data of any of the n-type layers 39R, 39G, and 39B and the NIR pixel data from at least one of the charge storage units 41 can be input to one output circuit 42. The output circuit 42 is basically a kind of selector that selects and outputs these pixel data.
The output unit 40 inputs a control signal from the control unit 6 of FIG. 1 via an external terminal (not shown), or the control signal is received by a control circuit inside the imaging device 3 that is controlled by the control unit 6. Given. Thereby, switch switching, charge accumulation control and output selection control are executed.

When visible light pixel data is output from the output unit 40, visible light pixel data is sequentially output from an unillustrated output signal line (also referred to as a vertical signal line) through a peripheral circuit in the imaging device 3, as shown in FIG. It is sent to the signal processing unit 5. The signal processing unit 5 performs necessary signal processing such as noise removal and color correction to generate a visible light image (in this example, an RGB image).
On the other hand, when non-visible light data (NIR pixel data) is output from the output unit 40, this may be used as configuration data of a bright and dark image. In this example, NIR pixel data is used for distance detection and biological detection. To do. This detection process is performed by the signal processing unit 5 of FIG.

Depending on the type of device, such as whether the imaging device 3 is a CCD or a CMOS image sensor, the configuration and location of the output unit 40 may differ.
For example, in the case of a CMOS image sensor, a circuit called a pixel circuit that performs signal charge resetting and amplification readout operations is provided for each pixel. Therefore, the output unit 40 can be formed in the pixel circuit for each pixel. In this case, although the scale of the pixel circuit is increased, the charge storage unit 41 is formed of, for example, a capacitor having a dielectric film sandwiched between two conductive layers, and is thus stacked on the upper layer of the substrate transistor circuit. The output circuit 42 can also be configured with a plurality of switches. Therefore, in the case of a CMOS image sensor, the output unit 40 can be mounted without increasing the pixel size.

On the other hand, in the case of a CCD, most of the pixel area is occupied by a vertical register and a light receiving unit for vertical charge transfer, and in a switch type, a transistor called a read gate is provided for each pixel. Therefore, the addition of a transistor is not as easy as a CMOS image sensor.
However, it is possible to form a transistor circuit with TFTs on a light shielding film that covers the readout gate and the vertical CCD widely. In the backside illumination type imaging device, a microlens array and a color filter are attached to the backside of the substrate, and light can be received from the backside of the substrate and discharged from the frontside. For this reason, even if a relatively large circuit is formed on the surface side, it is unlikely that the pixel size is increased.
Therefore, the output unit 40 can be mounted even when the imaging device 3 is a CCD.

  A configuration in which the output unit 40 is shared by a plurality of pixels is possible regardless of the type of the imaging device 3. For example, in the case of pixels commonly connected to one output signal line, the output unit 40 can be shared by further adding a switch for switching the input to the output unit 40 for a plurality of pixels.

[Distance measurement method]
The distance measurement to the subject will be described.
Several distance measurement methods using TOF (Time of Flight) that project light, detect reflected light from the subject 100, and measure the delay time have already been proposed.

In the embodiment of the present invention, the following method can be employed.
As shown in FIG. 1, invisible (NIR in this case) pulsed light is emitted toward the subject 100 from the first light emitting unit 2A.
Reflected light (return light) from the subject 100 is received by the imaging device 3, and the return light is photoelectrically converted by the internal invisible light (NIR) sensor unit to generate electric charges. The generated charges are distributed to the first and second charge storage units 41 and 41 by the switch 43 that operates alternately in synchronization with the emission of the pulsed light, and the output circuit 42 outputs an output signal line or register outside the pixel. Etc. are transferred.
At this time, the distance to the subject 100 is obtained from the distribution ratio of the charges distributed to the first and second charge storage units. At this time, there is a period in which the reflected light from the subject is not incident on the sensor unit corresponding to the pulsed irradiation light. During this period, the light photoelectrically converted by the sensor unit is background light of the subject or noise light from the surroundings. Therefore, the generated charge in this period is transferred to the third charge storage unit 41 by operating the switch 43. The background light charge is sent to the external signal processing unit 5 so as to be distinguishable from the detected charge, and is used by the signal processing unit 5 for processing for background light removal.
The period other than the background light or noise period is the distance measurement period.

FIG. 7 shows a setting example of distance measurement timing for a specific pulse waveform.
FIG. 7A shows a light emission pulse, and FIG. 7B shows a light reception pulse delayed in time with respect to the light emission pulse, that is, a pulse of a signal generated by photoelectric conversion (hereinafter referred to as a light reception pulse).
The light emission pulse shown in FIG. 7A shows only one pulse here, but in actuality, the light emission pulse is repeatedly emitted toward the subject periodically. Return light generated by reflection from the subject is received by the imaging device. The light reception start point at this time is delayed in time from the corresponding light emission pulse issuance start point. Since the time difference from the start of light emission to the start of light reception is approximately proportional to the distance of the subject, this time difference is the detection target.

In this example, a distance measurement period is set for time difference measurement. Here, the duration of the light reception pulse from the start of light reception to the end of light reception is defined as a distance measurement period. Since the distance measurement period only needs to be a period in which the light reception pulse is reliably maintained, it can be arbitrarily set as long as it is within the duration of the light reception pulse.
A control unit such as a CPU switches the switch 43 shown in FIG. 6 at a certain timing within the distance measurement period, so that the A charge in the light receiving pulse part before that and the B charge in the light receiving pulse part after that change. The charge of the received light pulse is divided into two. The distributed A charges and B charges are stored in two different charge storage units 41 and 41 shown in FIG. 6, and are read out as voltage values through output signal lines, for example, at different timings so that they can be distinguished from each other.

  The read A charge and B charge (two voltage values proportional to the charge amount) are input to the signal processing unit 5 shown in FIG. 1, and the ratio of both charges (both voltage values) is calculated. From the result of this calculation, the signal processing unit 5 calculates the distance to the subject that is approximately proportional to the time difference that is the detection target, that is, the time difference from the start of light emission to the start of light reception.

In addition, about the distance measuring method of this invention, the method is not limited to what was mentioned above. Other methods can be realized by changing the switch 43, the charge storage unit 41, the wiring, the signal processing unit 5 and the like at the subsequent stage from the non-visible light (NIR) sensor unit.
Further, since this distance measurement uses a sensor unit different from the RGB sensor unit, acquisition of RGB images and distance measurement can be performed simultaneously in parallel. Furthermore, since the distance measurement can be performed using pixel address data corresponding to the RGB image, the correspondence between the measurement result and the subject can be easily obtained. For example, when the image position of the subject (pixel address range) and the pixel address where the return light of distance measurement is detected are greatly different, the measurement result can be invalidated as an error.

[Biological detection method]
Next, a method for detecting a living body such as human skin will be described.
In FIG. 1, invisible light (first NIR light) having a first wavelength region is projected from the first light emitting unit 2 </ b> A toward the subject 100.
Reflected light (returned light) from the subject 100 is received by the imaging device 3 and subjected to photoelectric conversion by an internal invisible light (NIR) sensor unit.
Invisible light (second NIR light) in a wavelength region different from the first wavelength region is projected from the second light emitting unit 2B.
The reflected light (return light) from the subject 100 is photoelectrically converted by the imaging device 3 in the same manner.
The projection by the first light emitting unit 2A and the projection by the second light emitting unit 2B are controlled so as to be alternately performed. Further, a switch 43 that distributes the output of the invisible light (NIR) sensor unit to the plurality of charge storage units 41 is opened and closed in synchronization with the projection by the first and second light emitting units 2A and 2B. As a result, the reflected light from the first light emitting unit 2A is input to the first charge storage unit 41, and the reflected light from the second light emitting unit 2B is input to the other second charge storage unit 41 and stored. .

Each detection signal is taken into the signal processing unit 5 through the wiring. A first invisible light image is obtained from NIR pixel data generated by reflected light of the first NIR light emitted from the first light emitting unit 2A. A second invisible light image is acquired from NIR pixel data generated by the reflected light of the second NIR light emitted from the second light emitting unit 2B.
Since the first and second invisible light images are images of the same object captured at the same angle of view for the same time, the amount of data at each address in the image is basically the same if the wavelength of the irradiation light is the same. It becomes.

Here, the wavelengths of the two irradiation lights are selected so that the difference in reflectance between human skins becomes large. For this reason, when the difference is taken between the pixel data of each address in the first invisible light image and the second invisible light image, this difference increases only in the image portion of the human skin, and the increased portion is determined as a threshold value. Living body detection is performed by determining.
At that time, after correcting the sensitivity difference of the invisible light (NIR) sensor unit due to the wavelength difference of the irradiation light of the first light emitting unit 2A and the second light emitting unit 2B, the difference between the two is calculated. In this living body detection, since an RGB sensor unit and another sensor unit are used, acquisition of RGB images and living body detection can be performed simultaneously in parallel.
The result of living body detection is used for a process of extracting a human skin portion from an RGB image (visible light image).
In the imaging device 3 of the present embodiment, the visible light sensor unit and the invisible light sensor unit are arranged side by side in the light incident direction (overlapping in the substrate deep portion direction) in the device. Therefore, if the address of the RGB image and the address of the invisible light image are the same, it is guaranteed that the same point of the subject 100 is captured. Therefore, the accuracy of living body detection is high, and skin images can be extracted accurately.

[Near-infrared light image (bright and dark image)]
Next, a near infrared light image will be described.
The non-visible light (NIR) sensor unit of each pixel detects near infrared light (NIR), and the detection signal from the non-visible light (NIR) sensor unit is a normal near infrared light image (NIR). (Image).
The NIR image is output to the outside via an output unit (pixel circuit or vertical / horizontal transfer register, etc.) for each frame, as in a general CCD or CMOS image sensor, and is required by the signal processing unit 5 in FIG. Is processed.

When there is little near-infrared light, such as at night, the subject 100 is illuminated by one or both of the first and second light emitting units 2A and 2B. This near-infrared light image acquisition uses a different photosensor (sensor unit in the photosensor) than the RGB image acquisition time, so the acquisition of the RGB image and the acquisition of the near-infrared light image are performed simultaneously. Can be done in parallel.
When acquiring distance measurement, living body detection, RGB image, near infrared light image, acquisition of RGB image using RGB sensor part, distance measurement using NIR sensor part, living body detection, near infrared light image acquisition One of the three operations can be performed simultaneously. In order to execute a plurality of the above three operations using the NIR sensor unit, it is necessary to perform this in time series.

<2. Second Embodiment>
A second embodiment of the present invention will be described.
The difference from the first embodiment is the structure of the optical filter 32 and the photosensor 34 of the imaging device 3.
In the first embodiment, as the color filter of the imaging device 3, red, green, and blue primary colors (FIG. 3B) are used. In the second embodiment, NIR wavelengths are transmitted through complementary color filters (FIG. 8A) of cyan (Cy = B + G), magenta (Mg = B + R), yellow (Ye = G + R), and green (G). The one to which the function to be added is added (FIG. 8B) is used.

In the optical filter 32 shown in FIG. 8B, each color filter section has a characteristic of transmitting near-infrared (NIR) wavelengths in addition to transmission of each color. Each filter unit represents cyan (Cy) as B + G + NIR (hereinafter, Cy + NIR). Further, magenta (Mg) is represented as B + R + NIR (hereinafter, Mg + NIR), yellow (Ye) is represented as G + R + NIR (hereinafter, Ye + NIR), and green (G) is represented as G + NIR. Each filter unit has a light transmission characteristic.
The basic color filters of Cy, Mg, Ye, and G are already widely used (FIG. 8A). The method of forming the filter shown in FIG. 8B by adding NIR transmission characteristics to this basic configuration can be realized by a multilayer film structure method similar to the primary color filter described above.

FIG. 9 shows a schematic view of a device cross section.
The photosensor according to the second embodiment is also composed of a layer that detects RGB visible light close to the surface and a layer that detects NIR formed at a deep position.
For Cy, Mg, and Ye, two colors of visible light of RGB are mixed. For this reason, this photosensor requires two n-type layers for detecting two colors of RGB, respectively, and a p-type buffer layer PB for separating the layers is added.
The structure of the second embodiment also utilizes the fact that the depth entering the silicon substrate differs depending on the wavelength of light.

  The photo sensor that receives the light transmitted through the filter unit (Cy + NIR) has a depth that the third n-type layer 39B on the outermost surface side is strongly sensitive to blue light (about 0.2 to 0.5 μm from the surface). Is formed. The second n-type layer is formed at a depth (about 0.8 to 1.5 μm from the surface) that is strongly sensitive to green light. The n-type layer (n-type well 37) of the deepest first sensor part has a depth (30 μm from the surface) that is strongly sensitive to the emission wavelength (near infrared light) of the first light-emitting part 2A (FIG. 1). Or less).

  The B, G, and NIR light transmitted through the filter unit (Cy + NIR) is dispersed according to the depth of the sensor unit, where B is the third n-type layer 39B, G is the second n-type layer, and NIR is the first layer. It is accumulated in one layer of n-type well 37 and separated and output as pixel data of visible light or invisible light.

The photo sensor that receives the light transmitted through the filter unit (Mg + NIR) is formed with a depth (about 0.2 to 0.5 μm from the surface) at which the third n-type layer 39B has high sensitivity to blue light. Yes. The second n-type layer is formed at a depth (about 2.0 to 7.5 μm from the surface) having strong sensitivity to red light. The deepest n-type layer (n-type well 37) has a depth (about 30 μm or less from the surface) that is strongly sensitive to the emission wavelength (near infrared light) of the first light emitting portion 2A (FIG. 1). Is formed.
The light of B, R, and NIR that has passed through the optical filter (Mg + NIR) is dispersed according to the depth of the sensor portion, where B is the third n-type layer 39R, R is the second n-type layer, and NIR is the first Accumulated in one layer n-type well 37 and output as pixel data of visible light or invisible light.

The photo sensor that receives light transmitted through the filter unit (Ye + NIR) is formed with a depth (about 0.8 to 1.5 μm from the surface) at which the third n-type layer 39G is strongly sensitive to green light. Yes. The second n-type layer is formed at a depth (approximately 2.0 to 7.5 μm from the surface) that is highly sensitive to red light. The first n-type layer (n-type well 37) is formed at a depth (about 30 μm or less from the surface) that is strongly sensitive to the emission wavelength (near infrared light) of the first light emitting portion 2A (FIG. 1). ing.
The light of G, R, and NIR transmitted through the filter unit (Ye + NIR) is dispersed according to the depth of the photosensor, where G is the third n-type layer 39G, R is the second n-type layer, and NIR is the first It is accumulated in one layer of n-type well 37 and is separated and output as pixel data of visible light or invisible light.

In the photosensor that receives light transmitted through the filter portion (G + NIR), the sensor portion of the second layer is formed at a depth (about 0.8 to 1.5 μm from the surface) that is sensitive to green light and sensitive. The sensor part of the first layer is formed at a depth (about 30 μm or less from the surface) that is strongly sensitive to the emission wavelength (near infrared light) of the first light emitting part 2A (FIG. 1).
The G and NIR lights that have passed through the filter unit (G + NIR) are separated by the depth of the sensor unit, separated by the output unit formed in the pixel circuit layer 33, and output.
This is the same structure as the G + NIR part (FIG. 6) of the first embodiment.

The depth of each sensor unit is an example, and can be changed as long as the spectral characteristics and the detected light quantity of each light do not cause a problem in use.
Light mixed with two colors of RGB enters the photosensor from the filter portion of each pixel. This incident light is separated into RGB and detected by the sensor unit, and the primary color signals of the same color (for example, R is R) are added to convert them into RGB signals, which are used as normal RGB color images. The
The handling of the detection signal from the NIR sensor unit, that is, whether it is used as a bright / dark image or at least one of the distance measurement and the living body detection, is arbitrary as in the first embodiment. In addition, these processes themselves can use the method described in the first embodiment.

  Compared with the first embodiment, the second embodiment has the advantage that the amount of RGB light is doubled and the disadvantage that the manufacturing process is increased by the three-layer structure.

<3. Third Embodiment>
A third embodiment of the present invention will be described.
FIG. 10 schematically shows a cross-sectional configuration of the pixel array of the imaging device 3.
The difference from the first and second embodiments is the structure of the optical filter 32 and the photosensor 34 of the imaging device 3.
In the third embodiment, the imaging device 3 has no optical filter such as a primary color system or a complementary color system. The received light of all wavelengths enters the photosensor, and R, G, B, and NIR are detected by spectroscopy based on the depth of the sensor portion.

The photosensor according to the third embodiment has the same four-layer sensor unit (pn junction diode that accumulates charges in an n-type layer) in all pixels. Similar to the second embodiment, a p-type buffer layer PB is interposed between n-type layers.
The pn junction of the sensor portion of the fourth layer is formed at a depth (about 0.2 to 0.5 μm from the surface) that is highly sensitive to blue light. The pn junction of the sensor part of the third layer is formed at a depth (about 0.8 to 1.5 μm from the surface) that is strongly sensitive to green light. The sensor layer pn junction of the second layer is formed at a depth (about 2.0 to 7.5 μm from the surface) that is highly sensitive to red light. The pn junction of the sensor part of the first layer is formed at a depth (about 30 μm or less from the surface) that is strongly sensitive to the emission wavelength (near infrared light) of the first light emitting part 2A (FIG. 1).

The pn junction depth of each sensor unit is an example, and can be changed within a range in which the spectral characteristics and the detected light amount of each light do not cause a problem in use.
Light mixed with R, G, B, and NIR is incident on each pixel, and the light is separated by the photosensor 34 and output from each sensor unit. Each RGB pixel data is used as a normal RGB color image.
The handling of the detection signal from the NIR sensor unit, that is, whether it is used as a bright / dark image or at least one of the distance measurement and the living body detection, is arbitrary as in the first embodiment. In addition, these processes themselves can use the method described in the first embodiment.

  Compared to the other embodiments, the third embodiment has the advantage that the amount of RGB light is increased and the optical filter 32 is not necessary, but the four-layer structure is disadvantageous in that the manufacturing process is increased. is there.

<4. Fourth Embodiment>
A fourth embodiment of the present invention will be described.
FIG. 11 schematically shows a cross-sectional configuration of the pixel array of the imaging device 3.
The difference from the first embodiment (FIG. 6) is the structure of the optical filter 32 and the photosensor 34 of the imaging device 3.
As the optical filter 32, an RGB transmission filter that is a color filter of a general imaging device 3, which has a characteristic of transmitting near infrared (NIR) wavelengths at least for G and B, is used.

  In the fourth embodiment, the NIR transmission function of the optical filter 32 is not essential for the R pixel because the NIR is not detected by the photosensor. Therefore, the optical filter 32 of the fourth embodiment has the characteristics that the basic configuration of the RGB filter is such that only R is transmitted (IR is blocked), G + NIR is transmitted, and B + NIR is transmitted.

The fourth embodiment is characterized in that the NIR sensor unit is set to a shallower depth than the first to third embodiments.
In the first to third embodiments, the formation depth of the NIR sensor portion is about 30 μm or less. However, in the fourth embodiment, the depth of the NIR sensor portion is about 10 μm or less so as to facilitate the manufacturing process. And
When the depth of the NIR sensor portion is reduced, if the first embodiment is maintained, the R detection sensor portion and the NIR sensor portion are formed close to or at the same depth. Therefore, it becomes difficult to form the NIR sensor portion and the R sensor portion with a layer structure in the depth direction.

As a solution, an NIR sensor portion is not formed in a pixel having a filter portion that transmits red, and NIR detection is not performed. For pixels that do not perform NIR detection, NIR pixel data is generated by averaging and interpolating data from the NIR detection values of surrounding pixels. Alternatively, four pixels of RGGB are used as one point and the resolution is lowered.
Further, as a disadvantage of reducing the detection depth of NIR, naturally, the sensitivity of light of NIR wavelength is lowered. This can be dealt with by increasing the amount of light projected from the first and second light emitting sections 2A and 2B.

The photosensor 34R that receives light transmitted through the filter portion (R) has a depth (2.0 to 7.5 μm from the surface) where the n-type well 37 constituting the first layer sensor portion is highly sensitive to red light. Degree). R is detected by the first layer sensor.
The photosensor 34G that receives light that has passed through the filter section (G + NIR) and the photosensor 34B that receives light that has passed through the filter section (B + NIR) are formed in the same manner as in FIG.

  The depth of each sensor unit is an example, and can be changed as long as the spectral characteristics and the detected light quantity of each light do not cause a problem in use. Further, application to a CCD or CMOS image sensor, distance measurement, living body detection, and use as a near infrared light image are also the same as in the first embodiment. This point is common to the second and third embodiments.

<5. Fifth Embodiment>
A fifth embodiment of the present invention will be described.
FIG. 12 schematically shows a cross-sectional configuration of the pixel array of the imaging device 3.
The fifth embodiment (FIG. 12) differs from the second embodiment (FIG. 9) in the structure of the photosensor of the imaging device 3.
As in the second embodiment, cyan (Cy) is Cy + NIR (B + G + NIR), magenta (Mg) is Mg + NIR (B + R + NIR), yellow (Ye) is Ye + NIR (G + R + NIR), and green (Y). G) has G + NIR transmission characteristics.

  The structure of the photosensor shown in FIG. 12 is different from that of the second embodiment (FIG. 9) in that each of visible light Cy, Mg, Ye, and G is detected by one n-type layer. It is composed of a layer that detects visible light of Cy, Mg, Ye, and G close to the surface and a layer that detects NIR formed at a deep position, resulting in a total of two charge storage layer (n-type layer) structures. Yes. Even in a general imaging device 3, Cy, Mg, Ye, and G are detected by a single-layer photosensor, which is a proven method.

In the photosensor that receives the light transmitted through the filter portion (Cy + NIR), the sensor portion of the second layer is formed at a depth (about 10 μm or less) that is strong and sensitive to visible light. The sensor part of the first layer is formed at a depth (about 30 μm or less from the surface) that is strongly sensitive to the emission wavelength (near infrared light) of the first light emitting part 2A (FIG. 1).
Cy and NIR light transmitted through the filter unit (Cy + NIR) is dispersed by the depth of the photosensor, and Cy is detected from the second-layer visible light sensor unit and NIR is detected from the first-layer NIR sensor unit, respectively. The

In the photosensor that receives light transmitted through the filter portion (Mg + NIR), the sensor portion of the second layer is formed at a depth (about 10 μm or less) that is highly sensitive to visible light. The sensor part of the first layer is formed at a depth (about 30 μm or less from the surface) that is strongly sensitive to the emission wavelength (near infrared light) of the first light emitting part 2A (FIG. 1).
The light of Mg and NIR transmitted through the filter unit (Mg + NIR) is dispersed by the depth of the photosensor, and Mg is detected from the second layer sensor unit and NIR is detected from the NIR sensor unit of the first layer, respectively.

In the photosensor that receives light that has passed through the filter portion (Ye + NIR), the first-layer sensor portion is formed at a depth (about 10 μm or less) that is strong and sensitive to visible light. The sensor part of the second layer is formed at a depth (about 30 μm or less from the surface) that is highly sensitive to the emission wavelength (near infrared light) of the first light emitting part 2A (FIG. 1).
The light of Ye and NIR transmitted through the filter unit (Ye + NIR) is dispersed by the depth of the photo sensor, and Ye is detected from the visible light sensor unit of the second layer and NIR is detected from the NIR sensor unit of the first layer, respectively. The

In the photosensor that receives light transmitted through the filter portion (G + NIR), the sensor portion of the second layer is formed at a depth (about 10 μm or less) that is highly sensitive to visible light. The sensor part of the first layer is formed at a depth (about 30 μm or less from the surface) that is strongly sensitive to the emission wavelength (near infrared light) of the first light emitting part 2A (FIG. 1).
The G and NIR lights that have passed through the filter unit (G + NIR) are dispersed according to the depth of the photosensor, separated by the output unit formed in the pixel circuit layer 33, and output.

The depth of each sensor unit is an example, and can be changed as long as the spectral characteristics and the detected light quantity of each light do not cause a problem in use.
Cy, Mg, Ye, and G pixel data of each pixel is detected from the output of the sensor unit, and is converted into an RGB signal by calculating them and used as a normal RGB color image. The calculation method is the same as that of an image pickup apparatus using a general complementary color filter.
The handling of the detection signal from the NIR sensor unit, that is, whether it is used as a bright / dark image or at least one of the distance measurement and the living body detection, is arbitrary as in the first embodiment. In addition, these processes themselves can use the method described in the first embodiment.

  Compared with the second embodiment, the advantage is that the photo sensor is made from three layers to two layers and the manufacturing is simple, and visible light is detected by Cy, Mg, Ye, and G. There is a disadvantage in that it requires an operation to do.

<6. Sixth Embodiment>
FIG. 13 schematically shows a cross-sectional configuration of the pixel array of the imaging device 3.
The sixth embodiment (FIG. 13) differs from the first embodiment (FIG. 6) in the configuration of the output unit 40. For example, a common charge storage unit 41C common to three RGB pixels in the color array is added. In the present embodiment, the existing charge storage unit is referred to as a dedicated charge storage unit 41X.
Corresponding to this change in configuration, the switch 43 in FIG. 13 is configured as a switch for switching the intercept to four contact points.
The dedicated charge storage unit 41X is used for near-infrared images, living body detection, and distance measurement, and the common charge storage unit 41C is used for distance measurement. For example, when measuring the distance, if the photosensor receives a sufficient amount of light for distance measurement, the dedicated charge storage unit 41X is used, and if the light amount of only one pixel is insufficient, the common charge storage unit 41C is used. .

FIG. 14 shows a wiring diagram in the case where the charge storage unit is used in common.
In the case of FIG. 14, the pixel size of the distance image is increased to four times the area of the RGB image or NIR image, and the amount of light received can be increased accordingly.
On the other hand, the resolution of the distance image is reduced to ¼ times that of the RGB image or NIR image. The missing portion of the distance data for the RGB image is used by interpolating with the average value of the peripheral distance image. An interpolation method will be described later.

  When performing distance measurement and biological detection to acquire RGB images and near-infrared light images, acquisition of RGB images using RGB photosensors, distance measurement using NIR sensor unit, biological detection, near-infrared light One of the three image operations can be performed simultaneously. In order to perform a plurality of the above three operations using the NIR sensor unit, it is necessary to operate them in time series.

[Data interpolation method]
Next, a data interpolation method for the distance image will be described.
When distance measurement is performed using the common charge storage unit 41C, the resolution of the distance image is lower than that of the RGB image, NIR image, and living body detection image. For the lower resolution of the distance image, the average value of a plurality of areas is calculated and interpolated.

The pixel arrangement shown in FIG. 15 is taken as an example. 16 to 19 are explanatory diagrams of the interpolation method.
Since the distance measurement image is reflected light detection, the amount of data for each pixel is small, and therefore, the distance measurement image is synthesized in 2 × 2 data units here. This synthesis can be easily performed using the configuration shown in FIG.

As shown in FIG. 16, when the area 1 of the RGB image, the NIR image, and the living body detection image and the distance measurement area (NIR_1) match, the distance data of the area 1 becomes the measurement value of NIR_1.
As shown in FIG. 17, when there is no distance measurement area that coincides with the area 2 of the RGB image, NIR image, and living body detection image and the distance measurement area spans two distance measurement areas (NIR_1 and NIR_3), the two distance measurement areas An average value obtained by averaging the outputs is given as distance data. The distance data of area 2 is half of the distance data (NIR_1 + NIR_3).

The case of FIG. 18 is the same as that of FIG. 17, and the distance data of area 3 is half of the distance data (NIR_1 + NIR_2), and this is used as the distance data of area 3.
As shown in FIG. 19, when there is no distance measurement area that coincides with area 2 of the RGB image, NIR image, and living body detection image, and it spans four distance measurement areas (NIR_1 to 4), the four distance measurement areas The average value obtained by averaging the outputs is used as distance data. The distance data of area 4 is 1/4 of the distance data (NIR_1 + NIR_2 + NIR_3 + NIR_4), and this becomes the distance data of area 4.
In the above manner, data interpolation is performed for the missing portion of the distance data.

<7. Seventh Embodiment>
A seventh embodiment of the present invention will be described.
The seventh embodiment differs from the sixth embodiment in the optical filter and photosensor structure of the imaging device. In the sixth embodiment (FIGS. 13 and 14), the color filters 32 of the imaging device 3 are those of primary colors of red, green, and blue.
On the other hand, the seventh embodiment is based on the complementary color filter of FIG. 8A, and also has a characteristic of transmitting near-infrared (NIR) wavelengths as shown in FIG. 8B. Use.
The configuration of the photosensor 34 is the same as that in FIG. 8B, and the configuration of the pixel circuit layer 33 is the same as that in FIG.

FIG. 20 shows a wiring diagram according to the seventh embodiment in the case where the charge storage unit is commonly used for four pixels.
As already described with reference to FIG. 9, the cyan (Cy) filter unit has a transmission characteristic of B + G + NIR, and each pixel data of B, G, NIR is obtained from a pixel having this characteristic as shown in FIG. Output separately. Further, since the magenta (Mg) filter section has a transmission characteristic of B + R + NIR, each pixel data of B, R, NIR is separated and output from a pixel having the characteristic. Similarly, since the yellow (Ye) filter unit has G + R + NIR and the green (G) filter unit has G + NIR transmission characteristics, G, R, NIR pixel data and G, NIR pixel data are obtained from each pixel. Are output separately.
Such light reception for each color and its sensor structure have been described in the description of FIG.

Also in the present embodiment, as in the sixth embodiment, the mixed R, G, and B of each pixel are obtained from a photosensor that separates and outputs each RGB. The signals of the same color (for example, R is R) are added to convert them into RGB signals, which are used as normal RGB color images.
The detection signal from the NIR photosensor is handled in the same manner as in the first and sixth embodiments. Also, the interpolation method can be used as it is if the codes shown in FIG. 15 to FIG. 19 and the codes representing the filter characteristics correspond to the mixed color system, and the method itself is the same. Is omitted.
Compared with the sixth embodiment, the seventh embodiment has an advantage that the amount of RGB light is doubled and a disadvantage that the manufacturing process is increased by the three-layer structure.

<8. Eighth Embodiment>
An eighth embodiment of the present invention will be described.
The difference from the seventh embodiment is the photosensor structure of the imaging device.
As in the seventh embodiment, cyan (Cy) is Cy + NIR (B + G + NIR), magenta (Mg) is Mg + NIR (B + R + NIR), yellow (Ye) is Ye + NIR (G + R + NIR), and green (Y). G) has G + NIR transmission characteristics.
The photo sensor structure is different from the seventh embodiment in that visible light is detected by one layer. It is composed of a layer that detects visible light of Cy, Mg, Ye, and G close to the surface and a layer that detects NIR formed at a deep position, resulting in a total of two charge storage layer (n-type layer) structures. Yes. Since this detection structure has already been described with reference to FIG. 12, illustration is omitted here.
Even in a general imaging device, Cy, Mg, Ye, and G are detected by a single-layer photosensor, which is a proven method.

Cy, Mg, Ye, and G of each pixel are detected from the photosensor, are converted to RGB signals by calculating them, and are used as a normal RGB color image. The calculation method is the same as that of an image pickup apparatus using a general complementary color filter.
The detection signal from the NIR photosensor is handled in the same manner as in the first embodiment. Compared with the seventh embodiment, the advantage is that the photosensor is changed from three layers to two layers and manufacturing is simplified, and visible light is detected by Cy, Mg, Ye, and G. There is a disadvantage in that it requires an operation to do.

<9. Ninth to eleventh embodiments>
The following three embodiments relate to variations of the sixth to eighth embodiments. This modification can also be applied to the other embodiments shown in FIGS. 6, 9, and 12.
FIG. 21 schematically illustrates a cross-sectional configuration of the pixel array of the imaging device 3 according to the ninth embodiment. FIG. 22 shows a wiring diagram in the case where the charge storage unit is commonly used for four pixels. The contents of this modification will be described below mainly in the ninth embodiment with reference to FIGS. 21 and 22.
The ninth embodiment of the present invention differs from the sixth embodiment and the first embodiment related to FIG. 6 in the photosensor structure of the imaging device. As the color filter, a primary color filter is used, and red (R) has transmission characteristics of R + NIR, green (G) has G + NIR, and blue (B) has transmission characteristics of B + NIR, as in the first embodiment.

The difference in the photosensor structure is the structure of the NIR photosensor. In the sixth embodiment, one NIR detection photosensor is arranged for each pixel, and the charge storage unit 41 collects charges from a plurality of NIR photosensors.
On the other hand, in the ninth embodiment, as shown in FIG. 21, in the sixth embodiment, the NIR photosensor itself is integrated with a plurality of pixels, and an NIR photosensor having a large area extending over the plurality of pixels is provided. Is formed. Specifically, the first n-type layer denoted by reference numeral 71 is formed with a large area common to three pixels.
There is no change in the photosensor structure and the configuration of the optical filter (wavelength transmission characteristics) other than this point.

The sensor part of the second layer of each pixel detects red, green, and blue (RGB) as in the previous embodiments, and the detection signal from the sensor part (n-type layer) of the second layer is It is used as a normal color image (RGB image).
The first layer sensor unit having a large area common to the four pixels of RGGB detects NIR, and detection signals from the first layer sensor unit are used for distance measurement, living body detection, and near-infrared light image. Used.

As described above, the NIR photosensor is arranged over a plurality of RGB pixels, and detects NIR light transmitted through the RGB pixels.
A plurality of common charge storage units 41 </ b> C used in common for each pixel are connected to the first layer sensor unit via a switch 43. By controlling the switch 43, it is possible to transfer charges output from the first layer sensor unit (strictly, the n-type well 37 having a large area) to a plurality of charge storage units (FIG. 22). reference).
In the ninth embodiment, compared with the sixth embodiment, the advantage is that the configuration of the output unit 40 including the charge storage unit of the photosensor is simplified, and the resolution of the NIR image and the living body detection is reduced. However, there is a disadvantage that data interpolation is essential.

23 and 24 are schematic cross-sectional views and wiring diagrams corresponding to FIGS. 21 and 22 relating to the tenth embodiment.
In the tenth embodiment, as in FIG. 21, except that the n-type well 37 (the n-type well 37, which is the n-type layer of the first layer sensor portion) is formed in a large area common to four pixels, The photosensor structure is the same as that of the seventh embodiment and FIG. The configuration of the pixel circuit layer 33 is the same as that of the ninth embodiment.
The effects, advantages, and disadvantages are the same as in the ninth embodiment.

25 and 26 are schematic cross-sectional views and wiring diagrams corresponding to FIGS. 21 and 22 relating to the eleventh embodiment.
In the eleventh embodiment, as in FIG. 21, except that the n-type well 37 (the n-type well 37, which is the n-type layer of the first layer sensor portion) is formed in a large area common to four pixels, The photosensor structure is the same as that of the eighth embodiment and FIG. The configuration of the pixel circuit layer 33 is the same as that of the ninth embodiment. The effects, advantages, and disadvantages are the same as in the ninth embodiment.

  As in the above ninth to eleventh embodiments, increasing the area of the NIR sensor portion of the first layer is a basic unit of color arrangement such as RGGB or CyYeMgG (four pixels in this example). It is not necessary to limit to. The NIR light detection unit may be configured by extending the size of the NIR sensor unit to a size multiple of the basic unit of the color array.

  According to the first to eleventh embodiments, the distance image sensor, the living body detection sensor, the RGB image, and the near-infrared light image can be realized by one camera system. In addition, an imaging device for that purpose can be newly provided.

The present invention can also be applied to general RGB image and near-infrared light image video cameras and digital cameras, and Web cameras connected to the Internet. It can also be used in the field of personal computers and game controllers using operations such as motion recognition and gesture recognition.
It can also be used in the field of security discrimination that recognizes human faces and hands. In addition, it can be used in the field of robots and production equipment for obstacle recognition and object recognition.

According to the first to eleventh embodiments, the following effects can be obtained.
In the past, there has been a single imaging device that combines distance measurement and near-infrared light image, living body detection and near-infrared light image.

However, depending on the application (for example, motion recognition, gesture recognition, 3D image recognition of objects, human faces, and hands), distance measurement and RGB image, biological detection and RGB image, biological detection and distance measurement, biological detection and distance measurement And RGB images may be required. In this case, it can be realized only by the two-camera configuration with the RGB camera.
In the case of the two-camera configuration, since the viewpoints are different, positional displacement between the cameras occurs depending on the position of the subject. The frame synchronization of the two cameras is necessary. When zooming is desired, the camera synchronization control is necessary and the complicated configuration. Problems arise.
In the case of the two-camera configuration, since the viewpoints are different, the two images may be misaligned due to the parallax between the two cameras, and frame synchronization control is required between the two cameras. Problems such as the need for synchronous control of the field of view between the cameras and a complicated configuration arise.
In the first to eleventh embodiments of the present invention, the above problem is solved by realizing a single imaging device.

Next, a processing example of the acquired information will be described.
By acquiring the distance measurement result and the RGB image, and selecting the pixel of the RGB image based on the distance information, it is possible to extract only the subject having no viewpoint deviation in pixel units, for example.
For example, it is possible to extract a color image of only a face or a hand by acquiring a living body detection and an RGB image and selecting a pixel of the RGB image based on detection information of the living organism.
By acquiring biological detection and distance measurement and adding distance information to the biological detection image, it is possible to extract three-dimensional information of the face and hand.
It is possible to extract three-dimensional color information of the face and hands by acquiring a living body detection, distance measurement, and an RGB image, selecting a pixel of the RGB image according to the detection information of the organism, and adding the distance information.

Even before the present invention was devised, there was a technology itself that made RGB images and near-infrared light images compatible.
One is a type in which the infrared light cut filter is switched in and out mechanically, and is widely applied in camcorders and the like.
The other is a method that does not require switching of the infrared light cut filter. In this method, an image is acquired with optical filters of R + NIR, G + NIR, B + NIR, and NIR. The NIR component is deleted from the detection results of R + NIR, G + NIR, and B + NIR to obtain RGB signals, and at the same time, NIR signals are obtained from the pixels of the NIR filter.

Of these, the latter has the disadvantage of reducing the resolution by removing one G from the Bayer array of color filters.
By applying the present invention, it is not necessary to mechanically switch the infrared light cut filter, and the RGB image and the near-infrared light image can be compatible without reducing the resolution.
In addition, the application of the present invention reduces the size of a pixel with an area of ¼ without increasing the amount of emitted light for the shortage of light in distance measurement, which is a problem in pixel downsizing of an imaging device including a distance image sensor. It is possible to provide a method, an imaging device, and an imaging apparatus corresponding to

  DESCRIPTION OF SYMBOLS 1 ... Imaging device, 2 ... Light emission part, 2A, 2B ... 1st and 2nd light emission part, 3 ... Imaging device, 4 ... Lens, 5 ... Signal processing part, 6 ... Control part, 32 ... Optical filter, 33 ... Pixel circuit layer, 34 ... photo sensor, 35 ... semiconductor substrate, 36, 37 ... n-type layer (n-type well, etc.), 40 ... output unit, 41 ... charge storage unit, 41C ... common charge storage unit, 41X ... dedicated charge Accumulator, 42 ... output circuit, 43 ... switch.

Claims (21)

A pixel array in which an optical filter is disposed on the side on which image light is incident;
The pixels of the pixel array are
A first filter region that transmits visible light having a wavelength corresponding to at least one of a plurality of primary colors or complementary colors, and invisible light having a wavelength longer than visible light;
A visible light sensor unit having a main sensitivity to the visible light from the first filter region;
A non-visible light sensor unit which is formed at a position farther from the side where the image light enters than the visible light sensor unit, and has a main sensitivity to the non-visible light from the first filter region;
Have
An output unit that separates and outputs a signal of a visible light image generated by photoelectric conversion of the visible light sensor unit and a signal of a non-visible light image generated by photoelectric conversion of the invisible light sensor unit is provided in the pixel array. An imaging device provided for each pixel or shared by a plurality of pixels commonly connected to an output signal line. The output unit is
A plurality of charge storage units;
A switch that distributes the generated charges of the invisible light sensor unit to the plurality of charge storage units;
The imaging device according to claim 1, comprising: N × M charge storage units are provided in each output unit (N and M are each an integer of 2 or more),
A control circuit that controls the switch so as to collectively output N rows and M columns of received charges in the pixel arrangement of the non-visible light image as a detection unit, or input a control signal from the outside The imaging device according to claim 2, further comprising a terminal for applying to the switch. The imaging device according to claim 1, wherein the invisible light sensor unit has a size of N rows and M columns (N and M are each an integer of 2 or more) in the pixel array of the visible light sensor unit. The output unit is
A plurality of charge storage units;
A switch that distributes the generated charges of the invisible light sensor unit to the plurality of charge storage units;
Have
Control for controlling the switch so as to further output a plurality of received charges of the non-visible light sensor unit having a size corresponding to N rows and M columns of the pixel arrangement of the non-visible light image as a detection unit. The imaging device according to claim 4, further comprising a circuit or a terminal for inputting the control signal from the outside and applying the signal to the switch. The plurality of charge storage units include
Two or more common charge storage units that are connected to a plurality of the invisible light sensor units by the switch and receive the received light charges from all of the plurality of invisible light sensor units via the switch,
Connection to any one of the plurality of non-visible light sensor units is controlled by the other switch, and two or more dedicated charge accumulations are supplied with received light charges from the corresponding non-visible light sensor units via the corresponding switches. And
The imaging device according to claim 3 or 5, comprising: The first filter region is configured to transmit visible light of a plurality of colors;
The visible light sensor unit that photoelectrically converts the visible light of the plurality of colors includes a plurality of sensor units having main sensitivities in different wavelength regions, from a side where the image light enters toward a sensor unit corresponding to a wavelength region having a long wavelength. The imaging device according to claim 1, wherein the imaging device is arranged at a distant position. The first filter region is configured to transmit visible light of primary colors or complementary colors of three colors or four colors;
The visible light sensor unit that photoelectrically converts the three or four colors of visible light includes three or four sensor units having main sensitivities in different wavelength regions, and the sensor unit corresponding to a wavelength region having a longer wavelength is the image. The imaging device according to claim 7, wherein the imaging device is arranged at a position far from a light entering side. The said visible light sensor part which photoelectrically converts the said multiple colors visible light is a single sensor part which has a sensitivity in the visible light range corresponding to the said multiple colors visible light. Imaging device. The optical filter transmits a visible light having a wavelength corresponding to a color having a longest wavelength among a plurality of primary colors or a color having a longest wavelength among a plurality of colors of a complementary color, and blocks the invisible light. Has a filter area,
The imaging device according to claim 1, wherein a pixel including the second filter region of the pixel array includes a visible light sensor unit having main sensitivity to the visible light from the second filter region. An imaging device including a pixel array in which an optical filter is disposed on a side on which image light is incident;
The imaging device emits and projects at least one invisible light having a plurality of wavelength regions that are single or different from each other on the longer wavelength side than the wavelength region of visible light, and a part of the projection light reflected by the subject is the imaging device. A light emitting part that is directed to reach
A lens for imaging the return light on the imaging device;
A control unit for controlling the light emission timing of the light emitting unit;
A signal processing unit that processes a signal of a visible light image output from the imaging device;
Have
The pixels of the pixel array are
A first filter region that transmits visible light having a wavelength corresponding to at least one of a plurality of primary colors or complementary colors, and invisible light having a wavelength longer than visible light;
A visible light sensor unit having a main sensitivity to the visible light from the first filter region;
A non-visible light sensor unit which is formed at a position farther from the side where the image light enters than the visible light sensor unit, and has a main sensitivity to the non-visible light from the first filter region;
Have
An output unit that separates and outputs a signal of a visible light image generated by photoelectric conversion of the visible light sensor unit and a signal of a non-visible light image generated by photoelectric conversion of the invisible light sensor unit is provided in the pixel array. Provided for each pixel or shared by a plurality of pixels commonly connected to the output signal line,
The signal processing unit performs signal processing on the signal of the visible light image,
The signal processing unit and the control unit control light reception time and near infrared light emission of the imaging device, and calculate a distance to the subject based on the obtained signal of the invisible light image An imaging apparatus that executes at least one of a process and a biometric detection process for detecting an image of the skin of the subject. The light emitting unit is configured to be able to independently control the timing of light emission and stop of the one invisible light,
The output unit is
A plurality of charge storage units;
A switch that distributes the generated charges of the invisible light sensor unit to the plurality of charge storage units;
An output circuit that outputs a plurality of non-visible light images corresponding to the respective accumulated charge amounts from the plurality of charge storage units;
Have
The signal processing unit and the control unit control a switching timing of the switch with respect to a light reception time of the imaging device, a timing of light emission and stop of the invisible light, and the plurality of invisibles obtained by the control The imaging device according to claim 11, wherein a calculation process for measuring a distance to the subject can be executed based on a light image. The light emitting unit is configured to be capable of independently controlling the timing of light emission and stop of the first and second invisible light having different wavelength regions,
The output unit is
A plurality of charge storage units;
A switch that distributes the generated charges of the invisible light sensor unit to the plurality of charge storage units;
An output circuit that outputs a plurality of non-visible light images corresponding to the respective accumulated charge amounts from the plurality of charge storage units;
Have
The signal processing unit and the control unit control the light reception time of the imaging device, the switch switching timing, and the light emission and stop timings of the plurality of near-infrared light, and the plurality of non-lights obtained by the control are controlled. The imaging device according to claim 11 or 12, wherein a living body detection process for detecting an image of the skin of the subject can be executed based on a visible light image. The signal processing unit and the control unit switch the switch from a first charge accumulation unit to a second charge accumulation unit within a light receiving period of the return light of the imaging device, and the first and second charge storage units are switched by the switching. The imaging apparatus according to claim 12, wherein a distance to the subject is obtained based on a ratio of received light amounts of the return light distributed to the charge storage unit. The signal processing unit and the control unit set a distance measurement range in the image of the subject in the processing of the visible light image, and the pixel address of the imaging device corresponds to the set distance measurement range. The imaging device according to claim 11, wherein the arithmetic processing for measuring the distance is performed using an image portion of a visible light image. When a non-visible light image corresponding to the distance measurement range of the visible light image is not obtained, a plurality of other non-visible light images adjacent to the non-visible light image portion corresponding to the distance measurement range by a pixel address in the non-visible light image. The data of the non-visible light image part which acquires data from a non-visible light image part, interpolates the data of the acquired several non-visible light image parts, and produces | generates the data of the image part of the non-visible light image used for the said calculation process. Imaging device. The first and second invisible light emitted by the light emitting unit is invisible light in different wavelength regions, one of which is mainly absorbed by the skin of the living body and the other is mainly reflected by the skin of the living body,
The signal processing unit and the control unit are configured to switch the first and second living body detection images obtained from the first and second invisible return lights by switching the switches. A level difference between the first living body detection image obtained from the first charge storage unit and the second living body detection image obtained from the second charge storage unit is separated for each pixel. A range of the invisible light image in which the obtained level difference is equal to or greater than a preset threshold is recognized as a human skin range of the subject, and the visible light image range corresponding to the human skin range and a pixel address is defined as a living body. The imaging device according to claim 13, wherein the imaging device is extracted as an image. In the imaging device,
The first filter region is configured to transmit visible light of a plurality of colors;
The visible light sensor unit that photoelectrically converts the visible light of the plurality of colors includes a plurality of sensor units having main sensitivities in different wavelength regions, from a side where the image light enters toward a sensor unit corresponding to a wavelength region having a long wavelength. The imaging device according to claim 11, wherein the imaging device is arranged at a distant position. In the imaging device,
The first filter region is configured to transmit visible light of primary colors or complementary colors of three colors or four colors;
The visible light sensor unit that photoelectrically converts the three or four colors of visible light includes three or four sensor units having main sensitivities in different wavelength regions, and the sensor unit corresponding to a wavelength region having a longer wavelength is the image. The imaging device according to claim 18, wherein the imaging device is arranged at a position far from the light entering side. In the imaging device,
The visible light sensor unit that photoelectrically converts the visible light of the plurality of colors is a single sensor unit having sensitivity in a wide visible light range corresponding to the visible light of the plurality of colors. The imaging device described. The optical filter transmits a visible light having a wavelength corresponding to a color having a longest wavelength among a plurality of primary colors or a color having a longest wavelength among a plurality of colors of a complementary color, and blocks the invisible light. Has a filter area,
The imaging device according to claim 11, wherein a pixel including the second filter region of the pixel array includes a visible light sensor unit having main sensitivity to the visible light from the second filter region.

Images