JP5464008B2 - Image input device - Google Patents

Description

  The present invention relates to an image input device, and more particularly to an image input device that receives visible light to infrared light and forms an image.

  For example, a surveillance camera for monitoring at night is known that forms infrared images and receives reflected light from a subject to form a monochrome image. On the other hand, as a daytime monitoring camera, a camera that receives natural light reflected from a subject and forms a color image is known. However, the conventional surveillance camera has a problem in that the cost increases because different imaging devices are used for night photography and daytime photography.

  On the other hand, Patent Document 1 has an IR filter in addition to the RGB filter, and transmits the RGB filter in one shooting using an image sensor that receives a broadband light from visible light to infrared light. A color image reproducing device is disclosed that can acquire color information using the light that has been obtained and simultaneously acquire luminance information using the light that has passed through the IR filter.

JP 2007-184805 A

  According to the technique of Patent Document 1, it is possible to perform night shooting and daytime shooting using one image sensor, and it is also possible to superimpose a monochrome image and a color image. However, for example, a halogen lamp may be an object, but as shown in FIG. 1, the halogen light projected from the halogen lamp has an extremely large amount of infrared light component compared to the visible light component. When photographing is performed, overexposure may occur in the pixel that receives the projection light from the halogen lamp. If the entire exposure amount is reduced to avoid this, the signal amount of the visible light component (RGB) is remarkably reduced, and as a result, the S / N ratio of the chromaticity signal is greatly reduced and the noise is rough. There is a risk of forming a low-quality image. Further, the filter disclosed in Patent Document 1 has spectral characteristics such that only R + IR, only G + IR, only B + IR, and only IR are transmitted. However, forming a filter having such characteristics is not possible at present. It can be said that it is quite difficult.

  The present invention has been made in view of the above-described problems, and an image that can obtain an appropriate image according to shooting conditions using an imaging element that receives broadband light from visible light to infrared light. An object is to provide an input device.

The image input device of the first aspect of the present invention receives at least three types of optical filters that transmit light beams in different wavelength bands in a wavelength region including visible light and infrared light, and light beams that have passed through the optical filter. An image sensor including a plurality of pixels that output a signal corresponding to the amount of received light; and an arithmetic unit that performs an arithmetic process on the signal from the image sensor. An image input device for inputting,
The sensitivity of receiving a light beam in a wavelength band including infrared light is changed in a specific pixel among the pixels in accordance with imaging conditions.

An image input device according to a second aspect of the present invention receives at least three types of optical filters that transmit light beams in different wavelength bands in a wavelength region including visible light and infrared light, and light beams that have passed through the optical filter. An image sensor including a plurality of pixels that output a signal corresponding to the amount of received light; and an arithmetic unit that performs an arithmetic process on the signal from the image sensor. An image input device for inputting,
The infrared light sensitivity in the first pixel group is different from the infrared light sensitivity in the second pixel group.

  According to the image input device of the first aspect of the present invention, the sensitivity of receiving a light flux in a wavelength band including infrared light is changed in a specific pixel among the pixels according to shooting conditions. In general, since the incidence of visible light is large, the sensitivity of receiving a light flux in a wavelength band including infrared light is reduced for all pixels as a specific pixel, thereby increasing the S / N ratio. A high-quality color image is formed. On the other hand, in the case of night photography, since infrared light is generally incident, light beams in a wavelength band including infrared light are received for all pixels as specific pixels. Sensitivity is increased, so that the subject can be clearly photographed even in a dark scene. Furthermore, for example, a subject that partially includes a halogen lamp or the like may be shot at night shooting. In such a case, a pixel that receives halogen light having a large infrared light component is set as a specific pixel, and only the specific pixel is irradiated with infrared light. By reducing the sensitivity to receive the light flux in the wavelength band that contains, it is possible to receive an appropriate amount of visible light, and to receive an appropriate amount of infrared light for other pixels, thereby using a single image sensor Even in such a case, appropriate chromaticity information and luminance information can be obtained regardless of the shooting conditions.

  The “optical filter that transmits light beams having different wavelength bands” refers to an optical filter in which the wavelength bands of light to be transmitted do not completely match, and includes those in which the wavelength bands of transmitted light are partially different. Examples of such an optical filter include a W filter that transmits the entire visible light region and the infrared light region, a Ye filter that transmits the green and red component regions and the infrared light region of the visible light region, and the visible light region. Among them, there is a combination of an Ri filter that transmits the red component region and the infrared light region, and an IR filter that transmits only the infrared light region. In addition, for example, an IR filter that transmits only the infrared light region may be used in combination with an RGB filter used in a normal imaging device. Furthermore, “change the sensitivity of receiving a light beam in a wavelength band including infrared light” means that, for example, when infrared light is incident on a pixel and converted into a charge, the amount of discharge (amount discarded) of the charge Is changed according to the depth from the light receiving surface. Specifically, if the electric charge is discharged from a shallower portion, the sensitivity is lowered.

  According to an aspect of the present invention, the evaluation value is calculated by comparing the amount of infrared light projected from the subject and the amount of visible light, and the sensitivity of the specific pixel is changed based on the evaluation value. Can do. For example, when using the above-described W, Ye, Ri, and IR filters, the amount of visible light + infrared light output from a pixel that receives light that has passed through the W filter is A, and light that has passed through the IR filter is received. When the amount of infrared light output from a pixel is B, if B / A is an evaluation value and the evaluation value is greater than 0.5, a pixel that receives light that has passed through an IR filter (specification) If the evaluation value is smaller than 0.5, the sensitivity of the specific pixel can be made higher than the sensitivity of the other image.

  According to one aspect of the present invention, the sensitivity of the specific pixel can be changed based on the brightness level of the subject. For example, when the illuminance of the subject is high, the sensitivity of all the pixels is reduced as the specific pixel, assuming that the shooting is during the daytime. The sensitivity of the pixels can be increased. Further, when the brightness level of a part of the subject is high, the sensitivity is reduced by using a pixel that receives a light beam from a subject with a high brightness level as the specific pixel, and an appropriate amount of visible light is received. In pixels other than the specific pixel, an appropriate amount of infrared light can be received with the sensitivity higher than that of the specific pixel. The degree of brightness of the subject can be estimated in the process of performing exposure control and the like, and an evaluation value can be calculated accordingly.

  According to one aspect of the present invention, it is possible to calculate the ratio between the amount of natural light and the amount of artificial light included in the light projected from the subject, and change the sensitivity of the specific pixel based on the ratio. The evaluation value can be used as a value representing the ratio between the amount of natural light and the amount of artificial light, but is not limited thereto.

  According to one aspect of the present invention, the sensitivity of the specific pixel can be changed according to the type of the optical filter. For example, it is preferable to change the sensitivity of a pixel provided with an IR filter that transmits only infrared light.

  According to an aspect of the present invention, the specific pixel can be arranged in a partial region on the light receiving surface of the imaging element. For example, in the case of a fixed point monitoring camera, there may be a case where a scene where the halogen lamp is always at a specific position on the shooting screen may be monitored. In such a case, only the sensitivity of the pixels to which the halogen light is incident is reduced. If so, you can shoot high-quality color images at night.

  According to one aspect of the present invention, the position of the specific pixel on the light receiving surface of the image sensor can be changed in accordance with shooting conditions. For example, in the case of a surveillance camera capable of panning and tilting, a halogen lamp moves on the shooting screen according to the movement of the camera. By reducing only the pixel sensitivity, a high-quality color image can be taken regardless of the movement of the camera.

  According to one aspect of the present invention, a moving image including a plurality of frames can be formed by continuously capturing an image of a subject while changing the sensitivity of the specific pixel. The change in sensitivity may be continuous or stepwise, and the entire period may be changed periodically for one or more frames.

  According to an aspect of the present invention, a single image is continuously formed from a plurality of images captured under different conditions by continuously capturing a subject while changing the sensitivity of the specific pixel. Video can be formed.

  According to one aspect of the present invention, it is possible to vary the sensitivity for each specific pixel. In this case, the specific pixels may be all pixels, the sensitivity may be changed for each specific pixel, or the sensitivity may be changed for each group of pixels.

  According to one embodiment of the present invention, the sensitivity can be varied for each pixel.

  According to an aspect of the present invention, the imaging device captures charges generated by receiving light having a wavelength less than the boundary wavelength, converts the charges into a signal, and outputs the signal, and receives light having the wavelength longer than the boundary wavelength. Thus, a pixel having discharge means for discharging the electric charge generated by doing so can be included. As the boundary wavelength, there is a wavelength of around 700 nm that separates infrared light and visible light.

  According to an aspect of the present invention, the discharge unit is provided at a position deeper than the light incident surface than the capturing unit.

  According to one aspect of the present invention, the discharge amount of the charge discharged by the discharging means is variable according to a bias voltage control signal input from the outside.

  According to the second aspect of the present invention, since the sensitivity of the infrared light in the first pixel group is different from the sensitivity of the infrared light in the second pixel group, for example, a halogen lamp is used with a fixed surveillance camera or the like. When photographing a subject including the above, the position of a halogen lamp or the like is fixed on the photographing screen. Therefore, the pixels that receive halogen light having a large infrared light component are set as the first pixel group, and the other pixels are set. As the second pixel group, by reducing the sensitivity of only the first pixel group, it is possible to receive visible light with a sufficient exposure amount for the other pixels while suppressing overexposure, thereby allowing a single pixel. Even when this image sensor is used, appropriate chromaticity information and luminance information can be obtained regardless of the photographing conditions.

  According to an aspect of the present invention, the sensitivity of the infrared light in the first pixel group is lower than the sensitivity of the infrared light in the second pixel group, and the first pixel group is surrounded by the first pixel group. Two pixel groups can be arranged. For example, in a surveillance camera, a light may be lit on a subject at night. In such a case, visible light is reflected from the center of the subject, but the surroundings often remain dark. Therefore, by reducing the sensitivity of the first pixel group that receives visible light from the center of the subject and increasing the sensitivity of the second pixel group that receives light from the surroundings, a bright subject can be a dark subject. Can be monitored simultaneously.

  According to one aspect of the present invention, in the case of continuously capturing an image of a subject, a signal from the first pixel group and a signal from the second pixel group are alternately output, and thereby mainly visible. A moving image composed of a plurality of frames alternately including images based on light and images mainly based on infrared light can be formed.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 2 is a schematic diagram of the image input device MG according to the present embodiment. Such an image input device MG can be used for a surveillance camera, but its application is not limited thereto. The image input device MG includes a lens 101, an image sensor 102, an A / D converter 103, an image processor 104, a controller 105, a voltage controller 106, and a voltage generator 107. The lens 101 includes a diaphragm and a shutter (not shown) controlled by the control unit 105.

  The image pickup element 102, which will be described in detail later, includes a light receiving unit composed of a PD (photodiode), an output circuit that outputs a signal photoelectrically converted by the light receiving unit, and a drive circuit that drives the image pickup element 102. Original image data having different levels is generated. Here, as the image sensor 102, various image sensors such as a CMOS image sensor, a VMlS image sensor, and a CCD image sensor can be employed.

  In the present embodiment, the image sensor 102 captures a visible color image component with a pixel including a color filter, captures an infrared image component with a pixel including an infrared filter, and includes a transparent filter or no filter. The pixel has a function of capturing a luminance image component including a visible luminance image component and an infrared image component.

  The image processing unit 104 serving as a calculation means includes a calculation circuit and a memory used as a work area of the calculation circuit, and is a digital signal obtained by A / D converting the analog signal output from the image sensor 102 by the A / D conversion unit 103. , And after executing image processing to be described later, for example, it has a function of outputting to an unillustrated memory or display device.

  The control unit 105 includes a CPU and a memory that stores a program executed by the CPU, and has a function of controlling the entire image input device MG in response to an external control signal.

  The operation of the image input device MG will be described. Subject light is received by the light receiving surface of the image sensor 102 via the lens 101. Subject light is photoelectrically converted by the pixels of the image sensor 102, converted into an analog signal corresponding to the amount of incident light, input to the A / D converter 103, and converted into a digital signal. The digital signal of the A / D conversion unit 103 is input to the image processing unit 104, output as an image signal (RGB, YCC, etc.), displayed on a monitor (not shown), or stored in a memory.

  On the other hand, the image processing unit 104 creates exposure data and light source data for controlling the potential of the imaging element based on the brightness information (brightness level) and wavelength information of the subject, and inputs them to the control unit 105. . Based on the exposure data and the light source data, the control unit 105 drives and controls the voltage control unit 106 to generate a predetermined bias voltage in the voltage generation unit 107 and supply it to the image sensor.

  FIG. 3 is a diagram schematically showing a cross-sectional structure of one pixel of the image sensor 102 according to the present embodiment. In this pixel structure, on the surface of the p-type substrate 2 that is the first conductor, an n-type that is the second conductor serving as a photoelectric conversion portion is provided in a region partitioned by one element isolation region 3. 2 shows an example of a CMOS image sensor in which the photodiode 4 is formed. In the present embodiment, an n-type carrier discharge region 5, which is a second conductor, is embedded in the substrate deep portion of each photodiode 4. The carrier discharge region 5 is vertically connected to the contact portion 6 on the surface of the substrate 2 by an n-type region 7 in order to apply a bias voltage B. In FIG. 3, the boundary between the photodiode 4 and the transfer gate 8 coincides with the end of the carrier discharge region 5, but this is not restrictive. As long as the carrier can be completely transferred to the floating diffusion (hereinafter abbreviated as FD) 9 by the transfer gate 8, the carrier discharge region 5 may be extended in the direction of the transfer gate 8 as compared with FIG.

  FIG. 4 is a diagram showing an overall electrical configuration of the image sensor 102, and FIG. 5 is a diagram showing a circuit configuration for one pixel. FIG. 6 is a schematic diagram illustrating a part of the filter F included in the image sensor 102. FIG. 7 is a diagram illustrating the spectral characteristics of the filter F including the spectral sensitivity characteristics of the image sensor. Any structure having the same functions as those shown in FIGS. 3 to 7 is not limited to the structure shown in the figure.

  As shown in FIG. 6, the pixel filter repeatedly arranged in a matrix for each pixel is composed of four types of filters (also referred to as optical filters) W, Ye, Ri, and IR. More specifically, the filter Ri is arranged in the first row and the first column, the filter IR is arranged in the second row and the first column, the filter W is arranged in the first row and the second column, and the second row second The filter R, the filter IR, the filter W, and the filter Ye are arranged in a zigzag manner so that the filter Ye is arranged in a row. However, this is an example, and the filter R, the filter IR, the filter W, and the filter Ye may be arranged in a zigzag pattern in other patterns. Note that the filter W may be transparent, and in this case, the filter F includes three types of optical filters.

  The filter W transmits all visible light and infrared light, the filter Ye transmits green light to infrared light, the filter Ri transmits red light and infrared light, and the filter IR transmits only infrared light. To Penetrate. Each of these filters corresponds to one pixel. FIG. 7 shows an example of the spectral characteristics of the filter (W, Ye, Ri, IR). For example, the filter Ye has a cutoff frequency of around 500 nm, the filter Ri has a cutoff frequency of around 600 nm, The filter IR has a cutoff frequency around 700 nm. The filter W does not have a cutoff frequency. Such a filter F is usually inserted and disposed between the lens of the imaging device and the light receiving element. Here, the visible light wavelength region is 400 nm to 700 nm, and the infrared light wavelength region is 700 nm to 1100 nm.

  Instead of the filter configured as described above, Ye, M (magenta) + IR, C (cyan) + IR (where M + IR blocks only green light and C + IR blocks only red light) can be realized. . However, the filter (Ye, Ri, IR) can have a steep spectral transmission characteristic, and has a better spectral transmission characteristic than, for example, an M + IR filter or a C + IR filter. That is, each of the M + IR filter and the C + IR filter has a characteristic of shielding only the green light region and the red light region, which are partial regions in the center, of the visible light wavelength + infrared light wavelength band. It is difficult for such a filter to have a steep spectral transmission characteristic such as a filter (Ye, Ri, IR). Therefore, each of the M + IR filter and the C + IR filter cannot extract the RGB image components with high accuracy even if the calculation is performed. Therefore, by configuring the image sensor 102 using the filters (W, Ye, Ri, IR), the performance of the image sensor 102 can be improved.

  FIG. 8 is a block diagram illustrating a detailed configuration of the image processing unit 104. The image processing unit 104 includes an exposure correction unit 104a, a color interpolation unit 104b, a color signal generation unit 104c, a color space conversion unit 104d, and a light source detection unit 104e.

  The exposure correction unit 104 a receives a signal from the image sensor 102, generates exposure data, and outputs it to the control unit 105. Based on the exposure data, the control unit 105 (FIG. 2) adjusts the exposure time of the image sensor so that the exposure is appropriate.

  The color interpolation unit 104b receives an image component Ye that is an output signal of a pixel that has received a light beam that has passed through the filter Ye, an image component Ri that is an output signal of a pixel that has received the light beam that has passed through a filter Ri, and a light beam that has passed through a filter IR. Interpolation processing for interpolating the missing pixel data is performed on each of the image component IR that is an output signal of the pixel that has received the light and the image component W that is the output signal of the pixel that has received the light beam that has passed through the filter W, and the image component Each of Ri, image component IR, image component W, and image component Ye is converted into image data having the same number of pixels as the number of pixels of the image sensor 102. Note that missing pixel data is generated in the image components Ye, Ri, IR, and W because the filters (W, Ye, Ri, and IR) are arranged in a staggered pattern. Also. As the interpolation process, for example, a linear interpolation process may be employed.

  The light source detection unit 104e receives the image component Ye, the image component Ri, the image component IR, and the image component W that have been subjected to the interpolation processing by the color interpolation unit 104b, and compares them to obtain an evaluation value. Is output to the control unit 105 as light source data. As an example, for example, when the value of the image component W is A and the value of the image component IR is B, B / A can be the evaluation value. The control unit 105 controls the potential of the image sensor by changing a bias voltage value to be described later based on the evaluation value.

The color signal generation unit 104c synthesizes the image component Ye, the image component Ri, the image component IR, and the image component W that have been subjected to the interpolation processing by the color interpolation unit 104b, using the equation (1), and the color signal dR , DG, dB (RGB color signal) are generated.
dR = Ri-IR
dG = Ye-Ri (1)
dB = W-Ye

The color space conversion unit 104d converts the color signals dR, dG, and dB into a luminance signal Y (an example of a second intensity signal) and color difference signals Cb and Cr (an example of a chromaticity signal), as shown in Expression (2). The color difference signal Cb indicates a blue color difference signal, and the color difference signal Cr indicates a red color difference signal.
Y = 0.3dR + 0.59dG + 0.11dB
Cb = dB-Y (2)
Cr = dR−Y

Further, the color space conversion unit 104d converts the luminance signal Yadd (an example of the first intensity signal) obtained by adding the image components Ye, Ri, IR, and W, as shown in Expression (3), into the conversion target. Is calculated as a luminance signal of the color space.
Yadd = (1/4) x (Ri + IR + W + Ye) (3)

  Here, the luminance signal Yadd is calculated by the addition process. The noise component can be reduced as compared with the case where the luminance signal Y is calculated by the subtraction process.

  Further, in FIG. 8, the color space conversion unit 104d performs the smoothing process on the color difference signals Cb and Cr obtained by Expression (2) to calculate the color difference signals Cbs and Crs. Here, as the smoothing process, for example, a cascade filter process, which is a filter process that multi-resolutions the color difference signals Cb and Cr by repeatedly using a relatively small size low-pass filter such as 5 × 5, is adopted. Also good. Further, a filter process using a low-pass filter of a predetermined size having a relatively large size may be employed.

  Also, an edge-preserving filter that smoothes areas other than edges without blurring the subject that emits light (smoothing when the signal level difference between pixels is smaller than a certain reference value, and smoothing the part larger than the reference value Filter) processing that does not enable conversion may be employed. Note that it can be estimated that light emission is detected by comparing an infrared component and a visible light component.

  As described above, by performing the smoothing process on the color difference signals Cb and Cr, the noise components included in the color difference signals Cb and Cr are blurred, and the S / N ratio of the color difference signals Cb and Cr can be improved.

Further, as shown in the equation (4), the color space conversion unit 104d has a ratio RT1 of the luminance signal Yadd obtained by the equation (3) to the luminance signal Y obtained by the equation (2) (first ratio: RT1 = The color difference signals Crs and Cbs are corrected according to Yadd / Y) to obtain the color difference signals Crm and Cbm.
Crm = Crs × Yadd / Y
Cbm = Cbs × Yadd / Y (4)

  In this way, by correcting the color difference signals Crs and Cbs using the ratio RT1, the color difference signal and the luminance signal in the color space to be converted can be calculated with good balance. Without this processing, when the color signals dR ′, dG ′, and dB ′ are calculated, if the luminance signal Yadd is larger than the luminance signal Y, the vividness will be lost, and the luminance signal Yadd will be greater than the luminance signal Y. If it is small, it causes the problem of excessive vividness.

  Next, the operation of the image input device MG will be described. FIG. 9 is a flowchart illustrating the operation of the image input device MG according to the first embodiment. First, the control unit 105 causes the image sensor 102 to capture one frame of original image data. By this. Image components Ye, Ri, IR, and W are obtained (step S1).

  Here, the image sensor 102 captures the image component Ye with a pixel that receives the light beam that has passed through the filter Ye, images the image component Ri with a pixel that receives the light beam that has passed through the filter Ri, and the light beam that has passed through the filter IR. The image component IR is picked up by the pixels that receive the light, and the image component W is picked up by the pixels that receive the light flux that has passed through the filter W. When the image input device MG captures a moving image, the control unit 105 may cause the image sensor 102 to capture the original image data at a frame rate such as 30 fps or 60 fps. When the image input device MG captures a still image, the control unit 105 may cause the image sensor 102 to capture the original image data when the user presses the release button.

  Next, color interpolation processing is performed on the image components Ye, Ri, IR, and W by the color interpolation unit 104 b in the image processing unit 104. Next, the color signal generation unit 104c calculates the color signals dR, dG, and dB by calculating dR = Ri-IR, dG = Ye-Ri, and dB = W-Ye shown in Expression (1) (Step S2). ).

  Next, the color signal generation unit 104c calculates Y = 0.3dR + 0.59dG + 0.11dB, Cb = dB−Y, and Cr = dR−Y shown in Expression (2), thereby obtaining the luminance signal Y, the color difference signal Cr, Cb is calculated (step S3).

  Next, the color space conversion unit 104d performs a smoothing process on the color difference signals Cr and Cb to calculate the color difference signals Crs and Cbs (step S4).

  Next, the color space conversion unit 104d calculates Yadd = (1/4) × (Ri + IR + W + Ye) shown in Expression (1) to calculate the luminance signal Yadd (step S5).

In the present embodiment, the luminance signal Yadd is calculated by adding the image components Ri, IR, W, and Ye as shown in Expression (1). However, the present invention is not limited to this. For example, Expression (1) The luminance signal Yadd may be calculated by weighted addition as indicated by '.
Yadd = a · Ri + β · IR + γ · W + s · Ye (1) ′

  Here, a, β, γ, and s are weighting coefficients, and a + β + γ + s = 1. As a, β, γ, and s, for example, predetermined values may be adopted.

Next, the color space conversion unit 104d calculates the color difference signals Crm and Cbm by performing the calculation shown in Expression (4) (step S6). In FIG. 9, the color difference signals Crm and Cbm are collectively expressed as Cm, and the color difference signals CrS and Cbs are collectively expressed as Cs.

  Note that the RGB color signal generation unit (not shown) inversely converts Equation (2) to calculate the color signals dR ′, dG ′, and dB ′ from the luminance signal Yadd and the color difference signals Crm and Cbm (Step S7). .

  As described above, according to the image input device MG, the luminance signal Yadd is calculated using the equation (1). Therefore, the luminance signal Yadd having a high S / N ratio can be calculated even at night. Further, since the color difference signals Cr and Cb are smoothed, the color difference signals Crs and Cbs having a high S / N ratio can be calculated even at night.

  Further, the configuration of the image sensor 102 will be described in detail with reference to FIG. The charge obtained by the photodiode 4 is transferred to the FD 9 by applying a high-level transfer pulse φTX to the transfer gate 8 and is transferred to the FD 9, and is converted into a voltage value by the capacitance in the FD 9. And input to the amplification transistor 12. In the amplification transistor 12, the voltage amplified using the power supply voltage VDD is supplied to the selection signal φV from the row selection signal line LH to the gate of the row selection transistor 13. It is output to the signal line LV. On the other hand, the charge accumulated in the FD 9 is supplied with a high level reset pulse φRST from the row selection signal line LH to the reset gate 19 of the reset transistor 14, whereby the reset transistor 14 is turned on and reset to the reset voltage VRSB. Is done.

  Referring to FIG. 4, a large number of pixels are two-dimensionally arranged in the center (light receiving surface) of the image sensor 102, and the vertical scanning circuit 15 arranged at the peripheral edge is connected to the row selection signal line LH. The selection output is given, and the voltage outputted from each pixel to the vertical signal line LV is sequentially read out from the readout circuit 17 by the selective scanning of the horizontal scanning circuit 16. The vertical scanning circuit 15 and the horizontal scanning circuit 16 are realized by a shift register or the like.

  The readout circuit 17 includes two sample and hold circuits SH1 and SH2 connected to each column, that is, the vertical signal line LV, and a load transistor Q. The sample and hold circuits SH1 and SH2 include switches S1 and S2, capacitors C1 and C2, and amplifiers A1 and A2. Each vertical signal line LV is connected to capacitors C1 and C2 via switches S1 and S2, and the hold voltage is applied to the subtractor 18 via amplifiers A1 and A2. The amplifiers A1 and A2 in each column are driven by the horizontal scanning circuit 16.

  In such a readout circuit 17, in order to perform correlated double sampling, one of the switches S1 and S2 is turned on, and the transfer pulse φTX is applied to each pixel, so that one of the capacitors C1 and C2 is supplied from each pixel. Output voltage is held. The other of the switches S1 and S2 is turned on when the reset pulse φRST is applied to each pixel, and the output voltage from each pixel is held. The hold voltage is supplied to the subtractor 18 by selecting a pair of the amplifiers A1 and A2 by the horizontal scanning circuit 16, and thus the subtractor 18 provides the correlated double sample value with less influence of noise. Are output sequentially.

  FIG. 10 shows the pixel drive timing in the column selection period in detail. FIG. 10A shows the timing in so-called rolling shutter drive. During the column selection period, first, the reset gate 19 is turned on to reset the potential of the FD 9. The potential at that time is sampled and held in the capacitor C1 by turning on the switch S1. Next, the transfer gate 8 is turned on to end the accumulation time of the photodiode 4 and the accumulated charge is transferred to the FD 9. The potential of the FD 9 at that time is sampled and held in the capacitor C2 by turning on the switch S2. The charge accumulation period of the photodiode 4 is a period from when the transfer gate 8 turned on to transfer the charge is turned off after the transfer is turned on to when it is turned on and turned off again to transfer the charge in the next frame. .

  On the other hand, FIG. 10B shows timing in so-called global reset driving. In the rolling shutter drive of FIG. 10A, almost the entire imaging period is the charge accumulation period of the photodiode 4, whereas in this global reset drive, a mechanical shutter is used together, and in FIG. As shown, in the imaging period, a part of the period during which the shutter is open is the charge accumulation period. Therefore, by applying the bias voltage B only during this charge accumulation period, the transfer operation after the charge accumulation period and the circuit characteristics of the pixel portion can be stabilized in all pixels. Therefore, in this rolling shutter drive, the application period of the bias voltage B may be at least this charge accumulation period.

  In the imaging device 102 configured as described above, each pixel is divided into a plurality of groups, and different bias voltages B are applied from the contact portion 6 between the groups. FIG. 11 is a diagram for explaining the grouping. This example shows an example of realizing a color image based on visible light and a monochrome image based on infrared light. The two-dimensional array has 4n rows, and each pixel row has a group. The bias voltages BW, BYe, BRi, and BIR are applied from the contact portions 6W, 6Ye, 6Ri, and 6IR, respectively.

  FIG. 12 is a diagram schematically showing the energy potential distribution in a state where one pixel shown in FIG. 3 is cut along the A-A ′ section. The solid line shows the potential curve when the bias voltage B is low, while the dotted line shows the potential curve when the bias voltage B is high. The photodiode 4 is provided with an n-type region potential well 21 near the surface (incident light surface) and an n-type region potential well 23 at a deeper position (position far from the incident light surface). Charges generated by photoelectric conversion, which are divided by a boundary d1 between the n-type region of the photodiode 4 and the p-type region 22 thereunder, are accumulated in one of the potential wells 21 and 23. However, since light having a short wavelength is absorbed near the surface of the substrate 2 and light having a long wavelength penetrates and is absorbed to a deep portion of the substrate 2, the n-type region potential well 21 mainly transmits visible light. Charges are accumulated and mainly infrared light charges are discharged to the n-type region potential well 23. Here, the position of the boundary d1 changes depending on the bias voltage supplied from the outside. When the position of the boundary d1 is changed, the charges generated by the light on the longer wavelength side are accumulated in the potential well 21, and thus the spectral sensitivity becomes different. The boundary between the wavelength of the light beam in which charges are accumulated in the potential well 21 and the wavelength of the light beam in which charges are accumulated in the potential well 23 is referred to as a boundary wavelength, which is about 700 nm here.

  As shown in FIG. 12, the boundary that affects the spectral sensitivity is a relatively deep d1 position when the bias voltage B is low, and a relatively shallow d2 position when the bias voltage B is high. That is, when the bias voltage B is increased, the boundary moves to d2, which is shallower than d1, and as a result, as shown by the solid line to the dotted line in FIG. Thus, the sensitivity on the long wavelength side in the spectral sensitivity becomes relatively low. Therefore, by increasing the bias voltage B, the spectral characteristics of the filter F change from those shown in FIG. 7 to those shown in FIG. Note that the bias voltage can be changed steplessly, whereby the boundary between the potential wells 21 and 23 changes steplessly between d1 and d2.

  By utilizing this, for example, during daytime photography in which visible light is dominant, if the bias voltage B is set high, the sensitivity of the infrared light of each pixel becomes relatively low, so that the S / N ratio is high. A high-quality color image can be obtained. On the other hand, when shooting at night, when shooting must rely on infrared light, if the bias voltage B is set high, the sensitivity of infrared light in each pixel is relatively increased. Can capture dark subjects clearly. In other words, the same image sensor can be used to create a color image based on visible light and a highly sensitive color image to which infrared light is added, which contributes to cost reduction. Whether it is daytime shooting or nighttime shooting can be determined as follows. For example, when the light source detection unit 104e of the image processing unit 104 sets the value of the image component W as A and the value of the image component IR as B for all pixels, the control unit 105 sets B / A as an evaluation value. Output. When the evaluation value is greater than 0.5, the control unit 105 determines that the night-time shooting is performed with a large amount of infrared light. When the evaluation value is 0.5 or less, the control unit 105 has a relatively large amount of visible light. It can be determined that it is daytime shooting. Exposure data can also be used.

  FIG. 15 is a diagram schematically showing a cross-sectional structure of one pixel of an image sensor 102 according to another embodiment. FIG. 16 is a diagram schematically showing the energy potential distribution in a state where one pixel shown in FIG. 15 is cut along the A-A ′ section. The solid line shows the potential curve when the bias voltage B is low, while the dotted line shows the potential curve when the bias voltage B is high. In the above embodiment, the n region is formed on the p substrate to provide the charge discharging region. However, in this embodiment, the p layer is formed on the side closer to the incident surface side of the n substrate and the n substrate is formed. Is used as a charge discharge region. A bias voltage for controlling the position of the boundary d1 (d2) is applied to the n substrate. Since other than that is the same as that of embodiment mentioned above, description is abbreviate | omitted. The potential well 21 constitutes a capturing means that captures charges generated by receiving light having a wavelength less than the boundary wavelength (for example, 700 nm) that changes depending on the bias voltage, converts the charges into a signal, and outputs the signal. The discharge means for discharging the charges generated by receiving the light having the boundary wavelength or more is configured.

  Another example of controlling the bias voltage B in the present embodiment will be described. For example, when shooting at night, a subject such as a halogen lamp containing a large amount of infrared light may be included in the shooting screen. In such a case, the light source detection unit 104e of the image processing unit 104 obtains an evaluation value for each pixel, and the control unit 105 compares a pixel (specific pixel) with the evaluation value of 0.5 or less while performing night shooting. Assuming that the visible light is incident, the control unit 105 sets the bias voltage B to be applied high and relatively reduces the sensitivity of the infrared light of the specific pixel, thereby reproducing the S / N ratio of the color of the halogen light. A good, high-quality color image can be obtained. On the other hand, assuming that a large amount of infrared light is incident on a pixel having an evaluation value larger than 0.5 (a pixel other than the specific pixel), the control unit 105 sets the bias voltage B to be applied to be low and pixels other than the specific pixel. The sensitivity of infrared light can be kept relatively high. Furthermore, the sensitivity of a specific pixel can be changed according to the type of optical filter. That is, the sensitivity of a specific pixel provided with an IR filter that transmits only infrared light may be different from the sensitivity of a pixel provided with other filters. It can also be set according to the value of the evaluation value.

  According to yet another control example, the sensitivity of a specific pixel can be changed based on the luminance or illuminance of the subject. For example, when the illuminance of the subject is high, the sensitivity of all the pixels is reduced as it is shooting in the daytime, while when the illuminance of the subject is low, the sensitivity of all the pixels is increased as it is shooting at night be able to. Further, when the luminance of a part of the subject is high, the sensitivity is reduced by using a pixel that receives a light beam from the subject with high luminance as a specific pixel, and an appropriate amount of visible light is received. An appropriate amount of infrared light can be received by the sensitivity higher than that of the specific pixel. The luminance and illuminance of the subject can be estimated in the process of performing exposure control and the like, and the evaluation value can be calculated accordingly.

  According to yet another control example, the ratio of the amount of natural light and the amount of artificial light included in the light projected from the subject can be calculated, and the sensitivity of the specific pixel can be changed based on this ratio. The above evaluation value can be used as a value representing the ratio between the amount of natural light and the amount of artificial light. For example, when the ratio of natural light is high, the total sum of the evaluation values for all the pixels is small, so the bias voltage is changed so as to reduce the infrared light sensitivity of all the pixels as the specific pixels. On the other hand, when the proportion of artificial light is high, the sum of all the evaluation values for all pixels increases, so the bias voltage is changed so as to increase the infrared light sensitivity of all the pixels as the specific pixels.

  According to another control example, the infrared light sensitivity of pixels arranged in a part of the light receiving surface of the image sensor 102 as the specific pixel is made different from the infrared light sensitivity of other pixels. Thus, a bias voltage can be applied. For example, in the case of a fixed point monitoring camera, there is a case where a scene where the halogen lamp is always at a specific position on the shooting screen is sometimes monitored. If only the infrared light sensitivity is reduced, a high-quality color image can be taken even at night.

  According to yet another control example, the position of the specific pixel on the light receiving surface of the image sensor 102 can be changed in accordance with the shooting conditions. For example, in the case of a surveillance camera capable of panning and tilting, a halogen lamp or the like moves on the shooting screen as the camera moves. Therefore, a pixel on which halogen light or the like is incident is determined as a specific pixel based on the evaluation value, and only the infrared light sensitivity of the specific pixel on which the halogen light or the like is incident is changed by changing the specific pixel according to the movement of the camera. By reducing it, a high-quality color image can be taken regardless of the movement of the camera.

  According to yet another control example, in the case of continuously capturing an image of a subject, a signal from a specific pixel with increased infrared light sensitivity and a pixel with a lower infrared light sensitivity than the specific pixel (other than the specific pixel) Signal from the pixel) can be output alternately. In FIG. 17, when n is a natural number, the 2n-th frame is an image mainly based on infrared light based on a signal from a specific pixel with increased infrared light sensitivity, and the 2n + 1-th frame is specified. By making an image mainly based on visible light based on signals from pixels other than pixels, a moving image composed of a plurality of images can be formed. As a result, a moving image can be formed in which a color image based on a signal from a pixel receiving visible light and a monochrome image based on a signal from a pixel receiving infrared light are alternating frames. By combining these, it is possible to provide a moving image shot with an optimum spectral sensitivity characteristic for each subject.

  According to yet another embodiment, as shown in FIG. 18, the pixels of the image sensor 102 are divided into two groups, a first pixel group PX1 and a second pixel group PX2, and red pixels in the first pixel group PX1. The sensitivity of the external light can be made lower than the sensitivity of the infrared light in the second pixel group PX2, and the second pixel group PX2 can be arranged around the first pixel group PX1. For example, in a surveillance camera, a light may be lit on a subject at night. In such a case, visible light is reflected from the center of the subject, but the surroundings often remain dark. Therefore, by reducing the sensitivity of the first pixel group PX1 that receives visible light from the center of the subject and increasing the sensitivity of the second pixel group PX2 that receives light from the surroundings, a bright subject can be a dark subject. Can be monitored simultaneously. The sensitivity may be varied not only for each specific pixel but also for each pixel.

  According to yet another embodiment, when continuously imaging a subject, a signal from the first pixel group PX1 and a signal from the second pixel group PX2 are alternately output. As shown in FIG. 17, a moving image composed of a plurality of frames alternately including an image mainly based on visible light and an image mainly based on infrared light can be formed. By continuously capturing the subject while changing the sensitivity of the specific pixel, it is possible to form a moving image by continuously forming one image from a plurality of images captured under different conditions.

  In the above-described embodiment, the filter including the pixel filters W, Ye, Ri, and IR is shown. Spectral characteristics obtained by calculating R, G, B, and IR from this filter are shown in FIGS. By increasing the bias voltage B, the spectral characteristic of the filter changes from that shown in FIG. 19 to that shown in FIG.

  The image input device MG according to the second embodiment is characterized in that the calculation method of the color difference signals Crm and Cbm is different from that of the first embodiment. Note that the description of the same configuration as that of the first embodiment is omitted. The configuration of the image processing unit 104 is the same as that of the first embodiment, and will be described with reference to FIG.

  In this embodiment, the color space conversion unit 104d shown in FIG. 8 performs a smoothing process on the luminance signal Y obtained by Expression (2) to obtain the luminance signal Ys. Further, the luminance signal Yadds obtained by Expression (3) is subjected to smoothing processing to calculate the luminance signal Yadds. Here, as the smoothing process, a cascade filter process or a normal filter process may be employed.

Then, the color space conversion unit 104d, as shown in Expression (5), the ratio RT2 of the luminance signal Ys subjected to the smoothing process to the luminance signal Yadds subjected to the smoothing process (second ratio: RT2 = Ys / Yadds). Accordingly, the luminance signal Yadd is corrected, and the corrected luminance signal Yadd ′ is calculated as the luminance signal of the color space to be converted.
Yadd ′ = Yadd × Ys / Yadds (5)

  Since the luminance signal Y is obtained by the color space conversion process shown in Expression (2), the luminance signal Y is a signal that accurately reproduces the luminance of the image visually recognized by humans. On the other hand, since the luminance signal Yadd is calculated by the addition process shown in Expression (3), the luminance of the image viewed by humans cannot be accurately reproduced as compared with Expression (2). Therefore, as the luminance signal Yadds increases compared to the luminance signal Y3, that is, as the ratio RT2 decreases from 1, the luminance signal Yadd cannot accurately reproduce the luminance of the image viewed by humans. . Therefore, by obtaining the luminance signal Yadd ′ using the equation (5), the luminance signal Yadd can be converted into a signal that accurately reproduces the luminance of an image viewed by a human. Here, since the smoothed value is used for Ys, the noise component is reduced. However, unnatural artifacts may occur at the edge depending on the pattern.

Also, the color space conversion unit 104d. As shown in Expression (6), the color difference signals Crs and Cbs are corrected according to the above-described ratio RT1, and the color difference signals Crm ′ and Cbm ′ are calculated. Here, Yadd ′ is preferably used instead of Yadd.
Crm ′ = Crs × Yadd / Y
Cbm ′ = Cbs × Yadd / Y (6)

  Thereby, the balance between the luminance signal and the color difference signal in the color space to be converted is maintained, and the calculation accuracy of the color signals dR ′, dG ′, and dB ′ generated by the RGB color signal generation unit (not shown) in the subsequent stage is maintained. Can be increased.

  In addition, the color space conversion unit 104d compares the intensity of the infrared light and the intensity of the visible light. When the intensity of the infrared light is high, the luminance signal Yadd and the image component are reduced so that the weighting of the luminance signal Yadd is reduced. The luminance signal of the color space may be calculated by weighting and adding the signal Yt using Ye, Ri, IR, and W as they are.

In this case, if the color space conversion unit 104d determines that the ratio RT2 is smaller than the threshold Th1 indicating that the luminance signal Yadds is dominant, the intensity of infrared light is determined to be higher than the intensity of visible light. Good. Then, the luminance signal Yadd ′ may be calculated by weighted addition of the luminance signal Yadd and the signal Yt so that the weighting of the luminance signal Yday dd becomes small as shown in Expression (7).
Yadd ′ = Yt × (1−Ys / Yadds) + Yadd × (Ys / Yadds) (7)

  Yt represents a signal using any one of the image components Ye, Ri, IR, and W as they are. Specifically, when calculating the luminance signal Yadd ′ of a certain pixel, if the pixel corresponds to the W filter, Yt = W, and if it corresponds to the Ye filter, Yt = Ye, corresponding to the Ri filter. In this case, Yt = Ri, and in the case of corresponding to the IR filter, Yt = IR.

  When the luminance signal Yadds is dominant at the ratio RT2, the luminance signal Yadd ′ may greatly differ from the luminance of the image viewed by humans. Therefore, when the luminance signal Yadd ′ is large, the luminance signal Yadd ′ is calculated so that the ratio of the signal Yt to the luminance signal Yadd is large, whereby the luminance signal Yadd ′ is determined according to the luminance of the image viewed by humans. It can be a close signal.

  In addition, when the luminance signal Yadds is dominant at the ratio RT2, it indicates that infrared light is dominant. Therefore, the signal of the subject color between the pixels of Ye, Ri, IR, and W is displayed. Level difference can be ignored. That is, it can be regarded as equivalent to the case of the monochrome sensor. Therefore, the resolution (resolution) can be maximized by treating the pixels of Ye, Ri, IR, and W as independent signals without adding them.

  Furthermore, the color space conversion unit 104d may compare the intensity of infrared light and the intensity of visible light using the difference or ratio between the image component W and the image component IR. In this case, when IR / W is larger than a predetermined value close to 1, it is determined that the intensity of infrared light is stronger than the intensity of visible light, and when it is smaller than a predetermined value close to 0, the intensity of visible light is What is necessary is just to determine with it being stronger than the intensity | strength of infrared light.

  When W-IR is smaller than a predetermined value close to D, it is determined that the intensity of infrared light is stronger than the intensity of visible light. When IR is smaller than a predetermined value close to 0, the intensity of visible light is What is necessary is just to determine with it being stronger than the intensity | strength of infrared light.

Furthermore, since the intensity of infrared light becomes stronger than the intensity of visible light as the value of IR / W or IR-W increases, the weighting coefficient of the signal Yt is determined according to the value of IR / W or IR-W, The luminance signal Yadd ′ may be calculated by performing the calculation of Expression (7) ′.
Yadd ′ = k · Yt + Yadd · (1-k) (7) ′

  However, k ≦ 1, and k represents a weighting factor of a predetermined signal Yt that increases as IR / W or IR-W increases.

  Next, the RGB color signal generation unit (not shown) reversely converts the equation (2) to obtain the color signals dR ′, dG ′, and dB ′ from the luminance signal Yadd ′ and the color difference signals Crm ′ and Cbm ′. calculate.

  Next, the operation of the image input device MG will be described. FIG. 21 is a flowchart showing the operation of the image input apparatus MG according to the second embodiment.

  Steps S11 to S15 are the same as steps S1 to S5 shown in FIG. In step S16, the color space conversion unit 104d performs a smoothing process on the luminance signal Yadd to calculate the luminance signal Yadds, and also performs a smoothing process on the luminance signal Y to calculate the luminance signal Ys.

  Next, the color space conversion unit 104d calculates the luminance signal Yadd ′ by the calculation of Expression (5) when the ratio RT2 is equal to or greater than the threshold Th1, and when the ratio RT2 is less than the threshold Th1, the calculation of the Expression (7). A luminance signal Yadd ′ is calculated (step S17).

  Next, the color space conversion unit 104d obtains the color difference signals Crm ′ and Cbm ′ by the calculation of Expression (6) (Step S18). Next, the RGB color signal generation unit (not shown) inversely converts the equation (2) to obtain the color signals dR ′, dG ′, and dB ′ (step S19).

  As described above, according to the image input device MG according to the second embodiment, in addition to the effect of the first embodiment, when the ratio RT2 is equal to or greater than the threshold Th1, the luminance signal Yadd is expressed using Expression (5). ′ Is calculated, and when the ratio RT2 is less than the threshold Th1, the luminance signal Yadd ′ is calculated using the equation (7), and thus a luminance signal that accurately reproduces the luminance of the image visually recognized by humans is generated. be able to.

  Further, the color difference signals Crs and Cbs are corrected using the equation (6). Since the color difference signals Crm ′ and Cbm ′ are calculated, the luminance signal and the color difference signal can be obtained in a balanced manner in the color space to be converted.

  The color space conversion unit 104d may correct the color difference signals Crm ′ and Cbm ′ so that the color difference signals Crm ′ and Cbm ′ become lower as the intensity of infrared light approaches the intensity of visible light.

In this case, for example, the color space conversion unit 104d may calculate the color difference signals Crm ′ and Cbm ′ using Expression (8) instead of Expression (6).
Crm ′ = Crs × Yadd × RT2 / Y
Cbm ′ = Cbs × Yadd × RT2 / Y (8)

  That is, when the ratio RT2 is smaller than the threshold value TH1, the luminance signal Yadd ′ is dominant with respect to the luminance signal Y, and there is a high possibility that the night is being photographed. It may not be possible. Therefore, the color signals dR ′, dG ′, and dB ′ can be calculated so that the color difference signal is lowered by adopting the equation (8), and the influence of the color difference signal with low calculation accuracy is reduced. dR ′, dG ′, and dB ′ can be calculated visually without a sense of incongruity. Furthermore, the color signals dR ′, dG ′, and dB ′ may be further color-converted so as to be compatible with a standard display (for example, the IEC 61966-2-1 sRGB standard).

  Further, the formula used by comparing the ratio RT2 with the threshold is not limited to switching to (6) or (8), and the color difference signals Crs and Cbs are reduced in advance as the ratio RT2 decreases. The obtained correction value may be stored, and the color difference signals Crm ′ and Cbm ′ may be calculated using the correction value. In this case, in the equation (8), this correction value may be adopted instead of RT2.

  In the above embodiment, Ri pixels and IR pixels are used as the second to third pixels. However, the present invention is not limited to this, and cyan (C) pixels and magenta (M) pixels may be adopted. The good thing is as described above. That is, the first to fourth pixels may be Ye, M, C, and W pixels. In this case, incident light can be used more effectively, shot noise becomes lower, and noise components included in the original image data become lower. can do.

Further, in this case, assuming that image components obtained by Ye, C, M, and W image search are image components Ye, C, M, and W, the RGB color signal generation unit replaces Equation (1) with Equation (9). ) To obtain the color signals dR, dG, dB.
dR = W-C
dG = WM (9)
dB = W-Ye

In addition, the color space conversion unit 104d may obtain the luminance signal Yadd using Expression (10) instead of Expression (3).
Yadd = (1/4) × (Ye + C + M + W) (10)

  Here, weighted addition may be employed in the same manner as in equation (1) ′ instead of simple averaging as in equation (10). In the above embodiment, the color space conversion unit 104d performs the smoothing process on the color difference signals Cb and Cr. However, the present invention is not limited to this. That is, a smoothing processing unit may be provided before the color interpolation unit 104b, and the smoothing processing may be performed on the image components Ye, Ri, IR, and W so that the smoothing process by the color space conversion unit 104d may be omitted.

  Further, the smoothing process may be performed anywhere as long as the color signals dR, dG, and dB are calculated.

  Moreover, in the said embodiment, although the image pick-up element 102 was provided with four types of pixels, the 1st-4th pixel, it is not limited to this, What is necessary is just to be provided with at least 3 types of pixels.

  In this case, a type that can generate R, G, and B color signals from the spectral transmission characteristics of each pixel may be employed as the type of pixels that form the image sensor 102.

FIG. 22 is a flowchart according to a modification of the second embodiment. In this modification, when infrared light is dominant (the ratio is high), pixel data is weighted and added as it is to the luminance signal (11), and the gain of the chromaticity signal is lowered (12).
Yadd ′ = Yadds × Ys / Yadds
Yadd “= Yt × (1−Ys / Yadds) + Yadd ′ × (Ys / Yadds) (11)
Cadd ′ = Cadd × Ys / Yadds (12)

  The present invention can be applied to surveillance cameras and the like, but the application is not limited thereto.

It is a figure which shows the spectral characteristics of a halogen lamp, A vertical axis | shaft is light emission intensity and a horizontal axis is a wavelength. It is a schematic diagram of image input device MG concerning this embodiment. It is a figure which shows typically the cross-section of 1 pixel of the image pick-up element 102 which concerns on this Embodiment. 2 is a diagram illustrating an overall electrical configuration of an image sensor 102. FIG. 2 is a diagram illustrating a circuit configuration of one pixel of the image sensor 102. FIG. 3 is a schematic diagram illustrating a part of a filter F included in the image sensor 102. FIG. 6 is a diagram illustrating spectral characteristics of a filter F. FIG. 3 is a block diagram illustrating a detailed configuration of an image processing unit 104. FIG. It is a flowchart which shows operation | movement of the image input device MG concerning 1st Embodiment. It is a timing chart which shows the pixel drive timing in a column selection period. It is a figure for demonstrating the mode of the pixel group division which applies a bias voltage. FIG. 4 is a diagram schematically showing an energy potential distribution in a state where one pixel shown in FIG. 3 is cut along an A-A ′ section. It is a figure which shows the change of the spectral characteristics by a bias voltage change. It is a figure which shows the spectral characteristic of the filter F after a bias change. It is a figure which shows typically the cross-sectional structure of 1 pixel of the image pick-up element 102 which concerns on another example. FIG. 16 is a diagram schematically showing an energy potential distribution in a state where one pixel shown in FIG. 15 is cut along an A-A ′ section. It is a schematic diagram which shows the some example of a frame. It is a schematic diagram which shows an example of a photography screen It is a figure which shows the spectral characteristic of another filter. It is a figure which shows the spectral characteristic of another filter after a bias change. It is a flowchart which shows operation | movement of the image input device MG concerning 2nd Embodiment. It is a flowchart which shows operation | movement of the image input device MG concerning the modification of 2nd Embodiment.

3 element isolation region 4 photodiode 5 carrier discharge region 6 contact portion 6W, 6Ye, 6Ri contact portion 7 n-type region 8 transfer gate 11 transfer transistor 12 amplification transistor 13 row selection transistor 14 reset transistor 15 vertical scanning circuit 16 horizontal scanning circuit 17 Circuit 18 Subtractor 19 Reset gate 21 Potential well 21 n-type region potential well 23 n-type region potential well 101 Lens 102 Image sensor 103 Conversion unit 104 Image processing unit 104a Exposure correction unit 104b Color interpolation unit 104c Color signal generation unit 104d Color Spatial conversion unit 104e Light source detection unit 105 Control unit 106 Voltage control unit 107 Voltage generation unit F Filter LH Row selection signal line LV Vertical signal line MG Image input device PX1 First pixel group PX2 Second pixel group SH1, SH2 Sample hold circuit VDD Power supply voltage

Claims (6)

In a wavelength region including visible light and infrared light, at least three types of optical filters that transmit light fluxes in different wavelength bands, and a plurality of light filters that receive the light fluxes that have passed through the optical filter and output a signal corresponding to the amount of received light An image input device that includes an image pickup device including pixels and a calculation unit that performs calculation processing on a signal from the image pickup device, receives light projected from a subject, and inputs an image signal;
Differentiating the sensitivity of the infrared light in the first pixel group and the sensitivity of the infrared light in the second pixel group ;
When the subject is continuously imaged, the signal from the first pixel group and the signal from the second pixel group are alternately output, whereby an image mainly based on visible light and mainly infrared An image input apparatus that forms a moving image including a plurality of frames alternately including images based on light . The sensitivity of the infrared light in the first pixel group is lower than the sensitivity of the infrared light in the second pixel group, and the second pixel group is arranged around the first pixel group. The image input apparatus according to claim 1 . The image input apparatus according to claim 1 or 2, characterized in that it is possible to vary the sensitivity for each one pixel. The imaging device captures a charge generated by receiving light having a wavelength less than the boundary wavelength, converts it into a signal and outputs the signal, and discharges the charge generated by receiving light having the wavelength longer than the boundary wavelength. the image input apparatus according to claim 1, characterized in that it comprises pixels having the discharge means. The image input apparatus according to claim 4 , wherein the discharging unit is provided at a position deeper than a light incident surface than the capturing unit. 6. The image input apparatus according to claim 4, wherein the discharge amount of the charge discharged by the discharge means is variable according to a bias voltage control signal input from the outside.

Images