JP2012244533A - Imaging apparatus and image signal processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging apparatus which can output an excellent color image even in environment, typical of a nighttime mode, in which a visible light is weak, and to provide an image signal processing method.SOLUTION: The imaging apparatus includes a readout part which is arranged over a plurality of light receiving elements arrayed in two dimensions, and reads out a plurality of kinds of color signals transmitted through a plurality of kinds of color filters transmitting visible light and near-infrared light and then photoelectrically converted and a near-infrared signal transmitted through a near-infrared filter transmitting only near-infrared light and photoelectrically converted; a (k) calculation part 222 which calculates a mixing coefficient (k) indicative of the ratio of the near-infrared signal to be mixed with the plurality of kinds of color signals for each unit pixel according to the signal amount of the near-infrared signal; a luminance calculation part 223 which mixes the plurality of color signals and the near-infrared signal for each unit pixel based upon the mixing coefficient (k), and outputs the resulting signal as a luminance signal; and a color difference calculation part 215 which calculates a plurality of color component signals for each unit pixel using the plurality of kinds of color signals, the near-infrared signal, and the luminance signal, and outputs the signals.

Description

  The present invention relates to an imaging device and an image signal processing method that receive visible light and near infrared light.

  In recent years, as a security, network, in-vehicle application, etc., it has a function of capturing a color image using visible light as a light source during the daytime and capturing a black and white image by receiving near-infrared light at night. The demand for the so-called imaging device is rapidly increasing.

  Patent Document 1 discloses a conventional imaging device having a day / night function. Specifically, in the daytime, a light receiving path of a solid-state imaging device such as a MOS sensor or a charge coupled device (CCD) in which color filters of different colors are mounted on a light receiving device arranged two-dimensionally. A visible color image is captured by disposing an IR cut filter that blocks near-infrared light above. On the other hand, at night, by removing the IR cut filter, a monochrome image is captured by using all the signals of each pixel as luminance signals.

  Patent Document 2 discloses a conventional imaging device having a day / night function without disposing an IR cut filter in an optical system. Specifically, a plurality of visible light receiving elements equipped with different color filters and at least one near-infrared light receiving element are used, and in the daytime, the near-red light mixed in the signal of the visible light receiving element is used. A visible color image is captured by subtracting the external light component from the pure near-infrared light component obtained by the dedicated light-receiving element, and at night, a black-and-white image with the near-infrared signal incident on all pixels as the luminance signal Image.

  Also, in Patent Document 3, an object illuminated with red light and near infrared light (R + IR), green light and near infrared light (G + IR), or blue light and near infrared light (B + IR) with an external light source is monochrome solid. The prior art which improves a luminance signal using the IR signal transmitted simultaneously with visible light by imaging with an image pick-up element is disclosed.

US Pat. No. 4,316,659 JP 2008-35090 A JP-A-63-135121

  However, since the prior art disclosed in Patent Document 1 requires a mechanical switching device that attaches and detaches an IR cut filter, the cost increases as the number of parts increases, the reliability decreases due to fatigue of mechanical parts, and day and night There is a problem of mismatch between the optical system and the signal processing system due to sudden switching at the boundary, specifically, for example, generation of a low-quality image output period due to exposure deviation.

  In addition, the prior art disclosed in Patent Document 2 is a method that, once switched to the night mode, always forms a luminance signal based on near-infrared light and outputs only a black-and-white image. There is a problem that an image cannot be obtained.

  Also, in the prior art disclosed in Patent Document 3, once the mode is switched to night mode, a luminance signal based on near-infrared light is always formed and only a black and white image is output. There is a problem that a color image cannot be obtained and the mode cannot be switched within the frame.

  For this reason, for example, in cameras for security, networking, in-vehicle use, etc., it is difficult to capture an image using a weak color signal component that uses a streetlight or light leaked from a house as a light source even at night. It is in.

  The present invention has been made in view of the above problems, and provides an imaging apparatus and an image signal processing method that are capable of outputting a good color image even in an environment where a visible light signal is weak, represented by a night mode. Objective.

  In order to solve the above-described problems, an imaging device according to one embodiment of the present invention is two-dimensionally arranged, and is disposed on a plurality of light receiving elements that photoelectrically convert incident light and the plurality of light receiving elements. A plurality of types of color filters that transmit visible light and near-infrared light, a near-infrared filter that transmits only near-infrared light, and a plurality of types of color signals that are photoelectrically converted through the color filter; A reading unit that reads out from the plurality of light receiving elements photoelectrically converted near infrared signals that have passed through the near infrared filter, a light receiving element in which the near infrared filter is disposed, and the color adjacent to the light receiving element For each unit pixel configured with a light receiving element in which a filter is arranged, a mixing coefficient representing a ratio of the near-infrared signal to be mixed with the plurality of types of color signals read from the reading unit Calculated according to the amount of infrared signal A plurality of color signals read from the reading unit and the near-infrared signal are mixed for each unit pixel based on the mixing coefficient calculated by the coefficient calculating unit. And calculating a plurality of color component signals for each unit pixel using the luminance calculation unit that outputs the mixed signal as a luminance signal, the plurality of types of color signals and the near-infrared signal, and the luminance signal. And a color component calculation unit for outputting the output.

  According to the above aspect, by calculating the mixing coefficient according to the signal amount of the near-infrared signal, it is possible to colorize the image while effectively leaving the color component signal even if the visible light signal is weak. It becomes. In other words, the ratio of the near-infrared signal to the luminance signal can be adjusted according to the near-infrared signal amount.

  For example, if the mixing coefficient is reduced (if the ratio of the near-infrared signal in the luminance signal is reduced), the color component signal (for example, the color difference signal) becomes relatively large, and the image can be made closer to color from black and white. . Conversely, if the mixing coefficient is increased (if the ratio of the near-infrared signal occupying the luminance signal is increased), the color component signal (for example, the color difference signal) becomes relatively small, and the image can be made closer to color from black and white. .

  Therefore, it is possible to generate images in stages according to the mixing coefficient, such as black and white images, color images close to black and white, intermediate images between black and white and color images, black and white images close to color, and color images. Become. By controlling this for each unit pixel, even when outputting images in pseudo color mode using near-infrared light at night, depending on the properties of the object of interest, only a specific region of interest is converted into a high-resolution black-and-white image. Or red can be emphasized, and the object of interest can be easily recognized.

  The focal length of the condenser lens formed on the light receiving element on which the near-infrared filter is disposed is greater than the focal length of the condenser lens formed on the light receiving element on which the plurality of types of color filters are disposed. Short is preferred.

  Since the focal length is shorter as the wavelength is longer, the above-described aspect allows the near-infrared light signal having a wavelength longer than visible light to be efficiently incident on the light receiving element.

  The condensing lens may have an effective refractive index distribution generated by a light transmission film having a concentric structure divided by a line width equal to or shorter than the wavelength of incident light.

  As a result, the light collection efficiency with obliquely incident light is increased and the resolution and sensitivity are improved, and a solid-state imaging device with high definition and high color reproducibility can be realized.

  The refractive index of the condenser lens formed on the light receiving element on which the near-infrared filter is arranged is based on the refractive index of the condenser lens formed on the light receiving element on which the plurality of types of color filters are arranged. Is also preferably large.

  Since the focal length is shortened as the refractive index is increased, a signal of near infrared light having a wavelength longer than that of visible light can be efficiently incident on the light receiving element.

  The refractive index of the interlayer film formed on the light receiving element on which the near-infrared filter is disposed is larger than the refractive index of the interlayer film formed on the light receiving element on which the plurality of types of color filters are disposed. It is preferable.

  The refractive index of the interlayer film formed on the near-infrared light receiving element whose wavelength is longer than that of visible light is higher than the refractive index of the interlayer film formed on the light-receiving element for (visible light + near infrared light). Since it is large, a near-infrared light signal having a longer wavelength than visible light can be efficiently incident on the light receiving element.

  In addition, the present invention can be realized not only as an image pickup apparatus including such characteristic means, but also as an image signal processing method using the characteristic means included in the image pickup apparatus as a step.

  In order to solve the above problems, an image signal processing method according to one embodiment of the present invention includes a plurality of light receiving elements that are two-dimensionally arranged to photoelectrically convert incident light, and are disposed on the plurality of light receiving elements. An image signal processing method performed in an imaging apparatus including a plurality of types of color filters that transmit visible light and near infrared light and a near infrared filter that transmits only near infrared light, the color filter A step of reading out from the plurality of light receiving elements a plurality of types of color signals that have been transmitted through and photoelectrically converted, and a near infrared signal that has been transmitted through the near infrared filter and subjected to photoelectric conversion, and the near infrared filter For each unit pixel composed of the light receiving element in which the color filter is disposed and the light receiving element in which the color filter adjacent to the light receiving element is disposed, the unit pixel should be mixed with the plurality of types of color signals read in the reading step. A coefficient calculating step for calculating a mixing coefficient representing a ratio of the near-infrared signal in accordance with a signal amount of the near-infrared signal; and the plurality of types of color signals read in the reading step and the near-red A luminance output step of mixing an external signal for each of the unit pixels based on the mixing coefficient calculated in the coefficient calculation step, and outputting the mixed signal as a luminance signal, the plurality of types of color signals, A color component output step of calculating a plurality of color component signals for each unit pixel using the near-infrared signal, the luminance signal, and the luminance signal, and outputting the plurality of color component signals. Features.

  In the coefficient calculating step, the mixing coefficient is calculated so that the first area in the same frame image is a black and white image, and the second area different from the first area is a color image. A mixing coefficient may be calculated.

  This makes it possible to always output a color image regardless of nighttime or daytime. Furthermore, since only the specific attention area can be a high-resolution black-and-white image or color image, the resolution or visibility can be improved.

  Further, each of the plurality of types of color signals and the infrared signal has a normalized value, the luminance signal output in the luminance output step is Y, and red among the plurality of types of color signals. The luminance signal read from the light receiving element having the color filter (R + IR), the luminance signal read from the light receiving element having the green color filter (G + IR), and the light receiving element having the blue color filter (B + IR), where IR is the luminance signal read from the light receiving element having the near-infrared filter, and k is the mixing coefficient, Y = (R + IR) + (G + IR) + ( B + IR) + k × IR, and in the coefficient calculation step, the mixing coefficient may be calculated at a value within a range of −3 to +1.

  Thereby, the calculation of the luminance signal can be generated by a simple calculation.

  Further, in the color component output step, the components of the near-infrared signal component included in the plurality of types of color signals and the near-infrared signal so as to maximize the gain of the red component signal among the plurality of color component signals. The plurality of color component signals may be output by adjusting the amount.

  As a result, the near-infrared signal is preferentially added to the red color of the attention area in the color signal processing, whereby the red color is emphasized and displayed even in the area where the visible light is weak, thereby improving the visibility.

  In the color component output step, the gain of the red signal of the plurality of types of color signals is set to be higher than the gain of the other color signals so as to maximize the gain of the red component signal of the plurality of color component signals. And the plurality of color component signals may be output.

  As a result, even in a region where the visible light is weak, red is highlighted and the visibility can be improved.

  In the coefficient calculation step, the mixing coefficient is calculated according to a signal amount of the near-infrared signal for each minimum pixel row in one frame. In the luminance output step, the plurality of types of color signals are calculated. And the near-infrared signal are mixed based on the mixing coefficient, and the mixed signal is output as a luminance signal. In the color component output step, the plurality of types of color signals, the near-infrared signal, and the By calculating a plurality of color component signals using the luminance signal and outputting the plurality of color component signals, a color image and a black-and-white image mainly composed of luminance components by near-infrared light are minimized in one frame. It may be switched and displayed in units of one line.

  As a result, an optimum mixing coefficient can be obtained for each pixel row in the frame, and a monochrome image and a color image are generated in one frame, so that an image having the advantages of both can be provided to the user.

  According to the imaging device and the image signal processing method of the present invention, by calculating the mixing coefficient according to the signal amount of the near-infrared signal, the color component signal is effectively left even if the visible light signal is weak, Images can be colorized.

It is a block diagram which shows the main function structures of the imaging device which concerns on Embodiment 1 of this invention. It is a flowchart which shows an example of the mixing coefficient calculation process in a k calculation part. 3 is an operation flowchart illustrating an image signal processing method according to Embodiment 1 of the present invention. It is a block diagram which shows the main function structures of the imaging device which concerns on Embodiment 2 of this invention. It is a flowchart which shows an example of the mixing coefficient calculation process in a brightness | luminance calculation part.

  Hereinafter, embodiments according to the present invention will be described in detail with reference to the drawings.

(Embodiment 1)
The imaging apparatus according to the present embodiment reads a plurality of types of color signals and near-infrared signals corresponding to a plurality of colors of visible light from a solid-state imaging device having a plurality of light receiving elements arranged two-dimensionally, Depending on the signal amount of the near-infrared signal, a mixing coefficient indicating the mixing ratio of the near-infrared signal with respect to a plurality of types of color signals is calculated for each unit pixel, and the mixing coefficient is used to calculate a near-infrared signal and a plurality of types of color signals. Infrared signals are mixed, the mixed signals are output as luminance signals, and a plurality of color component signals are calculated and output using a plurality of types of color signals and near-infrared signals.

  Thereby, by calculating the mixing coefficient according to the signal amount of the near-infrared signal, even if the visible light signal is weak, it is possible to effectively leave the color component signal and colorize the image.

  In particular, even when an image is output in a pseudo color mode using near-infrared light at night, only a specific attention area is set to high resolution IR black and white (for example, a car license plate), or red is emphasized (for example, a sign) ) To make it easier to recognize the object of interest.

  First, the configuration of the imaging apparatus (camera) according to Embodiment 1 of the present invention will be described.

  FIG. 1 is a block diagram showing the main functional configuration of the imaging apparatus according to Embodiment 1 of the present invention. The imaging apparatus 100 described in the figure includes a solid-state imaging device 101, a signal processing unit 201, an imaging control unit 301, a light source control unit 401, an IR (near infrared light) light source 411, a lens 501, A diaphragm 502 is provided.

  The solid-state imaging device 101 includes a plurality of unit pixels 115, a vertical shift register 121, a horizontal shift register 122, a noise removal circuit 123, and an output amplifier 124, and photoelectrically converts incident light to generate a color signal. MOS type image sensor or the like.

  The plurality of unit pixels 115 receive (R + IR) pixels that receive red light and near infrared light, (G + IR) pixels that receive green light and near infrared light, and receive blue light and near infrared light (B + IR). A pixel and four pixels (IR) pixels that receive near-infrared light are defined as one unit. The (R + IR) pixel includes a light receiving element and an (R + IR) transmission filter 111 disposed on the light receiving element, and the (G + IR) pixel is a (G + IR) transmission disposed on the light receiving element and the light receiving element. The (B + IR) pixel includes a light receiving element and a (B + IR) transmission filter 113 disposed on the light receiving element, and the IR pixel is disposed on the light receiving element and the light receiving element. IR transmission filter 114.

  The (R + IR) transmission filter 111, the (G + IR) transmission filter 112, and the (B + IR) transmission filter 113 are color filters that transmit visible light (any one of R, G, and B) and near-infrared light (IR), respectively. It is. The IR transmission filter 114 is a near-infrared filter that suppresses visible light and transmits near-infrared light. The unit pixel 115 corresponds to one pixel in a color image or a monochrome image.

  The solid-state imaging device 101 selects each pixel row of the unit pixels 115 arranged two-dimensionally by the vertical shift register 121, and removes noise from the pixel signals of the unit pixels belonging to the selected pixel row by the noise removal circuit 123. The color signal is selected by the horizontal shift register 122 and the color signal for each unit pixel 115 is output from the output amplifier 124 to the signal processing unit 201. The vertical shift register 121, the horizontal shift register 122, and the selection circuit 125 are a plurality of types of color signals that are photoelectrically converted through the color filter, and near-infrared signals that are photoelectrically converted through the near-infrared filter. And function as a reading unit that reads from a plurality of light receiving elements.

  In order to easily realize the configuration of the pixel having the (R + IR) transmission filter 111, the (G + IR) transmission filter 112, the (B + IR) transmission filter 113, and the IR transmission filter 114, a photonic crystal color filter is used as a solid-state imaging device. It is preferable to accumulate on 101. An optical multilayer film obtained by alternately laminating low refractive index and high refractive index materials has a forbidden band that does not transmit light of a specific wavelength. In the present embodiment, the IR transmission filter is formed by adjusting the transmission band so that near-infrared light is transmitted.

  In the (R + IR) transmission filter 111, the (G + IR) transmission filter 112, and the (B + IR) transmission filter 113, materials having a low refractive index and a high refractive index are alternately laminated with a film thickness of λ / 4 (λ: wavelength). The optical multilayer film is formed by introducing a “defect layer” having a film thickness different from λ / 4. In other words, the defect layer disturbs the optical periodicity, and a transmission band can be generated in the forbidden band. Therefore, a transmission band having a desired wavelength band can be realized by appropriately designing the thickness of the defect layer. That is, the transmission filter 111 that transmits red light and near infrared light (R + IR), the transmission filter 112 that transmits green light and near infrared light (G + IR), and the blue light and near infrared light transmit ( B + IR) transmission filter 113 and IR transmission filter 114 that transmits only near-infrared light can be easily designed.

  Here, the focal length of the condensing lens formed on the light receiving element on which the IR transmission filter 114 is arranged has the (R + IR) transmission filter 111, the (G + IR) transmission filter 112, and the (B + IR) transmission filter 113 arranged thereon. It is preferable that the focal length of the condensing lens formed on the light receiving element is shorter.

  Since the focal length becomes shorter as the wavelength becomes longer, an IR signal having a wavelength longer than that of visible light can be efficiently incident on the light receiving element due to the relation of the focal length.

  In addition, the condensing lens formed on the light receiving element on which the IR transmission filter 114 is disposed has an effective refraction caused by a light transmission film having a concentric structure divided by a line width equal to or shorter than the wavelength of incident light. It may have a rate distribution.

  As a result, the light collection efficiency with obliquely incident light is increased and the resolution and sensitivity are improved, and a solid-state imaging device with high definition and high color reproducibility can be realized.

  The refractive index of the condensing lens formed on the light receiving element on which the IR transmission filter 114 is arranged is such that the (R + IR) transmission filter 111, the (G + IR) transmission filter 112, and the (B + IR) transmission filter 113 are received. You may make it larger than the refractive index of the condensing lens formed on the element.

  Since the focal length becomes shorter as the refractive index increases, an IR signal having a wavelength longer than that of visible light can be efficiently incident on the light receiving element due to the relationship of the refractive index of the condenser lens.

  The interlayer film formed on the light receiving element on which the IR transmission filter 114 is arranged is formed on the light receiving element on which the (R + IR) transmission filter 111, the (G + IR) transmission filter 112, and the (B + IR) transmission filter 113 are arranged. The refractive index may be larger than the interlayer film formed.

  The refractive index of the interlayer film formed on the near-infrared light receiving element whose wavelength is longer than that of visible light is higher than the refractive index of the interlayer film formed on the light-receiving element for (visible light + near infrared light). Since it is large, an IR signal having a wavelength longer than that of visible light can be efficiently incident on the light receiving element.

  The imaging control unit 301 includes an exposure time control unit 311 and a diaphragm control unit 312, and adjusts the diaphragm 502 and the exposure time of the unit pixel 115 based on a control signal from the near infrared signal control unit 221.

  The light source control unit 401 adjusts the irradiation amount of the IR light source 411 based on the control signal from the near infrared signal control unit 221.

  The lens 501 forms an image of incident light on the imaging area of the solid-state imaging device 101.

  The signal processing unit 201 includes an OB calculation unit 211, a low-pass filter (LPF) 212, a 4 × 4 matrix calculation unit 213, a white balance unit 214, a color difference calculation unit 215, a gain adjustment unit 216, and a γ correction unit. 217, a near-infrared signal control unit 221, a k calculation unit 222, and a luminance calculation unit 223, and performs image signal processing on the four types of pixel signals output from the unit pixel 115 of the solid-state imaging device 101. DSP.

  The signal processing unit 201 performs offset removal such as dark current on the four types of input image signals using the OB calculation unit 211. Subsequently, the noise signal is removed from the color signal by the low-pass filter 212.

  The 4 × 4 matrix calculation unit 213 obtains an R signal, a G signal, and a B signal by subtracting the IR signal multiplied by the specific gravity from the original signals of the (R + IR) pixel, the (G + IR) pixel, and the (B + IR) pixel. . At this time, the specific gravity coefficient is adjusted so that near infrared light disappears. That is, the 4 × 4 matrix operation unit 213 performs the 4 × 4 matrix operation on the (R + IR) signal, the (G + IR) signal, the (B + IR) signal, and the IR signal to obtain the R signal, the G signal, and the B signal. Output.

  The white balance unit 214 performs white balance and performs color correction between the three primary colors in the same manner as the normal three primary colors.

  The color difference calculation unit 215 uses a plurality of types of color signals, that is, the R signal, the G signal, and the B signal, and the luminance signal Y calculated by the luminance calculation unit 223 to generate a plurality of color component signals (RY signals). , BY signal) is calculated and output.

  The color signal output from the γ correction unit 217 is output as an image signal to a display device or the like together with the luminance signal output from the luminance calculation unit 223.

  In the present embodiment, a signal output from the light receiving element in which the IR transmission filter 114 is arranged is used in the calculation for creating the luminance signal. The luminance signal is calculated by the following equation 1.

  Luminance signal Y = (R + IR) pixel luminance signal + (G + IR) pixel luminance signal + (B + IR) pixel luminance signal + k × (IR) pixel luminance signal (Equation 1)

  That is, Y = (R + IR) + (G + IR) + (B + IR) + kIR. Here, k is a mixing coefficient indicating the mixing ratio of the IR signal with respect to the ((R + IR) + (G + IR) + (B + IR)) signal or a mixing coefficient indicating the mixing ratio of the IR signal with respect to the luminance signal Y. . In this case, k is determined within a range of −3 to +1.

  When k = −3, the luminance signal Y from which the IR signal is completely removed can be obtained. This is suitable for imaging in the daytime mode. When k = + 1, the luminance signal Y including the entire IR signal can be obtained. This is suitable for generating black and white images. When k is an intermediate value between -3 and +1, a color image is generated, and the degree of colorization is determined according to k. This is suitable for colorizing in the night mode.

  FIG. 2 is a flowchart illustrating an example of the mixing coefficient calculation process in the k calculation unit 222. In the figure, Ave (IR) is an average value of IR signals in the previous frame, and Th1 to Th4 are threshold values satisfying Th1> Th2> Th3> Th4.

  First, the k calculation unit 222 sets the mixing coefficient k = k1 when AVE (IR) is larger than the threshold Th1 (Yes in S21) (S22).

  Next, when AVE (IR) is equal to or smaller than the threshold Th1 (No in S21) and larger than the threshold Th2 (Yes in S23), the mixing coefficient k = k2 is set (S24).

  Next, when AVE (IR) is equal to or smaller than the threshold Th2 (No in S23) and larger than the threshold Th3 (Yes in S25), the mixing coefficient k = k3 is set (S26).

  Next, when AVE (IR) is equal to or smaller than the threshold Th3 (No in S25) and larger than the threshold Th4 (Yes in S27), the mixing coefficient k = k4 is set (S28).

  Next, when AVE (IR) is equal to or less than the threshold Th4, the mixing coefficient k = k5 is set (S29). Here, k1, k2, k3, and k4 are, for example, +1, 0, -1, -2, and -3. The mixing coefficient may be any number as long as it is three or more levels, or may be a continuous value.

  As described above, the k calculation unit 222 includes the light receiving element in which the IR transmission filter 114 is arranged and the light receiving element in which the (G + IR) transmission filter 112 and the (B + IR) transmission filter 113 are arranged adjacent to the light receiving element. For each unit pixel, a mixing coefficient k indicating the mixing ratio of the IR signal with respect to the (R + IR) signal, (G + IR) signal, and (B + IR) signal read from the solid-state imaging device 101 is determined according to the signal amount of the IR signal. It is a coefficient calculation part to calculate.

  The luminance calculation unit 223 uses the mixing coefficient k calculated by the k calculation unit 222 to mix the (R + IR) signal, (G + IR) signal, (B + IR) signal, and IR signal read from the solid-state imaging device 101. Then, the mixed signal is output as the luminance signal Y.

  The near-infrared signal control unit 221 controls the component amount of the IR signal used for image signal calculation processing in the signal processing unit 201 to a predetermined value (for each unit pixel). Specifically, the near-infrared signal control unit 221 outputs the IR signal used by the k calculation unit 222 to the k calculation unit 222.

  Further, the near-infrared signal control unit 221 controls the imaging control unit 301 according to the component amount of the IR signal used for the luminance signal calculation, and controls the exposure amount of the solid-state imaging device 101 to a predetermined value. Specifically, the near-infrared signal control unit 221 controls the exposure time control unit 311 to adjust the exposure time of the solid-state imaging device 101 to a predetermined value. Further, the near-infrared signal control unit 221 controls the diaphragm control unit 312 to adjust the amount of incident light to a predetermined value by adjusting the diaphragm 502 of the lens 501. Alternatively, the near infrared signal control unit 221 controls the light source control unit 401 to adjust the irradiation amount of the IR light source 411 that irradiates the subject with predetermined near infrared light.

  Thus, the near-infrared light component component amount applied to the luminance signal is controlled for each pixel by the near-infrared signal control unit 221, so that the pixel and the near-infrared light that output a color image in the same frame It is possible to switch the pixel that outputs a black and white image mainly including the luminance component by the unit of one pixel.

  Further, the near-infrared signal control unit 221 adjusts the gain by controlling the gain adjustment unit 216 according to the output of the white balance unit 214 for the component amount of the IR signal component applied to the color signal component. In particular, in order to obtain red color reproducibility such as signs required for in-vehicle cameras and security cameras, the component amount of the IR signal component is controlled so as to maximize the gain of the R signal. That is, the near-infrared signal control unit 221 determines a color gain for enhancing or suppressing the specific color according to the signal amount of the IR signal.

  The gain adjustment unit 216 adjusts the gains of the plurality of color component signals calculated by the color difference calculation unit 215 according to the color gain.

  Thereby, the signal processing unit 201 can give the maximum gain to the R signal component, for example, in the step of generating the color reproduction signal composed of only the visible light component excluding the IR signal component of each light receiving element. . Note that the near-infrared signal control unit 221 may receive a red enhancement instruction that is a specific color based on a user operation from the outside, and perform a process of relatively increasing the gain for the received color.

  Further, instead of emphasizing a specific color, suppression other than the accepted color may be performed. In this case, the gain of the specific color may be relatively decreased.

  Further, the near-infrared signal control unit 221 receives a red enhancement instruction that is a specific color based on a user operation from the outside, and performs a process of preferentially adding the luminance signal of the near-infrared signal to the received color. Thus, the received color may be emphasized.

  With the above configuration and signal processing, the signal processing unit 201 generates an R signal, a G signal, and a B signal from the four types of pixel signals output from the solid-state imaging device 101, and generates an IR signal for each unit pixel 115. . Then, a color image composed of a luminance signal and two color difference signals is generated. The luminance signal alone represents a black and white image.

  In order to generate the R signal, the G signal, and the B signal from the pixel signal output from the solid-state imaging device 101, for example, when generating the R signal, the (R + IR) pixel that is the output signal of the (R + IR) pixel. The R signal is derived by subtracting the IR pixel signal that is the output signal of the IR pixel from the signal.

  If the above configuration and signal processing are used, the mixing coefficient k is calculated according to the signal amount of the IR signal, so that the color component signal is effectively left even if the visible light signal is weak, and the image is colored. be able to. In other words, the ratio of the infrared signal to the luminance signal can be adjusted according to the amount of infrared signal. For example, if the mixing coefficient is decreased (the ratio of the infrared signal to the luminance signal is decreased), the color component signal (for example, the color difference signal) is relatively increased, and the image can be made closer to color from black and white. Conversely, if the mixing coefficient is increased (if the ratio of the infrared signal in the luminance signal is increased), the color component signal (for example, the color difference signal) is relatively reduced, and the image can be made closer to color from black and white.

  For example, in the daytime, the red, green, and blue signals can be obtained by subtracting the IR signal component of the IR pixel from the red, green, and blue original signals including the near infrared light component. Therefore, a mechanical IR cut filter is unnecessary even during the daytime. On the other hand, at night, a near-infrared light image is obtained by using signals of all pixels. That is, the resolution of the IR image is higher than that of the RGB image in the daytime because the IR components of (R + IR) pixels, (G + IR) pixels, and (B + IR) pixels are used.

  The imaging apparatus 100 according to the present embodiment can generate a black and white image, a color image close to black and white, an intermediate image between black and white and a color image, a black and white image close to color, and a color image according to the mixing coefficient. . An imaging device that outputs a visible color image even in a night mode in an environment where a visible light signal is weak can be realized in a solid-state imaging device that captures an image by receiving near-infrared light at night, such as a day / night camera. Even when an image is output in pseudo color mode using near infrared light at night, by controlling the image signal for each unit pixel, only a specific region of interest is made to have high resolution IR monochrome (for example, a car license plate). ) Or by emphasizing red (for example, a sign), the object of interest can be easily recognized.

  Note that the present invention can be realized not only as the imaging apparatus 100 including the above-described characteristic means, but also as an image signal processing method using the characteristic means included in the imaging apparatus 100 as a step. . Hereinafter, an image signal processing method performed in the imaging apparatus 100 will be described.

  FIG. 3 is an operation flowchart for explaining the image signal processing method according to the first embodiment of the present invention.

  First, the signal processing unit 201 passes through the (R + IR) transmission filter 111, the (G + IR) transmission filter 112, and the (B + IR) transmission filter 113 included in the solid-state imaging device 101, and (R + IR) signals photoelectrically converted and (G + IR) ) Signal, (B + IR) signal, and IR signal that has been subjected to photoelectric conversion through the IR transmission filter 114 are read out from a plurality of light receiving elements (S01: read step).

  Next, the k calculator 222 calculates, for each unit pixel, a mixing coefficient k indicating the mixing ratio of the IR signal with respect to the (R + IR) signal, the (G + IR) signal, and the (B + IR) signal read in step S01, as IR Calculation is performed according to the signal amount of the signal (S02: coefficient calculation step).

  Next, the luminance calculation unit 223 uses the (R + IR) signal, (G + IR) signal, (B + IR) signal, and IR signal read out in step S01 based on the mixing coefficient k calculated in step S02. Each unit pixel is mixed, and the mixed signal is output as a luminance signal Y (S03: luminance output step).

  Finally, the color difference calculation unit 215 uses the (R + IR) signal, the (G + IR) signal, the (B + IR) signal and the IR signal, and the luminance signal Y read out in step S01 to generate a plurality of colors for each unit pixel. The component signal is calculated and the plurality of color component signals are output (S04: color component output step).

  According to the above-described image signal processing method, by calculating the mixing coefficient k according to the signal amount of the IR signal, the color component signal is effectively left even if the visible light signal is weak, and the image is colored. be able to. For example, if the mixing coefficient is decreased (the ratio of the infrared signal to the luminance signal is decreased), the color component signal (for example, the color difference signal) is relatively increased, and the image can be made closer to color from black and white. Conversely, if the mixing coefficient is increased (if the ratio of the infrared signal in the luminance signal is increased), the color component signal (for example, the color difference signal) is relatively reduced, and the image can be made closer to color from black and white. Therefore, a visible color image can be output even in the night mode.

  In step S02, the mixing coefficient k is calculated so that the first area in the same frame image becomes a black and white image, and the mixing coefficient k is set so that the second area different from the first area becomes a color image. It may be calculated.

  This makes it possible to always output a color image regardless of nighttime or daytime. Therefore, even when an image is output in pseudo color mode using near infrared light at night, by controlling the image signal for each unit pixel, only a specific attention area is made to be a high resolution IR monochrome (for example, a car It is possible to make it easier to recognize the object of interest by emphasizing red (for example, a license plate) or red (for example, a sign). Furthermore, since only the specific attention area can be a high-resolution black-and-white image or color image, the resolution or visibility can be improved.

  Further, the (R + IR) signal, the (G + IR) signal, the (B + IR) signal, and the IR signal each have a normalized value, and in step S02, the mixing coefficient k is set to a value within a range from -3 to +1. You may calculate in.

  Thereby, the calculation of the luminance signal Y can be generated by a simple calculation according to the above Equation 1.

  In step S04, the near-infrared signal control unit 221 adjusts the component amount of the IR signal component so that the gain of the red component signal among the plurality of color component signals is maximized, whereby the plurality of colors. A component signal may be output.

  As a result, the near-infrared signal is preferentially added to the red color of the attention area in the color signal processing, whereby the red color is emphasized and displayed even in the area where the visible light is weak, thereby improving the visibility.

  In step S04, the near-infrared signal control unit 221 increases the gain of the R signal over the gains of the other color signals so as to maximize the gain of the red component signal among the plurality of color component signals. Thus, the plurality of color component signals may be output.

  As a result, even in a region where the visible light is weak, red is highlighted and the visibility can be improved.

  In step S02, the mixing coefficient k may be calculated for each minimum pixel row in one frame according to the signal amount of the IR signal.

  As a result, an optimum mixing coefficient can be obtained for each pixel row in the frame, and a monochrome image and a color image are generated in one frame, so that an image having the advantages of both can be provided to the user.

(Embodiment 2)
In the present embodiment, first, the luminance calculation unit further mixes a plurality of types of color signals and near-infrared signals by using a dummy mixing coefficient kd for suppressing the infrared signals. An example in which the luminance signal Yd is calculated and the color difference calculation unit 215 generates two color difference signals as color component signals using the dummy luminance signal Yd will be described.

  Second, the luminance calculation unit mixes the first predetermined number of consecutive rows and the second predetermined number of consecutive rows that are alternately repeated so that the first predetermined number of consecutive rows become a monochrome image. A configuration for calculating the coefficient and calculating the mixing coefficient so that the second predetermined number of continuous rows is a color image will be described.

  Third, the near-infrared signal control unit 221 determines a color gain for emphasizing or suppressing a specific color according to the signal amount of the near-infrared signal, and the gain adjustment unit 216 determines according to the color gain. An example of adjusting the gains of a plurality of color component signals calculated by the color difference calculation unit 215 will be described.

  First, the configuration of an imaging apparatus (camera) according to Embodiment 2 of the present invention will be described.

  FIG. 4 is a block diagram showing the main functional configuration of the imaging apparatus according to Embodiment 2 of the present invention. The imaging apparatus 200 shown in the figure includes a solid-state imaging device 101, a signal processing unit 251, an imaging control unit 301, a light source control unit 401, an IR (near infrared light) light source 411, a lens 501, A diaphragm 502 is provided. The imaging apparatus 200 illustrated in the figure is different from the imaging apparatus 100 illustrated in FIG. 1 in that a luminance calculation unit 423 is provided instead of the luminance calculation unit 223 as a configuration. Hereinafter, the imaging apparatus 200 according to the present embodiment will be described with a focus on different points while omitting the description of the same points as the imaging apparatus 100.

  In addition to the function of the luminance calculation unit 223, the luminance calculation unit 423 calculates a dummy luminance signal Yd, and supplies the calculated dummy luminance signal Yd to the color difference calculation unit 215 as the luminance signal Y. Here, the dummy luminance signal is a luminance signal in which the IR signal is completely suppressed, and is the luminance signal Y when k = −3.

  Thereby, the color difference calculation unit 215 calculates the color difference signal when the IR signal is completely suppressed even when the mixing coefficient is large. Thereby, two color difference signals with good color reproducibility can be generated.

  Further, the luminance calculation unit 423 calculates a mixing coefficient so that the first predetermined number of continuous rows become a black and white image, and the second predetermined number of continuous rows become a color image, as compared with the luminance calculation unit 223. The difference is that the mixing coefficient is calculated.

  FIG. 5 is a flowchart illustrating an example of the mixing coefficient calculation process in the luminance calculation unit 423.

  First, the luminance calculation unit 423 calculates k for the first predetermined number of consecutive rows in the loop 1 as in FIG. 2 (S51 to S53). As a result, the first predetermined number of consecutive rows becomes a color image.

  Next, the luminance calculation unit 423 determines k = 1 for the second predetermined number of consecutive frames in the loop 2 (S54 to S55). As a result, the second predetermined number of consecutive frames become a monochrome image.

  Note that (the first predetermined number and the second predetermined number) are both integers of 1 or more, and may be (1, 1), (1, 2), (2, 1), or the like.

  By the above configuration and signal processing, the black and white image and the color image are alternately generated for each predetermined number of rows, so that an image having the advantages of both can be provided to the user.

  The near-infrared signal control unit 221 receives a red enhancement instruction that is a specific color based on a user operation from the outside, and performs a process of preferentially adding a luminance signal of the near-infrared signal to the received color. The received color may be emphasized.

  As a result, the near-infrared signal is preferentially added to the red color of the attention area in the color signal processing, whereby the red color is emphasized and displayed even in the area where the visible light is weak, thereby improving the visibility.

  In addition, the luminance calculation unit 423 calculates the mixing coefficient so that the first predetermined number is a monochrome image on one line, and calculates the mixing coefficient so that the second predetermined number is a color image. A monochrome image and a color image may be switched for each row.

  As a result, an optimum mixing coefficient can be obtained for each pixel row in the frame, and a monochrome image and a color image are generated in one frame, so that an image having the advantages of both can be provided to the user.

  Note that the luminance calculation unit 423 calculates the mixing coefficient so that the first predetermined number of consecutive frames become a monochrome image, and calculates the mixing coefficient so that the second predetermined number of consecutive frames become a color image. You may control.

  The imaging device according to the present invention is not limited to the first and second embodiments. Other embodiments realized by combining arbitrary components in the first and second embodiments, and various modifications conceivable by those skilled in the art without departing from the gist of the present invention to the first and second embodiments. Variations obtained and various devices incorporating the imaging device according to the present invention are also included in the present invention.

  The imaging apparatus of the present invention can be applied to a camera that can colorize an image by effectively leaving a color component signal even if the visible light signal is weak, and in particular, a surveillance camera, a network camera, an in-vehicle camera, a digital camera, a mobile phone It is possible to improve the image quality of captured images at night.

100, 200 Imaging device 101 Solid-state imaging device 111 (R + IR) transmission filter 112 (G + IR) transmission filter 113 (B + IR) transmission filter 114 IR transmission filter 115 Unit pixel 121 Vertical shift register 122 Horizontal shift register 123 Noise removal circuit 124 Output amplifier 125 Selection circuit 201, 251 Signal processing unit 211 OB calculation unit 212 Low-pass filter (LPF)
213 4 × 4 matrix calculation unit 214 White balance unit 215 Color difference calculation unit 216 Gain adjustment unit 217 γ correction unit 221 Near infrared signal control unit 222 k calculation unit 223 423 Brightness calculation unit 301 Imaging control unit 311 Exposure time control unit 312 Aperture control unit 401 Light source control unit 411 IR (near infrared light) light source 501 Lens 502 Aperture

Claims (11)

A plurality of light receiving elements that are two-dimensionally arranged and photoelectrically convert incident light; and
A plurality of types of color filters that transmit visible light and near-infrared light, and a near-infrared filter that transmits only near-infrared light, disposed on the plurality of light-receiving elements;
A readout unit that reads out from the plurality of light receiving elements a plurality of types of color signals that have been photoelectrically converted through the color filter, and a near-infrared signal that has been photoelectrically converted through the near-infrared filter,
The plurality of types of colors read from the readout unit for each unit pixel composed of a light receiving element in which the near-infrared filter is arranged and a light receiving element in which the color filter adjacent to the light receiving element is arranged A coefficient calculation unit that calculates a mixing coefficient representing a ratio of the near-infrared signal to be mixed with a signal according to a signal amount of the near-infrared signal;
The plurality of types of color signals read from the reading unit and the near-infrared signal are mixed for each unit pixel based on the mixing coefficient calculated by the coefficient calculation unit, and the mixed signal is A luminance calculation unit for outputting as a luminance signal;
An image pickup apparatus comprising: a color component calculation unit that calculates and outputs a plurality of color component signals for each unit pixel using the plurality of types of color signals, the near-infrared signal, and the luminance signal. The focal length of the condensing lens formed on the light receiving element on which the near-infrared filter is arranged is shorter than the focal length of the condensing lens formed on the light receiving element on which the plurality of types of color filters are arranged. Item 2. The imaging device according to Item 1. The condenser lens is
The imaging device according to claim 2, wherein the imaging device has an effective refractive index distribution generated by a light transmission film having a concentric structure divided by a line width equal to or shorter than a wavelength of incident light. The refractive index of the condensing lens formed on the light receiving element on which the near-infrared filter is disposed is larger than the refractive index of the condensing lens formed on the light receiving element on which the plurality of types of color filters are disposed. The imaging device according to claim 1. The refractive index of the interlayer film formed on the light receiving element in which the near-infrared filter is disposed is larger than the refractive index of the interlayer film formed on the light receiving element in which the plurality of types of color filters are disposed. The imaging apparatus according to 1. A plurality of light receiving elements that are two-dimensionally arranged to photoelectrically convert incident light, and a plurality of types of color filters and near infrared light that are disposed on the plurality of light receiving elements and transmit visible light and near infrared light An image signal processing method performed in an imaging device including a near-infrared filter that transmits only light,
A readout step of reading out from the plurality of light receiving elements a plurality of types of color signals that have been photoelectrically converted through the color filter and a near-infrared signal that has been photoelectrically converted through the near-infrared filter;
The plurality of types read out in the readout step for each unit pixel composed of a light receiving element in which the near infrared filter is arranged and a light receiving element in which the color filter adjacent to the light receiving element is arranged. A coefficient calculating step for calculating a mixing coefficient representing a ratio of the near-infrared signal to be mixed with a color signal in accordance with a signal amount of the near-infrared signal;
The plurality of types of color signals read in the reading step and the near-infrared signal are mixed for each unit pixel based on the mixing coefficient calculated in the coefficient calculating step, and the mixing is performed. A luminance output step for outputting the signal as a luminance signal;
A color component that calculates a plurality of color component signals for each unit pixel and outputs the plurality of color component signals using the plurality of types of color signals, the near-infrared signal, the luminance signal, and the luminance signal An image signal processing method including an output step. In the coefficient calculation step,
Calculating the mixing coefficient so that the first region in the same frame image is a black and white image;
The image signal processing method according to claim 6, wherein the mixing coefficient is calculated so that a second area different from the first area is a color image. The plurality of types of color signals and the infrared signals each have normalized values;
The luminance signal output in the luminance output step is Y, the luminance signal read from the light receiving element having the color filter of red among the plurality of types of color signals (R + IR), and the color filter of green The luminance signal read from the light receiving element having (G + IR), the luminance signal read from the light receiving element having the blue color filter (B + IR), and read from the light receiving element having the near infrared filter If the luminance signal is IR and the mixing coefficient is k,
Y = (R + IR) + (G + IR) + (B + IR) + k × IR
Represented by
The image signal processing method according to claim 6 or 7, wherein, in the coefficient calculation step, the mixing coefficient is calculated at a value within a range of -3 to +1. In the color component output step,
The plurality of colors are adjusted by adjusting the component amounts of the near-infrared signal components included in the plurality of types of color signals and the near-infrared signal so as to maximize the gain of the red component signal among the plurality of color component signals. The image signal processing method according to claim 6, wherein a component signal is output. In the color component output step,
In order to maximize the gain of the red component signal of the plurality of color component signals, the gain of the red signal of the plurality of types of color signals is increased more than the gain of the other color signals, and the plurality of color components The image signal processing method according to claim 6, wherein a signal is output. In the coefficient calculating step, the mixing coefficient is calculated according to the signal amount of the near-infrared signal for every one pixel row in one frame,
In the luminance output step, the plurality of types of color signals and the near-infrared signal are mixed based on the mixing coefficient, and the mixed signal is output as a luminance signal.
In the color component output step, a color image is calculated by calculating a plurality of color component signals using the plurality of types of color signals, the near infrared signal, and the luminance signal, and outputting the plurality of color component signals. The image signal processing method according to claim 6, wherein a black-and-white image mainly including a luminance component by near-infrared light is switched and displayed in units of at least one line in one frame.

Images