WO2011001672A1

WO2011001672A1 - Imaging device and imaging method

Info

Publication number: WO2011001672A1
Application number: PCT/JP2010/004304
Authority: WO
Inventors: 田中圭介; 香山信三; 藤原一夫; 廣瀬裕
Original assignee: パナソニック株式会社
Priority date: 2009-06-30
Filing date: 2010-06-30
Publication date: 2011-01-06
Also published as: JP2011015087A

Abstract

An imaging device (100) is provided with: a plurality of color filters which include multiple types of filters (111, 112, 113) through which visible light passes and an IR pass filter (114) which blocks visible light and allows infrared light to pass through; a plurality of light-receiving elements which are each equipped with any one of the plurality of color filters and which are disposed in a two-dimensional arrangement; a read-out unit which reads out, from the plurality of light-receiving elements, multiple types of color signals corresponding to multiple colors of visible light, and an infrared signal; a luminance calculation unit (223) which uses a mixing coefficient indicating the mixing rate of the infrared signal to the multiple types of color signals to mix the multiple types of color signals with the infrared signal and output the result as a luminance signal; a color difference calculation unit (215) which uses the multiple types of color signals and the infrared signal to calculate and output a plurality of color component signals; and a k computing unit (222) which computes the mixing coefficient according to the signal quantity of the infrared signal.

Description

Imaging apparatus and imaging method

The present invention relates to an imaging device that receives visible light and near infrared light.

In recent years, for security, network, and in-vehicle applications, color images are captured using visible light as a light source during the day, near-infrared light is received at night, and a function that captures black-and-white images is used. The demand for equipment is growing rapidly.

In the prior art, this day / night function is used for solid-state imaging devices such as MOS sensors and charge coupled devices (CCD) in which color filters of different colors are mounted on two-dimensionally arranged light receiving elements in daytime imaging. By placing an IR cut filter that blocks near-infrared light in the front, a visible light color image is taken, and at night imaging, the IR cut filter is removed, and the signal of each pixel is used as a luminance signal. This was realized by capturing a monochrome image (Patent Document 1).

The first prior art requires a mechanical switching device that attaches and detaches an IR cut filter, which increases costs due to the increase in the number of parts, decreases the reliability due to fatigue of mechanical parts, and suddenly switches between day and night. There has been a problem of occurrence of a low quality image output period due to mismatch between the optical system and the signal processing system, for example, an exposure amount deviation. In order to solve such a problem, in the second prior art, an IR cut filter is not disposed in the optical system, and a plurality of visible light receiving elements mounted with different color filters and at least one near infrared light are provided. A dedicated light receiving element is used, and in the daytime, a visible color image is captured by subtracting the near infrared light component mixed in the signal of the visible light receiving element from the pure near infrared light component obtained by the dedicated light receiving element. Adopted a method of capturing a black and white image using a near-infrared signal incident on all pixels as a luminance signal (Patent Document 2). According to such a method, since the switching between the day and night imaging modes can be performed electronically, problems such as an increase in cost and a decrease in reliability associated with the switching of the mechanical switching device are solved.

This second prior art is a method that, once switched to the night mode, always forms a luminance signal based on near-infrared light and outputs only a monochrome image, so that a color image is obtained in the night mode. Is impossible.

Also, in Patent Document 3, by imaging an object illuminated with an external light source with R + IR, G + IR, or B + IR (monochrome CCD), the IR signal transmitted simultaneously with visible light is also used to improve the luminance signal. A light source device for an electronic endoscope and a signal processing method are described.

US Pat. No. 4,316,659 JP 2008-35090 A JP 2001-189926 A

However, for example, in an in-vehicle monitor camera, a weak color signal component using a streetlight or light leaked from a house as a light source can be used even at night, and an imaging apparatus that outputs such an image is strongly desired. It is rare.

An object of the present invention is to solve the problem of color image generation in the night mode that cannot be realized by the second prior art, and to provide an imaging apparatus and an imaging method for outputting a color image even in an environment where a visible light signal is weak There is to do.

In order to solve the above problems, an imaging device according to an embodiment of the present invention includes a plurality of colors including at least a plurality of types of color filters that transmit visible light and an infrared transmission filter that suppresses visible light and transmits infrared light. A plurality of light-receiving elements each mounted with a filter and any one of the plurality of color filters, and a plurality of light-receiving elements corresponding to visible light of the plurality of color filters from the plurality of light-receiving elements The plurality of types of color signals and the infrared signal are obtained by using a reading unit that reads out the types of color signals and infrared signals, and a mixing coefficient indicating a mixing ratio of the infrared signals with respect to the plurality of types of color signals. A luminance calculation unit that mixes and outputs as a luminance signal, a color component calculation unit that calculates and outputs a plurality of color component signals using the plurality of types of color signals and the infrared signal, and a signal of the infrared signal According to quantity And a coefficient calculating section for calculating the mixing coefficient Te.

According to this configuration, by calculating the mixing coefficient according to the signal amount of the infrared signal, the color component signal can be effectively left even if the visible light signal is weak, and the image can be colored. . In other words, the ratio of the infrared signal to the luminance signal can be adjusted according to the amount of infrared signal. For example, if the mixing coefficient is decreased (the ratio of the infrared signal to the luminance signal is decreased), the color component signal (for example, the color difference signal) is relatively increased, and the image can be made closer to color from black and white. Conversely, if the mixing coefficient is increased (if the ratio of the infrared signal in the luminance signal is increased), the color component signal (for example, the color difference signal) is relatively reduced, and the image can be made closer to color from black and white. As described above, a monochrome image, a color image close to black and white, an intermediate image between black and white and a color image, a black and white image close to color, and a color image can be generated according to the mixing coefficient.

Here, the mixing coefficient may take any one of three or more multi-stage values.

This configuration makes it easy to calculate the mixing coefficient because the mixing coefficient changes in a stepwise manner compared to a case where the mixing coefficient changes continuously in an analog manner.

Here, the color component calculation unit may generate two color difference signals as the color component signal using the luminance signal.

According to this configuration, it is possible to always output a color image composed of a luminance signal and two color difference signals both at night and in the daytime. The color image is colored according to the mixing coefficient. Of course, if only the luminance signal is used, the monochrome image can be used even in the daytime.

Here, the coefficient calculation unit may calculate the mixing coefficient according to a signal amount of the infrared signal with respect to the plurality of types of color signals.

According to this configuration, a more optimal mixing coefficient can be calculated as compared with the case where the mixing coefficient is calculated only from the signal amount of the infrared signal.

The plurality of types of color filters include a first color filter that transmits visible light and infrared light of a first color, and a second color filter that transmits visible light and infrared light of a second color. A third color filter that transmits visible light and infrared light of a third color, and the plurality of types of color signals correspond to the first, second, and third color filters. The luminance calculation unit calculates the luminance signal by adding the first, second, and third color signals and the infrared signal weighted by the mixing coefficient. It may be.

According to this configuration, since the first to third color filters receive not only visible light but also infrared light, the sensitivity of a black and white image by infrared light can be remarkably improved.

Here, each of the first, second, and third color signals and the infrared signal may have normalized values, and the mixing coefficient may be a value within a range of -3 to +1.

According to this configuration, the calculation of the luminance signal can be generated by a simple calculation.

Here, when the mixing coefficient is +1, the luminance calculation unit may output each of the first, second, and third color signals and the infrared signal as a luminance signal.

According to this configuration, it is possible to generate a high-resolution black-and-white image having the number of pixels twice as long as twice as large as that of a color image.

The plurality of types of color filters include a first color filter that transmits visible light of a first color and suppresses infrared light, and a first color filter that transmits visible light of a second color and suppresses infrared light. Including a two-color filter and a third color filter that transmits visible light of the third color and suppresses infrared light, and the plurality of types of color signals correspond to the first, second, and third color filters. The luminance calculation unit includes the first, second, and third color signals, and the luminance calculation unit adds the first, second, and third color signals and the infrared signal weighted by the coefficient to add the luminance. The signal may be calculated.

Here, each of the first, second, and third color signals and the infrared signal may have normalized values, and the mixing coefficient may be a value within a range of 0 to +1.

According to this configuration, the calculation formula of the luminance signal can be simplified.

Here, the imaging apparatus may further include an exposure control unit that controls the exposure amount of the plurality of light receiving elements in accordance with the signal amount of the infrared signal with respect to the plurality of types of color signals.

Here, the exposure control unit determines an exposure time of the plurality of light receiving elements according to a signal amount of the infrared signal with respect to the plurality of types of color signals, and exposes the plurality of light receiving elements. You may do it.

According to this configuration, it is possible to prevent a decrease in sensitivity by adjusting the exposure time.

Here, the imaging apparatus further controls the light emission amount of the infrared light source based on at least one of the signal amount of the plurality of types of color signals and the signal amount of the infrared signal, and an infrared light source that emits infrared light. You may make it provide a light source control part.

According to this configuration, it is possible to prevent a decrease in sensitivity by controlling the light emission amount of the infrared light source.

Here, the luminance calculation unit may calculate the mixing coefficient for each frame.

According to this configuration, an optimum mixing coefficient can be obtained for each frame.

Here, the luminance calculation unit is configured to mix the first predetermined number of consecutive frames into a monochrome image with respect to the first predetermined number of consecutive frames and the second predetermined number of consecutive frames that are alternately repeated. A coefficient may be calculated, and the mixing coefficient may be calculated so that the second predetermined number of consecutive frames is a color image.

According to this configuration, since the black and white image and the color image are alternately generated, an image having the advantages of both can be provided to the user.

Here, the luminance calculation unit calculates the mixing coefficient so that the first area in one frame is a black and white image, and a second area different from the first area in one frame is a color image. The mixing coefficient may be calculated as follows.

According to this configuration, since a monochrome image and a color image are generated within one frame, an image having the advantages of both can be provided to the user.

Here, the color component calculation unit may generate three primary color signals as the color component signals using the plurality of types of color signals.

According to this configuration, it is possible to simultaneously generate a monochrome image composed of luminance signals and a color image composed of three primary color signals.

Here, the luminance calculation unit further calculates a dummy luminance signal by mixing the plurality of types of color signals and the infrared signal using a dummy mixing coefficient for suppressing the infrared signal. The color component calculation unit may generate two color difference signals as the color component signal using the dummy luminance signal.

This configuration makes it possible to generate two color difference signals with good color reproducibility even when the mixing coefficient is large.

Here, the imaging apparatus further includes a color gain determination unit that determines a color gain for enhancing or suppressing a specific color according to the signal amount of the infrared signal, and the color component calculation unit according to the color gain. You may make it provide the gain adjustment part which adjusts the gain of the calculated several color component signal.

According to this configuration, it is possible to prevent the occurrence of a false color due to a specific color. Here, the false color refers to a phenomenon in which, for example, a leaf is not imaged in green.

An imaging method according to an aspect of the present invention includes a plurality of color filters including at least a plurality of color filters that transmit visible light, a plurality of color filters including an infrared transmission filter that suppresses visible light and transmits infrared light, and the plurality of color filters. An imaging method for generating an image composed of a luminance signal and a color component signal using a solid-state imaging device that includes each of the color filters and includes a plurality of light receiving elements arranged two-dimensionally. Then, a plurality of types of color signals and infrared signals corresponding to the plurality of colors of visible light are read from the plurality of light receiving elements, and the plurality of types of color signals are output according to the signal amount of the infrared signals. A mixing coefficient indicating a mixing ratio of the infrared signal is calculated, the plurality of types of color signals and the infrared signal are mixed using the mixing coefficient, and output as the luminance signal, and the plurality of types of colors Signal and front It calculates and outputs a plurality of said color component signals using the infrared signal.

According to this configuration, by calculating the mixing coefficient according to the signal amount of the infrared signal, it is possible to output the color image while effectively leaving the color component signal even if the visible light signal is weak. .

According to the present invention, by calculating the mixing coefficient according to the signal amount of the infrared signal, the color component signal can be effectively left even if the visible light signal is weak, and the image can be colored. . For example, it is possible to provide an imaging apparatus that outputs a visible color image even in a night mode in which the visible light signal is weak.

FIG. 1 is a block diagram illustrating a main functional configuration of the imaging apparatus according to the first embodiment. FIG. 2A is a flowchart illustrating an example of the mixing coefficient calculation process in the k calculation unit 222. FIG. 2B is a flowchart illustrating an example of the mixing coefficient calculation process in the k calculation unit 222. FIG. 3A is a chromaticity diagram of an image according to the prior art. FIG. 3B is a chromaticity diagram of an image in the first embodiment. FIG. 4 is a block diagram illustrating a main functional configuration of the imaging apparatus according to the second embodiment. FIG. 5A is a flowchart illustrating an example of the mixing coefficient calculation process of the luminance calculation unit 423. FIG. 5B is a flowchart illustrating color gain determination processing in the near-infrared signal control unit 221. FIG. 6 is a block diagram illustrating a main functional configuration of the imaging apparatus according to the third embodiment. FIG. 7 is a flowchart illustrating an example of the luminance signal calculation process of the luminance calculation unit 623. FIG. 8 is a block diagram illustrating a main functional configuration of the imaging apparatus according to the second embodiment. FIG. 9 is a block diagram illustrating a main functional configuration of an imaging apparatus according to a modification.

Hereinafter, embodiments of a solid-state imaging device according to the present invention will be described with reference to the drawings, taking a digital still camera as an example. In addition, although this invention is demonstrated using the following embodiment and attached drawing, this is for the purpose of illustration and this invention is not intended to be limited to these.

(Embodiment 1)
First, the configuration of the imaging device according to the embodiment of the present invention will be described. In the present embodiment, a plurality of types of color signals and infrared signals corresponding to a plurality of colors of visible light are read from the solid-state imaging device, and the red color corresponding to the plurality of types of color signals is read according to the signal amount of the infrared signal. A mixing coefficient indicating a mixing ratio of an external signal is calculated, and using the mixing coefficient, the plurality of types of color signals and the infrared signal are mixed and output as the luminance signal, and the plurality of types of color signals and An imaging apparatus that calculates and outputs a plurality of the color component signals using the infrared signal will be described. Accordingly, by calculating the mixing coefficient according to the signal amount of the infrared signal, it is possible to color the image while effectively leaving the color component signal even if the visible light signal is weak.

FIG. 1 is a block diagram illustrating a main functional configuration of the imaging apparatus according to the first embodiment.

As shown in FIG. 1, the imaging apparatus 100 according to the present embodiment includes an imaging element 101, a signal processing unit 201, an imaging control unit 301, a light source control unit 401, and an IR (infrared light) light source 411. A lens 501 and a diaphragm 502. The signal processing unit 201 includes red (R) + near infrared (IR), green (G) + near infrared (IR), blue (B) + near infrared (IR), and near infrared (IR). An image signal composed of a color signal and a luminance signal is processed for the image sensor 101 having the unit pixel 115 having one pixel as a unit.

The image sensor 101 includes an R + IR transmission filter 111, a G + IR transmission filter 112, a B + IR transmission filter 113, an IR transmission filter 114, a vertical shift register 121, a horizontal shift register 122, a noise removal circuit 123, and an output amplifier 124.

The signal processing unit 201 includes an OB calculation unit 211, a low-pass filter 212, a 4 × 4 matrix calculation unit 213, a white balance unit 214, a color difference calculation unit 215, a gain adjustment unit 216, a γ correction unit 217, and a near infrared signal control unit 221. , K calculation unit 222, and luminance calculation unit 223.

The imaging control unit 301 includes an exposure time control unit 311 and an aperture control unit 312.

The R + IR transmission filter 111, the G + IR transmission filter 112, and the B + IR transmission filter 113 include three types of filters that transmit visible light (any one of R, G, and B) and infrared light (IR). As an infrared transmission filter that suppresses visible light and transmits infrared light, the IR transmission filter 114 is provided as an infrared transmission filter that suppresses visible light and transmits infrared light. Four unit pixels 115 corresponding to the four types of filters correspond to one pixel in a color image or a monochrome image.

The lens 501 forms the incident light on the imaging region of the image sensor 101.

The image sensor 101 is a MOS type image sensor that photoelectrically converts incident light to generate a color signal.

The signal processing unit 201 is a DSP or the like that performs image signal processing on the four types of pixel signals received from the image sensor 101.

The image sensor 101 selects each row of the unit pixels 115 arranged two-dimensionally by the vertical shift register 121, selects the row signal by the horizontal shift register 122, and outputs a color signal for each unit pixel 115 from the output amplifier 124. To do. The vertical shift register 121, the horizontal shift register 122, and the selection circuit 125 function as a reading unit that reads a plurality of types of color signals and infrared signals corresponding to a plurality of colors of visible light from the plurality of unit pixels 115.

As shown in FIG. 1, the signal processing unit 201 generates pixel signals of red (R), green (G), and blue (B) filters from the four types of pixel signals output from the image sensor 101. One unit of near-infrared (IR) signal is generated for each unit pixel 115, and further, a color image composed of a luminance signal and two color difference signals is generated. The luminance signal alone represents a black and white image.

In order to generate an image signal of a filter pixel of red (R), green (G), and blue (B) from a pixel signal output from the image sensor 101, for example, in the case of red (R), visible light From the red (R) + near infrared (IR) output signal of the pixel having the R + IR transmission filter 111 for invisible light to the near infrared (IR) output signal of the pixel having a band filter for invisible light. By subtracting red (R), red (R) may be used as an image signal.

Also, if this pixel configuration is used, the red, green, and blue signals can be obtained by subtracting the IR signal component of the IR pixel from the original red, green, and blue signals including near-infrared light components in the daytime. Can be obtained. Thus, a mechanical IR cut filter is unnecessary even during the daytime.

On the other hand, near-infrared light images are obtained at night by using signals from all pixels. Therefore, the resolution of the IR image is higher than that of the RGB image in the daytime because the IR components of the R + IR pixel, the G + IR pixel, and the B + IR pixel are used.

In order to easily realize a pixel configuration having the R + IR transmission filter 111, the G + IR transmission filter 112, the B + IR transmission filter 113, or the IR transmission filter 114, a photonic crystal color filter was integrated on the image sensor. An optical multilayer film obtained by alternately laminating low refractive index and high refractive index materials has a forbidden band through which light does not pass. This time, the transmission band through which light is transmitted is designed to transmit near infrared light and used as an IR filter.

The R + IR transmission filter 111, the G + IR transmission filter 112, and the B + IR transmission filter 113 are formed on an optical multilayer film in which materials having a low refractive index and a high refractive index are alternately stacked with a film thickness of λ / 4 (λ: wavelength). Can be realized by introducing a “defect layer” different from λ / 4. Due to this defect layer, the optical periodicity is disturbed, and a transmission band can be generated in the forbidden band. By appropriately designing the thickness of the defect layer, a transmission band of a desired wavelength band can be realized. That is, R + IR pixels that transmit red light and near-infrared light, G + IR pixels that transmit green light and near-infrared light, B + IR pixels that transmit blue light and near-infrared light, and only near-infrared light transmit. A color filter having IR pixels can be easily designed. Simply adjusting the film thickness of each defect layer of the R + IR transmission filter 111, the G + IR transmission filter 112, and the B + IR transmission filter 113, red light and near infrared light, green light and near infrared light, blue light and near infrared light. A light transmission band can be formed.

The color signal of the unit pixel 115 on which the R + IR transmission filter 111, the G + IR transmission filter 112, the B + IR transmission filter 113, or the IR transmission filter 114 is mounted is output to the signal processing unit 201 by the output amplifier 124. The signal processing unit 201 performs offset removal such as dark current on the input image signal in the OB calculation unit 211. Subsequently, the noise signal is removed from the color signal by the low-pass filter 212.

The 4 × 4 matrix calculation unit 213 obtains R, G, B visible light color signals by the 4 × 4 matrix calculation unit 213 and obtains the IR signal multiplied by the specific gravity from the original signal of each pixel. The specific gravity coefficient is adjusted so that near infrared light disappears. That is, the 4 × 4 matrix calculation unit 213 outputs 3 signals of RGB by performing 4 × 4 matrix calculation on the (R + IR) signal, (G + IR) signal, (B + IR) signal, and IR signal.

The white balance unit 214 performs white balance and performs color correction between the three primary colors in the same manner as the normal three primary colors. Further, the color difference calculation unit 215 and the gain adjustment unit 216 perform signal processing. Finally, the color signal output from the γ correction unit 217 is output as an image signal to the display device or the like together with the luminance signal output from the luminance calculation unit 223.

The color difference calculation unit 215 calculates and outputs a plurality of color component signals (RY signal, BY signal) using a plurality of types of color signals (that is, R, G, B).

In this embodiment, a signal output from a light receiving element equipped with a color filter that passes only near-infrared light is used in an operation for creating a luminance signal. The luminance signal is calculated by luminance signal = (R + IR) pixel luminance signal + (G + IR) pixel luminance signal + (B + IR) pixel luminance signal + k × (IR) pixel luminance signal. That is, Y = (R + IR) + (G + IR) + (B + IR) + kIR. Here, k is a mixing coefficient indicating the mixing ratio of the IR signal with respect to the ((R + IR) + (G + IR) + (B + IR)) signal or a mixing coefficient indicating the mixing ratio of the IR signal with respect to the luminance signal Y. In this case, k is determined within a range of −3 to +1.

When k = -3, the luminance signal Y from which the IR signal is completely removed can be obtained. This is suitable for imaging in the daytime mode. When k = + 1, the luminance signal Y including the entire IR signal can be obtained. This is suitable for generating black and white images. When k is an intermediate value between -3 and +1, a color image is generated, and the degree of colorization is determined according to k. This is suitable for colorizing in the night mode.

The k calculation unit 222 calculates a mixing coefficient according to the signal amount of the IR signal.

The luminance calculation unit 223 mixes a plurality of types of color signals and the infrared signal using the mixing coefficient k, and outputs the result as a luminance signal Y.

FIG. 2A is a flowchart showing an example of the mixing coefficient calculation process in the k calculation unit 222. In the figure, Ave (IR) is an average value of IR signals in the previous frame, and Th1 to Th4 are thresholds satisfying Th1> Th2> Th3> Th4. As shown in the figure, the k calculator 222 sets the mixing coefficient k = k1 when AVE (IR) is larger than the threshold value Th1 (S21, S22), and the AVE (IR) is equal to or lower than the threshold value Th1. When the value Th2 is larger than the value Th2, the mixing coefficient k = k2 is set (S23, S24). When AVE (IR) is less than the threshold Th2 and larger than the threshold Th3, the mixing coefficient k = k3 is set (S25, S26). , AVE (IR) is set to the mixing coefficient k = k4 when the threshold value Th3 is equal to or smaller than the threshold value Th3 (S27, S28), and when AVE (IR) is equal to or smaller than the threshold value Th4 (S29). Here, k1, k2, k3, and k4 are, for example, +1, 0, -1, -2, and -3.

As described above, the k calculation unit 222 calculates the mixing coefficient k according to the signal amount of the IR signal.

Furthermore, a near-infrared signal control unit 221 that controls the component amount of the near-infrared signal used for calculation to a predetermined value (for each frame) is provided.

The near-infrared signal control unit 221 can control the imaging control unit 301 in accordance with the component amount of the near-infrared signal used for the luminance signal calculation, and can control the exposure amount of the imaging element 101 to a predetermined value. .

In order to adjust the exposure amount of the image sensor 101 to a predetermined value, the near infrared signal control unit 221 controls the exposure time control unit 311 to adjust the exposure time to a predetermined value.

Further, in order to adjust the exposure amount of the image sensor 101 to a predetermined value, the near-infrared signal control unit 221 controls the aperture control unit 312 and adjusts the aperture 502 of the lens 501 to adjust the incident light amount to a predetermined value. To do.

In order to adjust the exposure amount of the image sensor 101 to a predetermined value, the near-infrared signal control unit 221 controls the light source control unit 401 to change the irradiation amount of the IR light source 411 that irradiates the subject with predetermined near-infrared light. adjust.

The near-infrared signal control unit 221 adjusts the gain by controlling the gain adjustment unit 216 according to the output of the white balance unit 214 for the component amount of the near-infrared signal component applied to the color signal component. In particular, in order to obtain red color reproducibility such as signs required for in-vehicle cameras and security cameras, the component amount of the near-infrared signal component is controlled so as to maximize the gain of the red signal. In addition, the signal processing unit 201 controls the step of generating a color reproduction signal including only a visible light component excluding the near-infrared signal component of each light receiving element and the maximum gain for the red signal component.

Also, the near infrared signal control unit 221 controls the amount of near infrared light component applied to the luminance signal according to the intensity of the color reproduction signal.

In addition, the signal processing unit 201 controls the amount of near-infrared light component applied to the luminance signal by the near-infrared signal control unit 221 for each frame, so that the luminance component due to the color image frame and near-infrared light is mainly used. The black-and-white image frame is switched in units of at least one frame.

The signal processing unit 201 controls the near-infrared light amount component applied to the luminance signal by the near-infrared signal control unit 221 for each pixel, so that the near-red and red pixels that output a color image in the same frame are controlled. It is characterized in that pixels for outputting a black and white image mainly including a luminance component due to external light are switched in units of at least one pixel.

Therefore, when the imaging apparatus 100 according to the first embodiment is used, the imaging device that captures an image by receiving near-infrared light at night, such as a day / night camera, can be used even in the night mode in which the visible light signal is weak. As shown in the chromaticity diagram of the image subjected to the color signal processing shown in 3B, an imaging device that outputs a visible color image can be realized. FIG. 3A shows a chromaticity diagram of a black-and-white image (infrared image) in night mode in the prior art.

2A shows an example in which the k calculation unit 222 calculates the mixing coefficient k in accordance with the average signal amount of the IR signal, it may be as shown in FIG. 2B instead of FIG. 2A. 2B shows an example in which the k calculation unit 222 calculates the mixing coefficient in accordance with the average signal amount of the IR signal with respect to a plurality of types of color signals (R signal, G signal, and B signal).

In FIG. 2B, Ave (IR / (R + G + B)) is an average value of the ratio of the IR signal to the (R + G + B) signal in the previous frame, and Th11 to Th14 are threshold values satisfying Th11> Th12> Th13> Th14. It is.

In FIG. 2B, the k calculation unit 222 sets the mixing coefficient k = k11 when AVE (IR / (R + G + B)) is larger than the threshold Th11 (S51, S52), and the AVE (IR / (R + G + B)) is the threshold. When the value is less than Th11 and greater than the threshold value Th12, the mixing coefficient k is set to k2 (S53, S54), and when AVE (IR / (R + G + B)) is less than the threshold value Th12 and greater than the threshold value Th13, the mixing coefficient k is set. = K13 (S55, S56), and when AVE (IR / (R + G + B)) is less than or equal to the threshold Th13 and greater than the threshold Th14, the mixing coefficient k is set to k14 (S57, S58), and AVE (IR When (/ (R + G + B)) is equal to or less than the threshold Th14, the mixing coefficient k is set to k15 (S29). Here, k11, k12, k13, and k14 are, for example, +1, 0, -1, -2, and -3.

2A and 2B show an example in which the mixing coefficient k can take a value of 5 steps, but any number of steps may be used as long as it is 3 steps or more, or a continuous value may be used.

(Embodiment 2)
In this embodiment, first, the luminance calculation unit further mixes a plurality of types of color signals and the infrared signal by using a dummy mixing coefficient kd for suppressing the infrared signal, thereby performing a dummy. An example in which the luminance signal Yd is calculated and the color difference calculation unit 215 generates two color difference signals as the color component signals using the dummy luminance signal Yd will be described.

Second, the luminance calculation unit performs the mixing so that the first predetermined number of consecutive frames become a monochrome image with respect to the first predetermined number of consecutive frames and the second predetermined number of consecutive frames that are alternately repeated. A configuration for calculating a coefficient and calculating the mixing coefficient so that the second predetermined number of consecutive frames becomes a color image will be described.

Third, the near-infrared signal control unit 221 determines a color gain for enhancing or suppressing the specific color according to the signal amount of the infrared signal, and the gain adjustment unit 216 is calculated by 2215 according to the color gain. An example of adjusting the gains of a plurality of color component signals will be described.

FIG. 4 is a block diagram illustrating a main functional configuration of the imaging apparatus according to the second embodiment. This figure is different from FIG. 1 in that a luminance calculation unit 423 is provided instead of the luminance calculation unit 223. Hereinafter, the description of the same points will be omitted, and different points will be mainly described.

The luminance calculation unit 423 is different from the function of the luminance calculation unit 223 in that it calculates a dummy luminance signal Yd and supplies the calculated dummy luminance signal Yd to the color difference calculation unit 215 as the luminance signal Y. Here, the dummy luminance signal is a luminance signal in which the IR signal is completely suppressed, and is the luminance signal Y when k = −3.

Thereby, the color difference calculation unit 215 calculates the color difference signal when the IR signal is completely suppressed even when the mixing coefficient is large. Thereby, two color difference signals with good color reproducibility can be generated.

Further, the luminance calculation unit 423 is the same in that k is calculated for each frame as compared with the luminance calculation unit 223, but the mixing coefficient is calculated so that the first predetermined number of continuous frames become a monochrome image. The mixing coefficient is calculated so that the second predetermined number of consecutive frames become a color image.

FIG. 5A shows a flowchart showing an example of the mixing coefficient calculation process of the luminance calculation unit 423. In FIG. 5A, the luminance calculation unit 423 calculates k for the first predetermined number of consecutive frames in the loop 1 as in FIG. 2A or 2B (S51 to S53). As a result, the first predetermined number of consecutive frames becomes a color image.

Also, the luminance calculation unit 423 determines k = 1 for the second predetermined number of consecutive frames in the loop 2 (S54 to S55). As a result, the second predetermined number of consecutive frames become a monochrome image.

(The first predetermined number and the second predetermined number) are both integers of 1 or more, and may be (1, 1), (1, 2), (2, 1), for example.

In this way, since the black and white image and the color image are alternately generated, an image having the advantages of both can be provided to the user.

Compared to Embodiment 1, the near infrared signal control unit 221 has a function of determining a color gain for enhancing or suppressing a specific color according to the signal amount of the infrared signal.

The gain adjusting unit 216 is configured to adjust the gains of a plurality of color component signals calculated by 2215 according to the color gain.

FIG. 5B is a flowchart showing color gain determination processing in the near-infrared signal control unit 221.

As shown in the figure, the near-infrared signal control unit 221 receives a specific color enhancement instruction based on a user operation from the outside (S61, S63, S65), and performs a process of relatively increasing the gain for the received color. (S62, S64, S66).

In this way, it is possible to prevent the occurrence of a false color due to the specific color. Here, the false color refers to a phenomenon in which, for example, a leaf is not imaged in green.

It should be noted that the specific color may be suppressed instead of emphasizing the specific color. In this case, the gain of the specific color may be relatively decreased.

(Embodiment 3)
In this embodiment, an imaging apparatus that outputs a black and white image at a higher resolution than a color image will be described.

FIG. 6 is a block diagram illustrating a main functional configuration of the imaging apparatus according to the third embodiment. This figure is different from FIG. 1 in that a luminance calculation unit 623 is provided instead of the luminance calculation unit 223. Hereinafter, the description of the same points will be omitted, and different points will be mainly described.

In addition to the function of the luminance calculation unit 223, the luminance calculation unit 623 performs the first, second, and third color signals (R + IR signal, G + IR signal, B + IR signal), and IR signal when the mixing coefficient k is +1. Each is output as a luminance signal.

FIG. 7 is a flowchart showing an example of the luminance signal calculation process of the luminance calculation unit 623. As shown in the figure, the luminance calculation unit 623 calculates the luminance signal Y (Y = (R + IR) + (G + IR) + (B + IR) + kIR) and outputs it to the color difference calculation unit 215 (S70), and the k calculation unit 222. If k = 1 calculated by the above, Y is output as a luminance signal (S71, S73). On the other hand, if k = 1 is not satisfied, a monochrome image having four times the number of pixels is generated (y = R + IR, y = G + IR, y = B + IR, y = IR) (S73), and y is output as a luminance signal (S74).

This makes it possible to generate a high-resolution black-and-white image having twice as many pixels as twice as long as a color image.

(Embodiment 4)
In the present embodiment, in addition to the function of the imaging device of the first embodiment, the light emission amount of the infrared light source is controlled based on at least one of the signal amount of the plurality of types of color signals and the signal amount of the infrared signal. The configuration will be described.

FIG. 8 is a block diagram showing the main functional configuration of the imaging apparatus according to the second embodiment.

As shown in FIG. 8, the imaging apparatus 100 according to the present embodiment includes an imaging element 101, a signal processing unit 201, an imaging control unit 301, a light source control unit 401, an IR light source 411, a lens 501, and a diaphragm 502. I have. The signal processing method according to the present embodiment includes red (R) + near infrared (IR), green (G) + near infrared (IR), blue (B) + near infrared (IR), and near infrared. An image signal composed of a color signal and a luminance signal is processed with respect to the imaging apparatus 100 having the unit pixel 115 having (IR) four pixels as one unit.

Hereinafter, the configuration and signal processing method different from the embodiment will be described.

In order to adjust the exposure amount of the image sensor 101 to a predetermined value, the luminance calculation unit 223 controls the light source control unit 401 corresponding to the visible light intensity, and the IR light source 411 that irradiates the subject with predetermined near infrared light. Adjust the dose.

Therefore, when the imaging apparatus 100 according to the second embodiment is used, the imaging device that captures an image by receiving near-infrared light at night, such as a day / night camera, can be used even in the night mode in which the visible light signal is weak. As shown in the chromaticity diagram of the image subjected to the color signal processing shown in FIG. 2, an imaging device that outputs a visible color image can be realized.

In addition, as shown in FIG. 9, it is good also as an imaging device which produces | generates three primary color signals as said color component signal.

In this way, it is possible to simultaneously generate a black and white image made up of luminance signals and a color image made up of three primary color signals.

The imaging apparatus of the present invention can be used for surveillance cameras, network cameras, vehicle-mounted cameras, digital cameras, mobile phones, and the like, and can improve the image quality of captured images at night of these devices.

DESCRIPTION OF SYMBOLS 100 Image pick-up device 101 Image pick-up element 111 R + IR transmission filter 112 G + IR transmission filter 113 B + IR transmission filter 114 IR transmission filter 115 Unit pixel 121 Vertical shift register 122 Horizontal shift register 123 Noise removal circuit 124 Output amplifier 201 Signal processing section 211 OB calculation section 212 Low pass Filter 213 4 × 4 matrix calculation unit 214 White balance unit 215 Color difference calculation unit 216 Gain adjustment unit 217 γ correction unit 221 Near infrared signal control unit 222

k calculation unit

223, 423, 623 Brightness calculation unit 301 Imaging control unit 311 Exposure time Control unit 312 Aperture control unit 401 Light source control unit 411 IR light source 501 Lens 502 Aperture

Claims

A plurality of color filters including at least visible light and a plurality of color filters including an infrared transmission filter that suppresses visible light and transmits infrared light;
A plurality of light receiving elements each mounted with any one of the plurality of color filters and arranged two-dimensionally;
A readout unit that reads out a plurality of types of color signals corresponding to visible light of the plurality of color filters, and an infrared signal from the plurality of light receiving elements,
Using a mixing coefficient indicating a mixing ratio of the infrared signal with respect to the plurality of types of color signals, the plurality of types of color signals and the infrared signal are mixed and output as a luminance signal;
A color component calculator that calculates and outputs a plurality of color component signals using the plurality of types of color signals and the infrared signal;
An imaging apparatus comprising: a coefficient calculation unit that calculates the mixing coefficient according to a signal amount of the infrared signal.
The imaging apparatus according to claim 1, wherein the mixing coefficient takes any one of three or more multi-stage values.
The imaging apparatus according to claim 2, wherein the color component calculation unit generates two color difference signals as the color component signal using the luminance signal.
The imaging apparatus according to claim 3, wherein the coefficient calculation unit calculates the mixing coefficient according to a signal amount of the infrared signal with respect to the plurality of types of color signals.
The plurality of types of color filters include a first color filter that transmits visible light and infrared light of a first color, a second color filter that transmits visible light and infrared light of a second color, and a third color filter. A third color filter that transmits visible light and infrared light of the color
The plurality of types of color signals include first, second and third color signals corresponding to the first, second and third color filters,
The imaging apparatus according to claim 3, wherein the luminance calculation unit calculates the luminance signal by adding the infrared signals weighted by the first, second, and third color signals and the mixing coefficient.
The first, second and third color signals and the infrared signal each have normalized values;
The imaging apparatus according to claim 5, wherein the mixing coefficient is a value within a range of −3 to +1.
The imaging device according to claim 6, wherein the luminance calculation unit further outputs each of the first, second, and third color signals and the infrared signal as luminance signals when the mixing coefficient is +1.
The plurality of types of color filters include a first color filter that transmits visible light of a first color and suppresses infrared light, and a second color filter that transmits visible light of a second color and suppresses infrared light. And a third color filter that transmits visible light of the third color and suppresses infrared light,
The plurality of types of color signals include first, second and third color signals corresponding to the first, second and third color filters,
The imaging apparatus according to claim 3, wherein the luminance calculation unit calculates the luminance signal by adding the first, second, and third color signals and the infrared signal weighted by the coefficient.
The first, second and third color signals and the infrared signal each have normalized values;
The imaging apparatus according to claim 8, wherein the mixing coefficient is a value within a range of 0 to +1.
The imaging device further includes:
The imaging apparatus according to claim 3, further comprising an exposure control unit that controls an exposure amount of the plurality of light receiving elements in accordance with a signal amount of the infrared signal with respect to the plurality of types of color signals.
The exposure control unit
The imaging apparatus according to claim 10, wherein an exposure time of the plurality of light receiving elements is determined according to a signal amount of the infrared signal with respect to the plurality of kinds of color signals, and the plurality of light receiving elements are exposed.
The imaging device further includes:
An infrared light source that emits infrared light;
The imaging apparatus according to claim 1, further comprising: a light source control unit that controls a light emission amount of the infrared light source based on at least one of a signal amount of the plurality of types of color signals and a signal amount of the infrared signal.
The imaging apparatus according to claim 2, wherein the luminance calculation unit calculates the mixing coefficient for each frame.
The luminance calculation unit calculates the mixing coefficient for a first predetermined number of consecutive frames and a second predetermined number of consecutive frames that are alternately repeated so that the first predetermined number of consecutive frames become a black and white image. The imaging apparatus according to claim 13, wherein the mixing coefficient is calculated so that the second predetermined number of consecutive frames become a color image.
The luminance calculation unit calculates the mixing coefficient so that the first area in one frame becomes a monochrome image, and the second area different from the first area in one frame becomes a color image. The imaging apparatus according to claim 2, wherein the mixing coefficient is calculated.
The imaging apparatus according to claim 2, wherein the color component calculation unit generates three primary color signals as the color component signals using the plurality of types of color signals.
The luminance calculation unit further mixes the plurality of types of color signals and the infrared signal using a dummy mixing coefficient for suppressing the infrared signal, and calculates a dummy luminance signal,
The imaging apparatus according to claim 2, wherein the color component calculation unit generates two color difference signals as the color component signal using the dummy luminance signal.
The imaging device further includes:
A color gain determination unit that determines a color gain for enhancing or suppressing a specific color according to the signal amount of the infrared signal;
The imaging apparatus according to claim 2, further comprising: a gain adjustment unit that adjusts gains of a plurality of color component signals calculated by the color component calculation unit according to the color gain.
A plurality of color filters that transmit at least visible light, a plurality of color filters including an infrared transmission filter that suppresses visible light and transmits infrared light, and any one of the plurality of color filters are mounted. An imaging method for generating an image composed of a luminance signal and a color component signal using a solid-state imaging device including a plurality of light receiving elements arranged two-dimensionally,
A plurality of types of color signals corresponding to the plurality of colors of visible light and infrared signals are read from the plurality of light receiving elements,
According to the signal amount of the infrared signal, a mixing coefficient indicating a mixing ratio of the infrared signal with respect to the plurality of types of color signals is calculated,
Using the mixing coefficient, the plurality of types of color signals and the infrared signal are mixed and output as the luminance signal,
An imaging method for calculating and outputting a plurality of color component signals using the plurality of types of color signals and the infrared signal.