JP5228717B2 - Image input device - Google Patents

Description

  The present invention relates to an image input device that performs image processing on an image captured by an image sensor.

In recent years, automobiles equipped with an image sensor capable of capturing color image data are known. Further, as a prior art related to the present invention, Patent Document 1 discloses that visible image data including R, G, and B color components is extracted from an image captured by a pixel including R, G, and B filters. An image extraction means 120; an infrared image extraction means 130 for extracting infrared image data from an image captured by a pixel having an Ir filter; and HSV conversion of the visible image data to extract first luminance information; And luminance information extraction means 140 for extracting the second luminance information from the infrared image data, weighting the first luminance information with the weighting factor w1, weighting the second luminance information with the weighting factor w2 (w1 + w2 = 1), Pseudo color image generation means 170 for generating a pseudo color image, w1 = 1 on a clear day, w1 = 0 on dark at night, and 1>w1> 0 in an intermediate situation. Color image reproducing apparatus is disclosed.
JP 2007-184805 A

By the way, since there is a sufficient amount of visible light in the daytime, there is sufficient color information for white balance correction. Meanwhile, in the night, are mostly human engineering light, for greater high illumination of saturation, such as billboards, image sensor, it is difficult to receive the white light suitable for carrying out the white balance correction, nighttime In this case, it is difficult to perform accurate white balance correction.

In particular, when mounting the image sensor in an automobile, situations many occurs no human engineering light to the outside at night, this case, only the light of a headlight of a car becomes illumination light, the imaging sensor is sufficient light amount from the object Cannot be received, and it is difficult to perform accurate white balance correction.

  Furthermore, Patent Document 1 has no description about performing highly accurate white balance correction at night.

  An object of the present invention is to provide an image input device capable of performing highly accurate white balance correction even in an environment where the amount of illumination light is low.

(1) An image input device according to an aspect of the present invention includes an imaging device, a color signal generation unit that generates a color signal from image data captured by the imaging device, and a brightness detection unit that detects the brightness of the subject. When the brightness of the subject detected by the brightness detection unit falls below a predetermined reference value, the color signal is converted into white using a predetermined coefficient to reproduce white light from a predetermined artificial light source. A white balance correction unit that performs balance correction, the artificial light source is a headlight of an automobile, and the white balance correction unit is configured such that when the brightness detected by the brightness detection unit exceeds the reference value, based on the image data imaged by the imaging device, and calculates the white balance correction value, based on the white balance correction value, the white balance correcting the image data And features.

According to this configuration, when the brightness of the subject falls below a predetermined reference value, the color signal is subjected to white balance correction using a predetermined coefficient in order to reproduce white light from the predetermined artificial light source. Therefore, highly accurate white balance correction can be performed even in an environment where a sufficient amount of white light cannot be obtained for white balance correction.
Further, according to this configuration, when the image input device is mounted on an automobile, white balance correction with high accuracy can be performed at night.
Further, according to this configuration, it is possible to perform highly accurate white balance correction even in the daytime.

( 2 ) It is preferable that the brightness detection unit is configured by an exposure control unit that detects the brightness of a subject and sets an exposure time of the image sensor according to the detected brightness.

  According to this configuration, it is possible to detect the brightness without separately providing a sensor for detecting the brightness of the subject.

( 3 ) It is preferable that the said image pick-up element is comprised by the pixel provided with an infrared filter, the pixel which is not provided with a filter, and the pixel provided with a color filter.

  According to this configuration, an infrared image component can be imaged and a color image component can be imaged.

( 4 ) The image pickup element includes a first pixel, a second pixel, a third pixel, and a fourth pixel having a visible wavelength region and an infrared wavelength region as a sensitive wavelength band. Are arranged in a matrix, and the first pixel includes a first color filter that transmits light in the sensitive wavelength band excluding the blue region of the visible wavelength region, and the second pixel includes the visible pixel. A second color filter that transmits light in the sensitive wavelength band excluding a blue region and a green region of the wavelength region; and the third pixel transmits light in the sensitive wavelength band excluding the visible wavelength region. Preferably, the fourth pixel is not provided with a filter.

  According to this configuration, since all the pixels have sensitivity in the infrared wavelength region, an infrared image component having a high S / N ratio can be obtained.

  According to the present invention, highly accurate white balance correction can be performed even in an environment where the amount of illumination light is low.

  Hereinafter, an image input apparatus 1 according to an embodiment of the present invention will be described. FIG. 1 shows a block diagram of an image input apparatus 1 according to an embodiment of the present invention. As shown in FIG. 1, the image input apparatus 1 includes a lens 2, an image sensor 3, an image processing unit 4, a control unit 5, and an exposure control unit 6. Here, the image input device 1 is mounted on, for example, an automobile and images a subject around the automobile.

  The lens 2 includes an optical lens system that captures a light image of a subject and guides it to the image sensor 3. As the optical lens system, for example, a zoom lens, a focus lens, other fixed lens blocks, and the like arranged in series along the optical axis L of the optical image of the subject can be employed. The lens 2 includes a diaphragm (not shown) for adjusting the amount of transmitted light, a shutter (not shown), and the like, and the driving of the diaphragm and the shutter is controlled under the control of the control unit 5.

  The image pickup device 3 includes a light receiving unit composed of a PD (photodiode), an output circuit that outputs a signal photoelectrically converted by the light receiving unit, and a drive circuit that drives the image pickup device 3, and has a level corresponding to the amount of light. Generate image data. Here, as the imaging device 3, various imaging sensors such as a CMOS image sensor, a VMIS image sensor, and a CCD image sensor can be employed.

  In the present embodiment, the image sensor 3 captures a visible color image component with a pixel including a color filter, captures an infrared image component DBlk with a pixel including an infrared filter, and visualizes with a pixel not including the filter. The luminance image component DW including the luminance image component and the infrared image component DBlk is imaged.

  The image processing unit 4 includes an arithmetic circuit that performs an operation on the image signal, a memory that stores the image signal, and the like. The image data output from the image sensor 3 is A / D converted into a digital signal to be described later. After executing the processing, the data is output to a memory or a display device (not shown), for example.

  The control unit 5 includes a CPU and a memory that stores a program executed by the CPU, and outputs a control signal for controlling the image sensor 3 and the image processing unit 4 in response to a control signal from the outside.

  FIG. 2 is a diagram illustrating an arrangement of pixels of the image sensor 3. As shown in FIG. 2, the imaging device 3 includes a ye pixel (an example of a first pixel), an R pixel (an example of a second pixel), a Blk having a visible wavelength region and an infrared wavelength region as sensitive wavelength bands. Unit pixel portions 31 including pixels (an example of a third pixel) and W pixels (an example of a fourth pixel) are arranged in a matrix.

  In the case of FIG. 2, in the unit pixel unit 31, R pixels are arranged in the first row and first column, Blk pixels are arranged in the second row and 21st column, and W pixels are arranged in the first row and second column, R pixels, Blk pixels, W pixels, and ye pixels are arranged in a staggered manner, such that ye pixels are arranged in the second row and the second column. However, this is only an example, and the R pixel, Blk pixel, W pixel, and ye pixel may be arranged in a staggered pattern in another pattern.

  Since the ye pixel includes a ye filter (an example of a first color filter), a visible color image component of ye (hereinafter referred to as “ye image component”) is imaged, and the R pixel is an R filter (second filter). Since an R visible color image component (hereinafter referred to as “R image component”) is imaged and a Blk pixel has a Blk filter (an example of an infrared filter). Since the infrared image component DBlk is imaged and the W pixel does not include a filter, the luminance image component DW including the visible luminance image component and the infrared image component DBlk is imaged.

  FIG. 3 is a diagram showing the spectral transmission characteristics of the ye, R, and Blk filters. The vertical axis shows the transmittance (sensitivity), and the horizontal axis shows the wavelength (nm). A graph indicated by a dotted line shows the spectral sensitivity characteristics of the pixel in a state where the filter is removed. It can be seen that this spectral sensitivity characteristic has a peak near 600 nm and changes in a convex curve. In FIG. 3, 400 nm to 640 nm is a visible wavelength region, 640 nm to 1100 nm is an infrared wavelength region, and 400 nm to 1100 nm is a sensitive wavelength band.

  As shown in FIG. 3, the ye filter has a characteristic of transmitting light in the sensitive wavelength band excluding the blue region of the visible wavelength region. Therefore, the ye filter mainly transmits yellow light and infrared light.

  The R filter has a characteristic of transmitting light in a sensitive wavelength band excluding a blue region and a green region in the visible wavelength region. Therefore, the R filter mainly transmits red light and infrared light.

  The Blk filter has a characteristic of transmitting light in a sensitive wavelength band excluding the visible wavelength region, that is, in the infrared wavelength band. W indicates a case where no filter is provided, and all light in the sensitive wavelength band of the pixel is transmitted.

  Returning to FIG. 1, the exposure control unit 6 controls the exposure time of the image sensor 3 in accordance with the brightness of the subject. Here, the exposure control unit 6 detects the brightness of the subject using the luminance image component DW imaged by the image sensor 3 so that the exposure time is shorter as the subject is brighter and the exposure time is longer as the subject is darker. Set the exposure time. Specifically, the exposure control unit 6 determines the pixel value of the W pixel, that is, the luminance image component DW in a local region composed of pixels in a predetermined row × predetermined column in one frame of image data captured by the image sensor 3. The sum of the pixel values is calculated as a brightness value indicating the brightness of the subject. The exposure control unit 6 stores in advance a lookup table or function that defines the relationship between the brightness value and the exposure time of the image sensor 3. Then, the exposure control unit 6 determines an exposure time corresponding to the brightness value using this lookup table or function.

  FIG. 4 shows a block diagram of the image processing unit 4, the control unit 5, and the exposure control unit 6. The image processing unit 4 includes an evaluation unit 41, an exposure correction unit 42, a color interpolation unit 43, a color signal generation unit 44, an evaluation value calculation unit 45, a WB (white balance) correction unit 46, a color signal synthesis unit 47, and a color space. A conversion unit 48 is provided. The control unit 5 includes a coefficient determination unit 51, an exposure correction value calculation unit 52, and a WB correction value calculation unit 53.

  The evaluation unit 41 evaluates the size of the infrared image component DBlk in the luminance image component DW. Specifically, the evaluation unit 41 has a pixel value of W pixels, that is, a pixel of the luminance image component DW, in a local region composed of pixels in a predetermined row × predetermined column in one frame of image data captured by the image sensor 3. The evaluation value eDW is calculated by adding the values, and the pixel value of the Blk pixel, that is, the pixel value of the infrared image component DBlk is added to calculate the evaluation value eDBlk, and the value obtained by subtracting the evaluation value eDBlk from the evaluation value eDW Is calculated as an evaluation value e (= eDW−eDBlk). Then, the evaluation unit 41 evaluates the size of the infrared image component DBlk in the luminance image component DW based on the calculated evaluation value e. As the evaluation value e increases, the intensity of infrared light with respect to visible light decreases.

  The evaluation unit 41 employs eDW-eDBlk as the evaluation value e, but any value may be employed as long as the value indicates the size of the infrared image component DBlk in the luminance image component DW. For example, eDBlk / eDW may be adopted as the evaluation value e.

  The coefficient determination unit 51 evaluates a weighting coefficient k having a predetermined characteristic that increases as the evaluation value e increases, and a weighting coefficient kw that has a predetermined characteristic that decreases as the evaluation value e increases. The evaluation value e calculated by the unit 41 is used for determination.

  FIG. 5 is a graph showing the characteristics of the weighting factors k and kw. The vertical axis represents the weighting factors k and kw, and the horizontal axis represents the evaluation value e (= eDW−eDBlk). As shown in FIG. 5, the weighting factor k has a characteristic that increases linearly as the evaluation value e increases, and the weighting factor kw has a characteristic that decreases linearly as the evaluation value e increases, for example. I understand.

  In FIG. 5, experimentally obtained values are used as the slopes of the straight lines indicating the characteristics of the weighting factors k and kw. Specifically, the weighting factors k and kw have a relationship of k + kw = 1, and the slope of the straight line indicating the characteristics of the weighting factors k and kw is the maximum value of the evaluation value e for which the evaluation value e is predetermined. K = 1 and kw = 0, and k = 0 and kw = 1 when the evaluation value e indicates a predetermined minimum value of the evaluation value e.

  The exposure correction value calculation unit 52 calculates the luminance evaluation value edY by substituting the weighting coefficients k and kw determined by the coefficient determination unit 51 into the following equation, and divides the luminance evaluation value edY by a predetermined value. An exposure correction value H1 is calculated and output to the exposure correction unit 42. Here, as the predetermined value, the total number of pixels of the DBlk pixel and the DW pixel in the above-described local region can be employed.

edY = kw × eDW + k × (eDW−eDBlk)
The exposure correction unit 42 multiplies the exposure correction value H1 on each pixel of the ye image component Dye, R image component DR, infrared image component DBlk, and luminance image component DW captured by the image sensor 3, The exposure correction is performed on the ye image component Dye, the R image component DR, the infrared image component DBlk, and the luminance image component DW.

  The color interpolation unit 43 interpolates missing pixels of R pixels, Blk pixels, W pixels, and ye pixels arranged in a staggered manner, so that an R image component DR, an infrared image component DBlk, a luminance image component DW, and Each of the ye image components Dye is subjected to interpolation processing to generate an R image component DR, an infrared image component DBlk, a luminance image component DW, and a ye image component Dye having the same number of pixels as the number of pixels of the image sensor 3. Here, as the interpolation processing, for example, linear interpolation processing can be adopted.

  The color signal generation unit 44 combines the ye image component Dye, the R image component DR, the infrared image component DBlk, and the luminance image component DW that have been subjected to the interpolation processing by the color interpolation unit 43, using Expression (1). Thus, R (red), G (green), and B (blue) color signals dR, dG, and dB of each pixel are generated.

dR = DR-DBlk
dG = Dye-DR (1)
dB = DW-Dye
The evaluation value calculation unit 45 adds the values of the color signals dR, dG, and dB of each pixel in the local region described above, adds the value Rave of the color signal dR, the addition value Gave of the color signal dG, and the color signal dB. Is added to the controller 5 and output to the controller 5.

  When the brightness value calculated by the exposure control unit 6 is lower than a predetermined reference value, the WB correction value calculation unit 53 has predetermined coefficients ξ, η, ζ is set as the WB correction value Rg, Gg, Bg.

  Here, an automobile headlight is assumed as the artificial light source. The coefficients ξ, η, and ζ adjust the levels of the color signals dR, dG, and dB so that the reflected light becomes the reference white when the headlight irradiates the white plate coated with the reference white. A predetermined value that can be used is adopted, and a value that is calculated in advance by an experiment is adopted.

  The artificial light source is not limited to an automobile headlight, and various artificial light sources (for example, fluorescent lamps) used as illumination light at night can be used. In this case, coefficients ξ, η, ζ The value may be a value suitable for an artificial light source.

  Further, as the reference value of the brightness value, a predetermined value indicating that the brightness value is night can be adopted.

  On the other hand, when the brightness value calculated by the exposure control unit 6 is equal to or greater than a predetermined reference value, the WB correction value calculation unit 53 calculates the addition value Rave, the addition value Gave, and the addition value Bave according to Expression (2). The obtained values are set as WB correction values Rg, Gg, Bg.

Rg = Gave / Rave
Gg = Gave / Gave (2)
Bg = Gave / Bave
In other words, the WB correction value calculation unit 53 sets the WB correction values Rg, Gg, and Bg as fixed values by coefficients ξ, η, and ζ at night, and converts the WB correction values Rg, Gg, and Bg into image data in the daytime. Fluctuate accordingly.

  In Equation (2), the WB correction values Rg, Gg, and Bg are calculated based on the color signal dG. However, the present invention is not limited to this, and the WB correction values Rg, Gg and Bg may be calculated.

  The WB correction unit 46 uses the WB correction values Rg, Gg, and Bg set by the WB correction value calculation unit 53 as shown in Expression (3) as the color signals dR, The color signals dR ′, dG ′, and dB ′ are calculated by multiplying dG and dB, and the color signals dR, dG, and dB are WB corrected.

dR ′ = Rg × dR
dG ′ = Gg × dG (3)
dB ′ = Bg × dB
The color signal synthesis unit 47 weights the color signals dR ′, dG ′, and dB ′ with the weighting coefficient k determined by the coefficient determination unit 51 and interpolates the red signal interpolated by the color interpolation unit 43 as shown in Expression (4). The outer image component DBlk is weighted using the weighting coefficient kw determined by the coefficient determining unit 51, and each of the weighted weighted color signals dR ′, dG ′, and dB ′ is combined with the weighted infrared image component DBlk. The combined color signals Rwav, Gwav, and Bwav for each pixel are generated.

Rwav = k · dR ′ + kw · DBlk
Gwav = k · dG ′ + kw · DBlk (4)
Bwav = k · dB ′ + kw · DBlk
Here, the larger the evaluation value e, the larger the weighting coefficient k, and the smaller the weighting coefficient kw, the larger the evaluation value e.

  Therefore, in a bright scene such as clear sky, the ratio of the color signals dR ′, dG ′, and dB ′ representing the visible luminance image component in the composite color signals Rwav, Gwav, and Bwav is large, Equivalent images can be obtained.

  On the other hand, in a dark scene such as darkness, the ratio of the infrared image component DBlk increases in the composite color signals Rwav, Gwav, and Bwav, and a color signal having a good S / N ratio using a large number of luminance signals in the infrared wavelength region. Can be obtained.

  The color space conversion unit 48 generates the luminance signal Y and the color difference signals Cb, Cr using the combined color signals Rwav, Gwav, Bwav as shown in Expression (5).

  Here, the color difference signal Cb indicates a blue color difference signal, and the color difference signal Cr indicates a red color difference signal.

Y = 0.3Rwav + 0.59Gwav + 0.11Bwav
Cb = Bwav−Y (5)
Cr = Rwav-Y
Next, the operation of the image input apparatus 1 will be described. First, the control unit 5 causes the image sensor 3 to capture one frame of image data. Here, the imaging device 3 images the ye image component Dye by the ye pixel, images the R image component DR by the R pixel, images the infrared image component DBlk by the Blk pixel, and the luminance image component DW by the W pixel. Take an image. When the image input device 1 captures a moving image, the control unit 5 may cause the image sensor 3 to capture an image at a frame rate such as 1/30 s or 1/60 s. When the image input device 1 captures a still image, the control unit 5 may cause the image sensor 3 to capture an image when the user presses the release button.

  Here, the control unit 5 causes the exposure control unit 6 to calculate the brightness value from the luminance image component and to calculate the exposure time of the image pickup device 3 from the brightness value before causing the image pickup device 3 to pick up the subject. .

  Then, the image sensor 3 performs exposure with the exposure time calculated by the exposure control unit 6 to capture one frame of image data.

  Next, the evaluation unit 41 calculates the evaluation value eDW by adding the pixel values of the luminance image component DW in the local region, and calculates the evaluation value eDBlk by adding the pixel value of the infrared image component DBlk in the local region. The evaluation value e (= eDW−eDBlk) is calculated.

  Next, the coefficient determination unit 51 determines the weight coefficients k and kw from the evaluation value e. Next, the exposure correction value calculation unit 52 calculates the exposure correction value H1 from the weighting factors k and kw.

  Next, the exposure correction unit 42 multiplies the exposure correction value H1 by each pixel of the ye image component Dye, the R image component DR, the infrared image component DBlk, and the luminance image component DW, so that the ye image component Dye. , R image component DR, infrared image component DBlk, and luminance image component DW are corrected for exposure.

  Next, the color interpolation unit 43 performs an interpolation process on the exposure-corrected ye image component Dye, R image component DR, infrared image component DBlk, and luminance image component DW.

  Next, the color signal generation unit 44 generates the color signals dR, dG, and dB according to Expression (1). Next, the evaluation value calculation unit 45 calculates the addition values Rave, Gave, and Bave. Next, the WB correction value calculation unit 53 calculates the coefficients ξ, η, ζ as the WB correction values Rg, Gg, Bg when the brightness value calculated by the exposure control unit 6 is lower than the reference value. On the other hand, when the brightness value calculated by the exposure control unit 6 is greater than or equal to the reference value, the WB correction value calculation unit 53 calculates the WB correction values Rg, Gg, and Bg using Expression (2).

  Next, the WB correction unit 46 WB corrects the color signals dR, dG, and dB of each pixel using the WB correction values Rg, Gg, and Bg calculated by the WB correction value calculation unit 53, and the color signals dR ′, dG ′ and dB ′ are generated.

  Next, the color signal synthesis unit 47 generates the synthesized color signals Rwav, Gwav, and Bwav using Expression (4). Next, the color space conversion unit 48 generates the luminance signal Y and the color difference signals Cb and Cr using Expression (5).

  As described above, according to the image input apparatus 1, when the brightness value calculated by the exposure control unit 6 falls below a predetermined reference value, a predetermined coefficient for reproducing the light of the predetermined artificial light source white. The color signals dR, dG, dB are WB corrected using ξ, η, ζ. Therefore, highly accurate WB correction can be performed even at night when white light having a sufficient amount of light for performing WB correction cannot be obtained.

1 shows a block diagram of an image input apparatus 1 according to an embodiment of the present invention. It is a figure which shows the arrangement | sequence of the pixel of an image pick-up element. It is the figure which showed the spectral transmission characteristic of ye, R, Blk filter. The block diagram of an image processing part, a control part, and an exposure control part is shown. It is a graph which shows the characteristic of the weighting factors k and kw.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image input device 3 Image pick-up element 4 Image processing part 5 Control part 6 Exposure control part 44 Color signal generation part 45 Evaluation value calculation part 46 WB correction part 47 Color space conversion part 51 Coefficient determination part 52 Exposure correction value calculation part 53 WB correction Value calculator

Claims (4)

An image sensor;
A color signal generation unit that generates a color signal from image data captured by the image sensor;
A brightness detector for detecting the brightness of the subject;
When the brightness of the subject detected by the brightness detection unit falls below a predetermined reference value, the color signal is white balance corrected using a predetermined coefficient to reproduce white light of a predetermined artificial light source. A white balance correction unit
The artificial light source is an automobile headlight,
When the brightness detected by the brightness detection unit exceeds the reference value, the white balance correction unit calculates a white balance correction value based on image data captured by the image sensor, and the white balance An image input device that performs white balance correction on the image data based on a correction value .   2. The image according to claim 1, wherein the brightness detection unit is configured by an exposure control unit that detects the brightness of a subject and sets an exposure time of the imaging device in accordance with the detected brightness. Input device.   The image input device according to claim 1, wherein the image pickup device includes a pixel including an infrared filter, a pixel not including a filter, and a pixel including a color filter. The image pickup device has a matrix pixel unit unit including a first pixel, a second pixel, a third pixel, and a fourth pixel having a visible wavelength region and an infrared wavelength region as sensitive wavelength bands. Arranged in
The first pixel includes a first color filter that transmits light in the sensitive wavelength band excluding a blue region of the visible wavelength region,
The second pixel includes a second color filter that transmits light in the sensitive wavelength band excluding a blue region and a green region of the visible wavelength region,
The third pixel includes an infrared filter that transmits light in the sensitive wavelength band excluding the visible wavelength region,
The image input device according to claim 3, wherein the fourth pixel does not include a filter.

Images