JP5098908B2 - Image input device - Google Patents

Description

  The present invention relates to an image input device that performs image processing on an image captured by an image sensor.

2. Description of the Related Art In recent years, imaging devices that capture night scenes in color are known. Patent Literature 1 includes a visible image extraction unit 120 that extracts visible image data including R, G, and B color components from an image captured by a pixel including R, G, and B filters, and an Ir filter. Infrared image extraction means 130 for extracting infrared image data from an image picked up by pixels, first luminance information is extracted by HSV conversion of visible image data, and second luminance information is extracted from the infrared image data. Information extracting means 140 for extracting the first luminance information with weighting factor w1, weighting the second luminance information with weighting factor w2 (w1 + w2 = 1), and a pseudo color image generating means for generating a pseudo color image 170, a color image reproducing apparatus in which w1 = 1 is set on a sunny day, w1 = 0 is set in the dark at night, and 1>w1> 0 is set in an intermediate state.
JP 2007-184805 A

  By the way, in Patent Document 1, the luminance component V is obtained by converting the received light amounts of the red, green, and blue light receiving elements including the infrared component into the HSV color space (paragraph [0035]). The R, G, and B filters have different sensitivities in the infrared wavelength region, the sensitivities in the infrared wavelength region vary, and the sensitivities in some bands in the infrared wavelength region are low. Therefore, although the sensitivity in the visible wavelength region is different, no problem is considered in the case where an image pickup device including a plurality of types of filters having a constant sensitivity in the entire infrared wavelength region is employed.

  The object of the present invention is to solve the problem that occurs when an imaging device composed of pixels having a plurality of types of filters having a constant sensitivity in the entire infrared wavelength region, although the sensitivity in the visible wavelength region is different. It is to provide an image input device capable of performing the above.

  (1) An image input apparatus according to an aspect of the present invention captures an infrared image component with a pixel including an infrared filter, and a luminance image component including a visible luminance image component and an infrared image component with a pixel not including the filter. An image pickup device that picks up an image, an evaluation unit that evaluates the size of the infrared image component in the luminance image component, and a signal generation unit that generates a luminance signal based on the luminance image component and the infrared image component Then, based on the evaluation result by the evaluation unit, the edge component is emphasized so that the intensity of the edge component of the luminance signal increases as the size of the infrared image component in the luminance image component decreases. And an edge processing unit.

  The image sensor includes a pixel including an infrared filter and a pixel not including a filter. Here, the pixel provided with the infrared filter and the pixel not provided with the filter have a constant sensitivity over the entire infrared wavelength region. Therefore, this image sensor has a high resolution for infrared light but a low resolution for visible light. Therefore, this image sensor has a lower resolution as the intensity of the infrared light with respect to the visible light becomes weaker, and accordingly, the intensity of the edge component of the luminance signal also decreases as the intensity of the infrared light with respect to the visible light becomes weaker. To do. Therefore, the edge processing unit performs edge processing for enhancing the edge component of the luminance signal as the size of the infrared image component with respect to the luminance image component decreases. Thereby, it is possible to compensate for the decrease in the intensity of the edge component of the luminance signal due to the decrease in the resolution of the image sensor due to the decrease in the intensity of the infrared light with respect to the visible light.

  (2) The image sensor further includes a pixel including a color filter having a visible wavelength region and an infrared wavelength region as a sensitive wavelength band, and the signal generation unit is a color imaged by the pixel including the color filter. It is preferable to generate a color signal based on the image component, the luminance image component, and the infrared image component, and generate the luminance signal based on the generated color signal.

  In this case, a color image component is captured by a pixel including a color filter. Therefore, a color signal can be generated using the color image component, the luminance image component, and the infrared image component. In addition, since the color filter uses the infrared wavelength region as a sensitive wavelength band, the intensity of the edge signal of the luminance signal is reduced by applying the process (1) due to the reduction of the intensity of the infrared light with respect to the visible light. Can be supplemented.

  (3) The image pickup device includes a first pixel, a second pixel, a third pixel, and a fourth pixel having a visible wavelength region and an infrared wavelength region as sensitive wavelength bands. Are arranged in a matrix, and the first pixel includes a first color filter that transmits light in the sensitive wavelength band excluding the blue region of the visible wavelength region, and the second pixel includes the visible pixel. A second color filter that transmits light in the sensitive wavelength band excluding a blue region and a green region of the wavelength region; and the third pixel transmits light in the sensitive wavelength band excluding the visible wavelength region. Preferably, the fourth pixel is not provided with a filter.

  In this case, since the first pixel includes the first color filter, an image in a sensitive wavelength band excluding the blue region can be picked up as a visible color image component. Since the color filter is provided, an image in the sensitive wavelength band excluding the blue region and the green region can be captured as a color image component, and the third pixel is an image in the sensitive wavelength band excluding the visible wavelength region. Since the fourth image does not include a filter, an image in the sensitive wavelength band can be captured as the luminance image component.

  And since the unit pixel part provided with the 1st-4th pixel is arranged in matrix form, the image pick-up element can image a visible color image component, an infrared image component, and a luminance image component without bias. .

  (4) The edge processing unit includes: an extraction unit that extracts a high-frequency component from the luminance image component captured by the imaging device; and the infrared image component in the luminance image component based on an evaluation result by the evaluation unit. An adjustment unit for adjusting the intensity of the high-frequency component so that the intensity of the high-frequency component extracted by the extraction unit increases as the size decreases, a high-frequency component whose intensity is adjusted by the adjustment unit, and the signal generation It is preferable that a synthesis unit that enhances an edge component of the luminance signal by synthesizing with the luminance signal generated by the unit.

  In this case, since the intensity of the edge component of the luminance signal is adjusted by extracting the high frequency component from the luminance image component, adjusting the intensity of the high frequency component, and then combining it with the luminance signal, the edge component of the luminance signal is adjusted. The strength of the can be adjusted with high accuracy.

  (5) The evaluation unit may evaluate the size of the infrared image component in the luminance image component by calculating an evaluation value that is a value obtained by subtracting the infrared image component from the luminance image component. preferable.

  In this case, the size of the infrared image component in the luminance image component can be evaluated by simple processing.

  According to the present invention, the sensitivity to visible light is different when an imaging device including pixels having a plurality of types of filters having constant sensitivity in the entire infrared wavelength region is employed, although the sensitivity in the visible wavelength region is different. A decrease in the intensity of the edge component of the luminance signal due to a decrease in the intensity of external light can be compensated.

  Hereinafter, an image input apparatus 1 according to an embodiment of the present invention will be described. FIG. 1 shows a block diagram of an image input apparatus 1 according to an embodiment of the present invention. As shown in FIG. 1, the image input device 1 includes a lens 2, an image sensor 3, an image processing unit 4, and a control unit 5.

  The lens 2 includes an optical lens system that captures a light image of a subject and guides it to the image sensor 3. As the optical lens system, for example, a zoom lens, a focus lens, other fixed lens blocks, and the like arranged in series along the optical axis L of the optical image of the subject can be employed. The lens 2 includes a diaphragm (not shown) for adjusting the amount of transmitted light, a shutter (not shown), and the like, and the driving of the diaphragm and the shutter is controlled under the control of the control unit 5.

  The image pickup device 3 includes a light receiving unit composed of a PD (photodiode), an output circuit that outputs a signal photoelectrically converted by the light receiving unit, and a drive circuit that drives the image pickup device 3, and has a level corresponding to the amount of light. Generate image data. Here, as the imaging device 3, various imaging sensors such as a CMOS image sensor, a VMIS image sensor, and a CCD image sensor can be employed.

  In the present embodiment, the image sensor 3 captures a visible color image component with a pixel including a color filter, captures an infrared image component DBlk with a pixel including an infrared filter, and visualizes with a pixel not including the filter. The luminance image component DW including the luminance image component and the infrared image component DBlk is imaged.

  The image processing unit 4 includes an arithmetic circuit that performs an operation on the image signal, a memory that stores the image signal, and the like. The image data output from the image sensor 3 is A / D converted into a digital signal to be described later. After executing the processing, the data is output to a memory or a display device (not shown), for example.

  The control unit 5 includes a CPU and a memory that stores a program executed by the CPU, and outputs a control signal for controlling the image sensor 3 and the image processing unit 4 in response to a control signal from the outside. In addition, the control unit 5 receives an evaluation value, which will be described later, output from the image processing unit 4, and performs various processes, which will be described later, based on the evaluation value.

  FIG. 2 is a diagram illustrating an arrangement of pixels of the image sensor 3. As shown in FIG. 2, the imaging device 3 includes a ye pixel (an example of a first pixel), an R pixel (an example of a second pixel), a Blk having a visible wavelength region and an infrared wavelength region as sensitive wavelength bands. Unit pixel portions 31 including pixels (an example of a third pixel) and W pixels (an example of a fourth pixel) are arranged in a matrix.

  In the case of FIG. 2, in the unit pixel unit 31, R pixels are arranged in the first row and first column, Blk pixels are arranged in the second row and 21st column, and W pixels are arranged in the first row and second column, R pixels, Blk pixels, W pixels, and ye pixels are arranged in a staggered manner, such that ye pixels are arranged in the second row and the second column. However, this is only an example, and the R pixel, Blk pixel, W pixel, and ye pixel may be arranged in a staggered pattern in another pattern.

  Since the ye pixel includes a ye filter (an example of a first color filter), a visible color image component of ye (hereinafter referred to as “ye image component”) is imaged, and the R pixel is an R filter (second filter). Since an R visible color image component (hereinafter referred to as “R image component”) is imaged and a Blk pixel has a Blk filter (an example of an infrared filter). Since the infrared image component DBlk is imaged and the W pixel does not include a filter, the luminance image component DW including the visible luminance image component and the infrared image component DBlk is imaged.

  FIG. 3 is a diagram showing the spectral transmission characteristics of the ye, R, and Blk filters. The vertical axis shows the transmittance (sensitivity), and the horizontal axis shows the wavelength (nm). A graph indicated by a dotted line shows the spectral sensitivity characteristics of the pixel in a state where the filter is removed. It can be seen that this spectral sensitivity characteristic has a peak near 600 nm and changes in a convex curve. In FIG. 3, 400 nm to 640 nm is a visible wavelength region, 640 nm to 1100 nm is an infrared wavelength region, and 400 nm to 1100 nm is a sensitive wavelength band.

  As shown in FIG. 3, the ye filter has a characteristic of transmitting light in the sensitive wavelength band excluding the blue region of the visible wavelength region. Therefore, the ye filter mainly transmits yellow light and infrared light.

  The R filter has a characteristic of transmitting light in a sensitive wavelength band excluding a blue region and a green region in the visible wavelength region. Therefore, the R filter mainly transmits red light and infrared light.

  The Blk filter has a characteristic of transmitting light in a sensitive wavelength band excluding the visible wavelength region, that is, in the infrared wavelength band. W indicates a case where no filter is provided, and all light in the sensitive wavelength band of the pixel is transmitted.

  FIG. 4 shows a block diagram of the image processing unit 4 and the control unit 5. The image processing unit 4 includes an evaluation unit 41, a signal generation unit 42, an edge processing unit 43, and an evaluation value calculation unit 44. The control unit 5 includes a coefficient determination unit 51, an exposure correction value calculation unit 52, and a WB (white balance) correction value calculation unit 53. The signal generation unit 42 includes an exposure correction unit 421, a color interpolation unit 422, a color signal generation unit 423, a WB correction unit 424, a color signal synthesis unit 425, and a color space conversion unit 426.

  The evaluation unit 41 evaluates the size of the infrared image component DBlk in the luminance image component DW. Specifically, the evaluation unit 41 has a pixel value of W pixels, that is, a pixel value of the luminance image component DW, in a local region including pixels in a predetermined row × predetermined column in an image of one frame imaged by the image sensor 3. Is added to calculate the evaluation value eDW, and the pixel value of the Blk pixel, that is, the pixel value of the infrared image component DBlk is added to calculate the evaluation value eDBlk, and the evaluation value eDW is subtracted from the evaluation value eDBlk. Calculated as an evaluation value e (= eDW−eDBlk). Then, the evaluation unit 41 evaluates the size of the infrared image component DBlk in the luminance image component DW based on the calculated evaluation value e. As the evaluation value e increases, the intensity of infrared light with respect to visible light decreases.

  The evaluation unit 41 employs eDW-eDBlk as the evaluation value e, but any value may be employed as long as the value indicates the size of the infrared image component DBlk in the luminance image component DW. For example, eDBlk / eDW may be adopted as the evaluation value e.

  The coefficient determination unit 51 evaluates a weighting coefficient k having a predetermined characteristic that increases as the evaluation value e increases, and a weighting coefficient kw that has a predetermined characteristic that decreases as the evaluation value e increases. The evaluation value e calculated by the unit 41 is used for determination.

  FIG. 5 is a graph showing the characteristics of the weighting factors k and kw. The vertical axis represents the weighting factors k and kw, and the horizontal axis represents the evaluation value e (= eDW−eDBlk). As shown in FIG. 5, the weighting factor k has a characteristic that increases linearly as the evaluation value e increases, and the weighting factor kw has a characteristic that decreases linearly as the evaluation value e increases, for example. I understand.

  In FIG. 5, experimentally obtained values are used as the slopes of the straight lines indicating the characteristics of the weighting factors k and kw. Specifically, the weighting factors k and kw have a relationship of k + kw = 1, and the slope of the straight line indicating the characteristics of the weighting factors k and kw is the maximum value of the evaluation value e for which the evaluation value e is predetermined. K = 1 and kw = 0, and k = 0 and kw = 1 when the evaluation value e indicates a predetermined minimum value of the evaluation value e.

  The exposure correction value calculation unit 52 calculates the luminance evaluation value edY by substituting the weighting coefficients k and kw determined by the coefficient determination unit 51 into the following equation, and divides the luminance evaluation value edY by a predetermined value. An exposure correction value H1 is calculated and output to the exposure correction unit 421. Here, as the predetermined value, the total number of pixels of the DBlk pixel and the DW pixel in the above-described local region can be employed.

edY = kw × eDW + k × (eDW−eDBlk)
The exposure correction unit 421 multiplies the exposure correction value H1 by the respective pixels of the ye image component Dye, the R image component DR, the infrared image component DBlk, and the luminance image component DW captured by the image sensor 3. The exposure correction is performed on the ye image component Dye, the R image component DR, the infrared image component DBlk, and the luminance image component DW.

  The color interpolation unit 422 interpolates missing pixels of R pixels, Blk pixels, W pixels, and ye pixels arranged in a staggered manner, so that an R image component DR, an infrared image component DBlk, a luminance image component DW, and Each of the ye image components Dye is subjected to interpolation processing to generate an R image component DR, an infrared image component DBlk, a luminance image component DW, and a ye image component Dye having the same number of pixels as the number of pixels of the image sensor 3. Here, as the interpolation processing, for example, linear interpolation processing can be adopted.

  The color signal generation unit 423 combines the ye image component Dye subjected to the interpolation processing by the color interpolation unit 422, the R image component DR, the infrared image component DBlk, and the luminance image component DW by Expression (1). Thus, R (red), G (green), and B (blue) color signals dR, dG, and dB of each pixel are generated.

dR = DR-DBlk
dG = Dye-DR (1)
dB = DW-Dye
The evaluation value calculation unit 44 adds the values of the color signals dR, dG, and dB of each pixel in the above-described local region, adds the value Rave of the color signal dR, the addition value Gave of the color signal dG, and the color signal dB. Is added to the controller 5 and output to the controller 5.

  The WB correction value calculation unit 53 calculates the R gain Rgain, the G gain Ggain, and the B gain Bgain from the added value Rave, the added value Gave, and the added value Bave according to Expression (2).

Rgain = Gave / Rave
Ggain = Gave / Gave (2)
Bgain = Gave / Bave
The WB correction unit 424 uses the gains Rgain, Ggain, and Bgain calculated by the WB correction value calculation unit 53 as shown in Expression (3) as the color signals dR, dG, The color signals dR ′, dG ′, and dB ′ are calculated by multiplying dB. As a result, the color signals dR, dG, and dB are subjected to WB correction based on the color signal dG.

dR ′ = Rgain × dR
dG ′ = Ggain × dG (3)
dB ′ = Bgain × dB
Note that the WB correction unit 424 may perform WB correction using the color signal dR or the color signal dB as a reference instead of the color signal dG. In this case, the WB correction value calculation unit 53 may calculate a gain corresponding to the color signal dR or dB that is the reference of the WB correction unit 424.

  The color signal synthesis unit 425 weights the color signals dR ′, dG ′, and dB ′ with the weighting coefficient k determined by the coefficient determination unit 51 and interpolates the red signal interpolated by the color interpolation unit 422 as shown in Expression (4). The outer image component DBlk is weighted using the weighting coefficient kw determined by the coefficient determining unit 51, and each of the weighted weighted color signals dR ′, dG ′, and dB ′ is combined with the weighted infrared image component DBlk. The combined color signals Rwav, Gwav, and Bwav for each pixel are generated.

Rwav = k · dR ′ + kw · DBlk
Gwav = k · dG ′ + kw · DBlk (4)
Bwav = k · dB ′ + kw · DBlk
Here, the larger the evaluation value e, the larger the weighting coefficient k, and the smaller the weighting coefficient kw, the larger the evaluation value e.

  Therefore, when a subject such as clear sky is bright, the ratio of the color signals dR ′, dG ′, and dB ′ representing the visible luminance image component in the composite color signals Rwav, Gwav, and Bwav is large, and the subject that the person is viewing Can be obtained.

  On the other hand, when a subject such as darkness is dark, the ratio of the infrared image component DBlk is large in the composite color signals Rwav, Gwav, and Bwav, and a color with a good S / N ratio is obtained by using a large number of luminance signals in the infrared wavelength region. A signal can be obtained.

  The color space conversion unit 426 generates a luminance signal Y and color difference signals Cb, Cr using the combined color signals Rwav, Gwav, Bwav as shown in Expression (5).

  Here, the color difference signal Cb indicates a blue color difference signal, and the color difference signal Cr indicates a red color difference signal.

Y = 0.3Rwav + 0.59Gwav + 0.11Bwav
Cb = Bwav−Y (5)
Cr = Rwav-Y
The edge processing unit 43 includes an extraction unit 431, a coring unit 432, an adjustment unit 433, and a synthesis unit 434. The extraction unit 431 extracts the high frequency component Yh of the luminance image component DW interpolated by the color interpolation unit 422. Here, the extraction unit 431 extracts the high frequency component Yh as follows. The luminance image component DW is filtered using a broadband low-pass filter that passes through the band of the high-frequency component Yh to be extracted, and the broadband low-pass component YWW is extracted. On the other hand, the four image components of the luminance image component DW, the ye image component Dye, the R image component DR, and the infrared image component DBlk interpolated by the color interpolation unit 422 are averaged to generate an average image component YWL. . Then, the high-frequency component Yh is extracted by the broadband low-pass component YWW-average image component YWL.

  The extraction unit 431 may extract the high frequency component Yh by filtering the luminance image component DW using, for example, a predetermined row × predetermined high-pass filter.

  The coring unit 432 performs coring on the high frequency component Yh and removes weak noise included in the high frequency component Yh.

  The adjustment unit 433 sets a gain GG for increasing the strength of the high-frequency component Yh as the evaluation value e calculated by the evaluation unit 41 decreases, and multiplies the set gain GG by the high-frequency component Yh to thereby increase the high-frequency component Yh. Adjust the intensity.

  FIG. 6 is a graph showing the characteristics of the gain GG, where the vertical axis indicates the gain GG and the horizontal axis indicates the evaluation value e (= eDW−eDBlk). E-max is a predetermined maximum value of the evaluation value e. As shown in FIG. 6, the gain GG is LL when e is in the range of 0 to 1 / 3e-max, and is almost linear from LL when e is in the range of 1 / 3e-max to 2 / 3e-max. It gradually increases to HL, and becomes HL in the range of e from 2/3 e-max to e-max. The reason why the gain GG is gradually increased in this way is that the intensity of the edge component of the luminance signal Y gradually decreases with respect to the decrease in the intensity of the infrared light with respect to the visible light.

  Specifically, the adjustment unit 433 stores in advance a lookup table or function indicating the relationship between the evaluation value e and the gain GG shown in FIG. 6, and uses this lookup table or function to set the evaluation value e. Set the corresponding gain GG. Then, the strength of the high frequency component Yh is adjusted by multiplying the set gain GG by the high frequency component Yh.

  In FIG. 6, for example, 0.2 can be used as LL, and 0.8 can be used as HL. However, this is an example, and 0 or 0.1 as LL, 0.8 or 1 as HL. Other values such as and may be adopted. Further, in FIG. 6, the gain GG changes at 1 / 3e-max and 2 / 3e-max, but is not limited to this, and other values such as 1 / 4e-max and 3 / 4e-max are obtained. It may be changed with.

  The synthesizing unit 434 generates a luminance signal Y ′ in which the edge component is emphasized by synthesizing the high frequency component Yh whose intensity is adjusted by the adjusting unit 433 and the luminance signal Y generated by the color space converting unit 426. . Here, the synthesizing unit 434 adds the pixel values of the corresponding pixels of the high frequency component Yh and the luminance signal Y, or weights the high frequency component Yh and the luminance signal Y at a certain ratio, and sets the pixel values of the corresponding pixels. What is necessary is just to synthesize | combine the high frequency component Yh and the luminance signal Y by adding.

  As described above, the image processing unit 4 generates an image signal including the luminance signal Y and the color difference signals Cb and Cr.

  Next, the operation of the image input apparatus 1 will be described. First, the control unit 5 causes the image sensor 3 to capture an image of one frame. Here, the imaging device 3 images the ye image component Dye by the ye pixel, images the R image component DR by the R pixel, images the infrared image component DBlk by the Blk pixel, and the luminance image component DW by the W pixel. Take an image. When the image input device 1 captures a moving image, the control unit 5 may cause the image sensor 3 to capture an image at a frame rate such as 1/30 s or 1/60 s. When the image input device 1 captures a still image, the control unit 5 may cause the image sensor 3 to capture an image when the user presses the release button.

  Next, the evaluation unit 41 calculates the evaluation value eDW by adding the pixel values of the luminance image component DW in the local region, and calculates the evaluation value eDBlk by adding the pixel value of the infrared image component DBlk in the local region. The evaluation value e (= eDW−eDBlk) is calculated.

  Next, the coefficient determination unit 51 determines the weight coefficients k and kw from the evaluation value e. Next, the exposure correction value calculation unit 52 calculates the exposure correction value H1 from the weighting factors k and kw.

  Next, the exposure correction unit 421 multiplies each pixel of the ye image component Dye, the R image component DR, the infrared image component DBlk, and the luminance image component DW by multiplying the exposure correction value H1 by the ye image component Dye. , R image component DR, infrared image component DBlk, and luminance image component DW are corrected for exposure.

  Next, the color interpolation unit 422 interpolates the exposure-corrected ye image component Dye, R image component DR, infrared image component DBlk, and luminance image component DW.

  Next, the color signal generation unit 423 generates the color signals dR, dG, and dB using Expression (1).

  Next, the evaluation value calculation unit 44 calculates the addition values Rave, Gave, and Bave. Next, the WB correction value calculation unit 53 calculates gains Rgain, Ggain, and Bgain using the addition values Rave, Gave, and Bave. Next, the WB correction unit 46 WB corrects the color signals dR, dG, dB of each pixel using the gains Rgain, Ggain, Bgain.

  Next, the color signal synthesis unit 425 generates the synthesized color signals Rwav, Gwav, and Bwav using Expression (4). Next, the color space conversion unit 426 generates a luminance signal Y and color difference signals Cb and Cr using Expression (5).

  Next, the extraction unit 431 extracts the high frequency component Yh of the luminance image component DW. Next, the coring unit 432 removes weak noise from the high frequency component Yh. Next, the adjustment unit 433 sets the gain GG from the evaluation value e, multiplies the set gain GG by the high frequency component Yh, and adjusts the intensity of the high frequency component Yh.

  Next, the synthesizing unit 434 synthesizes the high-frequency component Yh whose intensity is adjusted with the luminance signal Y generated by the color space conversion unit 426, and the luminance signal Y ′ whose intensity of the edge component of the luminance signal Y is adjusted. Is generated.

  Thus, according to the image input device 1, the edge processing unit 43 increases the edge component of the luminance signal Y as the evaluation value e increases, that is, as the intensity of infrared light with respect to visible light decreases. Edge processing is performed. Thereby, it is possible to compensate for the decrease in the intensity of the edge component of the luminance signal due to the decrease in the resolution of the image sensor due to the decrease in the intensity of the infrared light with respect to the visible light.

1 shows a block diagram of an image input device according to an embodiment of the present invention. FIG. It is a figure which shows the arrangement | sequence of the pixel of an image pick-up element. It is the figure which showed the spectral transmission characteristic of ye, R, Blk filter. The block diagram of an image processing part and a control part is shown. It is a graph which shows the characteristic of the weighting factors k and kw. It is a graph which shows the characteristic of the gain which the adjustment part 433 sets.

Explanation of symbols

3 image sensor 4 image processing unit 5 control unit 41 evaluation unit 42 signal generation unit 43 edge processing unit 423 color signal generation unit 425 color signal synthesis unit 426 color space conversion unit 431 extraction unit 432 coring unit 433 adjustment unit 434 synthesis unit

Claims (5)

An image sensor that captures an infrared image component with a pixel including an infrared filter, and that captures a luminance image component including a visible luminance image component and an infrared image component with a pixel that does not include a filter;
An evaluation unit for evaluating the size of the infrared image component in the luminance image component;
A signal generation unit that generates a luminance signal based on the luminance image component and the infrared image component;
Edge processing for emphasizing the edge component so that the intensity of the edge component of the luminance signal increases as the size of the infrared image component in the luminance image component decreases based on the evaluation result by the evaluation unit An image input device. The imaging device further includes a pixel including a color filter having a visible wavelength region and an infrared wavelength region as a sensitive wavelength band,
The signal generation unit generates a color signal based on a color image component captured by a pixel including the color filter, the luminance image component, and the infrared image component, and based on the generated color signal, the luminance The image input device according to claim 1, wherein a signal is generated. The image pickup device has a matrix pixel unit unit including a first pixel, a second pixel, a third pixel, and a fourth pixel having a visible wavelength region and an infrared wavelength region as sensitive wavelength bands. Arranged in
The first pixel includes a first color filter that transmits light in the sensitive wavelength band excluding a blue region of the visible wavelength region,
The second pixel includes a second color filter that transmits light in the sensitive wavelength band excluding a blue region and a green region of the visible wavelength region,
The third pixel includes an infrared filter that transmits light in the sensitive wavelength band excluding the visible wavelength region,
The image input device according to claim 2, wherein the fourth pixel does not include a filter. The edge processing unit
An extraction unit that extracts a high-frequency component from a luminance image component imaged by the imaging device;
Based on the evaluation result by the evaluation unit, the intensity of the high-frequency component is adjusted so that the intensity of the high-frequency component extracted by the extraction unit increases as the size of the infrared image component in the luminance image component decreases. Adjusting part to be
A synthesis unit that enhances an edge component of the luminance signal by synthesizing the high-frequency component whose intensity is adjusted by the adjustment unit and the luminance signal generated by the signal generation unit. Item 4. The image input device according to any one of Items 1 to 3.   The evaluation unit evaluates the size of the infrared image component in the luminance image component by calculating an evaluation value that is a value obtained by subtracting the infrared image component from the luminance image component. The image input device according to claim 1.

Images