JP2017118283A - Camera system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a camera system that is capable of performing stereo photographing with respect to both of a visible image and an infrared image, and is capable of improving color reproducibility at the time of visible light photographing.SOLUTION: A camera system comprises: two imaging sensors 1; and two DBPFs 5 that have a characteristic of permeability in a visible light band and a second wavelength band, are provided so as to correspond to the two imaging sensors in a one-to-one manner, and are an optical filter. The camera system comprises two color filters that include at least four types of filter units having mutually different spectroscopy permeation characteristics corresponding to wavelengths in the visible light band and mutually approximate permeation characteristics in the second wavelength band, and are provided so as to correspond to the two imaging sensors in a one-to-one manner. The camera system measures a distance to an object 200 on the basis of two visible or infrared image signals.SELECTED DRAWING: Figure 54

Description

  The present invention relates to a camera system that performs shooting for distance measurement and image display.

  In recent years, improvement of image recognition technology in surveillance cameras and the like, and development of cameras used for automatic driving of automobiles have been promoted. In movies, 3D video by computer graphics is increasingly used, and not only computer graphics but also the frequency of use of 3D video by live action may increase. A stereo camera can be used as an operation input method in a video game.

  For example, a surveillance camera is known that detects distance using a stereo camera and detects the intrusion of a suspicious person (see, for example, Patent Document 1). Patent Document 1 describes a “distance measurement image recognition device using a stereo image” as a method for measuring the shape and distance of an object, and a technique called a stereo method for obtaining a distance from a stereo image is known. ing. In this stereo method, first, two left and right images called stereo images are input, and corresponding points of the left and right images are obtained by calculating feature amounts in the images. This corresponding point indicates where the target object at a certain position in the left image appears in the right image. Details of how to find these corresponding points are for example. Patent Document 2 describes this as an “image matching method”. If the corresponding points of the left and right images are obtained, the distance to the object surface can be calculated by the principle of triangulation. Therefore, the distance to the object and the shape of the object surface can be known.

  Patent Document 1 proposes a moving object recognition apparatus that can detect a moving object with high accuracy and high speed and measure its shape and distance by using a known correspondence relationship of stereo images. .

  Such an image recognition technique using a distance measuring technique based on the stereo method seems to be useful for a user interface in an electronic device including a surveillance camera, an automatic driving camera, a video game machine, and the like. A surveillance camera or an automatic operation camera performs night shooting within the shooting range. However, depending on the situation, sufficient lighting may not be obtained. For example, in order to make it difficult to understand the position of the monitoring camera, there may be a case where no visible light illumination is provided in the shooting range of the monitoring camera. In such a case, it is conceivable to perform infrared imaging using infrared illumination that is invisible to humans. In addition, in an autonomous driving camera, it is conceivable to perform infrared photographing using infrared light as illumination for illuminating a distant place in consideration of the influence of a headlight on an oncoming vehicle at night. In any case, infrared illumination is difficult to see for human eyes when it is necessary to shoot at night without lighting during daytime when there is a high possibility that the amount of visible light is sufficient without using illumination. It is conceivable to perform infrared imaging using the.

In consideration of such a situation, it is preferable that the camera can be used for photographing visible light and infrared light, depending on the application.
For example, in an imaging device such as a surveillance camera that continuously shoots day and night, infrared light is detected during shooting at night. Since a photodiode (light receiving element) that is a light receiving portion of an image sensor such as a CCD sensor or a CMOS sensor can receive light in the near-infrared wavelength band of about 1300 nm, if it is an image pickup apparatus using these image sensors, In principle, it is possible to photograph up to the infrared band.

  Since the wavelength band of light with high human visibility is 400 nm to 700 nm, when near-infrared light is detected by the image sensor, the image looks reddish to the human eye. For this reason, when shooting in a bright place in the daytime or indoors, in order to match the sensitivity of the image sensor with human visual sensitivity, an infrared cut filter that blocks infrared band light is provided in front of the image sensor, It is desirable to remove light having a wavelength of 700 nm or more. On the other hand, when shooting at night or in a dark place, it is necessary to perform shooting without providing an infrared cut filter.

  As such an image pickup apparatus, an image pickup apparatus for manually attaching / detaching an infrared cut filter and an image pickup apparatus for automatically inserting / removing an infrared cut filter are conventionally known. Furthermore, an imaging apparatus is disclosed that eliminates the need to insert and remove the infrared cut filter described above. For example, the second wavelength band which has a transmission characteristic in the visible light band, has a cutoff characteristic in the first wavelength band adjacent to the long wavelength side of the visible light band, and is a part of the first wavelength band An optical filter having transmission characteristics has been proposed (see, for example, Patent Document 3). According to this filter, light can pass through both the visible light band and the second wavelength band far from the visible light band on the long wavelength side of the visible light band, that is, on the infrared side.

  For example, the second wavelength band overlaps with the wavelength band of infrared illumination, and this filter is an optical filter that enables both visible light imaging and nighttime infrared light imaging using infrared light illumination. It is. Hereinafter, as described above, an optical filter that transmits light in the visible light band and the second wavelength band on the infrared side and blocks light in other wavelength bands is referred to as DBPF (double band pass filter). Called.

Japanese Patent Laid-Open No. 3-81878 JP-A-62-107386 Japanese Patent No. 5009395

  By the way, in DBPF of patent document 3, the light of the 2nd wavelength band contained in the infrared (near-infrared) wavelength band (relatively narrow wavelength band contained in the infrared wavelength band) is not cut off at all times. Will be transmitted. That is, unlike the case of using an infrared cut filter that cuts the longer wavelength side than the visible light band, in the imaging in the visible light band, the influence of the infrared light transmitted through the second wavelength band is not a little. Become.

  A color filter is used for an imaging sensor that performs color imaging as imaging in the visible light band. In the color filter, areas (filter units) of red, green, and blue are arranged in a predetermined pattern corresponding to each pixel of the image sensor. Each of these color regions basically has a light transmittance peak in the wavelength band of each color, and restricts (blocks) transmission of light in the wavelength bands of other colors.

  However, on the longer wavelength side than the visible light band, although the light transmittance differs depending on the region and wavelength of each color, light is basically transmitted. Therefore, if an infrared cut filter is used, there is no problem because light on the longer wavelength side than the visible light band is cut. However, when infrared light is transmitted in the second wavelength band on the infrared side as in the above DBPF, This infrared light passes through the color filter and reaches the photodiode (light receiving element) of the image sensor, increasing the amount of electrons generated by the photoelectric effect in the photodiode.

  Here, in performing both color photographing with visible light and photographing with infrared light illumination, for example, the above-described color filter in which each color region of red, green, and blue is arranged in a predetermined pattern is used. An infrared light region (infrared region) having a light transmittance peak in the second wavelength band is provided. That is, the arrangement (pattern) of the color filter includes four regions of red R, green G, blue B, and infrared IR. In this case, since the infrared light region blocks light in the visible light band and mainly transmits light in the second wavelength band, the light that has passed through the infrared light region of the color filter. It is conceivable to remove infrared light components from image signals of red, green, and blue colors using an infrared light image signal output from an imaging sensor that receives the light. However, even with such signal processing, it has been difficult to perform substantially the same color reproduction as in color photography when an infrared cut filter is used. In addition, when the distance is calculated by making a stereo, if there is a difference between the left and right signal levels, it is a factor causing an error in the parallax calculation.

  Here, for example, the surveillance camera is configured to be capable of ranging by the stereo method, and it is possible to shoot day and night using both visible light and infrared light. If we can make it as clear and high-resolution as possible, we will be able to greatly improve the ability to analyze the situation when a crime occurs. Also, in the automatic driving camera, when infrared and visible imaging are used together as described above, a clear image with less noise can be obtained, so that the accuracy of image recognition can be improved. From the above, stereo imaging of both infrared and visible images is possible with two cameras for stereo photography, and image quality such as noise, resolution and color reproducibility does not include normal infrared images It is desired to have a level equal to or higher than that of a visible image.

  The present invention has been made in view of the above circumstances, and provides a camera system capable of performing stereo photography with both a visible image and an infrared image and improving color reproducibility during visible light photography. Objective.

In order to solve the above problems, a camera system of the present invention includes two imaging sensors in which a light receiving element is disposed in each pixel,
A transmission characteristic in the visible light band, a cutoff characteristic in the first wavelength band adjacent to the long wavelength side of the visible light band, and a second wavelength band that is a part of the first wavelength band; Two optical filters having transmission characteristics and provided corresponding to the two imaging sensors,
Spectral transmission characteristics corresponding to the wavelength in the visible light band are different from each other, and the transmittance in the second wavelength band is approximately similar to each other. Two color filters,
Each of the image sensors has a lens for connecting an image, and two optical systems provided corresponding to the two image sensors,
A signal processing device capable of processing a signal output from each imaging sensor and outputting two visible image signals and two infrared image signals;
Using the two visible image signals and / or two infrared image signals output from the signal processing device, the distance to the subject reflected in the visible image by the visible image signal and / or the infrared image by the infrared image signal is determined. A distance measuring device for calculating,
The signal processing device is based on an infrared component corresponding to the second wavelength band by using a signal output from the imaging sensor corresponding to light transmitted through each type of the filter unit of the color filter. Calculating an infrared image signal and a visible image signal based on at least three color components corresponding to the visible light band from which the infrared component is subtracted;
In addition, when subtracting the infrared component from the color component, if the color component is equal to or higher than a pixel saturation level, control is performed to correct the infrared component subtracted from the color component so as to decrease the value. A subtraction control device is provided.

  According to such a configuration, each color signal including the infrared component exceeds the dynamic range and reaches the pixel saturation level, so that it is not clipped without exceeding the dynamic range in the clipped state, and the output level. When an infrared component (infrared signal) that may increase is subtracted from each color component (color signal), the output level of the infrared signal increases after each color signal is clipped. The output level will drop.

  As a result, when the brightness of the highlight portion decreases, the color signal is clipped as described above by performing control so as to reduce the value of the infrared signal subtracted from the color signal. It can be prevented that the luminance of the color signal after the infrared signal becomes high and the infrared signal is subtracted is lowered. For example, when the output level of each RGB color signal and the output level of the infrared signal are high, the output level of each RGB color signal including the infrared component is clipped over the dynamic range first, and is substantially red. A state in which the output level of each RGB color signal is further increased if it is not clipped by subtracting the infrared signal from the color signal because the infrared signal consisting only of the external component is further increased without exceeding the dynamic range. However, it is possible to prevent the output level of each color signal of RGB from decreasing and the brightness of the highlight portion from being overexposed to decrease.

  From the above, stereo imaging of both infrared and visible images is possible with two imaging devices for stereo shooting, and image quality such as noise, resolution, and color reproducibility is included, including normal infrared images There can be no more than the same level as the visible image. As a result, errors in distance measurement using a visible image can be suppressed, a more accurate distance can be measured, and a visible captured image with sufficient image quality can be displayed. When the distance is calculated using the signal, the signal levels of the two image signals can be matched, and the distance error can be reduced. That is, it is possible to suppress the occurrence of an error caused by recognizing different parts of the subject as the same part due to the difference between the signal levels of the two image signals and their changes. It should be noted that the distance to the subject is the distance between the camera system and the subject. At this time, the position of the camera system may be the position of the lens unit or the image sensor.

  In the configuration of the present invention, the color filter includes the red filter section that transmits the infrared component and the red component light in the visible light band, and the infrared component and the green component light in the visible light band. The green filter section that transmits light, the blue filter section that transmits light of the infrared component and the blue component of the visible light band, the infrared component, the red component, the green component, and the blue component It is preferable to include the white filter portion that transmits the light of the white component combined.

  According to such a configuration, in a color filter capable of outputting an infrared image in simultaneous photographing of a visible image and an infrared image, for example, an infrared that transmits only the infrared component without transmitting the visible component. Compared to the case of using the infrared filter portion of the component, by using the filter portion of the white component, the amount of information of the luminance component in the color filter is increased, corresponding to the characteristics of the human eye, An image with a high resolution can be output.

In the configuration of the present invention, the subtraction control device is
As correction for lowering the value of the infrared component, it is preferable to control the value of the infrared component so as not to be higher than a limit value determined based on the color component.

  According to such a configuration, the value of the infrared signal subtracted from the color signal is limited by the limit value and is clipped by the limit value, whereby each color signal reaches the saturation level and is clipped. Even if the value of the infrared signal that has not yet exceeded the saturation level in the state is high, the value subtracted from the color signal is basically the limit value, and the output level of the color signal obtained by subtracting the infrared signal is It can be prevented from becoming low.

  In the configuration of the present invention, it is preferable that the subtraction control device determines the limit value of the infrared component for each color component based on a gain of each color obtained by white balance processing.

  According to such a configuration, it is possible to subtract an infrared signal limited by an appropriate limit value in each color component (color signal), for example, each RGB signal.

  In the configuration of the present invention, the signal processing device may be configured such that a signal saturation level corresponding to a value obtained by subtracting the limit value of the infrared component from the pixel saturation level of the color component is substantially the same for each color. It is preferable to control.

  According to such a configuration, when the output level is the highest in each color signal, that is, when the signal saturation level is reached, the output level of each color signal is substantially the same, so that it is substantially white and high. A color such as blue is not attached to the light portion, and natural color reproducibility can be maintained by making the highlight portion white. The signal saturation level is a value level when each color signal of visible light is clipped at the pixel saturation level and the infrared signal limited by the limit value is subtracted.

  In the configuration of the present invention, it is preferable that the signal processing device corrects the signal levels of the two visible image signals and / or the signal levels of the two infrared image signals to be close to each other.

  According to such a configuration, when measuring the distance from the two visible image signals and the stereo image obtained from the two infrared image signals to the object (subject), the object in the two images as the stereo image An object (corresponding point between two images) cannot be accurately recognized due to a difference in signal level, and it is possible to suppress an increase in error due to an inaccurate parallax.

  According to the present invention, the distance of an object can be measured more accurately in both an infrared image and a visible image.

It is the schematic which shows the image sensor of the camera system of the 1st Embodiment of this invention. It is a graph which shows the transmittance | permeability spectrum of DBPF of the said image sensor, and a color filter similarly. It is a graph which shows the transmittance | permeability spectrum of DBPF and the color filter of the said image sensor, and the emission spectrum of infrared illumination. It is a graph which shows the transmittance | permeability spectrum of DBPF and the color filter of the said image sensor, and the emission spectrum of infrared illumination. FIG. 6 is a diagram for explaining the arrangement of color filters of the image sensor, wherein (a) is a diagram showing a conventional arrangement without an infrared filter section, and (b), (c), (d). FIG. 4 is a diagram showing an array having an infrared filter section. It is the schematic which shows the imaging device which has the said imaging sensor of the said camera system. FIG. 3 is a block diagram for explaining signal processing in a signal processing unit of the imaging apparatus. It is a figure for demonstrating the interior process in the signal processing of the said imaging device. It is the schematic which shows the imaging device of the 2nd Embodiment of this invention. It is a figure for demonstrating the arrangement | sequence of the color filter of the imaging sensor of the 3rd Embodiment of this invention. It is a graph which shows the transmittance | permeability spectrum of DBPF of the said image sensor, and a color filter similarly. It is a figure for demonstrating the arrangement | sequence of the color filter of an image sensor similarly. It is a figure for demonstrating the arrangement | sequence of a color filter similarly. It is a figure for demonstrating the arrangement | sequence of a color filter similarly. It is a figure for demonstrating the characteristic of each color filter similarly. It is a block diagram for demonstrating the signal processing in an imaging device equally. It is a figure for demonstrating the arrangement | sequence of the color filter of the 4th Embodiment of this invention. It is a graph which shows the transmittance | permeability spectrum of DBPF of the said image sensor, and the blue B of a color filter similarly. It is a graph which shows the transmittance | permeability spectrum of DBPF of the said image sensor, and green G of a color filter similarly. It is a graph which shows the transmittance | permeability spectrum of DBPF of the said image sensor, and red R of a color filter. It is a graph which shows the transmittance | permeability spectrum of DBPF of the said image sensor, and the clear C of a color filter similarly. It is a block diagram which shows the imaging sensor of the 5th Embodiment of this invention. It is sectional drawing which shows an image sensor same as the above. It is sectional drawing which shows the other example of an image sensor same as the above. It is a figure for demonstrating the output of the image signal of 2 systems by an image sensor similarly. It is a block diagram which shows the imaging device of the 6th Embodiment of this invention. It is the schematic which shows an image sensor same as the above. It is a figure for demonstrating the arrangement pattern of the color filter which has an infrared region similarly. It is a graph which shows the transmittance | permeability spectrum of DBPF of the said image sensor, and a color filter similarly. FIG. 10 is a diagram of an R signal for explaining the reason why the luminance decreases in the highlight portion. It is a figure of G signal for demonstrating the reason a brightness | luminance falls in a highlight part similarly. It is a figure of B signal for demonstrating the reason a brightness | luminance falls in a highlight part similarly. FIG. 11 is a diagram of an IR signal for explaining the reason why the luminance decreases in the highlight portion. It is a figure which shows the clip level corresponding to each RGB color signal of IR signal. FIG. 6 is a diagram for explaining subtraction of clipped IR signals from RGB color signals, where (a) shows the case of the R signal, (b) shows the case of the G signal, and (c ) Is a diagram showing a case of a B signal. FIG. 6 is a diagram for explaining white balance processing, where (a) shows a region set to white in the plane of the RY signal and BY signal, and (b) shows white. It is a figure which shows the range of a luminance level. FIG. 6 is a diagram for explaining a method for adjusting the output level of each RGB signal clip after the IR signal is subtracted; FIG. 2 is a block diagram illustrating a signal processing unit of the imaging processing apparatus. It is a block diagram which shows the signal processing part as another example same as the above. It is a figure for demonstrating the saturation level of the RGBC color filter of the 7th Embodiment of this invention. It is a figure for demonstrating that the signal level of IR signal calculated | required by calculation becomes higher than an actual IR signal. FIG. 6 is an RGBC signal diagram for explaining the reason why the brightness decreases in the highlight portion. It is a block diagram for demonstrating a separation device equally. It is a block diagram for demonstrating a clip level calculation device same as the above. It is a figure for demonstrating the clip level of an RGBC signal similarly. It is a figure for demonstrating the clip level of IR signal similarly. It is a figure for demonstrating the clip level of an RGBC signal similarly. It is a block diagram for demonstrating a separation device equally. It is a figure for demonstrating the clipping process of an RGBCY signal similarly. It is a figure for demonstrating the clip level of IR signal of the 8th Embodiment of this invention. It is a figure for demonstrating the clipping process of an RGBCY signal similarly. It is a block diagram for demonstrating a separation device equally. FIG. 2 is a block diagram for explaining an IR signal generation device. It is a block diagram for demonstrating the signal processing by the camera system of embodiment of this invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
Here, before describing the camera system of the present embodiment, the imaging sensor 1 used in the camera system and the imaging device 10 including the imaging sensor 1 will be described. The imaging sensor 1 and the imaging device 10 have different forms, and the first to fourth embodiments of the imaging sensor 1 and the imaging device 10 will be described before describing the camera system. The camera system is configured to be able to simultaneously capture a stereo image of a visible image and a stereo image of an infrared image using the imaging device 10. In addition, a camera system is basically configured by combining two sets of imaging devices 10 (some parts can be shared), but each imaging device 10 has a visible image and an infrared image as one optical system and one. The image pickup sensor 1 can take a picture.

  For example, as shown in FIG. 1, the imaging sensor (image sensor) 1 includes a sensor body 2 that is a CCD (Charge Coupled Device) image sensor, and red (R) and green (corresponding to each pixel of the sensor body 2. G), blue (B), infrared (IR) regions (filters of each color) arranged in a predetermined arrangement, a color filter 3, a cover glass 4 covering the sensor body 2 and the color filter 3, and a cover glass 4 and a DBPF (double band pass filter) 5 formed on the substrate 4.

  The sensor body 2 is a CCD image sensor, and a photodiode as a light receiving element is disposed in each pixel. The sensor main body 2 may be a CMOS (Complementary Metal Oxide Semiconductor) image sensor instead of the CCD image sensor.

  The sensor body 2 is provided with a color filter 3. The color filter 3 is obtained by adding an IR filter to the color filter 3x having three filter parts of R, G, and B arranged in each pixel in the general Bayer arrangement shown in FIG. Yes. In the Bayer array filter, the basic pattern is composed of 16 color filter sections of 4 rows (horizontal arrangement) × 4 columns (vertical arrangement). In terms of the number of rows and columns, the first row is F11, F12, F13, F14, the second row is F21, F22, F23, F24, the third row is F31, F32, F33, F34, and four rows The eyes are F41, F42, F43, and F44.

  In the Bayer array, eight filter units F12, F14, F21, F23, F32, F34, F41, and F43 are G, and four filter units F11, F13, F31, and F33 are R, and F22, Four filter parts F24, F42, and F44 are designated as B. The reason why the number of G filter units is twice the number of R and B filter units is based on the fact that human eyes are highly sensitive to green. In addition, even if the lower sensitivity is set to high resolution, it may not be recognized by the human eye, but if the higher sensitivity is set to high resolution, it is more likely to be recognized by the human eye. Be recognized. In this Bayer arrangement, the G filters are arranged in a checkered pattern every other row in the row direction (horizontal direction) and in the column direction (vertical direction). The filter parts are arranged without being adjacent to each other.

  On the other hand, as shown in FIG. 5B, the color filter 3 according to the present embodiment has four Rs by setting four of the eight G filter units in the Bayer array as IR. , G, 4 B, and 4 IR filters 3 a are included. That is, in the basic arrangement of 4 rows and 4 columns, four types of R, G, B, and IR filter units are arranged four by four, and the same type of filter units are adjacent to each other in the row direction and the column direction. The R, G, B, and IR filter units are arranged one by one in each column, and two types of filters of the R, G, B, and IR filter units are arranged every other row. Two parts are arranged respectively.

  Specifically, four filter units F11, F13, F32, and F34 are R, and four filter units F12, F14, F31, and F33 are IR, and four filter units F21, F23, F42, and F44 are used. The number of filter sections is G, and the four filter sections F32, F34, F41, and F42 are B.

  In this case, the reduction of the G filter unit may cause the resolution to appear to the human eye to be deteriorated, but each color including IR is evenly arranged to facilitate the interpolation (interpolation) process. . Further, the positions of the respective colors are arranged so as to be shifted by one column in the first, second and third and fourth lines. In other words, the color arrangement is reversed left and right in the first, second and third and fourth lines.

  In this manner, in the 4 × 4 arrangement, each color is arranged in each column and two colors are arranged in every other row, so the horizontal direction (horizontal direction) rather than the vertical direction (vertical direction). The resolution in the horizontal direction due to the IR filter portion can be suppressed. In addition, the color filter 3a includes a color filter 3a shown in FIG. 5B that is horizontally reversed, vertically inverted, and 180-degree rotated. Moreover, what rotated 90 degree | times clockwise and what rotated 270 degree | times may be included. However, those rotated 90 degrees and 270 degrees have higher resolution in the vertical direction than in the horizontal direction.

  Further, in the color filter 3, as in the color filter 3 b shown in FIG. 5C, 2 of the four Bs of the pattern of the color filter 3 x in the Bayer array described above without reducing G that is highly sensitive to humans. One may be IR. This color filter 3b has a basic arrangement of 4 rows and 4 columns, 8 filter units for G, 4 filter units for R, and B filter among 4 types of filter units of R, G, B, and IR. And two IR filter units are arranged, and the same type of filter units are arranged so as not to be adjacent to each other in the row direction and the column direction.

  Specifically, in the color filter 3b, eight filter units F12, F14, F21, F23, F32, F34, F41, and F43 are set to G, and four filter units F11, F13, F31, and F33 are set to G. R, two filter parts F22 and F44 are B, and two filter parts F24 and F42 are IR. In this case, the resolution of B is inferior to that of the Bayer array, but G and R have the same resolution as that of the Bayer array. Note that the color filter 3b includes a color filter 3b shown in FIG. 5C that is horizontally reversed, vertically inverted, and 180-degree rotated. Moreover, what rotated 90 degree | times clockwise and what rotated 270 degree | times become the same pattern as what turned right and left and what turned upside down.

  Further, in order to increase the resolution of the IR, two of the four B of the pattern of the color filter 3x in the Bayer arrangement described above are IR-reduced without reducing G as in the color filter 3c shown in FIG. And two of the four Rs may be IRs. That is, the color filter 3c has a basic arrangement of 4 rows and 4 columns, including 8 G filter units, 4 IR filter units, and 4 R filter units of R, G, B, and IR. Two filter units and two B filter units are arranged, and the same type of filter units are arranged so as not to be adjacent to each other in the row direction and the column direction.

  More specifically, as shown in FIG. 5D, in the color filter 3c, eight filter portions F12, F14, F21, F23, F32, F34, F41, and F43 are G, and F11, F33. These two filter sections are R, two filter sections F22 and F44 are B, and four filter sections F13, F24, F31 and F42 are IR. In this case, the resolution of R and B is lower than that of the Bayer array, but the resolution of G is maintained and the IR resolution can be ensured. Note that the color filter 3c includes a color filter 3b shown in FIG. 5D that is inverted horizontally, vertically inverted, and rotated 180 times. Moreover, what rotated 90 degree | times clockwise and what rotated 270 degree | times become the same pattern as what turned right and left and what turned upside down.

  Known filters can be used for the R filter portion, the G filter portion, and the B filter portion of the color filter 3, respectively. The transmittance spectra of the R filter unit, the G filter unit, and the B filter unit in the present embodiment are as shown in the graphs of FIGS. 2, 3, and 4. 2, 3 and 4 show the transmittance spectra of the red (R), green (G), blue (B), and infrared (IR) filters of the color filter 3, and the vertical axis. Indicates the transmittance, and the horizontal axis indicates the wavelength. The wavelength range in the graph includes a part of the visible light band and the near-infrared band, and indicates a wavelength range of 300 nm to 1100 nm, for example.

  For example, as shown by R (double line) in the graph, the R filter portion has a substantially maximum transmittance at a wavelength of 600 nm, and the long wavelength side has a substantially maximum transmittance even when the wavelength exceeds 1000 nm. It will be maintained.

  The G filter portion has a peak at which the transmittance is maximized at a portion where the wavelength is about 540 nm, as indicated by G (broken line having a wide interval) in the graph, and the transmittance at a portion where the wavelength is about 620 nm on the long wavelength side. There is a part that becomes minimal. Further, the G filter portion tends to rise on the longer wavelength side from the portion where the transmittance is minimized, and the transmittance is substantially maximum at about 850 nm. On the longer wavelength side, the transmittance is substantially maximum even when the wavelength exceeds 1000 nm. As shown by B (broken line with a narrow interval) in the graph, the B filter portion has a peak at which the transmittance is maximized at a portion where the wavelength is about 460 nm, and is transmitted through the portion at about 630 nm on the long wavelength side. There is a part where the rate is minimal. Further, the longer wavelength side tends to increase, and the transmittance is substantially maximized at about 860 nm. On the longer wavelength side, the transmittance is substantially maximized even if it exceeds 1000 nm. The IR filter section blocks light on the short wavelength side from about 780 nm, blocks light on the long wavelength side from about 1020 nm, and the portion of about 820 nm to 920 nm has a substantially maximum transmittance.

  The transmittance spectrum of each of the R, G, B, and IR filter units is not limited to that shown in FIG. 2 and the like, but the color filter 3 that is currently used generally has a transmittance close to this. It seems to show a spectrum. In addition, 1 on the horizontal axis indicating the transmittance does not mean that 100% of light is transmitted, but the color filter 3 indicates, for example, the maximum transmittance.

The cover glass 4 covers and protects the sensor body 2 and the color filter 3.
Here, the DBPF 5 is an optical filter formed on the cover glass 4. The DBPF 5 has a transmission characteristic in the visible light band, a cutoff characteristic in the first wavelength band adjacent to the long wavelength side of the visible light band, and a second wavelength that is a part of the first wavelength band. An optical filter having transmission characteristics in a band.

  That is, as shown in the graph of FIG. 2, the DBPF 5 is located at a position slightly apart from the visible light band indicated by DBPF (VR) on the long wavelength side with respect to the visible light band, as indicated by DBPF (solid line) in the graph. The transmittance of the two bands of the infrared band (second wavelength band) indicated by DBPF (IR) is high. Further, DBPF (VR) as a band having a high transmittance in the visible light band has a wavelength band of about 370 nm to 700 nm, for example. Further, DBPF (IR) as the second wavelength band having a high transmittance on the infrared side is, for example, a band of about 830 nm to 970 nm.

  In the present embodiment, the relationship between the transmittance spectrum of each filter section of the color filter 3 and the transmittance spectrum of DBPF 5 is defined as follows. That is, DBPF (IR), which is the second wavelength band that transmits infrared light in the transmittance spectrum of DBPF5, has an almost maximum transmittance in all of the R filter unit, the G filter unit, and the B filter unit. 2 is included in the wavelength band A shown in FIG. 2 where the transmittance is substantially the same in each filter unit, and is also included in the wavelength band B that transmits light at the substantially maximum transmittance of the IR filter unit. It has become.

  Here, the wavelength band A in which the transmittances of the R, G, and B filter portions are substantially the same is a portion where the transmittance difference of each filter portion is 10% or less. Note that, on the shorter wavelength side than the wavelength band A, the transmittances of the G and B filter units are lower than the R filter unit having the maximum transmittance. In DBPF5, the difference in transmittance of the R, G, and B filter portions is DBPF (VR), which is a portion with high transmittance in the visible light band, and the second wavelength band in the infrared light band. This corresponds to a portion where the transmittance for substantially blocking the light of DBPF 5 between DBPF (IR) and the portion having a high transmittance is minimal. That is, on the infrared side, the transmission of light in a portion where the difference in transmittance between the R, G, and B filter portions becomes large is cut, and the transmittance of each filter portion becomes substantially maximum on the longer wavelength side. Light is transmitted in the wavelength band A where the transmittance is substantially the same.

  From the above, in the present embodiment, DBPF 5 used in place of the infrared light cut filter has a region that transmits light not only in the visible light band but also in the second wavelength band on the infrared light side. Therefore, in color imaging with visible light, it is affected by the light that has passed through the second wavelength band. However, as described above, the second wavelength band has transmittance in each of the R, G, and B filter units. The light of different portions is not transmitted, and only the light in the wavelength band in which the transmittance of each filter unit is substantially maximized and has substantially the same transmittance is transmitted.

  Further, in the second wavelength band of DBPF 5, light in a portion where the transmittance is substantially maximum in the IR filter portion is transmitted. Therefore, when it is assumed that R, G, B, and IR filter units are respectively provided in four very close pixels irradiated with substantially the same light, the R filter unit is used in the second wavelength band. , G filter part, B filter part, IR filter part, light passes through in substantially the same manner, and as infrared light, light of substantially the same amount of light in each filter part including IR is image sensor body To the photodiode of the above-mentioned pixel. That is, the amount of light that passes through the second wavelength band on the infrared side of the light that passes through the R, G, and B filters is the same as the amount of light that passes through the IR filter section. When assumed as described above, basically, the output signal of the assumed pixel from the sensor body 2 receiving the light transmitted through the R, G, and B filters and the IR filter as described above are passed. R, G, and B obtained by cutting off the infrared light that has passed through the R, G, and B filter sections as the difference from the assumed pixel output signal from the sensor body 2 that has received the light as described above. The output signal of the visible light portion.

  Actually, as shown in each pattern of the color filter 3 (3a, 3b, 3c), one of the R, G, B, and IR filter sections is arranged in each pixel of the sensor body 2. Therefore, there is a high possibility that the amount of light of each color irradiated to each pixel is different. For example, in each pixel, using a well-known interpolation method (interpolation method), for each color of each pixel Luminance is obtained, and the difference between the interpolated R, G, and B luminances of the interpolated pixels and the interpolated IR luminance can be set as the R, G, and B luminances, respectively. Note that the image processing method for removing the infrared light component from the luminance of each color of R, G, B is not limited to this, and finally passes through the second wavelength band from each luminance of R, G, B. Any method may be used as long as it can cut the influence of light. In any method, the DBPF 5 cuts a portion where the transmittance of the R, G, B filter portion on the infrared side is different from 10%, that is, a portion where the transmittance is different from a predetermined ratio. This makes it easy to remove the influence of infrared light.

  The imaging sensor 1 is used as an imaging sensor in an imaging apparatus capable of both color photography and infrared photography. In general, it is conceivable to perform normal photographing by color photographing and perform infrared photographing using infrared light illumination that is difficult for humans to recognize without using visible light illumination at night. For example, in various types of surveillance cameras, it is conceivable to perform night photographing with infrared light using infrared light illumination at the time of night photographing in a place where night illumination is not required or preferably not illuminated at night. It can also be used for daytime photography and nighttime photography for wildlife observation.

  When infrared light photographing is used as night photographing, the amount of light is insufficient at night as in the case of visible light, so infrared light illumination is necessary. The transmittance spectrum of DBPF 5 shown in FIG. 3 takes into account the transmittance spectrum of each of the R, G, B, and IR filter sections and the emission spectrum of light for infrared illumination, for example, the infrared LED for illumination. Determined. 3, in addition to the transmittance spectra R, G, B, IR of the filter portions of the same colors as in FIG. 2 and the transmittance spectrum DBPF of DBPF5, an emission spectrum IR-light of LED illumination is shown.

  As in the DBPF shown in FIG. 2, the second wavelength band indicated by DBPF (IR), which is a part that transmits infrared light of the DBPF shown in FIG. 3, is an R filter unit, a G filter unit, and a B filter. 2 are included in the wavelength band A shown in FIG. 2 where the transmittance is substantially the same in each filter portion, and light is transmitted with the maximum transmittance of the IR filter portion. In the wavelength band B to be included.

  In addition, substantially the entire wavelength band that is the peak of the emission spectrum of infrared illumination included in both the wavelength band A and the wavelength band B described above is included in the DBPF (IR) wavelength band. Yes. Note that the second wavelength band indicated by DBPF (IR) does not have to be wider than the peak width of the optical spectrum of infrared light illumination when infrared light photography is performed under infrared light illumination instead of nighttime natural light. When the spectrum of the infrared light illumination is included in both the wavelength band A and the wavelength band B described above, the peak width of the emission spectrum of the infrared light illumination is approximately the same as the peak width having a peak at about 860, for example. You may provide the peak part of the transmittance | permeability of DBPF5 shown by DBPF (IR) as a 2nd wavelength band.

  That is, in FIG. 3, the peak in the emission spectrum of infrared illumination indicated by IR-light is on the short wavelength side of the above-described wavelength band A and wavelength band B, and the second wavelength of DBPF indicated by DBPF (IR) The band substantially overlaps the peak of the emission spectrum in the IR-light of the short wavelength side portion of the wavelength band A and the wavelength band B.

  Also, the graph shown in FIG. 4 is similar to the fluff shown in FIG. 3 in that the emission spectrum of infrared illumination is added to the graph shown in FIG. 2 and the infrared side transmittance of the DBPF5 transmittance spectrum is high. The second wavelength band indicated by a certain DBPF (IR) is matched with the peak of the emission spectrum indicated by IR-light of the above-mentioned infrared light illumination.

  In FIG. 4, infrared light illumination having a longer emission spectrum peak wavelength than that shown in FIG. 3 is used, and this peak is included in the above-described wavelength band A and wavelength band B. It exists on the long wavelength side of the wavelength band A and the wavelength band B. Correspondingly, the second wavelength band indicated by DBPF (IR) of DBPF 5 is provided so as to substantially overlap with the peak indicated by IR-light of the infrared illumination in the above-described wavelength band A and wavelength band B.

  The second wavelength band of DBPF 5 may be that shown in any of FIGS. 2, 3, and 4, and the second wavelength band is included in both wavelength band A and wavelength band B described above. It only has to be done. Further, when the wavelength band that is the peak of the emission spectrum of infrared light illumination used for nighttime infrared light photography is determined, the wavelength band is included in both the wavelength band A and the wavelength band B described above. In addition, it is preferable to match the second wavelength band of DBPF 5 with the peak of the emission spectrum of infrared illumination.

  In such an image sensor, the second wavelength band that transmits light on the infrared side of the DBPF 5 is the infrared side of each of the R, G, B, and IR filter units, and the transmittance of each filter unit is substantially the same. The maximum is included in the wavelength band A when the transmittance of each filter part is substantially the same, and is included in the wavelength band B where the transmittance of the IR filter part is substantially maximum. In other words, on the longer wavelength side of the visible light band, the transmittance of each of the R, G, and B filters is substantially maximum only for the R filter portion, and the transmittance of the G and B filter portions is substantially maximum. Due to the absence, the transmittances of the R, G, and B filter portions are not substantially the same, and different portions of light are cut by the DBPF 5.

  That is, in each filter part of R, G, B, IR, since the light of the second wavelength band is transmitted on the infrared side, the infrared side transmittance in each filter part is substantially the same, If the light in the second wavelength band is irradiated with the same light amount, the transmitted light amounts in the R, G, B, and IR filter units are the same. Thereby, as described above, the color based on the output signal from the pixel corresponding to each of the R, G, and B filter units is corrected, and the influence of the infrared light passing through the second wavelength band of the color at the time of color photographing is performed. It is possible to easily obtain an image in which the above is suppressed.

  Further, by making the second wavelength band correspond to the peak of the emission spectrum of the infrared light illumination included in the wavelength band A and the wavelength band B described above, the light of the infrared light illumination can be used efficiently, The width of the second wavelength band can be narrowed to reduce the influence of infrared light passing through the second wavelength band during color photography.

  FIG. 6 shows an imaging apparatus 10 using the imaging sensor 1 of the present embodiment. The imaging device 10 processes the imaging lens 11, the imaging sensor 1 including the DBPF 5, and the output signal 13 output from the imaging sensor 1, and performs the above-described interior processing or the second wavelength during color imaging. A signal processing unit (signal processing device) 12 that performs image processing for removing the influence of infrared light that has passed through the band, image processing such as gamma correction, white balance, and RGB matrix correction on the image signal is provided. A visible color image output signal 14 and an infrared light image output signal 15 can be output from the image processing unit.

  The lens 11 constitutes an optical system that forms an image on the imaging sensor 1 of the imaging device 10. The lens 11 is composed of a plurality of lenses, for example.

  FIG. 7 is a block diagram showing signal processing in the signal processing unit 12 of the imaging apparatus 10. Output signals from the R, G, B, and IR pixels are sent to the interior processing blocks 21r, 21g, 21b, and 21ir. In each of the interior processing blocks 21r, 21g, 21b, and 21ir, for example, as shown in FIG. 8, in the case of the above-described color filter 3b, all of the image data for each frame by interpolation processing (interpolation processing) Image data 20r in which all pixels are represented by red R, image data 20g in which all pixels are represented by green G, image data 20b in which all pixels are represented by blue B, and all pixels are infrared IR The R, G, B, and IR signals are converted so that the image data is represented by 20ir. A known method can be used as the interpolation processing method.

  Next, in the infrared light removal signal creation blocks 22r, 22g, 22b, and 22ir, in order to remove the influence of the infrared light received from the second wavelength band described above, signals of each color of R, G, and B are used. A signal to be subtracted from is generated from the IR signal. The signals created for each of R, G, and B by the infrared light removal signal creation blocks 22r, 22g, and 22b are subtracted from the signals of the R, G, and B colors. In this case, for the same pixel as described above, the IR signal is basically removed from the R, G, and B signals, which facilitates processing. Actually, the sensitivity differs for each color pixel depending on the characteristics of the filter section of each pixel, and therefore, a signal to be subtracted from each R, G, B signal for each R, G, B image from the IR signal. create.

Next, in the image processing block 23, the R, G, and B signals are converted into R, G, and B signals using a determinant to correct the colors, and a known RGB matrix process is performed. The well-known gamma correction which is the correction for the image output to a known white balance process and a display etc. which makes the output value of each signal of R, G, B become the same is performed. Next, in the luminance matrix block 24, a signal of luminance Y is generated by multiplying the R, G, B color signals by coefficients. Also, by dividing the signal of luminance Y from the signal of blue B and red R, the color difference signals of RY and BY are calculated, and Y, RY, and BY signals are output. To do.
The IR signal is basically output as a monochrome gradation image.

In such an imaging apparatus 10, as described in the above-described imaging sensor 1, it is possible to easily perform image processing for removing the influence of infrared light from a color image, and a visible color image having excellent color reproducibility. Can be obtained. In addition, the development cost of such an imaging apparatus can be reduced.
As described above, when the influence of infrared light that has passed through the second wavelength band is cut by image processing in visible light imaging, the R, G, and B filter units have mutually different transmittances. Since it is physically cut by the DBPF, in the image processing, it is only necessary to perform a process of cutting the IR light at a portion where the transmittance is substantially maximum on the infrared side of each of the R, G, and B filter portions. It will be. In this case, image processing becomes easy, and it becomes possible to obtain color image data having the same color reproducibility as when a conventional infrared light cut filter is used.

  For example, the light that passes through the R, G, and B filter sections and reaches the photodiode is the same as the visible light that has passed through the respective filter sections in the visible light region and the R, G, and B filter sections. The infrared light that has passed through the second wavelength band that is the same in the IR filter section. Therefore, for example, the output signal of the IR after the interpolation processing corrected according to the characteristics such as the sensitivity based on the filter unit of each color is subtracted from each output signal after the interpolation processing of the R, G, B of the imaging sensor 1. By doing this, it is possible to obtain color reproducibility close to that when an infrared cut filter is used.

  When the second wavelength band that transmits light includes a wavelength band in which the transmittance in each of the R, G, and B filter units has a difference of greater than 10% in transmittance, each filter unit substantially It is difficult to obtain the amount of infrared light that should be cut from the light that has passed through the image, and it is difficult to obtain image data having the same color reproducibility by using image processing as when an infrared light cut filter is used. .

  Next, an imaging device according to a second embodiment of the present invention will be described. As illustrated in FIG. 9, the imaging apparatus 10 a according to the second embodiment is configured such that the DBPF 5 is not provided in the imaging sensor 1 but the DPBF 5 is provided in the lens 11. The imaging device 10a processes the imaging lens 11 provided with the DBPF 5, the imaging sensor 1, and the output signal 13 output from the imaging sensor 1, and performs the above-described interior processing and the second wavelength during color imaging. And a signal processing unit 12 that performs image processing for removing the influence of infrared light that has passed through the band, image processing such as gamma correction, white balance, and RGB matrix correction on the image signal. A visible color image output signal 14 and an infrared light image output signal 15 can be output from the image processing unit.

  The DBPF 5 and the color filter 3 are the same as the DBPF 5 and the color filter 3 of the first embodiment. The transmittance of each of the R, G, B, and IR filter portions of the color filter 3 and the second of the DPBF 5 The relationship of the wavelength band DBPF (IR) is the same as that of the first embodiment. Therefore, unlike the first embodiment, even if the lens 11 is provided with the DBPF 5, the same operational effects as those of the imaging device 10 of the first embodiment can be obtained. The DBPF 5 is provided in the optical system of the imaging device 10a, and has a visible light band (DBPF (VR)) and a second wavelength band on the infrared side (DBPF (IR)) with respect to light reaching the imaging sensor 1. So that the light is blocked at the short wavelength side of the visible light band, the long wavelength side of the second wavelength band, and the wavelength band between the visible light band and the second wavelength band. It may be provided anywhere as long as it is.

  Next, an imaging sensor and an imaging apparatus according to the third embodiment of the present invention will be described. The imaging sensor 1 and the imaging apparatus according to the third embodiment are different in the configuration of a part of the color filter 3 and the IR component removal method from the RGB signals. The color filter 3 and the IR component removal method will be described below.

  In the present embodiment, the color filter 3e (RGBC configuration 1), for example, as shown in FIG. 10, two of the four B of the pattern of the color filter 3x in the Bayer array described above are represented as C. Two of the two R's are C, and four of the eight G's are C. That is, the color filter 3e has a basic arrangement of 4 rows and 4 columns, of 4 types of R, G, B, and C filter units, 4 G filter units, 8 C filter units, Two filter units and two B filter units are arranged, and the same type of filter units are arranged so as not to be adjacent to each other in the row direction and the column direction. Accordingly, the eight C filter units are arranged in a checkered pattern. Here, C indicates a transparent state as a clear filter part, and basically has transmission characteristics from the visible light band to the near-infrared wavelength band. Here, the visible light band C = R + G + B. Note that C, which is clear, can be referred to as white light, that is, white (W) because three colors of RGB are transmitted, and C = W = R + G + B. Therefore, C corresponds to the amount of light in substantially the entire wavelength band of the visible light band.

  Here, as shown in the graph showing the transmittance spectra (spectral transmission characteristics) of the color filter 3e and the DBPF 5 in FIG. 11, the R, G, and B filter sections have transmission characteristics on the long wavelength side of the visible light band. Even in the C filter section, which is a clear filter section, light is transmitted on the long wavelength side of the visible light band. On the other hand, by using DBPF5, similarly to the first embodiment, the infrared that transmits the longer wavelength side than the visible light band is limited to the second wavelength band, and R, The amount of light passing through the G, B, and C filter units and the DBPF 5 is almost the same (approximate) in each of the R, G, B, and C filter units, and in the visible light band, the R, G, B, and C filters The transmission characteristics differ according to the wavelength of the part.

  In the first and second embodiments as well, the infrared that transmits longer wavelengths than the visible light band is limited to the second wavelength band, and R, G, B, IR The amount of light passing through the filter unit and the DBPF 5 is substantially the same in each of the R, G, B, and IR filter units, and in the visible light band, transmission characteristics corresponding to the wavelengths of the R, G, B, and IR filter units are provided. Different.

  Thereby, also in the third embodiment, IR correction of each pixel can be performed with high accuracy, and a visible image with high color reproducibility can be generated. In other words, unlike the first embodiment, it does not include the above-described IR filter unit that has a cutoff characteristic in substantially the entire wavelength range of the visible light band and has a transmission characteristic in the infrared on a longer wavelength side than the visible light band. However, by providing the C filter unit, the IR signal can be calculated by the following equation.

In the following description, C (W), R, G, B, and IR indicate the level of the output signal from the imaging sensor 1, but C (W), R, G, and B indicate the level of the visible light band. It is assumed that the infrared component is not included.
Here, when the color filter 3e is designed as C = W≈R + G + B, and the IR signal to be removed from the RGB signals is IR ′,
IR ′ = ((R + IR) + (G + IR) + (B + IR) − (C + IR)) / 2 = IR + (R + G + B−C) / 2
IR′≈IR. IR represents an actual value obtained by measurement or the like, and IR ′ represents a value obtained by calculation. IR correction can be performed by subtracting IR ′ from each filter.
That is,
R filter (R + IR):
R ′ = (R + IR) −IR ′ = R− (R + G + B−C) / 2
G filter (G + IR):
G ′ = (G + IR) −IR ′ = G− (R + G + B−C) / 2
B filter (B + IR):
B ′ = (B + IR) −IR ′ = B− (R + G + B−C) / 2
C (= W) filter (W + IR):
W ′ = (C + IR) −IR ′ = C− (R + G + B−C) / 2
It becomes.

  Thus, even if a clear C filter unit is used instead of the IR filter unit in the color filter 3, the IR transmittance of each filter unit can be approximated by the DBPF 5, and the IR component is obtained as described above. By removing this from the signal of each filter unit, color reproducibility can be improved.

  Note that such calculation is performed by interpolation as described above, for example, R + IR, G + IR, B + IR, C + IR is obtained for each pixel, and the above calculation is performed for each pixel. The design is made such that C = W≈R + G + B. However, it is not always necessary to agree with this expression almost exactly, and if approximate, there may be a deviation due to an error or other necessity. There may be a deviation of about%.

  For C, since R + G + B, the amount of light received by the pixel is likely to be saturated and the C filter section reduces the amount of light received in the visible light band, and the infrared wavelength band and visible light band. The amount of received light may be lowered over a wavelength band including the above, or the charge accumulated with respect to the amount of received light may be reduced in the element portion constituting the pixel in each pixel. In that case, it is necessary to change the above formula accordingly.

FIG. 12 shows another arrangement of R, G, B, and C color filters. In a 2 × 2 arrangement, R, G, B, and C are evenly arranged one by one. is there.
Further, in the case of the conventional R, G, B Bayer array, as shown in FIG. 13, in the 2 × 2 array, R and B are arranged one by one and two G are arranged.
In addition, a 2 × 2 array of R, G, B, and IR color filters in which one of the conventional arrays that does not include C or IR is changed to IR is, as shown in FIG. G, B, and IR are arranged one by one.

As such color filters, RGB-C configuration 1 shown in FIG. 10, RGB-C configuration 2 shown in FIG. 12, conventional RGB array (Bayer array) shown in FIG. 13, RGB-IR array shown in FIG. In one example, there is a difference in characteristics as shown in FIG. C does not include color information such as RGB, but includes luminance information as the amount of light.
Therefore, the RGB-C (Configuration 1) sensor has a high luminance resolution due to the checkered arrangement of C, but the RGB pixels are sparse and asymmetrical, so the resolution is low and moire tends to occur. However, the resolution required for the color signal is lower than that of the luminance signal, which is 1/2 or less, so that there is no problem. Also, the sensitivity is high.

  RGB-C (Configuration 2) has the same luminance resolution and color resolution as the conventional RGB sensor, and has higher sensitivity than the RGB sensor. The RGB-IR sensor is provided with an IR that does not have transmission characteristics in the visible light band, so that the sensitivity is lower than that of the RGB sensor, and the luminance resolution is also lower. That is, the color filter having C is more likely to be advantageous in resolution and sensitivity than the color filter having IR of the first embodiment and the second embodiment.

FIG. 16 is a block diagram showing signal processing in the signal processing unit 12 of FIG. The imaging sensor 1 includes an RGB-C sensor (imaging sensor) 1 including the above-described RGB-C color filter 3e, and also includes a lens 11 and a DBPF 5 that constitute an optical system.
R + IR, G + IR, B + IR, and C + IR signals are input from the RGB-C sensor 1 to a separation device 51 that performs color separation, IR separation, and IR correction, and R, G, B, W, and IR signals are obtained and output. This process is performed based on the calculation using the above-described equation.

  Of the R, G, B, W, and IR signals output from the separation device 51, R, G, and B signals are sent to the color matrix device 52, where known RGB matrix correction and the like are performed, and RGB signals. Is output. Also. The R, G, B, W, and IR signals from the separation device 51 are sent to the luminance generation device 53, and a luminance signal is generated from each signal based on an equation for obtaining the set luminance.

  The RGB signals output from the color matrix device are input to the gamma processing and color difference generation device 54, where known gamma processing is performed, and for example, BY and RY signals are generated as the color difference signals. The Further, the signals output from the separation device 51 and the RGB-C sensor 1 are reduced in noise after being reduced in noise by the noise reduction device 56 as a signal of a predetermined wavelength band via a BPF (band pass filter) 55. The signal is amplified by the enhancement processing device 57 together with the luminance signal output from the generation device 53, and is output as a luminance signal (Y signal) of the luminance / chrominance signal through the gamma processing in the gamma processing device 58. The IR signal output from the separation device 51 is output as an IR signal via the enhancement processing device 59 and the gamma processing device 60. In the image signal processing, clip processing described later is performed, and the clip processing will be described later.

  Next, an imaging sensor and an imaging apparatus according to a fourth embodiment of the present invention will be described. In the fourth embodiment, each color of the color filter is generalized, and the color filter of the present invention is not limited to RGB-IR or RGB-C. Hereinafter, a method for removing an IR component in an imaging sensor including a color filter having a generalized four-color filter unit will be described. The four color (four types) filter units basically have different transmission characteristics according to the wavelength in the visible light band, and the wavelength band including the second wavelength band of the DBPF described above includes a visible light band. A third wavelength band having a transmittance difference of 10% or less with other filter units on the longer wavelength side is provided, and the third wavelength band includes the second wavelength band of DBPF5. ing. Thereby, when the color filter and DPBF 5 are used, the transmission characteristics according to the wavelength on the infrared side from the visible light band are approximated by the filter portions of the respective colors.

Furthermore, IR can be separated by designing a color filter under the following conditions in a filter arrangement of four types of pixels. As shown in FIG. 17, the filter arrangement is preferably a 2 × 2 arrangement, and each of the four types of filter sections A, B, C, and D is provided.
In addition, it is preferable to design each of the A, B, C, and D filter units so that the following relationship is established as much as possible in the visible wavelength band. That is, in the visible light band,
KaA + KbB + KcC + KdD≈0
And A, B, C, and D indicate the level of the output signal from the imaging sensor 1 in the visible light band of each filter unit.

In the IR region longer than the visible light band, the IR transmittance is substantially constant in the above-described third wavelength bands of the A, B, C, and D filter units. The IR transmittance may be approximately an integral multiple of a certain IR transmittance in each of the A, B, C, and D filter units. When designed in this way (here, IR transmittance is constant as described above),
Ka (A + IR) + Kb (B + IR) + Kc (C + IR) + Kd (D + IR) ≈IR (Ka + Kb + Kc + Kd)
Therefore, the IR signal is
IR ′ = (Ka (A + IR) + Kb (B + IR) + Kc (C + IR) + Kd (D + IR)) / (Ka + Kb + Kc + Kd)
Can be calculated by
The IR component contained in each pixel of A, B, C, and D can be corrected by the following calculation.
A ′ = (A + IR) − (Ka (A + IR) + Kb (B + IR) + Kc (C + IR) + Kd (D + IR)) / (Ka + Kb + Kc + Kd) = A− (KaA + KbB + KcC + KdD) / (Ka + Kb + Kc + Kd)
B ′ = B + IR− (Ka (A + IR) + Kb (B + IR) + Kc (C + IR) + Kd (D + IR)) / (Ka + Kb + Kc + Kd) = B− (KaA + KbB + KcC + KdD) / (Ka + Kb + Kc + Kd)
C ′ = C + IR− (Ka (A + IR) + Kb (B + IR) + Kc (C + IR) + Kd (D + IR)) / (Ka + Kb + Kc + Kd) = C− (KaA + KbB + KcC + KdD) / (Ka + Kb + Kc + Kd)
D ′ = D + IR− (Ka (A + IR) + Kb (B + IR) + Kc (C + IR) + Kd (D + IR)) / (Ka + Kb + Kc + Kd) = D− (KaA + KbB + KcC + KdD) / (Ka + Kb + Kc + Kd)
Here, the error is
(KaA + KbB + KcC + KdD) / (Ka + Kb + Kc + Kd). This error can be corrected in the RGB matrix.
Actually, the transmittance of the IR component for each filter unit is slightly different, so that correction is performed using a coefficient-corrected signal as described below.

  A '= A + IR * KIRa-KIRa (Ka (A + IR * KIRa) + Kb (B + IR * KIRb) + Kc (C + IR * KIRc) + Kd (D + IR * KIRd)) / (Ka * KIRa + Kb * KIRb + Kc * KIRc + Kd * KIRd)

  B '= B + IR * KIRb-KIRb (Ka (A + IR * KIRa) + Kb (B + IR * KIRb) + Kc (C + IR * KIRc) + Kd (D + IR * KIRd)) / (Ka * KIRa + Kb * KIRb + Kc * KIRc + Kd * KIRd)

  C '= C + IR * KIRc-KIRc (Ka (A + IR * KIRa) + Kb (B + IR * KIRb) + Kc (C + IR * KIRc) + Kd (D + IR * KIRd)) / (Ka * KIRa + Kb * KIRb + Kc * KIRc + Kdd + KIRd)

  D '= D + IR * KIRd-KIRd (Ka (A + IR * KIRa) + Kb (B + IR * KIRb) + Kc (C + IR * KIRc) + Kd (D + IR * KIRd)) / (Ka * KIRa + Kb * KIRb + Kc * KIRc + Kd * KIRd)

  The spectral transmission characteristics of each filter when using DBPF are as shown in FIG. As an example of the filter unit, four types of filter units of R + IR, G + IR, B + IR, and C + IR are used. However, the IR portion is constant or an integer multiple of each other, so that KaA + KbB + KcC + KdD≈0. As long as a color filter is designed, each filter unit is not limited to R + IR, G + IR, B + IR, and C + IR.

  18 shows spectral transmission combining the B filter unit and DBPF5, FIG. 19 shows spectral transmission combining the G filter unit and DBPF5, and FIG. 20 shows spectral transmission combining the B filter unit and DBPF5. FIG. 21 shows the spectral transmission obtained by combining the C (W) filter unit and the DBPF 5.

  Each spectral transmission characteristic is obtained by adding four transmittances of a visible R transmission region, a visible G transmission region, a visible B transmission region, and an IR transmission region, as shown in the above-described equations. . From this, the signal values of each visible R transmission region, visible G transmission region, visible B transmission region, and IR transmission region can be calculated from the values of four or more types of filters. The spectral transmission characteristics are determined based on the above-described formulas indicating the spectral transmission characteristics of A, B, C, and D. Six types are determined from combinations of spectral transmission characteristics of two of these filters.

  Next, a fifth embodiment of the present invention will be described. When the imaging sensor 1 that outputs a signal in the visible light band and an infrared signal on the longer wavelength side is mounted on a smartphone or the like, on the smartphone side, a circuit that processes a signal from the imaging sensor 1 is a main circuit. When the signal is on an SOC (System On Chip) functioning as an arithmetic processing unit, the RGB signal output from the image sensor includes an IR component as described above, and processing that removes the IR component is required. There is a possibility that the design change of the SOC will be forced. In addition to the SOC, the smartphone needs to be provided with a circuit for performing the IR correction described above. In these cases, when an imaging sensor capable of outputting both visible light and infrared image signals is introduced into a device such as a smartphone, an increase in cost occurs in a portion other than the imaging sensor.

  Therefore, in the present embodiment, a circuit that uses an image signal output from the visible light band from the image sensor 1 as a conventional signal is incorporated in the image sensor. As shown in FIG. 22, the imaging sensor 101 includes an imaging unit 102 and an IR correction / separation circuit 103 having substantially the same configuration as the above-described imaging sensor 1. For example, a visible RGB signal and an IR signal are output. Note that the IR correction / separation circuit 103 includes a circuit that performs processing related to clipping of a signal level described later.

In this case, since the RGB signal and the IR signal are output, the number of pins of the image sensor 101 increases. However, the increase in the number of pins can be suppressed by outputting with a serial output standard such as CSI-2. Here, the visible RGB signal is output as a RAW signal output of a RGBG Bayer array image sensor or a YUYV (YCb, YCr) signal. Further, by outputting the IR signal as a monochromatic signal, simultaneous visible and near-infrared imaging can be utilized by the RGB-IR sensor and the RGB-C sensor without changing the SOC of the smartphone or the future phone.

  The structure of the image sensor 101 is, for example, a stacked stack structure as shown in FIG. 23. For example, an integrated circuit chip constituting the IR correction / separation circuit 103 is mounted on one substrate 110, and A chip that constitutes the imaging unit 102 is placed over the top. A cover glass 111 is disposed on the imaging unit 102. Solder balls 115 are disposed on the bottom surface of the substrate 110.

  Further, as shown in FIG. 24, the structure of the image sensor 101 is such that two substrates 110 are stacked one above the other at an interval in the vertical direction, and the chip of the IR correction / separation circuit 103 is placed on the lower substrate 110. The chip of the imaging unit 102 may be disposed on the upper substrate 110, and the cover glass 111 may be disposed thereon.

  Currently, it is possible to realize a small package with a stacked stack structure. That is, the IR correction / separation circuit 103 and the imaging unit 102 can be stacked one above the other as described above to form a single package sensor. Even with this method, the image sensor 101 incorporating the IR correction / separation circuit can be realized with a single package. In this manner, a small sensor that can simultaneously capture small visible / near infrared light and can be used in a smartphone or the like can be realized without including a correction separation circuit. A smartphone has functions such as biometric authentication such as iris authentication and 3D capture using an IR sensor, and can capture a moving image / still image with a single sensor.

  That is, when performing biometric authentication or the like using an infrared sensor, for example, it may be possible to install the infrared sensor separately from the camera. Although deterioration of space efficiency etc. can be considered, these can be suppressed.

  FIG. 25 shows a schematic diagram of signal output similar to the RGB Bayer arrangement by IR correction + interpolation. In horizontal and vertical scanning, R / IR and G / B are output line-sequentially with R, IR, R, IR,... / G, B, G, B,. IR correction / IR separation is performed, and the position of the IR pixel is interpolated from the neighboring G signal to generate a G ′ signal, and R, G ′, R, G ′,... / G, B , G, B,..., And outputs visible R / G ′ and G / B signals and separated IR signals. Since the visible R / G ′ and G / B signals have the same output format as the conventional RGB Bayer array sensor, the signal processing side can perform processing using the same signal processing circuit. In addition, since the IR signal is a single color signal as it is, the signal processing unit can also perform processing by normal monochrome signal processing.

  With the above configuration, it is possible to process the output of the RGB-IR sensor with a conventional signal processing circuit with minimal changes to the processing circuit, and biometric authentication by IR imaging without increasing space on a smartphone or the like. It is possible to implement new functions such as 3D capture.

  In the above description, the RGB-IR sensor has been described. Even in the visible / near-infrared simultaneous imaging using the RGB-C sensor, an IR correction / separation circuit is incorporated in the sensor, and the conventional RGB Bayer array is used. A similar effect can be obtained by configuring the sensor output format to output a visible signal and an IR signal.

Next, a sixth embodiment of the present invention will be described.
As shown in FIG. 26, the imaging apparatus 10 according to the present embodiment processes a lens 11 that is an optical system for photographing, an imaging sensor 1 that includes a DBPF 5, and an output signal 13 that is output from the imaging sensor 1. The image processing includes image processing such as the above-described interior processing, image processing that removes the influence of infrared light that has passed through the second wavelength band during color shooting, gamma correction, white balance, and RGB matrix correction. And a signal processing unit (signal processing device: subtraction control device) 12. The signal processing unit 12 can output a visible color image output signal 14 (visible image signal) and an infrared light image output signal 15 (infrared image signal).

  The lens 11 constitutes an optical system that forms an image on the imaging sensor 1 of the imaging device 10. The lens 11 is composed of a plurality of lenses, for example.

  As shown in FIG. 27, the imaging sensor (image sensor) 1 includes a sensor body 2 that is a CCD (Charge Coupled Device) image sensor, and red (R) and green (corresponding to each pixel of the sensor body 2, for example. G), blue (B), infrared (IR) regions (filters of each color) arranged in a predetermined arrangement, a color filter 3, a cover glass 4 covering the sensor body 2 and the color filter 3, and a cover glass 4 and a DBPF (double band pass filter) 5 formed on the substrate 4.

  The sensor body 2 is a CCD image sensor, and a photodiode as a light receiving element is arranged in each pixel. The sensor main body 2 may be a CMOS (Complementary Metal Oxide Semiconductor) image sensor instead of the CCD image sensor.

  The sensor body 2 is provided with a color filter 3. Here, a Bayer array color filter that has red R, green G, and blue B regions but no infrared IR region has 16 regions of 4 × 4 in length, which is a basic pattern. Each region is a G region, four regions are R, and four regions are B. On the other hand, as shown in FIG. 28, as the color filter 3 of the present embodiment, four of the eight G regions in the Bayer array are IR regions, so that four Rs are obtained. G is 4, B is 4, IR is 4. Note that the color filter including the IR region is not limited to the color filter 3 illustrated in FIG. 28, and various color filters can be used. Each of the RGB regions is a general RGB filter, but has a transmittance peak in the wavelength range of each color and transparency in the near-infrared wavelength region. The R region was R + IR, the green region was G + IR, and the blue region was B + IR.

  The transmittance spectra of the R region, the G region, and the B region in the present embodiment are as shown in the graph of FIG. That is, the transmittance spectrum of each of the red (R), green (G), blue (B), and infrared (IR) filters of the color filter 3 is shown, the vertical axis indicates the transmittance, and the horizontal axis indicates the transmittance. It is a wavelength. The wavelength range in the graph includes a part of the visible light band and the near-infrared band, and indicates a wavelength range of 300 nm to 1100 nm, for example.

  For example, as indicated by R (double line) in the graph, the R region has a substantially maximum transmittance at a wavelength of 600 nm, and the long wavelength side maintains a substantially maximum transmittance even when the wavelength exceeds 1000 nm. It will be in the state. The G region has a peak at which the transmittance is maximized at a portion where the wavelength is about 540 nm, as shown by G (broken line having a wide interval) in the graph, and the transmittance is minimized at a portion where the wavelength is about 620 nm on the long wavelength side. There is a part that becomes. In the G region, the longer wavelength side tends to increase from the portion where the transmittance is minimized, and the transmittance is substantially maximum at about 850 nm. On the longer wavelength side, the transmittance is substantially maximum even when the wavelength exceeds 1000 nm. The region B has a peak at which the transmittance is maximized at a portion where the wavelength is about 460 nm, as shown by B (dashed broken line) in the graph, and is transmitted at a portion around 630 nm on the long wavelength side. There is a part where the rate is minimal. Further, the longer wavelength side tends to increase, and the transmittance is substantially maximized at about 860 nm. On the longer wavelength side, the transmittance is substantially maximized even if it exceeds 1000 nm. The IR region blocks light on the short wavelength side from about 780 nm, blocks light on the long wavelength side from about 1020 nm, and the portion of about 820 nm to 920 nm has a substantially maximum transmittance.

  The transmittance spectrum in each of the R, G, B, and IR regions is not limited to that shown in FIG. 29. However, the color filter 3 that is currently used generally has a transmittance spectrum close to this. It seems to show. In addition, 1 on the vertical axis indicating the transmittance does not mean that 100% of the light is transmitted, but in the color filter 3, for example, indicates the maximum transmittance.

  The cover glass 4 covers and protects the sensor body 2 and the color filter 3. Here, the DBPF 5 is an optical filter formed on the cover glass 4. The DBPF 5 has a transmission characteristic in the visible light band, a cutoff characteristic in the first wavelength band adjacent to the long wavelength side of the visible light band, and a second wavelength that is a part of the first wavelength band. An optical filter having transmission characteristics in a band. Note that the arrangement position of the DBPF 5 is not limited to the cover glass 4, and may be arranged at another location of the imaging sensor 1. Further, the arrangement position of the DBPF 5 is not limited to the image sensor 1 and may be arranged in an optical system that includes the lens 11 and forms an image on the image sensor 1.

  As shown in the graph of FIG. 29, the DBPF 5 has a visible light band indicated by DBPF (VR) as shown by a DBPF (solid line) in the graph, and a DBPF at a position slightly distant from the visible light band on the long wavelength side. The transmittance of two bands of the infrared band (second wavelength band) indicated by (IR) is high. Further, DBPF (VR) as a band having a high transmittance in the visible light band has a wavelength band of about 370 nm to 700 nm, for example. Further, DBPF (IR) as the second wavelength band having a high transmittance on the infrared side is, for example, a band of about 830 nm to 970 nm.

  In the present embodiment, the relationship between the transmittance spectrum of each region of the color filter 3 and the transmittance spectrum of DBPF 5 is defined as follows. That is, DBPF (IR), which is the second wavelength band that transmits infrared light in the transmittance spectrum of DBPF5, has almost maximum transmittance in each of the R region, the G region, and the B region. It is included in the wavelength band A shown in FIG. 29 in which the transmittance is substantially the same in the region, and is also included in the wavelength band B that transmits light with the substantially maximum transmittance in the IR region.

  Here, the wavelength band A in which the transmittance in each of the R, G, and B regions is substantially the same is a portion where the transmittance difference in each region is 10% or less. Note that, on the shorter wavelength side (wavelength band C) than the wavelength band A, the transmittance of the G and B regions is lower than the R region where the transmittance is substantially maximum. In DBPF5, the portion where the transmittance of each region of R, G, and B is different is DBPF (VR) which is a portion where the transmittance in the visible light band is high, and the second wavelength band in the infrared light band. This corresponds to the portion where the transmittance that substantially blocks the light of the DBPF 5 between the DBPF (IR) and the portion having a high transmittance is minimized. That is, on the infrared side, the transmission of light in a portion where the difference in transmittance between the R, G, and B regions is large is cut, and the transmittance in each region is substantially maximized on the longer wavelength side. The light is transmitted in the wavelength band A in which are substantially the same.

  From the above, in the present embodiment, DBPF 5 used in place of the infrared light cut filter has a region that transmits light not only in the visible light band but also in the second wavelength band on the infrared light side. Therefore, in color imaging with visible light, it is affected by light that has passed through the second wavelength band, but the second wavelength band has different transmittances in the R, G, and B regions as described above. A portion of light is not transmitted, and only the light in the wavelength band in which the transmittance of each region is substantially maximized and becomes substantially the same transmittance is transmitted.

  Further, in the second wavelength band of DBPF 5, light in a portion where the transmittance is substantially maximum in the IR region is transmitted. Therefore, when it is assumed that R, G, B, and IR regions are respectively provided in four pixels that are very close to each other and irradiated with substantially the same light, in the second wavelength band, the R region, G The light passes substantially the same in the region B, the region B, and the IR region, and the light on the infrared side has substantially the same amount of light in each of the regions including the IR. This leads to a photodiode. That is, the amount of light that passes through the second wavelength band on the infrared side of the light that passes through the R, G, and B filters is the same as the amount of light that passes through the IR region. When assumed as described above, basically, the output signal of the assumed pixel from the sensor body 2 receiving the light transmitted through the R, G, and B filters and the IR filter as described above are passed. The difference from the output signal of the pixel assumed as described above from the sensor main body 2 that has received the light is that of each of R, G, and B obtained by cutting the infrared light that has passed through the R, G, and B regions. It becomes the output signal of the visible light part.

  Actually, as shown in each pattern of the color filter 3 (3a, 3b, 3c), any one region of R, G, B, IR is arranged for each pixel of the sensor body 2. Since it is highly possible that the amount of light of each color irradiated to each pixel is different, for example, the brightness of each color of each pixel using a known interpolation method (interpolation method) in each pixel. And the difference between the R, G, and B luminances of each interpolated pixel and the interpolated IR luminance can be set as the R, G, and B luminances, respectively. Note that the image processing method for removing the infrared light component from the luminance of each color of R, G, B is not limited to this, and finally passes through the second wavelength band from each luminance of R, G, B. Any method may be used as long as it can cut the influence of light. In any method, the DBPF 5 cuts a portion where the transmittance of the R, G, B region on the infrared side is different from 10%, that is, a portion where the transmittance is different from a predetermined ratio. Processing that removes the influence of infrared light becomes easy.

  The imaging sensor 1 is used as an imaging sensor in an imaging apparatus capable of both color photography and infrared photography. In general, it is conceivable to perform normal photographing by color photographing and perform infrared photographing using infrared light illumination that is difficult for humans to recognize without using visible light illumination at night. For example, in various types of surveillance cameras, it is conceivable to perform night photographing with infrared light using infrared light illumination at the time of night photographing in a place where night illumination is not required or preferably not illuminated at night. It can also be used for daytime photography and nighttime photography for wildlife observation.

  When infrared light photographing is used as night photographing, the amount of light is insufficient at night as in the case of visible light, so infrared light illumination is necessary. The transmittance spectrum of DBPF 5 shown in FIG. 29 takes into account the transmittance spectrum of each region of R, G, B, and IR and the emission spectrum of light for infrared light illumination, for example, the infrared light LED for illumination. decide.

  In such an imaging sensor, the second wavelength band that transmits light on the infrared side of the DBPF 5 is the infrared side of each of the R, G, B, and IR regions, and the transmittance of each region is substantially maximum. Thus, it is included in the wavelength band A in which the transmittance in each region is substantially the same, and is included in the wavelength band B in which the transmittance in the IR region is substantially maximum. In other words, on the longer wavelength side than the visible light band, the transmittance of each of the R, G, and B filters is substantially maximum only for the filter portion of R, and the transmittance of the G and B regions is not substantially maximum. As a result, the transmittances of the R, G, and B regions are not substantially the same, but different portions of light are cut by the DBPF 5.

  That is, in each of the R, G, B, and IR regions, light in the second wavelength band is transmitted on the infrared side, so that the infrared side transmittance in each region is substantially the same. If the light having the same wavelength band is irradiated with the same light amount, the transmitted light amounts in the R, G, B, and IR regions are the same. Thus, as described above, the color based on the output signal from the pixel corresponding to each of the R, G, and B regions is corrected, and the influence of the infrared light passing through the second wavelength band of the color at the time of color photographing is affected. A suppressed image can be easily obtained.

  Further, by making the second wavelength band correspond to the peak of the emission spectrum of the infrared light illumination included in the wavelength band A and the wavelength band B described above, the light of the infrared light illumination can be used efficiently, The width of the second wavelength band can be narrowed to reduce the influence of infrared light passing through the second wavelength band during color photography.

  That is, by using the DBPF 5, it is possible to perform highly accurate correction by subtracting the value of each IR signal from the value of each RGB signal of the imaging sensor 1. Here, before describing the signal processing unit 12 in detail, an imaging processing method in the imaging apparatus will be described. For example, the light receiving components of the pixels of each color of the image sensor 1 are in a state in which an IR component is added to the components of each color as shown below.

R pixel R + IR
G pixel G + IR
B pixel B + IR
IR pixel IR

  Therefore, as shown below, IR correction is performed by removing the IR component from the light receiving components of the RGB pixels excluding the IR pixels.

R signal (R pixel output) − (IR pixel output) = (R + IR) −IR = R
G pixel (R pixel output) − (IR pixel output) = (G + IR) −IR = G
B pixel (R pixel output) − (IR pixel output) = (B + IR) −IR = B
As a result, the IR component that passes through the DBPF 5 and passes through the color filter can be excluded from each color region other than the IR of the color filter.

  However, the R pixel, the G pixel, and the B pixel have different sensitivities to each light source, and each pixel of the image sensor 1 has a dynamic range, and charges that exceed the dynamic range cannot be read out. The output is clipped and capped. That is, when the input light exceeds the dynamic range, the output signal is clipped and cut off. As a result, as described below, errors occur in the corrected R signal, G signal, and B signal, causing problems such as unnatural luminance level (the luminance of the highlight portion decreases) and coloring in highlights. .

  30 to 32 are diagrams for explaining a problem when the IR component is subtracted from each color component in a state where the dynamic range is exceeded. FIG. 30 shows the case of R, and FIG. And FIG. 32 shows the case of B. In the graphs shown in FIGS. 30 to 32, the vertical axis indicates the output level of the signal from each color pixel of the image sensor 1. The horizontal axis indicates the time lapse of the output level of one pixel of the image sensor 1, or the position on the pixel column (for example, the position of each pixel on the Y axis). Here, for example, the horizontal axis is the pixel position on the Y axis. Therefore, the graph shows the change in the output level of the signal depending on the position on the Y axis in each pixel of each color. FIG. 33 shows the output level of the IR signal for each color shown in FIGS. 30 to 32, and the position of each pixel on the Y-axis is the same as in the case of each RGB pixel in each graph described above. Indicates the output level.

  The upper graph shown in each of FIGS. 30 to 32 shows the output level of a signal that exceeds the dynamic range with the difference in position on the Y axis, that is, the output level of the signal of each pixel is As the position on the Y-axis changes to the right, it rises and then descends. In RGB except IR, the clipped state exceeds the dynamic range. Each pixel of the image sensor 1 can read the charge only up to the pixel saturation level, and the output level is clipped without reading the charge above the pixel saturation level. Further, since the output level of each RGB pixel signal includes the output level of the IR signal, it becomes R + IR, G + IR, and B + IR, respectively, and the RGB output level is RGB from the output level of the IR signal. It is higher by a single output level not including IR. Therefore, IR pixels do not exceed the dynamic range, and RGB easily exceeds the dynamic range.

  In FIGS. 30 to 32, a graph indicated by a dotted line exceeding the pixel saturation level indicates an output level when not clipped. The lower graphs shown in FIGS. 30 to 32 show a case where the IR output level is subtracted from the output level including the upper IR. Here, the upwardly convex portion indicates a case where the output level of the IR signal is subtracted from the output level of each of the RGB signals when not clipped as indicated by the dotted line. However, since RGB signals before subtraction are actually clipped, the output level of the IR, which has a mountain shape with a difference in position, is subtracted upward. As shown by the arrow from the lower graph, the lower graph becomes a state indicated by the lower graph.

  In this way, the output level should be a mountain-like output level where the output level at the center of each pixel in a row becomes higher, but the original output level is clipped beyond the dynamic range as described above, and the pixel saturation level is horizontal. By subtracting the output level of the IR that has a mountain shape without exceeding the dynamic range from the output level that has become, the portion with the highest output level is originally recessed.

  When the pixel saturation level is exceeded, the output level of the image signal of the pixel of each color reaches its peak, but at that time IR is still lower than the pixel saturation level, and the brightness is exceeded after other colors exceed the pixel saturation level. The higher the value, the higher the IR output level. That is, even if the luminance is further increased, the output level of each RGB pixel signal is not increased due to clipping, but the IR output signal subtracted from these is increased. Therefore, the output level obtained by subtracting the IR output level from the RGB output levels decreases as the luminance increases. As a result, the output level drops in the portion where the output level should be the highest, and the brightness drops in the highlight portion. In addition, the portion where all the RGB output levels exceed the pixel saturation level should be white, but there is a difference in the output level when convex downward in each RGB, and the highlight portion does not become white. It will be in a colored state.

  Therefore, when the output level of the IR signal is subtracted from the output level of the RGB pixel signal, the output level of the RGB pixel signal exceeds the dynamic range, and is clipped at the pixel saturation level. Subtract after lowering the output level of IR to be subtracted. That is, in the output level value of the IR signal subtracted from each RGB output level, as shown in FIGS. 33 and 34, the output level of the IR signal subtracted from the output level of each RGB pixel signal. Sets a limit value corresponding to each RGB signal (each component) so that the output level is clipped even if it is below the pixel saturation level without exceeding the dynamic range. The output level as described above is in a state of being clipped (limited so as to reach a peak) with a set limit value.

  In this way, when the IR signal clipped with the limit value set for each RGB is subtracted from each RGB signal, the IR signal to be subtracted is clipped with the limit value unlike the cases shown in FIGS. Therefore, as shown in FIGS. 35A, 35B, and 35C, when the luminance increases, the output level of each RGB signal does not decrease, and the signal saturation after subtraction is reduced. Clipped by level. This signal saturation level is a value obtained by subtracting each limit value corresponding to RGB of the IR signal from the pixel saturation level of each RGB signal. Therefore, a phenomenon such as a decrease in luminance at the highlight portion does not occur. Note that the signal saturation level that occurs after subtracting the IR signal of each RGB signal is less than the pixel saturation level because the limit value of the IR signal is subtracted from the pixel saturation level when each RGB signal exceeds the dynamic range. It becomes a low level. Further, the signal saturation level of each RGB signal after the IR signal subtraction differs for each color as described later. Note that the position where the output level of the IR signal shown in FIG. 34 is clipped (IR signal limit value) differs for each RGB signal to be subtracted and changes depending on the situation.

  That is, the limit value (clip level) of the output level of the IR signal when correcting each RGB signal depends on the spectral sensitivity of the image sensor 1, the color temperature of the light source, and the type of light source corresponding to each color of RGB. The appropriate clip level varies. This level is difficult to obtain uniformly by calculation, but can be determined by measuring the output level of each RGB signal and the output level of the IR signal output from the sensor for each light source (color temperature). Note that the spectral sensitivity of the image sensor 1 is determined by the image sensor 1. In addition, the color temperature is determined to some extent depending on the type of light source. Therefore, it is necessary to obtain the limit value of the IR signal value corresponding to each RGB signal to which the IR output level is clipped, based on the color temperature.

  In the camera, the level of each RGB signal changes due to the change in the color temperature of the light source. For example, for a light source with a low color temperature, the R signal increases and the B signal decreases. For a light source having a high color temperature, the B signal increases and the R signal decreases. As a result, when the color temperature is low, the image becomes reddish, and when the color temperature is high, the image becomes bluish. For this reason, the color reproducibility changes depending on the color temperature change of the light source. In order to stabilize this color reproducibility, white balance processing (WB) is performed to keep the RGB signal levels constant.

  The white balance processing is performed by measuring the color temperature of the light source from the color signal and adjusting the gain of each RGB color signal. At present, a method of performing white balance detection from an image signal and controlling based on the detection result is common. For example, an RGB gain adjustment circuit (included in the control circuit 21 in FIG. 38) and a white balance detection circuit 26 (shown in FIG. 38) constitute a feedback control loop. -Y signal, BY signal, or R, G, B signals are integrated. When integrating the RY signal and BY signal, control is performed so that these integral values become 0. When integrating the R, G, and B signals, the integral values are made equal. Control the gain of each RGB signal.

  At this time, the respective ratios of the R signal, the G signal, and the B signal can be obtained from the gain of each RGB signal, which is information for determining the color temperature of the light source. Based on this information, the limit value (clip level) for each RGB signal of the IR signal at the time of IR correction is determined.

  The video signal component method includes an RGB method using RGB corresponding to the three primary colors as a component and a color difference method using a luminance-color difference converted from RGB into a luminance signal and a color difference signal. As the color difference method, a method using Y, Cb and Cr, a method using Y, Pb and Pr are known. Y is luminance, Cb and Pb are (B (blue) -Y (luminance)) multiplied by a coefficient, and Cr and Pr are coefficients (R (red)-Y (luminance)). Multiplication is performed. Cb and Pb are BY signals, and Cr and Pr are RY signals.

For example, Y, Cb, and Cr are expressed by the following equations for R, G, and B.
Y = 0.299 * R + 0.587 * G + 0.114 * B
Cb = 0.564 * (B−Y) = − 0.169 * R−0.331 * G + 0.500 * B
Cr = 0.713 * (R−Y) = 0.500 * R−0.419 * G−0.081 * B
The color difference method reconstructs an RGB signal into a component that represents brightness (luminance) and a component that represents the difference between two color signals and the luminance signal (color difference). However, since the amount of color difference information is reduced by half during transmission, the amount of processing is 2/3 compared to RGB.

  Hereinafter, a method for setting the clip level (limit value) will be described using an example. First, in white balance processing, processing for obtaining the gain of each RGB signal will be described. The white balance detection is performed in this region by setting a region considered white and detecting the set white region. That is, when the BY signal and the RY signal of the color difference signal of each pixel in the region considered to be white are integrated, the BY signal and the R− signal are set so that the integrated value becomes zero. Adjust the gain of the Y signal. Alternatively, the gains of the R, G, and B signals are adjusted so that the values obtained by integrating the RGB signals are equal to each other.

An example of a region considered white in white balance detection is shown in FIGS. As shown in FIG. 36A, on the BY and RY planes, a white detection area is set in the vicinity of the movement locus due to the color temperature of the point that becomes white. In addition, as shown in FIG. 36B, a white detection range (for example, 70% or more and less than 105% of the white level) is set with respect to the luminance signal level. In the color difference method, when the color difference signal is in the white detection area and the luminance signal is in the white detection range, the pixel is in the white area, and R, G of the pixels in this area , B signals or color difference type BY signals and RY signals are integrated.
Alternatively, the gain of each R, G, B signal is adjusted so that the integral values of R, G, B are equal.

For example, the adjusted gains of R, G, B when the gains of the R, G, B signals are adjusted so that the integrated values of R, G, B are equal become information indicating the color temperature information. . In the white balance processing, the RGB signals after the white balance processing can be obtained by multiplying the corresponding RGB signals by the RGB gains thus adjusted. That is, each RGB signal after white balance is
(R signal after WB) = (Gain of R signal) × (R signal before WB)
(G signal after WB) = (Gain of G signal) × (G signal before WB)
(B signal after WB) = (Gain of B signal) × (B signal before WB)
It becomes.

  Next, correction of the value of the IR signal subtracted from each of the R (R + IR), G (G + IR), and B (B + IR) signals using the RGB gains adjusted by the white balance processing as described above. A method for obtaining a limit value as a value will be described. The limit value of the IR signal subtracted from the R (R + IR), G (G + IR), and B (B + IR) signals is the pixel saturation of the R (R + IR), G (G + IR), and B (B + IR) color signals. This value is the upper limit when the value of the IR signal is subtracted from each color signal when the level exceeds the level and clipping is performed at the pixel saturation level.

  The limit value (clip level) corresponding to each RGB signal of the IR signal is such that the saturation level of the pixel is Lsat, and when white is imaged at a certain color temperature, the ratio of the R signal to the IR signal is Kr. When the limit value (clip level) of the IR signal in the R signal is Lclip-R, it is expressed by the following equation.

Lsat = Lclip-R + Kr * Lclip-R
Lclip-R = Lsat / (1 + Kr)
Similarly, the ratio of the G signal to the IR signal is Kg, the ratio of the B signal to the IR signal is Kb, and the limit value (clip level) of the IR signal in the G signal is Lclip-G. , If the limit value (clip level) of the IR signal in the B signal is Lclip-B, then Lclip-G = Lsat / (1 + Kg)
Lclip-B = Lsat / (1 + Kb)
It becomes.
Now, when imaging white, assuming that the G signal ratio to the R signal is Kg / r and the B signal ratio to the R signal is Kb / r, as described above,
(R signal after WB) = (Gain of R signal) × (R signal before WB)
(G signal after WB) = (Gain of G signal) × (G signal before WB)
(B signal after WB) = (Gain of B signal) × (B signal before WB)
And therefore
(R signal before WB) = (R signal after WB) / (R signal gain)
(G signal before WB) = (G signal after WB) / (G signal gain)
(B signal before WB) = (B signal after WB) / (Gain of B signal)
It becomes.

Also, the R signal, G signal, and B signal after WB are equal to white. Therefore,
Kg / r = (R signal gain) / (G signal gain)
Kb / r = (R signal gain) / (B signal gain)
And also
Kg = Kr x Kg / r
Kb = Kr × Kb / r
Because
Lclip-G = Lsat / (1 + Kr × Kg / r)
Lclip-B = Lsat / (1 + Kr × Kb / r)
Thus, the red clip level Lclip-R, the green clip level Lclip-G, and the blue clip level Lclip-B of the IR signal can be obtained. Kb / r can be used as a parameter indicating the color temperature. At each color temperature, Kr with respect to Kb / r is measured and recorded in advance in a memory or the like, so that Kb / r and this Kb / r are obtained based on the gain obtained by white balance detection and the above formula. The clip level of the IR signal for color correction can be determined from Kr.

  In addition, when IR correction (calculation of IR signal clip level) is performed as described above, signal saturation level at the highlight of each RGB signal (corrected from each RGB signal clipped by reaching the pixel saturation level) The level of each signal when the subsequent IR signal (limit value: clip level) is subtracted is not necessarily the same. This causes the highlight to be colored and causes the image quality to deteriorate. For this reason, after white balance, as shown in FIGS. 37A, 37B, and 37C, correction is performed so that the highlight portion of RGB is clipped at the same level (RGB clip level). This eliminates the unnaturalness of highlighting and brightness gradation.

In FIG. 37, the vertical axis represents the output level when the IR signal is subtracted from the RGB (R + IR, G + IR, B + IR) signals, and the horizontal axis represents, for example, the position of the pixel on the image sensor 1 in the Y-axis direction. Or the time passage of one pixel is shown. In each graph of FIG. 37, when the RGB color signals have reached the pixel saturation level, the color temperature of the light source does not change, and the RGB clip levels (limit values) obtained as described above are constant. , I is limited (clipped) by a limit value (clip level) from each RGB signal of pixel saturation level
When the R signal is subtracted, each RGB signal has a constant pixel saturation level and each IR signal limit value corresponding to RGB is constant, so that the RGB signal after subtraction is in a constant state, that is, the signal saturation level. It becomes a certain state. However, as shown in FIG. 37, the signal saturation level is different for RGB, and as it is, white is not white because the output level of the RGB signal is different in the highlight portion. Therefore, the signal saturation level of the R signal having the lowest signal saturation level is set to the RGB clip level common to the RGB signals (R signal saturation level), and the signal saturation levels of the RGB signals are adjusted to the common RGB clip level.

  That is, as shown in FIG. 37A, by subtracting the IR signal in which the clip level (limit value) corresponding to the R signal is set from the R signal, the signal saturation level ( Clip level). Since the R signal has the lowest signal saturation level of each RGB signal after subtracting the IR signal, the signal saturation level and B of the G signal after subtracting the IR signal are based on the R signal as described above. As shown in FIGS. 37 (b) and 37 (c), the signal saturation level of the signal is lowered in accordance with the RGB clip level equal to the signal saturation level of the R signal, so that the RGB signals after the IR signal subtraction is performed. Adjust the signal saturation level. Thereby, the high level of each signal in a highlight part becomes the same, and coloring of a highlight can be prevented.

  In the configuration described above, the IR signal is clipped and then subtracted from the R signal, G signal, and B signal. However, the IR signal is a level at which the R signal, G signal, and B signal are saturated. (IR signal clipping level) or more, the IR signal may be subtracted from the R, G, and B signals after the gain is lowered by a multiplier. In this way, by configuring so as to suppress the subtraction amount, unnaturalness of luminance gradation of the R signal, the G signal, and the B signal in the highlight portion may be prevented. Also, a process of generating a luminance signal from the RGB signal and reducing the gain of the RY signal and BY signal to erase the color at a level higher than a portion where coloring occurs due to a difference in the signal saturation level of the RGB signal. Therefore, coloring may be prevented.

  FIG. 38 is a block diagram showing signal processing in the signal processing unit 12 (shown in FIG. 26) of the imaging apparatus 10 (shown in FIG. 26). If the output signals of the R, G, B, and IR pixels from the image sensor 1 (input signals from the image sensor 1 in this signal processing) are usually RAW outputs, R, G, B, and IR are lines. Since the signals are output sequentially or dot-sequentially, for example, a synchronization circuit (not shown) for each color signal is provided at the input portion of the RAW signal from the image sensor 1.

  In this case, in the image data for each frame by interpolation processing (interpolation processing) in the synchronization circuit, image data in which all pixels are represented by red R and image in which all pixels are represented by green G, respectively. Data, R, G, B, IR signals are converted so that all pixels are image data represented by blue B and all pixels are image data represented by infrared IR. In other words, R, G, B, and IR signals are output in all pixels. A known method can be used as the interpolation processing method.

  That is, the signal processing unit 12 includes synchronization circuits for R + IR, G + IR, B + IR, and IR sensor outputs (not shown), and the R + IR, G + IR, B + IR, and IR sensor outputs of FIG. It is a signal after passing. The signal processing unit 12 outputs the IR signal subtracted from the RGB signal from the limiters 27r, 27g, and 27b and the limiters 27r, 27g, and 27b for clipping at the clip level (limit value) determined for each RGB. Multipliers 28r, 28g, and 28b for multiplying the IR signal by a correction value, and the clipped IR signals output from the multipliers 28r, 28g, and 28b are subtracted from the R + IR, G + IR, and B + IR signals. Subtractors 23r, 23g, and 23b, multipliers 24r, 24g, and 24b for multiplying the RGB signals obtained by subtracting the IR signal by the RGB gains for white balance, and the white balance while the IR signal is subtracted Limiters 25r, 25g, and 25b for matching the signal saturation levels in the RGB signals for which It is provided.

  In addition, the signal processing unit 12 includes a control circuit 21 that calculates and outputs the clip level (limit value) of the IR signal for each RGB signal in the limiters 27r, 27g, and 27b. The control circuit 21 outputs correction values to the multipliers 28r, 28g, and 28b, and outputs RGB gains calculated for white balance to the multipliers 24r, 24g, and 24b, and the limiters 25r, 25g, and 25b. Output an RGB clip level (R signal saturation level) for matching the signal saturation level of each RGB signal. The signal processing unit also includes a white balance detection circuit 26 for obtaining a white balance gain from the RGB output signals from the signal processing unit 12.

  IR signals as IR sensor outputs are respectively sent to an R limiter 27r, a G limiter 27g, and a B limiter 27b provided in the signal processing unit 12, and for IR signals set for each color as described above. Clipped with the limit value (clip level). At this time, the R limiters 27r and G for the IR signals calculated by the control circuit 21 correspond to the red clip level Lclip-R, the green clip level Lclip-G, and the blue clip level Lclip-B, respectively. The limiter 27g and the B limiter 27b output to the limiter 27g, the limiter 27g for G, and the limiter 27b for B, respectively. Thereby, the output level exceeding the clip level (limit value) is clipped at the clip level by the limiters 27r, 27g, and 27b.

  The values output from the R limiter 27r, the G limiter 27g, and the B limiter 27b are IR signals that are subtracted from the RGB signals. The IR signals are multiplied by multipliers 28r and 28g. 28b, the correction value output from the control circuit 21 is multiplied. In this configuration, the IR component included in the R pixel, G pixel, and B pixel is substantially the same level as that of the IR pixel. However, the aperture difference between the R pixel, the G pixel, and the B pixel, or the amplifier in the sensor. There may be some error in the signal level due to variations in gain. That is, the values output from the limiters 27r, 27g, and 27b tend to be slightly larger than necessary values, for example, and the correction values are multiplied by the multipliers 28r, 28g, and 28b corresponding to the RGB signals. To correct.

  The subtractors 23r, 23g, and 23b subtract the IR signals corresponding to RGB that have been clipped and corrected in this way from the R + IR, G + IR, and B + IR signals. Each multiplier 24r, 24g, 24b multiplies the signal output from each subtractor 23r, 23g, 23b by each gain of RGB calculated by the white balance processing to obtain white balance. The RGB gains calculated by the control circuit 21 based on the RGB signals detected by the white balance detection circuit 26 are input to the multipliers 24r, 24g, and 24b. The RGB signals output from the multipliers 24r, 24g, and 24b are clipped by subtracting the clipped IR signal.

  That is, when the IR signal limited by the limit value as described above is subtracted in a state where each RGB signal is clipped at the pixel saturation level, the signal saturation level is a value obtained by subtracting the limit value from the pixel saturation level. Clipped with. If the signal saturation levels of these RGB signals are different, the highlight portion is colored. Therefore, the limiters 25r, 25g, and 25b are the same for each RGB signal in order to match the RGB signal saturation levels. The limit value is set so as to be the RGB clip level. Here, the signal saturation level of the R signal is set to the RGB clip level as a limit value, and the clip levels of the G signal and the B signal are matched.

  The signal processing as described above will be described by taking an R (R + IR) signal as an example. The R signal output from the imaging sensor 1 is interpolated by a synchronization circuit and is used as an R signal for every pixel used as an image by the imaging sensor 1. Similarly, the G signal, the B signal, and the IR signal are processed by the synchronization circuit, and are converted into the G signal, the B signal, and the IR signal for every pixel used as an image.

  The IR signal is sent to the limiter 27r. The limiter 27r outputs Lclip-R, which is the limit value of the IR signal for the R signal calculated by the control circuit 21 as described above, and this becomes the limit value of the limiter 27r. Therefore, the IR signal that has passed through the limiter 27r is clipped so as to reach the limit value. That is, the output level of the IR signal exceeding the limit value becomes the limit value.

  The IR signal clipped with the limit value is sent to the multiplier 28r, multiplied by the correction value calculated (or stored) by the control circuit 21 and corrected, and then sent to the subtractor 23r. It is done. The subtractor 23r receives the above-described synchronized R signal, and the IR signal that is limited and corrected as described above, and the value of the IR signal is subtracted from the R signal. Next, the R signal from which the IR signal has been subtracted is sent to the multiplier 24r. The multiplier 24r receives the R signal and the gain of the R signal obtained by the white balance detection circuit 26 and the control circuit 21, and multiplies the R signal by the gain.

  An R signal subjected to white balance processing is output from the multiplier 24r. This R signal is input to the limiter 25r. The RGB clip level described above is sent from the control circuit 21 to the limiter 25r, which is set as a limit value, and the R signal exceeding the RGB clip level is clipped. However, in this signal processing unit 12, the RGB clip level is equal to the R signal saturation level, and the R signal from which the IR signal has been subtracted is already clipped at the R signal saturation level. The process of 25r is not necessarily required. The G signal and the B signal are clipped at the RGB clip level by the limiters 25g and 25b with the RGB clip level equal to the R signal saturation level as a limit value.

  Similarly, the G signal and the B signal are processed, and the processed RGB signals are output as an output signal 14 of a visible color image as shown in FIG. The IR signal is output as an infrared image output signal 15 after passing through a synchronization circuit. The IR signal output as the infrared image signal is not limited by the above limit value.

  The block diagram shown in FIG. 39 is a modification of the block diagram shown in FIG. The difference between the block diagram shown in FIG. 39 and the block diagram shown in FIG. 38 is that each RGB signal after the IR signal is subtracted after the white balance multipliers 24r, 24g, and 24b in the block diagram of FIG. Whereas limiters 25r, 25g, and 25b for adjusting the signal saturation level are arranged, in FIG. 39, limiters 25r, 25g, and 25b for adjusting the clip level of each RGB signal after the IR signal is subtracted. Is followed by white balance multipliers 24r, 24g, and 24b.

  Thus, in FIG. 38, the signal saturation level is adjusted in each RGB signal after white balance is achieved, whereas in FIG. 39, the white balance processing is performed after the signal saturation level of each RGB signal is adjusted. Will do.

  According to such an imaging processing system and an imaging processing method, by using DBPF5, it is possible to perform color imaging with visible light and imaging with infrared light without switching use or non-use of an infrared cut filter. In the camera (imaging device), as described above, it is possible to prevent the brightness of the highlight portion from being lowered and the highlight portion from being colored instead of white.

  Next, a seventh embodiment will be described. In the sixth embodiment described above, the clip level (control value) when the RGB-IR color filter is used has been described, but in this embodiment, the RGB-C clip level will be described.

  In the RGB-IR imaging sensor 1, each RGB signal includes an IR component, whereas the IR signal includes only the IR component. Therefore, even if each RGB signal is saturated, the IR signal is saturated. do not do. Here, if the IR component whose level increases even after the RGB signal is saturated, the RGB component is saturated as described above, and then the IR component is removed to lower the saturation level or lower. In particular, as the amount of IR light increases, the RGB values after saturation become lower. On the other hand, in the RGB-C imaging sensor 1, C = R + G + B in the visible light band. Therefore, when the range is adjusted to RGB, the C pixel reaches a saturation level as shown in FIG. become.

  In this case, when the C pixel reaches the saturation level, C <R + G + B in the visible light band. Thus, when IR ′ is obtained by calculation as described above, IR ′ = IR + (R + G + B−C) / 2. Therefore, when C reaches the saturation level, (R + G + B−C) is less than 0. Therefore, as shown in FIG. 41, IR ′ obtained by calculation is greater than the actual IR.

  When the IR ′ component obtained by calculation from R, G, B, and C is removed when C reaches the saturation level as described above, the signal level reaches the peak at the saturation level for C. The IR component can be increased even after C reaches the saturation level, and after C is saturated, IR ′> IR as shown in FIG. As shown in FIG. 42, the signal level decreases as the actual light amount level increases. Further, since the RGB signal levels other than C are also IR '> IR after C is saturated, the signal level may decrease as the actual light quantity level increases, although not as high as C.

Therefore, in the seventh embodiment, clipping is performed so that each signal does not exceed a predetermined value, and an IR signal is separated from each RGB signal.
First. R / G / B / W / IR signal level is detected.
Next, the clip level is obtained by R, G, B, and W as follows.
(W clip level) = (calculated from the saturation level of W pixel (C pixel))
(R clip level) = (W clip level) (R level + IR level) / (W level + IR level)
(G clip level) = (W clip level) (G level + IR level) / (W level + IR level)
(B clip level) = (W clip level) (B level + IR level) / (W level + IR level)

  The level of the W clip is calculated based on a saturation level at which the signal level reaches its peak when the W signal level is detected and does not increase any more. Each RGB clip level is calculated by the above formula, and the clip level is calculated when the W level reaches the saturation level.

  When such clip processing is performed, the above-described separation device 51 is a block diagram as shown in FIG. The R, G, B, and W signal levels of each pixel are separated by the color separation device 61 by interpolation or the like and sent to the level detection / clip level calculation device 63 and the clip processing device 62. The level detection / clip level calculation device 63 calculates R, G, B, and W clip levels based on the above formula and the detected levels of R, G, B, and W. The R, G, B, and W clip levels calculated by the clip level calculation device 63 are sent to the clip processing device 62, and the R, G, B, and W signal levels input to the clip processing device 62 are clip levels. If it exceeds, clipping is performed. The R, G, B, and W signal levels output from the clip processing device 62 are input to the IR correction / IR generation device 64. The IR correction / IR generation device 64 removes the IR signal from the R, G, B, and W signals and generates an IR signal. As shown in FIG. 44, the clip level calculation device 63 obtains the IR signal level by the IR matrix device 66 based on the R, G, B, and W signal levels separated by the color separation device 61, and performs IR correction. The R, G, B, and W signal levels including the IR signal and the IR component are input to the device 67, and IR correction is performed to remove the IR signal level from the R, G, B, and W signal levels. An IR signal level is input from the IR matrix device 66 to the level detection device 68, and R, G, B, and W signal levels from which IR components are removed are input from the IR correction device 67.

  The R, G, B, W, and IR signal levels detected by the level detection device 68 are input to the clip level calculation device 69, and the clip level calculation device 69 outputs the R, G, B, and W signals as described above. The clip level for each signal level is calculated. Based on the R, G, B, and W clip levels obtained in this way, the clip processing device 62 shown in FIG. 45 when the R, G, B, and W signal levels exceed the clip level. As described above, each signal level reaches its peak at the clip level.

  Based on the R, G, B, and W signals clipped in this way, the IR component is obtained and the R, G, B, and W signals including the IR component are obtained in the same manner as in the third embodiment. From the IR component. At this time, as shown in FIG. 46, the IR signal obtained by calculation is also clipped at the IR signal level when W becomes the saturation level and reaches a peak state. As a result, as shown in FIG. 47, even after the IR component is removed from each of the R, G, B, and W signal levels, the clipped state is obtained at the clip level after the IR component is removed.

  In this case, the luminance signal obtained from the R, G, B, and W signal levels is also clipped. Therefore, by using a modification of the separation device 51 that performs color separation, IR separation, and IR correction as shown in FIG. 48, the luminance level is prevented from being clipped, and even after the W pixel reaches the saturation level. It is possible to give gradation to the luminance level.

  48, as shown in FIG. 48, the color separation device 61 separates the R, G, B, and W signal levels of each pixel by an interpolation method or the like, and performs level detection / clip level calculation. It is sent to the IR signal generation device 63 and the clip processing device 62. The level detection / clip level calculation device 63 calculates R, G, B, and W clip levels based on the above formula and the detected levels of R, G, B, and W. The R, G, B, and W clip levels calculated by the clip level calculation device 63 are sent to the clip processing device 62, and the R, G, B, and W signal levels input to the clip processing device 62 are clip levels. If it exceeds, clipping is performed. The R, G, B, and W signal levels output from the clip processing device 62 are input to the first IR correction / IR generation device 64. The first IR correction / IR generation device 64 removes the IR signal from the clipped R, G, B, and W signals and generates an IR signal. The RGB signals obtained by the first IR correction / IR generation device 64 are used as color signals, for example, color difference signals. Further, the second IR correction: IR generation device 65 of this modification is provided. The R, G, B, and W signals that are not clipped from the color separation device 61 are input to the second IR correction / IR generation device 65, IR generation and IR correction are performed, and IR components are removed. RGB signals are output. Here, the RGB signal is less likely to reach the saturation level than the W signal, and it is difficult to reach the peak. By using this RGB signal to calculate the luminance, as shown in FIG. 49, it is possible to prevent the luminance level from being clipped and to give a slight gradation even in a high luminance state.

Next, an eighth embodiment will be described. In the eighth embodiment, the clip level of each of the R, G, B, and W signals is determined as in the seventh embodiment. In the eighth embodiment, as in the seventh embodiment, the clip levels of the IR signal and the W signal are determined without determining the clip levels of the R, G, and B signals. As shown in FIG. 50, IR correction is performed by an IR signal clipped with a control value by limit processing. The clip level of W is set based on the saturation level of W.
(W clip level) = (calculated from the saturation level of W pixel (C pixel))
The IR clip level is determined based on the following equation.
(IR clip level) = (W clip level) (IR level) / (W level + IR level)
Thereby, as shown in FIG. When removing the IR signal level that is clipped as described above when the W signal is substantially saturated from the W signal level that reaches the saturation level, the W signal level is clipped at a signal level lower than the saturation level. Will be.

  When performing correction to remove the IR signal clipped from each RGB signal level, assuming that each RGB signal level does not reach the saturation level within the range of use conditions, as shown in FIG. When the IR signal is removed, each signal level can be increased without being clipped because the RGB signal level can be increased even after the IR signal reaches the clip level. It is possible.

  In the separation device 51 shown in FIG. 52 of the eighth embodiment, the color separation device 61 separates the R, G, B, and W signal levels of each pixel by interpolation or the like, and the IR signal generation device 71 The R, G, B, and W signals separated by the IR correction device 64 are output. The IR signal generation device 71 obtains the clip level of the IR signal to be generated, and when the clip level is exceeded, the IR signal to be clipped is output to the IR correction device 64. In the IR correction device 64, the IR signal is removed from the R, G, B, and W signal levels.

  As shown in FIG. 53, in the IR signal generation device 71, the R, G, B, and W signals are sent from the color separation device 61 to the IR correction device 72 and the IR matrix device 73, and are generated by the IR matrix device 73. The IR signal thus output is output to the IR correction device 72 and the limit processing device 74. The IR correction device 72 performs IR correction in which the IR signal is removed from each of the R, G, B, and W signal levels. The R, G, B, W signal and IR signal thus transmitted are sent to the level detection device 75, and the R, G, B, W signal and IR signal are sent to the limit level calculation device 76, and the IR signal limit level (clip) Level) is input to the limit processing device 74. When the IR signal level is input from the IR matrix device 73 to the limit processing device 74 and the IR signal level exceeds the limit level, the limit processing device 74 is clipped to the limit level.

In the eighth embodiment, similarly to the seventh embodiment, it is possible to prevent a situation in which the signal level of R, G, B, W is lowered in a situation where the signal level is increased and W Even in a state where the signal level is saturated, it is possible to give gradation to each signal level of RGB. In addition, it is possible to give gradation to the luminance signal in a situation where the W signal level is saturated. The luminance signal generation formula is
Y = (Kr * R + Kg * G + Kb * B + Kw * W) + Kir * IR
It can be.
Sensitivity can be controlled by changing the IR ratio. Note that when IR illumination is performed at night, the visible signal disappears and only the all-pixel IR signal is obtained. In this case, it is possible to generate a high-resolution and high-sensitivity luminance signal by turning off IR correction and generating a luminance signal from signals of all pixels. When IR illumination is built in the camera, signal processing is switched in conjunction with this.
In this case, the brightness generation formula for IR illumination is
Y = Kr * R + Kg * G + Kb * B + Kw * W (IR correction OFF)
It becomes.
In each pixel, the amount of received light of the infrared illumination that passes through the second wavelength band of the DBPF 5 and each filter is larger than the amount of received light in the visible light band. The level is output.

  Next, a camera system using the imaging sensor 1 and the imaging devices 10 and 10a according to the first to fourth embodiments as described above will be described. In the camera system, two imaging devices 10 (10a) according to any of the above-described embodiments are arranged on the left and right, and a stereo image composed of two images on the left and right can be captured with visible light and infrared light, respectively. ing.

  The camera system of this embodiment will be described with reference to FIG. Light from the target (subject) 200 is collected by a lens unit (lens, optical system) 201 and converted into an electrical signal by the image sensor 1 through the DBPF 5 and the color filter 3. The signal output from the imaging sensor 1 is input to the image input unit 202, the image signal correction processing unit 203, and the correction parameter calculation unit 204 as the signal processing unit 12 described above.

  In the signal processing unit 12, signals are input / output in the order of the image input unit 202, the image signal correction processing unit 203, and the correction parameter calculation unit 204. For example, the above-described interpolation processing is performed in the image input unit 202. The image signal correction processing unit 203 performs the above-described infrared component calculation, the above-described removal (subtraction) of the infrared component from each color component, the above-described clip processing when calculating and removing the infrared component, and the like. The correction parameter calculation unit 204 determines the clip level in the above-described clip processing, etc., and outputs it to the image signal correction processing unit 203 to be used when the above-described infrared signal is removed. The correction parameter calculation unit 204 also corrects clip level and signal level correction values (for example, the above-described values) so as to approximate the signal intensity (signal level) of each visible image output from the two left and right image signal correction processing units 203. Parameters such as addition, subtraction, multiplication, and division) are set on the visible image signal, infrared image signal, and the above-described infrared signal and each color signal, and two visible image signals (two The signal level of the infrared image signal is approximated. Regarding the correction amount in the image signal correction processing unit 203, the output from the two image signal correction processing units 203 is viewed and set to each level to match the level of the image signal. The process of matching the levels of the left and right image signals can be performed on both the infrared image signal and the visible image signal.

  That is, the correction parameter calculation unit 204 determines a correction amount based on the signal levels of the image signals output from the two image signal correction processing units 203, and the image signal output from the two image signal correction processing units 203 is determined. Try to approximate the signal level. This makes it possible to recognize different parts of the subject as the same part (corresponding point), for example, because the brightness levels of the two image data are different, and prevent errors in measurement distances and errors. can do.

  Note that the image sensor 1 of the camera system preferably uses RGB-C (W) as the color filter 3. As the infrared filter of the color filter 3, a filter that does not transmit visible light but transmits infrared light can be used. As described above, the red R, green G, and blue B filter sections and the clear C (white) By using the color filter 3 having the filter portion W), a C filter portion having high visible light luminance is provided in place of the infrared filter portion having zero visible light luminance as described above. , The luminance resolution becomes high.

  In addition, the camera system includes two sets of the imaging device 10 including the lens unit 201, the DBPF 5, the color filter 3, the imaging sensor 1, the image input unit 202, the image signal correction processing unit 203, and the like as described above. Stereo images can be obtained by shooting from two locations. The two lens units 201 are arranged so that their optical axes are parallel to each other. A distance calculation unit 205 as a distance calculation device of the camera system calculates the distance to the target 200 using two left and right visible image signals and two left and right infrared image signals respectively input from the two image signal correction processing units 203. measure. In this case, the same subject (corresponding point) is determined from the two images, the parallax as a difference in the position of the same subject on the image is detected, and the distance is obtained as in the conventional case. That is, the corresponding point for measuring the difference is determined by image recognition, and the distance is calculated based on the parallax that is the difference in the position of the corresponding point in the image. In addition, this camera system includes a suspicious person detection unit 206, and determines whether or not the person is a suspicious person based on the above-described distance between the visible image and the infrared image and the target 200.

  In this case, for example, it is determined whether or not the subject person is in the restricted entry area from the distance to the person who is the subject 200 and the position on the image, and it is determined that the subject person is in the restricted entry area. If you do, you may determine that you are a suspicious person. Further, it may be determined whether or not the person is registered by face recognition based on face data of a person registered in advance.

  Such a camera system is used, for example, in a surveillance camera, and can confirm a two-dimensional image or a three-dimensional image with human eyes, and can measure a distance to a subject, for example. Thereby, it is possible to grasp the positional relationship between a person, a car, an obstacle, and the like shown in the image from the distance, and it can be used for verification (analysis) of an automobile accident. It is also possible to measure the unevenness of the human face according to the distance to each part of the human face. For example, in face recognition, three-dimensional data of a face is acquired or three-dimensional data is It can be used for authentication and identification of people.

  In addition, according to this camera system, it is possible to photograph with visible light using natural light in the daytime, perform infrared photographing using infrared illumination at night, and obtain the distance from the visible image and perform infrared imaging. The distance can be obtained from the image. At this time, the visible imaging and the infrared imaging use the same imaging sensor 1 and lens unit 201 in the same imaging device 10, and the visible image and the infrared image are captured in the same range. , Taken with the same angle of view. For example, even when switching between infrared imaging and visible imaging at the time between day and night, the position of the target 200 does not change at the time of switching, and distance measurement can be continued. In addition, processing such as alignment of an infrared image and a visible image is not required during manufacturing or maintenance of the camera system.

  In addition, when using this camera system for automatic driving of automobiles, it is possible to measure the distance with both infrared and visible images, so the distance of the headlight can be measured with the light of the headlight at night and the headlight It is possible to measure the distance in a far place where the light does not reach with an infrared image using infrared illumination in consideration of the influence on the oncoming vehicle. At this time, when measuring the distance at the boundary between the illumination range of the headlight and the illumination range of the infrared illumination, since the positions of the infrared image and the visible image are as described above, the measured distance changes. Can be prevented.

1 imaging sensor 3 color filter 5 DBPF (optical filter)
11 Lens (optical system)
12 Signal processor (Signal processing device: Subtraction control device)
101 Lens unit (optical system)
105 Distance calculation unit (distance calculation device)

Claims (6)

Two imaging sensors in which a light receiving element is arranged in each pixel;
A transmission characteristic in the visible light band, a cutoff characteristic in the first wavelength band adjacent to the long wavelength side of the visible light band, and a second wavelength band that is a part of the first wavelength band; Two optical filters having transmission characteristics and provided corresponding to the two imaging sensors,
Spectral transmission characteristics corresponding to the wavelength in the visible light band are different from each other, and the transmittance in the second wavelength band is approximately similar to each other. Two color filters,
Each of the image sensors has a lens for connecting an image, and two optical systems provided corresponding to the two image sensors,
A signal processing device capable of processing a signal output from each imaging sensor and outputting two visible image signals and two infrared image signals;
Using the two visible image signals and / or two infrared image signals output from the signal processing device, the distance to the subject reflected in the visible image by the visible image signal and / or the infrared image by the infrared image signal is determined. A distance calculating device for calculating,
The signal processing device is a red based on an infrared component corresponding to the second wavelength band by using a signal output from the imaging sensor corresponding to light transmitted through each type of the filter unit of the color filter. Calculating an outer image signal and a visible image signal based on at least three color components corresponding to the visible light band from which the infrared component is subtracted;
In addition, when subtracting the infrared component from the color component, if the color component is equal to or higher than a pixel saturation level, control is performed to correct the infrared component subtracted from the color component so as to decrease the value. A camera system comprising a subtraction control device.   The color filter includes the red filter unit that transmits the infrared component and the red component light in the visible light band, and the green filter that transmits the infrared component and the green component light in the visible light band. Part, the blue filter part that transmits the infrared component and the blue component light in the visible light band, and the white component light that combines the infrared component, the red component, the green component, and the blue component The camera system according to claim 1, further comprising: the white filter unit that transmits light. The subtraction control device is:
The correction for lowering the value of the infrared component is controlled so that the value of the infrared component does not become higher than a limit value determined based on the color component. Camera system.   The camera system according to claim 3, wherein the subtraction control device determines the limit value of the infrared component for each color component based on a gain of each color obtained by white balance processing.   The signal processing device controls the signal saturation level corresponding to a value obtained by subtracting the limit value of the infrared component from the pixel saturation level of the color component to be substantially the same for each color. Item 5. The camera system according to Item 3 or Item 4.   The said signal processing device correct | amends so that the signal level of two visible image signals and / or the signal level of two infrared image signals may approximate each other. The camera system according to item.

Images