JP6734647B2 - Imaging device - Google Patents

Description

The present invention relates to an imaging device that captures a visible image and an infrared image.

2. Description of the Related Art Conventionally, in an imaging device such as a surveillance camera (hereinafter, simply referred to as an imaging device) that continuously shoots day and night, infrared light is detected and shooting is performed at night. Since a photodiode, which is a light receiving part of an image sensor such as a CCD sensor or a CMOS sensor, can receive light in the near infrared wavelength band of about 1300 nm, an image pickup device using these image sensors can detect light in the infrared band. In principle, it is possible to take a picture.

Since the wavelength band of light with high human visibility is 400 nm to 700 nm, when near infrared light is detected by the image sensor, the image will appear reddish to human eyes. Therefore, when shooting in a bright place in the daytime or indoors, an infrared cut filter that blocks light in the infrared band is provided in front of the image sensor in order to match the sensitivity of the image sensor with the human visibility. It is desirable to remove light with a wavelength of 700 nm or more. On the other hand, when shooting at night or in a dark place, it is necessary to perform shooting without providing an infrared cut filter.

As such an image pickup device, an image pickup device in which an infrared cut filter is manually attached/removed and an image pickup device in which an infrared cut filter is automatically inserted/removed are conventionally known. Further, Patent Document 1 discloses an imaging device that does not require the insertion and removal of the infrared cut filter described above.

For example, a second wavelength band that has a transmission characteristic in the visible light band, has a blocking characteristic in a first wavelength band adjacent to the long wavelength side of the visible light band, and is a part of the first wavelength band. An optical filter having a transmission characteristic has been proposed (for example, refer to Patent Document 1). According to this filter, light can be transmitted in both the visible light band and the second wavelength band apart from the visible light band on the long wavelength side of the visible light band, that is, the infrared side. For example, the second wavelength band overlaps with the wavelength band of infrared illumination, and is an optical filter that enables both visible light photography and nighttime infrared light photography using infrared light illumination. Hereinafter, an optical filter that transmits light in the visible light band and the second wavelength band on the infrared side as described above and blocks light in other wavelength bands is referred to as a DBPF (double band pass filter). To call. The DBPF is not limited to the one manufactured by the manufacturing method described in Patent Document 1.

Japanese Patent No. 5009395

By the way, in the DBPF of Patent Document 1, the light of the second wavelength band included in the infrared (near infrared) wavelength band (the relatively narrow wavelength band included in the infrared wavelength band) is not always blocked Will be transmitted. That is, unlike the case of using an infrared cut filter that cuts the longer wavelength side than the visible light band, in the imaging in the visible light band, the influence of the infrared light transmitted through the second wavelength band is not small. Become.

A color filter is used for an image sensor that performs color imaging as imaging in the visible light band. In the color filter, regions (filter portions) of red, green and blue colors are arranged in a predetermined pattern corresponding to each pixel of the image sensor. Each of these color regions basically has a peak of light transmittance in the wavelength band of each color and limits (blocks) the transmission of light in the wavelength bands of other colors.

However, on the longer wavelength side than the visible light band, although the transmittance differs depending on the region and wavelength of each color, light is basically transmitted. Therefore, if the infrared cut filter is used, there is no problem because light on the longer wavelength side than the visible light band is cut, but when infrared light is transmitted in the second wavelength band on the infrared side as in the DBPF described above, This infrared light passes through the color filter and reaches the photodiode (light receiving element) of the image sensor, increasing the amount of electrons generated by the photoelectric effect in the photodiode.

Here, in performing both color photography with visible light and photography with infrared light illumination, for example, in the color filter in which regions of each color of red, green, and blue are arranged in a predetermined pattern, It is conceivable to provide an infrared light region (infrared region) having a transmittance peak in the second wavelength band. For example, an array (pattern) of color filters is composed of four regions of red R, green G, blue B, and infrared IR. In this case, since the infrared light region blocks light in the visible light band and mainly transmits light in the second wavelength band, the light passing through the infrared light region of the color filter is used. It is conceivable to remove the infrared light component from the image signal of each color of red, green, and blue by using the image signal of infrared light output from the image sensor that receives the light.

As described above, when the DBPF is used instead of the infrared cut filter, light in the visible light band and infrared light in the second wavelength band are input to the photodiode of the image sensor. By this. The entire dynamic range of the image sensor is not used for visible light as in the case of using an infrared cut filter, but is used for visible light and infrared light in the second wavelength band. As a result, when the visible image signal is output, even if the component of the infrared light input as described above is removed, the S/N ratio may be deteriorated. For example, when the color temperature of illumination is low, the proportion of infrared light may be too high.
The graph of FIG. 27 shows the spectral distribution of light with different color temperatures, where the horizontal axis represents the wavelength of light (nm) and the vertical axis represents the spectral distribution peak value as 1 and the spectral distribution is expressed as a ratio. It was done. As shown in FIG. 27, if the color temperature is high, the ratio of infrared light to light in the visible light band is low, but the ratio of infrared light increases as the color temperature decreases. For example, in the wavelength range of 400 nm to 1300 nm, when the color temperature is 4000 K, the ratio on the long wavelength side of 700 nm is larger than that on the short wavelength side.
The color temperature of the morning sun and the setting sun is about 2000K, and the usual sun rays are 5000 to 6000K (6500K). Here, the light (tungsten lamp for photography/movies) taken in the studio is 3200K, and there is a possibility that lighting with a low color temperature may be used in shooting with a camera, and the photodiode of the image sensor receives light. The ratio of infrared light to be emitted becomes too high, and the S/N ratio of the visible image captured using the DBPF becomes low. It should be noted that even if it is not a light for studio shooting, warm-colored lighting has a low color temperature, and as described above, the S/N ratio of a visible image taken using DBPF may be low.

In addition, although the number of pixels in image sensors is being increased, when the number of pixels is increased, the sensitivity of the photodiode of the image sensor on the long wavelength side including infrared rays tends to be high, and the image pickup of high pixels There is a possibility that the S/N ratio of the visible image captured by using the above-mentioned DBPF as a sensor under illumination with a low color temperature may be further reduced.

The present invention has been made in view of the above circumstances, and improves the S/N ratio of a visible image when it is possible to capture a visible image and an infrared image using a DBPF instead of the infrared cut filter. It is an object of the present invention to provide an imaging device that can do the above.

In order to solve the above problems, an image pickup device of the present invention includes an image pickup sensor in which a light receiving element is arranged in each pixel
An optical system having a lens for forming an image on the image sensor,
A plurality of types of filter units are arranged in a predetermined arrangement corresponding to the arrangement of the pixels of the image sensor, and each type of filter unit has the first wavelength band adjacent to the long wavelength side of the visible light band. A filter having a transmission characteristic in a second wavelength band which is a part away from the visible light band and having spectral transmission characteristics different from each other according to a wavelength in the visible light band,
A signal processing device capable of processing a signal output from the image sensor to output a visible image signal and an infrared image signal,
The transmittance of the filter in the second wavelength band is lower than the transmittance of the filter in the visible light band.

In the above configuration of the present invention, the filter has a transmission characteristic in a visible light band, has a cutoff characteristic in a first wavelength band adjacent to a long wavelength side of the visible light band, and has a first wavelength band. An optical filter having a transmission characteristic in a second wavelength band, which is a part away from the visible light band,
A plurality of types of filter units are arranged in a predetermined arrangement corresponding to the arrangement of the pixels of the image sensor, and each type of filter unit has a transmission characteristic in the second wavelength band and a wavelength in the visible light band. With a color filter having different spectral transmission characteristics according to
It is preferable that the transmittance of the optical filter in the second wavelength band is lower than the transmittance of the optical filter in the visible light band.

According to such a configuration, in the filter, a plurality of types of filter units are arranged in a predetermined arrangement corresponding to the arrangement of the pixels of the image sensor, and each type of filter unit has a long wavelength in the visible light band. The second wavelength band, which is a part of the first wavelength band adjacent to the side away from the visible light band, has the transmission characteristic and the spectral transmission characteristics according to the wavelength in the visible light band are different from each other. further,
The transmittance of the filter in the second wavelength band is lower than the transmittance of the filter in the visible light band. The filter consists of an optical filter and a color filter.
Note that the transmittance indicates a ratio when the maximum (maximum) transmittance in the visible light band is set to 1 or 100, for example.
For example, the optical filter transmits light in the visible light band and infrared light in the second wavelength band, and each type of filter section of the color filter transmits infrared light in the second wavelength band and Although light in different visible wavelength bands is transmitted, the transmittance of the optical filter in the second wavelength band is lower than the transmittance of the optical filter in the visible light band. Accordingly, for example, when the image is captured by warm-colored illumination, it is possible to prevent the S/N ratio of the visible image from being deteriorated due to an excessively large amount of infrared light in the second wavelength band.
That is, by reducing the transmittance of the optical filter in the second wavelength band, it is possible to reduce the amount of infrared light received by the image sensor and reduce the influence on the visible image.

To reduce the amount of light in the second wavelength band that passes through the optical filter, it is conceivable to narrow the width of the second wavelength band in the optical filter, that is, the wavelength range. If the width of the second wavelength band is narrowed due to an error in the second wavelength band or a change in the wavelength of the transmitted light depending on the incident angle, the wavelength of the infrared LED used for infrared illumination is , And the second wavelength band may deviate.

That is, in the image capturing under the infrared illumination by the LED, the infrared illumination does not function due to the manufacturing error of the optical filter, or the infrared illumination does not function and becomes dark depending on the incident angle of light, that is, the direction of the subject. There is a possibility. On the other hand, when the amount of infrared light is reduced by lowering the transmittance, it is not necessary to make the width of the second wavelength band too narrow even if the amount of infrared light is greatly reduced. Illumination by the LED that emits infrared light in the wavelength band can be surely functioned.

By reducing the amount of infrared light that passes through the optical filter, the S/N ratio of a visible image to be captured is improved, but the amount of infrared light for capturing an infrared image is limited. Basically, in the case of shooting under a light whose color temperature is not so high, excess infrared light is only limited, and shooting with infrared light is sufficiently possible. In addition, at night or under dark lighting with high color temperature, it is possible to shoot using infrared lighting.

In the design of an image pickup device using an optical filter, a color temperature that is adjusted during shooting is set, and the transmittance of the second wavelength is set so that the S/N ratio of the color temperature becomes, for example, −3 dB or more. It is preferable to set. In this case, it is preferable that the visible light component in the signal output from the image sensor be 70% or more and the infrared component be 30% or less under the illumination of the color temperature to be adjusted. For example, in capturing a visible image, an infrared signal (infrared component) is removed from each color signal (color component) of red R, green G, blue B, etc., but the infrared component removed at this time is removed. Is preferably set not to exceed 30% of the color component before removal.

In this case, depending on how many times the lower limit of the color temperature of the illumination that can be used for shooting is set, as described above, the transmittance of the second wavelength band of the optical filter is the transmittance of the visible light band of the optical filter. It is preferable that the transmittance in the second wavelength band is 60% or less of the transmittance in the visible light band. In order to prevent the S/N ratio of the visible image from deteriorating due to the large amount of infrared light that passes through the optical filter and the color filter, even if the transmittance of the second wavelength band is set lower. Well, for example, the transmittance of the second wavelength band of the optical filter is 50% of the transmittance of the visible light band of the optical filter. It may be set to 40%, 30%, 25% or the like.

In addition, as described above, when the number of pixels of the image sensor increases and the number of pixels increases, that is, when the number of pixels on the image sensor decreases with an increase in the number of pixels, near-infrared rays on the longer wavelength side than the visible light band Since the sensitivity tends to increase, it is preferable to reduce the transmittance in the second wavelength band in response to the increase in the number of pixels of the image sensor and the decrease in the pixel size.

In the above configuration of the present invention, it is preferable that the transmittance of the optical filter in the second wavelength band is 60% or less of the transmittance of the optical filter in the visible light band.

With such a configuration, it is possible to reduce the amount of infrared light that passes through the optical filter and the color filter under illumination with a low color temperature, and to improve the S/N ratio of the visible image as described above. .. The transmittance in the second wavelength band may be further reduced as described above.

Further, in the configuration of the present invention, the optical filter,
A first filter having a transmission characteristic in the visible light band and the second wavelength band;
Having a transmission characteristic in the visible light band, having a transmission characteristic in the first wavelength band including the second wavelength band, the transmittance of the first wavelength band is greater than the transmittance of the visible light band. And a second filter that is low.

According to such a configuration, by using the first filter and the second filter, the visible light band and the second wavelength band have transmission characteristics as described above, and An optical filter having a transmittance in the second wavelength band lower than that in the transmittance can be obtained, and the above-described effects can be obtained.

In this case, depending on the model of the image pickup device, when the transmittance of the second wavelength band with respect to the visible light band of the optical filter is changed, it can be dealt with by changing the second filter, and a special transmittance spectrum can be obtained. There is a possibility that the setting of the transmittance can be changed more easily than changing the optical filter (first filter) that it has. In other words, the first filter can be used for general purposes, and the cost of the first filter can be reduced. In addition, the first filter and the second filter may be arranged so as to be in contact with each other so as to be overlapped with each other, or may be arranged apart from each other, or another optical member may be arranged therebetween. Good.

In the filter in such an image pickup device, a plurality of types of filter units are arranged in a predetermined arrangement corresponding to the arrangement of the pixels of the image sensor, and each type of filter unit is located on the long wavelength side of the visible light band. The second wavelength band, which has transmission characteristics in a second wavelength band that is a part of the first wavelength band adjacent to the first wavelength band adjacent to each other and has different spectral transmission characteristics according to wavelengths in the visible light band. Of the filter is lower than the transmittance of the filter in the visible light band.

Further, the filter has a transmission characteristic in a visible light band, a cutoff characteristic in a first wavelength band adjacent to a long wavelength side of the visible light band, and the visible light in the first wavelength band. An optical filter having a transmission characteristic in a second wavelength band that is a part away from the band;
A plurality of types of filter units are arranged in a predetermined arrangement corresponding to the arrangement of the pixels of the image sensor, and each type of filter unit has a transmission characteristic in the second wavelength band and a wavelength in the visible light band. With a color filter having different spectral transmission characteristics according to
It is preferable that the transmittance of the optical filter in the second wavelength band is lower than the transmittance of the optical filter in the visible light band.

According to the present invention, the S/N ratio of a visible image is improved when a visible image and an infrared image can be captured using an optical filter having a transparency in the visible light band and the infrared light band. be able to.

It is the schematic which shows the image sensor of the imaging device of the 1st Embodiment of this invention. 3 is a graph showing a DBPF of the image sensor and a transmittance spectrum of a color filter. 9 is a graph showing a DBPF transmittance spectrum of the image sensor. 9 is a graph showing a DBPF transmittance spectrum of the image sensor. 3 is a graph showing a DBPF of the image sensor, a transmittance spectrum of a color filter, and an emission spectrum of infrared illumination. 3 is a graph showing a DBPF of the image sensor, a transmittance spectrum of a color filter, and an emission spectrum of infrared illumination. FIG. 6 is a diagram for explaining an array of color filters of the image sensor, FIG. 6A is a view showing a conventional array without an infrared filter section, and FIGS. FIG. 6 is a diagram showing an array having an infrared filter section. FIG. 3 is a schematic diagram showing an image pickup apparatus having the image pickup sensor. FIG. 3 is a block diagram for explaining signal processing in a signal processing unit of the imaging device. FIG. 6 is a diagram for explaining interior processing in the signal processing of the image pickup apparatus. It is the schematic which shows the imaging device of the 2nd Embodiment of this invention. It is a figure for demonstrating the arrangement of the color filter of the image sensor of the imaging device of the 3rd Embodiment of this invention. 3 is a graph showing a DBPF of the image sensor and a transmittance spectrum of a color filter. FIG. 3 is a diagram for explaining an array of color filters of the image sensor in the same manner. FIG. 3 is a diagram for explaining an array of color filters of the same. FIG. 6 is a diagram for explaining an array of color filters of the same. FIG. 6 is a diagram for explaining the characteristics of each color filter. FIG. 3 is a block diagram for explaining signal processing in the image pickup apparatus. It is a figure for demonstrating the arrangement|sequence of the color filter of the 4th Embodiment of this invention. 9 is a graph showing the DBPF of the image sensor and the blue B transmittance spectrum of the color filter. 9 is a graph showing the DBPF of the image sensor and the transmittance spectrum of green G of the color filter. 9 is a graph showing the DBPF of the image sensor and the transmittance spectrum of red R of the color filter. 9 is a graph showing the transmittance spectra of the DBPF of the image sensor and the clear C of the color filter. It is a graph which shows the transmittance spectrum of DBPF which is the 1st filter of the optical filter of the imaging device of the 5th Embodiment of this invention. 6 is a graph showing a transmittance spectrum of an infrared neutral density filter that is a second filter of the optical filter. 3 is a graph showing a transmittance spectrum of an optical filter including the first filter and the second filter. It is a graph which shows the relationship between color temperature and spectral distribution.

Hereinafter, the image pickup apparatus according to the first embodiment of the present invention will be described with reference to the drawings.
The imaging device 10 (illustrated in FIG. 8) of the present embodiment includes an imaging sensor (image sensor) 1 shown in FIG. 1, for example. As shown in FIG. 1, the image sensor 1 includes a sensor main body 2 which is a CCD (Charge Coupled Device) image sensor, and red (R), green (G), and blue (B) corresponding to each pixel of the sensor main body 2. ), an infrared (IR) region (filter of each color) is arranged in a predetermined arrangement, a cover glass 4 covering the sensor body 2 and the color filter 3, and a cover glass 4 formed on the cover glass 4. And a DBPF (double band pass filter) 5.

The sensor body 2 is a CCD image sensor, and a photodiode as a light receiving element is arranged in each pixel. The sensor body 2 may be a CMOS (Complementary Metal Oxide Semiconductor) image sensor instead of the CCD image sensor. The sensor body 2 may be an image sensor, and the color filter 3 and the DBPF 5 may be attached to the image sensor.

A color filter 3 is provided on the sensor body 2. The color filter 3 has an IR filter section added to a color filter 3x having three R, G, and B filter sections arranged in each pixel in a general Bayer array shown in FIG. 7A. ing. In the Bayer array filter, the basic pattern is composed of 16 filter parts of each color in 4 rows (horizontal arrangement)×4 columns (vertical arrangement). In terms of the number of rows and columns, the first row is F11, F12, F13, F14, the second row is F21, F22, F23, F24, the third row is F31, F32, F33, F34, and the fourth row. The eyes are F41, F42, F43, and F44.

In the Bayer array, eight filter parts F12, F14, F21, F23, F32, F34, F41, and F43 are G, four filter parts F11, F13, F31, and F33 are R, and F22, Four filter units F24, F42, and F44 are designated as B. The reason why the number of G filter portions is twice the number of R and B filter portions is that human eyes have high sensitivity to green. It should be noted that even if the less sensitive one has a higher resolution, it may not be recognized by the human eye, but if the more sensitive one has a higher resolution, it is more likely to be recognized by the human eye, and a higher-resolution image will be obtained. Be recognized. In this Bayer array, G filters are arranged in alternate rows (horizontal direction) and column direction (vertical direction) to form a checkered pattern, and the remaining part has R filters and B filters. The filter parts are arranged without being adjacent to each other.

On the other hand, in the color filter 3 of the present embodiment, as shown in FIG. 7B, four R out of eight G filter units in the Bayer array are set to IR, so that R is four. , G of 4, B of 4, and IR of 4 are included. That is, in the basic array of 4 rows and 4 columns, four types of R, G, B, and IR filter units are arranged, and four filter units of the same type are adjacent to each other in the row direction and the column direction. Are arranged apart from each other, one R, G, B, and IR filter units are arranged in each column, and two types of R, G, B, and IR filter units are arranged every other row. There are two parts each.

Specifically, four filter parts F11, F13, F32, and F34 are set to R, four filter parts F12, F14, F31, and F33 are set to IR, and four filter parts F21, F23, F42, and F44 are set. The number of filter units is G, and the number of four filter units F32, F34, F41, and F42 is B.

In this case, since the number of G filter portions is reduced, the resolution may appear to be poor to the human eye, but the colors including IR are evenly arranged and the complementing (interpolation) processing is facilitated. .. In addition, the positions of the respective colors in the first and second rows and the third and fourth rows are laterally offset by one column. In other words, the first and second rows and the third and fourth rows are arranged such that the color arrangement is horizontally reversed.

By doing so, in the 4×4 arrangement, each color is arranged in each column and two colors are arranged in every other row, so that the horizontal direction (horizontal direction) rather than the vertical direction (vertical direction) is arranged. The resolution of the horizontal direction is increased, and the reduction of the horizontal resolution due to the provision of the IR filter can be suppressed. It should be noted that the color filter 3a includes a pattern obtained by horizontally inverting the pattern of the color filter 3a shown in FIG. 7B, a vertically inverted pattern, and a 180-degree rotated pattern. It may also include those rotated 90 degrees clockwise and those rotated 270 degrees. However, those rotated by 90 degrees and 270 degrees have higher resolution in the vertical direction than in the horizontal direction.

Further, in the color filter 3, as in the color filter 3b shown in FIG. 7C, two of the four Bs of the pattern of the Bayer array color filter 3x are not reduced without reducing G, which has high human sensitivity. One may be IR. In the color filter 3b, in the basic array of 4 rows and 4 columns, 8 G filter sections, 4 R filter sections, and 4 B filter sections out of 4 types of R, G, B, and IR filter sections. And two IR filter sections are arranged, and filter sections of the same type are arranged so as not to be adjacent to each other in the row direction and the column direction.

Specifically, in the color filter 3b, eight filter parts F12, F14, F21, F23, F32, F34, F41, and F43 are set to G, and four filter parts F11, F13, F31, and F33 are included. R, the two filter parts F22 and F44 are B, and the two filter parts F24 and F42 are IR. In this case, the resolution of B is inferior to that of the Bayer array, but the resolutions of G and R are the same as those of the Bayer array. It should be noted that the color filter 3b includes the pattern of the color filter 3b shown in FIG. 7C that is horizontally inverted, vertically inverted, and rotated 180 times. Further, a pattern rotated 90 degrees clockwise and a pattern rotated 270 degrees have the same pattern as a horizontally inverted pattern and a vertically inverted pattern.

Further, in order to increase the resolution of IR, two of four B of the pattern of the above-mentioned Bayer array color filter 3x are set to IR without reducing G unlike the color filter 3c shown in FIG. 7D. And two of the four Rs may be IR. That is, in the color filter 3c, in the basic array of 4 rows and 4 columns, among the four types of R, G, B, and IR filter sections, there are eight G filter sections, four IR filter sections, and four R filter sections. Two filter parts and two filter parts of B are arranged, and the filter parts of the same kind are arranged so as not to be adjacent to each other in the row direction and the column direction.

More specifically, as shown in FIG. 7D, in the color filter 3c, eight filter parts F12, F14, F21, F23, F32, F34, F41, and F43 are set to G, and F11 and F33. The two filter portions of No. 2 are designated as R, the two filter portions of F22 and F44 are designated as B, and the four filter portions of F13, F24, F31, and F42 are designated as IR. In this case, the resolutions of R and B are lower than those of the Bayer array, but the resolution of G is maintained and the resolution of IR can be secured. The color filter 3c includes the pattern of the color filter 3b shown in FIG. 7D which is horizontally inverted, vertically inverted, and rotated 180 times. Further, a pattern rotated 90 degrees clockwise and a pattern rotated 270 degrees have the same pattern as a horizontally inverted pattern and a vertically inverted pattern.

Well-known filters can be used for the R filter section, the G filter section, and the B filter section of the color filter 3. The transmittance spectra of the R filter unit, the G filter unit, and the B filter unit in the present embodiment are as shown in the graphs of FIGS. 2, 5, and 6. 2, 5 and 6 show the transmittance spectra of the red (R), green (G), blue (B) and infrared (IR) filters of the color filter 3, the vertical axis. Indicates the transmittance, and the horizontal axis indicates the wavelength. In the graph of the transmittance spectrum, the horizontal axis represents the unit of nm and the vertical axis represents the ratio, and, for example, the vertical axis represents the transmittance with respect to the peak value of 1 or 100. The wavelength range in the graph includes the visible light band and a part of the near-infrared band, and shows a wavelength range of 300 nm to 1100 nm, for example.

For example, as shown by R (double line) in the graph, the R filter portion has a maximum transmittance at a wavelength of 600 nm, and the long wavelength side thereof has a maximum transmittance even after exceeding 1000 nm. It will be maintained.

As indicated by G (broken dashed line) in the graph, the G filter section has a peak of maximum transmittance at a wavelength of about 540 nm and a transmittance at a wavelength of about 620 nm on the long wavelength side. There is a minimal part. Further, in the G filter part, the long wavelength side tends to increase from the part where the transmittance is minimum, and the transmittance becomes approximately maximum at about 850 nm. On the longer wavelength side, the transmittance is in the maximum state even if it exceeds 1000 nm. As indicated by B (narrow dotted line) in the graph, the B filter portion has a peak of maximum transmittance in a wavelength portion of about 460 nm, and a long wavelength side portion of about 630 nm, There are some parts where the transmittance is minimal. On the longer wavelength side, there is an increasing tendency, and the transmittance is approximately maximum at about 860 nm, and on the longer wavelength side, the transmittance is approximately maximum even if it exceeds 1000 nm. The IR filter part blocks light on the short wavelength side from about 780 nm, blocks light on the long wavelength side from about 1020 nm, and has a maximum transmittance at a portion of about 820 nm to 920 nm.

The transmittance spectra of the R, G, B, and IR filter parts are not limited to those shown in FIG. 2 and the like. However, the transmittance of the color filter 3 which is generally used at present is close to this. It seems to show a spectrum. In addition, 1 on the horizontal axis indicating the transmittance does not mean that 100% of light is transmitted, but indicates, for example, the maximum transmittance in the color filter 3.

The cover glass 4 covers and protects the sensor body 2 and the color filter 3.
The DBPF 5 is an optical filter formed on the cover glass 4 here. The DBPF 5 has a transmission characteristic in the visible light band, has a cutoff characteristic in the first wavelength band adjacent to the long wavelength side of the visible light band, and has a second wavelength that is a part of the first wavelength band. It is an optical filter having a transmission characteristic in the band.

That is, as shown in the graphs of FIGS. 2 and 3, the DBPF 5 is slightly separated from the visible light band indicated by DBPF (VR) on the long wavelength side with respect to the visible light band, as shown in the DBPF on the graphs. The transmittance of two bands in the infrared band (second wavelength band) indicated by the position DBPF (IR) is high. Further, DBPF (VR) as a band having a high transmittance in the visible light band is, for example, a wavelength band of about 370 nm to 700 nm. Further, the DBPF (IR) as the second wavelength band having a high transmittance on the infrared side is a band of about 830 nm to 970 nm, for example. The wavelength range in which the infrared light of DBPF (IR) is transmitted is not limited to the above-mentioned wavelength band, and the spectral characteristics of the color filter and the wavelength of the infrared illumination when equipped with the infrared illumination are taken into consideration. Is set.

In the present embodiment, the transmittance in the DBPF 5 is set to be different in the visible light band (DBPF(VR)) and the second wavelength band (DBPF(IR)) that is the infrared light band, and the visible light band is set. The transmittance of the second wavelength band that transmits infrared light is lower than the transmittance of the above. For example, as shown in FIGS. 2 and 3, in the DBPF 5, when the transmittance of the peak (maximum value (maximum value) of the visible light band) of the visible light band (DBPF(VR)) is 100, The peak transmittance of the second wavelength band (DBPF(IR)) is set to 50. That is, in the DBPF 5, the peak transmittance in the second wavelength band is 50% of the peak transmittance in the visible light band. In the DBPF 5, the transmittance in the second wavelength band is not limited to 50% of the visible light band, but is preferably 60% or less. The ratio of the transmittance of the second wavelength band to the visible light band in the DBPF is the lower limit of the color temperature of the illumination at the time of shooting, the characteristics of the image sensor 1 (area, number of pixels, etc.), and the width of the second wavelength band in the DBPF 5 ( Wavelength range). The transmittance in the visible light band and the second wavelength band, that is, the transmittance in the wavelength band within the predetermined wavelength range is the transmittance at the peak (maximum value) in the wavelength band in the transmittance spectrum. The average transmittance may be used in the wavelength band. That is, various average values, mode values, median values, etc. may be used as the representative values of the transmittance in the wavelength band.

In designing the image pickup apparatus, there are cases where the lower limit value of the color temperature of usable illumination is assumed. For example, if the lower limit value of the color temperature of the illumination is 3000 K, the ratio of the infrared light in the ratio of visible light to the infrared light in the illumination increases, and the image is captured using the DBPF 5 without using the infrared cut filter. In this case, the ratio of the infrared light signal to the visible light signal output from the image sensor 1 is high. That is, the proportion of infrared light in the dynamic range of the image sensor 1 increases as the color temperature of the illumination decreases, and the proportion of visible light in the dynamic range decreases, so that the S of the visible image captured with visible light is reduced. /N will decrease. For example, when about 30% of the signal output from the image sensor at the time of image pickup is infrared light and about 70% is visible light, for example, the S/N ratio of the visible image is about -3 dB, However, the S/N ratio of the visible image is preferably -3 dB or more.

Therefore, the ratio of the transmittance of the second wavelength band to the visible light band in the DBPF 5 is the lower limit color temperature of the illumination that can be used by the imaging device 10, the wavelength range (width) of the second wavelength band, and the imaging sensor 1. It is determined in consideration of the ratio of the visible light signal and the infrared light signal output from the device, and as described above, in the DBPF 5, the transmittance of the second wavelength band is set to 50% of the transmittance of the visible light band. However, for example, as shown in FIG. 4, the transmittance may be 25%. For example, it is preferable that the higher the resolution of the image sensor 1 is, the smaller the pixel area is, and the lower the color temperature of the illumination used is, the lower the transmittance of the second wavelength band is.

In the present embodiment, the relationship between the transmittance spectrum of each filter section of the color filter 3 and the transmittance spectrum of the DBPF 5 is defined as follows. That is, in DBPF (IR) that is the second wavelength band that transmits infrared light in the transmittance spectrum of DBPF 5, all of the R filter unit, the G filter unit, and the B filter unit have substantially maximum transmittance. Therefore, it is included in the wavelength band A shown in FIG. 2 in which the transmittances of the respective filter parts are substantially the same, and also included in the wavelength band B in which light is transmitted at the substantially maximum transmittance of the IR filter part. It has become.

On the shorter wavelength side than the wavelength band A, the transmittances of the G and B filter portions are lower than that of the R filter portion, which has a maximum transmittance. In the DBPF 5, a portion having a difference in transmittance between the R, G, and B filter portions is a portion having a high transmittance in the visible light band, DBPF (VR), and a second wavelength band in the infrared light band. Corresponds to the portion where the transmittance of the DBPF 5 between the DBPF (IR), which is a portion having a high transmittance, and which substantially blocks the light of the DBPF 5, is the minimum. That is, on the infrared side, the difference in transmittance between the R, G, and B filter portions becomes large, and the transmission of light at a portion where the transmittance may be lower than that of DBPF (IR) is cut. Light is transmitted in the wavelength band A in which the transmittance of each filter unit is surely higher than that of DBPF (IR) on the long wavelength side.

From the above, in the present embodiment, the DBPF 5 used in place of the infrared cut filter has a region that transmits light not only in the visible light band but also in the second wavelength band on the infrared light side. Therefore, in color imaging with visible light, light is affected by light that has passed through the second wavelength band. At this time, when the light passing through each of the RGB filters of the color filter 3 is received by the image sensor, by setting the transmittance of the second wavelength band to the transmittance of the visible light band of the DBPF 5 low, It is possible to prevent the S/N ratio of a visible image from being lowered due to visible light because the ratio of infrared light to visible light becomes too large.

Further, the second wavelength band of the DBPF 5 does not transmit the light of the portions having different transmittances in the R, G, and B filter portions, and the transmittances of the respective filter portions become substantially maximum, and the transmittances become substantially the same. Only the light in the wavelength band is transmitted.

Further, in the second wavelength low range of the DBPF 5, the light of the portion where the transmittance is substantially maximum in the IR filter portion is transmitted. Therefore, assuming that the R, G, B, and IR filter sections are respectively provided in four pixels that are extremely close to each other and are irradiated with substantially the same light, in the second wavelength band, the R filter section is provided. , G filter section, B filter section, and IR filter section pass light in substantially the same way, and as infrared side light, light having substantially the same light amount in each filter section including IR is obtained in the image sensor main body. The above-mentioned pixel photodiode is reached. That is, the amount of light that passes through the second wavelength band on the infrared side of the light that passes through the R, G, and B filters is the same as the amount of light that passes through the IR filter unit. When the above assumption is made, basically, the output signal of the pixel assumed as described above from the sensor main body 2 that receives the light transmitted through each of the R, G, and B filters and the IR filter is transmitted. The difference from the output signal of the pixel assumed as described above from the sensor main body 2 that receives the light is R, G, and B that cut the infrared-side light that has passed through the R, G, and B filter units, respectively. It becomes the output signal of the visible light part of.

Actually, as shown in each pattern of the color filter 3 (3a, 3b, 3c), one of the R, G, B, and IR filter portions is arranged in each pixel of the sensor body 2. Therefore, since it is highly possible that the respective light amounts of the respective colors of light with which each pixel is irradiated are different, for example, in each pixel, a well-known interpolation method (interpolation method) is used, It is possible to obtain the luminance, and use the difference between the interpolated R, G, and B luminance of each pixel and the interpolated IR luminance as the R, G, and B luminance, respectively. Note that the image processing method for removing the infrared light component from the brightness of each color of R, G, B is not limited to this, and finally the brightness of each of R, G, B passes through the second wavelength band. Any method may be used as long as it is a method capable of cutting the influence of the above-mentioned light. In any of the methods, the DBPF 5 cuts a portion where the transmittance of the R, G, and B filter portions on the infrared side is different from 20%, that is, a portion where the transmittance is different from a predetermined ratio, so that each pixel has , The processing for removing the influence of infrared light becomes easy.

The image sensor 1 is used as an image sensor in an image pickup apparatus capable of both color image pickup and infrared light image pickup. Generally, it is conceivable that normal photography is performed by color photography and infrared photography is performed at night using infrared illumination that is difficult for humans to recognize, without using visible illumination. For example, in various surveillance cameras and the like, it is conceivable to perform nighttime shooting with infrared light using infrared light illumination at the time of nighttime shooting in a place where nightlighting is not necessary or is preferably not illuminated at nighttime. It can also be used for daytime photography and nighttime photography for observing wildlife.

When infrared light photography is used as nighttime photography, infrared light illumination is necessary because even in the case of infrared light, the amount of light is insufficient at night as in the case of visible light. The transmittance spectrum of the DBPF 5 shown in FIG. 5 considers the transmittance spectrum of each of the R, G, B, and IR filter parts and the light for infrared light illumination, for example, the emission spectrum of the infrared LED for illumination. It was decided by. In FIG. 5, in addition to the transmittance spectra R, G, B, and IR of the filter parts of the respective colors similar to FIG. 2 and the transmittance spectrum DBPF of the DBPF 5, the emission spectrum IR-light of the LED illumination is illustrated.

The second wavelength band indicated by DBPF (IR), which is a portion of the DBPF shown in FIG. 5 that transmits infrared light, has an R filter section, a G filter section, and a B filter similar to the DBPF shown in FIG. All of the parts have a maximum transmittance and are included in the wavelength band A shown in FIG. 2 in which the transmittances of the respective filter parts are substantially the same, and the light is transmitted at the maximum transmittance of the IR filter part. It is included in the wavelength band B that is set.

In addition, almost the entire wavelength band that is the peak of the emission spectrum of infrared light illumination that is included in both the wavelength band A and the wavelength band B is included in the DBPF (IR) wavelength band. There is. It should be noted that when infrared light imaging is performed under infrared light illumination instead of natural light at night, the second wavelength band indicated by DBPF(IR) does not need to be wider than the peak width of the optical spectrum of infrared light illumination. When the spectrum of the infrared light illumination is included in both the wavelength band A and the wavelength band B, the peak width of the peak of the emission spectrum of the infrared light illumination, for example, having a peak at about 860 is approximately the same as the peak width. A peak portion of the transmittance of DBPF5 represented by DBPF(IR) may be provided as the second wavelength band.

That is, in FIG. 5, the peak in the emission spectrum of the infrared light illumination indicated by IR-light is on the short wavelength side of the wavelength band A and the wavelength band B described above, and the second wavelength of the DBPF indicated by DBPF(IR). The band is designed to substantially overlap the peaks of the emission spectrum in the IR-light in the short wavelength side portions of the wavelength band A and the wavelength band B.

Further, similarly to the graph shown in FIG. 5, in the graph shown in FIG. 6, the emission spectrum of infrared light illumination is added to the graph shown in FIG. 2 and the transmittance spectrum on the infrared side of the transmittance spectrum of DBPF5 is high. The second wavelength band represented by a certain DBPF (IR) is matched with the peak of the emission spectrum represented by IR-light of the above infrared light illumination.

In FIG. 6, as the infrared light illumination, one having a longer peak wavelength of the emission spectrum than that shown in FIG. 5 is used, and this peak is included in the above-mentioned wavelength band A and wavelength band B, and It exists on the long wavelength side of wavelength band A and wavelength band B. Correspondingly, the second wavelength band indicated by DBPF(IR) of DBPF 5 is provided so as to substantially overlap with the peak indicated by IR-light of infrared illumination within the above-mentioned wavelength band A and wavelength band B.

The second wavelength band of the DBPF 5 may be any of those shown in FIGS. 2, 5, and 6, and the second wavelength band is included in both the wavelength band A and the wavelength band B described above. It should be. Further, when the wavelength band that is the peak of the emission spectrum of the infrared light illumination used for night-time infrared light photography is determined, the wavelength band is included in both the wavelength band A and the wavelength band B described above. In addition, it is preferable to match the second wavelength band of the DBPF 5 with the peak of the emission spectrum of infrared light illumination.

In the image pickup apparatus 10 having such an image pickup sensor 1, the second wavelength band that transmits light on the infrared side of the DBPF 5 is on the infrared side of each of the R, G, B, and IR filter sections, and each filter is It is included in the wavelength band A in which the transmittance of each part is approximately the maximum and the transmittances of the respective filter parts are approximately the same, and is included in the wavelength band B in which the transmittance of the IR filter part is approximately the maximum. In other words, on the long wavelength side of the visible light band, the transmittance of each of the R, G, and B filters is substantially maximum only in the R filter section, and the transmittance of the G and B filter sections is substantially maximum. Since there is no such light, the light of different portions where the transmittances of the R, G, and B filter portions are not substantially the same are cut by the DBPF 5.

That is, in each of the R, G, B, and IR filter parts, the light of the second wavelength band is transmitted on the infrared side, so that the infrared side transmittances of all the filter parts are substantially the same, If the light in the second wavelength band is irradiated with the same amount of light, the amount of transmitted light in each of the R, G, B, and IR filter sections becomes the same. As a result, the color based on the output signal from the pixel corresponding to each of the R, G, and B filter units is corrected as described above, and the influence of the infrared light passing through the second wavelength band of the color at the time of color photographing is corrected. It is possible to easily obtain an image in which is suppressed.

Further, by making the second wavelength band correspond to the peak of the emission spectrum of the infrared light illumination included in the wavelength band A and the wavelength band B described above, the light of the infrared light illumination can be used efficiently, and By narrowing the width of the second wavelength band, it is possible to reduce the influence of infrared light passing through the second wavelength band during color photography.

FIG. 8 shows an image pickup apparatus 10 using the image pickup sensor 1 of the present embodiment. The image pickup apparatus 10 processes the image pickup lens 11, the image pickup sensor 1 including the DBPF 5, and the output signal 13 output from the image pickup sensor 1 to perform the above-described interior processing and the second wavelength during color image pickup. A signal processing unit (signal processing device) 12 that performs image processing for removing the influence of infrared light that has passed through the band, image processing such as gamma correction, white balance, and RGB matrix correction is provided. The visible color image output signal 14 and the infrared light image output signal 15 can be output from the image processing unit.

The lens 11 constitutes an optical system that forms an image on the image sensor 1 of the image pickup apparatus 10. The lens 11 is composed of, for example, a plurality of lenses.

FIG. 9 is a block diagram showing signal processing in the signal processing unit 12 of the image pickup apparatus 10. The output signals of the R, G, B, and IR pixels are sent to the interior processing blocks 21r, 21g, 21b, and 21ir. In each of the interior processing blocks 21r, 21g, 21b, and 21ir, for example, as shown in FIG. 10, in the case of the above-described color filter 3b, in the image data of each frame by interpolation processing (interpolation processing) Image data 20r in which pixels are represented by red R, image data 20g in which all pixels are represented by green G, image data 20b in which all pixels are represented by blue B, and all pixels are represented by infrared IR. The R, G, B, and IR signals are converted so as to obtain the represented image data 20ir. A known method can be used as the interpolation processing method.

Next, in the infrared light removal signal creation blocks 22r, 22g, and 22b, subtraction is performed from the signals of each color of R, G, and B in order to remove the influence of the infrared light received from the above-mentioned second wavelength band. The signal to be generated is generated from the IR signal. The signals created for each of R, G, and B in the infrared light removal signal creation blocks 22r, 22g, and 22b are subtracted from the signals of each color of R, G, and B. In this case, in the same pixel as described above, the IR signal is basically removed from each of the R, G, and B signals, which facilitates the processing. In reality, the sensitivity differs for each pixel of each color due to the characteristics of the filter unit of each pixel, so that the signal subtracted from each R, G, B signal for each R, G, B image is subtracted from the IR signal. create.

Next, in the image processing block 23, the R, G, and B signals are subjected to a well-known RGB matrix processing for converting the R, G, and B signals using a determinant to correct the color, and white in the image. A known white balance process is performed so that the output values of the R, G, and B signals are the same in the area where is, and a known gamma correction that is a correction for image output to a display or the like is performed. Next, in the luminance matrix block 24, the signals of R, G, and B colors are multiplied by a coefficient to generate a signal of luminance Y. Also, by dividing the signal of luminance Y from the signal of blue B and the signal of red R, the color difference signals of RY and BY are calculated, and the signals of Y, RY and BY are output. To do.
The IR signal is basically output as a black and white gradation image.

In the image pickup apparatus 10 as described above, as described in the above-mentioned image pickup sensor 1, image processing for removing the influence of infrared light from a color image can be easily performed, and a visible color image excellent in color reproducibility. Can be obtained. Further, it is possible to reduce the development cost of such an imaging device.
As described above, when the effect of infrared light that has passed through the second wavelength band is cut by image processing in visible light imaging, the portions of the R, G, and B filter portions that have greatly different transmissivities are: Since it is physically cut by the DBPF, in the image processing, processing for cutting the IR light at the portion where the transmittance is substantially maximum on the infrared side of each of the R, G, and B filter portions may be performed. It will be. In this case, image processing is facilitated, and it becomes possible to obtain color image data having the same color reproducibility as in the case of using the conventional infrared light cut filter.

For example, the light that passes through the R, G, and B filter portions and reaches the photodiode is the same as the visible light that has passed through the R, G, and B filter portions in the visible light region. , The IR light passes through the same second wavelength band as the infrared light. Therefore, for example, the IR output signal after the interpolation processing corrected according to the characteristics such as the sensitivity based on the filter unit of each color is subtracted from each output signal after the R, G, and B interpolation processing of the image sensor 1. By doing so, color reproducibility close to that when an infrared cut filter is used can be obtained.

Further, in the DBPF 5, as described above, the transmittance of the second wavelength band is set to, for example, 60% or less or 50% or less with respect to the transmittance of the visible light band, so that infrared light such as illumination light is taken at the time of photographing. Even if the ratio is high, the ratio of infrared light can be reduced by the DBPF 5 to improve the S/N ratio of the visible image.

Further, in a smartphone, for example, biometric authentication such as fingerprint authentication may be used, and it is considered to use iris authentication as biometric authentication. For example, it is conceivable to use the camera on the front side of the smartphone as the imaging device 10 of this embodiment. Normal imaging is performed as a visible image, and iris authentication is performed as an infrared image. In this case, it is considered that the visible image needs to have a high S/N ratio and that the image sensor has a large number of pixels for iris authentication. When using a smartphone camera for iris authentication, it is difficult to capture the pupil part of the eye so that it occupies most of the entire capture range. It is necessary to perform iris authentication with the pupil in the image. In this case, if the image of the iris portion becomes small and the number of pixels is not large, it becomes difficult to recognize the iris. In this case, it is necessary to use a high-pixel image sensor for infrared iris authentication and to capture a clear visible image as a smartphone camera. In such a case, by using the imaging device 10 as described above, it is possible to suppress deterioration of the S/N ratio of the visible image even when a high-pixel imaging sensor is used, and to capture an infrared image. Is possible. Further, when an LED that emits infrared light in a predetermined wavelength range is used for infrared illumination when photographing with infrared light for iris authentication, the wavelength range of the second wavelength band of the DBPF 5 is too narrow. Therefore, it is possible to prevent the wavelength of the second wavelength band of the DBPF 5 from being deviated due to a manufacturing error, a difference in the incident angle due to the shooting direction, and the like, and the wavelength of the LED illumination from not matching. The image pickup device 10 is preferably provided with an infrared illumination device such as an LED.

Next, an image pickup apparatus according to the second embodiment of the present invention will be described. As shown in FIG. 11, in the image pickup apparatus 10a according to the second embodiment, the image sensor 1 is not provided with the DBPF 5, but the lens 11 is provided with the DPBF. The image pickup apparatus 10a processes the image pickup lens 11 including the DBPF 5, the image pickup sensor 1, and the output signal 13 output from the image pickup sensor 1 to perform the above-described interior processing and the second wavelength during color image pickup. An image processing for removing the influence of infrared light that has passed through the band, a signal processing unit 12 for performing image processing such as gamma correction, white balance, and RGB matrix correction on the image signal. The visible color image output signal 14 and the infrared light image output signal 15 can be output from the image processing unit.

The DBPF 5 and the color filter 3 are the same as the DBPF 5 and the color filter 3 of the first embodiment, and the transmittance of each of the R, G, B, and IR filter portions of the color filter 3 and the second of the DPBF 5 are the same. The relationship of the wavelength band DBPF (IR) is also similar to that of the first embodiment. Therefore, unlike the first embodiment, even if the DBPF 5 is provided in the lens 11, the same operational effect as that of the imaging device 10 of the first embodiment can be obtained. The DBPF 5 is provided in the optical system of the image pickup device 10a, and for the light reaching the image pickup sensor 1, a visible light band (DBPF(VR)) and a second wavelength band on the infrared side (DBPF(IR)). And the light is blocked by the short wavelength side of the visible light band, the long wavelength side of the second wavelength band, and the wavelength band between the visible light band and the second wavelength band. It may be provided anywhere as long as it is.

Next, an image pickup sensor and an image pickup apparatus according to a third embodiment of the present invention will be described. The image sensor 1 and the image pickup apparatus according to the third embodiment are different from each other in the configuration of a part of the color filter 3 and the method of removing the IR component from each of the RGB signals. It is similar to the first embodiment, and the color filter 3 and the IR component removing method will be described below.

In the present embodiment, for example, as shown in FIG. 12, the color filter 3e (RGBC configuration 1) has two B's of the four B's of the pattern of the Bayer array color filter 3x as C, 4 Two of the two Rs are designated as C, and four of the eight Gs are designated as C. That is, the color filter 3e has four G filter units, four C filter units, and eight R filter units among the four types of R, G, B, and C filter units in the basic array of 4 rows and 4 columns. Two filter parts and two filter parts of B are arranged, and the filter parts of the same kind are arranged so as not to be adjacent to each other in the row direction and the column direction. Therefore, the eight C-shaped filter parts are arranged in a checkered pattern. Here, C indicates a transparent state as a clear filter portion, and basically has a transmission characteristic from the visible light band to the near infrared wavelength band. Here, the visible light band is used. In, C=R+G+B. Note that C that is clear can be called white light, that is, white (W) because three colors of RGB are transmitted, and C=W=R+G+B. Therefore, C corresponds to the amount of light in almost the entire wavelength band of the visible light band.

Here, as shown in the graph showing the transmittance spectra (spectral transmission characteristics) of the color filter 3e and the DBPF 5 in FIG. 13, each of the R, G, and B filter portions has a transmission characteristic on the long wavelength side of the visible light band. Also, the C filter portion, which is a clear filter portion, allows light to be transmitted on the long wavelength side of the visible light band. On the other hand, by using the DBPF 5, as in the first embodiment, the infrared ray that transmits the longer wavelength side than the visible light band is limited to the second wavelength band, and R, The amount of light passing through the G, B, and C filter parts and the DBPF 5 is approximately the same (approximate) in each of the R, G, B, and C filter parts, and in the visible light band, each of the R, G, B, and C filter parts is The transmission characteristics differ depending on the wavelength of the part.

In addition, also in the first and second embodiments, infrared rays that are transmitted on the longer wavelength side than the visible light band are limited to the second wavelength band, and R, G, B, and IR The amount of light passing through the filter unit and the DBPF 5 is substantially the same in each of the R, G, B, and IR filter units, and in the visible light band, the transmission characteristics according to the wavelengths of the R, G, B, and IR filter units are different.

Thereby, also in the third embodiment, it is possible to accurately perform IR correction of each pixel and generate a visible image with high color reproducibility. That is, unlike the first embodiment, the IR filter section having the cutoff characteristic in substantially the entire wavelength band of the visible light band and the transmission characteristic in the infrared on the longer wavelength side than the visible light band is not provided. Even if the C filter section is provided, the IR signal can be calculated by the following formula.

In the following description, C(W), R, G, B, and IR indicate the level of the output signal from the image sensor 1, while C(W), R, G, and B indicate the level of the visible light band. It does not include an infrared component.
Here, if the color filter 3e is designed as C=W≈R+G+B and the IR signal to be removed from each of the RGB signals is IR′,
IR′=((R+IR)+(G+IR)+(B+IR)−(C+IR))/2=IR+(R+G+BC)/2
IR′≈IR. In addition, IR shows an actual value calculated|required by measurement etc., IR' shows the value calculated|required. IR correction can be performed by subtracting IR' from each filter.
That is,
R filter (R+IR):
R'=(R+IR)-IR'=R-(R+G+BC)/2
G filter (G+IR):
G'=(G+IR)-IR'=G-(R+G+BC)/2
B filter (B+IR):
B'=(B+IR)-IR'=B-(R+G+BC)/2
C (=W) filter (W+IR):
W'=(C+IR)-IR'=C-(R+G+BC)/2
Becomes

Thereby, in the color filter 3, even if a clear C filter portion is used instead of the IR filter portion, the DBPF 5 can approximate the IR transmittance of each filter portion, and the IR component is obtained as described above. Then, by removing this from the signal of each filter unit, color reproducibility can be improved.

It should be noted that in such calculation, for example, R+IR, G+IR, B+IR, and C+IR are obtained at each pixel by the interpolation method as described above, and the above calculation is performed at each pixel. Although C=W≅R+G+B is designed, it is not necessary to match this formula with the approximate accuracy, and if approximated, there may be a deviation due to an error or other reason. There may be a deviation of about %.

Further, for C, since R+G+B results in a large amount of light received by the pixel and it is easy to saturate. Therefore, in the filter unit of C, the amount of light received in the visible light band is reduced, or the infrared wavelength band and the visible light band are reduced. It is also possible to reduce the amount of received light over a wavelength band including and, or to reduce the amount of charges accumulated in the amount of received light in the element portion forming the pixel in each pixel. In that case, it is necessary to change the above equations accordingly.

FIG. 14 shows another arrangement of R, G, B, and C color filters. In the 2×2 arrangement, R, G, B, and C are evenly arranged one by one. is there.
Also, in the case of the conventional R, G, B Bayer array, as shown in FIG. 15, in the 2×2 array, R and B are arranged one by one and two Gs are arranged.
Further, as shown in FIG. 16, a 2×2 array of R, G, B, and IR color filters in which G of one of the conventional arrays not including C or IR is changed to IR is The arrangement is such that G, B, and IR are arranged one by one.

As such a color filter, RGB-C configuration 1 shown in FIG. 12, RGB-C configuration 2 shown in FIG. 14, conventional RGB array (Bayer array) shown in FIG. 15, and RGB-IR array shown in FIG. In one example, there is a difference in characteristics as shown in FIG. Note that C does not include color information such as RGB, but includes brightness information as a light amount.
Therefore, the RGB-C (configuration 1) sensor has a high luminance resolution due to the checkerboard arrangement of C, but the RGB pixels are sparse and asymmetrical arrangement, so the resolution is low and moire is likely to occur. However, since the required resolution of the color signal is less than half that of the luminance signal, which is low, there is no problem. It also has high sensitivity.

The RGB-C (configuration 2) has the same luminance resolution and color resolution as the conventional RGB sensor, and has higher sensitivity than the RGB sensor. Since the RGB-IR sensor is provided with an IR having no transmission characteristic in the visible light band, it has lower sensitivity and lower luminance resolution than the RGB sensor. That is, the color filter having C is more likely to be advantageous in resolution and sensitivity than the color filter having IR in the first embodiment and the second embodiment described above.

FIG. 18 is a block diagram showing signal processing in the signal processing unit 12 described above. The image sensor 1 includes an RGB-C sensor (image sensor) 1 including the above-described RGB-C color filter 3e, a lens 11 forming an optical system, and a DBPF 5.
The R+IR, G+IR, B+IR, and C+IR signals from the RGB-C sensor 1 are input to a separation device 51 that performs color separation, IR separation, and IR correction, and R, G, and B, W, and IR signals are obtained and output. This processing is performed based on the calculation using the above formula.

The R, G, B, and R signals of the R, G, B, W, and IR signals output from the separation device 51 are sent to the color matrix device 52, and well-known RGB matrix correction and the like are performed to obtain RGB signals. Is output. Also. The R, G, B, W, and IR signals from the separation device 51 are sent to the luminance generation device 53, and a luminance signal is generated from each signal based on the equation for calculating the set luminance.

The RGB signals output from the color matrix device are input to the gamma processing and color difference generation device 54 and subjected to well-known gamma processing, and, for example, BY and RY signals are generated as color difference signals. It In addition, the signals output from the separation device 51 and the RGB-C sensor 1 are passed through a BPF (band pass filter) 55 as a signal in a predetermined wavelength band, noise is reduced by a noise reduction device 56, and then the luminance is reduced. It is amplified by the enhancement processing device 57 together with the brightness signal output from the generation device 53, and is output as a brightness signal (Y signal) of a brightness/color difference signal after undergoing gamma processing by the gamma processing device 58. The IR signal output from the separation device 51 is output as an IR signal through the enhancement processing device 59 and the gamma processing device 60. It should be noted that in the processing of the image signal, the clip processing described later is performed, and the clip processing will be described later.

Next, an image sensor and an image pickup apparatus according to a fourth embodiment of the present invention will be described. The fourth embodiment is a generalization of each color of the color filter, and shows that the color filter of the present invention is not limited to RGB-IR or RGB-C. Hereinafter, a method of removing the IR component in the image sensor including the color filter having the generalized four-color filter unit will be described. The four-color (four types) filter parts basically have different transmission characteristics according to the wavelength in the visible light band, and the visible light band is included in the wavelength band including the second wavelength band of the DBPF described above. A third wavelength band having a transmittance difference of 20% or less with the other filter unit on the longer wavelength side is provided, and the third wavelength band includes the second wavelength band of the DBPF 5. ing. As a result, when the color filter and the DPBF 5 are used, the transmission characteristics corresponding to the wavelength on the infrared side of the visible light band are approximated by the filter part of each color.

Further, in the filter arrangement of four kinds of pixels, the IR can be separated by designing the color filter under the following conditions. The filter arrangement is preferably a 2×2 arrangement, as shown in FIG. 19, in which four types of filter units A, B, C, and D are provided one by one.
Further, it is preferable to design each filter unit of A, B, C, and D so that the following relationship is established as much as possible in the visible wavelength band. That is, in the visible light band,
KaA+KbB+KcC+KdD≈0
And It should be noted that A, B, C, and D indicate the levels of the output signals from the image sensor 1 in the visible light band of each filter unit.

In the IR region on the longer wavelength side than the visible light band, the IR transmittance is substantially constant in the above-mentioned third wavelength band of each of the A, B, C, and D filter parts. The IR transmittance may be approximately an integral multiple of a certain IR transmittance in each of the A, B, C, and D filter units. When designed in this way (here, the IR transmittance is constant as described above),
Ka(A+IR)+Kb(B+IR)+Kc(C+IR)+Kd(D+IR)≈IR(Ka+Kb+Kc+Kd)
Therefore, the IR signal is
IR′=(Ka(A+IR)+Kb(B+IR)+Kc(C+IR)+Kd(D+IR))/(Ka+Kb+Kc+Kd)
Can be calculated by
The IR component contained in each pixel of A, B, C, and D can be corrected by the following calculation.
A′=(A+IR)−(Ka(A+IR)+Kb(B+IR)+Kc(C+IR)+Kd(D+IR))/(Ka+Kb+Kc+Kd)=A−(KaA+KbB+KcC+KdD)/(Ka+Kb+Kc+Kd)
B′=B+IR−(Ka(A+IR)+Kb(B+IR)+Kc(C+IR)+Kd(D+IR))/(Ka+Kb+Kc+Kd)=B−(KaA+KbB+KcC+KdD)/(Ka+Kb+Kc+Kd)
C′=C+IR−(Ka(A+IR)+Kb(B+IR)+Kc(C+IR)+Kd(D+IR))/(Ka+Kb+Kc+Kd)=C−(KaA+KbB+KcC+KdD)/(Ka+Kb+Kc+Kd)
D′=D+IR−(Ka(A+IR)+Kb(B+IR)+Kc(C+IR)+Kd(D+IR))/(Ka+Kb+Kc+Kd)=D−(KaA+KbB+KcC+KdD)/(Ka+Kb+Kc+Kd)
Where the error is
(KaA+KbB+KcC+KdD)/(Ka+Kb+Kc+Kd). This error can be corrected in the RGB matrix.
Actually, the transmittances of the IR components of the respective filter parts are slightly different, so that the correction is performed using the coefficient-corrected signals as described below.

A'=A+IR*KIRa-KIRa(Ka(A+IR*KIRa)+Kb(B+IR*KIRb)+Kc(C+IR*KIRc)+Kd(D+IR*KIRd))/(Ka*KIRa+Kb*KIRb+Kc*KIRc+Kd*KIRd)

B'=B+IR*KIRb-KIRb(Ka(A+IR*KIRa)+Kb(B+IR*KIRb)+Kc(C+IR*KIRc)+Kd(D+IR*KIRd))/(Ka*KIRa+Kb*KIRb+Kc*KIRc+Kd*KIRd)

C'=C+IR*KIRc-KIRc(Ka(A+IR*KIRa)+Kb(B+IR*KIRb)+Kc(C+IR*KIRc)+Kd(D+IR*KIRd))/(Ka*KIRa+Kb*KIRb+Kc*KIRc+Kd*KIRd)

D'=D+IR*KIRd-KIRd(Ka(A+IR*KIRa)+Kb(B+IR*KIRb)+Kc(C+IR*KIRc)+Kd(D+IR*KIRd))/(Ka*KIRa+Kb*KIRb+Kc*KIRc+Kd*KIRd)

The spectral transmission characteristics of each filter when using DBPF are as shown in FIG. As an example of the filter section, four kinds of filter sections of R+IR, G+IR, B+IR, and C+IR are used, but the IR section is constant or has an integral multiple relationship with each other, and KaA+KbB+KcC+KdD≈0. If a color filter is designed for each, the filter units are not limited to R+IR, G+IR, B+IR, and C+IR.

FIG. 20 shows the spectral transmission obtained by combining the B filter unit and the DBPF 5, FIG. 21 shows the spectral transmission obtained by combining the G filter unit and the DBPF 5, and FIG. 22 shows the spectral transmission obtained by combining the B filter unit and the DBPF 5. FIG. 23 shows the spectral transmission obtained by combining the C(W) filter unit and the DBPF 5.

Each spectral transmission characteristic is a sum of four transmittances of a visible R transmission region, a visible G transmission region, a visible B transmission region, and an IR transmission region, as shown in the above equations. .. Therefore, the signal values of each visible R transmission region, visible G transmission region, visible B transmission region, and IR transmission region can be calculated from the values of four or more types of filters.

Next, an image pickup apparatus according to the fifth embodiment of the present invention will be described. The image pickup apparatus according to the present embodiment includes an optical filter of the filter of the present invention as a DBPF 5 that serves as a first filter and an infrared neutral density filter (not shown) that serves as a second filter. The DBPF 5 in the present embodiment does not constitute the optical filter of the present invention by itself, but an optical filter is constituted by a combination with the infrared attenuating filter which is the second filter. Therefore, the filter of the present invention includes a color filter, a first filter (DBPF5), and a second filter (infrared attenuating filter).

The first filter (DBPF5) has a transmission characteristic in the visible light band (DBPF(VR)) and the above-mentioned second wavelength band (DBPF(IR)) as shown in the graph of the transmittance spectrum of FIG. , The visible light band (DBPF(VR)) and the second wavelength band (DBPF(IR)) have substantially the same transmittance.

On the other hand, the light attenuation filter of the second filter combined with the first filter has a substantially constant transmittance in the visible light band, as shown in the graph of the transmittance spectrum of FIG. 25, for example. Thus, the transmittance in the above-mentioned second wavelength band on the long wavelength side of the visible light band is low. In this embodiment, for example, the transmittance decreases linearly as the wavelength increases from a wavelength of slightly less than 700 nm.

As shown in the graph of the transmittance spectrum of FIG. 26, the transmittance when the first filter and the second filter are combined to form an optical filter, the transmittance is substantially the same as that of the single optical filter described above. Has a rate spectrum. That is, by combining the first filter and the second filter, the visible light band and the second wavelength band have transmission characteristics, and the transmittance of the visible light band causes the transmission of the second wavelength band. The rate is low, for example, 60% or less, 50% or less, 40% or less, 30% or less, 25% or less. Even in the image pickup apparatus including such an optical filter, the above-described effects can be obtained. In addition, since the optical filter is composed of the first filter and the second filter, it is possible to adjust the transmittance of infrared light only by changing the second filter. Polarization may be possible at a lower cost than changing. 24 to 26, the horizontal axis represents wavelength (nm) and the vertical axis represents transmittance as a percentage as described above.

1 Image sensor 2 Sensor body 3 Color filter 5 DBPF (optical filter)
11 lens (optical system)
12 Signal processing unit (signal processing device)

Claims (3)

An image sensor in which a light receiving element is arranged in each pixel,
An optical system having a lens for forming an image on the image sensor,
A color filter in which a plurality of types of filter portions having different spectral transmission characteristics according to wavelengths in the visible light band are arranged in a predetermined arrangement corresponding to the arrangement of the pixels of the image sensor ,
A transmission characteristic in the visible light band, a blocking characteristic for blocking light in a first wavelength band adjacent to a long wavelength side of the visible light band, and being a part of the first wavelength band; An optical filter having a transmission characteristic in two wavelength bands and having the cutoff characteristic in a region adjacent to the short wavelength side and the long wavelength side of the second wavelength band ,
A signal processing device capable of processing a signal output from the image sensor to output a visible image signal and an infrared image signal,
The color filter includes a third wavelength band, which is a wavelength band in which the transmittances of the filter units of the respective colors are similar to each other on the longer wavelength side than the visible light band,
The spectral transmission characteristic of the optical filter and the spectral transmission characteristic of each filter unit of the color filter are set so that the entire second wavelength band of the optical filter is included in the third wavelength band,
An image pickup apparatus, wherein the transmittance of the optical filter in the second wavelength band is lower than the transmittance of the optical filter in the visible light band. The image pickup apparatus according to claim 1 , wherein the optical filter has a transmittance in the second wavelength band of 60% or less of a transmittance of the optical filter in the visible light band. The optical filter,
A first filter having a transmission characteristic in the visible light band and the second wavelength band;
Wherein a visible light band transmission characteristic, the second has a transmission characteristic in the first wavelength band including a wavelength band, the transmittance of the first wavelength band region of transmittance the visible light band An image pickup apparatus according to claim 1 or 2, further comprising a second filter that is lower.

Images