JP2013115679A - Imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To inexpensively separate or combine a visible light image and an infrared light image and also improve resolution of the infrared light image by using an image sensor having a widespread color filter.SOLUTION: All pixels are subject to an arithmetic operation that separates signal components of RGB visible light and signal components of infrared light by finding solutions for simultaneous equations whose variables are RGB, from color signal components including the infrared light taken by an imaging element with a color filter of four colors of CMYG. Consequently, the visible light image composed of RGB signal components and the infrared light image composed of infrared light signal components are outputted. Alternatively, the outputted visible light image may be compositely outputted by being converted to a brightness signal including the infrared light and a color difference signal.

Description

The present invention relates to an imaging apparatus that captures an image by receiving reflected light from a subject using an imaging element having four color filters of cyan (C), magenta (M), yellow (Y), and green (G).

Image sensors are also sensitive to infrared light, so if you take it as it is, it is known that visible light and infrared light are received simultaneously, and the color reproducibility of the image deteriorates. Yes. For this reason, it is common to insert an infrared light cut filter between the lens and the image sensor to remove infrared light and improve color reproducibility.

By the way, since the infrared light cut filter is made of plastic or glass, the visible light component is also lost by about 1%. Further, since the infrared light component is lost as compared with the case where the infrared light cut filter is not used, the light amount is reduced by about 20 to 30%. For this reason, in Patent Document 1, it is possible to switch use / non-use of the infrared light cut filter. When the image is bright, the infrared light cut filter is used to improve color reproducibility. A technique for increasing the amount of light by not using a cut filter has been proposed. In particular, when infrared light illumination that cannot be seen by human eyes is used in combination, a clear image can be obtained even when the imaging range is so dark that it cannot be captured at all by visible light. However, when the infrared light cut filter is not used, the color reproducibility deteriorates, so that a monochrome image is output by expressing the color lightly or by completely deleting the color.

However, according to the technique disclosed in Patent Document 1, a mechanism for switching use / non-use of the infrared light cut filter is expensive, and the drive mechanism for that purpose causes a failure. For this reason, Patent Document 2 proposes a technique for separating infrared light without using a drive mechanism. Further, according to the technique disclosed in Patent Document 2, it is possible to synthesize a visible light image and an infrared light image when the imaging range is dark. Specifically, as shown in FIG. 4, the color filter arranged in the image sensor is preliminarily cut with red, green, and blue color filters and only infrared light. By using four color filters that transmit, simultaneous imaging of a visible light image and an infrared light image is possible. Accordingly, a visible light image with high color reproducibility is output when it is bright, and an infrared light image or an image obtained by combining a visible light image and an infrared light image is output when the imaging range is dark.

JP 2000-59798 A JP 2010-062604 A

  However, according to the technique disclosed in Patent Document 2, it is necessary to develop an image sensor having a special color filter. Since a different color filter is required for each pixel having a diameter of several micrometers, it is difficult to cope with the problem by replacing the color filter of the existing image sensor. In addition, since the filter that transmits infrared light is arranged in only one pixel per four pixels, the resolution of the infrared light image is ¼.

The present invention has been made to solve the above-described problems, and separates or synthesizes a visible light image and an infrared light image at low cost by using an image sensor having a widely used conventional color filter. An object of the present invention is to provide an imaging device that improves the resolution of an infrared light image.

In order to solve the above-described problem, the imaging apparatus of the present invention includes a CMYG complementary color filter, an imaging element, and color signal data C ′ including infrared light imaged by the imaging element via the CMYG complementary color filter, M ', Y', G 'are input, infrared color signal data R, G, B and infrared light data I are calculated and output using the following stored formula (A). It is characterized by comprising a light separation unit and a control unit for controlling them.
Formula (A)
R = Y'-G '
G = C ′ + Y′−M′−G ′
B = C'-G '
I = 2 × G′−C′−Y ′ + M ′

  According to the present invention, the infrared light separation unit performs infrared light together with R, G, and B color signal data from a signal acquired by a general CMYG complementary color image sensor by calculation based on the calculation formula (A). Data can be calculated.

  The imaging apparatus according to the present invention includes a first image output unit that receives the color signal data R, G, and B of the visible light, generates a visible light image, and outputs the generated image.

In the imaging apparatus of the present invention, the color signal data R, G, B of visible light output from the infrared light separation unit is input, converted into luminance signal field data L and color difference signal data Cb, Cr, and stored in advance. Infrared rays for calculating and outputting new luminance signal data L ′ from the luminance signal data L, the infrared light signal data I, and an arbitrary positive number t including zero using the following calculation formula (B) A synthesis unit;
And a second image output unit configured to generate and output an image from the new luminance signal data L ′ and the color difference data Cb and Cr.
Formula (B)
L ′ = L + I × t (t is any positive number including zero)

According to the present invention, the infrared light combining unit can include the infrared light data in the luminance signal data according to the arithmetic expression (B) for the visible light image output from the infrared light separating unit.

  According to the present invention, a visible light image can be generated and output by separating visible light data and infrared light data using an image sensor having a widely used conventional CMYG complementary color filter, An infrared light cut filter is not necessary, and an imaging apparatus can be configured at low cost. Also, by combining luminance signal data and infrared light data, and generating and outputting an image together with color difference data, a clear color image can be obtained even when the imaging range is dark without deteriorating color reproducibility. .

It is a figure which shows the structure of the imaging device which concerns on Embodiment 1 of this invention. It is a figure which shows the structure of the color filter used with the imaging device which concerns on Embodiment 1 of this invention. It is a figure which shows the structure of the imaging device which concerns on Embodiment 2 of this invention. It is a figure which shows the structure of the conventional color filter.

  DESCRIPTION OF EMBODIMENTS Hereinafter, embodiments for carrying out the present invention (hereinafter referred to as embodiments) will be described in detail with reference to the accompanying drawings. Note that the same number is assigned to the same element throughout the description of the embodiment.

(Configuration of Embodiment 1)
FIG. 1 is a block diagram illustrating a configuration of an imaging apparatus 10A according to the first embodiment. As shown in FIG. 1, the imaging apparatus 10A according to the first embodiment includes a lens 11, an image sensor 12, an infrared light separation unit 13, an image output unit 14, and a control unit 15a. The A subject image, which is reflected light from a subject incident through an optical system such as the lens 11, is photoelectrically converted by an image sensor 12 and supplied to an image output unit 14 via an infrared light separation unit 13 to be a signal. It is processed.

The image sensor 12 includes a widely used common CMYG complementary color filter 121 that transmits light from a subject, and an image sensor 122 that is configured by a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The image sensor 122 receives the reflected light that has passed through the color filter 121 and images the subject.

  The infrared light separation unit 13 uses R, G, and R, G, B, which are pre-stored arithmetic expressions using color signal data including infrared light imaged by the image sensor 122 as variables. A calculation unit for separating visible light data and infrared light data of B is included. The infrared light separation unit 13 performs an operation for separating visible light data and infrared light data for all the color signal data of the pixels included in the image sensor 122, and the first image output unit 14 uses this data as a basis. Generate an image. The control unit 15 a controls the image sensor 12, the infrared light separation unit 13, and the visible light image output unit 14.

  Alternatively, the infrared synthesizer 16 is further provided in the above configuration, and the R, G, B visible light data is converted into luminance data L and color difference data Cb, Cr, and the infrared rays separated into the luminance data L by the infrared separator 13. The data is added to obtain new luminance data L ′. This calculation is performed on the color signal data of all the pixels included in the image sensor 122, and the second image output unit 14 generates an image based on this data. The control unit 15 b controls the image sensor 12, the infrared light separation unit 13, and the visible light image output unit 14.

By the way, there are two types of color filters: primary color filters and complementary color filters. Generally, the primary color filter is excellent in color reproduction, and the complementary color filter is excellent in resolution. The primary color filter consists of four sets of red, green, blue and green. Two greens are included because much of the brightness felt by human eyes is due to green. The complementary color filters are a set of four colors of cyan, magenta, yellow, and green. Complementary colors are cyan, magenta, and yellow, but adding green to them increases the color reproducibility.

FIG. 2A shows the structure of a complementary color filter including a set of four colors of cyan, magenta, yellow, and green. In the imaging apparatus 10A according to the first embodiment, this CMYG complementary color filter 121 is used. Since each pixel that has passed through the CMYG complementary color filter 121 cannot recognize light that has not been transmitted, interpolation processing is performed. For example, in FIG. 2B, when attention is paid to the G11 pixel, only the amount of green light is output from this pixel. Since the amount of light of other colors for this pixel is often similar to the amount of light of that color for surrounding pixels, the average value of the amount of light for other colors of the surrounding pixels is calculated for the colors other than green in this pixel. Let it be the amount of light. This is interpolation.

If M00 is the magenta light amount in the pixel, C01 is the cyan light amount in the pixel, Y10 is the yellow light amount in the pixel, and G11 is the green light amount in the pixel, the following equation (1) The amount of light other than green in the G11 pixel can be obtained. This calculation formula is stored in the infrared light separation unit 13, and the calculation itself is also performed in the infrared light separation unit 13.

Formula (1)
M11 = (M00 + M02 + M20 + M22) / 4
C11 = (C01 + C21) / 2
Y11 = (Y10 + Y12) / 2

Color interpolation according to the equation (1) is the simplest and is called the binillia method. There is an interpolation method in which this is improved by further using the amount of ambient light for reasons such as poor resolution when performing color interpolation using this bilinear method, but the imaging apparatus according to the first embodiment described below In 10A, any interpolation method may be used as long as the light amounts of four colors C, M, Y, and G can be calculated for all pixels. This process is performed for all pixels. In preparation for such a case, for example, the pixel value on the left side cannot be used to obtain the cyan light amount because there is no pixel located on the left side in the pixel of M00 located at the end, for example. In many cases, extra pixels are arranged around the range of the image to be output in advance so that the same interpolation process can be performed in the entire range of the image to be output. Alternatively, special processing such as simply copying the value of C01 on the right side may be applied.

(Operation of Embodiment 1)
First, the operation of the infrared light separation unit 13 will be described. The CMYG complementary color filter 121 adds cyan light, magenta, yellow, and green color data to all of cyan, magenta, yellow, and green. That is, since the infrared light cut filter is not used, the infrared light data is also added to the value of the light amount of each pixel calculated by the infrared ray separation unit 13 by the above-described arithmetic expression (A), and this is used as it is. When an image is generated by one image output unit 14, the color reproducibility deteriorates. Therefore, in the imaging apparatus 10A according to the first embodiment, the infrared light separation unit 13 calculates the red, green, and blue color data by removing the infrared light component from the light amount value of each pixel by the arithmetic expression (A). In addition, infrared light data is also calculated.

Here, when color signal data including infrared light data is C ′, M ′, Y ′, and G ′, R, G, B data and infrared light data not including infrared light data are used. What is represented by I is shown in equation (2). This arithmetic expression is known as an arithmetic expression for performing color conversion from the complementary color system to the primary color system.

Formula (2)
C '= G + B + I
M ′ = R + B + I
Y '= R + G + I
G '= G + I

In the above equation (2), C ′, M ′, Y ′, and G ′ are cyan, magenta, yellow, and green, respectively, including infrared light data of that pixel or interpolated in a certain pixel. It is data of. R, G, and B are red, green, and blue data, respectively, that do not include infrared light data in the pixel. I is infrared light data in the pixel. Cyan includes green and blue, magenta includes red and blue, and yellow includes red and green. Here, each indicates that infrared data is included.

The arithmetic expression for calculating four values of R, G, B, and I from the four variables C ′, M ′, Y ′, and G ′ by this arithmetic expression (2) is as follows: , (4), (5), (6). All data output from the image sensor 12 is calculated as R, G, and B color data in the infrared light separation unit 13 using this arithmetic expression, and the infrared light data I is also calculated. Here, for the purpose of explaining the present invention, a process for deriving an arithmetic expression is also described. However, the arithmetic expression (A) stored in the infrared light separation unit 13 and used as the arithmetic expression is described above. ), That is, only the leftmost and rightmost sides of the arithmetic expressions (3), (4), (5), and (6).

Formula (3)
R = G-G + R + I-I
= (G + R + I)-(G + I)
= Y'-G '

Formula (4)
G = G + G-G + B-B + R-R + I + I-I-I
= (G + B + I) + (R + G + I)-(R + B + I)-(GI)
= C '+ Y'-M'-G'

Formula (5)
B = G-G + B + I-I
= (G + B + I)-(G + I)
= C'-G '

Formula (6)
I = G-G + I
= (G + I) -G
= G '-(C' + Y'-M'-G ')
= 2 × G′−C′−Y ′ + M ′

As described above, infrared light data can be calculated together with red, green, and blue data from a signal output from a general CMYG complementary color image sensor by a simple calculation. The infrared light separation unit 13 performs the calculation shown in the calculation formulas (3), (4), (5), and (6), that is, the calculation formula (A), for all the pixels, and visible light of red, green, and blue Output data. The first image output unit 14 generates and outputs a visible light image from the R, G, and B values calculated by the infrared light separation unit 13.

(Effect of Embodiment 1)
As described above, according to the imaging apparatus 10A according to the first embodiment, the infrared light separation unit 13 calculates the arithmetic expressions (3), (4), (5) from the signal acquired by the general image sensor 12. ), (6), that is, the calculation based on the calculation formula (A) is executed, and the infrared light data can be calculated together with the red, green and blue data. By performing this calculation for all the pixels, Visible light data having red, green, and blue elements that can form an image can be output. For this reason, visible light data can be obtained at low cost using an image sensor having a widely used general color filter without the need for an image sensor having a special color filter. One image output unit 14 can generate and output a visible light image. Moreover, since it can be realized with the same configuration as the normal camera component configuration except for removing the infrared light cut filter, the cost for the infrared light cut filter and its drive mechanism can be reduced, and the infrared light cut filter Visible light loss can also be prevented.

  In the imaging apparatus 10A according to the first embodiment, the infrared light separation unit 13 has been described as being configured independently of the control unit 15a, but the functions of the infrared light separation unit 13 are software. The same effect can be obtained even when the arithmetic expression (A) is stored in a storage unit (not shown) and is incorporated in the control unit 15a.

(Configuration of Embodiment 2)
FIG. 3 is a block diagram illustrating a configuration of the imaging apparatus 10B according to the second embodiment. The difference from Embodiment 1 shown in FIG. 1 is that an infrared light synthesis unit 16 is added. The infrared light synthesis unit 16 converts the visible light data and infrared light data output from the infrared light separation unit 13 into luminance data and color difference data including infrared light data, and outputs the second data. The unit 14 generates and outputs an image. Note that the control unit 15b and the infrared light synthesizer 16 separate the infrared light from the image sensor 12 so that the visible light image can be converted into a luminance signal and a color difference signal including infrared light. The unit 13 and the image output unit 14 are controlled. Other configurations are the same as those of the first embodiment shown in FIG.

(Operation of Embodiment 2)
Hereinafter, the infrared light synthesis method will be described. The visible light data and the infrared light data output from the infrared light separation unit 13 are input to the infrared light synthesis unit 16. The infrared light synthesizer 16 first converts the red, green, and blue values of the visible light data into luminance data and color difference data in accordance with the following general luminance / color difference arithmetic expression (7) stored in advance. .

Formula (7)
L = 0.3 × R + 0.6 × G + 0.1 × B
Cb = B−L
Cr = RL

In the above arithmetic expression (7), L represents luminance. Here, the brightness is the brightness perceived by humans and is usually expressed by Y. However, since Y is treated as yellow data here, the initial of brightness is taken. It was decided to indicate L. Cb and Cr are color differences and represent bluishness and redness. The color difference may take a negative value. If Cb is large, the color difference is bluish. If Cb is small, the color difference is reddish. If Cr is large, the color is reddish. If it is magenta, it is green if it is small, and if the absolute values of Cb and Cr are large, the color is dark, if it is small, the color is light, and if it is 0, it is white, black, or gray.

On the other hand, infrared light is essentially invisible to the human eye, but has properties similar to visible light. Therefore, if infrared light data is combined with visible light data, the amount of light increases, so that the subject can be captured even when the imaging range is dark. Therefore, in the imaging apparatus 10B according to the second embodiment, in order to synthesize the luminance data calculated by the infrared synthesizer 16 and the infrared light data output from the infrared separator 13, the arithmetic expression described above ( B) That is, the following equation (8) is used. In the imaging device 10 </ b> B according to the present embodiment, all input from the infrared light separation unit 13 using the arithmetic expression (8) for conversion into luminance data including infrared light stored in advance by the infrared light synthesis unit 16. The calculation is executed for the pixels of L, and L ′ is replaced as new luminance data and output. Note that the formulas Cb and Cr shown in the calculation formula (7) are used as they are for the color difference data so that the color reproducibility does not deteriorate even if the infrared light data is synthesized. Cb and Cr are calculated by the infrared ray synthesizer 16, and L ′, Cb, and Cr of all the pixels are output to the second image output unit 14, and an image is generated and output.

Formula (8)
L ′ = L + I × t

In the arithmetic expression (8), L ′ represents luminance data including infrared light data. The parameter t represents the degree to which the infrared light data is combined with the luminance data, and is in the range of 0 ≦ t (t is an arbitrary positive number including zero). Note that the value of the parameter t may be changed according to the situation where the control unit 15b determines the brightness of the imaging range. For example, if a coefficient shown in the range of 0 to 1 is applied to t, when the imaging range is bright, t = 0 is set to L ′ = L, which is the same as when no infrared light data is included. . Further, when the imaging range is dark, by setting t = 1, L ′ = L + I is obtained, and the original luminance data and infrared light data are evenly included.
For example, a table (not shown) that associates the brightness data with the value of the parameter t is stored in the infrared light combining unit 16 in advance. For all pixels in the imaging range, the value of the parameter t corresponding to the average value of the luminance data L obtained by the infrared light combining unit 16 according to the calculation formula (7) is selected from the above table, and calculated according to the calculation formula (8). Like.

(Effect of Embodiment 2)
According to the imaging device 10B according to the present embodiment described above, the infrared light combining unit 16 converts the luminance data into luminance data including the infrared light data according to the arithmetic expression (8) described above, It is possible to capture the subject even when the imaging range is dark by performing the processing on the pixel and generating and outputting the image as the image by the second image output unit 14 based on the data. In particular, according to the method disclosed in Patent Document 2, the infrared light component I is included in only one pixel per four, whereas the general CMYG complementary color used in the second embodiment is used. In the filter 121, since the infrared light data I is included in all the pixels, the resolution is superior to the method disclosed in Patent Document 2. In addition, since the amount of use of infrared light data can be gradually changed by changing the value of t in the calculation formula (8) in the infrared light separation unit 13, the infrared light as in Patent Document 1 is used. It is possible to avoid a sudden image change when switching from a visible light image to an infrared light image, which occurs when a cut filter is inserted and removed.

On the other hand, no matter how L 'changes, the color reproduction is hardly deteriorated because Cb and Cr indicate the hue and color density. Therefore, unlike the technique disclosed in Patent Document 1, it is not necessary to make the color darker or monochrome. By the way, although hue and color density do not change, the brightness may be greatly different between visible light and infrared light. For example, since the leaves of plants reflect a large amount of infrared light, the value of L ′ of the leaf portion is often larger than L, and the green color is maintained, but it becomes bright green. . In order to reduce this, it is possible to extract the edge component from the value of the infrared light component I and combine only the edge portion with L or perform processing such as a tone curve filter and then combine with L. Conceivable.

  In the imaging device 10B according to the second embodiment, the infrared light combining unit 16 has been described as being configured independently of the control unit 15b. However, the functions of the infrared light combining unit 16 are software. Even if it is realized and incorporated in the control unit 15b, the same effect can be obtained.

  As mentioned above, although preferred embodiment of this invention was explained in full detail, it cannot be overemphasized that the technical scope of this invention is not limited to the range as described in the said embodiment. It will be apparent to those skilled in the art that various modifications or improvements can be added to the above-described embodiments. Further, it is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention.

  DESCRIPTION OF SYMBOLS 10A, 10B ... Imaging device, 12 ... Image sensor, 13 ... Infrared light separation part, 14 ... Image output part, 15a, 15b ... Control part, 16 ... Infrared light composition Part

Claims (3)

A CMYG complementary color filter, an image sensor,
Color signal data C ′, M ′, Y ′, and G ′ including infrared light imaged by the image sensor through the CMYG complementary color filter are input, and the following arithmetic expression (A) stored in advance is used. An infrared light separating unit that calculates and outputs color signal data R, G, B and infrared light data I of visible light;
An imaging apparatus comprising: a control unit that controls these.
Formula (A)
R = Y'-G '
G = C ′ + Y′−M′−G ′
B = C'-G '
I = 2 × G′−C′−Y ′ + M ′   The imaging apparatus according to claim 1, further comprising a first image output unit that inputs the visible color signal data R, G, and B, and generates and outputs a visible light image. The visible light color signal data R, G, and B output from the infrared light separation unit are input, converted into luminance signal data L and color difference signal data Cb, Cr, and the following arithmetic expression (B ) To calculate and output new luminance signal data L ′ from the luminance signal data L, infrared light signal data I and an arbitrary positive number t including zero,
The imaging apparatus according to claim 1, further comprising a second image output unit that generates and outputs an image from the new luminance signal data L ′ and the color difference data Cb and Cr.
Formula (B)
L ′ = L + I × t (t is any positive number including zero)

Images