JP7150514B2 - Imaging device and imaging method - Google Patents

Description

The present invention relates to an imaging device and an imaging method.

Conventionally, an image sensor (image sensor) having spectral characteristics other than the three primary colors of light of red (R), green (G), and blue (B), called a multiband image sensor (multi-primary color image sensor), has been developed. . By acquiring various wavelength components independently, multiband image sensors can, for example, clearly display hazy scenery, or distinguish the color of leaves of plants that look the same green to the naked eye and determine the degree of growth. It is expected to be applied in various fields such as

As a technique for realizing this spectral characteristic, a technique of arranging a color filter having a spectral sensitivity characteristic that transmits only a desired wavelength band for each pixel of an image sensor is common. For example, Patent Document 1 proposes a pixel array of an image sensor that is combined with conventional RGB and has spectral characteristics corresponding to various wavelength bands.

Also, some multiband image sensors have spectral characteristics that include not only the visible light range visible to humans, but also the invisible light range such as ultraviolet light and infrared light. For example, in Patent Document 2, a pixel array is formed in which IR (Infra Red) pixels corresponding to infrared light are provided in addition to pixels corresponding to RGB, and IR pixel signals are used to prevent shading.

JP 2012-44519 A JP 2011-29810 A

A conventional imaging device does not require light other than light in the visible light region that can be visually recognized by humans, for example, light other than RGB (red, green, and blue) light. Therefore, the components in the ultraviolet light region can be excluded as a result because the ultraviolet light is absorbed by the characteristics of the glass lens. It was excluded by arranging. However, when it is desired to use the invisible light region, for example, when using infrared light, it is not possible to attach an infrared light cut filter.

RGB color filters are known to have spectral sensitivity characteristics that transmit not only visible light components but also infrared light components. Therefore, unless an infrared light cut filter is arranged in front of the image sensor, the RGB color filter transmits not only the components of the region corresponding to RGB but also the components of the infrared light region. As a result, there is a problem that the pixels of the image sensor cannot acquire correct luminance values for each wavelength component of RGB.

On the other hand, in Patent Document 2, luminance correction of RGB is performed by subtracting the luminance component obtained from the IR pixel from the luminance component of the pixel corresponding to RGB. The amount of infrared light transmitted through each of the RGB color filters has different characteristics depending on the wavelength, but the conventional technology does not take into account the variation in the spectral characteristics of the RGB color filters in the infrared region.

Therefore, if RGB luminance correction is simply performed using the luminance components of IR pixels that cover a wide wavelength range, even if the R, G, and B color filters detect wavelength components that have no sensitivity at all, Correction including that component may result in overcorrection. SUMMARY OF THE INVENTION It is an object of the present invention to prevent overcorrection and perform luminance correction with high precision.

In the imaging device according to the present invention, a plurality of visible light pixel regions respectively corresponding to a plurality of different color components in the visible light region and invisible light pixel regions corresponding to the color components in the invisible light region are arranged, and the plurality of an imaging device in which a plurality of pixels having different spectral sensitivity characteristics are arranged in each of the visible light pixel region and the invisible light pixel region of each of the plurality of visible light pixel regions, and the spectral sensitivity characteristics and the acquisition means for acquiring a correction coefficient for each pixel in each of the plurality of visible light pixel regions based on luminance information of pixels in the invisible light pixel regions ; and correction processing means for correcting luminance information of each pixel in each of the spectral wavelength bands of each of the plurality of pixels arranged in each of the plurality of visible light pixel regions and the invisible light pixel region, It is characterized by being narrower than the wavelength band of each color component corresponding to each pixel region .

According to the present invention, it is possible to prevent overcorrection in luminance correction and perform luminance correction with high precision.

It is a figure which shows the structural example of the imaging device in embodiment of this invention. It is a figure which shows the example of the pixel arrangement|sequence in a Bayer arrangement. It is a figure which shows the example of arrangement|positioning of an infrared-light cut filter. It is a figure which shows the example of the spectral sensitivity characteristic in a Bayer array. FIG. 3 is a diagram showing an example of an RGB+IR pixel array; FIG. 4 is a diagram showing an example of spectral sensitivity characteristics in an RGB+IR pixel array; FIG. 3 is a diagram illustrating an example of a pixel array of an imaging device according to the first embodiment; FIG. FIG. 4 is a diagram showing an example of spectral sensitivity characteristics related to the B component in the first embodiment; FIG. 4 is a diagram showing an example of spectral sensitivity characteristics related to IR components in the first embodiment; FIG. 4 is a diagram illustrating an example of luminance correction in the first embodiment; FIG. FIG. 4 is a diagram showing an example of spectral sensitivity characteristics for B and IR components in the first embodiment; FIG. 10 is a diagram showing an example of spectral sensitivity characteristics for B and IR components in the second embodiment; FIG. 11 is a diagram showing an example of spectral sensitivity characteristics for each RGB component and IR component in the third embodiment;

BEST MODE FOR CARRYING OUT THE INVENTION An embodiment of the present invention will be described below with reference to the drawings.

(First embodiment)
A first embodiment of the present invention will be described. FIG. 1 is a block diagram showing a configuration example of an imaging device according to one embodiment of the present invention. The imaging apparatus 100 in this embodiment has an optical lens 101 , an imaging element 102 , an image engine 103 , an image display element 114 , an image recording medium 115 and an image output terminal 116 . The imaging apparatus 100 further includes a mode microcomputer 117, a ROM (Read Only Memory) 118, a RAM (Random Access Memory) 119, an operation section 120, a power supply section 121, an oscillation section 122, and the like.

The imaging apparatus 100 condenses incident light relating to a video (object image) with an optical lens 101 and receives an image with an imaging device 102 . The optical lens 101 is made of a material such as glass or fluorite, and reduces incident light relating to an image (object image) to match the size of the imaging device 102 . For the imaging device 102, for example, a CCD (Charge Coupled Device) image sensor, a CMOS (Complementary Metal Oxide Semiconductor) image sensor, or the like is used.

The imaging device 102 performs photoelectric conversion for converting an optical signal into an electrical signal, and converts incident light relating to an image (object image) into an electrical signal. Since the electric signal obtained here is an analog value, the image sensor 102 also has an analog-to-digital conversion function for converting into a digital value. Note that the image sensor 102 may output an analog signal, and an analog-to-digital converter may be arranged after the image sensor 102 to convert it into a digital value.

The video engine 103 is an LSI (Large Scale Integration) specialized for control and processing related to video, and has various functions for driving and controlling the imaging apparatus 100 . The image engine 103 includes an image distribution unit 104, a visible image processing unit 107, an invisible image processing unit 108, a spectral characteristic comparison unit 109, a spectral characteristic storage unit 110, a correction coefficient acquisition unit 111, an image synthesis unit 112, and an additional information superimposition unit. 113. The image engine 103 also includes an image sensor control unit 105 that controls driving of the image sensor 102 and applies analog gain, and a lens control unit 106 that drives the optical lens 101 and controls image zooming, focusing, and the like.

The image distribution unit 104 supplies the visible image processing unit 107, the invisible image processing unit 108, and the spectral characteristic comparison unit 109 with the luminance information of each pixel converted into an electric signal by the image sensor 102 for each required wavelength. Distribute. For example, the image distribution unit 104 supplies the visible image processing unit 107 with luminance information obtained from pixels corresponding to a visible light region, which is a wavelength region visible to the human eye. In addition, the image distribution unit 104 supplies the invisible image processing unit 108 with luminance information obtained from pixels corresponding to the invisible light region, which is a wavelength region invisible to the human eye.

The visible image processing unit 107 performs processing related to luminance information of pixels corresponding to the visible light region supplied from the image distribution unit 104 . The visible image processing unit 107 performs luminance correction based on, for example, the correction coefficient supplied from the correction coefficient acquisition unit 111, and corrects luminance information of pixels corresponding to the visible light region.

The invisible image processing unit 108 performs processing related to luminance information of pixels corresponding to the invisible light region supplied from the image distribution unit 104 . The invisible image processing unit 108 performs luminance correction, for example, based on the correction coefficient supplied from the correction coefficient acquisition unit 111, and corrects luminance information of pixels corresponding to the invisible light region. In addition, the invisible image processing unit 108 visualizes luminance information obtained from pixels corresponding to the invisible light region, for example, so that the information can be visually recognized by human eyes.

The spectral characteristic comparison unit 109 acquires color filter characteristic information related to each color filter from the spectral characteristic storage unit 110, and compares and compares the luminance information of each pixel and the color filter characteristic information supplied from the video distribution unit 104. A result is output to the correction coefficient acquisition unit 111 . The spectral characteristic storage unit 110 stores pre-obtained spectral sensitivity characteristics of each color filter arranged for each pixel of the image sensor 102, that is, color filter characteristic information indicating the spectral sensitivity characteristic of each pixel of the image sensor 102. do.

A correction coefficient acquisition unit 111 acquires a correction coefficient for luminance information of each pixel of the image sensor 102 based on the comparison result of the spectral characteristic comparison unit 109 . The correction coefficients acquired by the correction coefficient acquisition unit 111 are supplied to the visible image processing unit 107 and the invisible image processing unit 108 .

The video synthesizing unit 112 appropriately synthesizes the video output from the visible video processing unit 107 and the video output from the invisible video processing unit 108 and outputs the synthesized video. For example, the image synthesizing unit 112 synthesizes the image from the visible image processing unit 107 subjected to luminance correction and the image visualized by the invisible image processing unit 108 so as to be visible to humans, and outputs the synthesized image.

The additional information superimposing unit 113 adds information such as OSD (On Screen Display) to the video output from the video synthesizing unit 112 as necessary, and sends the information to the video display element 114, the video recording medium 115, and the video output terminal 116. do.

The video display element 114 displays captured video, setting menus, and the like. The user can visually recognize the operation status of the image pickup apparatus 100 by the display on the image display element 114 . The image display element 114 is a small, low power consumption display device such as an LCD (Liquid Crystal Display) or an organic EL (Organic Electroluminescence).

The video recording medium 115 is a recording medium capable of recording video. The video recording medium 115 is a large-capacity recording medium such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive). The video output terminal 116 is an interface for outputting video to the outside, and is, for example, an analog composite video terminal or a digital video terminal. Examples of digital video terminals include terminals conforming to the SDI (Serial Digital Interface) standard and terminals conforming to the HDMI (registered trademark) (High Definition Multimedia Interface) standard.

A mode microcomputer 117 performs mode control of the imaging apparatus 100 . A ROM 118 is a nonvolatile memory that stores initial values and the like, and a RAM 119 is a volatile memory that temporarily stores programs, images, and the like. The operation unit 120 uses elements such as mechanical buttons and switches to accept user operations. The power supply unit 121 supplies power to each functional unit in the imaging device 100 , and the oscillator 122 generates a reference clock for generating a clock to be supplied to each functional unit in the imaging device 100 .

Next, the imaging device and its spectral sensitivity characteristics will be described. First, the reference technology leading to the present invention will be described, and then the imaging element and its spectral sensitivity characteristics in this embodiment will be described. In the following description, "R component color filter" indicates a filter whose main transmission wavelength band is the R component (red component), and "G component color filter" indicates a filter whose main transmission wavelength band is the G component ( green component). In addition, "B component color filter" indicates a filter whose main transmission wavelength band is the B component (blue component), and "IR component color filter" indicates a filter whose main transmission wavelength band is the IR component (infrared light component). indicates a filter that is

FIG. 2 is a diagram showing an example of a pixel array of an imaging device as a first reference technology. The pixel array shown in FIG. 2 is called a Bayer array, in which color filters of RGB colors are arranged in a checkered pattern. For example, the pixel 201 has an R component color filter, the pixels 202 and 203 have G component color filters, and the pixel 204 has a B component color filter.

Along with the development of development processing technology for interpolating pixels of a required color from surrounding pixels, the Bayer array is an array that is often used in single-chip imaging device cameras. The color filters for each of these RGB components have the characteristic of transmitting not only light of wavelengths corresponding to the respective colors but also infrared light. An IR) cut filter 301 is generally used.

FIG. 4 shows an example of the spectral sensitivity characteristics of an image sensor when color filters for each component of RGB are used. FIG. 4A is a diagram showing the transmittance of each component in the color filter in the configuration shown in FIG. 3, that is, in the state where the infrared light (IR) cut filter 301 is placed in front of the image sensor. FIG. 4B is a diagram showing the transmittance of each component in the color filter when the infrared light (IR) cut filter 301 is not arranged. 4(a) and 4(b) are indicated by a B component 401, a G component 402, and an R component 403, respectively.

As shown in FIG. 4(a), when an infrared light (IR) cut filter 301 is placed in front of the image sensor, the effect of the infrared light (IR) cut filter 301 reduces wavelengths longer than 700 nm, which are invisible to humans. of light is absorbed and unnecessary wavelengths are cut. On the other hand, when the infrared light (IR) cut filter 301 is not arranged as shown in FIG. It also has sensitivity to light in the area.

Here, the spectral sensitivity characteristics of the color filters for each component in the infrared region differ depending on the wavelength as shown in FIG. 4(b). For example, when focusing on light with a wavelength of 700 nm and when focusing on light with a wavelength of 850 nm, the spectral sensitivity characteristics of each of the RGB components are significantly different. In the 700 nm wavelength band, the transmittance of the R component color filter is high, but the transmittance of the B component color filter is low. On the other hand, in the 850 nm wavelength band, the transmittance of the B component color filter is high, but the transmittance of the R component color filter is low. In other words, it means that the characteristics of the color filter differ greatly depending on the wavelength band.

Next, as a second reference technology, consider a case where a pixel array of RGB+IR corresponding to infrared light (IR) as shown in FIG. 5 is adopted. This pixel array is a pixel array in which one of the two G-component color filters in the Bayer array is replaced with an IR-component color filter. FIG. 5 shows an example in which the pixels 203 in which the G component color filters are arranged in the example shown in FIG. 2 are replaced with pixels 501 in which the IR component color filters are arranged.

When adopting such an RGB+IR pixel arrangement, an infrared light cut filter cannot be arranged in front of the image sensor because the infrared light region is used. Therefore, the spectral sensitivity characteristic of the image sensor becomes as shown in FIG. 6, and an IR component 601 is added in this pixel array. In addition, since the infrared light cut filter is not arranged, it becomes obvious that the color filters of each component of RGB have the characteristic of transmitting the infrared light region.

In the RGB+IR pixel array as shown in FIG. 5, the infrared light component contained in the luminance component of the RGB pixels in which the color filters for each of the RGB components are arranged is converted to the IR component in the IR pixels in which the color filters for the IR components are arranged. Consider removing using the luminance component. As shown in FIG. 6, the characteristics of the IR component color filter cover a wide wavelength range in the same way as the RGB component color filter. I don't know if there are.

For example, assume that light with a wavelength of 900 nm is strong, and based on the luminance component of the IR pixel at that time, luminance correction is performed to remove the IR component from the RGB pixels. Here, as a correction method for removing the IR component, for example, a technique as described in Patent Document 2 may be used. In this case, as shown in FIG. 6, although the color filter for each component of RGB has almost no sensitivity to light with a wavelength of 900 nm, luminance correction is performed on the RGB pixels. . This results in over-correction, and the IR component contained therein cannot be accurately removed from the luminance components of the RGB pixels.

Next, the imaging device and its spectral sensitivity characteristics in this embodiment will be described. The present invention assumes an imaging apparatus having an imaging element with four or more spectral sensitivity characteristics including infrared light. FIG. 7A is a diagram showing an example of the pixel array of the imaging element in this embodiment. As shown in FIG. 7A, 1 to 16 color filters (CF) having different spectral sensitivities are arranged in the pixel array of the imaging element in this embodiment. The 16 pixels in which the color filters (CF1 to CF16) are arranged are taken as one group, and the color filters are arranged vertically and horizontally in the same arrangement. By making the spectral sensitivity characteristics of the respective color filters different from each other and further narrowing the spectral wavelength band, the imaging element in this embodiment can acquire luminance values for 16 wavelengths.

Further, the 16 pixels are divided into groups of 4 pixels, and the spectral sensitivity characteristics of the color filters of the 4 pixels are grouped together with relatively close components in terms of wavelength. FIG. 7(b) shows an example of the pixel array of the image pickup device in which the color filters are arranged in such a manner. In the example shown in FIG. 7B, wavelengths close to the R component (R1 to R4), wavelengths close to the G component (G1 to G4), wavelengths close to the B component (B1 to B4), Filters having spectral sensitivity characteristics for wavelengths (IR1 to IR4) close to IR components are summarized.

FIG. 7(c) shows a pixel array showing specific examples of corresponding wavelengths of each pixel. That is, the image sensor in this embodiment has a pixel region corresponding to each of the three colors (R component, G component, and B component) in the visible light region and a pixel region corresponding to the IR component in the invisible light region. , each of which has a pixel array in which a plurality of pixels with different spectral sensitivity characteristics are arranged.

With such a pixel array, by synthesizing and using the luminance values of four pixels close to the R component in which the color filters (R1 to R4) are arranged, the wavelength component equivalent to that of the R pixel in the conventional pixel array can be obtained. can generate pixels with Further, by performing similar processing on G pixels, B pixels, and IR pixels, pixels having wavelength components equivalent to those of G pixels, B pixels, and IR pixels in a conventional pixel array can be generated. It is possible to realize an imaging device with an RGB+IR pixel array. Such a configuration also has the advantage of being highly compatible with conventional video engines, since the concept of signal processing used in the conventional Bayer array can be used.

As a representative example, FIG. 8A shows the spectral sensitivity characteristics of four pixels forming the B component. For example, the spectral sensitivity characteristic of a pixel provided with a color filter (B1) is B component 1 (801), and the spectral sensitivity characteristic of a pixel provided with a color filter (B2) is B component 2 (802). Further, the spectral sensitivity characteristic of the pixel on which the color filter (B3) is arranged is B component 3 (803), and the spectral sensitivity characteristic of the pixel on which the color filter (B4) is arranged is B component 4 (804).

By using all the finely divided wavelength components in this way, it is possible to form a pixel covering a wide wavelength range close to the conventional B component. To form these pixels, four pixels are used for each color component, which is four times the normal number. Current super multi-pixel sensors such as 4K pixel sensors and 8K pixel sensors have a sufficient number of pixels, and are within a practical range for video. As described above, the imaging device of this embodiment can observe various wavelengths in detail in addition to the conventional RGB image and IR image.

Next, luminance correction processing using an imaging device having a pixel array having spectral sensitivity characteristics as illustrated in FIG. 7B will be described. As described above, an infrared light cut filter cannot be used in an image pickup apparatus that is premised on using an IR component that is in the infrared light region. In that case, some of the color filters for each component of RGB transmit infrared light. As an example, FIG. 8B shows spectral sensitivity characteristics of four pixels forming the B component without an infrared light cut filter. As shown in FIG. 8B, some of the color filters transmit light in the infrared region.

In order to simplify the explanation, the luminance correction will be explained below using a color filter with the highest transmittance for the B component as an example. FIG. 9 shows the spectral sensitivity characteristics of a pixel in which one of the narrow-band color filters (B3) forming the B component is arranged, and the spectral sensitivity of the pixels in which the color filters (IR1 to IR4) forming the IR component are arranged. It is a figure which shows together with a characteristic.

In FIG. 9, it is assumed that the spectral sensitivity characteristic of the pixel in which the color filter (B3) is arranged is B component 3 (803). In addition, the spectral sensitivity characteristic of the pixel on which the color filter (IR1) is arranged is IR component 1 (901), and the spectral sensitivity characteristic of the pixel on which the color filter (IR2) is arranged is IR component 2 (902). and In addition, the spectral sensitivity characteristic of the pixel on which the color filter (IR3) is arranged is IR component 3 (903), and the spectral sensitivity characteristic of the pixel on which the color filter (IR4) is arranged is IR component 4 (904). and

In the example shown in FIG. 9, the color filter (B3) for the B component transmits not only light with a wavelength of around 500 nm, but also light with a wavelength of around 800 nm, which is in the infrared region. On the other hand, the color filter (IR2), which is one of the color filters for the IR component, is a narrow-band color filter having spectral sensitivity near the transmission wavelength in the infrared region of the color filter (B3) for the B component. It is a filter.

For example, it is assumed that the light received by the imaging element has a strong component in the vicinity of 800 nm. At this time, it is detected that the pixel forming the B component in which the color filter (B3) is arranged has a light amount. In addition, a pixel in which an IR component color filter having spectral sensitivity to light with a wavelength of about 800 nm is arranged (in this example, a pixel in which a color filter (IR2) is arranged) is also detected as having a light amount.

In this case, if the characteristics of the narrow-band B component color filter and the narrow-band IR component color filter characteristics are known in advance, Using the obtained luminance value, luminance correction can be performed with high accuracy. That is, it is possible to appropriately remove the IR component contained in the luminance component of the B pixel by using the luminance value obtained from the pixel of the color filter having spectral sensitivity to light around 800 nm.

A specific correction procedure will be described together with the processing of the imaging device 100 . In the imaging apparatus 100, light of various wavelengths incident from the outside is condensed by an optical lens 101 and received by an imaging element 102 having a pixel array as shown in FIG. 7B. The luminance information of each wavelength converted into an electric signal by the imaging device 102 is distributed by the image distribution unit 104 for each required wavelength.

For example, the luminance information of the pixels corresponding to the visible light region, which is the wavelength region visible to the human eye, is divided into the visible image processing unit 107, and the luminance information of the pixels corresponding to the invisible light region, which is the wavelength region invisible to the human eye. The luminance information is divided into the invisible image processing unit 108 . Here, for example, when the IR component is strongly detected, there is a possibility that each component of RGB includes an excessive IR component.

Further, the luminance value of each wavelength input by the image distribution unit 104 and the color filter characteristic information acquired from the spectral characteristic storage unit 110 are compared by the spectral characteristic comparison unit 109, and the comparison result is obtained by the correction coefficient acquisition unit 111. is sent to Then, based on the comparison result of the spectral characteristic comparison unit 109, the correction coefficient acquisition unit 111 calculates a correction coefficient for luminance correction processing. The calculated correction coefficients are output to the visible image processing unit 107 and the invisible image processing unit 108, and the visible image processing unit 107 and the invisible image processing unit 108 perform luminance correction processing for each component based on the correction coefficients.

After a series of processes are completed, the output image of the visible image processing unit 107 whose brightness has been corrected and the image visualized by the invisible image processing unit 108 are appropriately synthesized by the image synthesizing unit 112 . After information such as OSD is added by the additional information superimposing unit 113 as necessary, the information is sent to the image display element 114, the image recording medium 115, and the image output terminal 116. FIG.

A specific example of luminance correction will be described with reference to FIG. Here, an example in which the amount of light is obtained with light having a wavelength of 800 nm as the IR component will be described. The spectral characteristic storage unit 110 stores a transmittance characteristic 1002 of a narrow-band B component color filter and a transmittance characteristic 1001 of a narrow-band IR component color filter for light with a wavelength of 800 nm. In other words, the spectral characteristic storage unit 110 stores information indicating the ratio of the amount of transmission of the IR component through the color filter and the amount of transmission of the B component through the color filter when the amount of incident light with a wavelength of 800 nm is the same. keeping.

Here, the luminance value obtained from the pixel (B pixel) in which the B component color filter is arranged is the sum of the B luminance component 1011 due to the B component light and the IR luminance component 1012 due to the IR component light. . A pixel (IR pixel) in which an IR component color filter is arranged obtains a luminance value 1021 of only the infrared light region, that is, only the light of the IR component.

Based on the transmittance characteristic ratio (transmission amount ratio) of the narrow-band B-component color filter and the IR-component color filter described above, the luminance value obtained by the B pixel is included in the luminance value obtained by the B pixel from the luminance value obtained by the IR pixel. A luminance value 1031 can be calculated from the IR component. For example, for light with a wavelength of 800 nm, the transmittance of the B component color filter is assumed to be (1/2) the transmittance of the IR component color filter. In this case, if the brightness value obtained from the IR pixel is 100, the brightness value of the IR component included in the brightness value obtained from the B pixel is 50.

By subtracting the calculated brightness value 1031 of the IR component from the brightness value obtained by the B pixel, the brightness value 1041 of the B pixel from which the IR brightness component 1012 of the IR component light has been appropriately removed is obtained. In this manner, overcorrection can be prevented for the B pixels in which the narrow-band B component color filter is arranged, and luminance correction can be performed with high accuracy. Here, the B pixel in which the narrow-band B component color filter is arranged has been described as an example, but the same applies to each of the other pixels.

According to the first embodiment, it is possible to acquire luminance information in a plurality of wavelength bands by using an imaging device having a pixel array in which a plurality of narrow wavelength band color filters having different spectral sensitivity characteristics are arranged. . In addition, the IR component included in the luminance information acquired from each pixel can be appropriately removed without overcorrection, and the luminance can be corrected with high accuracy.

(Second embodiment)
Next, a second embodiment of the invention will be described. In the following, only points of the imaging apparatus according to the second embodiment that are different from the above-described first embodiment will be described. As described above, the spectral sensitivity characteristics of the color filters of each component are various. As an example is shown in FIG. 8B, even a B-component color filter may have various transmission characteristics in the infrared region. In order to further improve the accuracy of the luminance correction processing described above, it is also possible to set the center of the transmission wavelength of the color filter for the IR component in accordance with the transmission characteristics of the infrared region of the color filter for each component of RGB. .

Hereinafter, the B component will be described as an example. For example, as shown in FIGS. 11(a) and 11(b), the peak wavelength in the infrared region of B components 1-4 (801-804) and the IR components 1-4 (901-904) There is a shift in the wavelength of the peak. This peak wavelength deviation affects the accuracy of luminance correction, and if the peak wavelength deviation is large, the accuracy of luminance correction decreases. Therefore, as shown in FIGS. 12(a) and 12(b), peak wavelengths in the infrared region of B components 1 to 4 (801 to 804) and IR components 1 to 4 (1201 to 1204) ), the filter characteristics are selected in advance so that the wavelengths of the peaks of ) match.

In this way, adjustment is performed in advance so that the spectral sensitivity characteristics in the infrared light region of the pixels in which the B component color filters are arranged and the spectral sensitivity characteristics of the pixels in which the IR component color filters are arranged are matched. As a result, there is no peak wavelength shift in the spectral sensitivity characteristics, and the accuracy of luminance correction can be improved. It should be noted that it is important to strike a balance in consideration of the spectral sensitivity characteristics of the pixels in which the R and G component color filters are arranged, since the pixels are not limited to those in which the B component color filter is arranged. It is also necessary to consider the wavelength in the optical region.

(Third Embodiment)
Next, a third embodiment of the invention will be described. Only the differences from the above-described embodiments in the imaging apparatus according to the third embodiment will be described below. As the number of types of color filters arranged on the image sensor 102 increases, the amount of information increases accordingly. As a result, the amount of processing in the image distribution unit 104, the spectral characteristic comparison unit 109, the spectral characteristic storage unit 110, the correction coefficient acquisition unit 111, and the like in the imaging apparatus 100 increases. An increase in the amount of information and the amount of processing may affect the processing capability, power consumption, and the like of the imaging device 100 .

Therefore, in order to reduce the processing, for each pixel region, for each pixel region, narrow wavelength components obtained from pixels in the pixel region are combined in advance to obtain combined luminance information, and luminance correction processing is performed. good too. For example, as shown in FIGS. 13A and 13B, narrow wavelength IR components 1 to 4 (1311 to 1314 ) can be used for luminance correction. For example, in the imaging device as shown in FIG. 7B, it is possible to perform correction processing using the luminance information of the IR pixels (IR1 to IR4) for the following synthesis result.
R (composite) = R1 + R2 + R3 + R4
G (composite) = G1 + G2 + G3 + G4
B (composite) = B1 + B2 + B3 + B4
By synthesizing narrow-wavelength R, G, and B components in advance in this manner, the amount of processing in the imaging apparatus 100 can be reduced, which is an effective technique for implementation with limited resources.

It should be noted that the above-described embodiments are not limited to being realized independently, and some of the above-described embodiments may be combined as appropriate. Further, in the above-described embodiment, an example in which each color component is composed of four pixels has been shown, but the correction accuracy can be improved by increasing the types of color filters.

Further, in the above description, it was mainly described that the luminance information of the RGB pixels is corrected using the IR components obtained by the IR pixels. Correction is also possible. Further, in order to correct the luminance information of a predetermined color (eg, R component) of RGB pixels, luminance correction is performed using luminance information of other colors (eg, G component and B component) different from the predetermined color. is also possible. Moreover, it is possible to develop not only the IR component but also the UV (Ultraviolet) component, which is in the ultraviolet light region. It is also possible to record information of each wavelength component on a recording medium or the like, and perform luminance correction using image processing software or the like after photographing.

(Another embodiment of the present invention)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

It should be noted that the above-described embodiments are merely examples of specific implementations of the present invention, and the technical scope of the present invention should not be construed to be limited by these. That is, the present invention can be embodied in various forms without departing from its technical concept or main features.

100: Imaging device 101: Optical lens 102: Imaging element 103: Image engine 104: Image distributing unit 105: Imaging element driving unit 106: Lens control unit 107: Visible image processing unit 108: Invisible image processing unit 109: Spectral characteristic comparison unit 110: Spectral characteristic storage unit 111: Correction coefficient acquisition unit 112: Video composition unit 113: Additional information superimposition unit

Claims (8)

A plurality of visible light pixel regions respectively corresponding to a plurality of different color components of the visible light region and an invisible light pixel region corresponding to the color components of the invisible light region are arranged, and the plurality of visible light pixel regions and the invisible light pixel regions are arranged. an imaging device in which a plurality of pixels having different spectral sensitivity characteristics are arranged in each of the optical pixel regions ;
A correction coefficient for each pixel in each of the plurality of visible light pixel regions is obtained based on the spectral sensitivity characteristics of each pixel in each of the plurality of visible light pixel regions and luminance information of pixels in the invisible light pixel region . acquisition means;
correction processing means for correcting luminance information of each pixel in each of the plurality of visible light pixel regions based on the correction coefficient ;
The spectral wavelength band of each of the plurality of pixels arranged in each of the plurality of visible light pixel regions and the invisible light pixel region is narrower than the wavelength band of each color component corresponding to each pixel region. Device. Comparing means for comparing the spectral sensitivity characteristic of each pixel in each of the plurality of visible light pixel regions with the luminance information of the pixels in the invisible light pixel region ,
2. The imaging apparatus according to claim 1, wherein said acquiring means acquires said correction coefficient based on the comparison result of said comparing means. In the visible light pixel area, a first pixel area in which the plurality of pixels having the spectral sensitivity characteristic related to the red component are arranged, and a second pixel area in which the plurality of pixels having the spectral sensitivity characteristic related to the green component are arranged. and a third pixel region in which the plurality of pixels having the spectral sensitivity characteristics related to the blue component are arranged ,
3. The imaging according to claim 1, wherein a fourth pixel region in which the plurality of pixels having the spectral sensitivity characteristic related to the infrared light component are arranged is arranged in the invisible light pixel region. Device. 4. The spectral sensitivity characteristic of each pixel in the invisible light pixel area is set according to the spectral sensitivity characteristic of each pixel in each of the plurality of visible light pixel areas. 1. The imaging device according to claim 1. The obtaining means associates a correction coefficient of each pixel in the visible light pixel region corresponding to a predetermined color component in the visible light region with a color component different from the predetermined color component in the visible light region. 5. The image pickup device according to claim 1 , wherein the image pickup device is obtained based on luminance information of each pixel in the visible light pixel region. The acquisition means acquires the correction coefficient for synthesized luminance information obtained by synthesizing luminance information of each pixel in each of the plurality of visible light pixel regions,
6. The imaging apparatus according to any one of claims 1 to 5 , wherein said correction processing means corrects said synthetic luminance information. A plurality of visible light pixel regions respectively corresponding to a plurality of different color components of the visible light region and an invisible light pixel region corresponding to the color components of the invisible light region are arranged, and the plurality of visible light pixel regions and the invisible light pixel regions are arranged. An imaging method for an imaging device having an imaging element in which a plurality of pixels having different spectral sensitivity characteristics are arranged in each of the optical pixel regions ,
the spectral wavelength band of each of the plurality of pixels arranged in each of the plurality of visible light pixel regions and the invisible light pixel region is narrower than the wavelength band of each color component corresponding to each pixel region;
A correction coefficient for each pixel in each of the plurality of visible light pixel regions is obtained based on the spectral sensitivity characteristics of each pixel in each of the plurality of visible light pixel regions and luminance information of pixels in the invisible light pixel region . an acquisition step;
and a correction processing step of correcting luminance information of each pixel in each of the plurality of visible light pixel regions based on the correction coefficient. A plurality of visible light pixel regions respectively corresponding to a plurality of different color components of the visible light region and an invisible light pixel region corresponding to the color components of the invisible light region are arranged, and the plurality of visible light pixel regions and the invisible light pixel regions are arranged. A program for causing a computer to execute processing in an imaging device having an imaging device in which a plurality of pixels having different spectral sensitivity characteristics are arranged in each of the optical pixel regions ,
the spectral wavelength band of each of the plurality of pixels arranged in each of the plurality of visible light pixel regions and the invisible light pixel region is narrower than the wavelength band of each color component corresponding to each pixel region;
A correction coefficient for each pixel in each of the plurality of visible light pixel regions is obtained based on the spectral sensitivity characteristics of each pixel in each of the plurality of visible light pixel regions and luminance information of pixels in the invisible light pixel region . an acquisition step;
and a correction processing step of correcting luminance information of each pixel in each of the plurality of visible light pixel regions based on the correction coefficient.

Images