JP2013187711A - Image processing apparatus, imaging apparatus, and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus, an imaging apparatus, an image processing method, etc. capable of noise reduction independent of the spatial or temporal characteristics of a pixel value.SOLUTION: The image processing apparatus comprises: an image acquiring unit which acquires an image captured by an imaging device including first to third filters with first to third transmittance characteristics; a multiband estimation unit which sets first to fifth bands corresponding to the overlapping portions and not-overlapping portions of the first to third transmittance characteristics and estimates the component values {b,b,b,g,r,r,r} of the first to fifth bands on the basis of the pixel values of first to third colors constituting the image; and a noise reduction processing unit which performs reconstitution of the pixel values {B',G',R'} of the first to third colors from the first to fifth bands.

Description

  The present invention relates to an image processing device, an imaging device, an image processing method, and the like.

  Due to the digitization of imaging devices such as cameras, microscopes, and endoscopes, there are remarkable improvements in and expansion of various imaging functions. Noise reduction processing is one of them, and various noise reduction processing has been proposed (for example, Patent Documents 1 and 2).

JP 7-231444 A JP 2002-158857 A

  However, since many of the conventionally proposed noise reduction processes are methods using the spatial or temporal characteristics of pixel values, there is a problem that the image quality is degraded by the noise reduction process.

  For example, a technique using spatial characteristics is typically adaptive filtering processing or processing of adding similar partial images. However, when trying to obtain a noise reduction effect by such a method, the resolution degradation and the noise reduction effect are traded off, such as crushing fine texture information or reducing high frequency components, reducing noise. The effect is naturally limited.

  As a technique using temporal characteristics, a technique of combining a plurality of continuously shot images and improving noise reduction by an addition effect is typical. However, in this method, since a plurality of images are captured, it is essential to align the same subject image between images when the subject moves or the image is shaken due to camera shake. However, it greatly affects the quality degradation of the composite image.

  According to some aspects of the present invention, it is possible to provide an image processing device, an imaging device, an image processing method, and the like that can perform noise reduction processing that does not depend on characteristics of spatial or temporal pixel values.

  One embodiment of the present invention is picked up by an imaging device including a first color filter having a first transmittance characteristic, a second color filter having a second transmittance characteristic, and a third color filter having a third transmittance characteristic. An image acquisition unit for acquiring the obtained image, and first to fifth bands corresponding to the overlapping portion and the non-overlapping portion of the first to third transmittance characteristics, and the first to third colors constituting the image A multi-band estimation unit that estimates the first to fifth band component values based on the first pixel value, and the first to third color pixel values are reconstructed from the first to fifth band component values. The present invention relates to an image processing apparatus including a noise reduction processing unit that performs processing as processing for reducing noise of the image.

  According to one aspect of the present invention, the component values of the first to fifth bands are estimated based on the pixel values of the first to third colors, and the first to third components are calculated from the component values of the first to fifth bands. The process of reconstructing the color pixel values is performed as a process of reducing image noise. This makes it possible to perform noise reduction processing that does not depend on the characteristics of the spatial or temporal pixel value.

  In the aspect of the invention, the multiband estimation unit may overlap the first band corresponding to a non-overlapping portion of the first transmittance characteristic, and the first transmittance characteristic and the second transmittance characteristic. The second band corresponding to a portion, the third band corresponding to a non-overlapping portion of the second transmittance characteristic, and the overlapping portion of the second transmittance characteristic and the third transmittance characteristic. A fourth band and the fifth band corresponding to a non-overlapping portion of the third transmittance characteristic may be set.

  In the aspect of the invention, the noise reduction processing unit may perform the process of adding the component values of the first and second bands as a process of reconfiguring the pixel values of the first color, The process of adding the 4-band component values is performed as a process of reconstructing the pixel values of the second color, and the process of adding the component values of the fourth and fifth bands is reconstructed of the pixel values of the third color You may carry out as processing to do.

  In the aspect of the invention, the noise reduction processing unit performs a vector quantization process on the component values of the first to fifth bands, and performs the vector quantization process of the first to fifth bands. Processing for reconstructing the pixel values of the first to third colors from the component values may be performed.

  Moreover, in one aspect of the present invention, the noise reduction processing unit is configured by the pixel values of the first to third colors as the number of combinations that can be taken by the vectors configured by the component values of the first to fifth bands. A process of quantizing the number of combinations that can be taken by the vector may be performed as the vector quantization process.

  In the aspect of the invention, the multiband estimator may include the first color pixel value, which is a value obtained by adding the component values of the first and second bands, and the component values of the second to fourth bands. A value obtained by adding the component value of the first band, the component value of the second band, and the component value of the third and fourth bands based on the pixel value of the second color that is a value obtained by adding And calculating the first band component value, the second band component value, and the first addition value based on the relational expression. Also good.

  In the aspect of the invention, the multiband estimation unit obtains the relational expression by using the component value of the first band as an unknown, and obtains the component values of the first and second bands expressed by the relational expression and the first band. An error evaluation value representing an error between one added value and the pixel values of the first and second colors is obtained, the unknown number that minimizes the error evaluation value is determined, and the determined unknown number and the relational expression are determined. Based on the above, the component values of the first and second bands and the first addition value may be determined.

  In the aspect of the invention, the multiband estimator may include a pixel value of the second color, which is a value obtained by adding the component values of the second to fourth bands, and the component values of the fourth and fifth bands. Based on the pixel value of the third color that is a value obtained by adding the second and third band component values based on the pixel value of the third color, and the component values of the fourth band, A relational expression between the component values of the fifth band is obtained, and the second addition value, the component value of the fourth band, and the component value of the fifth band are estimated based on the relational expression. The third band component value may be obtained from the first addition value and the fourth band component value, or from the second addition value and the second band component value.

  In the aspect of the invention, the multiband estimation unit may determine a pixel value between the first color pixel value and the previous second color pixel value based on the transmittance characteristics of the first and second color filters. The relative gain ratio may be corrected, and the first to fifth band component values may be estimated using the corrected pixel values of the first and second colors.

  In the aspect of the invention, the multiband estimation unit may calculate a ratio between the transmittance of the second color filter in the second band and the transmittance of the first color filter in the second band. The gain ratio may be corrected by multiplying the pixel value of the first color.

  In one embodiment of the present invention, light that has passed through an optical filter that divides a pupil of an imaging optical system into a first pupil and a second pupil is imaged on the imaging element, and the first pupil is the first pupil. , One of the second bands, the third band, and one of the fourth and fifth bands are transmitted, and the second pupil is the other of the first and second bands, and the third band. The multiband estimation unit transmits the other one of the fourth and fifth bands, and the multiband estimation unit performs the first to fifth based on the images obtained by imaging the transmitted light of the first pupil and the second pupil. The component value of the band may be estimated.

  In the aspect of the invention, the first to third colors are blue, green, and red, respectively, and the multiband estimation unit sets the component values of the first to fifth bands to the first blue component, respectively. You may obtain | require as a value, a 2nd blue component value, a green component value, a 1st red component value, and a 2nd red component value.

  Another aspect of the present invention relates to an imaging apparatus including the image processing apparatus described in any one of the above and the imaging element.

  Still another aspect of the present invention relates to an imaging apparatus including the image processing apparatus described in any one of the above, the imaging element, and the imaging optical system provided with the optical filter.

  According to still another aspect of the present invention, there is provided an imaging device including a first color filter having a first transmittance characteristic, a second color filter having a second transmittance characteristic, and a third color filter having a third transmittance characteristic. The first to fifth bands corresponding to the overlapping portion and the non-overlapping portion of the first to third transmittance characteristics are set, and the first to third colors constituting the image are acquired. A process of estimating component values of the first to fifth bands based on pixel values and reconstructing the pixel values of the first to third colors from the component values of the first to fifth bands is performed on the image. The present invention relates to an image processing method performed as a process for reducing noise.

FIG. 1A to FIG. 1D are explanatory diagrams of noise reduction processing by addition processing. Explanatory drawing about expansion of a dynamic range. Explanatory drawing about the noise reduction process by a vector quantization process. Explanatory drawing about the noise reduction process by a vector quantization process. Explanatory drawing about suppression of a quantization error. Explanatory drawing about the detailed example of a vector quantization process. 2 is a configuration example of an imaging apparatus according to the present embodiment. 1 is a configuration example of an image processing apparatus according to an embodiment. Explanatory drawing about a band division | segmentation method. Explanatory drawing about the unknown number estimation process in a 1st multiband estimation process. Explanatory drawing about the unknown number estimation process in a 1st multiband estimation process. The basic structural example of the imaging device in a 2nd multiband estimation process. Explanatory drawing about the band division | segmentation method in a 2nd multiband estimation process. FIG. 14A to FIG. 14F are explanatory diagrams of gain correction processing. Explanatory drawing about the unknown number estimation process in a 2nd multiband estimation process. FIG. 16A to FIG. 16F are explanatory diagrams of gain correction processing. Explanatory drawing about the unknown number estimation process in a 2nd multiband estimation process.

  Hereinafter, preferred embodiments of the present invention will be described in detail. The present embodiment described below does not unduly limit the contents of the present invention described in the claims, and all the configurations described in the present embodiment are indispensable as means for solving the present invention. Not necessarily.

1. Outline of this Embodiment As described above, many of the conventionally proposed noise reduction processes are methods using the spatial (positional) or temporal characteristics of pixel values, and the image quality is degraded by the noise reduction process. Resulting in. There is a need for a noise reduction method that is not affected by image quality as much as possible, but no effective method has yet been proposed.

  For example, in Patent Document 1, an original image is spatially averaged to generate an average image, a difference between the original image and the average image is extracted to generate difference data, and the difference data is vector-quantized to be vector-quantized. A method is disclosed in which an encoded image is generated and an original image is combined from the vector quantization code and an average image.

  In Patent Document 1, a proposal has been made to effectively reduce noise by vector quantization different from the conventional method. In this vector quantization, a spatial pixel value pattern in the same image is quantized, so that it is basically unaffected by resolution degradation and image quality degradation due to time axis blur correction. However, if the number of representative vectors for quantization decreases, the normal random noise decreases, but the quantization noise expands, and there is a problem that the image quality cannot be maintained.

Accordingly, in the present embodiment, imaging is performed with an imaging device having RGB three primary color filters, and component values of five bands {r ⁽¹⁾ , r ⁽²⁾ , g based on RGB pixel values obtained by imaging. , B ⁽¹⁾ , b ⁽²⁾ }. Then, component values {r ⁽¹⁾ , r ⁽²⁾ } constituting the R pixel value, component values {r ⁽²⁾ , g, b ⁽¹⁾ } constituting the G pixel value, and B pixel values are constituted. By adding the component values {b ⁽¹⁾ , b ⁽²⁾ } to reconstruct RGB pixel values, random noise in the captured image is reduced.

  In this way, since addition of pixel values at different positions and addition of pixel values at different imaging timings is not performed, it is possible to perform noise reduction processing that does not depend on the spatial or temporal characteristics of the pixel values. Thereby, the random noise of the captured image can be reduced without being affected by the reduction of high-frequency components due to spatial addition and the influence of image quality deterioration due to movement of the subject or camera shake.

  In Patent Document 2, spectral information is acquired from a multiband image, the principal component of the spectral information is analyzed, the multiband image is reconstructed as a sum of principal component vectors obtained by the analysis, and the reconstruction is performed. A technique for synthesizing the multiband image and the original multiband image with predetermined weighting to obtain a multiband image with high accuracy and good S / N is disclosed.

  However, Patent Document 2 is silent about a method of improving S / N by estimating a multiband image from RGB three primary color images and reconstructing the RGB three primary color images from the multiband image.

2. Noise reduction processing 2.1. Next, the noise reduction processing of this embodiment will be described in detail. In the following, a case where noise reduction processing is performed using a band component value obtained by the second multiband estimation process described later will be described as an example. However, the present embodiment is a first multiband estimation process described later. You may apply with respect to the obtained band component. The first multiband estimation process will be described in detail later with reference to FIGS. 9 to 11, and the second multiband estimation process will be described later in detail with reference to FIGS. Hereinafter, the image sensor is also referred to as an image sensor as appropriate. Further, {F _R , F _G , F _B } used below is a function of the wavelength λ, but the wavelength λ is omitted for the sake of simplicity.

The blue component {b ^R _B , b ^L _B } in FIG. 1 (A), the green component {b ^L _G , g _G , r ^R _G } in FIG. 1 (B), and the red component {r ^{R in} FIG. 1 (C). _{R 1} , r ^L _R } are band component values obtained by the second multiband estimation process from RGB pixel values {R, G, B} obtained by imaging. As will be described later with reference to FIG. 13 and the like, in the second multiband estimation process, component values of the first to fifth bands BD1 to BD5 are estimated. {B ^R _B , b ^L _B } are component values of the first and second bands that have passed through the blue filter (spectral characteristic F _B ) of the image sensor, and are component values that constitute the blue pixel value B. {B ^L _G , g _G , r ^R _G } are component values of the second to fourth bands that have passed through the green filter (spectral characteristic F _G ) of the image sensor, and component values constituting the green pixel value G It is. {R ^R _R , r ^L _R } are the component values of the fourth and fifth bands that have passed through the red filter (spectral characteristic F _R ) of the image sensor, and are the component values that constitute the red pixel value R.

As shown in FIGS. 1 (A) to 1 (C), a blue component {b ^R _B , b ^L _B }, a green component {b ^L _G , g _G , r ^R _G }, a red component {r ^R _R , A random noise component {n _R , n _G , n _R } is superimposed on r ^L _R }. This random noise component {n _R , n _G , n _R } is superimposed when a signal is output from the image sensor, and for example, noise generated by a circuit that reads out the accumulated charge of the pixel as an analog signal, Quantum conversion noise due to variations in photoelectric conversion.

As shown in the following formula (1), individual components are added for each color, and a blue component B ′, a green component G ′, and a red component R ′ are obtained again. Since the random noise component is relatively decreased with respect to each color component due to the addition effect, a noise reduction effect is expected.
B ′ = b ^R _B + b ^L _B ,
G ′ = b ^L _G + g _G + r ^R _G ,
R ′ = r ^R _R + r ^L _R (1)

For example, as the captured pixel value R, a signal {r ^R _R + r ^L _R } obtained by optically adding a band component is obtained. Therefore, the random noise component n _R added after the photoelectric conversion is superimposed on the pixel value R with the same phase for each band component. If it constituting an image in the pixel value R, an image is formed with a random noise component n _R course. On the other hand, in this embodiment, band component values {r ^R _R , r ^L _R } are estimated and separated using {R ′, G ′} which are independent pixel values. Therefore, the noise component of the band component value {r ^R _R , r ^L _R } is an independent random variable, and the noise component n _R ′ of the band component addition value {r ^R _R + r ^L _R } is an independent random variable. since the addition value, the dispersion is smaller than the original noise component n _R.

Moreover, when the above noise reduction process is applied to the 2nd multiband estimation process mentioned later, the effect which expands a dynamic range is acquired. That is, as shown in FIG. 12, when an optical filter RLT that divides the left and right pupils into different wavelength bands is used, the amount of light received by each pixel of the imaging sensor is less than the amount of light that passes through the imaging lens LNS. Become. In other words, the optical filter FLT acts as a neutral density filter. Therefore, as shown in A1 of FIG. 2, when the optical filter FLT is not, the received light amount of the pixel when the amount of light imaging lens LNS is transmitted becomes L ₀ is Max (pixel saturation amount of light, pixels However, if the optical filter FLT is provided as shown in A2, the amount of light received by the pixel is reduced until the amount of light transmitted through the imaging lens LNS becomes L ₁ (L ₁ > L ₀ ). Max is not reached. That is, the amount of light transmitted by the imaging lens LNS when the pixel reaches the amount of light Max (saturated amount of light) that can be received can be increased from L ₀ to L ₁ . In other words, a brighter subject can be imaged as compared to the case without the optical filter FLT. When combined with the noise reduction processing of the present embodiment, the range of brightness of a subject that can be photographed is widened and the noise (S / N) of the image is reduced, resulting in the effect of expanding the dynamic range.

2.2. Noise Reduction Processing by Vector Quantization In this embodiment, not only the above addition processing but also noise reduction processing by vector quantization may be performed.

Specifically, the pixel values of the captured image before multiband estimation are B (blue), G (green), and R (red), and the number of quantization steps of these pixel values is N (N is a natural number of 2 or more). Then, each pixel value can take the range of the following formula (2). The number of quantization steps N is the number of steps for quantizing the range (the range from the minimum value to the maximum value of the pixel value) set below the saturation value of the image sensor. For example, in the case of an 8-bit image, N = 256.
0 ≦ B <N, 0 ≦ G <N, 0 ≦ R <N (2)

  Here, the number of quantization steps N for R, G, and B is generally a common value because it is easier to handle the same, but even if the number of quantization steps N is intentionally different for each color. I do not care.

Since the pixel values B and R are defined as values obtained by adding two components {b ^R _B , b ^L _B } and {r ^R _R , r ^L _R }, respectively, the components b ^R _B and b ^L _B are defined. , R ^R _R , r ^L _R are independent values, each component before vector quantization can take the range of the following equation (3).
0 ≦ b ^R _B <N, 0 ≦ b ^L _B <N, 0 ≦ r ^R _R <N, 0 ≦ r ^L _R <N (3)

Since the pixel value G is defined as a value obtained by adding the three components {b ^L _G , g _G , r ^R _G }, the components b ^L _G , g _G , and r ^R _G are independent values. Then, each component before vector quantization can take the range of the following formula (4).
0 ≦ b ^L _G <N, 0 ≦ r ^R _G <N, 0 ≦ g _G <N (4)

Component value determined by the multi-band estimation processing ^{_{{b R B, b L B}} , b L G, g G, r R G, r R R, r L R} combination patterns of (component values in each pixel), 7-dimensional vector (multi-band vector) composed of seven components Φ ⁽⁷⁾ = (b ^R _B , b ^L _B , b ^L _G , g _G , r ^R _G , r ^R _R , r ^L _R ) . If the number of quantization steps for each component value is N, the range of the above equations (3) and (4) can be taken, so the number of vectors (number of combinations ⁾ that the 7-dimensional vector Φ ⁽⁷⁾ can take is the N ^7.

Further, a combination pattern of pixel values {R ′, G ′, B ′} of the three primary colors to be finally obtained after the noise reduction processing is considered as a three-dimensional vector Φ ⁽³⁾ composed of three components. If the number of quantization steps of each component value is N (for example, 256 gradations), each component value can take the range of the following equation (5), so the three-dimensional vector Φ ⁽³⁾ = (R ′, G The number of vectors (number of combinations) that ', B') can take is N ³ .
0 ≦ B ′ <N, 0 ≦ G ′ <N, 0 ≦ R ′ <N (5)

That is, the vector quantization process of the present embodiment is a process of converting a 7-dimensional vector Φ ⁽⁷⁾ having N ⁷ patterns into a 3-dimensional vector Φ ⁽³⁾ having N ³ patterns.

In the vector quantization processing of this embodiment, first, an occurrence probability distribution of a 7-dimensional vector Φ ⁽⁷⁾ having a pattern number N ⁷ is obtained, and a representative vector Φ ⁽⁷⁾ _m having a pattern number N ³ is obtained based on the occurrence probability distribution. Are set in a 7-dimensional space. Here, m = 0, 1, 2,..., (N ³ −1). N ³ representative vectors Φ _{⁽⁷⁾ m} pattern corresponds to the three-dimensional vector of ^{N 3} pattern [Phi ^(3), the representative vector [Phi _⁽⁷⁾ pixel value by adding the component values of _m {R ', G When ', B'} is obtained, one set of pixel values {R ', G', B '} corresponds to one representative vector Φ ⁽⁷⁾ _m . Since one representative vector Φ ⁽⁷⁾ _m may be a set of pixel values {R ′, G ′, B ′} in a three-dimensional space, even if it corresponds to a plurality of vectors in the seven-dimensional space. Good.

As shown in FIG. 3, taking a vector (r ^R , r ^L ) as an example, a representative vector Φ _m is set for a noise generation range NZ of the vector (r ^R , r ^L ). More specifically, as shown in FIG. 4, representative vectors Φ _K−1 , Φ _Ka , Φ _Kb , and Φ _{K + 1} are respectively set for the noise generation ranges NZ _K−1 , NZ _Ka , NZ _Kb , and NZ _{K + 1} . Set (K is a natural number of K <N). [Phi _K-1 is a vector of the pixel value R _K-1 when adding the component values of the vector, [Phi _Ka, [Phi _Kb is a vector of the pixel value R _K When adding the component values of the vector, [Phi _{K + 1} is a vector of the pixel value R _{K + 1} When adding the component values of the vector. Two representative vector [Phi _Ka in two-dimensional _space, [Phi _Kb is because it corresponds to one pixel value R _K in one-dimensional space, the first one-dimensional space representative vector [Phi _Ka, the _Kb [Phi as above Treat as two representative vectors.

Next, the 7-dimensional vector Φ ⁽⁷⁾ of each pixel is replaced with the corresponding representative vector Φ ⁽⁷⁾ _m , and vector quantization is performed. Next, from the replaced components {b ^R _B , b ^L _B , b ^L _G , g _G , r ^R _G , r ^R _R , r ^L _R } of the replaced representative vector Φ ⁽⁷⁾ _m , the three primary colors in each pixel Pixel values {R ′, G ′, B ′} are obtained. Specifically, the blue component B ′ is obtained by B ′ = b ^R _B + b ^L _B , the green component G ′ is obtained by G ′ = b ^L _G + g _G + r ^R _G , and the red component R ′ is R ′ = r ^{Obtained by R} _R + r ^L _R For example, it is assumed that 4 × 4 vectors (r ^R , r ^L ) exist in FIG. Since the representative vectors Φ _Ka and Φ _Kb correspond to one pixel value R _K , 4 × 4 vectors (r ^R , r ^L ) are quantized into three pixel values R _K−1 , R _K , and R _{K + 1} . Will be converted. Such quantization is performed so that N ⁷ vectors Φ ⁽⁷⁾ result in N ³ vectors Φ ⁽³⁾ .

By performing vector quantization as described above, the band divided component values pixel values ^{_{{b R B, b L B}} , b L G, g G, r R G, r R R, r L R} to The superimposed noise component can be reduced. Thereby, the noise of the pixel values {R ′, G ′, B ′} of the three primary colors obtained from the component values can be reduced.

Further, by obtaining the pixel value after vector quantization of the component value, the quantization error can be reduced as compared with the case of obtaining the pixel value from the component value and then quantizing the pixel value. Specifically, as shown in FIG. 5, it is assumed that component values {r ^R , r ^L } obtained by multiband imaging are added without vector quantization to obtain a pixel value R ′. In this case, for example, in the noise generation range NZ _Ka, there are ranges NZ ₁ and NZ ₂ (ranges indicated by hatching) outside the quantization width of the pixel value R ′. Since the component values {r ^R , r ^L } belonging to the ranges NZ ₁ and NZ ₂ are component values generated by superimposing noise on the vector Φ _Ka , the added value should be R _k originally. . However, since the component values {r ^R , r ^L } belonging to the ranges NZ ₁ and NZ ₂ are included in the quantization range of the pixel value R _{K + 1} , if the added value of the component values {r ^R , r ^L } is quantized The pixel value _{RK + 1} becomes an error different from the true value.

In this regard, according to the present embodiment, since the multiband component value can be estimated once from the pixel value and vector quantization can be performed on the component value, the component values in the ranges NZ ₁ and NZ ₂ as described with reference to FIG. Can be quantized to a true value. For example, the vector (r ^R , r ^L ) in FIG. 4 is obtained by quantizing the surrounding noise components. Then, by further quantizing the vector (r ^R , r ^L ) into a representative vector, the noise generation range vectors are collected into the representative vector. Thereby, since the vector of the noise generation range is replaced with a true value, the quantization error described with reference to FIG. 5 can be suppressed.

2.3. Detailed Example of Vector Quantization Processing Various vector quantization methods can be applied to the above-described vector quantization processing. In the following, a vector quantization process with a relatively light processing load will be described as an example.

First, as shown in the following equation (6), the 7-dimensional vector Φ _ij ⁽⁷⁾ of each pixel (i, j) is averaged in the image to obtain an average vector. Here, i and j are natural numbers, i is a pixel position (coordinate) in the horizontal scanning direction, and j is a pixel position (coordinate) in the vertical scanning direction. H is the total number of pixels of the image in the horizontal scanning direction, and V is the total number of pixels of the image in the vertical scanning direction. The average vector may be obtained from all the pixels in the image, or may be obtained from appropriately selected pixels.

Next, as shown in the following equation (7), a difference vector δ _ij between the 7-dimensional vector Φ _ij ⁽⁷⁾ and the average vector of each pixel (i, j) is obtained. The components of the difference vector δ _ij corresponding to the band component values {b ^R _B , b ^L _B , b ^L _G , g _G , r ^R _G , r ^R _R , r ^L _R } are respectively represented by {δ _ij (b ^R _{B ^{), δ ij (b L B}} ), δ ij (b L G), δ ij (g G), δ ij (r R G), δ ij (r R R), and δ _ij ^(r _L ^R)} Represent. Assume that the number of quantization steps of these components is N, and that a difference vector δ _ij of N ⁷ patterns can be defined. Consider specifying a representative vector of N ³ patterns from the N ⁷ patterns.

In order to change the N ⁷ pattern to the N ³ pattern, the quantization numbers of the seven components {b ^R _B , b ^L _B , b ^L _G , g _G , r ^R _G , r ^R _R , r ^L _R } are set. It is necessary to reduce from N to n (n <N). In order to change the N ⁷ pattern composed of these seven components to the N ³ pattern, n is obtained as in the following equation (8). If n obtained by the following equation (8) is not an integer, the decimal point may be rounded down.
n ⁷ = N ³ → log ₁₀ n ⁷ = log ₁₀ N ³ → n = 10 ^{( 3} log N ^{/ 7)} (8)

Next, as shown in FIG. 6, the occurrence probability distribution is obtained using the component δ _ij (b ^R _B ) as a random variable. A region between the maximum value (Max value) and the minimum value (Min value) of the occurrence probability distribution is divided into n quantization steps. At this time, the generation probability of each region is divided so as to be the same (or almost the same) p. Then, an average value of the components δ _ij (b ^R _B ) is obtained for each of the divided areas and set as a representative value of the area. Similarly, components ^{_{{δ ij (b L B)}} , δ ij (b L G), δ ij (g G), δ ij (r R G), δ ij (r R R), δ ij (r L R )}, Region division is performed to obtain a representative value for each region. And the vector which makes the calculated _| required representative value a component is set as a representative vector of difference vector (delta) _ij .

Next, the band component value of each pixel is quantized using the representative vector. That is, a representative vector having the closest Euclidean distance is obtained from the difference vector δ _{ij of} each pixel, and the representative vector is set as a quantization vector of the difference vector δ _ij .

2.4. Imaging Device FIG. 7 shows a configuration example of an imaging device that performs multiband estimation processing and noise reduction processing of the present embodiment. The imaging device of FIG. 7 includes an imaging lens LNS, an optical filter FLT, an imaging device 10, an imaging processing unit 20, an image processing device 25 (image processing unit), and a monitor display unit 40. The image processing device 25 includes an RGB image generation unit 30, an RGB filter characteristic data recording unit 50, a multiband estimation unit 70, a data compression unit 90, and a data recording unit 100.

  The imaging element 10 images a subject imaged by the imaging lens LNS and the optical filter FLT. The optical filter FLT has different transmission bands for the right pupil and the left pupil, as will be described later with reference to FIG. The imaging processing unit 20 performs control of imaging operation by the imaging element 10, processing for A / D conversion of analog pixel signals, and the like.

  The RGB image generation unit 30 performs a demosaicing process, an image quality improvement process, and the like of the RGB Bayer image obtained by imaging, and performs RGB3 plate image (pixel values R (i, j), G (i, j), B). (I, j)) is output to the multiband estimation unit 70. i and j are natural numbers representing the position (coordinates) of the image in the horizontal scanning direction and the vertical scanning direction, respectively.

The RGB filter characteristic data recording unit 50 stores data of transmittance characteristics (spectral characteristics) F _R , F _G , and F _B of the color filter of the image sensor 10, and outputs the data to the multiband estimation unit 70. To do. However, strictly speaking, the RGB components to be imaged are determined not only by the color filter but also by the spectral sensitivity characteristics of the imaging device 10 and the spectral characteristics of the imaging lens LNS. In the above description, the transmittance characteristic of the color filter is described as being an RGB filter characteristic. However, in the present embodiment, naturally, characteristic data including spectral characteristics of the image sensor 10 and the imaging lens LNS is defined as RGB filter characteristic data. doing.

The multiband estimation unit 70 performs the multiband estimation processing of the present embodiment on the RGB3 plate image, and performs a 5-band multiband image (component values {b ^R _B , b ^L _B }, {b ^L _G , g _G , R ^R _G }, {r ^R _R , r ^L _R }, (i, j are omitted for simplicity) in FIG. The noise reduction processing unit 80 applies the noise reduction of the present embodiment to the component values {b ^R _B , b ^L _B }, {b ^L _G , g _G , r ^R _G }, {r ^R _R , r ^L _R }. Processing is performed to reconstruct an RGB3 plate image (pixel values R ′ (i, j), G ′ (i, j), B ′ (i, j)). The monitor display unit 40 displays the RGB 3 plate image obtained by the noise reduction processing on a monitor (display device).

  The data compression unit 90 performs compression processing on the RGB 3 plate image data from the RGB image generation unit 30. The data recording unit 100 records compressed RGB3 plate image data and color filter transmittance characteristic data. These recorded data can be used for multiband estimation processing in post processing after shooting. The post processing may be performed by the image processing device 25 in the imaging device, or may be performed by an image processing device configured separately from the imaging device. Note that the data compression unit 90 performs compression processing on the RGB3 plate image data from the noise reduction processing unit 80, and the data recording unit 100 records the noise reduced and compressed RGB3 plate image data. Good.

2.5. Image Processing Device FIG. 8 shows a configuration example of an image processing device configured separately from the imaging device. 8 includes a data recording unit 200, a data decompression unit 210, a decompression data storage unit 220, a monitor image generation unit 230, a monitor image display unit 240, an image data selection unit 250, a selected frame storage unit 260, a multiband. An estimation unit 270, a noise reduction processing unit 275, and an image output unit 285 are included. As this image processing apparatus, for example, an information processing apparatus such as a PC is assumed.

  The data recording unit 200 is configured by, for example, an external storage device (for example, a memory card), and stores RGB3 plate image data and transmittance characteristic data recorded by the imaging device. The data decompression unit 210 performs a process of decompressing the RGB3 plate image data compressed by the imaging apparatus. The decompressed data storage unit 220 stores decompressed RGB3 plate image data.

  The monitor image generation unit 230 generates an image for monitor display from the RGB3 plate image data. The monitor image display unit 240 displays the monitor display image on the monitor. The image data selection unit 250 outputs the ID of the frame selected by the user (operator) to the decompressed data storage unit 220 via a user interface (not shown). The decompression data storage unit 220 outputs an image of a selected frame from among a plurality of RGB3 plate images (a plurality of still images or moving images). The selected frame storage unit 260 stores an image of the selected frame.

The multiband estimator 270 includes images of selected frames (pixel values R (i, j), G (i, j), B (i, j)) and color filter transmittance characteristics F _R , F _G , F based on and _B, performs the multi-band estimation process of the present embodiment, the component values ^{_{{b R B, b L B}} }, {b L G, g G, r R G}, {r R R, r L R }) Is output. The noise reduction processing unit 275 performs the noise reduction of the present embodiment on the component values {b ^R _B , b ^L _B }, {b ^L _G , g _G , r ^R _G }, {r ^R _R , r ^L _R }. Processing is performed to reconstruct an RGB3 plate image. The image output unit 285 outputs the reconstructed RGB3 plate image to an image output device such as a display device or a printer.

According to the above embodiment, as shown in FIG. 7, the image processing apparatus includes an image acquisition unit, a multiband estimation unit, and a noise reduction processing unit. Image acquisition unit includes an imaging and a third color filter having a second color filter and the third transmission characteristic F _R having a first color filter and the second transmittance characteristic F _G having a first transmittance characteristic F _B An image captured by the element 10 is acquired. As will be described later with reference to FIG. 13 and the like, the multiband estimation unit includes the first to fifth bands BD1 corresponding to the overlapping portion and the non-overlapping portion of the first to third transmittance characteristics {F _R , F _G , F _B }. Set BD5. As will be described later with reference to FIGS. 14A to 17 and the like, the multiband estimator performs the first to fifth bands based on the pixel values {B, G, R} of the first to third colors constituting the image. component values ^{_{{b R B, b L B}} , b L G, g G, r R G, r R R, r L R} to estimate. As described in FIGS. 1A to 1D and the like, the pixel values {B ′, G ′, R ′} of the first to third colors are reconstructed from the component values of the first to fifth bands. This processing is performed as processing for reducing image noise.

  In the present embodiment, the image processing apparatus corresponds to the image processing apparatus 25 in the imaging apparatus illustrated in FIG. 7 or the image processing apparatus configured separately from the imaging apparatus illustrated in FIG. The multiband estimation unit corresponds to the multiband estimation unit 70 in FIG. 7 or the multiband estimation unit 270 in FIG. 8. The noise reduction processing unit corresponds to the noise reduction processing unit 80 in FIG. 7 or the noise reduction processing unit 275 in FIG.

  In this way, by estimating the component values of the first to fifth bands from the pixel values of the first to third colors, and reconstructing the pixel values of the first to third colors from the estimated values, the image Noise can be reduced. That is, as described with reference to FIGS. 1A to 1D and the like, noise components can be separated into bands by estimation, and noise can be reduced by the addition effect when band component values are added. As a result, noise can be reduced without performing spatial noise reduction processing (for example, processing for averaging pixel values at different positions) or temporal noise reduction processing (for example, processing for averaging pixel values having different shooting frames). Therefore, for example, it is possible to reduce noise without causing a reduction in resolution due to a spatial noise reduction process or a decrease in moving object followability due to a temporal noise reduction process.

  In addition, as described with reference to FIG. 2 and the like, when combined with the second multiband estimation process, the optical filter FLT attenuates the reflected light from the subject, so that the maximum reflected light amount at which the pixel value is saturated is Increase. As a result, the dynamic range can be expanded.

In this embodiment, as described with reference to FIG. 1D and the like, the noise reduction processing unit performs processing for adding the component values {b ^R _B , b ^L _B } of the first and second bands to the first color. The pixel value B ′ is reconstructed, and the processing of adding the component values {b ^L _G , g _G , r ^R _G } of the second to fourth bands is performed to reconstruct the pixel value G ′ of the second color The process of adding the component values {r ^R _R , r ^L _R } of the fourth and fifth bands is performed as the process of reconstructing the pixel value R ′ of the third color.

Thus, the component values of the first to fifth band ^{_{{b R B, b L B}} , b L G, g G, r R G, r R R, r L R} first to third color from Pixel values {B ′, G ′, R ′} can be reconstructed. That is, as will be described later with reference to FIG. 13 and the like, by adding component values according to the correspondence between the first to fifth bands and the transmittance characteristics of the first to third color filters, the first to third bands are added. Color pixel values can be reconstructed. In the above description, the case where the component value is simply added without weighting has been described as an example. However, the addition processing of the present embodiment is not limited to this. For example, the component value is weighted and added. Various addition processes can be applied. In other words, it is sufficient if the addition process is such that a noise component reduction effect is expected by the addition process of the component values.

In this embodiment also, as described with reference to FIGS. 3 to 6, the noise reduction processing section, first to fifth band component values _^{b R ^{B _{of, b L B, b L G}} , g G, r R _G , r ^R _R , r ^L _R } is subjected to vector quantization, and the first to fifth band component values {b ^R _B , b ^L _B , b ^L _G , g subjected to the vector quantization processing A process of reconstructing the pixel values {B ′, G ′, R ′} of the first to third colors from _{G 1} , r ^R _G , r ^R _R , r ^L _R } is performed.

More specifically, the noise reduction processing unit is a vector Φ ⁽⁷⁾ = (b ^R _B , b ^L _B , b ^L _G , g _G , r ^R _G composed of the component values of the first to fifth bands. , R ^R _R , r ^L _R ) represents the number of combinations N ⁷ (number of patterns) that can be represented by a vector Φ ⁽³⁾ = (R ′, G ′, B ′ ⁾ composed of pixel values of the first to third colors. ) the process of quantizing the number of combinations N ³ where possible, performed as a vector quantization process.

  In this way, since multiband estimation is performed once in this embodiment, vector quantization can be performed on the component values of the first to fifth bands obtained by the multiband estimation. Accordingly, the quantization error as described with reference to FIG. 5 can be suppressed, and the pixel values of the first to third colors closer to the true value can be reconfigured. be able to.

3. First multiband estimation process 3.1. Band Division Method Next, processing for estimating a 5-band multiband image from an RGB image will be described in detail. First, the first multiband estimation process, which is a basic process example, will be described. In the following description, an image sensor having an RGB Bayer array will be described as an example. However, the present embodiment is not limited to this, and it is sufficient that the image sensor has an overlapping portion in the transmittance characteristics of the color filter.

In the present embodiment, RGB Bayer images are captured, and demosaicing processing is performed on the obtained Bayer images to generate three images for each RGB. The red, green, and blue pixel values of each pixel obtained by the demosaicing process are R, G, and B, respectively, and the pixel values R, G, and B and five-band wavelength components b ⁽¹⁾ , b ⁽²⁾ , The relationship between g, r ⁽¹⁾ and r ⁽²⁾ is modeled as shown in the following equation (9).
B = b ⁽¹⁾ + b ⁽²⁾ ,
G = r ⁽¹⁾ + g + b ⁽²⁾ ,
R = r ⁽¹⁾ + r ⁽²⁾ (9)

As shown in FIG. 9, the five-band components b ⁽¹⁾ , b ⁽²⁾ , g, r ⁽¹⁾ , and r ⁽²⁾ are components determined according to the spectral characteristics of the imaging system. FIG. 9 shows the transmittance characteristics F _R , F _G , and F _B of the color filter of the imaging sensor as the spectral characteristics of the imaging system. Strictly speaking, the spectral characteristics of the imaging system are, for example, excluding color filters. It also includes the spectral characteristics of the image sensor, the spectral characteristics of the optical system, and the like. In the following, for the sake of simplicity, it is assumed that spectral characteristics of the image sensor or the like are included in the transmittance characteristics F _R , F _G , and F _B of the color filter shown in FIG.

As shown in FIG. 9, the band component corresponding to the overlapping portion of the transmittance characteristic F _B of the blue filter and the transmittance characteristic F _G of the green filter is b ⁽²⁾ , and the transmittance characteristic F _{B of the} blue filter The component of the band corresponding to the non-overlapping part is b ⁽¹⁾ . Further, components of the band corresponding to the overlap between the transmittance characteristic F _G of the transmittance characteristic F _R and the green filters of the red filter is r ^(1), the non-overlapping portion of the transmission characteristic F _R of the red filter The corresponding band component is r ⁽²⁾ . Further, components of the band corresponding to non-overlapping portions of the transmission characteristic F _G of the green filter is g. Here, the non-overlapping portion is a portion that does not overlap with the transmittance characteristics of other color filters.

5-band of BD1~BD5, the transmittance characteristic F _R, F _G, may be determined depending on the shape and degree of overlap F _B, need not be the band itself bandwidth and the overlapping portion of the transmission characteristic . For example, the band of the overlapping portion of the transmittance characteristics F _G and F _B shown in FIG. 9 is about 450 nm to 550 nm, but the band BD2 only needs to correspond to the overlapping portion, and needs to be 450 nm to 550 nm. Absent.

3.2. {R ⁽¹⁾ , r ⁽²⁾ , g + b ⁽²⁾ } Estimation Process Next, using the above equation (9), the component values {r ⁽¹⁾ , r ⁽² ) are calculated from the pixel values {R, G}. ⁾ , G + b ⁽²⁾ } is obtained as in the following equation (10). Further, the following expression (11) is established from the above expression (9).
RG = [r ⁽¹⁾ + r ⁽²⁾ ]-[r ⁽¹⁾ + g + b ⁽²⁾ ]
= R ⁽²⁾ -[g + b ⁽²⁾ ] (10)
r ⁽¹⁾ = R−r ⁽²⁾ (11)

When r ⁽²⁾ is an unknown (dominant variable), the relational expression {r ⁽¹⁾ , r ⁽²⁾ , g + b ⁽²⁾ } is expressed by the following expression (12) from the above expressions (9) to (11). Is required.
r ⁽²⁾ = unknown (dominant variable),
r ⁽¹⁾ = R−r ⁽²⁾ ,
g + b ⁽²⁾ = r ⁽²⁾ -[RG] (12)

Since {R, G} is a known value detected, if the unknown r ⁽²⁾ is determined based on the above equation (12), {r ⁽¹⁾ , r ⁽²⁾ , g + b ⁽²⁾ } are all It will be decided. That is, the likelihood pattern of {r ⁽¹⁾ , r ⁽²⁾ , g + b ⁽²⁾ } can be specified.

FIG. 10 is a diagram showing this relationship in principle. As shown in FIG. 10, as unknowns ^{r (2), {r (} 1), r (2), g + b (2)} and _{{α R R, α Gr G} } obtain the value of error is minimal.

Here, α _R and α _Gr are values satisfying the following expression (13). α _R is an average value of {r ⁽¹⁾ , r ⁽²⁾ } with respect to R, and α _Gr is an average of {r ⁽¹⁾ , g + b ⁽²⁾ } with respect to G. The value is calculated. These may be determined in consideration of the component ratio of {r ⁽¹⁾ , r ⁽²⁾ } and {r ⁽¹⁾ , g + b ⁽²⁾ } from the color filter characteristics of the image sensor as shown in FIG. .
0 <α _R ≦ 1, 0 <α _Gr ≦ 1 (13)

A value E ^rg for evaluating an error between {r ⁽¹⁾ , r ⁽²⁾ , g + b ⁽²⁾ } and {α _R R, α _Gr G} is obtained by the following equation (14). Then, r ⁽²⁾ when the evaluation value E ^rg is minimized is obtained, and the obtained r ⁽²⁾ is substituted into the above equation (12), whereby {r ⁽¹⁾ , r ⁽²⁾ , g + b ^{( 2)} Determine the three values of}.
e ^r = [α _R R−r ⁽¹⁾ ] ² + [α _R R−r ⁽²⁾ ] ² ,
_{^{e g = [α Gr Gr (}} 1)] 2 + [α Gr G- (g + b (2))] 2,
^{E rg = e r + e g} (14)

3.3. {B ⁽¹⁾ , b ⁽²⁾ , g + r ⁽¹⁾ } In the same way, using the above equation (9), the component value {b ⁽¹⁾ , b ^{( 2)} , g + r ⁽¹⁾ } is obtained as in the following equation (15). Further, the following equation (16) is established from the above equation (9).
B−G = [b ⁽¹⁾ + b ⁽²⁾ ] − [r ⁽¹⁾ + g + b ⁽²⁾ ]
= B ⁽¹⁾ -[g + r ⁽¹⁾ ] (15)
b ⁽²⁾ = B−b ⁽¹⁾ (16)

When b ⁽¹⁾ is an unknown (dominant variable), the relational expression of {b ⁽¹⁾ , b ⁽²⁾ , g + r ⁽¹⁾ } is expressed by the following formula (9), (15), (16) 17).
b ⁽¹⁾ = unknown (dominant variable)
b ⁽²⁾ = B−b ⁽¹⁾ ,
g + r ⁽¹⁾ = b ⁽¹⁾ -[BG] (17)

Since {B, G} is a detected known value, if the unknown b ⁽¹⁾ is determined based on the above equation (17), {b ⁽¹⁾ , b ⁽²⁾ , g + r ⁽¹⁾ } are all It will be decided. That is, the likelihood pattern of {b ⁽¹⁾ , b ⁽²⁾ , g + r ⁽¹⁾ } can be specified.

FIG. 11 is a diagram showing this relationship in principle. As shown in FIG. 11, as the unknown b ⁽¹⁾ , a value that minimizes the error between {b ⁽¹⁾ , b ⁽²⁾ , g + r ⁽¹⁾ } and {α _B B, α _Gb G} is obtained.

Here, α _B and α _Gb are values satisfying the following expression (18). α _B is an average value of {b ⁽¹⁾ , b ⁽²⁾ } with respect to B, and α _Gb is an average of {b ⁽²⁾ , g + r ⁽¹⁾ } with respect to G. The value is calculated. These may be determined in consideration of the component ratio of {r ⁽¹⁾ , r ⁽²⁾ } and {r ⁽¹⁾ , g + b ⁽²⁾ } from the color filter characteristics of the image sensor as shown in FIG. .
0 <α _B ≦ 1, 0 <α _Gb ≦ 1 (18)

A value E ^bg for evaluating an error between {b ⁽¹⁾ , b ⁽²⁾ , g + r ⁽¹⁾ } and {α _B B, α _Gb G} is obtained by the following equation (19). Then, b ⁽¹⁾ when the evaluation value E ^bg is minimized is obtained, and the obtained b ⁽¹⁾ is substituted into the above equation (17), whereby {r ⁽¹⁾ , r ⁽²⁾ , g + b ^{( 2)} Determine the three values of}.
e ^b = [α _B B−b ⁽¹⁾ ] ² + [α _B B−b ⁽²⁾ ] ² ,
e ^g = [α _Gb G−b ⁽²⁾ ] ² + [α _Gb G− (g + r ⁽¹⁾ )] ² ,
^{E bg = e b + e g} (19)

Next, since the following equation (20) is established from the above equation (9), r ⁽¹⁾ and b ⁽²⁾ obtained by the above equations (12) and (17) are substituted into the following equation (20), and g Is calculated.
g = G− (r ⁽¹⁾ + b ⁽²⁾ ) (20)

From the three-band pixel values {R, G, B} of each pixel obtained by the demosaicing process as described above, the five-band pixel values {b ⁽¹⁾ , b ⁽²⁾ , g, r ⁽¹⁾ , r ⁽²⁾ } can be estimated.

According to the above embodiment, the image processing apparatus uses the first to fifth band component values {b ⁽¹⁾ based on the pixel values {R, G, B} of the first to third colors constituting the image. , B ⁽²⁾ , g, r ⁽¹⁾ , r ⁽²⁾ } are included. Multiband estimation unit, a first band BD1 corresponding to the non-overlapping portions of the first transmission characteristic F _B, the corresponding overlapping portion of the first transmission characteristic F _B and the second transmission characteristic F _G 2 a band BD2, the third band BD3 which corresponds to the non-overlapping portion of the second transmission characteristic _{F G,} fourth band corresponding to the overlapping portion of the second transmission characteristic _{F G} and third transmittance characteristic _{F R} BD4 If, setting the fifth band BD5 corresponding to the non-overlapping portion of the third transmission characteristic F _R.

  In the present embodiment, the image processing apparatus corresponds to the image processing apparatus 25 in the imaging apparatus illustrated in FIG. 7 or the image processing apparatus configured separately from the imaging apparatus illustrated in FIG. The multiband estimation unit corresponds to the multiband estimation unit 70 in FIG. 7 or the multiband estimation unit 270 in FIG. 8.

  Here, the overlapping portion of the transmittance characteristics is a region where the transmittance characteristics adjacent to each other on the wavelength axis overlap when the transmittance characteristics are expressed with respect to the wavelength axis as shown in FIG. . The overlapping portion is represented by a region where the transmittance characteristics overlap or a band where the bandwidths of the transmittance characteristics overlap. Further, the non-overlapping portion of the transmittance characteristic is a portion that does not overlap with other transmittance characteristics. That is, the portion obtained by removing the overlapping portion from the transmittance characteristic. The band corresponding to the overlapping portion or the non-overlapping portion is not limited to the band itself of the overlapping portion or the non-overlapping portion, and may be any band set corresponding to the overlapping portion or the non-overlapping portion. For example, the first to fifth bands may be set by dividing a band at a wavelength at which a predetermined transmittance and transmittance characteristics intersect.

In this way, the first to fifth band component values {b ⁽¹⁾ , b ⁽²⁾ , g, r ⁽ from the first to third color pixel values {R, G, B} of the captured image. ¹⁾ , r ⁽²⁾ } can be estimated. That is, as described in the above equation (9), the pixel values (for example, B, G) having adjacent transmittance characteristics include the component value (b ⁽²⁾ ) of the overlapping portion. By deleting the component value (b ⁽²⁾ ) of this overlapping portion by the pixel value difference (B−G) as in the above equation (15), the relational expression of the component values as in the above equation (17). And the component value can be estimated based on the relational expression.

  In addition, a multiband imaging system can be realized without greatly changing the existing imaging system. That is, multiband imaging can be performed without changing the RGB imaging element used in the conventional imaging apparatus to another special imaging element (for example, an imaging element having a multiband color filter). In addition, multiband imaging can be performed without using a special additional mechanism (for example, a rotary multiband color filter or a branching optical system). Since a special image sensor and an additional mechanism are unnecessary, it is possible to avoid a decrease in resolution and an increase in cost.

In the present embodiment, as described in the above equation (9), the pixel value B of the first color is a value obtained by adding the component values b ⁽¹⁾ and b ⁽²⁾ of the first and second bands. The pixel value G of the second color is a value obtained by adding the component values b ⁽²⁾ , g, r ⁽¹⁾ of the ^{second to} fourth bands. As described in the above equation (17), the multiband estimation unit determines the component value b ⁽¹⁾ of the first band and the second band based on the pixel values B and G of the first and second colors. A relational expression between the component value b ⁽²⁾ and the first addition value r ⁽¹⁾ + g that is a value obtained by adding the component values g and r ⁽¹⁾ of the third and fourth bands is obtained. Then, based on the relational expression, the multiband estimation unit obtains the first band component value b ⁽¹⁾ , the second band component value b ⁽²⁾ , and the first addition value r ⁽¹⁾ + g. presume.

  In this way, a state in which adjacent transmittance characteristics overlap each other can be expressed as an addition formula (the above formula (9)) of band component values. Then, a relational expression between band component values can be obtained from the addition expression, and the band component value can be estimated from the relational expression.

In this embodiment, as described in the above equation (17), the multiband estimation unit obtains a relational expression using the component value b ⁽¹⁾ of the first band as an unknown. As described with reference to FIG. 11 and the above equation (19), the multiband estimator includes the component values of the first and second bands and the first added values {b ⁽¹⁾ , b ⁽²⁾ , An error evaluation value E ^bg representing an error between g + r ⁽¹⁾ } and the pixel values {B, G} of the first and second colors is obtained, and an unknown b ⁽¹ that minimizes the error evaluation value E ^{bg ) And} the component values of the first and second bands and the first added value {b ⁽¹⁾ , b ^{(2 )} based on the determined unknown b ⁽¹⁾ and the relational expression (formula (17)). ⁾ , G + r ⁽¹⁾ }.

In this way, one unknown b ⁽¹⁾ necessary for determining three variables {b ⁽¹⁾ , b ⁽²⁾ , g + r ⁽¹⁾ } from two pixel values {B, G} is obtained. can be determined on the basis of the error evaluation value ^{E bg,} band component value by the determined unknown ^{{b (1), b (} 2), g + r (1)} can be determined.

Further, in the present embodiment, in the same manner as described above, the multiband estimation unit obtains a relational expression (the above expression (12)), and based on the relational expression, component values b ^{(2) of} the second and third bands. , G, the second added value g + b ⁽²⁾ , the fourth band component value r ⁽¹⁾ , and the fifth band component value r ⁽²⁾ are estimated (formula (9) (14)). Then, from the first added value g + r ⁽¹⁾ and the component value r ^{(1) of} the fourth band, or from the second added value g + b ⁽²⁾ and the component value b ^{(2) of} the second band, the component value of the third band. g is determined.

In this way, the band component values {r ⁽¹⁾ , r ⁽²⁾ , g + b ⁽²⁾ } are represented by relational expressions, and the band component values {r ⁽¹⁾ , r ⁽²⁾ are represented using the relational expressions. , G + b ⁽²⁾ } can be estimated. Then, using the obtained band component values {b ⁽¹⁾ , b ⁽²⁾ , g + r ⁽¹⁾ } and {r ⁽¹⁾ , r ⁽²⁾ , g + b ⁽²⁾ }, the first to fifth bands Component values {b ⁽¹⁾ , b ⁽²⁾ , g, r ⁽¹⁾ , r ⁽²⁾ } can be obtained.

In the present embodiment, the first to third colors are blue (B), green (G), and red (R), respectively. The multiband estimator uses the first to fifth band component values {b ⁽¹⁾ , b ⁽²⁾ , g, r ⁽¹⁾ , r ⁽²⁾ } as the first blue component values b ⁽¹⁾ , respectively. , Second blue component value b ⁽²⁾ , green component value g, first red component value r ⁽¹⁾ , and second red component value r ⁽²⁾ .

  In this way, for example, it is possible to perform 5-band multiband imaging using the RGB imaging device mounted in an existing camera system such as a Bayer array as it is. In the present embodiment, the first to third colors are not limited to RGB, and may be any color corresponding to three color filters in which the adjacent transmittance characteristics on the wavelength axis have overlapping portions.

4). Second multiband estimation process 4.1. Basic Configuration Example Next, a second multiband estimation process for performing pupil division using an optical filter will be described. FIG. 12 shows a basic configuration example of the imaging optical system in the present embodiment. As shown in FIG. 12, the imaging optical system includes an imaging lens LNS that forms an image of a subject on the sensor surface of the imaging device, and an optical filter FLT that separates a band between the first pupil and the second pupil.

In the following description, the first pupil is the right pupil and the second pupil is the left pupil. However, the present embodiment is not limited to this. That is, the separation direction of the pupil is not limited to the left and right, and it is sufficient that the first pupil and the second pupil are separated in an arbitrary direction perpendicular to the optical axis of the imaging optical system. In the following description, the wavelength division light and the band corresponding to the five-band components {b ^R , b ^L , g, r ^R , r ^L } are appropriately set to the same code {b ^R , b ^L , g, r ^R , r ^L }.

The optical filter FLT transmits the right pupil filter FL1 (first filter in a broad sense) having a characteristic of transmitting the wavelength division light {b ^R , g, r ^R } and the wavelength division light {b ^L , g, r ^L }. And a left pupil filter FL2 (second filter in a broad sense). The optical filter FLT is provided at a pupil position (for example, a diaphragm installation position) of the imaging optical system, and the filters FL1 and FL2 correspond to the right pupil and the left pupil, respectively.

As shown in FIG. 13, the right pupil filter FL1 is a filter that transmits the bands BD1, BD3, and BD4 corresponding to the components {b ⁽¹⁾ , g, r ⁽¹⁾ } described in FIG. That is, the wavelength division light {b ^R , g, r ^R } is transmitted light corresponding to the component {b ⁽¹⁾ , g, r ⁽¹⁾ }. The left pupil filter FL2 is a filter that transmits the bands BD2, BD3, and BD5 corresponding to the components {b ⁽²⁾ , g, r ⁽²⁾ } described in FIG. That is, the wavelength division light {b ^L , g, r ^L } is transmitted light corresponding to the component {b ⁽²⁾ , g, r ⁽²⁾ }.

As shown in FIG. 12, the imaging light transmitted through the imaging lens LNS and the optical filter FLT is input to an RGB Bayer array image sensor, and the light intensity of red (R), green (G), and blue (B). Is converted into a signal and acquired as data. As shown in FIG. 13, the wavelength division light {b ^R , g, r ^R } that has passed through the right pupil and the wavelength division light {b ^L , g, r ^L } that has passed through the left pupil have distinct wavelength bands. It is separated. On the other hand, the spectral characteristics {F _R , F _G , F _B } of the color filter of the image sensor are characteristics in which the wavelength bands of adjacent spectral characteristics overlap. Considering this overlapping state, the red, green, and blue pixel values R, G, and B in each pixel after the demosaicing process can be modeled as in the following equation (21).
R = g _R + r ^R _R + r ^L _R ,
_{^{G = b L G + g G}} + r R G,
B = b ^R _B + b ^L _B + g _B (21)

Here, as shown in FIG. 14 (C), the component {b ^R _B , b ^L _B , g _B } is the wavelength-divided light {b ^R , b ^L , g} that has passed through the blue filter having the spectral characteristic F _B. Corresponding to As shown in FIG. 14D, the component {b ^L _G , g _G , r ^R _G } corresponds to the wavelength-divided light {b ^L , g, r ^R } that has passed through the green filter having the spectral characteristic F _G. . Further, as shown in FIG. 16 (D), the component _{^{{g R, r R R,}} r L R} is the spectral characteristics _F wavelength division light passed through the red filter _{^{R {g, r R, r}} L} in Correspond. The superscript suffix for each component represents whether the right pupil “R” or the left pupil “L” has passed, and the subscript suffix represents the red filter “R”, the green filter “G”, and the blue filter It indicates which of “B” passed.

4.2. {B ^R , b ^L , (g + r ^R )} Estimation Processing Next, using FIG. 14A to FIG. 17, from the pixel values {R, G, B}, the components {b ^R _B , b ^L _B , A process for estimating g _B }, {b ^L _G , g _G , r ^R _G }, and {g _R , r ^R _R , r ^L _R } will be described.

First, using the above equation (21), the wavelength band {b ^L , g} overlapping with the pixel value {B, G} is removed based on the difference between the pixel values {B, G}, and the component b ^R and A process of deriving a relational expression of the components {b ^R , b ^L , (g + r ^R )} by obtaining the relationship of the component [g + r ^R ] is performed.

It should be noted here that, as shown in FIGS. 14A to 14D, the wavelength band b ^L corresponds to the component b ^L _B of the pixel value _B and the component b of the pixel value G. it is a ^L _G, component ^b _L ^B, the ^b _{L G,} the spectral characteristics _F B, is that the relative gain of _{F G} is multiplied. Therefore, the components b ^L _B and b ^L _G are different values by the relative gain, and it is necessary to correct the components b ^L _B and b ^L _{G to} be equal.

As shown in FIGS. 14C and 14D, the pixel value G is a reference (for example, “1”), the component ratio of (b ^L _B + g _B ) is k _B1, and the component ratio of b ^L _G If k _B2 , the following equation (22) is established. k _B1 / k _B2 is, for example, the gain ratio of the spectral characteristics F _B and F _G in the band b ^{L.
_{b L B + g B = (}} k B1 / k B2) × b L G (22)

Considering the gain of the spectral characteristics F _B in the bands b ^L and g, the component g _B is considered to be sufficiently smaller than the component b ^L _B. Therefore, in order to make the components b ^L _B and b ^L _G equal, the component ( b ^L _B + g _B ) and the component b ^L _G should be substantially equal. If the value obtained by correcting the component (b ^L _B + g _B ) is (b ^L _B ′ + g _B ′), the correction shown in the following equation (23) may be performed using the above equation (22).
^{_{b L B '+ g B'}} ≒ b L G = (k B2 / k B1) × (b L B + g B) (23)

Since the component (b ^L _B + g _B ) is included in the pixel value B, in order to correct the component (b ^L _B + g _B ), the pixel value B is eventually corrected. When this corrected B is B ′, the relationship of the following equation (24) is obtained.
B ′ = (k _B2 / k _B1 ) B (24)

From the above equation (24), B 'components ^{_{{b R B', b L}} B ', g B'} is the following equation (25).
^{_{b R B '= (k B2}} / k B1) × b R B,
b ^L _B '+ g _B ' ≈b ^L _G (25)

From the above equations (21) and (25), when the pixel value B ′ and the pixel value G are expressed using components {b ^R _B ′, b ^L _G , g _G , r ^R _G }, the following equation (26) It becomes like this.
_{^{B '= b R B' +}} (b L B '+ g B') = b R B '+ b L G,
_{^{G = b L G + (g}} G + r R G) (26)

Next, as shown in the following equation (27), the difference between the corrected pixel value B ′ and the pixel value G is taken to remove the overlapping component b ^L. Further, the following expression (28) is established from the above expression (26).
_{^{B'-G = [b R B}} '+ b L G] - [b L G + g G + r R G]
= B ^R _B '-(g _G + r ^R _G ) (27)
_{^{b L G = B'-b R}} B '(28)

When b ^R _B ′ is an unknown (dominant variable), the relational expression {b ^R _B ′, b ^L _G , (g _G + r ^R _G )} is expressed by the following expression (29) from the above expressions (27) and (28). It is required as follows.
b ^R _B '= unknown number (dominant variable)
_{^{b L G = B'-b R}} B '
_{^{g G + r R G = b}} R B '- (B'-G) (29)

Since {B ′, G} is a detected known value, if the unknown b ^R _B ′ is determined based on the above equation (29), {b ^R _B ′, b ^L _G , (g _G + r ^R _G ) } Are all determined. That is, it is possible to identify _{^{{b R B ', b L}} G, (g G + r R G)} likelihood pattern.

FIG. 15 is a diagram showing this relationship in principle. As shown in FIG. 15, 'as, ^{b _{R B'} unknown _{^{b R B, b L G,}} (g G + r R G)} and {B '/ 2, G / 2} value error is minimal Ask for. That is, by _obtaining b ^R _B ′ when the error evaluation value E _BG shown in the following equation (30) is minimized, and substituting the obtained b ^R _B ′ into the above equation (29), {b ^R _B ^', _b L _^G, determines the value of _{^{(g G + r R G)}} }.
^{e B = (B '/ 2} -b R B') 2 + (B '/ 2-b L G) 2,
_{^{e G = (G / 2-}} b L G) 2 + (G / 2- (g G + r R G)) 2,
E _BG = e ^B + e ^G (30)

As described above, the component {b ^R _B ′, b ^L _G , (g _G + r ^R _G )} can be estimated from the 2-band pixel values {B ′, G} of each pixel.

4.3. {(B ^L + g), r ^R , r ^L } Estimation Process Next, components {(b ^L + g), r ^R , r ^L } are estimated from the pixel values {G, R}. Specifically, using the above equation (21), the wavelength band {g, r ^R } overlapping with the pixel value {G, R} is removed based on the difference between the pixel values {G, R}, By ^obtaining the relationship between the component [b ^L + g] and the component r ^L, a process of deriving a relational expression of the component {(b ^L + g), r ^R , r ^L } is performed.

As shown in FIG. 16A to FIG. 16D, the component r ^R _G of the pixel value _G and the component r ^R _R of the pixel value R correspond to the wavelength band r ^R , but the component r ^R _G , R ^R _R are multiplied by the relative gains of the spectral characteristics F _G , F _R. Therefore, the components r ^R _G and r ^R _R have different values corresponding to the relative gain, and it is necessary to correct the components r ^R _G and r ^R _{R to} be equal.

As shown in FIGS. 16C and 16D, the pixel value G is a reference (for example, “1”), the component ratio of (g _R + r ^R _R ) is k _R1, and the component ratio of r ^R _G If k _R2 , the following equation (31) is established. k _R1 / k _R2 is, for example, the gain ratio of the spectral characteristics F _G and F _R in the band r ^{R.
_{g R + r R R = (}} k R2 / k R1) × r R G (31)

Band g, considering the gain spectral characteristic _{F R} in ^{r R,} since the components _{g R} considered sufficiently smaller than the component ^r _{R R,} in order to equalize component ^r _R ^G, the ^r _{R R,} the components ( g _R + r ^R _R ) and the component r ^R _G may be substantially equal. If the value obtained by correcting the component (g _R + r ^R _R ) is (g _R '+ r ^R _R '), the correction shown in the following equation (32) may be performed using the above equation (31).
^{_{g R '+ r R R'}} ≒ r R G = (k R2 / k R1) × (g R + r R R) (32)

Since the component (g _R + r ^R _R ) is included in the pixel value R, in order to correct the component (g _R + r ^R _R ), the pixel value R is eventually corrected. When this corrected R is R ′, the following equation (33) is obtained.
R ′ = (k _R2 / k _R1 ) R (33)

From the above equation (33), the component {g _R ′, r ^R _R ′, r ^L _R ′} of _R ′ becomes the following equation (34).
g _R '+ r ^R _R ' ≈r ^R _G ,
^{_{r L R '= (k R2}} / k R1) × r L R (34)

From the above formulas (21) and (34), when the pixel value G and the pixel value R ′ are expressed using the components {b ^L _G , g _G , r ^R _G , r ^L _R ′}, It becomes like this.
_{^{G = b L G + (g}} G + r R G),
R ′ = (g _R ′ + r ^R _R ′) + r ^L _R ′ = r ^R _G + r ^L _R ′ (35)

Next, as shown in the following equation (36), by taking the difference of the pixel values R 'and the corrected pixel value G, eliminating duplicate components r ^R. Further, the following equation (37) is established from the above equation (35).
_{^{G-R '= [b L}} G + g G + r R G)] - [r R G + r L R']
_{^{= (B L G + g G}} ) -r L R '(36)
_{^{r R G = R'-r L}} R '(37)

When r ^L _R ′ is an unknown (dominant variable), the relational expression of {r ^L _R ′, r ^R _G , (b ^L _G + g _G )} is expressed by the following expression (38) from the above expressions (36) and (37). It is required as follows.
r ^L _R '= unknown number (dominant variable),
_{^{r R G = R'-r L}} R ',
b ^L _G + g _G = r ^L _R '+ (G−R ′) (38)

Since {G, R ′} is a detected known value, if the unknown number r ^L _R ′ is determined based on the above equation (38), {r ^L _R ′, r ^R _G , (b ^L _G + g _G ) } Are all determined. That is, it is possible to specify a likelihood pattern of {r ^L _R ′, r ^R _G , (b ^L _G + g _G )}.

FIG. 17 is a diagram showing this relationship in principle. As shown in FIG. 17, the unknown r ^L _R ′ is a value that minimizes the error between {r ^L _R ′, r ^R _G , (b ^L _G + g _G )} and {G / 2, R ′ / 2}. As an unknown r ^L _R ′. That is, by _obtaining r ^L _R ′ when the error evaluation value E _GR shown in the following equation (39) is minimized, and substituting the obtained r ^L _R ′ into the above equation (38), {r ^L _{R ^', r R} ^G, determines the value of _{(b L G + g G)} }.
_{^{e G = (G / 2- (}} b L G + g G)) 2 + (G / 2-r R G) 2,
_{^{e G = (R '/ 2}} -r R G) 2 + (R' / 2- (r L R ')) 2,
E _GR = e ^G + e ^R (39)

As described above, the component {r ^L _R ′, r ^R _G , (b ^L _G + g _G )} can be estimated from the 2-band pixel value {G, R ′} of each pixel.

  In the above equations (30) and (39), the evaluation value is calculated by multiplying the pixel values {B ′, G}, {G, R ′} by a coefficient ½. It is not limited. As described in the first multiband estimation process (for example, the above equation (18)), the coefficient may be greater than 0 and 1 or less.

Needless to say, the method of calculating the evaluation values E BG and E GR is not limited to the above method, and various methods are conceivable. For example, taking the evaluation value E BG as an example, {L, G, B} obtained from {R, G, B} obtained by capturing a large number of target subjects (for example, natural subjects or medical in-vivo subjects) in advance. From B ′, G}, the occurrence pattern of the value {b R B ′, b L G , (g G + r R G )} is obtained as described above. Then, in the probability distribution of the occurrence pattern of {b R B ′, b L G , (g G + r R G )}, a pattern showing a peak with a high occurrence probability locally or a partial probability distribution region is set Then, a pattern {b R B ′, b L G , (g G + r R G )} indicating the average pattern of the pattern group is defined as a representative pattern. In this manner, a plurality of predetermined representative patterns are set. Then, when estimating from {B, G} acquired anew (by actual shooting), {b R B ′, b L G , (g calculate the G + r R G)}, it was determined by the calculation {b R B ', b L G, and (g G + r R G) }, Euclidean distance between the preset representative pattern as the evaluation value. Even if the representative pattern having the shortest Euclidean distance among the preset representative patterns, that is, the most similar representative pattern is set as the estimated pattern, and specified as the value of {b R B ′, b L G , (g G + r R G )} Good. When the representative pattern is set in advance, the {B ′, G} representative pattern may be obtained from the {B ′, G} occurrence pattern of the target subject. Then, when estimating from {B ′, G} actually photographed, the Euclidean distance between {B ′, G} obtained by photographing and the representative pattern is obtained as an evaluation value, and {B ′ having the smallest evaluation value is obtained. , G} may be performed on the representative pattern to specify the final estimated pattern {b R B ′, b L G , (g G + r R G )}.

4.4. Calculation of the component values then the values obtained in the above ^{_{{b R B ', b L}} G, (g G + r R G)}, {r L R', r R G, (b L G + g G)} Are used to construct the value of the component {b ^R _B , b ^L _B } constituting the pixel value B, the value of the component {b ^L _G , g _G , r ^R _G } constituting the pixel value _G , and the pixel value R The value of the component {r ^R _R , r ^L _R } to be calculated is calculated.

From the above equations (25) and (34), b ^R _B and r ^L _R are obtained by the following equation (40).
_{^{b R B = (k B1 /}} k B2) × b R B ',
r ^L _R = (k _R1 / k _R2 ) × r ^L _R ′ (40)

By using g _B << b ^R _B , g _R << r ^L _R , b ^L _B , r ^R _R is obtained by the following expression (41) from the above expression (21).
b ^L _B = B− (b ^R _B + g _B ) ≈B−b ^R _B ,
^{_{r R R = R- (r L}} R + g R) ≒ R-r L R (41)

From the above equation (21), b ^L _G and r ^R _G are obtained by the following equation (42).
_{^{b L G = G- (g G}} + r R G),
_{^{r R G = G- (g G}} + b L G) (42)

From the above equations (21) and (42), g _G is obtained by the following equation (43).
^{_{g G = G- (b L G}} + r R G) (43)

4.5. Acquisition of the right pupil images and Hidarihitomi image, as shown in the phase difference detection processing 12, by imaging the light transmitted through the optical filter FLT, and the subject image I ^R that has passed through the right pupil was passed through the left pupil A subject image ^IL is obtained. In FIG. 12, I ^R and I ^L are shown as an example when the sensor surface of the image sensor is at the defocus position PD, and h is an axis along the horizontal direction (the pupil separation direction in a broad sense). . The R, G, and B components of the images I ^R and I ^L are separated as shown in the following formula (44) from the components obtained above.
I ^R = (r ^R _R , r ^R _G , b ^R _B ), I ^L = (r ^L _R , b ^L _G , b ^L _B ) (44)

In this embodiment, the images I ^R and I ^L are acquired, the phase difference (image shift amount) at the defocus position of the images I ^R and I ^L is obtained, and the subject is measured using the principle of triangulation. Distance information may be obtained to obtain distance information to the subject. For example, the phase difference AF may be performed using the acquired phase difference. As shown in the above equation (44), the images I ^R and I ^L are each composed of components in each of the RGB bands as indicated by the subscript suffixes of the component values. Therefore, even when the color of the subject is extremely biased to any of RGB, the images I ^R and I ^L are not extremely dark, and the phase difference can be detected. The distance information can be acquired for each pixel in the image, and various application processes can be performed in addition to the phase difference AF using the distance information for each pixel. Such applied processing may be performed by the image processing apparatus shown in FIG.

4.6. Multiband Image Acquisition Processing At the time of focusing when the sensor surface of the image sensor is at the focus position PF, there is no shift between the images I ^R and I ^L. That is, r ^R _R and r ^L _R , r ^R _G and b ^L _G , b ^R _B and b ^L _B , and g _G in the same pixel all represent the color light of reflected light from the same minute region of the subject. That is, from the R, G, and B values of an arbitrary pixel of the image pickup device at the time of image formation, a color image (multi-band) of five bands {R ₁ , R ₂ , G, B ₁ , B ₂ } shown in the following formula (45) Band image) can be obtained.
_{^{R 1 = r R R, R}} 2 = r L R, G = r R G + g G + b L G, B 1 = b R B, B 2 = b L B (45)

According to the above embodiment, as described in FIGS. 14A to 14F, the multiband estimation unit is based on the transmittance characteristics F _B and F _G of the first and second color filters. , The relative gain ratio k _B1 / k _B2 between the pixel value B of the first color and the pixel value G of the second color is corrected, and the corrected pixel values B ′, G of the first and second colors are corrected. Is used to estimate the component values of the first to fifth bands.

More specifically, as described in the above equations (22) to (25), the multiband estimation unit calculates the transmittance k _B2 of the second color filter (F _G ) in the second band (b ^L ). The gain ratio is corrected by multiplying the pixel value B of the first color by the ratio k _B1 / k _B2 with the transmittance k _B1 of the first color filter (F _B ) in the second band (b ^L ).

In the color filter of the image sensor, the transmittance characteristic generally differs depending on the color, and therefore the gain differs depending on the color. According to the present embodiment, since there is a band corresponding to the overlapping portion of the transmittance characteristics, the relative gain ratio k _B1 / k _B2 in the band can be obtained, and the gain ratio is used to change the color. Therefore, different gains can be corrected.

In the present embodiment, as described with reference to FIG. 12, the imaging device (sensor surface of the imaging device) divides the pupil of the imaging optical system into the first pupil (for example, the right pupil) and the second pupil (the left pupil). The light transmitted through the optical filter FLT is imaged. As described in FIG. 13, the first pupil (first pupil filter FL1) has one of the first and second bands b ^R = b ⁽¹⁾ , the third band g, the fourth band, and the fifth band. On the other hand, it transmits r ^R = r ⁽¹⁾ . The second pupil (second pupil filter FL2) includes the other b ^L = b ^{(2) of} the first and second bands, the third band g, and the other r ^L = r ^{(2) of} the fourth and fifth bands. And transparent. The multiband estimator estimates component values of the first to fifth bands based on images obtained by imaging the transmitted light of the first pupil and the second pupil.

  In this way, band component values corresponding to the first pupil and the second pupil can be obtained respectively. Thereby, various application processing (for example, phase difference AF) using pupil division becomes possible.

In addition, since RGB components are included in the first pupil and the second pupil, respectively, color misregistration hardly occurs even in a defocused region of the captured image. That is, as shown in FIG. 12, the right pupil filter FL1 transmits the RGB band {b ^R , g, r ^R }, and the left pupil filter FL2 transmits the RGB band {b ^L , g, r ^L }. As can be seen from the above equation (21), the RGB components of the image include the transmission bands of the right pupil and the left pupil. Therefore, even if the component {b ^R , g, r ^R } that has passed through the right pupil and the component {b ^L , g, r ^L } that has passed through the left pupil are deviated at the defocus position (blurred region of the image), the RGB image In this case, color misregistration does not occur (or can be suppressed to a very small level), so that it is not necessary to perform color misregistration correction, and it is possible to view the image as it is.

  Further, as will be described later with reference to FIG. 2 and the like, the optical filter FLT attenuates the reflected light from the subject, so that the maximum reflected light amount that the pixel value is saturated increases. Thereby, it becomes possible to expand a dynamic range by combining with the noise reduction process of this embodiment.

In the above-described embodiment, the case where b ⁽¹⁾ and r ⁽¹⁾ are assigned to the first pupil has been described as an example. However, the present embodiment is not limited thereto, and b ⁽¹⁾ and b ^{(2 )} May be assigned to the first pupil, or any of r ⁽¹⁾ and r ⁽²⁾ may be assigned to the first pupil.

  Although the present embodiment has been described in detail as described above, it will be easily understood by those skilled in the art that many modifications can be made without departing from the novel matters and effects of the present invention. Accordingly, all such modifications are intended to be included in the scope of the present invention. For example, a term described with a different term having a broader meaning or the same meaning at least once in the specification or the drawings can be replaced with the different term in any part of the specification or the drawings. The configurations and operations of the multiband estimation unit, noise reduction processing unit, imaging device, image processing device, and the like are not limited to those described in the present embodiment, and various modifications can be made.

10 imaging device, 20 imaging processing unit, 25 image processing device,
30 RGB image generation unit, 40 monitor display unit,
50 RGB filter characteristic data recording unit, 70 multiband estimation unit,
80 noise reduction processing unit, 90 data compression unit, 100 data recording unit,
200 data recording unit, 210 data decompression unit, 220 decompressed data storage unit,
230 monitor image generation unit, 240 monitor image display unit,
250 image data selection unit, 260 selected frame storage unit,
270 multiband estimation unit, 275 noise reduction processing unit, 285 image output unit,
BD1~BD5 first to fifth ^{_{band, E BG, E GR, E}} bg, E rg evaluation value,
_{F B,} F G, F _R first to third transmittance characteristic, FL1 right pupil filter,
FL2 Hidarihitomi filter, FLT optical filter, ^{I L} Hidarihitomi image, ^{I R} right pupil images,
LNS imaging lens, PD defocus position, PF focus position,
NZ noise ^{range, b R, b L, g} , r R, component values of ^{r L} first to fifth band,
Φ vector, Φ _K representative vector, δ _ij difference vector

Claims (15)

Image acquisition for acquiring an image captured by an imaging device including a first color filter having a first transmittance characteristic, a second color filter having a second transmittance characteristic, and a third color filter having a third transmittance characteristic And
First to fifth bands corresponding to overlapping portions and non-overlapping portions of the first to third transmittance characteristics are set, and the first to third colors are configured based on pixel values of first to third colors constituting the image. A multiband estimation unit for estimating a component value of the fifth band;
A noise reduction processing unit that performs processing for reconstructing the pixel values of the first to third colors from the component values of the first to fifth bands as processing for reducing noise of the image;
An image processing apparatus comprising: In claim 1,
The multiband estimator is
The first band corresponding to the non-overlapping portion of the first transmittance characteristic, the second band corresponding to the overlapping portion of the first transmittance characteristic and the second transmittance characteristic, and the second transmittance. The third band corresponding to the non-overlapping part of the characteristic, the fourth band corresponding to the overlapping part of the second transmittance characteristic and the third transmittance characteristic, and the non-overlapping part of the third transmittance characteristic An image processing apparatus characterized by setting the fifth band corresponding to. In claim 2,
The noise reduction processing unit
The process of adding the component values of the first and second bands is performed as a process of reconstructing the pixel values of the first color, and the process of adding the component values of the second to fourth bands is performed for the second color An image processing apparatus characterized by performing processing for reconstructing pixel values and performing processing for adding the component values of the fourth and fifth bands as processing for reconstructing the pixel values of the third color. In any one of Claims 1 thru | or 3,
The noise reduction processing unit
Vector quantization processing is performed on the component values of the first to fifth bands, and the pixel values of the first to third colors are reproduced from the component values of the first to fifth bands subjected to the vector quantization processing. An image processing apparatus that performs processing to be configured. In claim 4,
The noise reduction processing unit
The process of quantizing the number of combinations that can be taken by the vector constituted by the component values of the first to fifth bands into the number of combinations that can be taken by the vector constituted by the pixel values of the first to third colors, An image processing apparatus characterized by performing vector quantization processing. In any one of Claims 1 thru | or 5,
The multiband estimator is
The pixel value of the first color, which is a value obtained by adding the component values of the first and second bands, and the pixel value of the second color, which is a value obtained by adding the component values of the second to fourth bands. Based on the first band component value, the second band component value, and a first added value that is a value obtained by adding the third and fourth band component values,
An image processing apparatus that estimates a component value of the first band, a component value of the second band, and the first addition value based on the relational expression. In claim 6,
The multiband estimator is
Obtaining the relational expression with the component value of the first band as an unknown,
Obtaining an error evaluation value representing an error between the component values of the first and second bands and the first addition value represented by the relational expression, and the pixel values of the first and second colors;
Determining the unknown to minimize the error estimate;
An image processing apparatus that determines the component values of the first and second bands and the first addition value based on the determined unknown and the relational expression. In claim 6 or 7,
The multiband estimator is
The pixel value of the second color, which is a value obtained by adding the component values of the second to fourth bands, and the pixel value of the third color, which is a value obtained by adding the component values of the fourth and fifth bands. Based on this, a relational expression between the second addition value, which is a value obtained by adding the component values of the second and third bands, the component value of the fourth band, and the component value of the fifth band is obtained. ,
Based on the relational expression, the second addition value, the component value of the fourth band, and the component value of the fifth band are estimated,
An image processing apparatus, wherein the component value of the third band is obtained from the component value of the first addition value and the fourth band or from the component value of the second addition value and the second band. In any one of Claims 1 thru | or 8.
The multiband estimator is
Based on the transmittance characteristics of the first and second color filters, the relative gain ratio between the pixel value of the first color and the pixel value of the previous second color is corrected, and the corrected first An image processing apparatus that estimates component values of the first to fifth bands using pixel values of the second color. In claim 9,
The multiband estimator is
The gain ratio is obtained by multiplying the pixel value of the first color by the ratio of the transmittance of the second color filter in the second band and the transmittance of the first color filter in the second band. An image processing apparatus that performs correction. In any one of Claims 1 thru | or 10.
On the imaging element, light that has passed through an optical filter that divides the pupil of the imaging optical system into a first pupil and a second pupil is imaged,
The first pupil transmits one of the first and second bands, the third band, and one of the fourth and fifth bands,
The second pupil transmits the other of the first and second bands, the third band, and the other of the fourth and fifth bands,
The multiband estimator is
An image processing apparatus that estimates component values of the first to fifth bands based on the images obtained by imaging the transmitted light of the first pupil and the second pupil. In any one of Claims 1 thru | or 11,
The first to third colors are blue, green, and red, respectively.
The multiband estimator is
An image processing apparatus for obtaining component values of the first to fifth bands as a first blue component value, a second blue component value, a green component value, a first red component value, and a second red component value, respectively. . An image processing device according to any one of claims 1 to 12,
The imaging element;
An imaging apparatus comprising: An image processing apparatus according to claim 11;
The imaging element;
The imaging optical system provided with the optical filter;
An imaging apparatus comprising: Obtaining an image captured by an imaging device including a first color filter having a first transmittance characteristic, a second color filter having a second transmittance characteristic, and a third color filter having a third transmittance characteristic;
First to fifth bands corresponding to overlapping portions and non-overlapping portions of the first to third transmittance characteristics are set, and the first to third colors are configured based on pixel values of first to third colors constituting the image. Estimate the component value of the fifth band,
An image processing method, wherein the process of reconstructing the pixel values of the first to third colors from the component values of the first to fifth bands is performed as a process of reducing noise of the image.

Images