JP4834938B2 - Two-plate image capture device - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a two-plate image capturing device.
[0002]
[Prior art]
2. Description of the Related Art Conventionally, a two-plate type image capturing device having two image pickup units is known.
In this type of apparatus, first, a light image of a subject is color-separated into a first light image and a second light image using a dichroic mirror or the like. Among these, a 1st optical image is imaged by one imaging part. Further, the second light image is spatially separated by the color filter array and then picked up by the other image pickup unit.
The two-plate type image capturing device processes the image data from the two-plate imaging unit to generate color image data.
[0003]
[Problems to be solved by the invention]
By the way, such a two-plate type image capturing device has a problem that false colors are likely to be generated when processing image data of two plates.
Accordingly, an object of the present invention is to provide a two-plate image capturing device that improves such false colors.
[0004]
[Means for Solving the Problems]
In order to solve the above-described problems, the present invention is configured as follows.
[0005]
<Claim 1>
The two-plate type image capturing device according to claim 1 optically decomposes a light image formed by the photographing optical system to form a first light image composed of a first color component and a second light image composed of another color component. A color separation optical system divided into: a first image pickup unit that picks up a first light image and generates first image data; and a second light image through a micro filter mixedly arranged on the image plane, A color filter array that spatially divides into a second color component and a third color component; a second light image is captured through the color filter array, second image data corresponding to the second color component, and a third color component A second imaging unit that generates third image data corresponding to the optical image ; and optically blurring the second light image by a predetermined shift width to reduce moire generated in the second image data and the third image data. facilities arithmetic processing of the spatial frequency filter to the first image data; a low-pass filter and A filter calculation unit that brings the spatial frequency band of the first image data after the calculation process closer to the spatial frequency band of the second image data and the third image data; the second image data, the third image data, and the post-calculation process A color difference calculation unit that calculates a color difference based on the first image data and generates color difference data; an image output unit that outputs color image data based on the first image data and the color difference data, and a filter calculation unit The arithmetic processing has a spatial frequency characteristic substantially equivalent to this optical low-pass filter, and the first imaging unit and the second imaging unit are arranged with their pixel positions spatially shifted with reference to the optical image; The filter operation unit aligns the pixel positions of the first image data to the third image data by performing local product-sum operation on the first image data by using an anisotropic weighting coefficient matrix; Color difference data is generated based on the first image data to the third image data in which the pixel positions are aligned by the arithmetic unit; the image output unit phase-shifts the pixel positions of the color difference data and is output from the first imaging unit. Color image data is output after aligning the first image data and the pixel position.
[0006]
<Claim 2>
The two-plate image capturing device according to claim 2 is the two-plate image capturing device according to claim 1, wherein the optical low-pass filter has a four-point separation characteristic that shifts the light image by pixel pitch in two vertical and horizontal directions of the screen.
The filter operation unit
0 0 0
0 1 1
0 1 1
The first image data is subjected to a local product-sum operation using a 3-by-3 weighting coefficient matrix consisting of:
[0010]
DETAILED DESCRIPTION OF THE INVENTION
Embodiments according to the present invention will be described below with reference to the drawings.
[0011]
<< First Embodiment >>
FIG. 1 is a block diagram of a two-plate image capturing device 11 in the first embodiment.
In FIG. 1, a photographing lens 12 is attached to a two-plate image capturing device 11. A color separation optical system 13 such as a dichroic mirror is disposed in the image space of the photographing lens 12. The color separation optical system 13 reflects the first light image of the G component and transmits the second light image including the RB component.
[0012]
In the image forming optical path of the second light image, an image sensor 16a is arranged via an optical low-pass filter 14a and a color filter array 15a.
An imaging element 16b is disposed in the imaging optical path of the first light image via a dummy optical member 14b having the same optical thickness as that of the optical low-pass filter 14a. Instead of attaching such a dummy optical member 14b, the reflection side dimension of the color separation optical system 13 may be designed to be thicker by the optical thickness.
[0013]
The G signal (corresponding to the first image data) of the image sensor 16b is digitized via the A / D converter 17b and then input to the signal processor 18. On the other hand, the RB signal (corresponding to the second image data and the third image data) of the image sensor 16a is also digitized via the A / D converter 17a and then input to the signal processor 18.
The signal processing unit 18 performs signal processing such as black level correction, gamma correction, and white balance correction on these signals.
[0014]
The G signal processed by the signal processing unit 18 is input to a filter calculation unit 19 for color difference calculation. The G signal calculated by the filter calculation unit 19 is input to the color difference calculation unit 20.
On the other hand, the RB signal processed by the signal processing unit 18 is also input to the color difference calculation unit 20. The color difference calculation unit 20 calculates a color difference between the RB signal and the G signal in units of pixels, and generates color difference signals (RG) and (BG).
[0015]
The color difference signals (RG) and (BG) are input to the color difference interpolation unit 21 and undergo color difference interpolation processing. In the color difference interpolation here, it is preferable to perform an interpolation calculation focusing on the direction in consideration of the direction in which the color difference signal and the G signal are similar.
The color difference signals (RG) and (BG) interpolated by the color difference interpolation unit 21 are input to the image output unit 24 after passing through the color difference low-pass filter 22 as necessary.
[0016]
On the other hand, the G signal of the signal processing unit 18 is also input to the image output unit 24 after being subjected to edge enhancement processing by the edge enhancement processing unit 23 as necessary.
In the adder in the image output unit 24, the G signal and the color difference signals (RG) and (BG) are added in units of pixels to generate an R signal and a B signal. The image output unit 24 outputs the RGB color image signal thus generated to the outside.
[0017]
[Correspondence with Invention]
Here, the correspondence between the invention and this embodiment will be described. Note that the correspondence relationship here illustrates one interpretation for reference, and does not limit the present invention.
The color separation optical system described in the claims corresponds to the color separation optical system 13.
The first imaging unit recited in the claims corresponds to the imaging element 16b.
The color filter array described in the claims corresponds to the color filter array 15a.
The second imaging unit recited in the claims corresponds to the imaging element 16a.
The filter calculation unit described in the claims corresponds to the filter calculation unit 19.
The color difference calculation unit described in the claims corresponds to the color difference calculation unit 20.
The image output unit described in the claims corresponds to the image output unit 24.
The optical low-pass filter described in the claims corresponds to the optical low-pass filter 14a.
[0018]
[Features of the first embodiment]
Hereinafter, feature points of the first embodiment will be described in detail.
FIG. 2 is a diagram illustrating a state of a pixel array in the image sensors 16a and 16b. In FIG. 2, the orientation of the image plane is shown in order to make the explanation easy to understand.
Points Z and Z 'in FIG. 2 indicate the positions of corresponding image points on the imaging surface. The output image of the image sensor 16a is spatially divided into a checkered R component and B component by the color filter array 15a.
[0019]
As shown in FIG. 2, the image sensors 16 a and 16 b are arranged so that the phases of the pixels coincide with each other using the optical image on the imaging surface as a reference of the position. In the assembly adjustment process of the two-plate type image capturing device 11, the output images of the image sensors 16a and 16b are displayed on the monitor in an overlapping manner, and the positions of the image sensors 16a and 16b are adjusted so that both output images match. Realized by combining.
[0020]
FIGS. 3A to 3C are diagrams illustrating the relationship between the optical characteristics of the optical low-pass filter 14 a and the calculation processing of the filter calculation unit 19.
As shown in FIG. 3A, the optical low-pass filter 14a has a four-point separation characteristic, and is shifted by a shift width of “1 / √2 times the pixel pitch P” in each of two diagonal directions of the pixel array. Blur the light image.
[0021]
FIG. 3B shows the aperture of the RB pixel enlarged by the optical low-pass filter 14a. By replacing the aperture overlap with a weighting factor, a weighting factor matrix having a spatial frequency characteristic substantially equivalent to that of the optical low-pass filter 14a is created.
[0022]
FIG. 3C is a weighting coefficient matrix of the G signal created in this way. This weighting coefficient matrix is
0 1 0
1 4 1
0 1 0
It consists of the elements of 3 rows and 3 columns. The filter operation unit 19 performs a local product-sum operation on the G signal using the weight coefficient matrix to generate a G signal for color difference calculation. (“1/8” shown in FIG. 3 is a divisor for level adjustment after the arithmetic processing, and is executed by bit shift arithmetic or the like)
[0023]
In this way, by executing a calculation process substantially equivalent to the optical low-pass filter 14a in the filter calculation unit 19, the band difference between the G-difference calculation G signal and the RB signal is substantially eliminated, and a false signal generated by the band difference is generated. The color difference signal is reliably reduced. As a result, it is possible to improve the false color generated by the false color difference signal.
[0024]
By the way, the image output unit 24 uses a G signal that has not been band-limited (that has not undergone the calculation process of the filter calculation unit 19) when generating a final RGB signal. This G signal is a component with high visual sensitivity and greatly contributes to sharpness. Therefore, by using the G signal whose band is not limited in the image output unit 24, it is possible to output color image data with high sharpness.
Next, another embodiment will be described.
[0025]
<< Second Embodiment >>
The configuration of the two-plate image capturing device in the second embodiment is the same as that of the first embodiment (FIGS. 1 and 2). Therefore, here, the reference numerals in FIG. 1 are used as they are in the description, and a duplicate description of the configuration is omitted.
[0026]
FIGS. 4A to 4C are diagrams illustrating the relationship between the optical characteristics of the optical low-pass filter 14 a and the calculation processing of the filter calculation unit 19.
As shown in FIG. 4A, the optical low-pass filter 14a has a four-point separation characteristic and blurs the optical image with a shift width of the pixel pitch P in each of the two vertical and horizontal directions of the pixel array.
[0027]
FIG. 4B shows the aperture of the RB pixel enlarged by the optical low-pass filter 14a. By replacing the aperture overlap with a weighting factor, a weighting factor matrix having a spatial frequency characteristic substantially equivalent to that of the optical low-pass filter 14a is created.
[0028]
FIG. 4C is a weighting coefficient matrix of the G signal created in this way. This weighting coefficient matrix is
1 2 1
2 4 2
1 2 1
It consists of the elements of 3 rows and 3 columns. The filter operation unit 19 performs a local product-sum operation on the G signal using the weighting coefficient matrix to generate a G signal for color difference calculation. (“1/16” shown in FIG. 4 is a divisor for level adjustment after arithmetic processing, and is executed by bit shift arithmetic or the like)
[0029]
In this way, by executing a calculation process substantially equivalent to the optical low-pass filter 14a in the filter calculation unit 19, the band difference between the G signal for color difference calculation and the RB signal is substantially eliminated, and the false difference generated by the band difference is eliminated. The color difference signal is reliably reduced. As a result, it is possible to reliably improve false color generation due to false color difference signals.
[0030]
By the way, the image output unit 24 uses a G signal that has not been band-limited (that has not undergone the calculation process of the filter calculation unit 19) when generating a final RGB signal. This G signal is a component with high visual sensitivity and greatly contributes to sharpness. Therefore, the image output unit 24 can output color image data with high sharpness by using the G signal that is not band-limited.
Next, another embodiment will be described.
[0031]
<< Third Embodiment >>
FIG. 5 is a block diagram of a two-plate image capturing device 31 in the third embodiment. Note that in the third embodiment, the same reference numerals are shown in FIG. 5 for the same constituent elements as those in the first embodiment (FIG. 1), and a duplicate description thereof is omitted here.
[0032]
A structural feature of the third embodiment is that a color difference low-pass filter 22 is provided in the image output unit 24 as shown in FIG.
Hereinafter, specific feature points of the third embodiment will be described in detail.
FIG. 6A is a diagram illustrating a state of a pixel array in the image sensors 16a and 16b. In FIG. 6A, the orientation of the image plane is shown in order to make the explanation easy to understand. Points Z and Z ′ in FIG. 6A indicate the positions of the corresponding image points on the imaging surface.
[0033]
As shown in FIG. 6A, when the optical image on the imaging surface (for example, image points Z and Z ′) is used as a reference, the pixel positions of the imaging device 16a and the imaging device 16b are shifted by half phase in the vertical and horizontal directions. This is realized by shifting the pixel positions of the image sensors 16a and 16b while enlarging and displaying the output images of the image sensors 16a and 16b on the monitor.
6B to 6D are diagrams illustrating the relationship between the optical characteristics of the optical low-pass filter 14a and the calculation processing of the filter calculation unit 19.
[0034]
As shown in FIG. 6B, the optical low-pass filter 14a has a four-point separation characteristic and blurs the optical image with a shift width of the pixel pitch P in each of the two vertical and horizontal directions of the pixel array.
[0035]
FIG. 6C shows the aperture of the RB pixel enlarged by the optical low-pass filter 14a. By replacing the aperture overlap with a weighting factor, a weighting factor matrix having a spatial frequency characteristic substantially equivalent to that of the optical low-pass filter 14a is created.
[0036]
FIG. 6D is a weighting coefficient matrix of the G signal created in this way. This weighting coefficient matrix is
0 0 0
0 1 1
0 1 1
Is composed of anisotropic elements of 3 rows and 3 columns. The filter operation unit 19 performs a local product-sum operation on the G signal using the weighting coefficient matrix to generate a G signal for color difference calculation. (“1/4” shown in FIG. 6 is a divisor for level adjustment after the arithmetic processing, and is executed by bit shift arithmetic or the like)
[0037]
At this time, by using the anisotropic weighting coefficient matrix, the G signal for color difference calculation shifts the pixels by half phase in the vertical and horizontal directions. As a result, the G signal and the RB signal that are shifted on the imaging surface coincide with each other. Therefore, both signals can be subtracted as they are in units of pixels, and the color difference signals (RG) and (BG) can be directly calculated.
[0038]
Furthermore, the band difference between the G signal for color difference calculation and the RB signal is almost eliminated by the calculation processing of the filter calculation unit 19. In particular, an anisotropic weighting coefficient matrix is used here. For this reason, the reference range of pixels in the local product-sum operation is narrow. As a result, it is possible to prevent the mid-range component from being lost due to the arithmetic processing. Therefore, it is possible to align the band characteristics of the G signal for color difference calculation and the RB signal up to the middle band component, and it is possible to further improve false color generation.
[0039]
By the way, the color difference signals (R−G) and (B−G) generated in this way are out of phase with the G signal to be finally output. Therefore, the color difference low-pass filter 22 performs a local product-sum operation using a weighting coefficient matrix as shown in FIG. 6E on the color difference signal after interpolation processing (or the color difference signal before interpolation processing). The weighting coefficient matrix shown in FIG. 6E is composed of elements that are point-symmetric with the weighting coefficient matrix shown in FIG. By the arithmetic processing of the color difference low-pass filter 22, noise reduction and phase shift of the color difference signals (RG) and (BG) are executed at a time.
[0040]
In the adder in the image output unit 24, the color-difference signals (RG) and (B-G) after the phase shift and the G signal are added in units of pixels to generate an R signal and a B signal. The image output unit 24 outputs the RGB color image signal thus generated to the outside.
[0041]
<< Additional items of embodiment >>
In the above-described embodiment, the color separation optical system 13 using a dichroic mirror or the like is used. However, the color separation optical system of the present invention is not limited to this. For example, the first light image and the second light image may be generated using a color filter after the light image is decomposed by a half mirror or the like.
[0042]
In the above-described embodiment, the G component image signal is generated on the reflected light path side of the color separation optical system 13. In this way, by reflecting a visually sensitive component (pseudo luminance component), optical image degradation, image distortion, and imaging when passing through an optical element for color separation (such as an optical thin film) Misalignment can be avoided. As a result, it is possible to generate a good color image signal having a visually sensitive component. However, the present invention is not limited to this. For example, the color separation optical system 13 may of course transmit the G component and reflect the RB component.
[0043]
In the above-described embodiment, the GRB primary color components are assigned to the first to third color components, respectively. However, the present invention is not limited to specific color components. For example, YRB and other arbitrary color components may be assigned to the first color component to the third color component.
[0044]
In the above-described embodiment, spatial frequency filter processing by local product-sum operation is performed. However, the present invention is not limited to this local product-sum operation. In general, any arithmetic processing that manipulates a spatial frequency component may be used. For example, the image data may be converted to the spatial frequency domain by discrete cosine transform, discrete wavelet transform, or other orthogonal transform, and the filter operation calculation process may be performed directly on the spatial frequency domain. Further, the arithmetic processing is not limited to the product-sum operation. Of course, you can use median value calculation, maximum value calculation, or minimum value calculation.
[0045]
In the above-described embodiment, RGB signals are output as final color image data. However, the present invention is not limited to this. For example, color image data according to another color system may be output. Alternatively, color image data having luminance / color differences such as YCbCr may be output.
[0046]
In the above-described embodiment, the color difference data interpolation process is performed. However, the present invention is not limited to this. For example, the color difference data may be output in a state where the color difference is thinned out (for example, a state where the color difference thinning ratio is 4: 2: 2) without performing interpolation processing.
[0047]
In the third embodiment, a local product-sum operation is performed on the color difference signal to shift the phase of the color difference signal. However, the phase shift is not limited to the local product-sum operation. For example, a phase shift may be realized by reproducing a continuous waveform band-limited based on the Nyquist frequency from a discrete color difference signal and re-sampling the continuous waveform. In this case, the effect that the high frequency component of the color difference signal does not disappear can be obtained.
[0048]
In the above-described embodiment, the description is made on the assumption of a square pixel. However, the present invention is not limited to this. Of course, the present invention may be applied to rectangular pixels and diagonally arranged pixels (so-called honeycomb pixels).
[0049]
【The invention's effect】
In the present invention, when calculating the color difference, the first image data is subjected to arithmetic processing to limit the spatial frequency band. By this arithmetic processing, the band difference between the first image data and the third image data is reduced, and the false color difference signal generated due to the band difference is reduced. As a result, it is possible to improve false color generation due to false color difference signals.
[Brief description of the drawings]
FIG. 1 is a block diagram of a two-plate image capturing device 11 according to a first embodiment.
FIG. 2 is a diagram illustrating a state of a pixel array in the image sensors 16a and 16b.
FIG. 3 is a diagram illustrating a weighting coefficient matrix in the first embodiment.
FIG. 4 is a diagram illustrating a weighting coefficient matrix according to the second embodiment.
FIG. 5 is a block diagram of a two-plate image capturing device 31 in a third embodiment.
FIG. 6 is a diagram illustrating a calculation process in the third embodiment.
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 11, 31 Two-plate image capture device 12 Shooting lens 13 Color separation optical system 14a Optical low-pass filter 14b Dummy optical member 15a Color filter array 16a, 16b Image sensor 17a, 17b A / D conversion unit 18 Signal processing unit 19 Filter calculation Unit 20 color difference calculation unit 21 color difference interpolation unit 22 color difference low-pass filter 23 edge enhancement processing unit 24 image output unit

Claims (2)

A color separation optical system that optically separates a light image formed by the photographing optical system into a first light image composed of a first color component and a second light image composed of another color component;
A first imaging unit that captures the first light image and generates first image data;
A color filter array that spatially divides the second light image into a second color component and a third color component through a micro filter mixedly arranged on the image plane;
A second imaging unit that captures the second light image via the color filter array and generates second image data corresponding to the second color component and third image data corresponding to the third color component When,
An optical low-pass filter that optically blurs the second light image by a predetermined shift width to reduce moire generated in the second image data and the third image data;
A spatial frequency filter calculation process is performed on the first image data, and the spatial frequency band of the first image data after the calculation process is brought close to the spatial frequency band of the second image data and the third image data. A filter operation unit;
A color difference calculation unit that calculates a color difference based on the second image data, the third image data, and the first image data after the calculation process, and generates color difference data;
An image output unit that outputs color image data based on the first image data and the color difference data ;
The calculation processing of the filter calculation unit has a spatial frequency characteristic substantially equivalent to the optical low-pass filter,
The first imaging unit and the second imaging unit are arranged with pixel positions spatially shifted with respect to the optical image,
The filter operation unit aligns the pixel positions of the first image data to the third image data by performing a local product-sum operation with an anisotropic weight coefficient matrix on the first image data,
The color difference calculation unit generates the color difference data based on the first image data to the third image data in which pixel positions are aligned by the filter calculation unit,
The image output unit phase-shifts the pixel position of the color difference data, aligns the pixel position with the first image data output from the first imaging unit, and then outputs the color image data. A two-plate image capturing device characterized by the above. The two-plate image capturing device according to claim 1 ,
The optical low-pass filter has a four-point separation characteristic that shifts a light image by a pixel pitch in two vertical and horizontal directions of a screen,
The filter operation unit
0 0 0
0 1 1
0 1 1
A two-plate type image capturing device , wherein a local product-sum operation is performed on the first image data using a 3-by-3 weighting coefficient matrix comprising:

Images