JP2001169301A - Multi-board type image pickup device and recording medium for processing image data - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enhance the resolution of image data picked up by image pickup elements 214a and 214b. SOLUTION: Image processing sections 31a, 431b amplify object image signals picked up by image pickup elements 214a and 214b that are placed on a conjugate image forming face with a shift by 1/2 pixel pitch in longitudinal and lateral directions respectively up to a prescribed level. Analog/digital converter circuits 432a, 432b respectively convert the image signals picked up by the image pickup elements 214a and 214b into digital signals, and DSP 433a, 433b apply image processing to the digital signals independently. The two image data after the image processing are respectively interpolated so as to have all color data corresponding to each pixel position and the two image data after the interpolation processing are arranged at an interval of one in longitudinal and lateral directions respectively. The two image data after the arrangement are combined in a checkered pattern and data of blank pixels in the checkered pattern are interpolated and calculated in response to the orientation similar to that of color data that are closest to the data of the blank pixels.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a multi-plate type image pickup device for picking up the same subject image by a plurality of image pickup devices and synthesizing image data outputted from each image pickup device, and a program for synthesizing image data. The recorded recording medium.

[0002]

2. Description of the Related Art For example, as described in JP-A-10-248069 (hereinafter referred to as a first prior art),
In an imaging device that captures a subject image using an imaging device such as a CCD, two imaging devices are arranged obliquely shifted by a half pixel pitch corresponding to the pixel arrangement, and the two imaging devices have the same configuration. There is known an imaging method called a so-called spatial pixel shift method, in which the resolution of image data is improved by imaging the subject and synthesizing the image data output from each imaging element.

According to the first prior art, single-chip interpolation is performed on data from each image sensor so as to have all of R, G, and B colors corresponding to the pixel positions of the image sensor. After that, the two pieces of image data after the single-plate interpolation processing are combined. In the single-chip interpolation processing, image information from each image pickup device to which a Bayer array color filter is attached is expanded to a double-density pixel surface. For a line having a pixel output of a color of interest in each of the expanded image information, the existing pixel output is internally divided to interpolate the pixel information at a vacancy. With respect to a line sandwiched between two lines in which the pixel information is interpolated, the pixel information is interpolated by internally dividing each pixel in the two lines. In the combining process of two images, corresponding two pixels of the two pieces of image information obtained by the interpolation process are added to each other for each of the R, G, and B colors to combine the images. According to this method, since double-density linear interpolation is performed for each single board before image synthesis, it is possible to prevent generation of a false signal at an edge portion that occurs when interpolation is performed using the pixel output of another image sensor. it can. Furthermore, since the image information of the two boards is added at the time of synthesis, an effect of improving the S / N ratio can be obtained.

[0004] Also, Proceedings of the Institute of Image Electronics Engineers of Japan 98-08-04
(The second prior art) has a high-definition single image obtained by combining two images captured by a single-plate color sensor using a Bayer array color filter without performing single-plate interpolation. An image processing algorithm is disclosed. According to the second conventional technique, data is arranged in half pixels on a double-density pixel surface corresponding to pixel shift for an image photographed by two color sensors to which a Bayer array color filter is attached. After that, the image interpolation / synthesis processing is performed. The interpolation / synthesis process uses the same color component as the target pixel existing in the normal direction with respect to the pixel shift direction for the pixel position having a shooting signal of a color different from the interpolation target color. Generate components. This is performed by detecting a ridge direction of an edge line by referring to peripheral pixels at a blank pixel position where no photographic pixel exists, and generating a color component at the pixel of interest by performing linear interpolation in the detected ridge direction. According to this method, a color component is generated similarly from a photographing signal of a color different from the interpolation target color, and a color component is generated in consideration of the directionality, so that the resolution of the image after the synthesis processing can be increased. it can.

Further, Japanese Patent Application Laid-Open No. H10-155158 (hereinafter referred to as a third prior art) discloses a pixel signal output from two image sensors arranged in a checkered pattern corresponding to three primary colors. On the other hand, there is disclosed an imaging apparatus which improves the resolution more than twice without performing single-plate interpolation. According to the third conventional technique, with respect to a pixel signal photographed by two imaging elements in a Bayer array, data is arranged in half the pixels on the double-density pixel surface in accordance with the pixel shift, and then the pixel signal is Interpolation / synthesis processing is performed. In the interpolation / synthesis processing, the R and G color signals around the pixel having the B color signal are regarded as the same luminance signal, the local pixel correlation is calculated, and the pixels are adjacent in the direction in which the calculated correlation becomes maximum. A luminance signal Y is generated by averaging the two luminance signals. The high-frequency luminance signal YH is generated by removing the low-frequency component from the luminance signal Y generated for half the pixels on the double-density pixel surface, and the R image captured by the image sensor is obtained.
This is performed by adding the high-frequency luminance signal YH to a signal obtained by interpolating the entire surface of the double-density pixel surface from the color, G, and B color signals using a low-pass filter. According to this method, the high-frequency component of the luminance signal is generated using the pixel signals of the specific two primary colors among the pixel signals of the three primary colors output by the two image pickup devices, so that the image of the image after the synthesis processing is generated. Higher resolution can be achieved.

[0006]

However, the first to third prior arts described above have the following problems. That is, in the first prior art, since the existing pixel output is only linearly interpolated in the single-chip interpolation process, for example, when the signal of the interpolation target color has a peak component at the target pixel position, Even if linear interpolation is performed, a sufficient peak value cannot be obtained, so that a false color may be generated. In addition, since the direction of pixel output is not taken into account during interpolation, edges may not be sharpened.

In the second prior art, raw data captured by two color sensors is arranged on a double-density pixel surface in accordance with pixel shift, and interpolation / synthesis processing is performed on the arranged data. Done. As a result, when the respective color sensors are imaged with different gains, jaggies occur in the data after the arrangement, which may affect the detection of the ridge direction of the edge line. Further, since it is intended to interpolate and combine at double density from the beginning, versatility such as handling only data by one color sensor is not considered.

In the third prior art, the R and G colors are treated as equivalent luminance signals, and the luminance signal at the B color position is interpolated and generated from the luminance signals based on these two color signals.
When the color and the G color are not equal, a false color may occur. If the direction is detected without considering the difference between the R color and the G color, the edge may not be sharpened. Further, there is a problem that the resolution of the color signal is not considered at all, and the extraction of the high-frequency luminance signal for the purpose of increasing the resolution is performed only for half of the pixels on the double-density pixel surface. . Furthermore, when two imaging elements are imaged with different gains, jaggies occur in the data arranged on the double-density pixel surface corresponding to the pixel shift, which may affect the high-frequency luminance signal. There is.

An object of the present invention is to perform interpolation using similarity between a color to be interpolated and a different color and interpolation in consideration of the directionality of an image pickup signal as single-chip interpolation and image data synthesis after single-chip interpolation. An object of the present invention is to provide a multi-chip type imaging device which has general versatility capable of handling data for each veneer and outputs a high-definition composite image. Another object of the present invention is to perform interpolation using similarity between a color to be interpolated and a different color and interpolation taking into account the directionality of image data by single-chip interpolation and image data synthesis after single-chip interpolation. It is another object of the present invention to provide a recording medium storing a program for performing image composition by incorporating the program into a computer.

[0010]

FIG. 1 shows an embodiment of the present invention.
The present invention will be described with reference to FIG. (1) According to the first aspect of the present invention, the same subject image is captured by a plurality of color imaging devices shifted by at least one half pixel in both the vertical and horizontal directions with respect to the direction of the pixel arrangement, and
The present invention is applied to a multi-plate imaging apparatus that obtains a subject image by combining imaging data output from two color imaging apparatuses 214a and 214b. Then, for each of the imaging data output from the two color imaging devices 214a and 214b, at least one of interpolation using the directionality of the image data and interpolation using the imaging data of a plurality of color components is used. Generated by single-chip interpolation means 433a-b, 436a-b, 439 for generating all color component data corresponding to all pixels of the color imaging devices 214a, 214b, and single-chip interpolation means 433a-b, 436a-b, 439 The above-mentioned object is achieved by providing the synthesizing means 433a-b and 436a-b, 439 for synthesizing the two sets of all color component data thus obtained. (2) The invention according to claim 2 is the multi-panel imaging apparatus according to claim 1, wherein the combining means 433a-b, 436a-b, 439 perform the combining using the directionality based on the two all-color component data. It is characterized by the following. (3) The invention of claim 3 is the multi-plate image pickup device according to claim 1 or 2, wherein the two color image pickup devices 214a and 214 are provided.
When b is displaced by 1/2 pixel in both the vertical and horizontal directions with respect to the direction of the pixel arrangement, the combining means 433a to 436a
B, 439 are color component data arranged in every other row in both the vertical and horizontal directions corresponding to the pixel arrangement direction for each of the two color component data, and one color component arranged every other row Data arranging means 433a-b, 436a-, arranging the other color component data arranged every other row between the data to form a checkerboard-shaped color component data array for each color component.
b, 439, and a similarity calculating unit 439 for calculating at least two vertical and horizontal similarities in a checkerboard pattern color component data array by the data arranging units 433a-b, 436a-b, 439, and a similarity calculating unit 439. And a direction classifying unit 439 for classifying the directionality of the similarity from the similarity calculated by the above, and in a checkered pattern color component data array for each color component, arranged in the direction classified by the directionality classifying unit 439. Interpolating means 433a-b, 436a-b, and 439 for generating interpolation data of a blank pixel position for each color component using the color component data thus obtained. (4) According to a fourth aspect of the present invention, there is provided the multi-plate type imaging apparatus according to the first or second aspect, wherein the two color imaging apparatuses 214a and 214 are provided.
When b is displaced by 1/2 pixel in both the vertical and horizontal directions with respect to the direction of the pixel arrangement, the combining means 433a to 436a
B, 439 are data arranging means 433a to 436a for arranging two full-color component data so as to overlap with each other for each color component by a half pixel pitch in both the vertical and horizontal directions corresponding to the pixel arrangement direction. b, 439, and data arrangement means 433a-b, 436
With respect to the color component data superimposed for each color component according to a to b, 439, the average of the two color component data constituting the overlapping portion is obtained for all the overlapping portions surrounded by the vertical and horizontal 1/2 pixel pitch. First calculating means 439 for calculating a value
A similarity calculating unit 439 for calculating the similarity in at least two directions in the average value of the color component data calculated by the first calculating unit 439; and the similarity calculated from the similarity calculated by the similarity calculating unit 439. Direction classification means 439 for classifying the direction of the degree, and a direction classification means for the color component data average value calculated by the first calculation means 439 for each color component.
A second calculating unit 439 for calculating weighted average again by using a predetermined number of color component data arranged in the direction classified by 439 and calculating composite data of a target pixel position for each color component.
And characterized in that: (5) The invention according to claim 5 is the multi-chip image pickup apparatus according to claim 3 or 4, wherein the similarity calculating means 439 includes all colors generated by the single-chip interpolation means 433a-b, 436a-b, 439. The similarity is calculated using the component data. (6) According to a sixth aspect of the present invention, in the multi-panel imaging apparatus according to the third or fifth aspect, the similarity calculating means 439 determines the similarity to a blank pixel position and the similarity to a pixel position around the blank pixel. Is used to calculate the degree of similarity. (7) According to the invention of claim 7, in the multi-chip imaging device according to claim 3 or 4, the directional classification means 439 determines that a difference between the similarities in each direction calculated by the similarity calculating means 439 is a predetermined value. When less than, the similarity in each direction is classified as being equal. (8) According to an eighth aspect of the present invention, in the multi-plate type imaging apparatus according to the fourth aspect, the two color imaging devices 214a and 214b capture a subject image by changing the exposure amount. (9) According to the ninth aspect of the present invention, the same subject image is captured by a plurality of color image capturing apparatuses shifted by at least one half pixel in both the vertical and horizontal directions with respect to the pixel array direction.
The present invention is applied to a multi-plate imaging apparatus that obtains a subject image by combining imaging data output from two color imaging apparatuses 214a and 214b. Then, single-plate interpolation means 433a-b, 436a-b, 439 for generating all color component data corresponding to all pixels of the respective imaging data output from the two color imaging devices 214a, 214b
And combining means 433a-b, 436a-b, 439 for performing combining using the directionality of the two full-color component data generated by the single-plate interpolation means 433a-b, 436a-b, 439. The above-mentioned object is achieved. (10) According to a tenth aspect of the present invention, there is provided the multi-chip type imaging apparatus according to the ninth aspect, wherein the single-plate interpolation means 433a-b, 436a-b, 439
For each of the imaging data output from the two color imaging devices 214a and 214b, at least one of interpolation using the directionality of the image data and interpolation using the imaging data of a plurality of color components is used. Two color imaging devices
It is characterized in that all color component data is generated corresponding to all pixels 214a and 214b. (11) According to the eleventh aspect of the present invention, the same subject image is captured by a plurality of color imaging devices shifted in at least one of the vertical and horizontal directions in the pixel array direction by at least one-half pixel, and at least two color imaging devices 214a and 214b The present invention is applied to a multi-panel imaging apparatus that obtains a subject image by combining imaging data output from a camera. Then, for each of the imaging data output from the two color imaging devices 214a and 214b, at least one of interpolation using the directionality of the image data and interpolation using the imaging data of a plurality of color components is used. Single-chip interpolation means 433a-b, 43 for generating data corresponding to at least the luminance component corresponding to all the pixels of the color imaging devices 214a, 214b
6a-b, 439, and combining means 433a-b, 436a-b, 439 for combining two data corresponding to the luminance components generated by the single-plate interpolation means 433a-b, 436a-b, 439. By
Achieve the above objectives. (12) The recording medium for image data processing according to the twelfth aspect, wherein two images obtained by capturing the same subject image by a plurality of color imaging devices shifted by at least one half pixel in both the vertical and horizontal directions with respect to the pixel arrangement direction. Single-chip interpolation processing for generating all color component data corresponding to all pixels using at least one of interpolation using image data directionality and interpolation using image data of a plurality of color components for data And a program for performing a synthesizing process for synthesizing two all-color component data generated by the single-plate interpolation process,
By executing this program, the above-described object is achieved. (13) The invention according to claim 13 is the image data processing recording medium according to claim 12, wherein the plurality of color imaging devices are arranged by being shifted by 1/2 pixel in both the vertical and horizontal directions in the pixel arrangement direction. In the combining process, the color component data is arranged every other row in both the vertical and horizontal directions corresponding to the direction of the pixel arrangement of the image data for each of the two color component data, and one color arranged every other row A data array process in which the other color component data arranged every other row is arranged between component data to form a checkerboard-like color component data array for each color component, and a checkerboard-like color component data by the data array process In the array, a similarity calculation process of calculating at least two similarities in the vertical and horizontal directions, and a directionality for classifying the directionality of the similarity from the similarity calculated by the similarity calculation process Classification processing and interpolation data of blank pixel positions are generated for each color component using color component data arranged in the direction classified by the directional classification process in a checkerboard pattern color component data array for each color component. And an interpolation process. (14) The invention according to claim 14 is the image data processing recording medium according to claim 12, wherein the plurality of color imaging devices are arranged by being shifted by 1/2 pixel in both the vertical and horizontal directions with respect to the direction of the pixel arrangement. The synthesizing process is a data arranging process of arranging two full-color component data so as to overlap with each other by shifting by 1/2 pixel pitch in both the vertical and horizontal directions corresponding to the pixel arrangement direction of the image data for each color component. With respect to the color component data superimposed and arranged for each color component by the processing, all the overlapping portions surrounded by the vertical and horizontal 1/2 pixel pitch constitute this overlapping portion 2
A first calculation process of calculating an average value of two color component data, a similarity calculation process of calculating a similarity in at least two directions in the average value of the color component data calculated by the first calculation process, Directional classification processing for classifying the directionality of the similarity from the similarity calculated by the degree calculation processing, and directional classification with respect to the average value of the color component data calculated in the first calculation processing for each color component. And a second calculation process of calculating a composite data of the pixel position of interest for each color component by weighting and averaging again using a predetermined number of color component data arranged in the direction classified by the process. .

In the section of the means for solving the above-mentioned problems, which explains the configuration of the present invention, the drawings of the embodiments are used to make the present invention easy to understand. However, the present invention is not limited to this.

[0012]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS -First Embodiment- An embodiment of the present invention will be described below with reference to the drawings.
As shown in FIG. 1, the single-lens reflex digital still camera 100 according to the first embodiment includes a camera body 200, a finder device 50 detachable from the camera body 200,
The camera includes a photographing lens 1 which has a built-in lens 91 and a stop 92 and is attached to and detached from the camera body 200. The subject light L passes through the photographing lens 1 and is incident on the camera body 200. The quick return mirror 3, which is a half mirror, transmits light L1 for focus detection (auto focus: AF) and reflected light L2 for finder observation. Is separated into Passing light L
1 passes through the quick return mirror 3 in the finder observation state, is condensed by the field lens 209, is further reflected by the mirror 210, and is guided to the mirror 211 described later. On the other hand, the reflected light L2 is
To form a subject image on the finder mat 4 conjugate with the primary imaging surface 16 of the taking lens 1, and the image is enlarged by the finder optical system 207 via the pentaprism 205 and observed by the photographer 18. Is done.

When the digital still camera 100 is released, the half mirror 3 rotates to the position shown by the broken line 3 ', and the subject light L proceeds to the mirror 210 without being separated.

The above-described field lens 209 is provided behind the quick return mirror 3 (right side in FIG. 1). A mirror 210 fixed at an angle θ (for example, 45 degrees) with respect to the horizontal direction is provided behind the primary imaging surface 16 of the photographing lens 1, and the light beam of the subject light L is
Reflects toward 1. The light beam incident on the mirror 211 is reflected horizontally by the mirror 211 in the depth direction of the drawing. As shown in FIG. 2 as seen through the digital still camera 100 from the back, the light beam reflected rightward in FIG. 2 by the mirror 211 enters a reduction optical system 212 described later.

FIG. 3 shows an image forming system including a reduction optical system 212, a beam splitter 219, and image pickup devices 214a and 214b which are arranged behind the above-mentioned primary image forming surface 16 of the photographing lens 1 (in FIG. 3, upper right). It is a perspective view of an image optical system. The relay lens 2 is provided inside the reduction optical system 212.
A shutter plate 208 and an aperture plate 215 are provided adjacent to each other, sandwiched between 12a and 212b. The shutter plate 208 and the aperture plate 215 are formed in a disk shape, and the shutter plate 208 and the aperture plate 215 are rotated by step motors 408 and 415 (FIG. 2) provided at the center of rotation of the disk.

As shown in FIG. 3, the shutter plate 208 has a complete light blocking portion 208a for blocking all subject light beams, an opening portion 208b for passing all subject light beams, and a pair of opening portions arranged in the horizontal direction of the object scene. 208c and 208 having
d, and a pair of openings 208 arranged in the vertical direction of the object scene
e and 208f, a pair of openings 208g and 208h arranged in a direction oblique at 45 degrees to the horizontal direction of the object scene, and a pair of openings 208 arranged in a direction orthogonal to the oblique 45 degrees.
i and 208j are provided. Then, the opening 208
c, opening 208e, opening 208g, and opening 2
08i includes polarizing plates 308c, 308e, 308g, and 308i that transmit the P-polarized component from the subject light beam, and the opening portions 208d, 208f, 208h, and 208j transmit the S-polarized component from the subject light beam. Polarizing plates 308d, 308f, 308h, and 308j are provided, respectively. Complete shading part 2 for normal shooting
08a and the opening 208b are used.
c to 208j are used during a focus detection operation by a pupil division method described later.

On the other hand, the diaphragm plate 215 has a diaphragm opening 215.
a to 215 g are provided. With reference to the area of the opening 215a through which all the subject light beams pass, the aperture area of the aperture openings 215b to 215g is
The opening area is set so as to be halved in order to 215 g. As shown in FIG. 3, by alternately arranging openings having a large opening area and openings having a small opening area, the distance between adjacent openings is not reduced. On the exit side of the reduction optical system 212 (upper right of the relay lens 212b in FIG. 3), the polarization beam splitter 21 is disposed.
9 are provided. The polarization beam splitter 219 is
It transmits the P-polarized light component and reflects the S-polarized light component of the incident subject light beam. The P-polarized light component of the light beam transmitted through the polarization beam splitter 219 is incident on the image sensor 214a,
The S-polarized component of the light beam reflected by the polarization beam splitter 219 is incident on the image sensor 214b. Image sensor 214
a and 214b are imaged on the primary imaging surface 16
Are reduced by the action of the field lens 209 and the relay lenses 212a and 212b. Although not shown in FIG. 3, as shown in FIG.
4b and the polarization beam splitter 219, a CCD
Protective glasses 213a and 213b and optical low-pass filters 223a and 223b are provided, respectively. When a high resolution is required for the images obtained by the imaging devices 214a and 214b, the optical
Both low pass filters 223a and 223b are removed. Further, the image pickup devices 214a and 214b
Are arranged on the image planes conjugate to each other and shifted by a half pixel pitch.

FIG. 4 is a block diagram showing a circuit of the digital still camera 100 according to this embodiment. CPU4
When the control program stored in the ROM 443 is started, the control unit 39 appropriately controls each block in FIG. 4 based on an operation signal input from the operation member 407. An operation member 407 includes a main switch, a release button,
This is a mode changeover switch described later. The digital still camera 100 has two operation modes, a recording mode in which a subject image is captured and image data is recorded, and a reproduction mode in which recorded image data is read and reproduced, and these operation modes are the above-described modes. Switching is performed based on an operation signal input from a changeover switch.

The image sensor 214 already shown in FIG.
a and 214b are CCDs having the same number of pixels,
The optical image formed on each pixel is photoelectrically converted into an electric image signal. Digital signal processors (hereinafter referred to as DSPs) 433a and 433b supply horizontal drive signals to the image sensors 214a and 214b, respectively, and control image sensor drive circuits 434a and 434b to control the vertical directions of the image sensors 214a and 214b. A drive signal is supplied.

The image processing units 431a and 431b have a CPU
439, the image sensors 214a and 214
The image signal photoelectrically converted at b is sampled at a predetermined timing and amplified to a predetermined signal level. The set value of the amplification factor for amplifying the image signal is changed between the time of shooting and the time of focus detection described later. In order to shorten the exposure time at the time of focus detection, for example, at the time of focus detection, it is set to about four times that at the time of shooting. Analog / digital conversion circuit
The amplified image signals output from the image processing units 431a and 431b are input to input ports of 432a and 432b (hereinafter, referred to as A / D conversion circuits). Then, the image data converted into digital signals by the A / D conversion circuits 432a and 432b is converted into the above-described DSPs 433a and 433a.
The image data is input to the image data 33b, and image processing such as contour compensation, gamma correction, and white balance adjustment is performed on each image data in parallel.

The DSPs 433a and 433b control the data buses connected to the buffer memories 436a and 436b and the memory card 424, and temporarily store the processed image data in the buffer memories 436a and 436b. The stored image data is read out from the memories 436a and 436b, and is read from the image sensor 214a.
And the two image data according to 214b are combined as described later. After that, for example, a predetermined format process is performed for JPEG compression, and the image data after the format process is data-compressed at a predetermined ratio by the JPEG method and recorded on the memory card 424. Also, DSP4
Reference numerals 33 a and 433 b store the image data after the image processing in the frame memory 435, display the image data on the LCD 220 (FIG. 2) provided on the back of the camera body 200, and record the captured image recorded from the memory card 424. The data is read and expanded, and the captured image data after expansion is stored in the frame memory 435 and displayed on the LCD 220. Further, the DSPs 433a and 433b record the above-described image data on the memory card 424, and buffer memory 436a and 436b of the decompressed photographed image data.
It manages the timing of data input / output during recording to the server.

The buffer memory 436a stores the CCD 21
4a, and the buffer memory 436b stores image data from the CCD 214b.
It is used to reduce the difference in the speed of input / output of image data to / from the memory card 424 and the difference in processing speed between the CPU 439 and the DSPs 433a and 433b. The colorimetric element 417 measures the color temperature of the subject and its surroundings, and outputs the measurement result to the colorimetric circuit 452. The colorimetric circuit 452 performs a predetermined process on the analog signal that is the colorimetric result output from the colorimetric element 417, converts the analog signal into a digital signal, and converts the converted digital signal into a C signal.
Output to PU439. The above-described white balance adjustment is performed based on the digital output signal. The timer 445 has a built-in clock circuit and outputs time data corresponding to the current time to the CPU 439. This time data is recorded on the memory card 424 together with the image data described above.

The diaphragm driving circuit 453 is provided with the diaphragm plate 2 described above.
A step motor 415 for rotating the aperture plate 215 is driven so that a predetermined one of the fifteen openings 215a to 215g is selected. The measurement of the exposure for setting the aperture is performed based on the image signals output from the image sensors 214a and 214b. The shutter drive circuit 454 drives the step motor 408 for rotating the shutter plate 208, and controls the shutter open time by switching between the complete light blocking portion 208a provided on the shutter plate 208 and the opening 208b through which all light beams pass. I do. The actual control of the exposure time is controlled by performing an electronic shutter operation described later while the shutter plate 208 is switched to the opening 208b. At the time of focus detection, the openings 208c to 208j are switched.

The lens drive circuit 430 moves a focusing lens built in the photographic lens 1 to perform focus adjustment. The in-finder display circuit 440 causes the in-finder display element 426 to display the settings in the various operations described above. The interface 448 is provided so as to exchange predetermined data such as captured image data with a predetermined external device (not shown).

-Focus Detection-The focus detection operation of the digital still camera 100 will be described. The CPU 439 controls the aperture driving circuit 453
Is controlled to drive the stepping motor 415 and the diaphragm plate 21
Among the openings formed in 5, the opening 215b through which all light beams pass is set on the optical axis. Further, the shutter drive circuit 454 is controlled to drive the step motor 408 so that, for example, the centers of the openings 208 e and 208 f formed in a direction perpendicular to the optical axis of the shutter plate 208 are on the optical axis. set. That is, the opening 20
8e and 208f are arranged vertically side by side when the camera is held in a horizontal position.

The primary image that has entered the photographic lens 1 and formed on the primary imaging surface 16 is reflected by mirrors 210 and 211 as a subject light beam and is incident on the shutter plate 208.
The incident subject light flux is emitted as a focus detection light flux of a P-polarized light component and a focus detection light flux of an S-polarized light component. That is, due to the action of the polarizing plate 308e (FIG. 3) provided in the opening 208e and transmitting P-polarized light, the light beam transmitted through the opening 208e becomes only a P-polarized component. On the other hand, the opening 208
Due to the action of the polarizing plate 308f provided in f and transmitting S-polarized light, the light beam transmitted through the opening 208f becomes only the S-polarized component. For this reason, the focus detection light flux of the P-polarized light component transmitted through the opening 208e is transmitted without being reflected by the polarization beam splitter 209, and is incident on the image sensor 214a. The focus detection light flux of the S-polarized light component transmitted through the opening 208f is reflected without passing through the polarization beam splitter 209, and is incident on the image sensor 214b. As described above, the light beam transmitted through the opening 208e enters the image sensor 214a, and the light beam transmitted through the opening 208f enters the image sensor 214b.
The focus detection operation is performed by comparing the two object image data based on the image signal output from 4b and detecting the contrast.

In the optical system shown in FIG.
Observing the image transmitted through the apertures 208e and 208f of the E.8 corresponds to observing the object field by dividing the object field into two pupils shifted in the horizontal direction by the action of the mirrors 210 and 211. Called split. The direction of pupil division is changed by rotating the shutter plate 208. For example, the openings 208c and 208d formed in the horizontal direction with respect to the optical axis of the shutter plate 208 so as to be orthogonal to the direction in which the openings 208e and 208f are arranged.
Is set to be on the optical axis, which corresponds to observing the object scene by dividing the pupil in the vertical direction. If the object scene contains a horizontal line or the like and contrast cannot be obtained from the subject image divided horizontally in the pupil, that is, if it is determined that focus detection is difficult, change the direction of the pupil division in this way. The focus detection operation is performed. The direction of pupil division is horizontal and vertical with respect to the object field,
A direction of 45 degrees oblique to the horizontal direction by g and 208h and a direction of -45 degrees oblique to the horizontal direction by the openings 208i and 208j can be selected, and these oblique directions are selected according to the subject. You.

-Single-chip interpolation of image data- The imaging elements 214a and 214b are constituted by CCDs on which color filters of the same Bayer arrangement are on-chip. By the operation of the polarization beam splitter 219, a mirror image of an image incident on the image sensor 214a is projected on the image sensor 214b. FIG. 3 shows how “P” is imaged on each of the imaging elements 214a and 214b when “P” is imaged on the primary image plane 16. Therefore, the image data captured by the image sensor 214b is rearranged in the reverse order of the pixels after the image is read, and is arranged in the pixel data of the image data captured by the image sensor 214a. FIG. 5A shows the arrangement of the pixels of the imaging element 214a, and FIG. 5B shows the arrangement of the pixels of the imaging element 214b.

In the single-chip interpolation, the color information at all the pixel positions is R, R, and B for each image data by the image pickup devices 214a and 214b configured as described above.
The interpolation calculation is performed so as to cover all of the G and B colors. That is, in FIG. 5A, the pixel data indicated by B112 or the like is only the data of B color and has no data of G and R colors, and similarly, the pixel data indicated by R121 and the like is only the data of R color and is G and B data. There is no color data. Therefore, FIGS. 5A and 5B
In all the pixel positions of the image data as shown in (1), the missing color data is calculated by the interpolation processing. The procedure of single-plate interpolation is similarity calculation, similar direction classification,
G color interpolation for interpolating G color data using similar directions,
This is performed by R color interpolation and B color interpolation in which data of R and B colors are interpolated using data after G color interpolation.

FIG. 6 is a table showing the pixel positions of the image data shown in FIGS. 5 (a) and 5 (b). The similarity calculation calculates the pixel arrangement of the pixel array around the coordinate [i, j] with respect to the pixel position where there is no G color data, for example, the position of the R color data indicated by the coordinate [i, j] in FIG. This is performed to check whether there is similarity between the data in the vertical direction and the data in the horizontal direction. The vertical similarity Cv0 [i, j] with respect to the coordinates [i, j] is given by the following equation (1).
, And the horizontal similarity Ch0 with respect to the coordinates [i, j]
[I, j] is calculated by the following equation (2).

(Equation 1) Here, G [i, j-1] is the output value of the G color data indicated by the coordinates [i, j-1], and Z [i, j] is the R color data indicated by the coordinates [i, j]. Alternatively, it is the output value of the B color data.

The absolute value of the first term in the above equations (1) and (2) is compared between the G color components to detect a rough direction. The absolute values of the second and third terms of the above equations (1) and (2) are
A fine similarity that cannot be detected in the first term is detected. That is, when the detection is performed between the G color and the R color component and between the G color and the B color component that are finer than the pixel pitch between the G color components, the similarity of the high frequency components can be detected. By adding the first term and the second and third terms, the advantages of both are utilized simultaneously. The vertical similarity and the horizontal similarity obtained by the above equations (1) and (2) are calculated for each coordinate, and the vertical similarity and the horizontal similarity at the target coordinate [i, j] are calculated. And the similarity in the vertical and horizontal directions around the target coordinate [i, j] are added, for example, according to a weighting table as shown in FIG. The vertical similarity Cv [i, j] is calculated by the following equation (3), and the horizontal similarity Ch [i, j] is calculated by the following equation (4). FIG.
Is used for the purpose of judging the continuity of the data at the target coordinate [i, j] and its surroundings.

(Equation 2)

The vertical similarity Cv calculated by the above equations (3) and (4)
The direction of similarity is classified based on [i, j] and the horizontal similarity Ch [i, j]. For example, taking the case where the same data is arranged in the vertical direction as an example, the vertical similarity calculated by the above equation (1) is 0.
As a result, the vertical similarity calculated by the above equation (3) also becomes zero. Therefore, the smaller the calculated similarity, the stronger the similarity of the data in that direction. FIG.
Is a table for classifying similar directions from the vertical similarity Cv [i, j] and the horizontal similarity Ch [i, j]. When the vertical similarity Cv [i, j] is equal to or smaller than the horizontal similarity Ch [i, j], it is determined that the similarity is strong in the vertical direction (case 1), and the horizontal similarity Ch [i, j] is the vertical similarity. When Cv [i, j] or less, it is determined that the similarity is strong in the horizontal direction (case 2).
In order to avoid erroneous determination due to noise included in the output of the pixel data, the vertical similarity Cv [i, j] and the horizontal similarity Ch [i, j]
If the absolute value of the difference is equal to or smaller than the predetermined value T, it is determined that the image is similar in both the vertical and horizontal directions (case 3). When the calculated value of the similarity is standardized in the range of 0 to 255, the predetermined value T is set to about 0 to 5 at the normal exposure level.

G color interpolation is performed using the classified similar directions. The G color interpolation is performed, for example, by using the coordinates [i, j] in FIG.
When it is determined that the position of the R color data indicated by is similar in the vertical direction, it is calculated by the following equation (5), and when it is determined that the similarity is in the horizontal direction, it is calculated by the following equation (6).

(Equation 3) Referring to FIG. 9 for explaining the G color interpolation,
The average value of the first term in (5) represents the average value 91 calculated at the coordinates [i, j] from the output values of the G color data at the coordinates [i, j-1] and [i, j + 1]. . The second in the above equation (5)
The term is R at coordinates [i, j] and coordinates [i, j-2] and [i, j + 2].
It represents the amount of change calculated at the coordinates [i, j] from the output value of the color data. Although only the average value 91 is obtained by averaging the output values of the G color data, the G color interpolation value 92 is obtained by adding the variation calculated from the R color data output value.
Is obtained. That is, as shown in FIG. 9, even when there is a peak of the output value at the target coordinate [i, j] where the G color data is to be interpolated, the change amount calculated assuming that the output difference between the G color data and the R color data is constant is constant. G that has a peak by being added
A color interpolation value 92 can be obtained. Such interpolation is called color difference interpolation, and the above equation (6) performs similar color difference interpolation in the horizontal direction.

If the similar direction is classified into both the vertical and horizontal directions, the G color interpolation values are calculated by the above equations (5) and (6), and the average of the two calculated G color interpolation values is calculated. The color interpolation value is used.

Using the G color data for all pixels obtained by the G color interpolation as described above, the R and B color data are interpolated. The R color interpolation is performed, for example, using the coordinates [i-
The positions where there is no R color data represented by [1, j-1], [i-1, j], [i, j-1] are calculated by the following equations (7) to (9), respectively.

(Equation 4) The average value of the first term in the above equations (7) to (9) represents the average value calculated from the output values of the R color data adjacent to the coordinates to be subjected to the R color interpolation. The coordinates in which the second term in (9) is the object of the R color interpolation and the G adjacent to this coordinate
It represents the amount of change calculated from the output value of the color data. That is, as in the case of the color difference interpolation performed in the G color interpolation, the average value of the output values of the R color data is added with the change amount calculated assuming that the output difference between the G color and the R color data is constant. Even when there is a peak of the output value at the target coordinates for interpolating the R color data, an R color interpolation value having a peak can be obtained.

Similarly, for the B color interpolation, for example, FIG.
For positions where there is no B color data represented by coordinates [i, j], [i-1, j], [i, j-1], the following equations (10) to (12)
Is calculated by Since the sample frequencies of the R color and the B color are lower than the sample frequency of the G color, the high frequency component of the G color data is reflected using the color difference RG and the color difference BG.

(Equation 5) If the single-chip interpolation described above is performed on image data obtained by the image sensor 214a and the image sensor 214b,
Two sets of image data having data for three colors of G, R, and B corresponding to all pixel positions are obtained.

-Synthesis of image data after single-chip interpolation- Two sets of image data after single-chip interpolation are synthesized as one image data. The image data obtained by the image sensor 214a and the image sensor 214b after the single-chip interpolation is subjected to a predetermined γ-conversion in order to match human visual characteristics, and a high-luminance-side gradation is compressed from the linear gradation data. Converted to data. In the present embodiment, image data is synthesized by performing tone conversion on image data obtained by the image sensor 214a and the image sensor 214b after single-chip interpolation, and then synthesizing them into one image. Image sensor 2 after interpolation
It is possible to cope with any case where the image data by the image pickup device 14a and the image sensor 214b is combined into one image data and then the gradation conversion is performed.

The synthesis procedure is performed by arranging data, calculating similarity, classifying similar directions, and performing interpolation using similar directions. FIGS. 10A and 10B show the image pickup device 214a.
FIG. 11 is a diagram showing pixels and single-chip interpolation of image data according to FIGS. The data arrangement is shown in FIG.
The image data of FIG. 10B is arranged vertically and horizontally every other row (FIGS. 10C and 10D), and the image according to FIG. The data strings are combined in a checkered pattern on the double-density pixel surface so as to be arranged (FIG. 10E). In the first embodiment, FIG.
As shown in (e), the data of the blank portion of the image data arranged in a checkered pattern is obtained by interpolation.

The similarity calculation is performed by using coordinates [i, j] in FIG. 11 representing a position where no data exists in the data arranged in a checkered pattern in FIG. For the indicated position, the coordinates [i, j]
Is performed to check whether there is similarity between the data in the vertical direction and the horizontal direction in the data array.
The vertical similarity Dv0 [i, j] for the coordinates [i, j] is
The horizontal similarity Dh0 [i, j] with respect to the coordinate [i, j] is calculated by the following equation (14). Note that FIG.
Indicates that the data of each color of R, G, and B exists on the memory at the coordinate position by the above-described single-chip interpolation.

(Equation 6)

The above equations (13) and (14) are for detecting the directionality by comparing adjacent same-color components using the data of each color. The vertical similarity and the horizontal similarity obtained by the above equations (13) and (14) are calculated for each coordinate,
The vertical similarity and the horizontal similarity at the target coordinate [i, j] and the vertical similarity and the horizontal similarity around the target coordinate [i, j] are shown in FIG. The final vertical similarity Dv [i, j] is calculated by the following equation (15), and the horizontal similarity Dh [i, j] is calculated by the following equation (16). Is calculated. As described above, the continuity between the target pixel and its surrounding pixels can be determined from the weighting table shown in FIG.

(Equation 7)

Vertical similarity calculated by the above equations (15) and (16)
The similarity direction is classified based on Dv [i, j] and the horizontal similarity Dh [i, j]. As in the case of single-plate interpolation, the smaller the calculated similarity, the stronger the similarity of the data in that direction. FIG. 12 shows the vertical similarity Dv [i, j] and the horizontal similarity Dh.
It is a table | surface which classifies a similar direction from [i, j]. Vertical similarity Dv
When [i, j] is equal to or less than the horizontal similarity Dh [i, j], it is determined that the similarity is strong in the vertical direction (case 1), and the horizontal similarity Dh [i, j] is the vertical similarity Dv [i, j]. j] In the following cases, it is determined that the similarity is strong in the horizontal direction (case 2). The vertical similarity Dv [i, j] and the horizontal similarity are used to avoid the influence of noise at the time of imaging included in the output of the pixel data and the misjudgment due to the shift of the fine gain adjustment between the two image sensors 214a and 214b. When the absolute value of the difference between Dh [i, j] is equal to or smaller than a predetermined value T2, it is determined that the two directions are similar in both the vertical and horizontal directions (case 3). The predetermined value T2 is such that the calculated value of the similarity is 0 to 2
When standardized to the range of 55, the normal exposure level is set to about 0 to 5.

Using the classified similar directions, the above-mentioned blank data of the image data is interpolated for each color. For example, when it is determined that the position indicated by the coordinates [i, j] in FIG. 11 is similar in the vertical direction, the following expressions (17) to (19) are obtained.
, And when it is determined to be similar in the horizontal direction, the following equation (2)
0) to (22).

(Equation 8)

When the similar directions are classified into the vertical and horizontal directions, the interpolation values of each color are calculated by the above equations (17) to (19) and (20) to (22), and the calculated values are calculated for each color. The average of the two interpolated values is taken as the interpolated value. As described above, the pixel sequence of the image data by the image sensor 214a (FIG. 10A) and the pixel sequence of the image data by the image sensor 214b (FIG. 10B) are arranged in a checkered pattern, By interpolating the blank part of
Pixels of the original image data and double-density image data having a resolution twice as high both vertically and horizontally are synthesized.

The operation of the digital still camera 100 thus configured in the recording mode will be described. FIG.
14 is a flowchart showing a program started by the half-press switch. The CPU 439 receives a half-press operation signal and a full-press operation signal in conjunction with a release button (not shown), which is one of the operation members 407. It is determined that the half-press operation signal has been input (step S1).
Y), and the amplification factor for amplifying the image signal in the image processing units 431a and 431b is changed to a predetermined value at the time of focus detection. Subsequently, an opening 215a on the aperture plate 215 through which all light beams pass is set on the optical path (step S2), and openings 208e and 208f formed in a direction perpendicular to the optical axis of the shutter plate 208 are placed on the optical path. (Step S3). On the other hand, if it is not determined in step S1 that the half-press operation signal has been input (N in step S1), the process waits for the input of the half-press operation signal.

In step S4, image data based on the image signals picked up by the image sensor 214a and the image sensor 214b is read, and it is determined whether or not the image data has a contrast (step S5). When it is determined that there is contrast (Y in step S5), the process proceeds to step S15, and it is determined that there is no contrast (step S5).
N), the shutter plate 208 is rotated, and the openings 208c and 208d formed in the horizontal direction with respect to the optical axis are set so as to be on the optical path (step S6). In step S7, the image data based on the image signals captured by the image sensor 214a and the image sensor 214b is read again, and it is determined whether or not there is a contrast in the focus detection area in the image data (step S8). ). When it is determined that there is contrast (Y in step S8), step S1
5, when it is determined that there is no contrast (N in step S8), the shutter plate 208 is rotated again, and the openings 208g and 20 formed obliquely with respect to the optical axis.
8h is set on the optical path (step S
9).

In step S10, the image sensor 214 is again activated.
The image data based on the image signal a and the image signal captured by the image sensor 214b is read, and it is determined whether or not the image data has a contrast in the focus detection area (step S11). When it is determined that there is a contrast (Y in step S11), the process proceeds to step S15, and when it is determined that there is no contrast (N in step S11), the shutter plate 208 is rotated again, and the openings 208g and 2
08h and openings 208i and 2
08j is set on the optical path (step S
12). In step S13, the image sensor 21
4a and image data based on the image signal imaged by the image sensor 214b are read, and it is determined whether or not there is contrast in the focus detection area in the image data.
(Step S14). Determined to have contrast
(Y in step S14) and the process proceeds to step S15, where it is determined that there is no contrast (N in step S14),
A warning indicating that focusing is impossible is displayed on the display element 426 in the viewfinder via the display circuit 440 in the viewfinder (step S).
17), and return to step S1.

In step S15, the two image data output from the image sensor 214a and the image sensor 214b are compared, and focus detection is performed by pupil division. Then, based on the obtained focus detection data, the focusing lens of the photographing lens 1 is driven to the in-focus position by the lens driving circuit 430 (step S16). When it is determined that the full-press operation signal has been input (Y in step S18), the quick return mirror 3 is flipped up, and step S1 is performed.
An imaging sequence subsequent to 9 is executed. On the other hand, when it is determined that the full-press operation signal is not input (step S1).
8) returns to step S1.

In step S19, the shutter plate 208
Is set on the optical path, and the exposure time and the aperture value are calculated based on the outputs from the image sensor 214a and the image sensor 214b so as to obtain an appropriate exposure (step S20). ). And
The amplification factor for amplifying the image signal in the image processing units 431a and 431b is changed from the set value at the time of focus detection to a predetermined set value at the time of shooting. Based on the value calculated in step S20, one of the openings 215a to 215h on the diaphragm plate 215 is set on the optical path (step S21), and the charges accumulated in the imaging elements 214a and 214b are discharged. That is, a so-called electronic shutter operation is performed (step S22). In step S23, the image sensor 2
14a and 214b are exposed for a predetermined time under the same conditions. After the exposure is completed, the complete light shielding portion 208a on the shutter plate 208 is set on the optical path in step S24,
Image signals are read from 4a and 214b (step S25). The read image signal is DSP433
The predetermined image processing is performed in steps a and 433b, and the above-described single-chip interpolation and combination processing are performed on the image data on which the image processing has been performed (step S26). The synthesized image data is compressed in a predetermined format, and the compressed image data is recorded on the memory card 424 (step S27), and the processing in FIGS.

In the first embodiment described above, the following operation and effect can be obtained. (1) A plurality of color data, for example, RGB during single-chip interpolation
(Cy, Mg, Ye, G, etc. in the case of a complementary color filter), examine the similarity of the target image data to the surrounding data, and perform G color interpolation in consideration of the directionality of the similarity. Since the R and B color data are interpolated from the obtained G color data by color difference interpolation, the two image pickup devices 21
It is possible to obtain high-resolution single-chip interpolated image data having all color data for each of the image data 4a and 214b. Therefore, even when the captured image has a frequency component close to the Nyquist frequency of the image sensor, the image is recovered by G color interpolation, and the R color,
The B color is also restored to a high resolution. Further, since the single-chip interpolation image data has all the color data, versatility such as displaying the monitor using the single-chip interpolation image data alone for a simple monitor display can be enhanced. (2) Since the image pickup devices 214a and 214b use CCDs having the same Bayer array and are arranged on the conjugate image plane with a shift of 1/2 pixel pitch in the vertical and horizontal directions, the image data from both image pickup devices is combined to obtain 1 The image data can be converted into double-density image data having twice the number of pixels in the vertical and horizontal directions of the screen as compared with the number of pixels of one image sensor. further,
Two sets of image data having all the color data obtained by interpolating the image data obtained by the respective image sensors on a single-chip basis are used, and the image data is synthesized in consideration of the similar directionality. Therefore, corresponding to all the pixel positions at which the density is doubled, the data of each color having one cycle of four pixels on the double-density screen is divided by 1 /.
Since correct color data is increased at positions shifted by four periods, false color is reduced, and high-quality image data with improved resolution of all color data is obtained.

(3) The image processing units 431a and 431b corresponding to the image pickup devices 214a and 214b,
Since the conversion circuits 432a and 432b, the DSPs 433a and 433b, and the buffer memories 436a and 436b are provided independently, it is possible to simultaneously perform single-chip interpolation of image data by the imaging devices 214a and 214b in parallel. The effect of shortening the processing time can be obtained as compared with performing the single-plate interpolation processing sequentially.

In the above description, the similarity Dv0 [i, j] in the vertical direction and the similarity Dh in the horizontal direction are calculated using the above equations (13) and (14).
0 [i, j] is calculated, but the luminance Y at the pixel position where the R, G, and B color data exists is calculated by the following equation (23).
[I, j] is calculated, and using this luminance, the following equations (24) and (2)
The vertical similarity Dv0 [i, j] and the horizontal similarity Dh0 [i, j] may be calculated according to 5).

(Equation 9)

Second Embodiment In the second embodiment, each of the following synthesis procedures of image data after single-plate interpolation differs from the synthesis procedure according to the first embodiment. The synthesizing procedure according to this embodiment is performed by arranging data, calculating similarity, classifying similar directions, and performing weighted averaging using similar directions for two sets of image data after single-plate interpolation. In the second embodiment, the image sensors 214a and 214b are exposed under different exposure conditions.

-Synthesis of Image Data after Single-Chip Interpolation- In the second embodiment, as in the first embodiment, the image data from the image pickup device 214a and the image pickup device 214b after single-plate interpolation are respectively applied. A case where the image is first combined with a single image after performing the gradation conversion,
It is possible to cope with any case where the image data by the image pickup device 14a and the image sensor 214b is combined into one image data and then the gradation conversion is performed.

FIGS. 15A and 15B are diagrams showing pixels after single-chip interpolation of image data by the image pickup devices 214a and 214b. The arrangement of the data is such that the image data of FIG. 15B is superimposed on the pixel data of FIG. As shown in FIG. 15C, each overlapping portion surrounded by the overlapped 1/2 pixel pitch is represented by an average of data constituting the overlapping portion. For example, taking the upper left portion of the overlap surrounded by the half pixel pitch as an example, since it is composed of pixel data 111 and pixel data 211, (111 + 211) / 2
It expresses. The overlapping portion surrounded by the half pixel pitch corresponds to one pixel on the double density pixel surface.

FIG. 16 is a diagram showing the positions of the arrangement data shown in FIG. 15C arranged based on the average value of the two sets of image data. The similarity calculation is performed, for example, by using coordinates [i,
j] is similar to the data arranged in four directions of the data array in the vertical direction, the horizontal direction, the 45 ° diagonal direction, and the 135 ° diagonal direction with the coordinate [i, j] as the center. This is done to see if there is any potential. Coordinates [i,
j], the vertical similarity Dv [i, j] is given by the following equation (26).
The horizontal similarity Dh [i, j] for coordinates [i, j] is
The similarity D45 [i, j] of the coordinate [i, j] in the diagonal 45-degree direction is calculated by (27), and the similarity D135 of the coordinate [i, j] in the diagonal 135-degree direction is calculated by the following equation (28). [I, j] is given by the following equation (2)
It is calculated in 9). In FIG. 16, the description “RGB” indicates that the data of each color of R, G, and B exists in the memory at the coordinate position by the above-described single-chip interpolation and the data arrangement processing.

(Equation 10)

The above equations (26) to (29) are used to detect the directionality by comparing adjacent same color components using the data of each color. The similarity in each direction obtained by the above equations (26) to (29) is calculated for each coordinate, and the similarity direction is classified based on the calculated similarity in each direction. As in the case of single-plate interpolation, the smaller the calculated similarity, the stronger the similarity of the data in that direction. Therefore, the minimum similarity Dm among the similarities in each direction calculated for each coordinate is Dm.
in [i, j] is detected. In order to avoid an erroneous determination due to noise included in the output of the pixel data, the maximum similarity Dmax [i,
j] to detect whether the difference between Dmax and Dmin is greater than a predetermined value T3. FIG. 17 is a table for classifying similar directions. When the difference is larger than the predetermined value T3 and the minimum similarity Dmin [i, j] is the vertical similarity Dv [i, j], it is determined that the similarity is strong in the vertical direction (case 1). When the difference is larger than the predetermined value T3 and the minimum similarity Dmin [i, j] is the horizontal similarity Dh [i, j], it is determined that the similarity is strong in the horizontal direction (case 2).

Further, the difference between Dmax [i, j] and Dmin [i, j] is larger than a predetermined value T3, and the minimum similarity Dmin [i, j] is a diagonal 45-degree similarity D45 [i, j]. , It is determined that the similarity is strong in the 45-degree diagonal direction (case 3), the difference is larger than the predetermined value T3, and the minimum similarity Dmin [i, j] is 1
When the degree of similarity is 35 degrees D135 [i, j], it is determined that the similarity is strong in the diagonal 135 degrees direction (case 4). Dmax [i, j] and Dm
When the difference from in [i, j] is equal to or smaller than the predetermined value T3, it is determined that the directions are similar in all directions (case 5). When the calculated value of the similarity is standardized in the range of 0 to 255, the predetermined value T3 is set to about 0 to 5 at the normal exposure level.

Using the classified similar direction, the target pixel data is re-synthesized by weighted average with the image data adjacent in the similar direction. For example, for a position indicated by coordinates [i, j] in FIG.
When it is determined to be similar in the vertical direction, it is calculated by the following equations (30) to (32).
(35), when it is determined to be similar in the diagonal direction of 45 degrees, it is calculated by the following formulas (36) to (38). ).

[Equation 11]

If a similar direction is classified as all directions,
The weighted average value of each color is calculated by the above equations (30) to (32) and (33) to (35), and the calculated two weighted average values are reaveraged for each color to re-synthesize pixel data. Value. Note that the re-synthesis loop processing according to the above equations (30) to (41) includes a method of overwriting the R, G, and B data of each pixel each time the processing of each pixel data ends, and a method of overwriting until the loop processing ends. There are two methods of storing the result in another buffer memory without using it, but either method may be used.

As described above, since the image synthesizing process is as follows, double-density image data having twice the vertical and horizontal resolution with respect to the pixels of the image data before the synthesis are synthesized. The pixel sequence of the image data by the image sensor 214a (FIG. 15A) and the pixel sequence of the image data by the image sensor 214b (FIG. 15B) are shifted with a pitch of 1/2 pixel vertically and horizontally. For each overlapping portion surrounded by a 1/2 pixel pitch, an average of data constituting the overlapping portion is calculated. The average value of the calculated data is recombined using the directionality of the similarity.

-Wide Dynamic Range- As described above, in the second embodiment, so-called wide dynamic range photographing is performed in which an image is taken while changing the exposure conditions of the image pickup devices 214a and 214b. FIG. 18 is a diagram illustrating a relationship between an input value and an output value of the imaging device. A curve A is a curve showing a relationship between an input value and an output value of the imaging device under standard exposure conditions, and the output value is a value after performing an A / D conversion of the output of the imaging device and performing gamma correction. Under a standard exposure condition, when a subject having a uniform reflectance of 18% (called a standard reflector) is photographed,
The standard exposure time t0 is determined so that the output value satisfies the following equations (42) and (43).

(Equation 12) According to the above equation, when a subject whose input value to the imaging device is 1024 is photographed, the output value from the imaging device is recorded as 256.

The curve B indicates that the exposure time t1 of the image sensor is t0 /
4 is a curve showing a relationship between an input value and an output value of the imaging device when the number is 2. Since the exposure amount is halved compared to the curve A,
When a subject having an input value of 2048, which is twice as long as the exposure time t0, is photographed, the output value from the imaging device becomes 256
Is recorded as Curve C represents the exposure time of the image sensor at t.
6 is a curve showing the relationship between the input value and the output value of the imaging device when the value is 2. The value of the exposure time t2 is determined so as to satisfy the following equation (44).

(Equation 13) The exposure time t2 = 1.67 · t0 is calculated by the above equation. That is, when a standard reflector having an input value of 128 is photographed, the gamma is obtained from the image pickup device (curve B) with the exposure time t1 = t0 / 2 and the image pickup device (curve C) with the exposure time t2 = 1.67 · t0. The exposure time t2 is calculated so that the average of the two output values output after the correction becomes equal to the output value (curve A) of the imaging device under the standard exposure condition. Since the exposure amount of the curve C is 1.67 times as large as that of the curve A, the output value is recorded as 256 when a subject having an input value 614 of 1 / 1.67 times the exposure time t0 is photographed. You.

In the second embodiment, the image pickup device 214a
Is the exposure time of t1, and the exposure time of the image sensor 214b is t2.
Exposure. Then, by performing the synthesis according to the above-described synthesis procedure, the image sensor 214 a
(Curve B) and the output value of the image sensor 214b (Curve C) are combined. As shown in the curve D, when the input value of the imaging device is low such that the input value of the imaging device is 200 or less, the output value is equivalent to the output value of the curve A exposed under the standard exposure condition, and the input value of the imaging device is Is higher than 200, the output value becomes lower than the output value of the curve A exposed under the standard exposure condition, and the dynamic range is widened.

In the range where the output value of the image data by the image sensor 214b of the curve C does not exceed 255, the resolution is doubled in the vertical and horizontal directions with respect to the pixels of the image data before the synthesis by combining as described above. Also, the image sensor 2
Even after the output value of the image data by 14b reaches the saturation level of 256, the output value of the image data by the image sensor 214a is 256 or less and does not reach the saturation level and outputs a value corresponding to the input value of the image sensor 214a. Therefore, the curve D obtained by synthesizing the output values of the image data from the image sensors 214a and 214b is a continuous value until the input value becomes 2048. As described above with reference to FIG. 15C, when the output data from the two image sensors 214a and 214b are overlapped, the overlap is determined for each overlapped portion surrounded by the overlapped 1/2 pixel pitch. Since the data from both image sensors constituting the portion are averaged, the result of the combination according to this embodiment is the average value of the output values output from the two image sensors.

The exposure / image processing including the above-described exposure time determination will be described with reference to the flowchart of FIG. The process shown in FIG. 19 is started by a half-press switch.
In the flowchart of the program shown in FIG. 7, the processing is performed after the photographing lens 1 is driven to the in-focus position by the lens driving circuit 430 in step S16. In FIG. 19, it is determined that the full-press operation signal has been input (step S10).
1), the quick return mirror 3 of FIG. 1 is flipped up, and a shooting sequence subsequent to step S102 is executed. On the other hand, when it is determined that the full-press operation signal is not input (N in step S101), the input of the full-press operation signal is waited.

In step S102, the opening 208b of the shutter plate 208 shown in FIG. 2 through which all light beams pass is set on the optical path, and the image pickup device 214a and the image pickup device 214
The brightness of the subject is measured based on the output from b. In step S103, the reference exposure time t0 and the aperture value are calculated based on the measured subject luminance so that an appropriate exposure is obtained. The exposure time t1 of the image sensor 214a obtained by multiplying the reference exposure time t0 by 1/2 is determined in step S10.
4, and the exposure time t2 of the image sensor 214b obtained by multiplying the reference exposure time t0 by 1.67 by the above equation (41) is calculated in step S105. In step S106, one of the openings 215a to 215h on the aperture plate 215 is set on the optical path based on the aperture value calculated in step S103, and the image pickup devices 214a and 214
The exposure operation of 4b is performed. The exposure time is controlled by discharging the charges accumulated in the imaging elements 214a and 214b, and then controlling the exposure time in the next step S107.
Control is performed so that the time until the signal charge is read out from 214b becomes the exposure time t1, t2, respectively. Therefore,
First, the charge stored in the image sensor 214b is discharged, and then the charge stored in the image sensor 214a is discharged, so that charge storage is started in each image sensor.

After the exposure is completed, in step S107, the complete light-shielding portion 208a on the shutter plate 208 is set on the optical path, and an image signal is read out from the image pickup device 214a to obtain a DS.
Predetermined image processing is performed in P433a. Step S1
08, the image signal is read from the image sensor 214b and D
Predetermined image processing is performed in SP433b. Step S
In 109, the above-described single-chip interpolation and synthesis processing are performed on the image data on which the image processing has been performed. The synthesized image data is compressed according to a predetermined format, and the compressed image data is recorded on the memory card 424 (step S110), and the processing in FIG. 19 ends.

In the second embodiment described above, the following operation and effect can be obtained. (1) As shown in FIG.
The output data of 14a and 214b are superimposed, and the data of the two image sensors constituting the overlapped portion surrounded by the superimposed 1/2 pixel pitch are added and averaged. Therefore, even when there is a difference between the exposure conditions and the output value between the two image sensors, a saw-toothed jagged appearance due to the output value difference between the two image sensors 214a and 214b appears in the combined image data. Without
High quality image data can be obtained. Also, by adding, the shot noise component included in the image data captured by each image sensor can be reduced to 1 / √2, so that the S / N ratio of the image data can be improved. (2) Two image sensors 214a and 21 after single-plate interpolation
At the time of combining the image data by 4b, one pixel data is combined by weighting and averaging single-plate four-pixel data having different phases in consideration of similar directionality. Therefore, even when the image to be imaged has a high frequency component equal to or higher than the Nyquist frequency of the single image sensor, the correctly resolved data is included in one of the image data of the two image sensors 214a and 214b interpolated by the single image. If you have
A similar directionality can be determined. As a result, even if an inaccurate portion of a picture having a frequency component equal to or higher than the Nyquist frequency occurs in the image data obtained by one of the image sensors after the single-chip interpolation, an effect of correcting this effect can be obtained. That is, by performing the low-pass filter processing along the determined direction, the data in which the false color has occurred is corrected, and a high-quality image is obtained.

(3) The standard exposure time t
Exposure is performed for an exposure time t1 that is 1/2 of 0, and the image sensor 214b is exposed.
Is exposed for an exposure time t2 that is 1.67 times the standard exposure time t0, so that image data from these two imaging elements is synthesized. Therefore, when a low-luminance subject is photographed, an output value of image data equivalent to the case of the standard exposure time t0 can be obtained, and when a high-luminance subject is photographed, the output value of the image data is suppressed. As compared with the case where the exposure is performed at the exposure time t0, it is possible to photograph a subject having a higher luminance, and the dynamic range can be expanded.

In the above description, the four directions of similarity Dv [i, j], Dh [i, j], D45 [i,
j] and D135 [i, j] are calculated,
The luminance Y [i, j] is calculated for each pixel position according to (45), and the similarities Dv [i, j] and Dh [i in each direction are calculated using the luminance according to the following equations (46) to (49). , j], D45 [i, j] and D135
[I, j] may be calculated.

[Equation 14]

When the difference between Dmax [i, j] and Dmin [i, j] is equal to or smaller than a predetermined value T3, it is determined that the directions are similar in all directions (case 5). To calculate | Dv [i, j] -Dh [i,
j] | ≦ T3 and | D45 [i, j] −D135 [i, j] | ≦ T3, it may be determined to be similar in all directions.

Third Embodiment Display LCD 22 of digital still camera 100 described above.
The display operation to 0 will be described. The image data finally recorded on the memory card 424 by the digital still camera 100 is obtained by synthesizing the image data captured by the two image sensors 214a and 214b.
However, when the digital still camera 100 is provided with the display LCD 220 for performing viewfinder display and monitor display, an image before shooting and an image after shooting can be displayed on the LCD 220.

-Display of Image Data- FIG. 20 is a flowchart illustrating a process of displaying image data on the display LCD 220 in the digital still camera 100 according to the present embodiment. Step S40
At 1, it is determined whether the digital still camera 100 is set to a recording mode or a reproduction mode. When it is determined that the digital still camera 100 has been set to the recording mode by the mode changeover switch (the recording mode side in step S401), step S402 is performed.
It is determined whether or not a half-press operation signal has been input. On the other hand, when it is determined that the digital still camera 100 is set to the playback mode (the playback mode side of step S401), the process proceeds to step S411. It is determined in step S402 that the half-press operation signal has been input (step S4
The process proceeds to step S403 and the process proceeds to step S403. When it is determined that no input has been made (N in step S402), the process waits for the input of a half-press operation signal.

In step S403, image data based on the output of the image sensor 214a is taken into the DSP 433a. The image data captured by the DSP 433a includes R, G, and B colors at each pixel position on the image sensor 214a.
When predetermined image processing including pixel interpolation processing for calculating color data is performed, the predetermined image processing is temporarily stored in the buffer memory 436a, and then stored in the frame memory 435 to be displayed on the display LCD.
Displayed at 220 (step S404). The image displayed at this time is a monitor subject image before shooting called a through image. In step S405, the exposure times and the aperture values of the image sensors 214a and 214b are calculated based on the outputs from the image sensors 214a and 214b so that appropriate exposure is obtained. The image sensors 214a and 214b are exposed for a predetermined time, two image data based on the outputs from the image sensors 214a and 214b are compared, and focus detection by pupil division is performed as described above (step S406). Then, based on the obtained focus detection data, the focusing lens of the photographing lens 1 is driven to the in-focus position by the lens driving circuit 430.

It is determined that the full-press operation signal has been input.
(Y in step S407) and the quick return mirror 3
Jumps up, and the photographing process in step S408 is executed. On the other hand, when it is determined that the full-press operation signal is not input (N in step S407), the process returns to step S402. In step S408, an opening 208b of the shutter plate 208 through which all light beams pass is set on the optical path,
Aperture plate 21 based on the value calculated in step S405
When any of the openings 215a to 215h on the optical path 5 is set on the optical path, the image sensors 214a and 214b are exposed for a predetermined time.

In step S409, the image sensor 21
The DSP 433a and 433b simultaneously perform predetermined image processing on the image signals read from the 4a and 214b, respectively, and combine the image data after the image processing by the two imaging elements into one image data. The subject image after the image processing and the synthesis processing is displayed on display LCD 220. This display image is a photographed subject image called a freeze image. As the image data after the image processing, for example, data compressed in a predetermined format such as JPEG is recorded on the memory card 424.

In step S410, it is determined whether or not the recording / reproducing mode has been changed. If it is determined that the mode has been changed to the reproducing mode (Y in step S410), the process proceeds to step S411. When it is determined that the recording mode is maintained (N in step S410), the process returns to step S402. When the composite image data finally recorded on the memory card 424 is displayed on the display LCD 220 in step S411, it is determined whether or not a key operation has been performed to display another frame (step S412). . When it is determined that no key operation is performed (N in step S412), the display performed in step S411 is continued, and when it is determined that the key operation is performed (Y in step S412), the key operation is performed. The image of the frame, for example, the image of the first recorded frame is displayed on display LCD 220 (step S413). In step S414, it is determined whether the recording / playback mode has been changed again. When it is determined that the mode has been changed to the recording mode (Y in step S414), the process returns to step S402, and it is determined that the playback mode is maintained. At this time (N in step S414), the process returns to step S412.

In the third embodiment described above, the following operation and effect can be obtained. (1) When displaying a through-the-lens image before shooting, the live view is displayed on the display LCD 220 based on an image signal output from one of the two image sensors, and a frozen image after the image is displayed. In this case, since the image signals from the two image sensors are combined and displayed, the combining process of combining the image signals from the two image sensors is not required during live view display, and quick display is performed. be able to. (2) When displaying a frozen image after shooting, image data from the two image sensors is combined and displayed on the display LCD 220. Therefore, if the LCD 220 has a sufficient display resolution, the resolution is high. A high-quality freeze image can be displayed.

In the above description, in step S403, the image data based on the output of the
Although the image before photographing is displayed on the display LCD 220 by taking it into the P433a (step S404), the image may be performed based on the output of the image sensor 214b instead of the image sensor 214a.

In the above description, step S409
Performs predetermined image processing on image signals read out from the imaging elements 214a and 214b in the DSP 433.
a and 433b are performed simultaneously, and the image data after image processing by the two image sensors is combined into one image data and displayed on the display LCD 220. One of the image data by the image sensors 214a and 214b is used. One of them may be displayed. In this case,
It is possible to display an image faster than displaying it on the LCD 220 after combining it into one image.

Further, the combined image data is recorded on the memory card 424 in step S409. The image data may be separately recorded on the memory card 424. In this case, the amount of memory used may be smaller than in the case of recording image data whose data number has been increased by interpolation at the time of synthesis, so that more image data can be recorded on the memory card 424. However, when displaying the last recorded image in step S411, the image data output from both of the imaging elements 214a and 214b, each of which is subjected to image processing and separately recorded on the memory card 424 is read and combined. Then, the display is performed.

-Fourth Embodiment- In the first and second embodiments described above, data of all colors is interpolated corresponding to each pixel position by single-chip interpolation.
When two image data after single-chip interpolation are combined, all the color data are combined at each pixel position on the double-density pixel surface and calculated. However, in the fourth embodiment, when only one color component data representing a luminance signal corresponding to each pixel position is interpolated by single-chip interpolation and two image data after single-chip interpolation are combined, At each position on the double-density pixel surface, only the one color component data is synthesized and calculated, and other color component data is calculated by linear interpolation from color difference data with color component data representing a luminance signal.

The synthesizing procedure according to this embodiment performs the single-chip interpolation processing in the first embodiment on the G color, and uses the two sets of G-color image data after the single-chip interpolation to perform the first implementation. Is performed on the G color in the form of
The composite data g corresponding to all pixel positions on the double-density pixel surface is calculated. The R and B color data are arranged on the double density pixel surface. Color difference data between the synthesized data g and the R and B color data
(Cr = R-g, Cb = B-g) are calculated respectively.
This is performed by linearly interpolating the calculated color difference data and calculating the R and B color data using the interpolation result.

FIG. 21 is a diagram showing, for example, the combined data g of G color combined on the double-density pixel surface according to the first embodiment. In FIG. 21, the numbers described in the pixel positions are the image pickup devices 214a described in FIG.
And the numbers representing the pixel positions of 214b. R
The arrangement of the color and B color data is arranged corresponding to the pixel position where the number is written on the double-density pixel surface, that is, the pixel position of the image sensors 214a and 214b. FIG. 22 is a diagram in which R color data is arranged on a double density pixel surface, and FIG. 23 is a diagram in which B color data is arranged on a double density pixel surface.

At the pixel position where the R color data is arranged in FIG. 22, the color difference data Cr = R−g of the G color combined data g and the R color data is calculated. Then, linear interpolation is performed on the blank portion of the color difference data using the existing color difference data. For example, in FIG. 22, the chrominance data at 131 pixel positions existing between the 121 and 141 pixel positions in the first row is the average (intermediate) value of the chrominance data at the 121 and 141 pixel positions. The chrominance data at the pixel positions existing between are assumed to be the average (intermediate) value of the chrominance data at the pixel positions 121 and 131. Similarly, for the second, fifth, and sixth rows, color difference data is interpolated by linear interpolation using color difference data existing in the horizontal direction. Further, in FIG. 22, there is no color difference data in the horizontal direction for the third, fourth, seventh and eighth rows, so if the color difference data is interpolated by performing linear interpolation in the vertical direction using the color difference data interpolated as described above, Color difference data C at each pixel position on the density pixel surface
r is generated (FIG. 24).

When the color difference data Cr is obtained for all the pixel positions on the double-density pixel surface, the combined R color data for each pixel position is calculated by r = Cr + g. If the same processing as the above processing is performed for the B color, the color difference data C
Using b = B−g, the B color data after synthesis is b = Cb + g
Is calculated by As described above, g, r, and b color data can be calculated for all pixel positions on the double-density pixel surface.

In the above description, the combined data g of G is generated on the combined double-density pixel surface by the combining method in the first embodiment. However, the combined data g may be generated in the second embodiment. it can.

The fourth embodiment has the following operation and effect. Only the G color component data representing the luminance signal corresponding to each pixel position is interpolated by single-chip interpolation, and the combination of the two image data after the single-chip interpolation is performed at each pixel position on the double-density pixel surface. Only these are combined and calculated. Since the color data of the R and B color components is calculated by linearly interpolating the color difference data with the synthesized data g of the G color, single-chip interpolation is performed for all the color component data corresponding to each pixel position. In addition, compared with the case where all the color component data are synthesized corresponding to each pixel position on the double-density pixel plane, the processing at the time of single-chip interpolation and synthesis can be reduced. As a result, the effect of shortening the synthesis processing time can be obtained.

In the above description, the case of the color filters of the R, G, and B primary colors has been described.
The present invention is also applicable to a case where a complementary color filter such as G or Ye is used.

Although the digital still camera has been described in the first to fourth embodiments, the above-described single-chip interpolation and the synthesizing process after the single-chip interpolation are performed in the form of software on a CD-ROM or a floppy disk. It can also be stored as an image processing program on a recording medium such as a personal computer and used when performing image processing on a personal computer. In this case, the image sensor 2
Based on the image signals output from 14a and 214b, both digitized image data were recorded on a large-capacity recording medium for image data, respectively, and the recording medium was set on a personal computer to capture both image data. Above, the above-described single-chip interpolation and the synthesis processing after the single-chip interpolation are performed by the image processing program.

Instead of reading the image processing program from a recording medium having the image processing program recorded thereon by a personal computer, the image processing program may be transmitted using a transmission medium such as the Internet. In this case, the transmitted image processing program is read by a personal computer, and the above-described single-chip interpolation and the synthesis processing after the single-chip interpolation are performed by the personal computer.

In the above description, a single-lens reflex digital still camera has been described. However, the present invention can be applied to a digital still camera in which a lens cannot be replaced and a digital video camera which can also capture a moving image.

The correspondence between each component in the claims and each component in the embodiment of the present invention will be described.
9 corresponds to a single-plate interpolation unit, a combination unit, a data arrangement unit, an interpolation unit, and a data arrangement unit, and the CPU 439 corresponds to a similarity calculation unit, a direction classification unit, a first calculation unit, and a second calculation unit.

[0094]

As described in detail above, in the present invention,
The following effects are obtained. (1) According to the first to fourteenth aspects of the present invention, single-chip interpolation is performed on image data output from two imaging devices that have captured the same subject image, and all color component data are provided corresponding to all pixels. It was made to synthesize from. Since the single-chip interpolation performs at least one of interpolation using the directionality of image data and interpolation using imaging data of a plurality of color components, the resolution can be increased for all pixels. Further, by synthesizing the image data after the single-plate interpolation, an effect of further improving the resolution of the image data after the synthesis can be obtained. In particular, according to the second and ninth aspects of the present invention, since the composition is performed using the directionality of the data after the single-plate interpolation, the edges of the subject image are composed smoothly. (2) In addition to the above (1), two sets of image data after single-plate interpolation are arranged in a checkerboard pattern, and the data arranged in a checkerboard pattern is added to the above (1). The data at the blank pixel position is interpolated and synthesized for each color component in accordance with the similar directionality of. Therefore, image data having all the color components at double density composed of data twice as high in the vertical and horizontal directions with respect to the pixel arrangement of the image data after the single-plate interpolation before the synthesis is synthesized, and high-resolution image data is obtained. be able to. (3) In the invention of claims 4, 5, 7, 8, and 14,
In addition to (1), two sets of image data after single-chip interpolation are overlapped with each other shifted by 1/2 pixel pitch in the vertical and horizontal directions corresponding to the direction of the pixel arrangement of the imaging device, and surrounded by the overlapped 1/2 pixel pitch. The data from the respective imaging devices constituting the overlapped portion are averaged. Therefore, for example, even when the exposure conditions are different between the two imaging devices, the sawtooth-like jaggedness due to the difference in the exposure conditions of the two imaging devices does not appear in the synthesized image data, and high-quality image data is obtained. Is obtained. Furthermore, since the average value data is synthesized using the average value data existing in similar directions, high resolution can be achieved, and the average value data can be combined with the shot noise of the imaging device included in the image data. It becomes small by the random noise component being averaged,
High-quality image data with a good S / N ratio can be obtained. (4) According to the seventh aspect of the present invention, in addition to the configuration of the third and fourth aspects, when the similarity of data is classified, when the similarity of the directions is less than a predetermined value, each direction has the same similarity. Since the classification is made as the degree, for example, when there is much noise, incorrect classification is prevented. (5) According to the second and ninth aspects of the present invention, since the directionality is checked for a plurality of all color component data, similar directions of data can be accurately grasped for all colors. (6) According to the eleventh aspect of the present invention, the image pickup data output from the two image pickup devices that picked up the same subject image are each subjected to single-chip interpolation, and at least luminance component data is provided for all pixels, and then combined. I did it. Since the single-chip interpolation performs at least one of the interpolation using the directionality of the imaging data and the color difference interpolation, it is possible to increase the resolution of the luminance component corresponding to all pixels. Further, by synthesizing the image data after the single-chip interpolation, an effect of further improving the resolution of the luminance component of the image data after the synthesis can be obtained.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating a configuration of a single-lens reflex digital still camera according to a first embodiment.

FIG. 2 is a view of the single-lens reflex digital still camera of FIG. 1 as viewed from the back.

FIG. 3 is a diagram illustrating an imaging optical system of the single-lens reflex digital still camera in FIG. 1;

FIG. 4 is a block diagram showing a circuit of the single-lens reflex digital still camera in FIG. 1;

FIGS. 5A and 5B are diagrams illustrating (a) the arrangement of the pixels of the imaging element 214a and (b) the arrangement of the pixels of the imaging element 214b.

FIG. 6 is a diagram showing pixel positions in FIGS. 5 (a) and 5 (b).

FIG. 7 is a diagram illustrating weighting for calculating a similarity.

FIG. 8 is a diagram for classifying similar directions from vertical similarities Cv [i, j] and horizontal similarities Ch [i, j].

FIG. 9 is a diagram illustrating G color interpolation.

10A shows pixels after single-chip interpolation of image data by the image sensor 214a, FIG. 10B shows pixels after single-chip interpolation of image data by the image sensor 214b, and FIG. (D) is a diagram in which (b) is arranged in every other row and column, and (e) is a diagram in which (c) and (d) are combined in a checkered pattern.

FIG. 11 is a diagram illustrating positions of checkerboard-shaped arrangement data.

FIG. 12 shows a vertical similarity Dv [i, j] and a horizontal similarity Dh [i, j].
It is a figure which classifies a similar direction from.

FIG. 13 is a flowchart of a first half of a program started by a half-press switch.

FIG. 14 is a flowchart of the latter half of the program started by the half-press switch.

FIG. 15A shows pixels of image data by the image sensor 214a, (b) pixels of image data by the image sensor 214b, and FIG. 15C shows image data of (a) and (b) in the vertical and horizontal directions. It is the figure which shifted by 1/2 pixel pitch, respectively, and overlapped.

FIG. 16 is a diagram showing the position of data arranged according to FIG. 15 (c).

FIG. 17 is a diagram for classifying similar directions.

FIG. 18 is a diagram illustrating a relationship between an input value and an output value of the imaging device.

FIG. 19 is a flowchart of exposure / image processing including exposure time determination.

FIG. 20 is a flowchart of image data display processing of the digital still camera according to the third embodiment.

FIG. 21 is a diagram illustrating combined data g of G colors combined on a double-density pixel surface.

FIG. 22 is a diagram in which R color data is arranged on a double density pixel surface.

FIG. 23 is a diagram in which B-color data is arranged on a double-density pixel surface.

FIG. 24 shows color difference data Cr generated on a double-density pixel surface.
FIG.

[Explanation of symbols]

1. Photographic lens, 3. Quick return mirror, 4. Focusing plate, 16.
Primary imaging plane, 50: finder device, 200
... Camera body, 208 ... Shutter plate, 2
08a: complete light shielding portion, 208b to 208j: opening portion,
209: field lens, 210, 211 ... mirror, 212: reduction optical system, 212a, 212
b: relay lens, 213a, 213b: protective glass,
214a, 214b: image sensor, 215: aperture plate,
215a to 215g: opening, 219: polarizing beam splitter, 220: display LCD, 2
23a, 223b: Optical low-pass filter, 308c to 308j: Polarizing plate, 4
07 ... operating member, 424 ... memory card, 433a, 433b ... DSP, 435 ... frame memory, 436a, 436b ... buffer memory, 4
39 ... CPU

 ──────────────────────────────────────────────────続 き Continued on the front page F term (reference) 5C065 AA03 BB26 CC02 CC03 CC08 CC09 DD02 DD18 EE06 GG02 GG13 GG18 GG21 GG29 GG30 GG32

Claims (14)

[Claims] 1. A plurality of color image pickup devices which are shifted by 1/2 pixel in at least one of the vertical and horizontal directions with respect to the direction of pixel arrangement, pick up the same subject image, and obtain image data output from at least two color image pickup devices. In a multi-panel imaging apparatus for obtaining a subject image by combining images, at least interpolation using the directionality of the image data and imaging data of a plurality of color components are used for each of the imaging data output from the two color imaging apparatuses. Single-chip interpolation means for generating all-color-component data corresponding to all pixels of the color image pickup device using one of the interpolations, and the two full-color-component data generated by the single-chip interpolation means. A multi-panel imaging apparatus, comprising: a synthesizing unit for synthesizing. 2. The multi-panel imaging apparatus according to claim 1, wherein said synthesizing means performs synthesizing using a directionality based on said two all-color component data. 3. The multi-chip imaging device according to claim 1, wherein the two color imaging devices are arranged so as to be shifted by a half pixel in both the vertical and horizontal directions with respect to the pixel arrangement direction. With respect to the two all-color component data, color component data is arranged every other row in both the vertical and horizontal directions corresponding to the direction of the pixel arrangement for each color component, and one color component data arranged every other row is A data arrangement means for arranging the other color component data arranged in every other row to form a checkerboard pattern color component data array for each color component; and the checkerboard pattern color component data array by the data arrangement means A similarity calculating means for calculating at least two similarities in the vertical and horizontal directions; and a directionality for classifying the directionality of the similarity from the similarity calculated by the similarity calculating means. In the checkerboard-like color component data array for each color component, interpolation data of a blank pixel position is used for each color component using color component data arranged in the direction classified by the directionality classifier. A multi-chip imaging device comprising: an interpolation unit that generates the image data. 4. The multi-chip imaging device according to claim 1, wherein the two color imaging devices are arranged so as to be displaced by 1/2 pixel in both the vertical and horizontal directions with respect to the pixel arrangement direction. Means for arranging the two all-color component data so as to overlap each other for each color component by a half pixel pitch in both the vertical and horizontal directions corresponding to the pixel arrangement direction; With respect to the color component data superimposed for each component, for all the overlapped portions surrounded by the vertical and horizontal 1/2 pixel pitch, a first value for calculating the average value of the two color component data constituting the overlapped portion is calculated. A calculating unit; a similarity calculating unit configured to calculate a similarity in at least two directions in an average value of the color component data calculated by the first calculating unit; A direction classifying unit that classifies the directionality of the similarity from the similarity calculated by the similarity calculating unit; and an average value of the color component data calculated by the first calculating unit for each color component, Second calculating means for calculating weighted average again by using a predetermined number of the color component data arranged in the direction classified by the directionality classifying means and calculating composite data at a target pixel position for each color component. A multi-panel imaging device characterized by the above-mentioned. 5. The multi-chip imaging device according to claim 3, wherein said similarity calculating means calculates the similarity using all color component data generated by said single-chip interpolation means. Multi-plate type imaging device. 6. The multi-chip imaging device according to claim 3, wherein the similarity calculating means uses the similarity to the blank pixel position and the similarity to pixel positions around the blank pixel. A multi-panel imaging apparatus, wherein a similarity is calculated. 7. The multi-panel imaging device according to claim 3, wherein the directionality classifying unit is configured to determine whether a difference between similarities in each direction calculated by the similarity calculating unit is smaller than a predetermined value. The multi-panel imaging device is characterized in that the similarities in the respective directions are classified as being equal. 8. The multi-chip imaging device according to claim 4, wherein the two color imaging devices pick up the subject image by changing an exposure amount. 9. A plurality of color imaging devices which are shifted by 1/2 pixel in at least one of the vertical and horizontal directions with respect to the direction of the pixel array, and image the same subject image, and obtain image data output from at least two color imaging devices. A multi-chip imaging apparatus that obtains a subject image by combining; a single-chip interpolation unit configured to generate all color component data corresponding to all pixels of respective imaging data output from the two color imaging apparatuses; A multi-chip imaging device, comprising: synthesizing means for synthesizing using the directionality of the two all-color component data generated by the interpolating means. 10. The multi-chip imaging device according to claim 9, wherein said single-chip interpolation means uses at least the directionality of the image data for each of the imaging data output from said two color imaging devices. A multi-panel imaging apparatus, wherein all color component data is generated corresponding to all pixels of the two color imaging apparatuses by using one of interpolation and interpolation using imaging data of a plurality of color components. 11. A plurality of color image pickup devices which are shifted by 1/2 pixel in at least one of the vertical and horizontal directions with respect to the direction of the pixel array, and pick up the same subject image, and obtain image data output from at least two color image pickup devices. In a multi-panel imaging apparatus for obtaining a subject image by combining images, at least interpolation using the directionality of the image data and imaging data of a plurality of color components are used for each of the imaging data output from the two color imaging apparatuses. Single-chip interpolation means for generating data corresponding to at least a luminance component corresponding to all pixels of the color imaging apparatus using one of the interpolations, and corresponding to the luminance component generated by the single-chip interpolation means And a synthesizing means for synthesizing two pieces of data. 12. At least the directionality of the image data is used for two image data obtained by capturing the same subject image by a plurality of color imaging devices shifted in at least one of the vertical and horizontal directions with respect to the direction of the pixel array. A single-chip interpolation process for generating all-color-component data corresponding to all pixels by using either one of interpolation using image data of a plurality of color components and interpolation performed by using the single-chip interpolation process. And a program for performing a synthesizing process for synthesizing the two color component data. 13. The image data processing recording medium according to claim 12, wherein the plurality of color image pickup devices are arranged so as to be shifted by a half pixel in both the vertical and horizontal directions with respect to the pixel arrangement direction. Is for the two color component data, color component data is arranged in every other row in both the vertical and horizontal directions corresponding to the direction of the pixel arrangement of the image data for each color component, and one color component is arranged in every other row A data arrangement process for arranging the other color component data arranged in every other row between data to form a checkered pattern color component data array for each color component; and the checkerboard pattern color component by the data arrangement process In the data array, a similarity calculation process for calculating at least two similarities in the vertical and horizontal directions, and a directionality of the similarity calculated from the similarities calculated by the similarity calculation process. In the directional classification process for classifying, in the checkered pattern color component data array for each color component, the interpolation data of the blank pixel position is calculated using the color component data arranged in the direction classified by the directional classification process. A recording medium for image data processing, comprising: an interpolation process for each color component. 14. The image data processing recording medium according to claim 12, wherein the plurality of color image pickup devices are arranged so as to be shifted by 1/2 pixel in both the vertical and horizontal directions with respect to the pixel arrangement direction. Is a data arrangement process in which, for each of the two color component data, a data arrangement process is performed for each color component so as to be superimposed and shifted by a half pixel pitch in both the vertical and horizontal directions corresponding to the pixel arrangement direction of the image data; With respect to the color component data superimposed and arranged for each color component by the processing, the average value of the two color component data constituting the overlapping portion is calculated for all the overlapping portions surrounded by the vertical and horizontal 1/2 pixel pitch. A first calculating process, and a similarity calculating a similarity in at least two directions in an average value of the color component data calculated by the first calculating process. Calculation processing; directionality classification processing for classifying the directionality of the similarity from the similarity calculated by the similarity calculation processing; and calculation of the color component data calculated in the first calculation processing for each color component. A second method of weighting and averaging the average value again using a predetermined number of the color component data arranged in the direction classified by the directionality classification process and calculating composite data at the target pixel position for each color component A recording medium for image data processing, comprising: a calculating process.