JP5860251B2 - Imaging apparatus, control method therefor, and program - Google Patents

Description

  The present invention relates to a technique for generating a binocular stereoscopic image using one imaging optical system.

  In recent years, imaging devices such as display devices that allow users to perform binocular stereoscopic viewing using various methods and digital cameras that can generate binocular stereoscopic images that can be browsed on such display devices have become widespread. ing.

  The human brain compares the images captured by the left and right eyes and can perceive distance, that is, a solid by the parallax, which is a shift that occurs between the images. And right-eye images. Some imaging devices capable of generating a binocular stereoscopic image have two imaging optical systems that are spaced apart from each other, and are used for the left eye and the right eye using the imaging optical system. It is possible to take an image.

  On the other hand, it is also possible to capture a binocular stereoscopic image having parallax even in an imaging apparatus having one imaging optical system. For example, in Patent Document 1, a microlens array is provided, and light beams that have passed through different positions of the exit pupil of the imaging optical system are imaged on a pair of light receiving elements arranged corresponding to one microlens. Techniques for taking images for the left eye and the right eye are disclosed.

  In a binocular stereoscopic image, the parallax is 0 on the reference plane where the gazing point exists in the depth direction, but the object perceived to exist at a position far away from or close to the plane is the reference. The further away from the surface, the more parallax occurs. In other words, by controlling the maximum amount of positional deviation of the image between the left-eye and right-eye images, it is possible to control the stereoscopic effect in the depth direction that is perceived by the viewer.

  For this reason, some imaging devices capable of capturing a binocular stereoscopic image can capture an image after determining the stereoscopic effect perceived by the viewer. For example, in the case of an imaging apparatus having two imaging optical systems, the amount of image misregistration between images can be reduced by changing the cut-out position and the image enlargement ratio in each captured left-eye and right-eye image. Can be controlled.

JP-A-58-24105

  However, in the case of an imaging apparatus having one imaging optical system, the stereoscopic effect that can be perceived by the viewer is limited by the specifications of the imaging optical system. For example, as shown in FIG. 11A, a pair of light-receiving elements 1102 and 1103 for the left eye and the right eye arranged corresponding to one microlens 1101 are different from each other in the exit pupil 1104. The light flux that has passed through the regions 1105 and 1106 is imaged. At this time, the light beams that are refracted by the microlens and enter the light receiving element 1102 for the left eye and the light receiving element 1103 for the right eye have a predetermined incident angle range.

  The intensity distribution of the incident light beam at the incident angle is as shown in FIG. In FIG. 11B, the incident angle of light incident on the light receiving element is incident at each incident angle with the incident angle from the right side being positive on the horizontal axis and the incident angle from the left side being negative on the horizontal axis. It is a figure which made the intensity | strength of the light beam the vertical axis | shaft. In the drawing, the intensity distribution of the incident light beam to the light receiving element 1102 for the left eye is shown as 1111, and the intensity distribution of the light beam incident to the light receiving element 1103 for the right eye is shown as 1112. Since the intensity peak is the position of the center of gravity in the region where each incident light beam passes through the exit pupil, the maximum parallax in the left-eye and right-eye images generated by the light receiving element is the regions 1105 and 1106. It is limited to the distance formed by the center of gravity position.

  That is, the maximum parallax in the binocular stereoscopic image captured by one imaging optical system is determined according to the radius of the exit pupil when the aperture is in the open state, and thus for the left eye and the right eye In the case of expanding the parallax of the image for use, it is necessary to use an imaging optical system having a large exit pupil.

  The present invention has been made in view of the above-described problems, and an object thereof is to extend parallax in a binocular stereoscopic image obtained by a light beam that has passed through different regions of an exit pupil.

In order to achieve the above object, an imaging apparatus of the present invention has the following configuration.
An imaging device that includes light-receiving elements for the left eye and the right eye , converts light beams that have passed through the same imaging optical system having a diaphragm, and outputs images for the left eye and the right eye, and for the left eye and the right Generating a binocular stereoscopic image using control means for controlling the aperture of the diaphragm so that the parallax in the image for the eye changes and the image for the left eye and the right eye output by the imaging device and means, and generating means, each image pixel for the left eye and right eye parallax is captured for each of the different first and second states of the image of the left-eye and right-eye By subtracting a value between states, an image for binocular stereoscopic viewing in which a parallax is expanded from an image for a left eye and a right eye in any state is generated.

  With such a configuration, according to the present invention, it is possible to extend parallax in a binocular stereoscopic image obtained by a light beam that has passed through different regions of the exit pupil.

1 is a block diagram showing a functional configuration of a digital camera 100 according to an embodiment of the present invention. The figure which showed the pixel structure of the CMOS type solid-state image sensor concerning embodiment of this invention. The figure which showed the internal structure of the light receiving element of the CMOS type solid-state image sensor which concerns on embodiment of this invention The figure for demonstrating the read-out operation | movement of the imaging part 106 which concerns on embodiment of this invention. The figure for demonstrating the concept of the parallax expansion which concerns on embodiment of this invention. Another figure for demonstrating the concept of parallax expansion which concerns on embodiment of this invention Flowchart of stereoscopic image shooting processing according to an embodiment of the present invention Flowchart of photographing processing according to a modification of the present invention The figure for demonstrating the further method of the parallax expansion which concerns on the modification of this invention. The figure for demonstrating another method for solving the subject of this invention The figure for demonstrating the concept which acquires the image for binocular stereoscopic vision using the light beam which passed through the area | region where an exit pupil differs.

[Embodiment]
Hereinafter, a preferred embodiment of the present invention will be described in detail with reference to the drawings. Note that, in the embodiment described below, the present invention is applied to a digital camera that can capture a binocular stereoscopic image using a light beam that has passed through different regions of the exit pupil, as an example of an imaging device. An example will be described. However, the present invention can be applied to any device that can capture a binocular stereoscopic image using a light beam that has passed through different regions of the exit pupil.

<Functional configuration of digital camera 100>
FIG. 1 is a block diagram showing a functional configuration of a digital camera 100 according to an embodiment of the present invention.

  The CPU 101 controls the operation of each block included in the digital camera 100. The CPU 101 controls the operation of each block by, for example, reading a stereoscopic image shooting process program, which will be described later, stored in the ROM 102, developing it in the RAM 103 and executing it.

  The ROM 102 is a rewritable nonvolatile memory such as an EEPROM, for example, and stores operation parameters necessary for the operation of each block, settings of the digital camera 100, and the like in addition to a parallax expansion processing program. The RAM 103 is a volatile memory, and stores not only a development area for a parallax expansion processing program but also intermediate data output in the operation of each block.

  The imaging optical system 104 includes an objective lens, a diaphragm, an eyepiece lens, and the like, and forms a subject image on a light receiving element of the imaging unit 106. In the description of this embodiment, the diaphragm is described as a mechanical diaphragm, but the present invention can be implemented even with a transmissive liquid crystal diaphragm that can control the shape, diameter, and the like of the diaphragm.

  The imaging unit 106 is a block including an imaging element such as a CCD or CMOS sensor and a readout circuit, for example, and photoelectrically converts a subject image formed on the element and outputs an analog image signal.

(Configuration of image sensor)
In the present embodiment, since the images for the left eye and the right eye are generated using the light fluxes that have passed through different positions of the exit pupil of the imaging optical system 104, one pixel of the image sensor is the left eye as shown in FIG. And a pair of light receiving elements 201 and 202 for the right and right eyes. A pair of light receiving elements corresponding to one pixel is provided with a microlens 203 for each pixel of the image sensor so that light beams that have passed through different areas of the exit pupil are imaged.

  FIG. 3 shows a configuration of the left-eye light receiving element 201 or the right-eye light receiving element 202 included in one pixel, taking a CMOS sensor as an example.

  The photodiode 301 for generating the optical signal charge is grounded on the anode side in the drawing. The cathode side of the photodiode 301 is connected to the gate of the amplification MOS transistor 304 via the transfer MOS transistor 302. The gate of the amplification MOS transistor 304 is connected to the source of a reset MOS transistor 303 for resetting it. The drain of the reset MOS transistor 303 is connected to the power supply voltage VDD. Further, the amplification MOS transistor 304 has a drain connected to the power supply voltage VDD and a source connected to the drain of the selection MOS transistor 305.

  The gate terminal of the transfer MOS transistor 302 is driven by the Ptx signal, and the gate terminal of the reset MOS transistor 303 is driven by the Pres signal. The gate terminal of the selection MOS transistor 305 is driven by the Psel signal, and the optical signal (pixel signal) generated by the photodiode 301 is output from the Vout terminal at an appropriate timing.

  Specifically, pixel signals generated by the left-eye and right-eye light-receiving elements are separately read out by the readout circuits (CH1) and (CH2) according to the readout timing input by the TG 105 under the control of the CPU 101. Read out and output.

  FIG. 4 is a diagram showing a circuit configuration necessary for reading pixel signals in the CMOS sensor. As shown in FIG. 4, the outputs of the left-eye light receiving element 201 and the right-eye light receiving element 202 are connected to different readout circuits. Since the unit pixel is composed of the left-eye light-receiving element 201 and the right-eye light-receiving element 202 as described above, the unit pixel is shown by a broken line. An image sensor comprising 8 pixels in a row is shown.

  Here, reading of the pixel signal for the right eye will be described below. Note that the readout circuit (CH1) similarly performs readout of the pixel signal for the left eye.

  The vertical shift register 401 receives the signal voltage via the row selection lines (vertical scanning lines) Pres, Ptx, and Psel in accordance with the timing signal input from the TG 105, and receives the signal voltages for the left eye 201 and the right eye. Applied to 202. In each light receiving element, the gate of the transfer MOS transistor 302 is connected to the row selection line Ptx, the gate of the reset MOS transistor 303 is connected to the row selection line Pres, and the gate of the selection MOS transistor 305 is connected to the row selection line Psel. Thus, the light receiving elements in the same row are commonly connected to the same vertical scanning line. Further, the source of the selection MOS transistor 305 of each light receiving element is connected to a terminal Vout of a vertical signal line arranged extending in the vertical direction, and is connected to a constant current source 407 as a load means.

  The output of each light receiving element is read out by the reading circuit 402 via the vertical signal line. The readout circuit 402 outputs a pixel signal to an n-channel MOS transistor 403 and a noise signal to an n-channel MOS transistor 404 among pixel signals on a vertical signal line as an input. The signal output to each n-channel MOS transistor is input to the differential amplifier 405, and the difference between the pixel signal and the noise signal is output as an analog image signal. Note that the reading circuit 402 includes a reading circuit for each column. The horizontal shift register 406 controls ON / OFF of the n-channel MOS transistor 403 and the n-channel MOS transistor 404.

  As described above, the imaging unit 106 according to the present embodiment includes a pair of light receiving elements in one pixel, acquires left-eye and right-eye images using light beams that have passed through different regions of the exit pupil, and performs analog front Output to the end (AFE) 107.

  The AFE 107 performs signal amplification and A / D conversion processing on the input analog image signal according to the timing signal input from the TG 105, and outputs image data for the left eye and right eye.

  A DSP (Digital Signal Processor) 108 is an image processing circuit that executes correction processing such as color conversion and white balance adjustment on input image data, encoding processing into a predetermined recording format, and the like.

  The photometry circuit 109 is a circuit for measuring the subject brightness, and the subject brightness information acquired by the circuit is used for determining the aperture value and the exposure amount in normal image shooting.

  The display unit 110 is a display device included in the digital camera 100 such as a small LCD, for example, and is used for displaying a photographed image and an image recorded on a recording medium described later. In the present embodiment, it is assumed that the display unit 110 is a display device that displays images for the left eye and the right eye so that the user can perform autostereoscopic viewing. Further, by sequentially outputting analog image signals captured by the imaging unit 106 to the display unit 110, the display unit 110 functions as an electronic viewfinder.

  The recording medium 111 is a recording device that is detachably connected to the digital camera 100 such as a built-in memory included in the digital camera 100, a memory card, or an HDD. It is assumed that an image photographed by the digital camera 100 is recorded on the recording medium 111 after being converted into a predetermined encoding format.

  The operation unit 112 is a user interface provided in the digital camera 100 such as a power button, a shutter button, and a mode change switch. The operation unit 112 outputs a control signal indicating the operation content to the CPU 101 in response to the operation of each operation member by the user.

<Concept of parallax expansion>
Here, the concept of the method for extending the parallax in the binocular stereoscopic image obtained by the light flux that has passed through different areas of the exit pupil in the present invention will be described with reference to FIG.

  For example, when the aperture of the imaging optical system 104 is in an open state, the optical images formed on the left-eye light receiving element 501 and the right-eye light receiving element 502 are different at the exit pupil 503 as shown in FIG. It is a light beam that has passed through each of the regions 504 and 505. The intensity distribution with respect to the incident angle of the light beam at this time is as shown in FIG. 5D as described above with reference to FIG. 11, and the parallax in the binocular stereoscopic image obtained by the light beam is open. It is determined according to the radius L1 of the exit pupil at the time of aperture.

  When the aperture of the imaging optical system 104 is in the aperture state, the optical image formed on the left-eye light receiving element 501 and the right-eye light receiving element 502 is narrower than the exit pupil 503 as shown in FIG. In the exit pupil 511, the light beams have passed through different regions 512 and 513, respectively. At this time, the intensity distribution with respect to the incident angle of the light beam is as shown in FIG. 5E, and the parallax in the binocular stereoscopic image obtained by the light beam is the radius L2 (L2 <L2 <L2 <L L1).

  In the present invention, the pixel values of the left-eye and right-eye image data obtained by changing the aperture state in this way are subtracted to pass through the regions 521 and 522 as shown in FIG. Image data corresponding to the optical image formed by the light beam is acquired. That is, as a result of the subtraction, the intensity distribution with respect to the incident angle of the light flux becomes as shown in FIG. 5F, and the sensitivity is relatively high with respect to the light from the outer periphery of the lens. The distance L3 can be extended from the distance between the centers of gravity when the aperture is open. That is, the parallax in the left-eye and right-eye images determined according to the distance between the centers of gravity can be expanded.

Note that in the case of an imaging optical system in which the aperture diameter and the exit pupil diameter match, the light beams reaching the left-eye and right-eye light receiving elements shown in FIGS. 5A to 5C pass through the exit pupil. The distances L1, L2, and L3 between the centers of gravity of the regions to be processed are as follows.
L1 = D1 / 2
L2 = D2 / 2
L3 = L1 + L2
Here, D1 is the aperture diameter when the aperture is fully open, and D2 is the aperture diameter in the aperture state.

In addition, the output Pb of the left-eye light receiving element 501 and the right-eye light receiving element 502 in the case of FIG. _5B depends on the optical performance of the imaging optical system 104 and the arrangement of the microlenses and light receiving elements of the imaging unit 106. changes by the condenser performance due to _{it, P a × (D2 / D1} ) based on the output _{P a} in the open aperture state _{≦ P b ≦ P a × (} D2 / D1) 2
Included in the range. Also in this case, the output P _c in the image data subtracted to extend the parallax is
P _c = P _a −P _b
It becomes.

  Since the aperture of the aperture is controlled step by step by a drive system such as a stepping motor, the left and right eyes obtained in the aperture state are obtained from the left-eye and right-eye image data obtained in the open state. Even if the image data for eyes is subtracted, an image having a desired parallax may not be obtained. In this case, for example, a binocular stereoscopic image having a desired parallax may be acquired by subtracting image data obtained in different aperture states as shown in FIG.

  As described above, the CPU 101 may calculate a combination of two aperture values selected so that the binocular stereoscopic image has a desired parallax. In addition, information on a combination of two aperture values for shooting so as to obtain a desired parallax may be stored in advance in the ROM 102, for example. In this way, the CPU 101 can determine two types of aperture values to be photographed in order to obtain a binocular stereoscopic image having a set parallax.

  The combination of the two aperture values has the following tendency depending on the desired parallax. That is, when a large parallax is required, the aperture value is determined so that the aperture of the aperture on the open side becomes larger in order to capture a light beam incident from the outside. Further, in order to increase the distance between the centers of gravity of the areas where the light flux corresponding to the image data after the subtraction passes through the exit pupil, the aperture of the closed side is closer to the aperture of the open side, that is, the aperture of the open side The aperture value is determined so as to reduce the difference between the aperture of the aperture and the aperture of the aperture on the closed side. Thus, by using the present invention, the parallax in the left-eye and right-eye image data obtained by the light flux that has passed through different regions of the exit pupil is theoretically close to a value close to the diameter of the exit pupil. Can be extended.

<Stereoscopic image shooting processing>
A specific process of the stereoscopic image shooting process of the digital camera 100 of the present embodiment having such a configuration will be described with reference to the flowchart of FIG. The processing corresponding to the flowchart can be realized by the CPU 101 reading out a corresponding processing program stored in, for example, the ROM 102, developing it in the RAM 103, and executing it. Note that this stereoscopic image shooting processing is performed by, for example, a shooting instruction by a user's shutter button operation in a state in which the shooting mode of the digital camera 100 is set to a parallax enhancement mode for shooting a binocular stereoscopic image with a large parallax. It is assumed that the process starts when is entered. That is, in the binocular stereoscopic image obtained by the stereoscopic image capturing process, the parallax between the left-eye image and the right-eye image in the image is expanded more than the image obtained in the normally open aperture state. Shall.

  In step S <b> 701, the CPU 101 sets the aperture value to an open side value (first aperture value) such as F2.0, executes the shooting process, and stores the obtained aperture side image data in the RAM 103. As described above, the aperture opening side image data is composed of image data for the left eye and for the right eye.

  In step S <b> 702, the CPU 101 determines whether or not saturated pixels are included in the left-eye and right-eye image data of the aperture opening side image data captured in step S <b> 701. Specifically, the CPU 101 acquires the pixel value of each pixel of the left-eye image data and the right-eye image data, and determines whether there is a pixel indicating the maximum value. If the CPU 101 determines that saturated pixel is included in the left-eye and right-eye image data of the aperture-opening side image data, the process proceeds to S707, and if it is determined that the saturated pixel is not included, the processing is performed in S703. Move to.

  When there is a saturated pixel, even if subtraction is performed on image data captured with a different aperture value, the image data obtained from the light beam that has passed through the corresponding exit pupil region is strictly different. For this reason, if there is a saturated pixel in the stereoscopic image capturing process, it is determined that the parallax cannot be expanded, and the aperture opening side image data is recorded in the recording medium 111 as a binocular stereoscopic image in S707 described later. To do.

  In step S <b> 703, the CPU 101 sets the aperture value to a closed value (second aperture value) such as F8.0, and performs shooting processing with the same exposure time as when shooting aperture-open image data. The closed aperture side image data is stored in the RAM 103. In other words, by taking images with the same exposure time, it is possible to avoid the generation of a saturation signal and to properly handle aperture closed side image data when subtracting aperture open side image data and aperture closed side image data. The image data from which the luminous flux component is removed can be obtained.

  In step S <b> 704, the CPU 101 causes the DSP 108 to perform subtraction between the aperture opening side image data and the aperture closing side image data for the left-eye image data and the right-eye image data. For example, the DSP 108 reads out each image data stored in the RAM 103, subtracts pixel values from the left-eye image data and the right-eye image data, and stores the obtained image data in the RAM 103.

In step S <b> 705, the CPU 101 causes the DSP 108 to execute a gain correction process so as to compensate for the average luminance reduced by the subtraction because the average luminance of the image is reduced by subtraction of the pixel value. At this time, the gain correction amount g is approximately g = (d1 / (d1-d2)) ² where the aperture diameter on the aperture open side is d1 and the aperture diameter on the aperture close side is d2.
It becomes. However, since the gain correction amount changes due to the optical performance of the imaging optical system as described above, it may be obtained experimentally in advance.

  Further, since the aperture diameter (opening state) is different between the aperture open side image data and the aperture close side image data, the peripheral light amount drop characteristics (signal level attenuation rate in the peripheral pixels) are different. For this reason, the gain correction processing includes processing for correcting the peripheral light amount drop characteristic so as to match the other image data. For example, the peripheral light amount drop characteristic is experimentally measured in advance for each of the aperture values, and the difference in the peripheral light amount drop is obtained for the two selected aperture values and applied to one of them. That's fine. In this way, by performing subtraction using an image combined with the peripheral light amount drop characteristic, an appropriate subtraction result can be obtained even in the peripheral pixels.

  In step S <b> 706, the CPU 101 causes the DSP 108 to convert the left-eye and right-eye image data for which gain correction has been performed into a predetermined binocular stereoscopic image data recording format. Then, the CPU 101 records the binocular stereoscopic image data in the obtained recording format on the recording medium 111, and completes the stereoscopic image capturing process.

  If it is determined in step S702 that the aperture-open side image data includes a saturated pixel, the CPU 101 records the aperture-open side image data in the DSP 108 in advance in step S707. Convert to format. Then, the CPU 101 records the binocular stereoscopic image data in the obtained recording format on the recording medium 111, and completes the stereoscopic image processing.

  If there is a saturated pixel in the aperture open side image data in S702, binocular stereoscopic image data is generated using the aperture open side image data without capturing the aperture closed side image data. Although described as a thing, implementation of this invention is not restricted to this. For example, an exposure time that eliminates saturated pixels may be set, and the aperture opening side image data shooting process may be executed again using the exposure time.

  As described above, the imaging apparatus of the present embodiment can expand the parallax in the binocular stereoscopic image obtained by the light flux that has passed through different areas of the exit pupil. Specifically, the imaging apparatus includes an imaging optical system including a diaphragm and a plurality of pixels each including a left-eye and right-eye light receiving elements, and each of the left-eye and right-eye light receiving elements is left. An imaging device that outputs images for the eyes and the right eye. Then, for each of the left-eye and right-eye images, the aperture opening is in the closed state from the pixel value of the first image output by the image sensor when the aperture opening is in the open state. In this case, the third image obtained by subtracting the pixel value of the second image output by the image sensor is output as binocular stereoscopic image data. At this time, as the third image data, for example, the left-eye image obtained from the left-eye pixel 501 and the right-eye image obtained from the right-eye pixel 502 in FIG. 5 are separately recorded as one scene. . In the playback device, it is possible to view the captured image stereoscopically by reproducing the left-eye image recorded as the one scene with the left eye and the right-eye image with the right eye. is there.

  In this way, even with a monocular imaging device, an image having the same parallax as that of a binocular imaging device can be acquired, so an image with effectively adjusted parallax is presented to the user. can do.

[Modification]
In the above-described embodiment, the operation only in the parallax enhancement mode has been described. However, an example in which the digital camera 100 includes a normal shooting mode and a mode in which different parallaxes can be set for a binocular stereoscopic image to be shot. explain.

<Shooting process>
A specific process of the photographing process of the digital camera 100 according to the present modification having the same configuration as that of the first embodiment will be described with reference to the flowchart of FIG. The processing corresponding to the flowchart can be realized by the CPU 101 reading out a corresponding processing program stored in, for example, the ROM 102, developing it in the RAM 103, and executing it. The main photographing process is described as being started when a photographing instruction is input by a user operating a shutter button in a state where the digital camera 100 is set to a photographing mode.

  In step S <b> 801, the CPU 101 determines whether the current shooting mode of the digital camera 100 is set to a mode for shooting binocular stereoscopic image data. Specifically, the CPU 101 determines whether the current shooting mode setting stored in the RAM 103 corresponds to any of the parallax enhancement mode, the parallax reduction mode, and the normal parallax mode, for example. The outline of the mode for capturing image data for binocular stereoscopic vision described in the present embodiment is as follows.

Normal parallax mode: Binocular stereoscopic images obtained with the aperture fully open (eg, F2.0)
Mode for recording as image data / Parallax reduction mode: Records binocular stereoscopic image data with a smaller parallax than the normal parallax mode
Mode (for example, F16, which is 6 stops from the fully open position, and 7 stops)
Subtract the image data obtained in F22)
-Parallax enhancement mode: Records binocular stereoscopic image data with a larger parallax than the normal parallax mode
Mode (for example, full aperture F2.0, one stop from full aperture)
(Subtract the image data obtained in F2.8)

  When the CPU 101 determines that the current shooting mode of the digital camera 100 is set to a mode for shooting binocular stereoscopic image data, the CPU 101 moves the process to S803, and is set to another shooting mode. If it is determined that there is, the process proceeds to S802.

  In step S <b> 802, the CPU 101 performs normal AE shooting according to the exposure amount determined by the photometry circuit 109, converts the image data obtained by the DSP 108 into a recording format, records the recording data on the recording medium 111, and completes the main shooting process. .

  If the CPU 101 determines that the mode for capturing binocular stereoscopic image data is set, the CPU 101 determines in step S803 whether the current shooting mode is a parallax enhancement mode. If the CPU 101 determines that the current shooting mode is the parallax enhancement mode, the process proceeds to S804. If the CPU 101 determines that the current shooting mode is not the parallax enhancement mode, the process proceeds to S805.

  In step S804, the CPU 101 performs shooting for F2.0 and F2.8 similarly to the stereoscopic image shooting process of the above-described embodiment, and subtracts the image data for the left eye and the right eye for recording. Binocular stereoscopic image data is acquired and recorded in the recording medium 111.

  In step S805, the CPU 101 determines whether the current shooting mode is the parallax reduction mode. If the CPU 101 determines that the current shooting mode is the parallax reduction mode, the process proceeds to S806. If the CPU 101 determines that the current shooting mode is not the parallax reduction mode, the process proceeds to S807.

  In step S806, the CPU 101 captures F16 and F22 in the same manner as the stereoscopic image capturing process of the above-described embodiment, and subtracts the image data for the left eye and the right eye to record the binocular stereoscopic images for recording. Image data is acquired and recorded in the recording medium 111.

  When determining in step S805 that the current shooting mode is not the parallax reduction mode, the CPU 101 determines that the current shooting mode is the normal parallax mode. In step S <b> 807, the CPU 101 performs shooting at F2.0, causes the DSP 108 to convert the obtained image data for the left eye and right eye into image data for binocular stereoscopic viewing in a recording format, and records the data in the recording medium 111. To do.

  Although the present embodiment has been described on the assumption that binocular stereoscopic image data can be captured by switching parallax in three stages depending on the shooting mode, the present invention is not limited to this. For example, whether the parallax target value selected by the user can be dealt with by changing the aperture value or whether a combination of two different aperture values is selected and the pixel values of the two obtained images are subtracted. Judgment processing may be performed after determination.

  In this way, when shooting binocular stereoscopic image data, the user can select a desired parallax according to the scene. Further, the degree of freedom in selecting parallax can be improved without being limited to the exit pupil of the imaging optical system.

  In the above-described embodiment and the modification, the parallax in the binocular stereoscopic image is expanded by subtracting the left-eye image and the right-eye image obtained by photographing with two different aperture states. Although the method to do was demonstrated, implementation of this invention is not restricted to this.

  When a plurality of light receiving elements are provided in a unit pixel as in the present invention, a readout circuit is required for each of the light receiving elements. For this reason, if the installation space for the readout circuit is secured, it is conceivable that the light receiving area of the light receiving element is limited, for example, as shown in FIG. However, as shown in FIG. 9A, in the imaging device, the above processing is performed using image data obtained by narrowing the occupation range of the left-eye and right-eye light-receiving elements 901 and 902 in the vertical direction. By doing so, the same effect can be obtained. That is, even if the light flux that can be received by the unit pixel is limited, the exit pupil area corresponding to the difference between the pixel signals obtained at different apertures is reduced only in the vertical direction as shown in FIG. 9B. Therefore, the position of the center of gravity of the region does not change regardless of the light receiving area. That is, it is possible to further expand the parallax as in the above-described embodiment and modification.

  In the example of FIG. 9A, transfer MOS transistors 903 and 904 and reset MOS transistors 905 and 906 connected to the photodiodes that are the respective light receiving elements are arranged in order to efficiently arrange the components of the light receiving elements. And are arranged. At this time, for members other than the light receiving elements 901 and 902, it is assumed that a metal light shielding layer is disposed immediately above so that light does not enter.

  Further, for example, by using a transmissive liquid crystal or the like as described above, the region passing through the exit pupil is controlled by the aperture without subtracting the pixel value of the image data photographed at two different aperture values. It is also possible to do. Specifically, for example, the parallax can be easily extended by using a diaphragm as shown in FIG. 10A, or the parallax can be easily reduced by using a diaphragm as shown in FIG.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (12)

An image sensor that includes light-receiving elements for the left eye and the right eye , converts light beams that have passed through the same imaging optical system having a diaphragm, and outputs images for the left eye and the right eye;
And control means for controlling the pre Kishibo Rino opening so that parallax is changed in the image for the left eye and right eye,
Generating means for generating a binocular stereoscopic image using the left-eye and right-eye images output by the imaging device;
The generating means calculates the pixel values of the left-eye and right-eye images captured for each of the first state and the second state in which the parallax in the left-eye and right-eye images is different between the states. An image pickup apparatus that generates the binocular stereoscopic image in which the parallax is expanded from the left-eye image and the right-eye image in any state by performing subtraction . The second state is a state where the parallax in the left-eye and right-eye images is smaller than the first state,
The generating means uses each pixel value of the left-eye image and the right-eye image captured in the first state based on the pixel value of each of the left-eye image and the right-eye image captured in the first state. The imaging apparatus according to claim 1, wherein the image for binocular stereoscopic vision is output by subtracting a value. The imaging device receives light beams that have passed through different regions of the exit pupil of the imaging optical system by the left-eye and right-eye light-receiving elements, and outputs left- eye and right-eye images, respectively. The imaging apparatus according to claim 1 or 2.   4. The exposure time according to the image captured in the first state and the exposure time according to the image captured in the second state have the same length. 5. The imaging apparatus according to item 1.   The generation means includes a light amount attenuation rate of peripheral pixels generated by the diaphragm in an image captured in the first state and a light amount attenuation rate of peripheral pixels generated in the image captured in the second state. 5. The binocular stereoscopic image is generated after correcting the attenuation rate of the light quantity of peripheral pixels of at least one of the images so as to be the same. The imaging device according to item.   The imaging device according to claim 1, wherein the generation unit compensates an average luminance to a predetermined luminance value when generating the binocular stereoscopic image.   When the image picked up in the first state includes a saturated pixel, the generating means converts the left-eye and right-eye images picked up in the first state into the binocular stereoscopic view. The image pickup apparatus according to claim 1, wherein the image pickup apparatus outputs the image as an image. A setting unit for setting a parallax generated by the binocular stereoscopic image output by the generation unit;
Wherein, the larger the parallax set by the setting means, the first state of the left eye and the image for the left eye and the right eye of the second state the parallax of the image for the right eye 8. The imaging apparatus according to claim 1, wherein an aperture of the diaphragm in the first state and the second state is controlled so that a difference from the parallax in the second state is small. .   The imaging apparatus according to claim 8, wherein the setting unit sets a parallax generated by the binocular stereoscopic image according to a shooting mode set by a user. Control method for an image pickup apparatus including left-eye and right-eye light-receiving elements, and an image sensor that converts a light beam that has passed through the same imaging optical system having a diaphragm and outputs left-eye and right-eye images Because
Control means, and a control step of controlling the pre Kishibo Rino opening such that the parallax of the image of the left-eye and right-eye changes,
A generating unit that generates a binocular stereoscopic image using the left-eye and right-eye images output by the imaging device; and
In the generating step, the generating means includes pixel values of each of the left-eye and right-eye images captured in each of the first state and the second state in which parallaxes in the left-eye and right-eye images are different. A method for controlling an imaging apparatus, wherein the binocular stereoscopic image in which the parallax is expanded from the left-eye image and the right-eye image in any state is generated by subtracting between the states .   The program for functioning a computer as each means of the imaging device of any one of Claims 1 thru | or 9.   A computer-readable recording medium on which the program according to claim 11 is recorded.

Images