JP2012124650A - Imaging apparatus, and imaging method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To shorten time lag of a stereoscopic image output in an imaging apparatus for continuously producing stereoscopic images through prescribed image processing for images obtained in a pupil division system.SOLUTION: Calculating distance information from image data A and B of an n-th (n=1,2,3..) frame until a release button 19 is fully pushed, producing virtual image data B' by applying the distance information corresponding to the image data A and B of the n-th frame to the image data A of an n+1-th frame which are obtained from a CCD 16, and producing three dimentional image data of the n-th frame based on the image data A of the n+1-th frame and the image data B' to display it as a through image on an LCD 30 are repeated. Thus, the through image is displayed from a viewpoint image of a posterior frame using the distance information calculated for a previous frame, so that delay in displaying the latest through image can be prevented from being caused by a state of waiting calculation of the distance information.

Description

  The present invention relates to an image pickup apparatus and an image pickup method in which subject images that have passed through different regions in two directions of a photographing lens are respectively formed on an image pickup device and different viewpoint images are acquired. In particular, the present invention relates to an imaging apparatus and an imaging method for displaying a shooting angle of view of an image having parallax as a through image in a three-dimensional manner.

  2. Description of the Related Art Conventionally, there are stereoscopic imaging apparatuses that acquire different viewpoint images by forming subject images that have passed through different regions in two directions of a photographing lens, respectively, on imaging elements.

  The optical system shown in FIG. 13 divides a subject image that has passed through different regions in the left and right directions of the main lens 1 and the relay lens 2 by a mirror 4 and supplies the image to the image sensors 7 and 8 via the imaging lenses 5 and 6, respectively. An image is formed.

  FIGS. 14A to 14C are diagrams illustrating separation states of images formed on the image sensor due to differences in the front pin, focus (best focus), and rear pin, respectively. In FIG. 14, the mirror 4 shown in FIG. 13 is omitted in order to compare the difference in separation due to focus.

  As shown in FIG. 14B, the focused image of the pupil-divided images is formed (matched) at the same position on the image sensor, but FIGS. 14A and 14C. As shown in FIG. 5, the images to be the front pin and the rear pin are formed (separated) at different positions on the image sensor.

  Therefore, by acquiring the subject image that is pupil-divided in the left-right direction via the imaging elements 7 and 8, it is possible to acquire a left viewpoint image and a right viewpoint image (3D image) having different viewpoints according to the subject distance. .

  According to Patent Document 1, the electronic camera includes an imaging unit, a light amount distribution detection unit, an image shift amount detection unit, and an image processing unit. The imaging unit photoelectrically converts a subject image by the photographing optical system to generate photographed image data. The light quantity distribution detection unit detects the light quantity distribution by the light flux from the subject passing through different optical paths. The image shift amount detection unit detects image shift amounts at a plurality of locations in the photographing screen based on the light amount distribution. The image processing unit generates stereogram image data by changing the positional relationship of the subject in the captured image data in the horizontal direction based on the image shift amounts at a plurality of locations. Further, based on the input from the input unit, the stereogram image data is corrected by changing the positional relationship of the subject in the stereogram image data in the horizontal direction. Since the stereogram image data can be corrected, it is possible to obtain a natural stereogram image with less discomfort for the user.

  In Patent Document 2, pattern matching is performed on the left-eye image and the right-eye image captured by the left and right cameras with respect to the stereo image of either the left-eye image or the right-eye image as a reference. Then, a matching image for each pixel is specified, an intermediate position image is obtained by interpolating the inter-pixel distance between the matching images for each left and right pixel, and the right is obtained by extrapolating the inter-pixel distance between the matching images for each left and right pixel. A multi-view image creation method for obtaining an outside image and a left outside image is disclosed.

  In Patent Document 3, the stereo matching processing unit (object detection means) 43 is processed by the image processing units 4A and 4B, and the above-described space setting unit in the two image data A and B stored in the memory unit 31. One or more corresponding points (objects) corresponding to each other with respect to the search space set by 42 are detected. The distance calculation unit (position calculation unit) 44 calculates the three-dimensional coordinate value (position information) of the corresponding point detected by the stereo matching processing unit 83 described above. The configurations of the photographing lenses 21A2 and 21B2 of the photographing unit 20A and the photographing unit 20B are different. The photographing lens 21A2 includes a zoom lens and a zoom lens driving unit (driving unit) (not shown) that drives the zoom lens. The photographing lens 21B2 includes: A fixed focus lens having a field angle equal to the wide angle end of the zoom lens of the photographing lens 21A2 is provided. With such a configuration, the cost is reduced.

  Patent Documents 4 to 6 are examples of a three-dimensional image generation technique using a single optical system. For example, in Patent Document 4, a large number of pixels are arranged on the same imaging surface, and in a solid-state imaging device that generates an image signal by photoelectrically converting a subject image formed on the imaging surface, Disclosed is a solid-state imaging device that is divided into groups and has different light-receiving incidence angles of pixels in each group.

  Patent Documents 7 to 11 are examples of a method for searching corresponding points between different viewpoint images, a technique for acquiring depth information by stereo matching, and a three-dimensional image generation technique using a two-dimensional image and distance information (depth information). .

JP 2007-104248 A JP 2009-124308 A JP2008-92007, paragraphs 0047-0048, 0071 JP2003-7994 Japanese Patent Laid-Open No. 2001-12916 JP 2001-016611 Japanese Unexamined Patent Publication No. 08-331607 JP 2008-141666 A JP 2009-14445 JP 2008-116309 Japanese Unexamined Patent Publication No. 2000-102040

  As illustrated in FIG. 11, the display of a through image (live view) in a normal compound eye camera that displays left and right viewpoint images with the same image quality as a stereoscopic image is based on the left and right viewpoint images captured for each frame. It is only necessary to repeat displaying the three-dimensional image.

  On the other hand, as illustrated in Patent Document 3, the left and right imaging units such as the left and right imaging units have different shooting pixel numbers, or one imaging unit can acquire color information but the other imaging unit cannot acquire color information. In the case of a compound eye camera that cannot generate a stereoscopic image for display immediately from the left and right viewpoint images, such as when the specifications of the viewpoint are different or when the resolution of the viewpoint image is different from the resolution of the monitor, in addition to calculating the distance information, For example, it is necessary to repeatedly generate and display a three-dimensional image from color information of an image of an imaging unit having a large number of pixels and an imaging unit capable of acquiring color information.

  The same applies to a camera that acquires a three-dimensional image from a single optical system as in Patent Documents 4 to 6.

  As illustrated in FIG. 12, in such a camera, since it takes time to generate a three-dimensional image, the time lag from the viewpoint image capturing to the through image display is increased as compared with a normal camera. Therefore, it is difficult to use because it hinders confirmation of the angle of view by the displayed through image.

  Further, in the pupil division type stereoscopic imaging device, since the baseline length is small, the stereoscopic effect of the image tends to be weak. Although it is possible to adjust the amount of parallax by performing image processing such as stereo matching or extrapolation or interpolation processing of parallax information on the captured image, the display of a through image is delayed due to an increase in image processing time. In recent years, cameras without a viewfinder have become mainstream, and if the through image display is delayed, there is a possibility of missing a photo opportunity, which is difficult to use.

  The present invention provides a technique for shortening the time lag of stereoscopic image output in an imaging apparatus that continuously generates a stereoscopic image through predetermined image processing on an image obtained by the pupil division method.

  The present invention provides a single imaging optical system and an imaging unit in which subject images that have passed through a first region and a second region having different predetermined directions of the imaging optical system are divided into pupils and formed respectively. An imaging unit capable of photoelectrically converting each of the subject images that have passed through the first area and the second area and continuously outputting a set of viewpoint images including the first image and the second image; An imaging control unit that controls the imaging unit so that a set of viewpoint images is output at the first time point and the second time point, which are arbitrary two time points before and after, and the viewpoint output from the imaging unit at the first time point A parallax information calculation unit that performs stereo matching based on a set of images to calculate parallax information, and generates or generates virtual parallax information by extrapolating or interpolating the parallax information corresponding to the first time point. The first image and / or the first output at time 2 Provided is an imaging apparatus including a stereoscopic image generation unit that generates a set of viewpoint images having desired parallax based on two images and generates a stereoscopic image from the set of images having desired parallax.

  Here, the parallax information includes subject distance information or a parallax map.

  The imaging control unit controls the imaging unit so that a set of viewpoint images is sequentially output at each of a plurality of different second time points after the first time point, and the stereoscopic image generating unit is set to the first time point. Virtual parallax information is generated by extrapolating or interpolating corresponding parallax information, and has a desired parallax based on the virtual parallax information and the first image and / or the second image respectively output at the second time point A plurality of sets of viewpoint images are generated, and a plurality of stereoscopic images are sequentially generated from the plurality of sets of viewpoint images having a desired parallax.

  The imaging control unit controls the imaging unit so that a set of viewpoint images is output at a second time point when the parallax information calculation process corresponding to the first time point by the parallax information calculation unit is completed by a predetermined ratio.

  The disparity information calculation unit includes a plurality of different processing systems capable of independently calculating a plurality of pieces of disparity information corresponding to different sets of viewpoint images, and corresponds to a plurality of different first time points. Each piece of parallax information is calculated by a different processing system.

  In response to the input of a predetermined recording instruction, the imaging apparatus extrapolates or interpolates disparity information corresponding to the first time point to generate virtual disparity information, and outputs the virtual disparity information and the first time point. A recording unit that generates a set of viewpoint images having a desired parallax based on the first image and / or the second image, and records reproduction information of a stereoscopic image based on the set of viewpoint images having the desired parallax Is provided.

  A display unit for displaying a stereoscopic image generated by the stereoscopic image generation unit is provided.

  The resolution of the second image is lower than the resolution of the first image and lower than the display resolution of the display unit.

  The first image includes luminance information and color information, and the second image includes only luminance information.

  The stereoscopic processing unit includes a blur processing unit that blurs the parallax information, and the stereoscopic image generation unit generates virtual parallax information by extrapolating or interpolating the parallax information corresponding to the first time point blurred by the blur processing unit. Based on the information and the first image and / or the second image output at the second time point, a set of viewpoint images having a desired parallax is generated, and a stereoscopic image is generated from the set of images having the desired parallax .

  The desired amount of parallax includes a specified value, a fixed value, or a value corresponding to the distance of the subject distance input from a predetermined operation unit.

  The present invention provides a single imaging optical system and an imaging unit in which subject images that have passed through a first region and a second region having different predetermined directions of the imaging optical system are divided into pupils and formed respectively. An imaging unit capable of photoelectrically converting each of the subject images that have passed through the first area and the second area and continuously outputting a set of viewpoint images including the first image and the second image; An imaging control unit that controls the imaging unit so that a set of viewpoint images is output at the first time point and the second time point, which are arbitrary two time points before and after, and the viewpoint output from the imaging unit at the first time point A parallax information calculation unit that performs stereo matching based on a set of images and calculates parallax information; and applying the parallax information corresponding to the first time point to the first image output at the second time point, A virtual second image having a parallax between the second time points To provide an imaging apparatus and a stereoscopic image generation unit for generating stereoscopic image with the first image output from the virtual second image.

  The imaging control unit controls the imaging unit so that a set of viewpoint images is sequentially output at each of a plurality of different second time points after the first time point, and the stereoscopic image generating unit includes a plurality of second image points A plurality of virtual second images having parallax with the first image by applying disparity information corresponding to the first time point to a plurality of first images included in a set of a plurality of viewpoint images output at each time point And a plurality of stereoscopic images are sequentially generated from a plurality of virtual second images corresponding to the first images respectively output at a plurality of second time points.

  The imaging control unit controls the imaging unit so that a set of viewpoint images is output at a second time point when the parallax information calculation process corresponding to the first time point by the parallax information calculation unit is completed by a predetermined ratio.

  The disparity information calculation unit includes a plurality of different processing systems capable of independently calculating a plurality of pieces of disparity information corresponding to different sets of viewpoint images, and corresponds to a plurality of different first time points. Each piece of parallax information is calculated by a different processing system.

  In response to the input of a predetermined recording instruction, the imaging apparatus applies the disparity information corresponding to the first time point to the first image output at the first time point, thereby reducing the parallax between the first image and the first image. A recording unit configured to generate a virtual second image and record reproduction information of a stereoscopic image based on the first image and the virtual second image output at the first time point.

  A display unit for displaying a stereoscopic image generated by the stereoscopic image generation unit is provided.

  The resolution of the second image is lower than the resolution of the first image and lower than the display resolution of the display unit.

  The first image includes luminance information and color information, and the second image includes only luminance information.

  A blur processing unit that blurs the disparity information, and the stereoscopic image generation unit applies the disparity information corresponding to the first time point blurred by the blur processing unit to the first image output at the second time point. A virtual second image having parallax with one image is generated, and a stereoscopic image is generated from the first image and the virtual second image output at the second time point.

  The stereoscopic image generation unit generates virtual parallax information by extrapolating or interpolating the parallax information corresponding to the first time point, and generates the virtual parallax information and the first image and / or the second image output at the second time point. A set of viewpoint images having a desired parallax is generated based on the image, and a stereoscopic image is generated from the set of images having the desired parallax.

  In this way, the parallax of the stereoscopic image can be adjusted arbitrarily.

  The desired amount of parallax includes a specified value, a fixed value, or a value corresponding to the distance of the subject distance input from a predetermined operation unit.

  The present invention provides a single imaging optical system and an imaging unit in which subject images that have passed through a first region and a second region having different predetermined directions of the imaging optical system are divided into pupils and formed respectively. An imaging apparatus comprising: an imaging unit capable of photoelectrically converting subject images that have passed through the first area and the second area, respectively, and continuously outputting a set of viewpoint images including the first image and the second image Is a step of controlling the image capturing unit so that a set of viewpoint images is output at a first time point and a second time point, which are arbitrary two time points that fluctuate in time, and output from the image capturing unit at the first time point A step of calculating parallax information by performing stereo matching based on the set of viewpoint images thus generated, extrapolating or interpolating the parallax information corresponding to the first time point to generate virtual parallax information, The first image output at time 2 and / Or on the basis of the second image to generate a set of viewpoint images having a desired parallax, to provide an imaging method of performing, and generating a stereoscopic image from a set of images having a desired parallax.

  The present invention provides a single imaging optical system and an imaging unit in which subject images that have passed through a first region and a second region having different predetermined directions of the imaging optical system are divided into pupils and formed respectively. An imaging apparatus comprising: an imaging unit capable of photoelectrically converting subject images that have passed through the first area and the second area, respectively, and continuously outputting a set of viewpoint images including the first image and the second image Is a step of controlling the image capturing unit so that a set of viewpoint images is output at a first time point and a second time point, which are arbitrary two time points that fluctuate in time, and output from the image capturing unit at the first time point A step of calculating parallax information by performing stereo matching based on the set of viewpoint images, and applying the parallax information corresponding to the first time point to the first image output at the second time point, A virtual second image having parallax between Generating a stereoscopic image from the first image output at 2 from the virtual second image, to provide an imaging method of execution.

  According to the present invention, a stereoscopic image is generated from the disparity information calculated at the first time point and the viewpoint image output at the second time point after the first time point. In this way, a stereoscopic image can be immediately generated based on the viewpoint image acquired at the second time point later, and the disparity information calculated at the first time point and the viewpoint image output at the first time point are used. A new stereoscopic image can be displayed faster than generating a stereoscopic image.

Block diagram of a camera according to the first embodiment The figure which shows the structural example of the pupil division parallax image acquisition image pick-up element CCD. The figure which showed 1 pixel of 1st, 2nd pixel each 3 is an enlarged view of the main part of FIG. Timing chart of through image display processing according to the first embodiment Timing chart of through image display processing according to the second embodiment Timing chart of through image display processing according to the third embodiment Block diagram of a camera according to the fourth embodiment Timing chart of through image display processing according to the fourth embodiment Timing chart of moving image recording process according to fifth embodiment Timing chart of live view display processing in a normal imaging device Timing chart of through image display processing in an imaging apparatus that performs distance calculation The figure which shows an example of the conventional monocular three-dimensional imaging device The figure which shows the separation state of the image imaged on an image sensor

<First Embodiment>
FIG. 1 is a block diagram showing an embodiment of a camera 1 according to the first embodiment.

  The camera 1 records captured images on a memory card 54, and the overall operation of the apparatus is controlled by a central processing unit (CPU) 40.

  The camera 1 is provided with an operation unit 38 such as a shutter button, a mode dial, a playback button, a MENU / OK key, a cross key, and a BACK key. A signal from the operation unit 38 is input to the CPU 40, and the CPU 40 controls each circuit of the camera 1 based on the input signal. For example, lens drive control, aperture drive control, shooting operation control, image processing control, image data control, Recording / reproduction control, display control of the liquid crystal monitor 30 for stereoscopic display, and the like are performed.

  The ROM 10 stores programs executed by the CPU 40 and various data necessary for control, pixel defect information of the CCD 16, various constants / information relating to camera operation, and the like.

  The shutter button is an operation button for inputting an instruction to start shooting, and includes a two-stage stroke type switch having an S1 switch that is turned on when half-pressed and an S2 switch that is turned on when fully pressed. The mode dial is selection means for selecting one of an auto shooting mode for shooting a still image, a manual shooting mode, a scene position such as a person, a landscape, a night view, and a moving image mode for shooting a moving image.

  The playback button is a button for switching to a playback mode in which a still image or a moving image of a stereoscopic image (3D image) or a planar image (2D image) that has been recorded is displayed on the liquid crystal monitor 30. The MENU / OK key is an operation key that has both a function as a menu button for instructing to display a menu on the screen of the liquid crystal monitor 30 and a function as an OK button for instructing confirmation and execution of selection contents. It is. The cross key is an operation unit for inputting instructions in four directions, up, down, left, and right, and functions as a button (cursor moving operation means) for selecting an item from the menu screen or instructing selection of various setting items from each menu. To do. The up / down key of the cross key functions as a zoom switch for shooting or a playback zoom switch in playback mode, and the left / right key functions as a frame advance (forward / reverse feed) button in playback mode. . The BACK key is used to delete a desired object such as a selection item, cancel an instruction content, or return to the previous operation state.

  In the photographing mode, the image light indicating the subject is a solid-state image sensor (hereinafter referred to as “CCD”, which is a phase difference image sensor capable of acquiring a pupil-divided parallax image via the imaging lens 12 and the diaphragm 14, but may be a CMOS or the like. Yes) The image is formed on 16 light receiving surfaces. The imaging lens 12 is driven by a lens driving unit 36 controlled by the CPU 40, and focus control, zoom (focal length) control, and the like are performed. The diaphragm 14 is composed of, for example, five diaphragm blades, and is driven by a diaphragm driving unit 33 controlled by the CPU 40. For example, the diaphragm value (F value) F2.8 to F11 is controlled in five stages in increments of 1AV. The

  Further, the CPU 40 controls the diaphragm 14 via the diaphragm driving unit 33, and performs charge accumulation time (shutter speed) in the CCD 16, image signal readout control from the CCD 16, and the like via the CCD control unit 32.

<CCD configuration example>
FIG. 2 is a diagram illustrating a configuration example of the CCD 16.

  As shown in FIG. 2A, the CCD 16 has odd-line pixels and even-line pixels arranged in a matrix, and photoelectric conversion is performed on the pixels of these two lines. Further, the image signals for the two planes can be read out independently. The plurality of light receiving elements corresponding to each pixel group form an effective pixel region for obtaining an effective imaging signal and an optical black region (hereinafter referred to as “OB region”) for obtaining a black level reference signal. The OB region is actually formed so as to surround the effective pixel region.

  As shown in FIG. 2B, the odd-numbered lines (1, 3, 5,...) Of the CCD 16 include pixels having R (red), G (green), and B (blue) color filters. The GRGR... Pixel array lines and the BGBG... Pixel array lines are alternately provided. On the other hand, as shown in FIG. 2C, the pixels in the even lines (2, 4, 6,...) Are similar to the odd lines in the GRGR... Pixel array line and the BGBG. Are alternately arranged, and the pixels are shifted in the line direction by a half pitch with respect to the pixels of the even-numbered lines.

  The arrangement region of the first pixel and the second pixel constituting the image signal for two surfaces may or may not be the same. For example, the first pixel exists over the entire effective pixel area, but the existence area of the second pixel may be only a specific area such as an AF area. The second pixels may be densely arranged over the entire effective pixel area or within a specific area, or may be roughly arranged.

  Specifically, as shown in FIGS. 2D and 2E, the second pixel number may be lower than the first pixel number. 2D and 2E show the first pixel in the white portion and the second pixel in the black portion. The color filter in the black portion may be RGB (FIG. 2D) or only G (FIG. 2E) that takes luminance information. In the case of FIG. 2D, there is an advantage that it is not necessary to change the arrangement of the RGB color filters between the first pixel and the second pixel. In the case of FIG. 2E, the information of each second pixel is defocused. There is an advantage that can be used for quantity detection.

  3 is a diagram showing each of the imaging lens 12, the diaphragm 14, and the first and second pixels of the CCD 16 in FIG. 2A, and FIG. 4 is an enlarged view of the main part of FIG.

  As shown in FIG. 4A, a light beam passing through the exit pupil is incident on a normal CCD pixel (photodiode PD) through the microlens L without being restricted.

  On the other hand, as shown in FIG. 4B, a light shielding member 16A is formed on the first pixel and the second pixel of the CCD 16. When imaging with the camera 1 placed horizontally, the light shielding member 16A shields the right half or the left half of the light receiving surface of the first pixel and the second pixel (photodiode PD). Alternatively, when imaging is performed with the camera 1 placed vertically, the light shielding member 16A shields the upper half or the lower half of the light receiving surface of the first pixel and the second pixel (photodiode PD). An opening 16B of the light shielding member 16A is located at a position deviated by a predetermined amount Δ from the optical axis Z of the microlens L in the right, left, upper, or lower direction (for example, leftward from the optical axis in FIG. 4B). Is provided. The light beam passes through the opening 16B and reaches the light receiving surface of the photodiode PD. That is, the light shielding member 16A functions as a pupil division member.

  Note that the first pixel and the second pixel are different in regions (right half / left half or upper half / lower half) where the light flux is restricted by the light shielding member 16A. For example, if the left half of the luminous flux is restricted at the first pixel and the right half of the luminous flux is restricted at the second pixel, the right viewpoint image is obtained from the first pixel and the left viewpoint image is obtained from the second pixel. . Alternatively, when the upper half of the luminous flux is restricted at the first pixel and the lower half of the luminous flux is restricted at the second pixel, the lower viewpoint image is obtained from the first pixel and the upper viewpoint image is obtained from the second pixel. .

  Therefore, as shown in FIG. 14, the outputs of the first pixel and the second pixel are out of phase or in phase with each other depending on the state of the rear pin, the in-focus state, and the front pin. Since the phase difference between the output signals of the first pixel and the second pixel corresponds to the defocus amount of the imaging lens 12, AF control of the imaging lens 12 can be performed by detecting this phase difference (phase difference AF). .

  The CCD 16 having the above configuration is configured such that the first pixel and the second pixel have different regions (right half and left half) where the light flux is limited by the light shielding member 16A. Without limiting to the light shielding member 16A, the microlens L and the photodiode PD may be shifted relative to each other in the left-right direction, and the light flux incident on the photodiode PD may be limited by the shifting direction. By providing one microlens for two pixels (the first pixel and the second pixel), the light beam incident on each pixel may be limited, or the pupil is divided by a mirror (for example, FIG. 13) But you can. In short, the application range of the present invention is a camera that acquires a phase difference image by pupil division.

  Returning to FIG. 1, the signal charge accumulated in the CCD 16 is read out as a voltage signal corresponding to the signal charge based on the readout signal applied from the CCD controller 32. The voltage signal read from the CCD 16 is applied to the analog signal processing unit 18 where the R, G, and B signals for each pixel are sampled and held, amplified, and then applied to the A / D converter 20. The A / D converter 20 converts R, G, and B signals that are sequentially input into digital R, G, and B signals and outputs them to the image input controller 22.

  The digital signal processing unit 24 performs predetermined processing such as offset control, gain control processing including white balance correction and sensitivity correction, gamma correction processing, YC processing, etc., on the digital image signal input via the image input controller 22. Perform signal processing.

  Here, as shown in FIGS. 2B and 2C, the first image data read from the first pixel of the odd line of the CCD 16 is processed as the left viewpoint image data and read from the second pixel of the even line. The second image data to be processed is processed as right viewpoint image data. Similarly, in the case of FIGS. 2D and 2E, the first image data read from the first pixel is processed as the left viewpoint image data, and the second image data read from the second pixel of the even line is Processed as right viewpoint image data. The first image data is not necessarily left viewpoint image data, and the second image data is not necessarily right viewpoint image data, and vice versa.

  The left viewpoint image data and right viewpoint image data (3D image data) processed by the digital signal processing unit 24 are input to the VRAM 50. The VRAM 50 includes an A area and a B area each storing 3D image data representing a 3D image for one frame. In the VRAM 50, 3D image data representing a 3D image for one frame is rewritten alternately in the A area and the B area. The written 3D image data is read from an area other than the area in which the 3D image data is rewritten in the A area and the B area of the VRAM 50. The 3D image data read from the VRAM 50 is encoded by the video encoder 28 and output to a stereoscopic display liquid crystal monitor (LCD) 30 provided on the back of the camera, whereby a 3D subject image is displayed on the liquid crystal monitor 30. Displayed on the display screen.

  The liquid crystal monitor 30 is a stereoscopic display unit that can display stereoscopic images (left viewpoint image and right viewpoint image) as directional images having predetermined directivities by a parallax barrier, but is not limited thereto, and is not limited thereto. The left viewpoint image and the right viewpoint image may be viewed separately by using a lens, or by wearing dedicated glasses such as polarized glasses or liquid crystal shutter glasses.

  In addition, when the shutter button of the operation unit 38 is first pressed (half-pressed), the CPU 40 starts the AF operation and the AE operation, and the focus lens in the imaging lens 12 is aligned via the lens driving unit 36. Control to come to the focal position. The image data output from the A / D converter 20 when the shutter button is half-pressed is taken into the AE detection unit 44.

  The AE detection unit 44 integrates the G signals of the entire screen, or integrates the G signals with different weights in the central and peripheral portions of the screen, and outputs the integrated value to the CPU 40. The CPU 40 calculates the brightness of the subject (shooting Ev value) from the integrated value input from the AE detection unit 44, and the aperture value of the diaphragm 14 and the electronic shutter (shutter) of the CCD 16 that can obtain appropriate exposure based on this shooting Ev value. (Speed) is determined according to a predetermined program diagram, the aperture 14 is controlled via the aperture drive unit 33 based on the determined aperture value (normal aperture control), and CCD control is performed based on the determined shutter speed. The charge accumulation time in the CCD 16 is controlled via the unit 32. Note that the brightness of the subject may be calculated based on an external photometric sensor.

  Here, the predetermined program diagram corresponds to a combination of the aperture value of the aperture 14 and the shutter speed of the CCD 16 or a combination of these and the imaging sensitivity (ISO sensitivity) corresponding to the brightness of the subject (imaging EV value). The following shooting (exposure) conditions are designed. By photographing under the photographing conditions determined according to the program diagram, it is possible to photograph the first image and the second image having a desired parallax regardless of the brightness of the subject.

  For example, the predetermined program diagram has a constant F value of 1.4 (AV = 1), and when the shooting EV value is from 7 to 12, only the shutter speed is set to 1/60 seconds (TV) according to the shooting EV value. = 6) to 1/2000 (TV = 11). When the photographing EV value is smaller than 7 (when dark), the ISO sensitivity is increased from 100 to 200,400,800, every time the photographing EV value becomes 1 EV with the F value = 1.4 and the shutter speed = 1/60 seconds. It is designed to be 1600,3200. That is, when the subject becomes dark, the aperture 14 is not opened, and the brightness is compensated by increasing the shutter speed and ISO sensitivity.

  Note that if the F value to be fixed is increased, the parallax is weakened, and if the F value to be fixed is decreased, the parallax is increased. Therefore, the F that is fixed according to the amount of parallax instructed by the user via the operation unit 38. Switch between values and program diagrams. For example, when weak parallax is instructed, program diagram A with F value = 5.6, when standard parallax is instructed, program diagram B with F value = 2.8, and when strong parallax is instructed, F value = Normal aperture control is performed along the program diagram C of 1.4.

  In this parallax priority program diagram, since the F value is fixed at a constant value, the first image and the second image having a desired parallax can be taken. However, if the shooting EV value is greater than 12 (when the shutter speed reaches the maximum value), the camera 1 is overexposed and cannot be shot. However, if a configuration that can automatically reduce the amount of light by adding an ND filter is added to the camera 1. Even if the shooting EV value is larger than 12, shooting is possible.

  The AF processing unit 42 is a part that performs contrast AF processing or phase difference AF processing. When performing contrast AF processing, high-frequency components of image data within a predetermined focus area (such as a rectangular area in the center of the screen) of at least one of left viewpoint image data and right viewpoint image data are extracted. The AF evaluation value indicating the in-focus state is calculated by integrating the high frequency component. The AF control is performed by controlling the focus lens in the imaging lens 12 so that the AF evaluation value is maximized. Further, when performing the phase difference AF processing, the phase difference between the image data corresponding to the first pixel and the second pixel in the predetermined focus area in the left viewpoint image data and the right viewpoint image data is detected, and this A defocus amount is obtained based on information indicating the phase difference. AF control is performed by controlling the focus lens in the imaging lens 12 so that the defocus amount becomes zero.

  When the AE operation and the AF operation are completed and the shutter button is pressed in the second stage (full press), it corresponds to the first pixel and the second pixel output from the A / D converter 20 in response to the press. The image data for two images, the left viewpoint image (first image) and the right viewpoint image (second image), are input from the image input controller 22 to the memory (SDRAM) 48 and temporarily stored.

  The two pieces of image data temporarily stored in the memory 48 are appropriately read out by the digital signal processing unit 24, where predetermined signals including generation processing (YC processing) of luminance data and color difference data of the image data are performed. Processing is performed. The YC processed image data (YC data) is stored in the memory 48 again. Subsequently, the two pieces of YC data are respectively output to the compression / decompression processing unit 26, and after a predetermined compression process such as JPEG (joint photographic experts group) is executed, the YC data is stored in the memory 48 again.

  A multi-picture file (MP file: a file in a format in which a plurality of images are connected) is generated from two pieces of YC data (compressed data) stored in the memory 48, and the MP file is generated by the media controller 52. It is read and recorded in the memory card 54.

  The defocus map creation unit 61 not only calculates the phase difference corresponding to the first pixel and the second pixel for each of the small areas included in the predetermined focus area, but substantially covers the entire effective pixel area. The calculation is performed for each of the plurality of small regions. The plurality of small areas that substantially cover the entire effective pixel area need not completely cover the entire effective pixel area, and may be densely or roughly arranged over the entire effective pixel area. That's fine. For example, each of the divided areas obtained by dividing the effective pixel area in a matrix by a predetermined unit (for example, 8 × 8 pixels), or less (for example, 1 × 1 pixel), or more (for example, 10 × 10 pixels). The phase difference is calculated. Alternatively, the phase difference is calculated for each divided region in a predetermined unit with a predetermined pitch (for example, one divided region or more or less) from the outer edge of the effective pixel region. In short, it is assumed that the phase difference is calculated over the entire effective pixel area, but it is not necessarily calculated for all of the small areas constituting the effective pixel area.

  The defocus map creation unit 61 obtains a defocus amount corresponding to each of the small areas based on the phase difference calculated for each of the small areas. A set of defocus amounts corresponding to each of the small areas obtained over the entire effective pixel area is referred to as a defocus map. The defocus map creation unit 61 has a volatile storage medium such as a RAM, and temporarily stores the obtained defocus map. The defocus map creation unit 61 detects feature points and corresponding points between the viewpoint images in the stereo matching processing unit 83, and generates a defocus map based on a difference in position information between the feature points and the corresponding points. You may create it.

  The restoration filter storage unit 62 is configured by a nonvolatile storage medium such as a ROM, and the image height (distance from the image center, typically the optical axis center L of the imaging lens 12) of each small region in each viewpoint image. ) And a restoration filter corresponding to the defocus amount (or subject distance).

  The restoration unit 63 deconvolves the small area with the restoration filter selected for each small area of each viewpoint image, and restores the corresponding small area of the viewpoint image. By this signal processing, the blur of the optical system, in particular, the blur of the image corresponding to the peripheral pixels on the light receiving surface in which the unbalanced incidence of the light flux occurs is removed.

  The stereo matching processing unit (object detection means) 83 detects one or more corresponding points (objects) corresponding to each other between the two image data A and B stored in the memory 48 at the same timing. . As a processing method in the stereo matching processing unit 83, a known technique using a region-based method, a segment-based method, an isoluminance line method, or the like can be used. Further, the stereo matching may be passive or active. In addition, stereo matching between images having different numbers of pixels can be performed based on known techniques such as Patent Documents 3 and 7 to 11.

  The distance calculation unit 84 calculates the three-dimensional coordinate value (distance information or parallax map) of the corresponding point detected by the stereo matching processing unit 83 described above. As the calculation method in the distance calculation unit 84, a known technique for calculation based on the principle of triangulation can be used. Note that the stereo matching processing unit 83 and the distance calculation unit 84 may be configured by a program, or may be configured by an IC, an LSI, or the like. As will be described later, the distance calculation unit 84 may have a plurality of processing systems capable of simultaneously processing the distance calculation. Note that the distance information and the parallax map are technically equivalent, and the following processing on the distance information can also be performed on the parallax map.

  The acquisition pixel of 1st image data and the acquisition pixel of 2nd image data may be the same structures, and may not be so. The first image data has color information and luminance information, but the second image data may have only luminance information. Alternatively, the second pixel may be a monochrome CCD that can receive an infrared region, and active stereo matching using infrared light as in Patent Document 7 may be employed.

  In the following, in order to enable stereo matching, both the first image data and the second image data include at least luminance information. Furthermore, in order to add color information to the three-dimensional image, at least one of the first image data and the second image data includes color information. Here, for simplification of description, the first image data (image data A) includes luminance information and color information, and the second image data (image data B) includes only luminance information. With such a configuration, the manufacturing cost of the CCD 16 can be reduced. However, when displaying the through image in black and white, the color information of the image data A is not essential.

  The resolution of the image data A is equal to or higher than the display resolution of the LCD 30 and the resolution of the image data B is lower than the display resolution of the LCD 30. That is, in order to display a three-dimensional image on the LCD 30, it is necessary to calculate distance information, extrapolate or interpolate distance information, and adjust the size of the acquired image data. .

  The three-dimensional image processing unit 85 applies the distance information calculated by the distance calculation unit 84 to the image data A acquired from the CCD 16 to correspond to a virtual viewpoint position different from the viewpoint position of the image data A. Image data B ′ can be generated. This is the same as in Patent Document 2. Moreover, the three-dimensional image generation from the distance information can be performed based on known techniques such as Patent Documents 7 to 11.

  The deviation (parallax amount) between the viewpoint position of the image data A and the viewpoint position of the image data B ′ may be a fixed value or an arbitrary set value. For example, the three-dimensional image processing unit 85 generates virtual image data B ′ according to the parallax amount set from the operation unit 38. This is the same as Patent Document 1. Alternatively, virtual image data B ′ is generated according to the fixed amount of parallax stored in the ROM 10.

  Alternatively, the three-dimensional image processing unit 85 sets a parallax amount according to the distance of the subject distance in the AF area focused by the AF processing, and virtual image data B ′ according to the set parallax amount. Is generated. Specifically, if the distance to the focused subject is long, the three-dimensional image processing unit 85 increases the amount of parallax by extrapolation. That is, when the distance to the in-focus subject where the defocus amount is 0 is long and it is difficult to obtain a stereoscopic effect, the parallax amount can be increased to provide a stereoscopic effect. In order to allow the user to view a stereoscopic image with an appropriate amount of parallax, an upper limit for increasing the amount of parallax may be provided. Furthermore, if the distance to the focused subject is close, the three-dimensional image processing unit 85 can also reduce the amount of parallax by interpolation.

  In any case, the three-dimensional image processing unit 85 uses the three-dimensional coordinate values (distance information) of the corresponding points between the stereo-matched image data A and image data B so as to have a desired amount of parallax with the image data A. ) Is interpolated or extrapolated to obtain image data B ′ (intermediate position image, right outer image, or left outer image). Therefore, even when the base line length is reduced and the stereoscopic effect is weakened by the camera 1, a stereoscopic live view image in which the parallax amount is increased to a desired value can be output without delay.

  The three-dimensional image processing unit 85 generates three-dimensional image data from the image data A and the image data B ′. By repeating the generation of the virtual image data B ′ and the generation of the three-dimensional image data each time new image data A is acquired, a stereoscopic live view image having a desired amount of parallax is continuously displayed on the LCD 30. .

  FIG. 5 is a timing chart of the through image display process according to the first preferred embodiment of the present invention. This process is started in response to the shooting mode being set to “still image recording” or “moving image recording”.

  In S <b> 1, the CPU 40 controls to acquire the left and right image data A and B of the first frame from the CCD 16.

  In S <b> 2, the CPU 40 causes the stereo matching processing unit 83 and the distance calculation unit 84 to perform stereo matching and distance information calculation. The stereo matching processing unit 83 performs stereo matching based on the acquired image data A and B of the first frame. The distance calculation unit 84 calculates distance information for each corresponding point detected by the stereo matching processing unit 83. In the present embodiment, it is assumed that the time required for the stereo matching and the calculation of distance information corresponds to the acquisition interval of the left and right viewpoint image data.

  The CPU 40 performs control so that the image data A and B of the second frame are acquired from the CCD 16 during the stereo matching and the calculation of the distance information.

  In S <b> 3, the three-dimensional image processing unit 85 interpolates or extrapolates the distance information regarding the first frame image data A and B calculated by the distance calculation unit 84 to generate virtual distance information. Then, the three-dimensional image processing unit 85 is a set of parallax images having a desired parallax amount d based on the virtual distance information and the second frame image data A and / or image data B acquired from the CCD 16. Some image data A ′ and image data B ′ are generated.

  The image data A ′ and the image data B ′ may not be directly output from the CCD 16, but have image information (luminance information and color information) equivalent to that case. The three-dimensional image processing unit 85 generates the first frame of three-dimensional image data based on the image data A ′ and the image data B ′. The display control unit 35 causes the LCD 30 to display the 3D image data generated by the 3D image processing unit 85. The three-dimensional image data displayed on the LCD 30 has a desired parallax amount d.

  Similarly, until the release button 19 is fully pressed, distance information is calculated from the left and right image data A and B of the n (n = 1, 2, 3...) Frame, and the n-th image data A and B are calculated. The virtual distance information of the nth frame is generated by extrapolating or interpolating the distance information corresponding to B. Then, based on the virtual distance information of the nth frame and the image data A and / or the image data B of the (n + 1) th frame, image data A ′ and image data that are a set of parallax image data having a desired parallax amount d B ′ is generated. Then, based on the image data A ′ and B ′, the generation of n-th frame three-dimensional image data having a desired parallax amount d is repeatedly displayed on the LCD 30 as a through image.

  Any of the following three patterns may be employed as a method by which the three-dimensional image processing unit 85 generates image data A ′ and B ′ that are a set of parallax images having a desired parallax amount d.

  (1) By extrapolating or interpolating the distance information of the nth frame, the coordinates of each pixel of the image data A and the coordinates of each pixel of the image data B are respectively outside or inside by d / 2. The virtual distance information moved to is calculated. Then, according to the virtual distance information, the coordinates of each pixel of the image data A in the (n + 1) th frame and the coordinates of each pixel of the image data B are moved outward or inward by d / 2, respectively, and the pixel color is also changed to the original color. Move in order of arrangement. This method is effective when the number of pixels of the image data A is equal to the number of pixels of the image data B.

  (2) By extrapolating or interpolating the distance information of the nth frame, virtual distance information is calculated by moving the coordinates of each pixel of the image data B of the nth frame outward or inward by d. Then, according to the virtual distance information, the coordinates of each pixel of the image data B of the (n + 1) th frame are moved outward or inward by d, and the pixel colors are also moved in the original arrangement order. This method is effective when the number of pixels of the image data A is larger than the number of pixels of the image data B. In this case, the image data A ′ for the (n + 1) th frame may be the same as the image data A for the (n + 1) th frame. Similarly to Patent Document 2, the number of pixels of the image data B ′ can be made equal to that of the image data A ′.

  (3) By extrapolating or interpolating the distance information of the nth frame, virtual distance information is calculated by moving the coordinates of each pixel of the image data A of the nth frame outward or inward by d. Then, according to the virtual distance information, the coordinates of each pixel of the image data A of the (n + 1) th frame are moved outward or inward by d, and the pixel colors are also moved in the original arrangement order. This method is effective when the number of pixels of the image data B is larger than the number of pixels of the image data A. In this case, the image data B ′ of the (n + 1) th frame may be the same as the image data B of the (n + 1) th frame. Similarly to Patent Document 2, the number of pixels of the image data A ′ can be made equal to that of the image data B ′.

  In this way, since the through image is displayed from the viewpoint image of the subsequent frame using the distance information calculated for the previous frame, it is possible to prevent the latest through image display from being delayed due to the calculation of distance information. it can. It should be noted that the image data to be calculated for the distance information and the image data from which the three-dimensional image is generated have different acquisition timing, so the three-dimensional position of the subject in the through image is different from the actual one, but the image data for one frame The difference in subject position as viewed from the observer is considered to be small because of a shift corresponding to the acquisition time of.

  Furthermore, instead of generating a set of viewpoint images having a desired parallax based on extrapolation / interpolation of subject distance information, a parallax map (from a reference viewpoint image, eg, image data A, to another viewpoint image, eg, A set of viewpoint images having a desired parallax can be generated based on the extrapolation / interpolation of the parallax up to the corresponding point to the image data B and representing another viewpoint image). That is, the distance calculation unit 84 calculates a parallax map indicating the parallax for each corresponding pixel between the image data A and the image data B corresponding to the nth frame. The parallax map is calculated by stereo matching or the like. The three-dimensional image processing unit 85 generates a virtual parallax map by extrapolating or interpolating the parallax of each pixel of the parallax map corresponding to the nth frame. The movement of each pixel of the image data A and / or the image data B of the (n + 1) th frame based on the virtual parallax map of the nth frame is similar to the above three patterns of the image data A and the image data B corresponding to the (n + 1) th frame. Perform for both or either. In this manner, a set of viewpoint images having a desired parallax can also be generated by extrapolating or interpolating the parallax of each pixel of the parallax map.

  In the following, for the sake of simplicity, the distance information of the subject will be described, but the same processing holds for the parallax map.

Second Embodiment
When the time required for the stereo matching and distance information calculation is longer than the acquisition interval of the left and right image data A and B for one frame, the distance information of the nth frame is applied to the n + 1 frame image data to obtain a three-dimensional An image cannot be generated. Therefore, the calculated same distance information is applied to the image data of a plurality of frames acquired after the calculation of the distance information to generate a plurality of frames of three-dimensional image data.

  FIG. 6 is a timing chart of the through image display process according to the second preferred embodiment of the present invention.

  S11 is the same as S1, and the first frame of image data A and B is acquired from the CCD 16.

  S12 is the same as S2, and the stereo matching processing unit 83 performs stereo matching based on the acquired left and right image data A and B of the first frame. However, the left and right image data B of the second frame are discarded without being stored in the memory unit 31. The distance calculation unit 84 calculates distance information (distance information) of the corresponding points detected from the image data A and B of the first frame by the stereo matching processing unit 83. It is assumed that the time required for the stereo matching and distance information calculation exceeds the acquisition interval of the left and right image data, and corresponds to, for example, twice. That is, unlike the first embodiment, the calculation of distance information is not completed between the acquisition of the image of the previous frame and the acquisition of the image of the next frame.

  In S13, the CPU 40 performs control so that the image data A and B of the third frame are acquired from the CCD 16 during the stereo matching and the calculation of the distance information.

  In S14, the three-dimensional image processing unit 85 extrapolates or interpolates the distance information regarding the first frame image data A and B calculated by the distance calculation unit 84, and generates virtual distance information for the first frame. The three-dimensional image processing unit 85 uses the virtual distance information of the first frame and the image data A and / or image data B of the third frame acquired from the CCD 16 to generate the first frame having a desired parallax amount d. Image data A ′ and B ′ are generated. The three-dimensional image processing unit 85 generates three-dimensional image data for the first frame based on the image data A ′ and the image data B ′ for the first frame. The display control unit 35 displays the generated 3D image data on the LCD 30. The three-dimensional image data displayed on the LCD 30 has a desired parallax amount d.

  During this time, the CPU 40 performs control so that image data for the fourth frame is acquired from the CCD 16. Further, the CPU 40 controls to start stereo matching and distance information calculation based on the image data A and B of the third frame.

  In S15, the three-dimensional image processing unit 85 has a desired parallax amount d based on the virtual distance information of the first frame and the image data A and / or image data B of the fourth frame acquired from the CCD 16. Image data A ′ and image data B ′ for the second frame are generated. Then, based on the image data A ′ and image data B ′ of the second frame, 3D image data of the second frame is generated, and the display control unit 35 displays the generated 3D image data on the LCD 30. The three-dimensional image data displayed on the LCD 30 has a desired parallax amount d.

  During this time, the CPU 40 controls the image data of the fifth frame from the CCD 16 to be acquired.

  Similarly, until the release button 19 is fully pressed, the distance information is repeatedly calculated from the image data A and B of n (n = 1, 2, 3,...) Frames, and the distance information of the n frames is calculated. Virtual distance information of the nth frame is generated by extrapolation or interpolation. Then, based on the virtual distance information of the nth frame and the image data A and / or image data B of the (n + 2) th frame, 3D image data of the nth frame having a desired parallax amount d is generated, and the through image is displayed on the LCD 30. Display as. Further, based on the virtual distance information of the nth frame and the image data A and / or the image data B of the (n + 3) th frame, three-dimensional image data of the (n + 1) th frame having a desired parallax amount d is generated and the through image is displayed on the LCD 30. Display as.

  Thereby, it is possible to more effectively prevent a delay in displaying a through image having a desired parallax amount d due to waiting for calculation of distance information. In addition, the frame rate for displaying the through image can be made faster than the distance calculation.

<Third Embodiment>
In the first and second embodiments, when the coordinate value calculation takes time because the resolution of the image is high, or when the coordinate value calculation time changes depending on the information amount of image data such as image color information, the through image frame Disappears periodically. Therefore, the acquisition of the left and right image data of the next frame may be started when the coordinate value calculation is completed at a predetermined rate.

  FIG. 7 is a timing chart of the through image display process according to the preferred third embodiment of the present invention.

  S21 is the same as S11, and the CPU 40 controls to acquire the first frame image data A and B from the CCD 16 in accordance with the setting of the photographing mode.

  S22 is the same as S12, and the stereo matching processing unit 83 performs stereo matching based on the acquired image data A and B of the first frame. The distance calculation unit 84 calculates distance information of corresponding points detected from the image data A and B by the stereo matching processing unit 83.

  In S23, the CPU 40 controls the acquisition of the second frame image data A and B from the CCD 16 when the coordinate value calculation is completed at a predetermined rate, for example, when 70% is completed. It is arbitrary how to set the completion rate of the coordinate value calculation in conjunction with the start of acquisition of the image data A and B. For example, the length of time from the start of coordinate value calculation corresponding to the image data A and B of the first frame until the complete calculation of distance information is substantially the same as the time required for completion of image data acquisition In that case, it may be controlled to start the acquisition of the second frame of image data from the CCD 16 when that point arrives. In this way, a three-dimensional image can be generated immediately after obtaining the image data.

  In S <b> 24, the CPU 40 extrapolates or interpolates the distance information corresponding to the first frame image data A and B calculated from the distance calculation unit 84 to generate the first frame virtual distance information. Then, based on the virtual distance information of the first frame and the image data A and / or image data B of the second frame, the first frame image data A ′ and B ′ having a desired parallax d is generated. The dimensional image processing unit 85 is controlled. The three-dimensional image processing unit 85 generates three-dimensional image data for the first frame based on the image data A ′ and the image data B ′ for the first frame. The three-dimensional image data of the first frame obtained from the display control unit 35 is output to the LCD 30 and displayed as a through image. The three-dimensional image data displayed on the LCD 30 has a desired parallax amount d.

  During this time, the CPU 40 controls the stereo matching processing unit 83 and the distance calculation unit 84 so as to start the stereo matching and distance information calculation corresponding to the image data A and B of the second frame.

  Thereafter, similarly, the CPU 40 controls to acquire the n + 1-th frame image data A / B from the CCD 16 when the distance information calculation based on the n-th frame image data A / B acquired from the CCD 16 is completed by a predetermined ratio. To do. Further, the CPU 40 extrapolates or interpolates the distance information corresponding to the n-th frame image data A and B calculated from the distance calculation unit 84 to generate the n-th frame virtual distance information. Based on the virtual distance information and the n + 1 frame image data A and / or image data B, the n frame image data A ′ and B ′ having a desired parallax d are generated, and the image data A ′ and the image data are generated. Based on the data B ′, the 3D image processing unit 85 is controlled to generate the 3D image data of the nth frame.

  As described above, by making the image acquisition interval of each frame variable according to the calculation timing of the distance information, it is possible to quickly generate a three-dimensional image using the latest image data.

<Fourth embodiment>
In the third embodiment, when there are a plurality of sets of the stereo matching processing unit 83 and the distance calculation unit 84 and the parallel processing of independent distance information calculation is possible for each of the image data of the plurality of frames, the image of the latest frame It is preferable to perform control so that distance information calculation corresponding to the latest frame is executed for an empty set for which distance information calculation is not performed at the time of data acquisition.

  FIG. 8 shows a block diagram of a digital camera 1 according to a preferred fourth embodiment of the present invention. The same blocks as those already described are denoted by the same reference numerals. The digital camera 1 includes a plurality of sets, here two sets of stereo matching processing units 83a and distance calculation units 84a, stereo matching processing units 83b, and distance calculation units 84b as an example.

  FIG. 9 is a timing chart of the through image display process according to the preferred fourth embodiment of the present invention.

  In S41, the CPU 40 controls to acquire the first frame image data A and B from the CCD 16 according to the setting of the shooting mode.

  In S42, the CPU 40 controls the stereo matching processing unit 83a / distance calculation unit 84a to start calculating distance information corresponding to the image data A / B of the first frame. The stereo matching processing unit 83a performs stereo matching based on the acquired image data A and B of the first frame. The distance calculation unit 84a calculates distance information of corresponding points detected by the stereo matching processing unit 83a.

  At the same time, the CPU 40 controls to acquire the second frame image data A and B from the CCD 16.

  In S43, the CPU 40 controls the stereo matching processing unit 83b and the distance calculation unit 84b to start calculating distance information corresponding to the image data A and B of the second frame. The stereo matching processing unit 83b performs stereo matching based on the acquired second frame image data A and B. The distance calculation unit 84b calculates distance information of corresponding points detected from the image data A and B of the second frame by the stereo matching processing unit 83.

  At the start of the distance information calculation by the stereo matching processing unit 83b and the distance calculation unit 84b, the distance information calculation by the stereo matching processing unit 83a and the distance calculation unit 84a may be continued, and the calculation of the distance information of both is independent. Done independently.

  Further, the CPU 40 controls to acquire the third frame image data A and B from the CCD 16.

  In S44, the CPU 40 extrapolates or interpolates the distance information corresponding to the first frame image data A and B calculated in S42 to generate virtual distance information for the first frame in a three-dimensional manner. The image processing unit 85 is controlled. Then, based on the virtual distance information of the first frame and the image data A and / or image data B of the third frame, the CPU 40 obtains the first frame of image data A ′ / B ′ having a desired parallax amount d. The 3D image processing unit 85 is controlled to generate a 3D image of the first frame based on the image data A ′ and the image data B ′.

  At the same time, the CPU 40 controls the stereo matching processing unit 83a / distance calculation unit 84a to start calculating distance information corresponding to the image data A / B of the third frame. However, it is assumed that the calculation of the distance information corresponding to the image data A and B of the first frame has been completed before the calculation of the distance information corresponding to the image data A and B of the third frame is started. Calculation of the distance information corresponding to the second frame image data A and B may not be completed.

  Similarly, the CPU 40 controls the three-dimensional image processing unit 85 so as to generate a three-dimensional image of the nth frame (n = 1, 2,...). If n is an odd number, the CPU 40 controls the stereo matching processing unit 83a / distance calculation unit 84a to start calculating distance information corresponding to the image data A / B acquired in the nth frame, where n is If the number is even, the CPU 40 controls the stereo matching processing unit 83b and the distance calculation unit 84b to start calculating distance information corresponding to the image data A and B acquired in the nth frame.

  Then, the CPU 40 extrapolates or interpolates the distance information corresponding to the n-th frame image data A and B, thereby generating the n-th frame virtual distance information, and the n-th frame virtual distance. Based on the information and the image data A and / or B of the (n + 2) th frame, the nth frame of image data A ′ / B ′ having a desired parallax amount d is generated, and the nth frame of image data A ′ and image data Based on B ′, the 3D image processing unit 85 is controlled so as to generate a 3D image of the nth frame. The n-th frame three-dimensional image data obtained from the three-dimensional image processing unit 85 is output to the LCD 30 and displayed as a through image. The three-dimensional image data displayed on the LCD 30 has a desired parallax amount d.

  Similarly, when there are three or more sets of the stereo matching processing unit 83 and the distance calculation unit 84, processing is performed with the set of the free stereo matching processing unit 83 and the distance calculation unit 84, and the calculation has been completed for any one of the sets. The virtual distance information is generated by extrapolating or interpolating the latest distance information of the image, and based on the virtual distance information and the acquired latest image data A and / or image data B, a desired distance information is generated. Image data A and B having a parallax amount d are generated, and a three-dimensional frame image is generated. The number of necessary and sufficient sets corresponding to the acquisition interval of the left and right image data A and B depends on the calculation time interval of the distance information. In FIG. 9, as an example, the calculation time is about 1.5 times the acquisition interval of the left and right image data, so two sets for separately processing image data for two frames are sufficient.

  In this way, by separately calculating the distance information of the image data of different frames in the set of the plurality of stereo matching processing units 83 and the distance calculation unit 84, even when the calculation of the distance information takes more than one frame acquisition interval. Display delay of live view can be prevented.

<Fifth Embodiment>
In the first to fourth embodiments, the reproduction information of a three-dimensional image is recorded as follows when the release button 19 is fully pressed.

  FIG. 10 shows a timing chart of the moving image recording process according to the preferred fifth embodiment of the present invention. This process is started when the shooting mode is set to “moving image shooting” and a moving image recording start instruction such as a full press of the release button 19 is input.

  S51 and 52 are the same as S1 and 2. As a result, the first frame image is acquired and distance information corresponding to the first frame image is obtained.

  In S <b> 53, the three-dimensional image processing unit 85 interpolates or extrapolates the distance information regarding the first frame image data A and B calculated by the distance calculation unit 84 to generate virtual distance information for the first frame. The three-dimensional image processing unit 85 uses the virtual distance information of the first frame and the first frame image data A and / or image data B acquired from the CCD 16 to generate the first frame having a desired parallax amount d. Image data A ′ and image data B ′ are generated, and a three-dimensional image is generated from the images A ′ and B ′ in the first frame. The CPU 40 controls the media control unit 37 to record the generated three-dimensional image. Further, the virtual image data B ′ is generated by applying the distance information corresponding to the images A and B to the image A of the first frame acquired in S51, and from the images A and B ′ of the first frame. It is also possible to generate a three-dimensional image and control the media control unit 37 to record the generated three-dimensional image. Others are the same as S3.

  In the same manner, display of a through image and recording of a three-dimensional image are continued until a moving image recording end instruction is input. As the reproduction information of the three-dimensional image, the image of each frame and the distance information may be recorded in association with each other as they are. This is because it is possible to generate a three-dimensional image during reproduction based on this reproduction information. The circuit for generating a three-dimensional image from the reproduction information may be the three-dimensional image processing unit 85 or a circuit different from that. The timing of recording on the external recording medium 36 is also arbitrary, and the obtained images may be sequentially recorded, or may be recorded collectively in response to the input of the moving image recording end instruction. In the former case, it is preferable that a three-dimensional image of each frame is generated in accordance with the timing at which the viewpoint image of each frame is acquired.

  As for the recording of playback information, the delay from the real time does not become a problem as in the case of displaying a through image. Therefore, the playback information of a three-dimensional image is recorded in parallel with the display of a through image as in the first embodiment. To do. For recorded reproduction information, there is no deviation between distance information and image information.

<Sixth Embodiment>
In the first to fifth embodiments, since the distance information and the color information are not obtained from the same frame image, the stereoscopic effect of the three-dimensional image may be different from the actual depth of the subject. In order to reduce the deviation of the three-dimensional effect, the following is performed.

  The blur processing unit 86 compares the distance information between adjacent pixels, and blurs the distance information corresponding to the adjacent pixels when the difference in the distance information between the adjacent pixels exceeds a predetermined threshold. For example, the blur processing unit 86 smoothes the distance information corresponding to the adjacent pixels.

  The three-dimensional image processing unit 85 interpolates or extrapolates the n-th frame distance information smoothed by the blurring processing unit 86 instead of the n-th frame distance information calculated by the distance calculation unit 84, thereby obtaining the n-th frame. Generate virtual distance information. The three-dimensional image processing unit 85 has a desired parallax amount d based on the virtual distance information of the nth frame and the image data A and / or image data B of the (n + 1) th frame acquired from the CCD 16. The image data A ′ and image data B ′ of the nth frame are generated, and a three-dimensional image of the nth frame is generated from the images A ′ and B ′ of the nth frame. Further, the three-dimensional image processing unit 85 applies the distance information smoothed by the blurring processing unit 86 to the image data A instead of the distance information calculated by the distance calculation unit 84, thereby generating virtual image data B ′. 3D image data can also be generated. As a result, it is possible to prevent the deviation between the contour portion of the subject and the portion where the depth changes from being noticeable in the three-dimensional image.

  30 ... LCD, 40 ... CPU, 83 ... stereo matching processing unit, 84 ... distance calculation unit, 85 ... three-dimensional image processing unit, 86 ... blurring processing unit

Claims (23)

A single imaging optical system;
An imaging unit in which subject images that have passed through a first area and a second area having different predetermined directions of the imaging optical system are divided into pupils, respectively, and the first area and the second area are imaged An imaging unit capable of photoelectrically converting each of the subject images that have passed through the region and continuously outputting a set of viewpoint images including the first image and the second image;
An imaging control unit that controls the imaging unit so that a set of viewpoint images is output at a first time point and a second time point, which are arbitrary two time points before and after in time;
A parallax information calculation unit that performs stereo matching based on a set of viewpoint images output from the imaging unit at the first time point and calculates parallax information;
Virtual parallax information is generated by extrapolating or interpolating the parallax information corresponding to the first time point, and based on the virtual parallax information and the first image and / or the second image output at the second time point A stereoscopic image generation unit that generates a set of viewpoint images having a desired parallax and generates a stereoscopic image from the set of images having the desired parallax;
An imaging apparatus comprising: The imaging control unit controls the imaging unit so that the set of viewpoint images is sequentially output at each of a plurality of different second time points after the first time point;
The stereoscopic image generation unit generates virtual parallax information by extrapolating or interpolating the parallax information corresponding to the first time point, and outputs the virtual parallax information and the first image output at the second time point and 2. A plurality of sets of viewpoint images having desired parallax are generated based on the second image, and a plurality of stereoscopic images are sequentially generated from the plurality of sets of viewpoint images having the desired parallax. Imaging device.   The imaging control unit controls the imaging unit to output a set of viewpoint images at a second time point when the parallax information calculation process corresponding to the first time point by the parallax information calculation unit is completed by a predetermined ratio. The imaging device according to claim 1 to be controlled.   The disparity information calculating unit has a plurality of different processing systems capable of independently calculating a plurality of pieces of disparity information respectively corresponding to sets of different viewpoint images, and corresponds to a plurality of different first time points. The imaging apparatus according to claim 1, wherein each piece of parallax information to be calculated is calculated by a different processing system.   In response to the input of a predetermined recording instruction, the imaging apparatus extrapolates or interpolates disparity information corresponding to the first time point to generate virtual disparity information, and the virtual disparity information and the first disparity information A set of viewpoint images having a desired parallax is generated based on the first image and / or the second image output at the time, and reproduction information of a stereoscopic image is generated based on the set of viewpoint images having the desired parallax. The imaging device according to any one of claims 1 to 4, further comprising a recording unit for recording.   The imaging device according to claim 1, further comprising a display unit that displays a stereoscopic image generated by the stereoscopic image generation unit.   The imaging device according to claim 6, wherein the resolution of the second image is lower than the resolution of the first image and lower than the display resolution of the display unit.   The imaging apparatus according to claim 1, wherein the first image includes luminance information and color information, and the second image includes only luminance information. A blur processing unit that blurs the parallax information;
The stereoscopic image generation unit generates virtual disparity information by extrapolating or interpolating the disparity information corresponding to the first time point blurred by the blur processing unit, and generates the virtual disparity information and the second time point. 9. The set of viewpoint images having a desired parallax is generated based on the first image and / or the second image output in step 1, and a stereoscopic image is generated from the set of images having the desired parallax. Imaging device.   The imaging apparatus according to claim 1, wherein the desired amount of parallax includes a specified value, a fixed value, or a value corresponding to a distance of a subject distance input from a predetermined operation unit. A single imaging optical system;
An imaging unit in which subject images that have passed through a first area and a second area having different predetermined directions of the imaging optical system are divided into pupils, respectively, and the first area and the second area are imaged An imaging unit capable of photoelectrically converting each of the subject images that have passed through the region and continuously outputting a set of viewpoint images including the first image and the second image;
An imaging control unit that controls the imaging unit so that a set of viewpoint images is output at a first time point and a second time point, which are arbitrary two time points before and after in time;
A parallax information calculation unit that performs stereo matching based on a set of viewpoint images output from the imaging unit at the first time point and calculates parallax information;
Applying disparity information corresponding to the first time point to the first image output at the second time point to generate a virtual second image having a parallax with the first image, the second image A stereoscopic image generating unit that generates a stereoscopic image from the first image output at the time and the virtual second image;
An imaging apparatus comprising: The imaging control unit controls the imaging unit so that the set of viewpoint images is sequentially output at each of a plurality of different second time points after the first time point;
The stereoscopic image generation unit applies the disparity information corresponding to the first time point to a plurality of first images included in a set of a plurality of viewpoint images respectively output at the plurality of second time points. A plurality of virtual second images having parallax with one image are generated, and a plurality of stereoscopic images are generated from the plurality of virtual second images corresponding to the first images respectively output at the plurality of second time points. The imaging device according to claim 11, which is sequentially generated.   The imaging control unit controls the imaging unit to output a set of viewpoint images at a second time point when the parallax information calculation process corresponding to the first time point by the parallax information calculation unit is completed by a predetermined ratio. The imaging device according to claim 11 to be controlled.   The disparity information calculating unit has a plurality of different processing systems capable of independently calculating a plurality of pieces of disparity information respectively corresponding to sets of different viewpoint images, and corresponds to a plurality of different first time points. The imaging apparatus according to claim 11, wherein each piece of parallax information to be calculated is calculated by a different processing system.   In response to the input of a predetermined recording instruction, the imaging device applies disparity information corresponding to the first time point to the first image output at the first time point, and A recording unit that generates a virtual second image having a parallax between them and records reproduction information of a stereoscopic image based on the first image output at the first time point and the virtual second image. 14. The imaging device according to any one of 14.   The imaging device according to claim 11, further comprising a display unit that displays a stereoscopic image generated by the stereoscopic image generation unit.   The imaging device according to claim 16, wherein the resolution of the second image is lower than the resolution of the first image and lower than the display resolution of the display unit.   The imaging device according to claim 11, wherein the first image includes luminance information and color information, and the second image includes only luminance information. A blur processing unit that blurs the parallax information;
The stereoscopic image generation unit applies disparity information corresponding to the first time point blurred by the blur processing unit to the first image output at the second time point, and The imaging according to any one of claims 11 to 18, wherein a virtual second image having parallax is generated at step S3 and a stereoscopic image is generated from the first image output at the second time point and the virtual second image. apparatus.   The stereoscopic image generation unit generates virtual parallax information by extrapolating or interpolating the parallax information corresponding to the first time point, and generates the virtual parallax information and the first image output at the second time point and / or The imaging according to any one of claims 11 to 19, wherein a set of viewpoint images having a desired parallax is generated based on the second image, and a stereoscopic image is generated from the set of images having the desired parallax. apparatus.   21. The imaging apparatus according to claim 20, wherein the desired amount of parallax includes a designated value, a fixed value, or a value corresponding to the distance of the subject distance input from a predetermined operation unit. A single imaging optical system and an imaging unit in which subject images that have passed through a first region and a second region having different predetermined directions of the imaging optical system are each divided into pupils and imaged; An imaging apparatus comprising: an imaging unit capable of photoelectrically converting subject images that have passed through the first area and the second area, respectively, and continuously outputting a set of viewpoint images including the first image and the second image;
Controlling the imaging unit so that a set of viewpoint images is output at a first time point and a second time point, which are arbitrary two time points before and after in time;
Calculating parallax information by performing stereo matching based on a set of viewpoint images output from the imaging unit at the first time point;
Virtual parallax information is generated by extrapolating or interpolating the parallax information corresponding to the first time point, and based on the virtual parallax information and the first image and / or the second image output at the second time point Generating a set of viewpoint images having a desired parallax, and generating a stereoscopic image from the set of images having the desired parallax;
An imaging method for executing. A single imaging optical system and an imaging unit in which subject images that have passed through a first region and a second region having different predetermined directions of the imaging optical system are each divided into pupils and imaged; An imaging apparatus comprising: an imaging unit capable of photoelectrically converting subject images that have passed through the first area and the second area, respectively, and continuously outputting a set of viewpoint images including the first image and the second image;
Controlling the imaging unit so that a set of viewpoint images is output at a first time point and a second time point, which are arbitrary two time points before and after in time;
Calculating parallax information by performing stereo matching based on a set of viewpoint images output from the imaging unit at the first time point;
Applying disparity information corresponding to the first time point to the first image output at the second time point to generate a virtual second image having a parallax with the first image, the second image Generating a stereoscopic image from the first image output at the time and the virtual second image;
An imaging method for executing.

Images