JP2014036362A - Imaging device, control method therefor, and control program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable easy generation of a stereo image by suppressing a load on a user irrespective of a photographing scene.SOLUTION: A first image and a second image mutually different in depth of field are photographed under control of a CPU 113. A difference amount calculation unit 119 obtains a difference amount by calculating a pixel value difference between mutually corresponding areas in the first image and the second image. A depth direction determination unit 120 determines depth in an image for each of the areas in accordance with the difference amount, and a pseudo stereo image generation unit 123 generates a right-eye image and a left-eye image as a stereo image from the first image and the second image in accordance with the depth.

Description

  The present invention relates to an imaging apparatus such as a digital camera or a video camera, a control method thereof, and a control program, and more particularly to an imaging apparatus having a pseudo stereo image generation function.

  Generally, a natural image input device used when displaying a stereo image has two or more cameras. These cameras are arranged so that their optical axes are parallel to each other, or arranged so that their optical axes intersect with a slight convergence, and the images photographed by the cameras are directly input to the display device as images for the left and right eyes. Is done.

  On the other hand, in recent years, a camera that includes a plurality of optical systems and captures images for the left and right eyes has been put into practical use.

  As described above, when obtaining a stereo image, it is necessary to prepare a plurality of cameras or to specially configure one camera. For this reason, a camera is inevitably expensive to obtain a stereo image.

  By the way, the depth direction or depth amount of the subject is estimated according to various information included in the single or plural two-dimensional image signals, and a pseudo stereo image signal is created based on the depth information obtained as a result of the estimation. Technology is known. As one of such techniques, for example, there is a 2D-3D conversion technique.

  For example, for a single still image taken by a monocular imaging device, when a photographer specifies an area to which a stereoscopic effect is to be applied, the depth direction before and after the area is determined in the area.

  Here, a pseudo stereo image is generated by applying any one of a plurality of depth models set in advance to a specified region (see Patent Document 1).

  In addition, for a plurality of still images taken with a monocular imaging device and having different illumination conditions, a pseudo-stereo image is generated by subtracting pixel values to estimate a three-dimensional depth direction. Yes (see Patent Document 2).

JP 2003-47027 A JP 2004-200383 A

  However, in the method described in Patent Document 1, it is necessary for the photographer to designate a near area (area to be stereoscopically displayed) in the target image, which places a burden on the photographer.

  On the other hand, with the technique described in Patent Document 2, 2D-3D conversion cannot be performed in a scene where it is difficult to change the illumination conditions (for example, flash photography prohibited area).

  Accordingly, an object of the present invention is to provide an imaging apparatus capable of easily generating a pseudo stereo image regardless of a shooting scene, a control method thereof, and a control program.

  In order to achieve the above object, an imaging apparatus according to the present invention includes an imaging unit that captures a first image and a second image having different depths of field, and the first image and the second image. Difference amount calculation means for obtaining a difference amount by obtaining a difference between pixel values in a corresponding area, depth determination means for determining a depth in the image for each of the areas according to the difference amount, and the depth according to the depth And stereo image generation means for generating a right-eye image and a left-eye image from the first image or the second image.

  A control method according to the present invention is a control method for an imaging apparatus that generates a stereoscopic image that is a stereoscopic image from an image obtained by photographing a subject, and includes a first image and a second image having different depths of field. An imaging step of capturing an image of the image, a difference amount calculating step of obtaining a difference value by obtaining a difference between pixel values in regions corresponding to each other for the first image and the second image, and the region corresponding to the difference amount A depth determination step for determining a depth in the image for each of the above, and a stereo image generation step for generating a right-eye image and a left-eye image as the stereo image from the first image or the second image according to the depth, It is characterized by providing.

  A control program according to the present invention is a control program used in an imaging apparatus that generates a stereoscopic image, which is a stereoscopic image, from an image obtained by photographing a subject. An imaging step for capturing a first image and a second image having different depths, and a difference amount calculating step for obtaining a difference amount by obtaining a difference between pixel values in regions corresponding to each other with respect to the first image and the second image A depth determination step for determining a depth in the image for each of the regions according to the difference amount, and a right-eye image as the stereo image from the first image or the second image according to the depth, and And a stereo image generation step of generating a left-eye image.

  According to the present invention, it is possible to easily generate a stereo image by reducing the burden on the photographer regardless of the shooting scene.

It is a block diagram which shows the example about the structure of the imaging device by the 1st Embodiment of this invention. It is a figure for demonstrating the outline | summary about the production | generation of the pseudo | simulation stereo image in the camera shown in FIG. 1, (A) is a figure which shows an example of an image in a state with a deep depth of field, (B) is a depth of field. The figure which shows an example of the image in a shallow state, (C) is a figure which shows the image obtained as a result of comparing the image shown in (A) and the image shown in (B), (D) is shown in (C) The figure which shows the state which extracted the near side area | region from the image, (E) is a figure which shows the state which shifted the near side area | region shown to (D) right and left, (F) is the input image for left eyes, and the input image for right eyes FIG. 4G is a diagram showing an image of a stereo image. 3 is a flowchart for explaining the acquisition of a pseudo stereo image performed in the camera shown in FIG. 1. It is a figure which shows the relationship between an aperture value, shutter speed, and exposure amount. It is a figure for demonstrating the determination of the imaging | photography parameter in the time of the 2nd imaging | photography performed in the imaging | photography parameter determination part shown in FIG. FIG. 2 is a diagram for explaining calculation of a difference amount by a difference amount calculation unit shown in FIG. 1, in which (A) sets shooting parameters so that a depth of field is set so that all subjects are in focus from a certain viewpoint; FIG. 5B is a diagram illustrating an image when the image is captured, and FIG. 5B is a diagram illustrating an image captured when the imaging parameter is set so that the depth of field is narrowed from the same viewpoint as FIG. (C) is a figure which shows an example of calculation of difference amount, (d) is a figure which shows the other example of difference amount calculation. It is a figure for demonstrating the process by the depth direction determination part shown in FIG. 1, (A) is a figure which shows the state which mapped the difference amount obtained by the difference amount calculating part, (B) is a binarization process. It is a figure which shows the result of having performed. It is a block diagram which shows an example of the structure about the camera by the 2nd Embodiment of this invention. It is a figure for demonstrating the structure of the image pick-up element used with the camera shown in FIG. 8, (A) is a figure which shows a part of image pick-up surface (imaging surface) of an image pick-up element, (B) is a pixel of an image pick-up element. (C) is a figure which shows incidence | injection of the light with respect to a pixel. FIG. 9 is a flowchart for explaining a pseudo stereo image acquisition performed in the camera shown in FIG. 8. FIG.

  Hereinafter, an imaging device according to an embodiment of the present invention will be described with reference to the drawings.

[First Embodiment]
FIG. 1 is a block diagram showing an example of the configuration of the imaging apparatus according to the first embodiment of the present invention.

  In FIG. 1, the imaging apparatus 100 may be, for example, a so-called compact digital still camera or a movie camera integrated with a lens, or a digital single-lens reflex camera with an interchangeable taking lens.

  In addition, the imaging apparatus 100 is a camera that has an interchangeable photographic lens, does not include a reflex mirror or a focusing screen, and instead has an electronic viewfinder device (EVF) and a display device for displaying a live view image. There may be.

  In the following description, it is assumed that the imaging apparatus 100 is a photographic lens interchangeable type and is a digital still camera that shoots while viewing an electronic viewfinder (EVF) image displayed on the display unit 114.

  An imaging apparatus (hereinafter simply referred to as a camera) 100 has an interchangeable photographic lens unit 101. The photographing lens unit 101 includes a plurality of lens groups, a diaphragm, a focus ring, and the like. In the illustrated example, a photographing lens 101a, an aperture 102, a focusing lens 103, an aperture driving unit 121, and a focusing lens driving unit 122 are shown.

  The diaphragm 102 includes a plurality of blades (not shown). When the aperture drive unit 121 is driven under the control of the aperture control unit 111, the aperture 102 adjusts the amount of incident light by changing the aperture shape at the time of photographing.

  The focusing lens 103 is a lens for changing the focal position. The focusing lens driving unit 122 is driven under the control of the focusing lens control unit 110 to perform focusing. The photographer can drive the focusing lens 103 by manually adjusting a focus ring (not shown) provided in the photographing lens unit 101.

  A system control CPU (hereinafter simply referred to as a CPU) 113 controls the entire camera. A memory 117 is connected to the CPU 113, and various data are stored in the memory 117.

  The CPU 113 records the image data obtained as a result of shooting on the recording medium 118, for example. As the recording medium 118, a CF (Compact Flash (registered trademark)) card or an SD card is used.

  An optical image is formed on the image sensor 104 via the taking lens unit 101. In the illustrated example, a CCD or CMOS sensor is used as the image sensor 104, and the image sensor 104 outputs an analog signal (image signal) corresponding to the optical image.

  The image signal is converted into a digital signal by the A / D conversion unit 105 and given to the signal processing unit 106. The signal processing unit 106 performs signal processing on, for example, a Bayer array digital signal obtained as a result of adding pixel signals, and provides the signal to the CPU 113.

  Under the control of the CPU 113, the image processing unit 107 performs image processing such as white balance adjustment and noise removal on the digital signal input from the signal processing unit 106 to obtain image data.

  On the other hand, the CPU 113 sequentially displays an image corresponding to the digital signal input from the signal processing unit 106 on the display unit 114 in real time so as to function as an electronic viewfinder (EVF). The display unit 114 is composed of, for example, a TFT liquid crystal.

  The CPU 113 displays a parameter setting screen on the display unit 114. The photographer can set or change shooting parameters such as shutter speed, F value, and ISO sensitivity using the shooting parameter setting unit 115 on the parameter setting screen.

  The photographer who is a user can switch the shooting mode by the shooting mode switching unit 116. Here, for example, there are a manual mode and a pseudo stereo image shooting mode as the shooting mode.

  The electronic shutter control unit 108 changes the shutter speed according to the parameter specified by the shooting parameter setting unit 115 or the shooting mode specified by the shooting mode switching unit 116 under the control of the CPU 113. The ISO sensitivity control unit 109 changes the ISO sensitivity according to the parameter specified by the shooting parameter setting unit 115 or the shooting mode specified by the shooting mode switching unit 116 under the control of the CPU 113.

  The focusing lens control unit 110 performs focusing by controlling the driving of the focusing lens driving unit 122 during autofocus (AF) or the like under the control of the CPU 113. The diaphragm control unit 111 adjusts the diaphragm 102 by drivingly controlling the diaphragm driving unit 112 according to the parameter specified by the shooting parameter setting unit 115 or the shooting mode specified by the shooting mode switching unit 116 under the control of the CPU 113. .

  When the pseudo stereo image shooting mode is selected, the shooting parameter determination unit 112 determines shooting parameters when shooting a second image (hereinafter referred to as “image B”) to be shot for difference comparison. It should be noted that determination of shooting parameters when shooting the image B (second image) will be described later.

  When the pseudo stereo image shooting mode is selected, the difference amount calculation unit 119 calculates the difference amount between the first image (hereinafter referred to as “image A”) and the image B. Then, the difference amount calculation unit 119 gives the difference amount obtained as a result of the calculation to the depth direction determination unit 120 via the CPU 113.

  The depth direction determination unit 102 compares the difference amount with a predetermined threshold value, and determines the near image area and the background image area in the image A (first image) according to the comparison result, as will be described later. The pseudo stereo image generation unit 123 generates a pseudo stereo image under the control of the CPU 113 as described later.

  FIG. 2 is a diagram for explaining an outline of generation of a pseudo stereo image in the camera shown in FIG. FIG. 2A is a diagram illustrating an example of an image in a state where the depth of field is deep, and FIG. 2B is a diagram illustrating an example of an image in a state where the depth of field is shallow. 2C shows an image obtained as a result of comparing the image shown in FIG. 2A with the image shown in FIG. 2B, and FIG. 2D shows the image shown in FIG. It is a figure which shows the state which extracted the near side area | region from the image shown to (). 2E is a diagram showing a state where the near side region shown in FIG. 2D is shifted to the left and right, and FIG. 2F is a diagram showing a left-eye input image and a right-eye input image. is there. FIG. 2G shows a stereo image.

  In FIGS. 2A and 2B, when taking a still image, the depth of field is changed in the state where the main subject 200 is focused from the same viewpoint. To do. Note that the object 201 is located behind the main subject 200.

  In FIG. 2A, shooting is performed with the depth of field deepened, and both the main subject 200 and the object 201 appear to be in focus within the depth of field. On the other hand, in FIG. 2B, shooting is performed in a state where the depth of field is shallower than in FIG. 2A and the main subject 200 is in focus, but the object 201 is not covered. It is out of focus and out of focus.

  Subsequently, in the images shown in FIGS. 2A and 2B, the components of the specific area are compared and calculated. Here, similarly to the arrangement of pixels, the difference amounts of these images are mapped to obtain a video of a portion having a difference between both images (see FIG. 2C).

  From the characteristics of the depth of field, it can be said that the portion with little change between images is close to the in-focus position, and the portion with large change is far from the in-focus position. Further, the in-focus position is generally the main subject, and the main subject is most often located in the foreground in the image. For this reason, the foreground and the background can be roughly separated on the basis of the main subject.

  The difference amount obtained in FIG. 2C is compared with a threshold value set in advance, and processing such as binarization is performed. As shown in FIG. It is determined whether it belongs to a white area) or a background area (black area). Then, as shown in FIG. 2 (E), a parallax is assigned to the pixels belonging to the front area and shifted to the left or right to generate a stereo image for the left and right eyes.

  When the left-eye input image 202 and the right-eye input image 203 illustrated in FIG. 2F are input to a stereoscopic image display device such as a lenticular method or a parallax barrier method, an image having a stereoscopic effect can be displayed. That is, a stereo image is displayed as shown in FIG.

  FIG. 3 is a flowchart for explaining the acquisition of the pseudo stereo image performed in the camera shown in FIG.

  When acquiring the pseudo stereo image, that is, when generating the pseudo stereo image, the photographer selects the pseudo stereo image shooting mode by the shooting mode switching unit 116 (step S301). Next, the CPU 113 focuses on the main subject according to the set shooting parameter and performs the first shooting to obtain an image (step S302).

  In setting the shooting parameters, the shooting parameters set by the shooting person using the shooting parameter setting unit 115 may be used, or predetermined shooting parameters may be used.

  Here, an image obtained by the first shooting is referred to as “image A”. Subsequently, the CPU 113 provides the photographing parameter determination unit 112 with the F value, shutter speed, and ISO sensitivity in the photographing parameters when the image A is photographed. The shooting parameter determination unit 112 determines shooting parameters so that the exposure amount is equal to the exposure amount when the image A is shot (hereinafter referred to as “A exposure amount”) and the depth of field changes (step S303). The determination of the shooting parameter for changing the depth of field will be described later.

  Next, the CPU 113 sets the shooting parameters determined in step S303 as shooting parameters for the second shooting (step S304). Then, the CPU 113 executes the second shooting to obtain an image (step S305). Here, an image obtained by the second shooting is referred to as “image B”.

  Subsequently, the CPU 113 provides the difference amount calculation unit 119 with the image A (first image) and the image B (second image) taken continuously. Then, the difference amount calculation unit 119 calculates the difference amount between the image A and the image B (step S306). The calculation of the difference amount between the two images will be described later.

  The CPU 113 gives the difference amount obtained by the difference amount calculation unit 119 to the depth direction determination unit 120. Then, the depth direction determination unit 120 determines the near area and the background area (that is, the depth direction) as described later (step S307).

  Subsequently, under the control of the CPU 113, the pseudo stereo image generation unit 123 generates a right eye image and a left eye image (pseudo stereo image) in a pseudo manner based on the depth direction (2D3D conversion processing: step S308). When the generation of the pseudo stereo image ends, the CPU 113 ends the pseudo stereo image acquisition process.

  By the way, the following conditions 1 to 3 are taken into consideration for the determination of the shooting parameters at the time of the second shooting described above.

  Condition 1: Exposure amounts of image A and image B are equivalent.

  Condition 2: The F value of the image B is as far as possible from the F value when the image A is captured.

  Condition 3: The shutter speed when shooting the image B should not be slower than the shutter speed when shooting the image A.

  The above condition 1 is a condition for improving the calculation accuracy by combining the output gains of the image A and the image B. Condition 2 is a condition for making the change in the depth of field the greatest, and the difference amount becomes more prominent as the depth of field differs between the image A and the image B. Condition 3 is a condition for user convenience and prevention of camera shake.

  Here, the parameters are determined by assigning priorities in the order of condition 1> condition 2> condition 3.

  Generally, the following formula (1) is used when calculating the exposure amount.

Ev = Av + Tv (1)
Here, Ev is the exposure amount and represents the degree of exposure. Av (aperture priority) is a value in which the aperture value F1 is set to 0 and 1 is added every time the aperture is reduced by one step. Tv (shutter speed priority) is a value that is incremented by 1 each time the shutter speed is increased by 1 step and incremented by 1 each time the shutter speed is decreased by 1 step.

  According to the equation (1), for a predetermined shooting parameter, it can be obtained when the aperture value is reduced by one step and the shutter speed is decreased by one step, and when the aperture value is opened by one step and the shutter speed is increased by one step. Ev values are equal to each other.

  FIG. 4 is a diagram showing the relationship between the aperture value, shutter speed, and exposure amount.

  In FIG. 4, Ev when the image is taken with the aperture value F8 and the shutter speed 125/1 is Ev = 6 + 7 = 13. On the other hand, Ev when the image is taken with the aperture value F4 and the shutter speed 500/1 is Ev = 4 + 9 = 13. Therefore, the exposure amounts of the two images taken with the aperture value and the shutter speed are equal to each other.

  Expression (1) is for obtaining the exposure amount at the photographing sensitivity (ISO) 100, and it is necessary to perform ISO sensitivity correction for the calculation of exposure amounts other than ISO 100. Here, the Iv (exposure correction) value is set in the equation (1) in consideration of the correction according to the ISO sensitivity setting.

  The Iv value is a value that is incremented by 1 every time the photographing sensitivity is increased by one step with ISO 100 as 0. The exposure amount considering the Iv value is expressed by the following equation (2).

Ev = Av + Tv-Iv (2)
Then, if the formula (2) is modified, the Av value, the Tv value, and the Iv value can be obtained by the formulas (3) to (5), respectively.

Av = Ev−Tv + Iv (3)
Tv = Ev−Av + Iv (4)
Iv = Av + Tv−Ev (5)
FIG. 5 is a diagram for explaining the determination of shooting parameters at the time of the second shooting performed by the shooting parameter determination unit 112 shown in FIG.

  When the shooting parameter determination process is started, the shooting parameter determination unit 112 first calculates an Ev value when the image A is shot (step S401). Subsequently, the shooting parameter determination unit 112 determines whether or not the Av value when the image A is shot is smaller than the median value in the settable Av value range (step S402).

  If the Av value at the time of shooting image A is smaller than the median value (YES in step S402), shooting parameter determining unit 112 sets the Av value at the time of shooting image B to the maximum value (step S403). Then, the shooting parameter determination unit 112 determines whether or not an Ev value equivalent to the Ev value when the image A is shot when the image B is shot can be obtained by adjusting the Iv value (step S404).

  If the Ev value of the image B and the Ev value of the image A can be made equivalent by adjusting the Iv value (YES in step S404), the imaging parameter determination unit 112 determines that the Ev value of the image B and the Ev value of the image A are An equivalent Iv value is set (step S405), and the photographing parameter determination process is terminated.

  Now, it is assumed that the Av value and the Tv value can be set to 10 levels of 0 to 10, respectively, and the Iv value can be set to 5 levels of 0 to 5. It is assumed that the image A is shot with shooting parameters of Av value = 5, Tv value = 6, Iv value = 0, and Ev value = 11.

  In this case, Av value = 10 (maximum value) is set in step S403. In step S404, it is determined whether an equivalent Av value can be obtained by adjusting the Iv value using the Tv value obtained when the image A is captured.

  Using the above formula (5), Iv = 10 + 6-11 = 5. Since this Iv value is within the settable Iv value range, the imaging parameter determination unit 112 determines that the Iv value can be adjusted. In step S404, the adjusted Iv value = 5 is set as the Iv value when the image B is captured.

  As a result, shooting parameters that satisfy all of the above conditions 1 to 3 are determined as shooting parameters when shooting the image B.

  If the Ev value of image B and the Ev value of image A cannot be made equivalent by adjusting the Iv value (NO in step S404), shooting parameter determining unit 112 sets the Iv value to the maximum value (step S406). Then, the shooting parameter determination unit 112 determines whether or not an Ev value equivalent to the Ev value obtained when the image A is captured when the image B is captured can be obtained by adjusting the Tv value (step S407).

  If the Ev value of image B and the Ev value of image A can be made equivalent by adjusting the Tv value (YES in step S407), the imaging parameter determination unit 112 determines that the Ev value of image B and the Ev value of image A are An equivalent Tv value is set (step S408), and the shooting parameter determination process is terminated.

  On the other hand, if the Ev value of image B and the Ev value of image A cannot be made equivalent by adjusting the Tv value (NO in step S407), shooting parameter determining unit 112 sets the Tv value to the minimum value (step S409). . Then, the shooting parameter determination unit 112 calculates an Av value equivalent to the Ev value of the image A (step S410), and ends the shooting parameter determination process.

  As described above, the Av value and the Tv value can be set to 10 levels of 0 to 10, respectively, and the Iv value can be set to 5 levels of 0 to 5, and Av value = 4, Tv value = 6, Iv value = 0, Then, it is assumed that the image A is shot with the shooting parameters where the Ev value = 10.

  In this case, since the Ev value of the image B and the Ev value of the image A cannot be made equal by adjusting the Iv value, Iv value = 5 (maximum value) is set in step S406. In step S407, it is determined whether or not adjustment by the Tv value is possible.

  Here, using Equation (4), Tv = 10−10 + 5 = 5, and the Tv value is within the settable Tv value range. Therefore, the imaging parameter determination unit 112 performs the above Tv in step S408. The value is determined as a Tv value when the image B is captured.

  As a result, shooting parameters satisfying the above conditions 1 and 2 are determined as shooting parameters when shooting the image B.

  In step S402 described above, if the Av value when the image A is captured is equal to or greater than the median value (NO in step S402), the capturing parameter determination unit 112 sets the Av value when capturing the image B to the minimum value. (Step S411). Then, the shooting parameter determination unit 112 determines whether or not an Ev value equivalent to the Ev value obtained when the image A is captured when the image B is captured can be obtained by adjusting the Tv value (step S412).

  When the Ev value of the image B and the Ev value of the image A can be made equivalent by adjusting the Tv value (YES in step S412), the imaging parameter determination unit 112 determines that the Ev value of the image B and the Ev value of the image A are An equivalent Tv value is set (step S413), and the shooting parameter determination process is terminated.

  As described above, it is assumed that the Av value and the Tv value can be set to 10 levels of 0 to 10, respectively, and the Iv value can be set to 5 levels of 0 to 5, Av value = 6, Tv value = 4, Iv value = 0, Then, it is assumed that the image A is shot with the shooting parameters where the Ev value = 10.

  In this case, since the Av value when the image A is captured is equal to or greater than the median value, Av value = 0 (minimum value) is set in step S41104. In step S412, it is determined whether or not the Ev value of the image B can be equivalent to the Ev value of the image A by adjusting the Tv value.

  Since Tv = 10−0 + 0 = 10 according to the above equation (4), by adjusting the Tv value, an Ev value equivalent to the Ev value of the image A can be obtained when the image B is captured. Accordingly, in step S413, the above Tv value is determined as the Tv value when the image B is captured.

  As a result, shooting parameters that satisfy all of the above conditions 1 to 3 are determined as shooting parameters when shooting the image B.

  On the other hand, if the Ev value of image B and the Ev value of image A cannot be made equal by adjusting the Tv value (NO in step S412), shooting parameter determining unit 112 sets the Tv value to the maximum value (step S414). . Then, the imaging parameter determination unit 112 determines whether or not an Ev value equivalent to the Ev value when the image A is captured when the image B is captured can be obtained by adjusting the Iv value (step S415).

  If the Ev value of the image B and the Ev value of the image A can be made equivalent by adjusting the Iv value (YES in step S415), the imaging parameter determination unit 112 determines that the Ev value of the image B and the Ev value of the image A are An equivalent Iv value is set (step S416), and the photographing parameter determination process is terminated.

  If the Ev value of image B and the Ev value of image A cannot be made equivalent by adjusting the Iv value (NO in step S415), shooting parameter determining unit 112 sets the Iv value to the maximum value (step S417). Then, the shooting parameter determination unit 112 calculates an Av value that is equivalent to the Ev value of the image A (step S418), and ends the shooting parameter determination process.

  It is assumed that the Av value and the Tv value can be set to 10 levels of 0 to 10, respectively, and the Iv value can be set to 5 levels of 0 to 5, Av value = 6, Tv value = 6, Iv value = 4, and Ev value = Assume that the image A is shot with the shooting parameter of 8.

  In this case, since the Ev value of the image B and the Ev value of the image A cannot be made equivalent by adjusting the Tv value, Tv value = 10 (maximum value) is set in step S414. In step S415, it is determined whether or not the Ev value of the image B can be equivalent to the Ev value of the image A by adjusting the Iv value.

  At this time, since Av value = 0 (minimum value) is set in step S411, Iv = 0 + 10−8 = 2 according to the above equation (5). Therefore, by adjusting the Iv value, an Ev value equivalent to the Ev value of the image A is obtained when the image B is captured. Therefore, in step S416, the above Iv value is determined as the Iv value when the image B is captured.

  As a result, shooting parameters satisfying the above conditions 1 and 2 are determined as shooting parameters when shooting the image B.

  FIG. 6 is a diagram for explaining the calculation of the difference amount by the difference amount calculation unit 119 shown in FIG. FIG. 6A is a diagram showing an image when shooting is performed with the shooting parameters set so that all the subjects are in focus from a certain viewpoint, and FIG. FIG. 7 is a diagram showing an image that is an image taken by setting shooting parameters such that the depth of field is narrowed from the same viewpoint as in FIG. 6A. FIG. 6C is a diagram illustrating an example of the difference amount calculation, and FIG. 6D is a diagram illustrating another example of the difference amount calculation.

  In FIG. 6B, only the main subject is in focus, and the difference amount calculation unit 119 determines the pixel values of the regions at the same position in the images shown in FIGS. 6A and 6B. Is subtracted to calculate the difference amount.

  For example, the region 500 shown in FIG. 6A and the region 502 shown in FIG. 6B correspond to regions at the same position (corresponding regions), and similarly, the region 501 and the region 503 correspond to regions at the same position.

  In the example shown in the figure, pixel values are subtracted from each other for a 3 × 3 (Pixel) region with the pixel of interest at the center. Then, the difference amount calculation unit 119 obtains the sum of absolute values of the difference values obtained by subtraction for each pixel, and calculates the difference amount in each region.

  As a result, the areas 500 and 502 that are in focus in FIG. 6A but not in focus in FIG. 6B have a larger difference amount (see FIG. 6C). On the other hand, the difference amount is small in the regions 501 and 503 in which both FIGS. 6A and 6B are in focus (ideally 0: see FIG. 6D).

  In the example illustrated in FIG. 6, the difference calculation is performed for each 3 × 3 area for convenience of explanation, but is appropriately determined within a settable range for the area where the difference calculation is performed. Further, when calculating the difference amount, after subtracting pixel values at the same position for two images, low-pass filter processing may be performed.

  FIG. 7 is a diagram for describing processing by the depth direction determination unit 120 illustrated in FIG. 1. FIG. 7A is a diagram illustrating a state in which the difference amount obtained by the difference amount calculation unit is mapped, and FIG. 7B is a diagram illustrating a result of binarization processing.

  Here, a case where the depth direction is determined with two values will be described for convenience of explanation.

  When the difference amount in each region obtained by the difference amount calculation unit 119 is received, the depth direction determination unit 120 maps the difference amount to obtain, for example, the image shown in FIG.

  In FIG. 7A, regions 600 to 602 indicate a no-difference region, a small difference amount region, and a large difference amount region, respectively. The depth direction determination unit 120 compares these difference amounts with a preset threshold value and binarizes the difference amount. For example, when the difference amount exceeds the threshold value, the depth direction determination unit 120 sets the area to “1”. On the other hand, if the difference amount is equal to or smaller than the threshold value, the depth direction determination unit 120 sets the area to “0”. Then, the depth direction determination unit 120 obtains a binarization result illustrated in FIG. 7B, for example.

  In FIG. 6B, an area 603 indicates an area without a difference (that is, an area “0”), and an area 604 indicates an area where a difference occurs (that is, an area “1”). In this case, it can be said that the region having no difference from the characteristics of the depth of field, that is, the region 603 is a subject existing at the in-focus position.

  In general, since the photographer focuses on the main subject, the main subject is often positioned at the forefront of the composition. For this reason, the depth direction determination unit 120 determines that the region 603 is a front side (front side) region, and the region 604 is a back side (background direction) region.

  In the example shown in FIG. 7, the front-rear direction of the region in the image is determined by binarization. However, a plurality of threshold values may be set to determine the positional relationship in a plurality of stages.

  The pseudo stereo image generation unit 123 receives the near side region and the far side region obtained by the depth direction determination unit 120, assigns parallax to the pixels belonging to the near region, and shifts to the left or right, and stereo for the left and right eyes An image (a pseudo stereo image that is a stereoscopic image) is generated. Then, the CPU 113 displays the pseudo stereo image on the display unit 114.

  As described above, in the first embodiment of the present invention, the photographer does not need to perform a special operation, and can determine the front-rear direction of each region of the image by a simple calculation based on optical characteristics. This can reduce the burden on the photographer.

  Furthermore, it is not necessary to perform strobe light emission and the like, and a pseudo stereo image can be taken in a wide variety of shooting scenes.

[Second Embodiment]
Subsequently, a camera according to a second embodiment of the present invention will be described.

  FIG. 8 is a block diagram showing an example of the configuration of a camera according to the second embodiment of the present invention.

  In FIG. 8, the same components as those in the camera shown in FIG. Further, since the camera shown in FIG. 8 is different from the camera shown in FIG.

  A camera 700 illustrated in FIG. 8 includes a region addition image acquisition unit (also referred to as a pupil-divided image acquisition unit) 712 described later. The region addition image acquisition unit 712 includes an A / D conversion unit 105 and a signal processing unit 106. It is arranged between. Then, a digital signal that is the A / D conversion unit 105 is input to the region addition image acquisition unit 712.

  The area addition image acquisition unit 712 adds the image signals (digital signals) obtained from the plurality of photoelectric conversion areas in the image sensor 104 to generate a plurality of image signals having different depths of field.

  In the following description, a pair of image signals having different depths of field generated by the region addition image acquisition unit 712 are referred to as an image A and an image B, respectively. These images A and B are given to the signal processing unit 106. The signal processing unit 106 performs signal processing on the image signals of the Bayer array obtained as a result of adding the pixel signals as described above for each of the images A and B.

  FIG. 9 is a view for explaining the structure of the image sensor 104 used in the camera 700 shown in FIG. 9A is a diagram illustrating a part of the imaging surface (imaging plane) of the imaging device 104, and FIG. 9B is a diagram illustrating the pixels of the imaging device 104 in an enlarged manner. FIG. 9C shows the incidence of light on the pixel.

  As shown in FIG. 9A, the imaging element 104 has a plurality of pixels arranged in a two-dimensional matrix. In the image sensor 104, RGB color filters are provided on the pixels, and R pixels 801, G pixels 802, and B pixels 803 having different spectral characteristics are regularly arranged. Each pixel includes a microlens 807 and a photoelectric conversion region (photoelectric conversion unit).

  In FIG. 9B, a pixel 804 includes a photoelectric conversion region 805 indicated by diagonal lines and a photoelectric conversion region 806 indicated by mesh lines, and these photoelectric conversion regions 805 and 806 are arranged concentrically. The areas of the photoelectric conversion regions 805 and 806 are equal to each other.

  In FIG. 9C, a light beam indicated by a dotted line (broken line) is incident on the photoelectric conversion region 805 and undergoes photoelectric conversion. On the other hand, a light beam indicated by a solid line enters the photoelectric conversion region 806 and undergoes photoelectric conversion.

  From the relationship between the luminous flux and the area of the photoelectric conversion region, the pixel signal (one of the first image and the second image) obtained in the photoelectric conversion region 806 is the pixel obtained in the photoelectric conversion regions 805 and 806. This is equivalent to a pixel signal obtained by changing the aperture one stage with respect to an addition signal (the other of the first and second images) obtained by adding the signals. That is, it is possible to acquire two image progressions having different depths of field in one shooting.

  In the example shown in FIG. 9, the photoelectric conversion area is divided into two areas. However, if the photoelectric conversion area is divided into three or more areas, a plurality of image signals having different subject depths can be obtained. When the photoelectric conversion area is divided into three or more areas, the images obtained in the innermost area of the concentric circles and the images obtained in all the areas have two different depths of field. It becomes an image.

  FIG. 10 is a flowchart for explaining the pseudo-stereo image acquisition performed in the camera 700 shown in FIG. In FIG. 10, the same steps as those in FIG. 3 are denoted by the same reference numerals.

  When the pseudo stereo image shooting mode is selected in step S301, the CPU 113 performs shooting while focusing on the main subject according to the set shooting parameters (step S902).

  In setting the shooting parameters, the shooting parameters set by the shooting person using the shooting parameter setting unit 115 may be used, or predetermined shooting parameters may be used.

  Subsequently, the pupil divided image acquisition unit 712 generates two image signals having different depths of field using the image signals obtained by the plurality of photoelectric conversion regions (that is, by pupil division) (step S903). Then, the two image signals are subjected to signal processing by the signal processing unit 106 and then given to the CPU 113 as images A and B.

  At this time, the signal processing unit 106 adjusts the gain so that the levels of the two image signals are equivalent under the control of the CPU 113, and processes the exposure of the two image signals having different depths of field equivalently. (Step S904).

  Thereafter, as described with reference to FIG. 3, the processes of steps S306 to S308 are performed, and the pseudo stereo image acquisition process is ended.

  As described above, in the second embodiment of the present invention, since a pseudo stereo image can be obtained by one shooting, in addition to the effects of the first embodiment, the burden on the photographer can be further reduced. it can.

  As is clear from the above description, in the example shown in FIG. 1, the photographing lens unit 101, the image sensor 105, the A / D conversion unit 105, the signal processing unit 106, and the system control CPU 113 function as an imaging unit. Further, the difference amount calculation unit 119 and the system control CPU 113 function as a difference amount calculation unit.

  Further, the depth direction determination unit 120 and the system control CPU 113 function as depth determination means, and the pseudo stereo image generation unit 123 and the system control CPU 113 function as stereo image generation means.

  In addition, in the example shown in FIG. 8, the region addition image acquisition unit 712, the signal processing unit 106, and the system control CPU 113 function as a region addition unit, and the signal processing unit 106 and the system control CPU 113 function as a gain adjustment unit. To do.

  As mentioned above, although this invention was demonstrated based on embodiment, this invention is not limited to these embodiment, Various forms of the range which does not deviate from the summary of this invention are also contained in this invention. .

  For example, the function of the above embodiment may be used as a control method, and this control method may be executed by the imaging apparatus. Further, a program having the functions of the above-described embodiments may be used as a control program, and the control program may be executed by a computer included in the imaging apparatus. The control program is recorded on a computer-readable recording medium, for example.

  Each of the above control method and control program has at least an imaging step, a difference amount calculating step, a depth determining step, and a stereo image generating step.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various recording media, and the computer (or CPU, MPU, etc.) of the system or apparatus reads the program. To be executed.

104 Image sensor 105 A / D converter 113 CPU for system control
114 Display unit 115 Shooting parameter setting unit 116 Shooting mode switching unit 119 Difference amount calculation unit 120 Depth direction determination unit 123 Pseudo stereo image generation unit 712 Region addition image acquisition unit (pupil division image acquisition unit)

Claims (12)

Imaging means for capturing a first image and a second image having different depths of field;
Difference amount calculation means for obtaining a difference amount by obtaining a difference between pixel values in regions corresponding to each other for the first image and the second image;
Depth determining means for determining the depth in the image for each of the regions according to the difference amount;
Stereo image generation means for generating a right-eye image and a left-eye image from the first image or the second image according to the depth;
An imaging apparatus comprising:   The imaging apparatus according to claim 1, wherein the imaging unit obtains the first image and the second image by continuous shooting. The imaging means includes an imaging device in which pixels are arranged in a two-dimensional matrix, and a photoelectric conversion unit provided in each of the pixels is concentrically divided into a plurality of photoelectric conversion regions,
Furthermore, the imaging means includes area addition means for obtaining the first image and the second image having different subject depths by adding image signals obtained from the plurality of photoelectric conversion areas. The imaging device according to claim 1.   The difference amount calculation unit obtains a difference between pixel values for each pixel in the corresponding region, and uses a sum of absolute values of the differences as a difference amount. The imaging device described.   The depth determination means compares the difference amount with a preset threshold value, and determines a depth indicating whether the region is located on the near side or the far side according to the comparison result. The imaging apparatus according to claim 1, wherein the imaging apparatus is characterized.   The imaging apparatus according to claim 2, wherein when the second image is captured, the imaging unit sets an exposure amount equivalent to an exposure amount when the first image is captured.   The imaging unit, when capturing the second image, performs imaging by changing the depth of field with respect to the depth of field at the time of capturing the first image. 6. The imaging device according to 6.   8. The image pickup unit according to claim 7, wherein when the second image is taken, the shutter speed is set so as not to be slower than a shutter speed at which the first image is taken. Imaging device.   The imaging apparatus according to claim 3, further comprising: a gain adjusting unit that adjusts a gain so that levels of the first image and the second image obtained by the region adding unit are equivalent.   The region adding means outputs one of the first image and the second image according to an image signal output from the innermost photoelectric conversion region among the plurality of photoelectric conversion regions as the first image. 10. The method according to claim 3, wherein the second image is generated by adding all image signals output from the plurality of photoelectric conversion regions and generating the other of the first image and the second image. Imaging device. A method for controlling an imaging apparatus that generates a stereoscopic image that is a stereoscopic image from an image obtained by photographing a subject,
An imaging step of capturing a first image and a second image having different depths of field;
A difference amount calculating step of obtaining a difference amount by obtaining a difference between pixel values in regions corresponding to each other with respect to the first image and the second image;
A depth determining step for determining a depth in the image for each of the regions according to the difference amount;
A stereo image generation step of generating a right-eye image and a left-eye image as the stereo image from the first image or the second image according to the depth;
A control method comprising: A control program used in an imaging apparatus that generates a stereoscopic image that is a stereoscopic image from an image obtained by photographing a subject,
In the computer provided in the imaging device,
An imaging step of capturing a first image and a second image having different depths of field;
A difference amount calculating step of obtaining a difference amount by obtaining a difference between pixel values in regions corresponding to each other with respect to the first image and the second image;
A depth determining step for determining a depth in the image for each of the regions according to the difference amount;
A stereo image generation step of generating a right-eye image and a left-eye image as the stereo image from the first image or the second image according to the depth;
A control program characterized by causing

Images