JP6336316B2 - Imaging apparatus, control method, and program - Google Patents

Description

  The present invention relates to an imaging apparatus, a control method, and a program, and more particularly to a technique for generating an image focused on a predetermined subject distance after shooting.

  By adjusting the aperture of an imaging apparatus such as a digital camera, the user can capture an image having a desired depth of field. On the other hand, it is required to adjust the depth of field of a shooting scene not only during shooting but also after shooting.

  Among imaging devices, there is an imaging device capable of recording intensity distribution of a light beam from a subject incident on an imaging surface as light field information together with information on the incident direction. In such an imaging apparatus, an image having a predetermined depth of field can be generated (reconstructed) from the light space information recorded by shooting. Also, from the light space information, it is possible to generate not only an image in which the depth of field is adjusted but also a so-called refocused image that focuses on a subject different from the subject focused at the time of shooting. Japanese Patent Application Laid-Open No. 2004-228561 discloses an imaging device that has a replaceable photographic lens and can record light space information.

JP 2011-043581 A

  By the way, depending on the aperture diameter at the time of shooting, an image in which the depth of field is adjusted from the obtained light space information or an image focused on a desired subject may not be generated. That is, in order to generate such an image, it is necessary to cause the imaging device to receive a light beam having various incident directions. However, by setting the aperture diameter to be small, the incident light beam is limited, and an image of a desired condition can be obtained. It may not be generated. Therefore, in order to be able to generate an image with a desired condition after shooting, it is necessary to provide an appropriate lower limit value for the aperture diameter at the time of shooting. However, there has been no image pickup apparatus that performs such control conventionally. In Patent Document 1, an upper limit is set on the aperture diameter that can be set in order to prevent stray light from occurring when the F value of the imaging lens is set to a value smaller than the F value of the microlens (the effective aperture is large). Although the setting of the value is disclosed, the setting of the lower limit value is not disclosed.

  The present invention has been made in view of the above-described problems of the prior art, and an object thereof is to provide an imaging apparatus, a control method, and a program that output light space information that can generate images with different conditions after imaging. To do.

In order to solve this problem, for example, an imaging apparatus of the present invention has the following configuration. That is, an imaging device that outputs a pixel signal that records a light flux that has passed through each of the divided pupil regions and that can generate a reconstructed image that is focused on one of a plurality of subject distances, There are control means for controlling the lower limit of the aperture diameter of the aperture to a value that allows the light beam that has passed through the plurality of divided pupil regions to pass through the aperture, and change means for changing the aperture diameter of the aperture when photographing the pixel signal. In the case where a photographing mode for outputting a pixel signal capable of generating a reconstructed image with a plurality of different conditions is set, the changing means is set when the aperture diameter of the diaphragm that is lower than the lower limit is set by the user. The aperture diameter of the diaphragm is changed to a lower limit value .

  According to the present invention, it is possible to output light space information capable of generating images with different conditions after shooting.

1 is a block diagram illustrating a functional configuration example of a digital camera 100 and a lens 102 as an example of an imaging apparatus according to an embodiment of the present invention. Rear view of digital camera 100 1 schematically shows an optical system of a digital camera 100 according to Embodiment 1. FIG. The figure which shows the example of a display of the display part 258 at the time of imaging | photography. 7 is a flowchart showing a series of processing when shooting is performed in the refocus priority shooting mode according to the first embodiment. The figure for demonstrating the setting of the aperture value which concerns on Embodiment 1. FIG. 7 is a flowchart showing a series of image generation processing according to the first embodiment. 7 is a flowchart showing a series of image shift processing according to the first embodiment. The figure which shows the reconstruction process of the image typically 8 is a flowchart showing a series of processing when shooting is performed in the refocus priority shooting mode according to the second embodiment. The figure which shows typically the relationship between the magnitude | size of the aperture diameter of the aperture stop 140, and the state in which the light ray from a to-be-photographed object image-forms on the image pick-up element 252.

(Embodiment 1)
Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. In the following, as an example of the imaging apparatus, an example in which the present invention is applied to a digital camera capable of recording light space information (LF data) composed of pixel values having information on the intensity and incident angle of a light beam from a subject. explain. However, the present invention is not limited to a digital camera that can record LF data, but can be applied to any device that can acquire LF data by adjusting an aperture value. In the following description, an image obtained by reconstructing LF data is referred to as a reconstructed image.

(1 Digital camera configuration)
FIG. 1 is a block diagram illustrating a functional configuration example of a digital camera 100 and a lens 102 as an example of an imaging apparatus according to an embodiment of the present invention. In the camera system according to the present embodiment, a photographing optical system 101 is provided in a lens 102 that can be attached to and detached from the digital camera 100.

  The light flux from the subject forms an image on the imaging surface of the imaging element 252 via the photographing optical system 101. The photographing optical system 101 includes a zoom lens 110, and can be driven in the optical axis direction by a zoom lens driving unit 111 using an ultrasonic motor or a stepping motor as a driving source. The zoom lens voltage driver 113 generates a voltage for driving and controlling the zoom lens driving unit 111.

  The zoom position detection unit 112 is a zoom encoder that detects the position of the zoom lens 110 along the optical axis I. The zoom position detection unit 112 outputs a pulse signal corresponding to the focal length value to the zoom lens control unit 105. The zoom lens control unit 105 controls the zoom lens driving unit 111 so that, for example, the zoom lens 110 is driven to a position on the optical axis I where the focal length is set by the user using a switch (not shown) on the digital camera 100 or the lens 102. Follow-up control is performed. Further, the zoom lens control unit 105 takes in the pulse signal output from the zoom position detection unit 112. The zoom lens control unit 105 calculates a drive signal based on a position (target position) on the optical axis I that becomes a focal length set by the user, current position information of the zoom lens 110, and the like, and uses the digital drive signal as a zoom lens. Output to the voltage driver 113. The zoom lens voltage driver 113 is a driver unit that supplies power to the zoom lens driving unit 111 in accordance with an input driving signal (driving voltage). The zoom lens voltage driver 113 switches the drive signal and applies a voltage to the zoom lens drive unit 111 to drive the zoom lens drive unit 111.

  The photographing optical system 101 includes an AF lens 120 and can be driven in the optical axis direction by an AF lens driving unit 121 using an ultrasonic motor or a stepping motor as a driving source. The AF lens voltage driver 123 generates a voltage for driving and controlling the AF lens driving unit 121. The focus position detection unit 122 is a focus encoder that detects the position of the AF lens 120 along the optical axis I. The focus position detection unit 122 outputs a pulse signal corresponding to the subject distance value to the AF lens control unit 104. The AF lens control unit 104 controls the AF lens driving unit 121, and the AF lens 120 is driven to a position on the optical axis I that is the subject distance detected by the operation of the release switch 191 of the user's digital camera 100, for example. This is the part that performs the follow-up control. The AF lens control unit 104 captures a pulse signal output from the focus position detection unit 122. The AF lens control unit 104 calculates a drive signal based on the position (target position) on the optical axis I that is the subject distance, the current position information of the AF lens 120, and the like, and this digital drive signal is sent to the AF lens voltage driver 123. Output.

  The AF lens voltage driver 123 is a driver unit that supplies power to the AF lens driving unit 121 in accordance with an input driving signal (driving voltage). The AF lens voltage driver 123 performs switching on the driving signal, applies a voltage to the AF lens driving unit 121, and drives the AF lens driving unit 121.

  The photographing optical system 101 has a diaphragm 140, and the size of the aperture of the diaphragm 140 is controlled by a diaphragm driving unit 141 that uses a stepping motor or the like as a driving source. The aperture control unit 106 controls the aperture drive unit 141 to calculate an aperture value that is an appropriate exposure amount according to, for example, the brightness of the subject to be photographed, and to calculate a drive signal that is the aperture value to drive the aperture. Output to the unit 141. Further, as will be described later, the aperture control unit 106 calculates a drive signal according to the aperture value set by the user with the aperture setting switch 194 or an instruction from the body CPU 109 and outputs the drive signal to the aperture drive unit 141.

  The lens CPU 103 is a central processing unit that performs various controls on the lens 102 side. In the lens CPU 103, a zoom lens control unit 105, an AF lens control unit 104, and an aperture control unit 106 are provided. The lens CPU 103 can communicate with the body CPU 109 via a lens contact 190 provided between the lens 102 and the digital camera 100. The EEPROM 131 is a non-volatile storage unit that stores lens data that is various kinds of unique information regarding the lens 102. The lens memory 132 is a volatile storage unit that stores various data when the lens CPU 103 performs arithmetic processing.

  The image sensor 252 is a CMOS sensor that is one of solid-state image sensors, for example. Since random access to an arbitrary pixel is possible and readout that is thinned out for display can be easily performed, an image can be displayed on the display portion 258 at a high display rate. The image sensor 252 utilizes the above-described features, and performs a display image output operation (reading in a region where a part of the light receiving region of the image sensor 252 is thinned) and a high-definition image output operation (reading in the entire light receiving region). Do.

  A microlens array (MLA) 20 is disposed between the imaging element 252 and the photographing optical system 101, and the microlenses are arranged in a lattice pattern on the MLA20. One microlens is associated with a plurality of photoelectric conversion elements of an image sensor, and light incident on the microlens is imaged on the plurality of photoelectric conversion elements and subjected to photoelectric conversion. Since each of the photoelectric conversion elements associated with one microlens outputs a pixel signal corresponding to the light beam that has passed through each of the divided pupil regions, LF data is obtained. Details of the functions and arrangement of the MLA 20 will be described later with reference to FIG.

  Further, as will be described later, the body CPU 109 appropriately adjusts the photographing optical system 101 based on the focus evaluation amount obtained from the image sensor 252 and the appropriate exposure amount. By this adjustment, reflected light from the subject is exposed to the image sensor 252 with an appropriate amount of light, and a subject image is formed in the vicinity of the image sensor 252. As will be described later, the body CPU 109 determines the main subject based on the defocus amount (blur amount) of the captured image, and controls the driving of the AF lens 120 via the lens CPU 103.

  The image processing unit 150 includes an A / D converter, a white balance circuit, a gamma correction circuit, an interpolation calculation circuit, and the like. The image processing unit 150 performs various processes on the LF data input from the image sensor 252 and records it. Generate an image for Various processes performed by the image processing unit 150 include an image shift process, an image generation process, and a correlation calculation process, which will be described later. In addition, the image processing unit 150 outputs the generated image to the recording unit, and compresses an image, a moving image, audio, and the like using a predetermined method. Further, the image processing unit 150 outputs an image generated according to an instruction from the body CPU 109 to the display unit 258.

  The body CPU 109 is a programmable processor such as a CPU or MPU, for example, and performs various controls of the entire camera system by reading out a program stored in a non-volatile memory (not shown), developing the program in the memory 197, and executing the program.

  When the release switch 191 is operated, the body CPU 109 detects that the release switch 191 is half-pressed or fully pressed, and controls the driving of the image sensor 252 and the operation of the image processing unit 150. The body CPU 109 communicates with the lens CPU 103 in order to control the aperture driving unit 141 based on an aperture value calculated according to a shooting mode described later. The body CPU 109 further controls the state of each segment of the information display device that displays information on a liquid crystal monitor or the like by the display unit 258.

  The release switch 191 detects a half-pressing operation of a release button (not shown) (SW1 is turned on) or a release button full-pressing operation (SW2 is turned on). A series of shooting preparation operations (photometry operation, focus adjustment operation, etc.) is started by detecting the half-press operation of the release button, and a shooting operation (from the image sensor 252) by detecting the release button full-press operation (SW2 ON). Recording of the read LF data on a recording medium) is started.

  FIG. 2 shows an example of a rear view of the digital camera 100 of the present embodiment. The selection switch 192 is a selection switch that can be operated in an arbitrary direction, and is a switch that is used when a user selects a well-known shooting menu item (not shown) or a user selects a main subject as will be described later.

  The decision switch 193 is a switch used when the user decides an operation in a well-known shooting menu (not shown) or when the user decides a main subject as will be described later. The aperture setting switch 194 is a switch that allows the user to set an aperture value by operating an operation dial (not shown).

  A main switch 195 is a switch for starting up the digital camera 100. The main switch 195 is a switch that can select two positions (195-1, 195-2). When the main switch 195 is set to the position 195-1, “OFF” is indicated, and the digital camera 100 enters a sleep state. When the main switch 195 is set to the position 195-2, “ON” is indicated, and the digital camera 100 is changed from the sleep state to the drive state, and accepts the operation of various switches to operate for taking a still image or a moving image. It will be in the state to perform.

  The mode switch 196 is a switch for setting the shooting mode of the digital camera 100. The user can set at least a mode in which shooting that gives priority to the shooting conditions set by the user and a “refocus priority shooting mode” can be set. In this embodiment, the “refocus priority shooting mode” refers to, for example, focusing on a main subject and then focusing on a subject distance different from that at the time of shooting (for example, focusing on a subject other than the main subject). This is a shooting mode for acquiring LF data that can generate an image.

  The memory 197 includes a processing circuit necessary for recording in addition to an actual storage unit.

  The display unit 258 is attached to the back surface of the digital camera 100, and displays image data generated from LF data at the time of shooting or playback, for example. Further, when displaying information such as the photographing environment and the state of the photographing optical system 101, the body CPU 109 causes the display unit 9 to display GUI data indicating the information. For example, if the display unit 258 is configured by an organic EL element, power consumption can be reduced, and the display unit 258 can be thinned. As a result, power saving and downsizing of the digital camera 100 can be achieved.

  FIG. 3 is a diagram schematically showing a photographing optical system in the present embodiment. In the present embodiment, LF data obtained by acquiring angle information in addition to the position of the light beam is acquired. In the present embodiment, the MLA 20 is disposed in the vicinity of the imaging plane of the photographing optical system 101 for acquiring LF data, and a plurality of pixels are associated with one microlens constituting the MLA 20.

  FIG. 3A schematically shows the relationship between the image sensor 252 and the MLA 20. FIG. 3B is a diagram schematically illustrating the correspondence between the pixels of the image sensor 252 and the MLA 20. FIG. 3C shows that the pixels provided under the MLA 20 by the MLA 20 are associated with a specific pupil region.

  As shown in FIG. 3A, the MLA 20 is provided on the image sensor 252, and the front principal point of the MLA 20 is arranged in the vicinity of the imaging plane of the photographing optical system 101. In the present embodiment, the MLA 20 on the image sensor 252 is arranged so that the front principal point of each microlens is in the vicinity of the imaging plane of the photographing optical system 101. 3A, the MLA 20 is arranged so as to cover a photoelectric conversion element (hereinafter referred to as a pixel) of the image sensor 252, as shown in the front view (a view of the MLA 20 viewed in the negative z-axis direction). Yes.

  Here, as shown in FIG. 2B, a plurality of pixels of the image sensor 252 are associated with each microlens of the MLA 20. A grid-like frame shown in FIG. 3B indicates each pixel of the image sensor 252. Reference numerals 20a, 20b, 20c, and 20d denote the respective microlenses constituting the MLA 20. As shown in FIG. 3B, a plurality of pixels are assigned to one microlens, and in the example of FIG. 3B, 25 pixels are provided for one microlens. (The size of each microlens is 5 × 5 times the size of the pixel).

  FIG. 3C shows a correspondence relationship between the light beam focused on the pixel of the image sensor 252 associated with each microlens of the MLA 20 and the pupil region of the photographing optical system 101. In FIG. 3C, for convenience, the pixel of the image sensor 252 associated with the microlens and the lens shown in the XZ section passing through the principal point of one microlens and orthogonal to the Y axis, and the emission orthogonal to the Z axis. The pupil region showing the pupil plane 30 (XY plane) is shown on the same plane. In FIG. 3C, pixels 21, 22, 23, 24, and 25 correspond to, for example, the pixels 21a to 25a in FIG. 3B, and light beams that have passed through the divided pupil regions are imaged. That is, each of the pixels 21 to 25 is designed to be conjugate with a specific divided pupil region on the exit pupil plane of the photographing optical system 101 by the micro lens of the MLA 20. In the example of FIG. 3C, the pixel 21 and the region 31 correspond to the pixel 22 and the region 32, the pixel 23 and the region 33, the pixel 24 and the region 34, and the pixel 25 and the region 35, respectively. Accordingly, by determining the position of the microlens and the distance between the exit pupil plane and the image sensor 6, information on the incident angle of the light beam incident on each of the pixels associated with the microlens can be substantially acquired. .

  FIG. 4 shows the state of the subject image displayed on the display unit 258 during so-called live view shooting. In FIG. 4, the display unit 258 displays a subject image 258a and shooting information (258b) such as shooting conditions (shutter speed, aperture value, ISO sensitivity, etc.), exposure level display, and battery level display. In the example of FIG. 4, the main subject 310 is a main subject, and the other subjects 320, 330, or 340 are subjects other than the main subject.

  A plurality of AF detection frames 301 are provided in the subject image 258a display unit. Since the AF detection frame 302 selected based on the focus detection result from the image sensor 252 is displayed in another display form (display with different colors, etc.), the user knows the main subject portion in focus. Can do.

  If the selection switch 192 and the determination switch 193 of the digital camera 100 are used, the user can detect an AF at any position desired to be focused from among the plurality of AF detection frames 301 regardless of the focus detection result from the image sensor 252. A frame can be selected. Based on the AF detection frame 302 selected by the user, the captured image is prevented from being blurred by calculating the distance to the main subject 310 from the focus detection result from the image sensor 252.

(2 Overview of image generation processing and image shift)
An overview of image generation processing and image shift processing for generating a reconstructed image will be described with reference to FIG.

FIG. 9B shows pixels that are added when each pixel of the reconstructed image is generated with respect to the surface (acquisition surface) on which the imaging element 252 captures the light beam related to the subject image. In FIG. 9B, each of the pixels X1 _{, i} , X2 _{, i} , X3 _{, i} , X4 _{, i} , and X5 _{, i} is defined by i through each of the divided pupil regions 1-5. The pixel corresponding to the light beam incident on the microlens at the position is shown. In FIG. 9, for simplicity of explanation, the LF data defines a divided pupil region obtained by dividing the pupil into five in one direction, and records the light flux that has passed through each of the divided pupil regions for each microlens. To do. Accordingly, in FIG. 9B, pixels corresponding to the same microlens are arranged in the vertical direction, and pixels having the same divided pupil region through which the corresponding light flux has passed are arranged in the horizontal direction. In relation to the physical position, X _{1, i} is attached with data obtained from the area 21 in FIG. 3C, and X _{2, i} is attached with data obtained from the area 22 in FIG. The letters 3, 4, and 5 indicate that they correspond to the areas 23, 24, and 25, respectively.

In order to generate a reconstructed image in a focus state corresponding to the acquisition surface, pixel values of pixels corresponding to each microlens may be added (image generation processing) as shown by a broken line in FIG. 9B. Specifically, by obtaining the sum S _i of the pixel values of X _{1, i} , X _{2, i} , X _{3, i} , X _{4, i} , X _{5, i} , the microlens at the position defined by i The pixel value of the reconstructed image corresponding to can be obtained. Further, when only a pixel value corresponding to the light beam that has passed through the central divided pupil region 3 is used when generating a reconstructed image, it is equivalent to a state in which the pupil is narrowed, that is, the light beam incident on the image sensor is divided into the center. An image equivalent to a case where the light flux passing through the pupil is limited is obtained. For this reason, the obtained reconstructed image is an image having a deeper depth of field than an image obtained by adding all the pixels.

Next, as shown in FIG. 9A, consider a case where a reconstructed image is generated for a surface (reconstructed surface 1) that is defined closer to the subject than the acquisition surface. As described with reference to FIG. 3, the imaging optical system 101 of the present embodiment limits the luminous flux incident on each pixel to a specific divided pupil region, and thus the incident angle is known. The position of each pixel on the reconstruction plane is reconstructed along this angle. Specifically, it is assumed that a segmented pupil region with a subscript of 1 such as X _{1, i} is incident at an angle as indicated by 41 in FIG. The divided pupil regions 2, 3, 4, and 5 correspond to 42, 43, 44, and 45, respectively. At this time, the light beam incident on the microlens X _i on the reconstruction surface 1 is scattered and incident from X _{i −2} to X _{i + 2} on the acquisition surface. More specifically, it is distributed in X1 _{, i-2} , X2 _{, i-1} , X3 _{, i} , X4 _{, i + 1} , and X5 _{, i + 2} . In order to generate a reconstructed image in the focus state corresponding to the reconstructed plane 1, it is understood that pixels shifted according to the incident angle may be added, as indicated by a broken line in FIG. In order to generate a reconstructed image on the reconstruction plane 1, one with a divided pupil region of 1 has two pixels on the right, one with two divided pupil regions has one pixel on the right, and one with three divided pupil regions has no shift. When the divided pupil region is 4, the pixel is shifted to the left by 1 pixel, and when the divided pupil region is 5, the pixel is shifted to the left by 2 pixels. Thereby, the shift amount according to the incident angle can be given to each pixel. Shifting the corresponding pixel on the reconstruction surface based on the incident direction of the light beam in this way is called image shift processing.

Thereafter, by adding in the vertical direction of FIG. 9A (that is, for each divided pupil region), the pixel value of the reconstructed image in the focus state corresponding to the reconstruction plane 1 can be obtained. Specifically, the pixel value of this reconstructed image is the sum S _{i of} X _{1, i-2} , X _{2, i-1} , X _{3, i} , X _{4, i + 1} , X _{5, i + 2.} Therefore, the integral value in the angular direction of the light incident on X _i can be obtained. Thereby, a reconstructed image on the reconstructed surface 1 is obtained.

Here, as shown in FIG. 9 (d), if the reconstructed surface 1 and the X _i is the bright _{spot, X 1, i-2,} X 2 on the obtained surface light _{flux, i-1,} X _{3, i} , X _{4, i + 1} , X _{5, i + 2} are dispersed and are in a so-called blurred state. However, when a reconstructed image on the reconstruction surface 1 is generated, a bright spot is generated again on X _i and an image with high contrast is obtained. As can be seen from FIG. 9C, an image can be generated on the reconstruction plane 2 as well as on the reconstruction plane. What is necessary is just to reverse the direction to shift, if the directions which arrange | position a reconstruction surface differ.

  By the image generation process and the image shift process described above, a reconstructed image to be focused on an arbitrary subject (for example, the subject 320, 330, or 340 shown in FIG. 4) that is not focused at the time of shooting. Can be generated.

(3 series of shooting operations in refocus priority shooting mode)
A series of processes when the user performs shooting while setting the shooting mode to the refocus priority shooting mode will be described with reference to the flowchart shown in FIG.

  The processing corresponding to the flowchart can be realized by the body CPU 109 reading, for example, a processing program stored in the nonvolatile memory, developing the program in the memory 197, and executing it. This process is started when, for example, an operation on the main switch 195 is detected.

  First, in S1010, the body CPU 109 determines whether or not the main switch 195 is ON. If the main switch 195 is OFF, the body CPU 109 ends the process because the digital camera 100 enters the sleep state. The body CPU 109 advances the process to S1020 when the main switch 195 is ON.

  In step S1020, the body CPU 109 determines whether the mode switch 196 is set to the refocus priority shooting mode. The body CPU 109 advances the process to S1030 when the shooting mode is set to the refocus priority shooting mode, and advances the process to S1250 when the shooting mode is not set to the refocus priority shooting mode.

  In step S <b> 1030, the body CPU 109 sets a limit on the side on which the aperture value is increased (hereinafter referred to as the small aperture side) with respect to the aperture value set by the user using the aperture setting switch 194. The restriction of the aperture value on the small aperture side will be described with reference to FIG.

  As shown in FIG. 3C, the pixels 21 to 25 correspond to, for example, the pixels 21a to 25a in FIG. 3B, and light beams that have passed through the divided pupil regions are imaged. On the other hand, when the aperture diameter of the stop 140 is reduced in the photographing optical system 101, a part of the light beam that has passed through the divided pupil region is blocked, and the light beam is imaged on a limited pixel, for example, the pixel 23 (corresponding pixel 23a). The Further, as shown in FIG. 9, in order to generate a reconstructed image corresponding to a reconstructed surface different from the acquisition surface, it is necessary to receive a light beam that has passed through a plurality of divided pupil regions having different incident directions. is there. In other words, when the light beam is restricted to the central pixel below each microlens by the diaphragm 140, for example, the image shift process is restricted and a reconstructed image cannot be generated.

For this reason, an aperture value F _min for defining the lower limit value of the aperture diameter when controlling the aperture 140 is set. The aperture value F _min is set so that the lower limit value of the aperture diameter of the aperture is set to a value of an aperture diameter through which a light beam that has passed through a plurality of divided pupil regions can pass. When the divided pupil region is represented by corresponding pixels, for example, the aperture value F _min is an F number of each microlens 20a constituting the MLA 20, and f is a column of a pixel group arranged in a grid with respect to one microlens. It is defined as follows, where n is the number of pixels.
F _min = nf / 3 (when n is an odd number)
F _min = nf / 2 (when n is an even number)
In FIG. 6A, for example, when the number of pixels for one column is five (n = odd number), the pixels arranged around the pixel 23a arranged at the center of the microlens 20a emit light from the subject. It shows a state of receiving light. Each pixel indicated by hatching in FIG. 6A is a pixel that receives a light beam from a subject. That is, the light beam from the subject is incident on the pixels 22a, 24a, 23aa, 23ab, 23ac, 23ad, 23ae, and 23af arranged around the pixel 23a with the aperture diameter corresponding to the aperture value _Fmin . Similarly, in FIG. 6B, for example, when the number of pixels for one column is four (n = even number), the luminous flux from the subject is applied to the pixels 22a, 23a, 22aa, and 22ab arranged at the center of the microlens 20a. Incident. Thus, by making it possible to receive a light beam that has passed through a plurality of divided pupil regions, a reconstructed image can be generated from LF data.

The body CPU 109 sets F _min as the limit value on the small aperture side. The aperture value that can be set by the aperture setting switch 194 is, for example, that the aperture value on the open side is limited from F2.8, which is the open aperture value of the photographing optical system 101 of the lens 102, to F _min on the small aperture side. It becomes.

  In S1040, the body CPU 109 determines whether or not the release switch 191 is half-pressed. When the body CPU 109 determines that the counter-pressing operation is being performed, the process proceeds to S1050.

In S1050, the body CPU 109 reads the aperture value set by the user with the aperture setting switch 194, and determines whether the limit value of the aperture value ( _Fmin ) set in S1030 has been exceeded. That is, the body CPU 109 determines whether the aperture diameter of the aperture designated by the user is below the lower limit value of the aperture diameter that can generate the reconstructed image. The body CPU 109 advances the process to S1050 when the aperture value set by the user exceeds the limit value, and advances the process to S1300 when it does not exceed the limit value.

  In step S <b> 1300, the body CPU 109 transmits the aperture value set by the user read in step S <b> 1050 to the lens CPU 103 via the lens contact 190. The aperture control unit 106 controls the aperture drive unit 141 according to the set aperture value so that the aperture 140 has an aperture diameter corresponding to the aperture value set by the user. Thereafter, the body CPU 109 advances the process to step S1080.

  In S1060, the body CPU 109 displays a warning message such as “the aperture value has exceeded the limit value. The setting value will be changed to the limit value” in the display unit 258, and the aperture value set by the user exceeds the limit value. The body CPU 109 changes the set aperture value to the limit value and advances the process to S1070.

  In S1070, the body CPU 109 transmits the aperture value of the limit value set in S1060 to the lens CPU 103 via the lens contact 190. Then, the diaphragm control unit 106 drives and controls the diaphragm driving unit 141 in accordance with the diaphragm value of the limit value so that the diaphragm 140 has an aperture diameter corresponding to the diaphragm value that is the set limit value. Thereafter, the body CPU 109 advances the process to S1080.

  In S1080, the body CPU 109 obtains data by exposing the image sensor 252 for an appropriate time and reading (A / D conversion). Note that an appropriate exposure amount in photographing may be calculated from the exposure time and the exposure amount.

  In S1090, the body CPU 109 performs correlation calculation processing. In the correlation calculation process, a focus position by so-called phase difference AF can be obtained. In the correlation calculation process, calculation is repeatedly performed in accordance with a preset number of evaluation positions (so-called number of AF detection frames). In the correlation calculation process, the correlation calculation is performed within the range of the number of pixels corresponding to the size of the appropriately set evaluation frame (evaluation target region). The correlation calculation may be performed by, for example, Σ | Ai−Bi |. Here, Ai indicates the luminance of the i-th pixel that has passed through a specific pupil region. Bi represents the luminance of the i-th pixel that has passed through a pupil region different from Ai. For example, in FIG. 3, the pixel array corresponding to the pixel 22 may be Ai, and the pixel array corresponding to the pixel 24 may be Bi. Further, the correlation calculation evaluation values are obtained by the number of preset evaluation points (corresponding to the focus depth at which the focus search is performed). In the evaluation formula Σ | Ai−Bi | shown above, the location where the correlation value is small corresponds to the location with the best focus state. The point with the best correlation evaluation value is stored as the best focus position based on the correlation amount evaluation.

  In S1100, the body CPU 109 calculates the defocus amount in each AF detection frame 301 from the focus position obtained in S1090.

  In S1110, the body CPU 109 determines the subject that the user is the main subject from the defocus amount in each AF detection frame 301 obtained in S1100. Specifically, the body CPU 109 determines, for example, the AF detection frame 301 that has a specific value and a defocus amount in the vicinity thereof from the distribution state of the defocus amount in each AF detection frame 301 and the difference between the absolute values. Identify areas with large numbers. The body CPU 109 determines that the subject in the area is the main subject 310 and displays the subject with the smallest defocus amount in the area as the AF detection frame 302 at the focus position (see FIG. 4). Alternatively, when the user designates the AF detection frame 302 at a position to be focused by the selection switch 192 as described above, the body CPU 109 performs AF with a defocus amount similar to the defocus amount in the designated AF detection frame 302. A detection frame may be specified. At this time, the subject in the area where the AF detection frame is distributed may be determined as the main subject 310. When the user designates the main subject 310, if the display such as “Please select / determine the main subject” is displayed in the body CPU 109 display unit 258, the convenience for the user is improved. Also, a subject (for example, subjects 330 and 340) that is outside the AF detection region (the region where the AF detection frame 301 is not disposed) within the photographing range (subject image 258a in FIG. 4) is also processed by the processing of S1060. Focus amount is obtained. Therefore, the body CPU 109 may store the defocus amount of subjects other than the main subject 310 in the memory 197.

  In S1120, the body CPU 109 transmits the calculation result of S1110 to the lens CPU 103 via the lens contact 190, and drives the AF lens 120. The body CPU 109 moves the AF lens 120 to a position where the main subject 310 obtained in S1110 is in a focused state.

  In step S <b> 1130, the image processing unit 150 performs image generation processing in accordance with an instruction from the body CPU 109. Details of the operation of the image generation processing will be described later with reference to FIG. 8, and an image of the main subject 310 formed on the image sensor 252 is thereby generated in the memory 197.

  In S1140, the body CPU 109 displays the image focused on the main subject 310 generated in S1130 on the display unit 258 as a preview image before photographing.

  In S1150, the body CPU 109 determines whether the aperture value has been changed by the user operating the aperture setting switch 194. This is because, after confirming the preview image displayed on the display unit 258 in S1140, the user may shoot after changing the aperture value before shooting. The body CPU 109 compares the already obtained aperture value with the aperture value of the aperture setting switch 194, and if the aperture value is different, the process proceeds to S1050, and if not changed, the process proceeds to S1150.

  In S1160, the body CPU 109 determines whether or not the release switch 191 has been fully depressed (whether or not SW2 is turned on). The body CPU 109 returns to S1060 when the release switch 191 is not fully depressed, and proceeds to S1170 when all the release switches 191 are depressed.

  In S <b> 1170, the body CPU 109 controls the photographing operation, and the image of the main subject 310 formed on the image sensor 252 is recorded as a recorded image processed by the image processing unit 150 through a known image processing technique, or the memory 197 or not illustrated. Record on a recording medium.

  In S1180, the body CPU 109 displays the recorded image generated in S1170 on the display unit 258. Thereby, the user can confirm and acquire an image focused on the main subject 310 immediately after shooting.

  In S1190, the body CPU 109 determines whether or not the main switch 195 is turned off. If the main switch 195 is turned off, the process returns to S1040. If the main switch 195 is turned on, the series of processes is terminated. To do.

  On the other hand, in S1250, the body CPU 109 releases the restriction on the aperture value. As a result, the user can photograph the subject with the aperture value intended by the user.

  In S1260, a normal photographing operation is performed and the process is terminated. Since the operation in S1260 is a known operation, detailed description thereof is omitted.

(4 Image generation processing operations)
The operation of the above-described image generation process will be described with reference to FIG. In this embodiment, a reconstructed image is generated for previewing the acquired LF data in S1140. For this reason, an image in which the incident light beam is reproduced on the acquisition surface is generated, so that the image generation process is performed without performing the image shift process.

  In step S <b> 1610, the image processing unit 150 initializes the area data for the addition process in step S <b> 1640 (fills with 0). The size of the data area at this time may be the number of micro lenses of the MLA 20.

  S1620 to S1660 form a loop based on the number of micro lenses of MLA20. In S1620, a loop calculation is executed according to the number of microlenses constituting the MLA 20. For example, in the example illustrated in FIG. 3, the number of micro lenses is the number obtained by dividing the number of pixels of the original image sensor by 25 (number of pupil divisions). When the image processing unit 150 performs the loop processing for the number of microlenses, the process proceeds to S1670.

  S1630 to S1660 further form a loop according to the number of pupil divisions. When the image processing unit 150 performs the loop processing for the number of pupil divisions, the process proceeds to S1660. That is, the process returns to S1620 again.

In step S1640, the image processing unit 150 adds the pixel values of the pixels corresponding to the microlens that is the processing target. When the loop calculation is executed by the number corresponding to the number of pupil divisions, for example, in the example shown in FIG. 3, light beams from 25 pupil positions are received with respect to the pixels divided into 25 pixels. Pixels are added. In the example of FIG. 9, the sum S _{i of} pixel values is obtained for each microlens X _i . If the total shift amount of the pixel values is not an integer multiple of the pixels, in S1640, addition is performed while being appropriately divided (addition may be appropriately performed according to the overlapping area).

  In step S1670, the process returns to the calling source S1130.

  The series of operations in the present embodiment has been described above. In this embodiment, the limit value of the aperture value is defined by the F number of the microlens and the number of pixels that receive the light beam. However, the method for setting the limit value of the aperture value is not limited to this. The limit value may be set based on the feature amount of the subject obtained by image analysis. For example, it is conceivable to limit the aperture value within a range where the refocusing of the face area is possible based on the subject distance and depth information of the face area obtained by a known face detection process.

  As described above, in this embodiment, the lower limit value of the aperture diameter of the stop is set to a value that allows a light beam that has passed through a plurality of divided pupil regions to pass, so that reconfigurable LF data can be acquired. . That is, it is possible to output LF data that can generate images with different conditions after shooting. Further, in this embodiment, when the refocus priority shooting mode is set, the aperture value is limited to an aperture value that can be reconstructed LF data. This makes it possible to output LF data that can generate images with different conditions after shooting even when the user attempts to acquire LF data by increasing the aperture value.

  Furthermore, in this embodiment, when the aperture value is limited to an aperture value that can be reconstructed LF data, the user is notified. Thus, even when the user performs an operation with an increased aperture value, the user can recognize that LF data capable of generating an image with different conditions is output after shooting.

(Embodiment 2)
In the first embodiment described above, a method has been described in which when the refocus priority shooting mode is set, control is performed so that the user cannot set an aperture value at which a reconstructed image with different conditions cannot be generated from LF data. In the present embodiment, a method for recording LF data capable of generating a reconstructed image under different conditions while presenting an image corresponding to a user's desired aperture value while allowing the user to set a desired aperture value Will be described. In the description of the present embodiment, the functional configuration of the digital camera 100 is assumed to be the same as that of the first embodiment, and the description thereof is omitted.

(1 Outline of generation of reconstructed image when aperture is set)
The principle of generating a corresponding reconstructed image from LF data when a user has set an aperture value at which a reconstructed image with different conditions cannot be generated from LF data will be described with reference to FIG.

  FIG. 11 shows the relationship between the aperture diameter of the diaphragm 140 and the luminous flux from the object (subject) imaged on the image sensor 252. In FIG. 11, 51 indicates an object plane, 51a and 51b indicate appropriate points on the object, 52 indicates a pupil plane of the photographing optical system 101, and 61 and 62 indicate specific microlenses on the MLA 20, respectively. . Further, the imaging element 252, the MLA 20, and the areas 31 to 35 correspond to those shown in FIG.

  Illustrated by a solid line is a light beam that is emitted from the point 51a on the object and passes through the regions 31 and 33 on the pupil plane, and is represented by a broken line that is emitted from the point 51b on the object and passes through the regions 31 and 33 on the pupil plane. did. In the example of FIG. 11, as described with reference to FIG. 3, the MLA 20 is arranged in the vicinity of the imaging plane of the photographing optical system 101, so that the imaging device 6 and the pupil plane 52 of the photographing optical system 101 are in a conjugate relationship. . Furthermore, the object plane 51 and the MLA 20 are in a conjugate relationship. Therefore, the light beam emitted from the point 51a on the object reaches the microlens 61, the light beam emitted from the point 51b reaches the microlens 62, and the light beams that have passed through the regions 31 to 35 respectively correspond to those provided below the microlens. Reach the pixel.

  For example, when the aperture diameter of the aperture 140 is set to a predetermined aperture value, the luminous flux from the subject through the aperture diameter is the area indicated by the arrow 71 on the pupil plane 52 (all the areas 32 to 34 and the area 31 and 35). The light beams that have passed through the pupil plane 52 reach the corresponding pixels (pixels 21 to 25 in FIG. 3C) under the MLA 20. By performing image generation processing on the information of the pixels that receive each light beam, it is possible to obtain a subject image having a depth of field corresponding to a set predetermined aperture value.

  On the other hand, by performing image shift processing, a reconstructed image equivalent to a state in which the aperture is reduced can be obtained. It is assumed that a virtual aperture diameter corresponding to a desired reconstructed image aperture value is given by an arrow 72. The luminous flux from the subject becomes a luminous flux that passes through the area indicated by the arrow 72 (almost all of the areas 32 to 34) on the pupil plane 52. The luminous flux reaches the corresponding pixels (pixels 22 to 24 in FIG. 3C) under the MLA 20 respectively. That is, a reconstructed image equivalent to a state where the aperture is further reduced is generated by using a pixel value corresponding to the light beam that has virtually reached. That is, a reconstructed image can be generated using information of the pixels 22 to 24 among the pixels 21 to 25 that actually receive the light flux. Therefore, it is possible to obtain a reconstructed image having a smaller aperture diameter by using the pixel information that is actually acquired by limiting the aperture to a larger aperture diameter. In other words, a reconstructed image that has been changed to the depth of field set by the user after shooting can be obtained.

  FIG. 10 is a flowchart showing a series of processing when shooting is performed in the refocus priority shooting mode according to the second embodiment. Hereinafter, the operation when the user performs the aperture value control process after shooting will be described with reference to FIG.

  The operations from S3010 to S3060, and S3310, S3320, and S3400 correspond to S1010 to S1060, and S1250, S1260, and S1300 described in the first embodiment, respectively, and thus the description thereof is omitted.

  In S3070, the body CPU 109 stores the aperture value set by the user used for comparison in S3050 (S1050) in the memory 197, and advances the process to S3080.

  Since the operations from S3080 to S3190 correspond to S1070 to S1180 described in the first embodiment, the description thereof is omitted.

  In step S3200, the body CPU 109 waits for an instruction from the user to change whether or not to change the depth of field for the captured image focused on the main subject captured in step S3180. Specifically, an image focused on the main subject 310 is displayed on the display unit 258 and a display such as “Do you want to change the aperture value?” Is displayed to change the depth of field by the user. Check whether or not. At this time, “Yes” and “No” are displayed, and if “Yes” is selected by the selection switch 192 or the like, the process proceeds to S3210. On the other hand, when the display of “No” is selected, the body CPU 109 advances the process to S3290.

  In step S3210, the body CPU 109 acquires the aperture value set by the user and stored in the memory 197 in step S3070.

  In S3220, the body CPU 109 determines whether the aperture value set by the user exceeds the limit value on the small aperture side. If it is determined that the limit value is exceeded, the process proceeds to S3220. If it is determined that the limit value is not exceeded, the process proceeds to S3500. In step S3500, when the body CPU 109 acquires the aperture value input by the user using the aperture setting switch 194, the process proceeds to step S3230.

  In step S3230, the body CPU 109 performs image shift processing on each microlens position to generate a reconstructed image corresponding to the aperture value using the aperture value set by the user or the aperture value acquired in step S3500. The operation of the image shift process is shown in FIG. The body CPU 109 advances the process to step S1710, and causes the image processing unit 150 to execute image shift processing.

  S1710 to S1760 form a loop, and the loop calculation is executed by the number corresponding to the number of pupil divisions. For example, in the example shown in FIG. 3, the pupil region is divided into 25 areas, so that calculations are performed according to the 25 pupil positions. As described with reference to FIG. 9, in the generation of a reconstructed image, the amount of image shift differs when the incident angle is different even on the same reconstructed surface. The loop processing according to the number of pupil divisions is for calculating the shift amount of each pixel according to each incident angle.

  In step S1710, the image processing unit 150 determines whether the processing has been performed for the number of pupil divisions. If the number of processes is smaller than the number of pupil divisions, the process proceeds to step S1730. If the number of processes reaches the number of pupil divisions, The process ends the loop and proceeds to S1770.

  In step S1720, the image processing unit 150 acquires pupil position information. The pupil position information is information indicating which division pupil region receives the light beam in which the correspondence between each pixel on the image sensor 252 and the MLA 20 is stored.

  In step S <b> 1730, the image processing unit 150 calculates the image shift amount in each divided pupil region currently being processed based on the acquired pupil position information.

  In S1740, pixels that have obtained light rays having the same incident angle (obtained light rays from the same divided pupil region) are shifted based on the information in S1730. For example, pixels 25a and 25b in FIG. 3B correspond to pixels that obtain light rays having the same incident angle. There are as many such pixels as the number of microlenses constituting the MLA 20.

  In step S1760, the image processing unit 150 returns the processing to the calling source S3230.

  In S3240, the image processing unit 150 operates the image generation process using the pixels shifted in S3230. As a result, a reconstructed image having a depth of field obtained by the aperture value set by the user is generated.

  In S3250, the body CPU 109 displays the reconstructed image generated in S3240 on the display unit 258 as a preview image.

  In step S3260, the body CPU 109 waits for an input from the user with respect to the displayed reconstructed image. Specifically, together with the image with the changed depth of field displayed on the display unit 258, for example, “Are you sure you want to use this depth of field?” It is confirmed whether an image having a depth of field is obtained. At this time, when “OK” and “Cancel” are further displayed and “OK” is selected by the user, the body CPU 109 determines that an image having the depth of field desired by the user has been obtained, The process proceeds to S3270. On the other hand, if “cancel” is selected by the user, it is determined that an image having the depth of field desired by the user has not been obtained, and the process proceeds to S3600.

  In step S3270, the body CPU 109 confirms whether the generated reconstructed image is to be stored in the memory 197 or a recording medium (not illustrated). Specifically, the body CPU 109 displays the reconstructed image on the display unit 258 and also displays, for example, “Do you want to save?” To confirm whether the reconstructed image is to be saved. At this time, the body CPU 109 further displays “Yes” and “No”, and if “Yes” is selected by the user, the process proceeds to S3280. On the other hand, if “NO” is selected by the user, the body CPU 109 advances the process to step S3700.

  In S3280, the body CPU 109 stores the reconstructed image in the memory 197 or a recording medium (not shown) with a name different from the name of the image focused on the main subject 310 recorded in S3180, and advances the process to S3290.

  In S3290, the body CPU 109 performs the same process as S1190, and ends the series of operations.

  On the other hand, in S3600, the body CPU 109 determines whether or not the user has changed the aperture value by operating the aperture setting switch 194. If the body CPU 109 determines that the aperture value has been changed from the signal from the setting switch 194, the process returns to S3230 and a series of operations are repeated. On the other hand, when the aperture value has not been changed, the process returns to S3600 again to wait for input from the user. In step S3700, the body CPU 109 deletes the reconstructed image generated in the memory 197 in step S3240, and proceeds to step S3290.

  In the second embodiment, the process is executed after confirming whether the user has changed the depth of field after shooting. However, for example, when the aperture value set by the user is smaller than the limit value, the user convenience is further improved by setting a shooting mode that automatically changes the depth of field after shooting. Can do. In the present invention, the digital camera 100 and the lens 102 are detachable. However, the present invention is not limited to this, and a digital camera in which the digital camera 100 and the lens 102 are integrated can provide the same effects as described above. Needless to say.

  As described above, in the present embodiment, reconfigurable LF data is acquired by setting the lower limit value of the aperture diameter of the diaphragm to a value that allows the light beam that has passed through the plurality of divided pupil regions to pass through. A reconstructed image was generated based on an instruction from the user. Thereby, even when the user increases the aperture value beyond the predetermined aperture value at the time of shooting, the user's desired reconstructed image can be generated.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

DESCRIPTION OF SYMBOLS 100 ... Digital camera, 109 ... Body CPU, 140 ... Aperture, 141 ... Aperture drive part, 196 ... Mode switch, 252 ... Image sensor, 258 ... Display part

Claims (7)

An image pickup apparatus that outputs a pixel signal that records a light flux that has passed through each of the divided pupil regions and that can generate a reconstructed image focused on any of a plurality of subject distances,
Control means for controlling the lower limit value of the aperture diameter of the diaphragm for photographing the pixel signal to a value at which the light flux that has passed through the plurality of divided pupil regions can pass through the aperture ;
Changing means for changing the aperture diameter of the diaphragm,
In the case where a photographing mode for outputting a pixel signal capable of generating a reconstructed image with a plurality of different conditions is set, the changing means is set when the aperture diameter of the diaphragm that is lower than the lower limit value is set by the user. An imaging apparatus , wherein the aperture diameter of the diaphragm is changed to the lower limit value . The imaging according to claim 1 , further comprising notification means for notifying that the aperture diameter of the diaphragm has been changed when the aperture diameter of the diaphragm is changed to the lower limit value by the changing means. apparatus. Imaging means for performing imaging and outputting the pixel signal;
The imaging apparatus according to claim 1 or 2, characterized by further comprising a generating means for generating a reconstructed image from the pixel signal outputted by the imaging means. When the aperture diameter of the diaphragm that is lower than the lower limit value is set by the user, the generation unit reconstructs the pixel signal that is captured and output with the aperture diameter of the lower limit value and that corresponds to the set aperture diameter The imaging apparatus according to claim 3 , wherein an image is generated. Further comprising display means for displaying the reconstructed image generated by the generating means;
The display means displays a reconstructed image corresponding to the set aperture diameter after shooting when the user sets a diaphragm aperture diameter lower than the lower limit value and performs shooting. The imaging device according to claim 4 . A method of controlling an imaging apparatus that outputs a pixel signal that records a light flux that has passed through each of divided pupil regions and that can generate a reconstructed image focused on one of a plurality of subject distances. ,
A control step in which the control means controls the lower limit value of the aperture diameter of the diaphragm when photographing the pixel signal to a value at which the light beam that has passed through the plurality of divided pupil regions can pass through the aperture ;
A changing step of changing the aperture diameter of the diaphragm,
In the changing step, when a photographing mode for outputting a pixel signal capable of generating a reconstructed image having a plurality of different conditions is set, and when an aperture diameter of the diaphragm lower than the lower limit value is set by the user A method for controlling an imaging apparatus , wherein the aperture diameter of the diaphragm is changed to the lower limit value . The program for functioning a computer as each means of the imaging device of any one of Claims 1 thru | or 5 .

Images