JP2013254114A - Imaging device, control method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To speed up an autofocus operation in an imaging device capable of generation of an image focused at any focus distance after imaged.SOLUTION: An imaging device is configured to: set the number of pixels of a generation reconstruction image for focus evaluation; determine pixels of a reconstruction image capable of generation to be used in generation of each pixel of a generation reconstruction image from the set number of pixels and the upper limit number of pixels of a reconstruction image capable of generation from pixels of an image signal; and as to each pixel of the generation reconstruction image, of the pixels of the image signal corresponding to pixels of the reconstruction image capable of generation to be used in generation, select a predetermined number of pixels as pixels to be used in generation. Then, when the set number of pixels of the generation reconstruction image is smaller than the upper limit number of pixels, the imaging device selects pixel of at least one corresponding image signal as to each pixel of the generation reconstruction image capable of generation to be used in generation.

Description

  The present invention relates to an imaging apparatus, a control method, and a program, and more particularly to a technique for generating an image focused on an arbitrary focal length after shooting.

  In an imaging apparatus such as a digital camera, shooting is performed by adjusting the focus according to a subject to be shot. A general imaging apparatus in recent years has a so-called autofocus (AF) function for automatically performing such focus adjustment. Focus detection methods used for AF include, for example, a phase difference detection method and a contrast detection method. The former is mainly used for a digital single-lens reflex camera, and the latter is mainly used for a compact digital camera.

  The contrast detection method detects the in-focus state based on the signal strength of the obtained image signal, etc., so the focal length is in focus compared to the phase difference detection method that detects image displacement between images obtained for different optical axes. Can be detected strictly. On the other hand, in focus focal length detection using a contrast detection method, it is necessary to perform focus evaluation on a plurality of images taken with different focal lengths. For this reason, the detection of the focal focal length using the contrast detection method requires an evaluation time for a plurality of images and the drive time of the focus lens, and the detection requires more time than the phase difference detection method.

  On the other hand, in Patent Document 1, in the focus focal length detection using the contrast detection method, an image signal obtained by reducing the number of pixels read out for the captured image signal is used for focusing evaluation. An imaging apparatus that reduces the time required for the focus evaluation process is disclosed.

  On the other hand, in recent years, there is an imaging apparatus that can generate an image focused on an arbitrary focal length after shooting. In such an imaging apparatus, microlenses are assigned to a plurality of photoelectric conversion elements in the imaging element, and different pupil regions of the imaging optical system are assigned to each of the photoelectric conversion elements associated with one microlens. The light beam that has passed through is imaged. By rearranging the output signals of the pixels of the image sensor obtained by the imaging device and adding the pixel values for the number of pupil divisions, the pixels of the image focused on an arbitrary focal length are generated.

  Patent Document 2 discloses an image for focus detection focused on a plurality of focal lengths from an output signal obtained by one imaging in an imaging apparatus capable of generating an image focused on an arbitrary focal length after shooting. A method of detecting a focal length that is focused on a subject to be imaged by generating is disclosed.

JP 2008-199477 A JP 2009-258610 A

  However, the method of detecting the focal length in the imaging apparatus as in Patent Document 2 rearranges the pixels of the output signal obtained by one imaging in accordance with the focal length, and uses the rearranged output signal. An image for focus detection is generated. For this reason, since it takes time to generate a pixel for focus detection and focus evaluation on the image to generate an image for focus detection that focuses on one focal length, detection of the focal length using a contrast detection method The shortening of the time required for this was not sufficient.

  On the other hand, it is conceivable to reduce the time required for generating and focusing evaluation of one focus detection image by thinning out pixels read out from the image sensor as in Patent Document 1 and performing focus evaluation. . However, a method for generating a thinned image suitable for focus detection has not been proposed so far for an imaging apparatus capable of generating an image focused on an arbitrary focal length after shooting.

  The present invention has been made in view of the above-described problems, and an imaging apparatus, a control method, and an imaging apparatus that speed up an autofocus operation in an imaging apparatus capable of generating an image focused at an arbitrary focal length after shooting. The purpose is to provide a program.

In order to achieve the above object, an imaging apparatus of the present invention has the following configuration.
An imaging unit that images a subject and outputs the obtained image signal, each of the pixels of the image signal corresponding to a light flux in which a combination of a pupil region and an incident direction of the imaging optical system that has passed is different Determining a focal length of a subject to be focused in the generated reconstructed image, set for a generated reconstructed image generated from the image signal output by the imaging means, and a focal length determined by the determining means Accordingly, a generation unit that generates a generated reconstructed image focused on a focal length by adding pixel values of pixels of a predetermined number of image signals corresponding to each pixel of the generated reconstructed image, and generated by the generation unit An imaging device comprising: an evaluation unit that performs focusing evaluation on the generated generated reconstructed image; and a setting unit that sets the number of pixels of the generated reconstructed image generated by the generating unit, and a setting unit The reconfigurable image pixels that can be used to generate each pixel of the generated reconstructed image are determined from the set number of pixels of the generated reconstructed image and the upper limit number of reconstructed images that can be generated from the pixels of the image signal. And selecting means for selecting a predetermined number of pixels among the pixels of the image signal corresponding to the pixels of the reconfigurable reconfigurable image used for generation for each pixel of the generated reconstructed image, When the number of pixels of the generated reconstructed image set by the setting means is smaller than the upper limit number of pixels, at least one corresponding image signal pixel is selected for each of the reconfigurable image pixels used for generation It is characterized by that.

  With such a configuration, according to the present invention, it is possible to speed up the autofocus operation in the imaging apparatus capable of generating an image focused at an arbitrary focal length after shooting.

The block diagram which showed the function structure of the camera system which concerns on embodiment of this invention The figure for demonstrating the correspondence of the micro lens array which concerns on embodiment of this invention, and an image pick-up element. The flowchart which illustrated AF process performed with the digital camera 100 which concerns on embodiment of this invention 6 is a flowchart illustrating various processes executed by the digital camera 100 according to the embodiment of the invention. The figure for demonstrating the production | generation method of the reconstruction image which concerns on embodiment of this invention. The figure for demonstrating the production | generation method of the thinning | reconstruction reconstructed image which concerns on embodiment of this invention. The figure which showed the structural example of the optical system which can apply embodiment of this invention The figure for demonstrating the AF detection operation of the conventional contrast detection type accompanying a lens drive

[Embodiment]
Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. In addition, one embodiment described below describes an example in which the present invention is applied to a camera system that can generate an image focused on an arbitrary focal length after shooting as an example of an imaging apparatus. However, the present invention can be applied to any device capable of generating an image focused on an arbitrary focal length and performing a focus detection operation.

In this specification, the following terms are defined and described.
・ "Light Field (LF) data"
The image signal output from the imaging part 102 which the digital camera 100 of this embodiment has. Each pixel of the image signal indicates a signal intensity corresponding to a light beam having a different combination of the pupil region and the incident direction of the imaging optical system 202 that has passed. The LF data is also called ray space information.
・ "Reconstructed image"
An image generated from LF data and focused at an arbitrary focal length. Specifically, the pixels of the LF data are rearranged according to the pixel arrangement on the focal plane (reconstruction plane) corresponding to the focal length to be generated, and the pixel values of the pixels corresponding to one pixel of the reconstructed image are summed up. Get the pixel value of the pixel. The number of pixels to which the normal pixel values are added is the pupil division number.

《Camera system configuration》
FIG. 1 is a block diagram showing functional configurations of a digital camera 100 and a lens 200 included in a camera system according to an embodiment of the present invention.

<Configuration of digital camera 100>
The camera control unit 101 is a CPU, for example, and includes a ROM and a RAM (not shown). The camera control unit 101 controls the operation of each block of the digital camera 100 by reading an operation program for auto focus (AF) processing, which will be described later, stored in the ROM, developing it in the RAM, and executing it. In addition, the camera control unit 101 sends information on the focal position determined by analysis of the image signal output from the imaging unit 102 and information on the aperture value corresponding to the determined exposure setting to the lens 200 via the electrical contact 107. Send.

  The imaging unit 102 is an imaging element such as a CCD or a CMOS sensor. The imaging unit 102 performs exposure and readout of each photoelectric conversion element included in the imaging device based on the timing signal generated by the camera control unit 101 based on the set exposure setting, and an analog image of the obtained LF data The signal is output to the image processing unit 103. Specifically, the imaging unit 102 photoelectrically converts an optical image formed on the light receiving surface of the imaging element by the imaging optical system 202 of the lens 200, and outputs an analog image signal.

  Further, in the digital camera 100 of the present embodiment, the microlens array 108 in which the microlenses 20 are arranged in a lattice shape as shown in FIG. One microlens 20 of the microlens array 108 is associated with a plurality of photoelectric conversion elements (pixels) of the image sensor 109 as shown in FIG. The light beam that has entered through the imaging optical system 202 of the lens 200 is divided into pupils by being imaged by the microlenses 20 on the pixels of the associated image sensor 109. That is, the light flux that has passed through the pupil region of the imaging optical system 202 corresponding to the associated pixel position is imaged on each pixel of the associated image sensor 109. In the example of FIG. 2B, 4 × 4 = 16 pixels are associated with one microlens 20, and the number of pupil divisions of the imaging optical system 202 is 16.

  FIG. 2C is a diagram illustrating a correspondence relationship between a pixel associated with one microlens and a pupil region of an exit pupil of the imaging optical system 202 through which a light beam formed on each pixel passes. 2C, for the sake of simplicity, it is assumed that four pixels 21 to 24 in the horizontal direction with respect to one microlens 20 are arranged side by side on a horizontal line passing through the center of the microlens 20. . At this time, each pixel is designed by the microlens 20 so as to be conjugate with the pupil regions 31 to 34 on the exit pupil plane. In the example of FIG. 2C, the pixel 21 has a pupil region 31, the pixel 22 has a pupil region 32, the pixel 23 has a pupil region 33, and the pixel 24 has a conjugate relationship with the pupil region 34.

  The image processing unit 103 performs predetermined image processing on the image signal of the LF data output from the imaging unit 102. Specifically, the image processing unit 103 performs A / D conversion processing, white balance adjustment processing, gamma correction processing, interpolation calculation processing, and the like on the input analog image signal, and generates a recording image (LF data). . In the present embodiment, the image processing unit 103 also performs processing for generating a reconstructed image from the obtained LF data. Further, the image processing unit 103 executes compression processing of data such as an image, a moving image, and sound according to a predetermined encoding method. Various image processes may be realized by a dedicated circuit or the like.

  The memory 104 includes a memory element and a processing circuit that reads from and writes to the memory element. The memory 104 outputs to the storage element and stores an image to be output to the display unit 105. The memory 104 stores encoded images, moving images, audio data, and the like.

  The display unit 105 is a display device provided in the digital camera 100 such as an LCD. The display unit 105 displays a reconstructed image or the like generated from LF data obtained by imaging.

  The operation detection unit 106 detects an operation performed on a user interface such as a release button of the digital camera 100. Specifically, when the operation detection unit 106 detects that the user has operated, for example, a release button, the operation detection unit 106 outputs a control signal corresponding to the operation to the camera control unit 101.

<Configuration of lens 200>
The lens control unit 201 is a CPU, for example, and incorporates a ROM and a RAM (not shown). The lens control unit 201 controls the operation of each block by reading the operation program of each block included in the lens 200 stored in the ROM, developing the program in the RAM, and executing the program. When the lens control unit 201 receives information on the focal position and the aperture value from the camera control unit 101 via the electrical contact 107, the lens control unit 201 transmits the information to the lens driving unit 203 and drives the corresponding optical member of the imaging optical system 202. Let

  The imaging optical system 202 includes a lens group, a diaphragm, and the like that the lens 200 has. In the present embodiment, the imaging optical system 202 includes at least a focus lens, a shift lens, and a diaphragm. The lens driving unit 203 performs drive control of the focus lens, the shift lens, and the diaphragm of the imaging optical system 202 according to the information input from the lens control unit 201. Note that a camera shake detection sensor (not shown) is connected to the lens control unit 201 of this embodiment, and the lens driving unit 203 drives the shift lens according to the output of the sensor input from the lens control unit 201.

<AF processing>
A specific process of the AF process of the digital camera 100 of the present embodiment having such a configuration will be described with reference to the flowchart of FIG. The process corresponding to the flowchart can be realized by the camera control unit 101 reading, for example, a corresponding processing program stored in the ROM, developing it in the RAM, and executing it. This AF processing is started when the camera control unit 101 detects that the release button is half-pressed while the digital camera 100 is activated in the shooting mode, for example, based on a control signal from the operation detection unit 106. It will be described as being done.

  In step S <b> 301, the camera control unit 101 causes the imaging unit 102 to image a subject and acquires LF data. Specifically, the camera control unit 101 acquires the LF data by causing the image processing unit 103 to apply A / D conversion processing to the analog image signal related to the LF data output from the imaging unit 102 with a predetermined exposure setting. To do.

  In step S <b> 302, the camera control unit 101 executes an evaluation distance determination process for determining a focal distance that is an evaluation target for focus evaluation. That is, the camera control unit 101 determines the focal length of the subject to be focused in the generated reconstructed image for the reconstructed image (generated reconstructed image) generated for the focus evaluation.

<Evaluation distance determination process>
Here, the details of the evaluation distance determination process of the present embodiment will be described using the flowchart of FIG.

  In step S401, the camera control unit 101 determines whether it is the first focal length determination related to the focus evaluation. That is, when the AF process is executed, the camera control unit 101 determines whether or not the evaluation distance determination process has been executed in order to determine the focal distance for which the focus evaluation is performed first. That is, the camera control unit 101 determines whether or not to generate a reconstructed image for the first time after performing focus evaluation on the acquired LF data. If the camera control unit 101 determines that it is the first determination of the focal length for the focus evaluation, it moves the process to S402, and if it determines that it is the second time or later, it moves the process to S405.

  In step S <b> 402, the camera control unit 101 determines whether it is possible to acquire information on the focal length set at the previous shooting. The focal length information set at the time of the previous shooting may be acquired from, for example, LF data stored in the memory 104 or may be stored in the RAM. Note that the focal length information set at the time of the previous shooting may be the focal length at which the subject determined by the previous AF process is focused. If the camera control unit 101 determines that the information on the focal length set at the previous photographing can be acquired, the process proceeds to S403. If the camera control unit 101 determines that the information cannot be acquired, the process proceeds to S404.

  In step S <b> 403, the camera control unit 101 sets the focal length (initial value) to be evaluated according to a predetermined rule based on the focal length set at the previous shooting, and completes the focal length determination process.

  In step S404, the camera control unit 101 sets the focal length of the evaluation target as the focal length of the imaging surface, that is, the LF data acquired in step S301 of the AF process, as the focal length (initial value) at the time of the previous shooting. Complete the decision process.

  If it is determined in S401 that it is the second or later focal length determination related to the focus evaluation, in S405, the camera control unit 101 performs a rough focus search in which the current focal point search is performed with a rough interval for changing the focal length of the evaluation target. It is determined whether the search is a key search or a fine search performed in detail. In this embodiment, the camera control unit 101 first searches for a change behavior of a rough contrast value in a coarse tone search for the focal length, and receives the result for a focal length range in which an appropriate in-focus position is considered to exist. By performing the fine adjustment search, the processing time required for the AF processing is shortened. This is the same as the method of detecting the focal length at which the subject is focused by performing lens driving as shown in FIG. 8, and this is realized without lens driving in the camera system of this embodiment. If the camera control unit 101 determines that the current focus search is a coarse search, the process proceeds to S406. If the camera control unit 101 determines that the current focus search is a fine search, the process proceeds to S407.

  In step S <b> 406, the camera control unit 101 obtains the currently set evaluation target focal length by adding or subtracting the coarse interval search focal length setting interval value according to the search direction. The focal length is set to the evaluation target focal length and the focal length determination process is completed. Note that the setting interval of the focal length for coarse tone search is preferably about 10 to 25 FΔ (F: F number, Δ: 20 μm), for example.

  In step S <b> 407, the camera control unit 101 adds or subtracts the value of the setting interval of the focal length for fine tuning search to the currently set focal length of the evaluation target in accordance with the search direction, and obtains the obtained focus. The distance is set to the focal length of the evaluation target, and the focal length determination process is completed. Note that the setting interval of the focal length for fine tuning search is preferably about 1 to 5 FΔ, for example.

  In the present embodiment, whether the search is a coarse search or a fine search is determined by, for example, storing information on the current search method in the RAM and referring to the information by the camera control unit 101. Although described as a thing, it is not restricted to this. For example, when the AF processing is started, the coarse tuning search value is set as an initial value as the focal length setting interval value, and the camera control unit 101 determines that the fine tuning search may be shifted to the value. The value may be changed to a predetermined fine tuning search value. In this case, the camera control unit 101 may determine whether the current focus search is a coarse search or a fine search based on whether the value of the focal distance setting interval is equal to or greater than a threshold value.

  After determining the focal length that is the evaluation target of the focus evaluation in this way, the camera control unit 101 moves the AF process to S303.

  In step S <b> 303, the camera control unit 101 executes a pixel selection process for determining pixels of LF data used for generating a reconstructed image for performing focus evaluation. In other words, in this step, the camera control unit 101 determines pixels that are not used in the generation of the reconstructed image for which the focus evaluation is performed, that is, pixels to be thinned out, and determines other pixels as pixels that are used for generating the reconstructed image.

<Pixel selection processing>
Here, the pixel selection processing of the present embodiment will be described in detail using the flowchart of FIG.

  In step S <b> 411, the camera control unit 101 acquires pixel arrangement information of LF data on the reconstruction plane corresponding to the focal length for which focus evaluation is performed.

  For example, as shown in FIG. 5A, the light beam that has passed through the areas 1 to 4 of the exit pupil forms an image on the corresponding photoelectric conversion element of the imaging element 109 with respect to one microlens of the microlens array 108. Consider the case. In this embodiment, since a color image is generated by performing development processing from the reconstructed image, any of a plurality of color filters is applied to the microlens of the microlens array 108 according to a specific arrangement. In the example of FIG. 5A, the relationship between the microlens to which the R or G color filter arranged on one horizontal line in the microlens array 108 is applied and the pixel of the image sensor 109 is shown. ing. In the figure, the hatched pixel corresponds to the color signal corresponding to R, and the non-hatched pixel corresponds to the color signal corresponding to G.

  As shown in FIG. 5B, for example, when a reconstructed image is generated for the focal length corresponding to the imaging surface 501, each pixel of the reconstructed image corresponds to the sum of light fluxes passing through one microlens. To do. That is, a light beam that passes through each pupil region (pupil regions 1 to 4 in the illustrated example) of the imaging optical system 202 and reaches one microlens forms a pixel corresponding to the microlens. . For this reason, each pixel of the reconstructed image having the focal length corresponding to the imaging surface 501 is obtained by adding the pixel values of the pixels associated with each microlens.

  In contrast, for example, a reconstruction surface 502 corresponding to a focal length longer than the imaging surface 501 is considered. As shown by the line segment connecting the imaging surface 501 and the reconstruction surface 502 in FIG. 5B, the light beam formed on each pixel when the imaging element 109 is on the reconstruction surface 502 is incident in the incident direction. Accordingly, it can be reproduced by changing the pixel arrangement of the LF data obtained on the imaging surface. That is, for each pixel of the reconstructed image generated for the reconstruction plane 502, the light flux that has passed through each pupil region for a pixel that exists at the same position as each pixel of the reconstructed image on the imaging surface in the changed pixel arrangement The pixel value is obtained by summing the pixel values.

  In this step, the camera control unit 101 acquires information on the pixel arrangement of the LF data related to the generation of the reconstructed image, which is changed according to the focal length in this way.

  In step S412, the camera control unit 101 determines whether the current focus search is a coarse search or a fine search. If the camera control unit 101 determines that the current focus search is a coarse adjustment search, the process proceeds to S413. If the camera control unit 101 determines that the current focus search is a fine adjustment search, the process proceeds to S414.

  In step S413, the camera control unit 101 determines pixels of LF data used for generating a reconstructed image for performing focus evaluation in the coarse tone search. In this embodiment, in order to reduce the processing time required for the coarse search, the reconstructed image for performing the focus evaluation at the coarse search is smaller in number of pixels (the first pixel number) than the reconstructed image at the fine search. ) Is used. Therefore, in this step, the camera control unit 101 performs a process of thinning out the pixels of the LF data used for generating the reconstructed image.

  For example, as shown in FIG. 6A, a case where discrete thinning is performed on the pixel arrangement of the LF data on the imaging surface 601 is considered. In the drawing, two types of R and G color filters are applied to the microlenses of the microlens array 108, and the thinning rate 1 / n is 1/2. In the figure, the thinned pixels are black pixels in which the numbers of the corresponding pupil regions are highlighted.

In a state (n = 1) where thinning is not performed as shown in FIG. 4B, the reconstructed image on the imaging surface 401 is for a set of microlenses to which adjacent R and G color filters are applied. One pixel after development is generated. That is, in the example of FIG. 4B, the pupil division number N is 4, and the color filter number s is 2.
N × s × n = 4 × 2 × 1 = 8
Thus, 8 pixels of the LF data correspond to 1 pixel of the developed image obtained by developing the reconstructed image. That is, the pixel having the cumulative pixel value of the pixels 1 to 4 associated with the microlens to which the R color filter is applied and the pixels 1 to 4 associated with the microlens to which the G color filter is applied. One pixel of the developed image is generated by the pixel having the cumulative pixel value.

On the other hand, when thinning is performed as shown in FIG. 6A, one pixel of the developed image obtained by developing the reconstructed image includes N × s × n = 4 × 2 × 2 = 16
Thus, 16 pixels of the LF data indicated by a frame in FIG. That is, four of the pixels of the reconstructed image that can be generated from the LF data (two pixels in the developed image) are referred to for generating two pixels of the reconstructed image that has been thinned out (one pixel in the developed image). . In other words, the number of pixels of a reconstructed image (decimated reconstructed image) obtained by performing 1/2 decimation from the one-dimensional LF data as illustrated is the number of pixels of the reconstructed image obtained without performing decimation. That is, it becomes 1/2 with respect to the upper limit number of pixels.

  Since FIG. 6A illustrates an example of performing thinning for a one-dimensional pixel array, the thinning rate and the number of pixels of the thinned reconstructed image / the maximum number of pixels are the same. However, when thinning is performed for each of the horizontal direction and the vertical direction of the two-dimensionally arranged LF data, it is easy that the horizontal thinning rate × the vertical thinning rate becomes the number of pixels of the thinned reconstructed image / the maximum number of pixels. Will be understood.

  By the way, when a reconstructed image that is focused on an arbitrary focal length is generated using LF data, the following problem occurs in the method of simply thinning out the LF data output from the imaging unit 102.

  Discrete decimation is performed at a decimation rate of 1/2 on the LF data output from the imaging unit 102, that is, the pixel array of the LF data on the imaging surface 601, and a thinned reconstructed image on the reconstruction surface 602 is generated. Think about the case. At this time, in the pixel array on the reconstruction plane 602, the distribution of thinned pixels is biased. In such a case, the light beam in the incident direction corresponding to the thinned pixel is not considered in one pixel of the thinned reconstructed image generated with respect to the reconstructed surface 602. For example, the image is formed in the thinned pixel region. The thin line or the like is not reflected in the thinned reconstructed image. That is, for example, even when the main subject has a thin line, the thinned-out reconstructed image cannot take into account the contrast of the thin line, so the detected focal length is the focal length at which the main subject is strictly in focus. It may not be possible.

  For this reason, in the digital camera 100 of the present embodiment, the characteristics of the LF data are used to determine discrete thinned pixels for the pixel arrangement of the LF data on the reconstruction plane 602 as shown in FIG. Determine the pixels used to generate the image. By doing so, an optical image of a region that is not reflected by the thinned-out pixels being determined in a fixed pattern as in the past can be considered in the reconstructed image for focus evaluation with a small number of pixels. it can.

In addition, a situation in which a thin line is not reflected by reducing the number of pixels of the reconstructed image for focusing evaluation as described above may also occur depending on the method of selecting the thinned pixels. Even when the thinned pixels are determined as pixels corresponding to the pupil regions 2 and 3 on the reconstruction plane for generating the reconstructed image, the light flux in a specific direction is not reflected in the reconstructed image. For this reason, in the digital camera 100 of this embodiment, the pixel of LF data used for the production | generation of a reconstructed image is determined according to the following rules.
1. Select at least one pixel of LF data corresponding to each pixel for each of a plurality of pixels of the reconstructed image having an upper limit number of pixels that can be generated from LF data, which is referred to for generating one pixel of the thinned reconstructed image (To make the reconstruction surface discrete and reflect more frequency information of the subject image)
2. Among the plurality of pixels of the reconstructed image having the upper limit number of pixels, which are referred to for generation of one pixel of the thinned reconstructed image, pixels of LF data corresponding to the light fluxes that have passed through different pupil regions between at least two pixels. Select to include

By doing in this way, the situation where a thin line is not considered in the thinning-out reconstructed image mentioned above can be avoided. In addition, when a color filter is applied to a microlens like this embodiment, you may use the following rules.
3. LF corresponding to a light beam that has passed through a different pupil region at a pixel corresponding to at least a different color filter among a plurality of pixels of the reconstructed image having the upper limit number of pixels, which is referred to for generation of one pixel of the thinned reconstructed image Select to include data pixels

  Note that the pixels of the LF data used for generating the reconstructed image may be stored in the ROM as table data for each reconstruction plane for each of the coarse search and fine search according to such rules. That is, the camera control unit 101 refers to the table stored in the ROM, and determines a pixel of LF data used for generating a reconstructed image for performing focus evaluation.

  In step S414, the camera control unit 101 determines a pixel of LF data used for generating a reconstructed image for performing focus evaluation in fine adjustment search. In the digital camera 100 according to the present embodiment, it is assumed that the reconstructed image having the upper limit number of pixels is used as a reconstructed image for focus evaluation without thinning out the pixels of the LF data during fine adjustment search. In this embodiment, the reconstructed image having the upper limit number of pixels at the time of fine adjustment search is described as being used for focus evaluation. However, the number of pixels of the reconstructed image generated at the time of fine adjustment search is larger than the number of pixels at the time of coarse search. And the number of pixels (second pixel number) equal to or less than the upper limit number of pixels. That is, it is sufficient if the image has the number of pixels that can be evaluated for focusing with reference to more detailed information than in the coarse tone search.

  In this way, after selecting the pixels of the LF data used for generating the reconstructed image according to a predetermined rule, the camera control unit 101 moves the AF process to S304.

  In step S304, the camera control unit 101 transmits information about the pixel arrangement of the LF data on the reconstruction plane corresponding to the focal length for which the focus evaluation is performed to the image processing unit 103, and reconstructs the LF data pixels according to the information. Shift to the pixel arrangement for generation of.

  In step S <b> 305, the camera control unit 101 executes reconstructed image generation processing that causes the image processing unit 103 to generate a reconstructed image.

<Reconstructed image generation processing>
Here, the details of the reconstructed image generation process of this embodiment will be described with reference to the flowchart of FIG.

In S421, the image processing unit 103 secures an area for storing the pixel value of each pixel of the reconstructed image to be generated in the memory 104, and initializes (fills with 0) the data in the area. At this time, the capacity of the data area may be a capacity corresponding to the number of microlenses constituting the microlens array 108. It is preferable that the data gradation of the pixel value of each pixel of the reconstructed image can store the product of the data gradation of each pixel in the LF data and the number of pupil divisions. For example, if the data of one pixel of LF data is 8 bits and the exit pupil is divided into 16, if there are 12 bits (= 8 bits + log ₂ 16), it is not necessary to consider data overflow.

  Then, the image processing unit 103 generates a reconstructed image used for focusing evaluation by executing the loop processing of S422 to S427 as many times as the number of microlenses constituting the microlens array 108. Note that reference numbers are assigned to the microlenses of the microlens array 108 in the order of horizontal scanning starting from the upper left, for example, and the microlens corresponding to the number of execution times of the loop processing is the processing target.

  In step S422, the image processing unit 103 determines whether the number of executions of the loop processing in steps S422 to S427 has been executed the same number of times as the number of microlenses. If the same number of times as the number of microlenses has not been executed, the image processing unit 103 moves the process to S423, and if it has been executed, completes the reconstructed image generation process. Note that the number of microlenses may be information stored in the ROM in advance, or may be obtained by dividing the number of pixels of the image sensor 109 by the number of pupil divisions.

  In addition, the image processing unit 103 executes the loop processing of S423 to S426 the same number of times as the number of pupil divisions of the exit pupil of the imaging optical system 202, so that the pixels of the LF data existing at the position associated with one microlens. Add pixel values for. Note that the LF data pixels present at the position associated with one microlens array are assigned with reference numbers in the order of horizontal scanning starting from the upper left, for example, and have reference numbers corresponding to the number of loop processing executions. A pixel of LF data is a processing target.

  In step S423, the image processing unit 103 determines whether the number of executions of the loop processing in steps S423 to S426 has been executed the same number of times as the number of pupil divisions. If it is not executed, the process proceeds to S424. If it is executed, the loop process is exited, and the process returns to S422.

  In step S424, the image processing unit 103 determines whether the pixel of the LF data to be processed is a thinned pixel. Specifically, the image processing unit 103 receives pixel information used to generate a reconstructed image obtained as a result of the pixel selection process from the camera control unit 101, and the pixel to be processed is a thinned pixel according to the information. Determine whether or not. If the image processing unit 103 determines that the pixel to be processed is a thinned pixel, the process proceeds to the loop end S426. If the image processing unit 103 determines that the pixel to be used for generating a reconstructed image, the process proceeds to S425.

  In step S425, the image processing unit 103 adds the pixel value of the pixel of the LF data to be processed to the value in the storage area of the memory 104 for the corresponding pixel of the reconstructed image. In addition, when performing thinning, the corresponding reconstructed image pixels are all added to the value of the same region as the LF data pixels used for generating each pixel of the thinned reconstructed image as described with reference to FIG. Shall be. At this time, it goes without saying that pixels of LF data corresponding to different color filters are stored in different regions. Further, for example, when the shift amount of the pixel position of the LF data is not an integer value due to the set focal length, a plurality of pixel values are internally divided in this step, and a storage area for the pixel of the corresponding reconstructed image Add to the value of.

  By executing the reconstructed image generation process in this way, the image processing unit 103 can generate a reconstructed image for performing focus evaluation corresponding to the reconstructed surface. After completing the reconstructed image generation process, the camera control unit 101 moves the AF process to S306.

  In step S <b> 306, the camera control unit 101 causes the image processing unit 103 to execute a contrast evaluation process for acquiring a contrast evaluation value indicating the degree of focus of the reconstructed image generated in step S <b> 305.

<Contrast evaluation processing>
Here, the contrast evaluation processing of the present embodiment will be described in detail with reference to the flowchart of FIG.

  In step S431, the image processing unit 103 sets the number of evaluation points to be subjected to contrast evaluation and the size of the evaluation frame. Note that the evaluation score and the size of the evaluation frame may be set in advance by the user, for example. As the number of evaluation points increases, it becomes possible to cover a wider range of the reconstructed image as an evaluation target, but the time required for contrast evaluation becomes longer. In addition, when the size of the evaluation frame increases, it is possible to perform contrast evaluation even for textures that do not have a pattern locally, but it is a so-called perspective conflict that simultaneously evaluates images of subjects with different distances. Incidence increases. In the present embodiment, it is assumed that the number of evaluation points and the size of the evaluation frame are set in consideration of such problems.

  In step S432, the image processing unit 103 applies filter processing to the pixel values of the reconstructed image. In the filter processing, it is preferable to perform filtering focusing on a lower frequency component during the coarse search, and performing filtering focusing on the high frequency compared with the coarse search during the fine search. By doing so, it is possible to improve the focusing accuracy of the focal length for focusing on the subject finally determined from the contrast evaluation value while avoiding the local minimum.

  The image processing unit 103 can obtain a contrast evaluation value for each evaluation frame of the evaluation score sentence by executing the loop processing of S433 to 436 for the evaluation score. In this loop process, the image processing unit 103 selects an evaluation frame for evaluation in accordance with a preset evaluation order and performs the following calculation.

  In step S433, the image processing unit 103 determines whether the number of executions of the loop processing in steps S433 to S436 has reached the evaluation score. If the image processing unit 103 determines that the number of executions of the loop processing has reached the evaluation score, it moves the process to S437, and if it determines that the number has not reached, it moves the process to S434.

  In S434, the image processing unit 103 peaks and holds the maximum pixel value for all horizontal lines of the reconstructed image in the evaluation frame.

  In step S435, the image processing unit 103 calculates a contrast evaluation value by adding the maximum values peak-held for all horizontal lines.

  In step S437, the image processing unit 103 outputs the contrast evaluation value calculated for each evaluation frame to the camera control unit 101, and completes the contrast evaluation process.

  In the present embodiment, it is assumed that the total value of the maximum pixel value values for all horizontal lines in the evaluation frame is output as the contrast evaluation value as described above, but the calculation of the contrast evaluation value accompanying the change in the focal length is described. May use other calculation methods. For example, a non-dimensional evaluation value obtained by dividing the secondary contrast evaluation value, which is the sum of squares of adjacent pixel output differences, by the square of the primary contrast, which is the absolute sum of adjacent pixel output differences, is used. The method can be used.

  After receiving the contrast evaluation value for the reconstructed image for which the focus evaluation is performed in this way, the camera control unit 101 stores the evaluation value in the RAM in association with the focal length information determined in S302, and performs AF processing. Move to S307.

  In step S <b> 307, the camera control unit 101 determines whether the best focal length for focusing on the subject has been detected based on the change in the contrast evaluation value for each focal length accumulated in the RAM. Specifically, the camera control unit 101 determines whether or not the peak value of the focal length that gives the highest contrast evaluation value is obtained as a result of the coarse adjustment search and the fine adjustment search. If the camera control unit 101 determines that the best focal length for focusing on the subject has been detected, it moves the process to S309. If it determines that it has not detected it, it moves the process to S308.

  In S <b> 308, the camera control unit 101 determines whether to shift from the coarse search to the fine search based on the change (evaluation result) of the contrast evaluation value for each focal length accumulated in the RAM, and the search direction. Is determined, and the process returns to S302. That is, in this step, the camera control unit 101 updates information necessary for determining the focal length for the next focus evaluation.

  In step S309, the camera control unit 101 transmits information on the focal length determined to be detected in step S307 to the lens control unit 201 of the lens 200 via the electrical contact 107, and drives the focus lens to the focal length. The lens control unit 201 transmits the received focal length information to the lens driving unit 203, and drives the focus lens to a position corresponding to the focal length.

  In this way, it is possible to detect the focal length that focuses on the subject without driving the lens from the LF data obtained by one imaging. In addition, the time required for generation of a reconstructed image and contrast evaluation can be shortened by appropriately reducing the number of pixels of the reconstructed image related to the focus evaluation during the coarse search and the fine search.

  In the present embodiment, the pixel value of each pixel of the reconstructed image to be evaluated for focusing is described as the sum of the pixel values of the LF data pixels corresponding to the number of pupil divisions. It is not limited. For example, the pixel value of each pixel of the reconstructed image may be a predetermined number that is equal to or greater than the number of pupil divisions.In this case, the total pixel value is the same level as the pixel value for the number of pupil divisions. Adjustment may be performed according to the number of added pixels.

  Next, another example of an optical system applicable to this embodiment will be described with reference to FIG. FIG. 7 is a diagram schematically illustrating a state in which a light beam from an object (subject) forms an image on the image sensor 109. FIG. 7A corresponds to the optical system described with reference to FIG. 2, and is an example in which the microlens array 108 is disposed in the vicinity of the imaging surface of the imaging optical system 202. FIG. 7B shows an example in which the microlens array 108 is arranged closer to the object than the imaging plane of the imaging optical system 202. FIG. 7C shows an example in which the microlens array 108 is arranged on the side farther from the object than the imaging plane of the imaging optical system 202.

  In FIG. 7, the same reference numerals are assigned to configurations common to those in FIG. 3, and redundant description is omitted. Reference numeral 51 denotes an object plane, and 51a and 51b are arbitrary points on the object plane. Reference numeral 52 denotes a pupil plane of the imaging optical system 202, and 61, 62, 71, 72, 73, 81, 82, 83, and 84 denote specific microlenses on the microlens array 108, respectively.

  7B and 7C, the virtual image sensor 109a and the virtual microlens array 108a are shown to clarify the correspondence with FIG. 7A. Also, a light beam passing from the point 51a on the object plane through the pupil regions 31 and 33 on the pupil plane 52 is indicated by a solid line, and a light beam passing from the point 51b on the object plane to the pupil regions 31 and 33 on the pupil plane 52 is indicated by a broken line. did.

  In the example of FIG. 7A, as described with reference to FIG. 3, the microlens array 108 is disposed in the vicinity of the imaging surface of the imaging optical system 202, so that the imaging device 109 and the pupil plane 52 of the imaging optical system are conjugated. Are in a relationship. Further, the object plane 51 and the microlens array 108 are in a conjugate relationship. For this reason, the light beam from the point 51a on the object plane 51 reaches the microlens 61, the light beam from the point 51b reaches the microlens 62, and the light beams that have passed through the pupil regions 31 to 35 correspond to the microlens. Reach the selected pixel.

  In the example of FIG. 7B, the light beam from the imaging optical system 202 is imaged by the microlens array 108, and the imaging element 109 is provided on the imaging surface. By arranging in this way, the object plane 51 and the image sensor 109 have a conjugate relationship. The light beam that has passed through the pupil region 31 of the pupil plane 52 from the point 51a on the object plane 51 reaches the micro lens 71, and the light beam that has passed through the pupil region 33 on the pupil plane from the point 51a reaches the micro lens 72. Further, the light beam that has passed through the pupil region 31 of the pupil plane 52 from the point 51b on the object plane 51 reaches the micro lens 72, and the light beam that has passed through the pupil region 33 of the pupil plane 52 from the point 51b reaches the micro lens 73. . The light beam that has passed through each microlens reaches a pixel provided so as to correspond to the microlens. Thus, the light beam from the object plane forms an image at different positions on the imaging surface of the image sensor 109 according to the emission position and the passing pupil region. If these are rearranged at the positions on the imaging surface of the virtual imaging element 109a, the same information as the information obtained on the imaging surface of FIG. 7A can be obtained. That is, information on the pupil region (incident angle) that has passed and the position on the image sensor 109 can be obtained.

  In the example of FIG. 7C, the microlens array 108 re-images the light beam from the imaging optical system 202 (referred to as re-image because a light beam once formed is diffused). The imaging surface of the image sensor 109 is disposed on the re-imaging plane. By arranging in this way, the object plane 51 and the image sensor 109 have a conjugate relationship. The light beam that has passed through the pupil region 31 of the pupil plane 52 from the point 51a on the object plane 51 reaches the micro lens 82, and the light beam that has passed through the pupil region 33 of the pupil plane 52 from the point 51a reaches the micro lens 81. Further, the light beam that has passed through the pupil region 31 of the pupil plane 52 from the point 51b on the object plane 51 reaches the micro lens 84, and the light beam that has passed through the pupil region 33 of the pupil plane 52 from the point 51b reaches the micro lens 83. The light beam that has passed through each microlens reaches a pixel provided so as to correspond to the microlens.

  Similarly to the case of FIG. 7B, if the pixel signals obtained by the image sensor 109 are rearranged to the positions on the image plane of the virtual image sensor 109a, they can be obtained on the image plane of FIG. 7A. Information similar to information can be obtained. That is, information on the pupil region (incident angle) that has passed and the position on the image sensor 109 can be obtained.

  FIG. 7 shows a configuration example in which pupil division is performed using the microlens array 108 (phase modulation element) to acquire position information and angle information of a light beam. Other configurations can be used as long as they can acquire (equivalent to limiting). For example, a configuration in which a pattern mask (gain modulation element) configured by repeating basic patterns is used instead of the microlens array 108 may be used.

  As described above, the imaging apparatus of the present embodiment can speed up the autofocus operation in the imaging apparatus that can generate an image focused on an arbitrary focal length after shooting. Specifically, the imaging apparatus sets the number of pixels of the generated reconstructed image for focus evaluation, and determines the generated reconstructed image from the number of pixels and the upper limit number of pixels of the reconstructed image that can be generated from the pixels of the image signal. A reconstructable image pixel that can be used for generating each pixel is determined. Then, for each pixel of the generated reconstructed image, a predetermined number of pixels are selected as pixels used for generating the generated reconstructed image among the pixels of the image signal corresponding to the pixels of the reconfigurable reconfigurable image used for generation. . At this time, if the set number of pixels of the generated reconstructed image is smaller than the upper limit number of pixels, the imaging apparatus determines at least one corresponding image signal pixel for each of the reproducible reconstructed image pixels used for generation. select.

[Other Embodiments]
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (7)

An imaging unit that images a subject and outputs the obtained image signal, each of the pixels of the image signal corresponding to a light flux in which a combination of a pupil region and an incident direction of the imaging optical system that has passed is different ,
A determination unit configured to set a generated reconstructed image generated from the image signal output by the imaging unit, and to determine a focal length of a subject to be focused in the generated reconstructed image;
In accordance with the focal length determined by the determining means, the pixel values of the pixels of the predetermined number of the image signals corresponding to the pixels of the generated reconstructed image are added together, thereby focusing on the focal length. Generating means for generating a generated reconstructed image;
An evaluation unit that performs a focus evaluation on the generated reconstructed image generated by the generation unit,
Setting means for setting the number of pixels of the generated reconstructed image generated by the generating means;
The generation possibility used for generating each pixel of the generated reconstructed image from the number of pixels of the generated reconstructed image set by the setting means and the upper limit number of pixels of the reconstructed image that can be generated from the pixels of the image signal A pixel of the reconstructed image is determined, and for each pixel of the generated reconstructed image, the predetermined number of pixels are selected from the pixels of the image signal corresponding to the pixels of the reconfigurable reconstructable image used for generation And a selection means to
The selection means has at least one correspondence for each pixel of the reconfigurable reconfigurable image used for the generation when the number of pixels of the generated reconstructed image set by the setting means is smaller than the upper limit number of pixels. An image pickup apparatus that selects pixels of the image signal.   When the number of pixels of the generated reconstructed image is smaller than the upper limit number of pixels, the selection unit selects different pupil regions between at least two pixels of the reconfigurable image that can be used for generation. 2. The imaging apparatus according to claim 1, wherein the predetermined number of pixels of the image signal are selected so as to include the corresponding pixels of the image signal that have passed. The imaging means has a plurality of color filters, and the pixels of the image signal are color signals corresponding to any of the plurality of color filters,
The generation unit generates each pixel of the generated reconstructed image using a pixel of the image signal corresponding to each of the plurality of color filters,
When the number of pixels of the generated reconstructed image is smaller than the upper limit number of pixels, the selection unit is configured to select the plurality of pixels of the image signal corresponding to each pixel of the reconfigurable reconfigurable image used for the generation. The imaging apparatus according to claim 1, wherein a pixel that passes through a different pupil region among pixels corresponding to each of the color filters is selected. The determining unit changes an interval between the focal length currently set and the focal length to be set next, according to an evaluation result by the evaluation unit,
The setting means sets the number of pixels of the generated reconstructed image to a first number of pixels smaller than the upper limit number of pixels when the interval is larger than a threshold, and when the interval is smaller than the threshold, The number of pixels of the generated reconstructed image is determined to be a second number of pixels that is larger than the first number of pixels and equal to or less than the upper limit number of pixels. Imaging device.   The imaging apparatus according to claim 1, wherein the predetermined number is a pupil division number. An imaging unit that images a subject and outputs the obtained image signal, each of the pixels of the image signal corresponding to a light flux in which a combination of a pupil region and an incident direction of the imaging optical system that has passed is different ,
A determination unit configured to set a generated reconstructed image generated from the image signal output by the imaging unit, and to determine a focal length of a subject to be focused in the generated reconstructed image;
In accordance with the focal length determined by the determining means, the pixel values of the pixels of the predetermined number of the image signals corresponding to the pixels of the generated reconstructed image are added together, thereby focusing on the focal length. Generating means for generating a generated reconstructed image;
An evaluation unit that performs focusing evaluation on the generated reconstructed image generated by the generation unit;
A setting step in which the setting unit of the imaging apparatus sets the number of pixels of the generated reconstructed image generated by the generating unit;
Each of the generated reconstructed images is selected from the number of pixels of the generated reconstructed image set in the setting step and the upper limit number of pixels of the reconstructed image that can be generated from the pixels of the image signal. A pixel of the reconstructable image that can be generated used for generating a pixel is determined, and for each pixel of the generated reconstructed image, among the pixels of the image signal corresponding to the pixel of the reconfigurable image that can be generated that is used for generation And a selection step of selecting the predetermined number of pixels,
In the selection step, when the number of pixels of the generated reconstructed image set in the setting step is smaller than the upper limit number of pixels, the selection unit is configured to generate pixels that can be generated for the generation. A control method for an imaging apparatus, wherein at least one corresponding pixel of the image signal is selected.   The program for functioning a computer as each means except the imaging means of the imaging device of any one of Claims 1 thru | or 5.

Images