JP2013145980A - Imaging device, control method thereof, image processing apparatus, image generation method, program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To display an image in-focus over a wide range, while reducing the processing load for reconstruction during live view driving.SOLUTION: The image processing apparatus has, between a photographic lens and an image sensor consisting of multiple photoelectric conversion elements, such a microlens array that one microlens corresponds to the multiple photoelectric conversion elements. The image processing apparatus includes a region splitting unit for splitting a captured image acquired by the image sensor into first multiple split regions and second multiple split regions, a distance information acquisition unit for acquiring the distance information of a subject from the first split region, and a refocus image generation unit for generating a reconstruction image set at a predetermined focal position for the second split region. The refocus image generation unit sets a predetermined focal position for each second split region, based on the distance information acquired from the first split region.

Description

  The present invention relates to an imaging device, and more particularly to an imaging device having means for acquiring information on the incident direction of light using a microlens array and reconstructing a refocused image.

  2. Description of the Related Art In recent years, an imaging apparatus (light field camera) that can acquire not only the intensity distribution of light but also information on the incident direction of light has been proposed in an imaging apparatus such as an electronic camera.

  For example, according to Non-Patent Document 1, a microlens array is arranged between a photographic lens and an image sensor, and one microlens is associated with a plurality of pixels of the image sensor, thereby allowing light that has passed through the microlens. Is acquired for each incident direction by a plurality of pixels. In addition to generating a normal captured image for the pixel signal (light field) acquired in this way, a technique called “Light Field Photography” is applied to any image plane (refocus plane). A focused image can be reconstructed after shooting.

Ren.Ng and 7 others, “Light Field Photography with a Hand-Held Plenoptic Camera”, Stanford Tech Report CTSR 2005-02

  In the light field camera described in Non-Patent Document 1, in order to increase the depth of the reconfigurable refocus plane, the aperture of the photographing lens is set close to the open position to acquire larger angle information of light. Is desirable. In that case, the depth of field is generally shallow.

  On the other hand, when the depth of field is shallow, it is assumed that it is difficult to confirm the composition including the background. In particular, at the time of live view (LV) driving mounted on an electronic camera in recent years, there is a risk of hindering confirmation of the composition which is one of the purposes.

  In other words, the light field camera can refocus to an arbitrary focal position after shooting, but it is desirable to focus on a wider range than the main subject during live view driving.

  However, narrowing down the aperture to increase the depth of field not only causes a problem in the S / N ratio due to a decrease in the amount of incident light, but also means that some angle information of incident light is lost. To do. This is contrary to the original use and purpose of the imaging apparatus that can be refocused after shooting.

  In addition, reconstructing and displaying all refocusable refocus planes to display images that are in focus over a wider range other than the main subject requires a large amount of computation, especially when driving live view. Is not realistic.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to display an image focused in a wide range while reducing the processing load for reconstruction during live view driving. It is.

  An imaging apparatus according to the present invention includes an image processing apparatus having a microlens array in which one microlens corresponds to each of the plurality of photoelectric conversion elements between an imaging lens and an imaging element including a plurality of photoelectric conversion elements. An area dividing unit that divides a captured image acquired by the image sensor into a plurality of first divided areas and a plurality of second divided areas, and distance information of a subject is acquired from the first divided areas. Distance information acquisition means for performing refocusing image generation means for generating a reconstructed image set at a predetermined focal position with respect to the second divided region, wherein the refocus image generation means includes The predetermined focus position is set for each of the second divided areas based on distance information acquired from one divided area.

  According to the present invention, it is possible to display an image focused in a wide range while reducing the processing load for reconstruction during live view driving.

1 is an overall block diagram of an imaging apparatus according to an embodiment of the present invention. The figure explaining the structure of an image pick-up element and a micro lens array. 2A and 2B illustrate a configuration of a photographing lens, a microlens array, and an imaging element. The figure explaining the correspondence of the pupil region of a photographic lens, and a light reception pixel. The figure explaining the passage area of a refocus image generation light ray. 6 is a flowchart for explaining an image reconstruction operation of the image processing circuit. The figure explaining the area | region dividing method of distance information. The figure explaining the selection method of a refocus surface. The figure explaining the reconstruction processing method for every refocus plane.

  Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

  FIG. 1 is a diagram illustrating a block configuration of an imaging apparatus 100 that is an example of an image processing apparatus according to an embodiment of the present invention. In FIG. 1, 101 is a photographing lens, 102 is a microlens array, and 103 is an image sensor. The light that has passed through the photographing lens 101 forms an image near the focal position of the photographing lens 101. The microlens array 102 includes a plurality of microlenses 1020, and is arranged in the vicinity of the focal position of the photographing lens 101 so that light that has passed through different pupil regions of the photographing lens 101 is divided and emitted for each pupil region. It has the function to do. The image sensor 103 is a photoelectric conversion element represented by a CMOS image sensor or a CCD image sensor. By arranging a plurality of pixels so as to correspond to one of the plurality of microlenses 1020, the light emitted by the microlens 1020 divided for each pupil region is received while maintaining the division information, and the data It has a function of converting into a processable image signal.

  Reference numeral 104 denotes an analog signal processing circuit (AFE), and 105 denotes a digital signal processing circuit (DFE). The analog signal processing circuit 104 performs correlated double sampling processing, signal amplification, reference level adjustment, A / D conversion processing, and the like on the image signal output from the image sensor 103. The digital signal processing circuit 105 performs digital image processing such as reference level adjustment on the digital image signal output from the analog signal processing circuit 104.

  Reference numeral 106 denotes an image processing circuit, 107 denotes a memory circuit, and 108 denotes a recording circuit. The image processing circuit 106 performs predetermined image processing and a refocusing operation, which is a feature of this embodiment, on the image signal output from the digital signal processing circuit 105. The memory circuit 107 and the recording circuit 108 are recording media such as a nonvolatile memory or a memory card that records and holds an image signal output from the image processing circuit 106.

  Reference numeral 109 denotes a control circuit, 110 denotes an operation circuit, and 111 denotes a display circuit. The control circuit 109 comprehensively drives and controls the entire imaging apparatus such as the imaging element 103 and the image processing circuit 106. The operation circuit 110 receives a signal from an operation member provided in the imaging apparatus 100 and reflects a user command to the control circuit 109. The display circuit 111 displays an image after shooting, a live view image, various setting screens, and the like.

  This embodiment is characterized by the image processing circuit 106. The image processing circuit 106 (refocus image generation means) sets an arbitrary focal position (refocus plane) from the image data (captured image) by performing a calculation process using a technique called “Light Field Photography”. An image (reconstructed image, output image) is reconstructed.

  Next, the configuration of the photographic lens 101, the microlens array 102, and the imaging element 103 in the imaging apparatus of the present embodiment will be described.

  FIG. 2 is a diagram of the image sensor 103 and the microlens array 102 observed from the optical axis Z direction in FIG. One microlens 1020 is arranged so as to correspond to a plurality of unit pixels 201. A plurality of unit pixels 201 behind one microlens are collectively defined as a pixel array 20. In the present embodiment, it is assumed that the pixel array 20 includes a total of 25 unit pixels 201 of 5 rows and 5 columns.

  FIG. 3 is a diagram in which light emitted from the photographing lens 101 passes through one microlens 1020 and is received by the image sensor 103 as observed from a direction perpendicular to the optical axis Z. Light emitted from each of the pupil regions a1 to a5 of the photographic lens 101 and passed through the microlens 1020 forms an image on the corresponding rear unit pixels p1 to p5, respectively.

  FIG. 4A is a view of the aperture of the taking lens 101 as viewed from the optical axis Z direction. FIG. 4B is a view of one microlens 1020 and the pixel array 20 disposed behind the microlens 1020 when viewed from the optical axis Z direction. As shown in FIG. 4A, when the pupil region of the photographing lens 101 is divided into the same number of regions as pixels behind one microlens, one pixel is emitted from one pupil division region of the photographing lens 101. Light is imaged. However, here, it is assumed that the F-numbers of the photographing lens 101 and the microlens 1020 are substantially the same.

  The correspondence between the pupil division areas a11 to a55 of the photographing lens 101 shown in FIG. 4A and the pixels p11 to p55 shown in FIG. 4B is point-symmetric when viewed from the optical axis Z direction. Therefore, the light emitted from the pupil division region a11 of the photographing lens 101 forms an image on the pixel p11 in the pixel array 20 behind the microlens. Similarly, the light emitted from the pupil division area a11 and passing through another microlens 1020 forms an image on the pixel p11 in the pixel array 20 behind the microlens.

  Here, a method for calculating a focal position (refocus plane) corresponding to an arbitrary subject position in the screen will be described.

  As described with reference to FIG. 4, each pixel of the pixel array 20 receives light that has passed through different pupil regions with respect to the photographing lens 101. By combining a plurality of pixel signals from these divided signals, a pair of signals divided into pupils in the horizontal direction is generated.

  Expression (1) is obtained by integrating the light passing through the left area (pupil areas a1 to a2) of the exit pupil of the photographing lens 101 with respect to each pixel of a certain pixel array 20. This is applied to a plurality of pixel arrays 20 arranged in the horizontal direction, and a subject image constituted by these output signal groups is defined as an A image. Expression (2) is obtained by integrating the light passing through the right region (pupil regions a4 to a5) of the exit pupil of the photographing lens 101 for each pixel of a certain pixel array 20. This is applied to a plurality of pixel arrays 20 arranged in the horizontal direction, and a subject image constituted by these output signal groups is defined as a B image. A correlation operation is performed on the A image and the B image, and an image shift amount (pupil division phase difference) is detected. Furthermore, the focal position corresponding to an arbitrary subject position in the screen can be calculated by multiplying the image shift amount by the focal position of the photographing lens 101 and a conversion coefficient determined by the optical system.

  Next, image reconstruction processing at an arbitrarily set focal position (refocus plane) with respect to imaging data acquired by the configuration of the photographing lens 101, the microlens array 102, and the imaging element 103 will be described.

  FIG. 5 shows from which the light passing through a certain pixel on the refocus plane set arbitrarily is emitted from which pupil division region of the photographing lens and which microlens is incident from a direction perpendicular to the optical axis Z. It is a figure. In FIG. 5, the position of the pupil division area of the photographing lens is the coordinates (u, v), the pixel position on the refocus plane is the coordinates (x, y), and the position of the microlens on the microlens array is the coordinates (x ′, y ′), the distance from the taking lens to the microlens array is F, and the distance from the taking lens to the refocus plane is αF. α is a refocus coefficient for determining the position of the refocus plane and can be arbitrarily set by the user. In FIG. 5, only the directions of u, x, and x ′ are shown, and v, y, and y ′ are omitted. As shown in FIG. 5, the light passing through the coordinates (u, v) and the coordinates (x, y) reaches the coordinates (x ′, y ′) on the microlens array. The coordinates (x ′, y ′) can be expressed as in Equation (3).

  When the output of the pixel receiving this light is L (x ′, y ′, u, v), the output E (x, y) obtained with the coordinates (x, y) on the refocus plane is L Since (x ′, y ′, u, v) is integrated with respect to the pupil region of the photographing lens, Equation (4) is obtained.

  In Expression (4), since the refocus coefficient α is determined by the user, if (x, y) and (u, v) are given, the position (x ′, y ′) of the microlens where the light enters can be known. . Then, the pixel corresponding to the position (u, v) is known from the pixel array 20 corresponding to the microlens. The output of this pixel is L (x ', y', u, v). E (x, y) can be calculated by performing this operation for all pupil division regions and integrating the obtained pixel outputs. If (u, v) is the representative coordinates of the pupil division area of the taking lens, the integral of equation (4) can be calculated by simple addition.

  Next, an image reconstruction operation in the image processing circuit 106, which is a feature of the present embodiment, will be described with reference to an LV display image generation flowchart in FIG.

  In this embodiment, the light field camera having the configuration described so far is used to record light intensity and direction as a continuous moving image in time series, and for each frame of the captured moving image. By generating a refocused image, it is possible to obtain an image focused on a wide area of the object scene.

  In FIG. 6, in step S <b> 601, image data obtained from the image sensor 103 is acquired.

  In step S602, the input imaging data is divided into predetermined distance information acquisition areas (hereinafter referred to as “D area”). Here, since the number of divisions of the D area (first division region) is reflected in the focusing accuracy at the time of reconstruction, it is desirable that the division is performed for each pixel array 20 having the minimum position resolution. However, the number of pixel arrays 20 can be determined as appropriate according to restrictions on the calculation capability, frame rate, and the like of the image processing circuit 106, such as forming a single D area. That is, the division number of the D area may be decreased as the frame rate is higher or the calculation capability of the image processing circuit 106 is lower.

  In step S603, the above-described formulas (1) and (2) are applied to the imaging data divided into the D areas, and the subject distance representing each D area is calculated. Here, when a plurality of pixel arrays 20 are combined to form one D area (for example, 5 × 5), the result of only the central pixel array 20 may be substituted, or a 5 × 5 pixel. The average value of the array 20 may be used. Further, the subject distance information for each D area obtained in this manner is used as a distance information map.

  In step S604, a refocus plane is selected according to the above-described distance information map. At this time, the refocus plane to be selected is a method of obtaining the mode value from the frequency distribution of the distance information map (hereinafter referred to as distance information histogram), or a method of applying curve fitting with a predetermined function form. Selected by. In order to reduce the load of reconstruction processing, at least one of the refocus surfaces can be set based on a distance determined from the focus lens position at the time of shooting. In this case, it is possible to omit the reconstruction process, such as simply adding and outputting pixel signals for each pixel array 20. Although it is desirable that the optimum number of refocus surfaces be determined according to the refocusable range and the depth of field of the optical system, in the present embodiment, the description will be made assuming that there are three refocus surfaces.

  In step S605, the input imaging data is divided into reconstruction areas (hereinafter referred to as R areas) based on the distance information map. Here, the R area (second divided region) may coincide with the D area. In this case, step S605 is included in step S602.

  In step S606, the above-described reconstruction process is performed for each R area on the refocus plane to which each R area belongs.

  In step S607, the R areas subjected to the reconstruction process are connected to generate a one-frame image.

  In step S608, the joined image is displayed on the display circuit 111.

  In step S609, it is determined whether or not the LV display is continued. If the LV display is continued, the process returns to S601. If the process is finished, the image reconstruction operation in the image processing circuit 106 is completed.

  Next, the area dividing method of distance information in step S602 and step S603 will be described with reference to FIG. FIG. 7A shows the acquired imaging data. In this imaging data, the subject distance is calculated for each predetermined D area. FIG. 7B is a schematic diagram expressing these subject distances as a contrast diagram.

  Next, the method for selecting the refocus plane in step S604 will be described in detail. FIG. 8 is a schematic diagram of a distance information histogram in which the subject distance information for each D area obtained in step S603 is represented with the horizontal axis as the focus lens position and the vertical axis as the frequency. Here, when an image is captured, the mode is most frequently displayed in three distance areas: a short distance (refocus plane (a)), a medium distance (refocus plane (b)), and a long distance (refocus plane (c)). The imaging conditions where values appear are assumed.

  In general, it is reasonable to consider that the mode value of the distance information histogram corresponds to the position where a representative subject exists. By detecting this mode value, it is possible to select a refocus plane on which a representative subject exists. The detection method is not limited to the mode value, and a known method such as curve fitting using a predetermined function form can be applied.

  Alternatively, since the focus lens position at the time of shooting is known in advance, the position may be selected as at least one refocus plane. In FIG. 8, this means that the refocus plane (b) matches the focus lens position at the time of shooting. Thereby, an in-focus image can be acquired without performing the reconstruction process, and the load of the reconstruction process can be reduced. In this case, it is not possible to focus completely on the mode value existing at the middle distance, but it is considered that shifting appropriately within the range of the depth of field is allowed.

  In addition, when a characteristic distribution does not appear in the distance information histogram and the position of the subject cannot be specified, the refocus plane can be arbitrarily selected within the refocusable range. In this case, it is desirable to select the refocus plane with an interval of about the depth of field.

  Next, the reconstruction processing method for each refocus plane in step S605, step S606, and step S607 will be described in detail.

  FIGS. 9A, 9B, and 9C are schematic views showing R areas to be reconfigured in a plurality of refocus planes selected in step S604, and FIG. 9D shows step S606. It is a schematic diagram which shows the synthesized image which connected the small area reconfigure | reconstructed for every refocus surface. In FIG. 9, the solid line represents a focused subject, and the broken line represents a non-focused subject.

  First, the input imaging data is divided into R areas based on the distance information histogram. Here, the division method of the R area may be determined in advance, or may be the same as that of the D area. Alternatively, it can be determined according to the distance information histogram, such as increasing the area of the area where the subject distance information is out of the focus lens position at the time of shooting. For example, as shown in FIG. 9B, the R area reconstructed by the refocus plane (b) close to the focus lens position at the time of shooting is the same as the D area, whereas, as shown in FIG. 9C, The area of the R area reconstructed by the refocus plane (c) far from the focus lens position at the time of shooting can be increased.

  Each R area is associated with at least one of the selected refocus planes. In FIG. 8, the R area belonging to the distance area (A) is the refocus plane (a), the R area belonging to the distance area (B) is the refocus plane (b), and the R area belonging to the distance area (C) is It means to correspond to the refocus plane (c).

  Further, additional information indicating that refocusing is not performed can be associated with an R area that does not belong to any distance region, that is, it is determined that the distance cannot be calculated and is outside the refocusable range.

  The R area belonging to the distance area (A) in FIG. 8 is reconstructed on the refocus plane (a). In this case, “person” in FIG. 9A corresponds to the subject existing at a short distance. Hereinafter, the reconstruction process is similarly performed on the corresponding refocus planes in the R area belonging to the distance area (B) and the distance area (C). In this case, “gate” and “house” in FIGS. 9B and 9C correspond to subjects existing at a medium distance and a long distance, respectively.

  Further, for the R area corresponding to the focus lens position at the time of shooting and the R area determined to be out of the refocusable range (for example, “sky” in FIG. 9), the refocus processing is not performed, for example, the pixel array The pixel signals are simply added every 20 and output.

  Furthermore, for the R areas belonging to the boundaries of the distance regions (A), (B), and (C), the focal position near the middle of each refocus plane for the purpose of relaxing the discontinuity after synthesis. It is also possible to reconfigure using Further, the focus position at this time or the number of R areas to which this focus position is applied (histogram class number from the boundary) is weighted according to the relative distance between each refocus plane and the focus lens position at the time of shooting. It does not matter.

  As described above, a plurality of refocus planes are set, and refocus processing is performed on the refocus plane associated with at least one for each R area, thereby reducing the processing load for reconstruction and An image focused on a wide area of the field can be obtained.

  The processing operation of the present embodiment may be performed at all times during live view driving, and is performed only when there is a user instruction (for example, only when a function similar to a general narrowing button is provided). May be. Furthermore, the processing operation of this embodiment may be performed not only when the live view is driven but also when being played back by another playback device. In other words, the image processing apparatus according to the present embodiment does not necessarily have the image sensor as illustrated, and acquires an image acquired from the illustrated image sensor as an input image and performs the processing operation according to the present embodiment. It may be.

As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.
(Other embodiments)
The object of the present invention can also be achieved as follows. That is, a storage medium in which a program code of software in which a procedure for realizing the functions of the above-described embodiments is described is recorded is supplied to the system or apparatus. The computer (or CPU, MPU, etc.) of the system or apparatus reads out and executes the program code stored in the storage medium.

  In this case, the program code itself read from the storage medium realizes the novel function of the present invention, and the storage medium and program storing the program code constitute the present invention.

  Examples of the storage medium for supplying the program code include a flexible disk, a hard disk, an optical disk, and a magneto-optical disk. Further, a CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD-R, magnetic tape, nonvolatile memory card, ROM, or the like can also be used.

  Further, by making the program code read by the computer executable, the functions of the above-described embodiments are realized. Furthermore, when the OS (operating system) running on the computer performs part or all of the actual processing based on the instruction of the program code, the functions of the above-described embodiments are realized by the processing. Is also included.

  Furthermore, the following cases are also included. First, the program code read from the storage medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. Thereafter, based on the instruction of the program code, the CPU or the like provided in the function expansion board or function expansion unit performs part or all of the actual processing.

Claims (10)

An imaging apparatus having a microlens array in which one microlens corresponds to the plurality of photoelectric conversion elements between a photographing lens and an imaging element composed of a plurality of photoelectric conversion elements,
Area dividing means for dividing a captured image acquired by the imaging element into a plurality of first divided areas and a plurality of second divided areas;
Distance information acquisition means for acquiring distance information of a subject from the first divided region;
Refocus image generation means for generating a reconstructed image set at a predetermined focal position with respect to the second divided region;
The image pickup apparatus, wherein the refocus image generation unit sets the predetermined focal position for each of the second divided regions based on distance information acquired from the first divided region.   The imaging apparatus according to claim 1, wherein the first divided area and the second divided area match.   The imaging apparatus according to claim 1, wherein the refocus image generation unit generates one image by connecting the refocus images reconstructed for each of the second divided regions.   The imaging apparatus according to claim 1, wherein the refocus image generation unit sets a plurality of the predetermined focal positions.   The imaging apparatus according to claim 4, wherein the refocus image generation unit sets at least one of the plurality of predetermined focal positions to be set as a focus lens position at the time of photographing. A method for controlling an imaging apparatus having a microlens array in which one microlens corresponds to the plurality of photoelectric conversion elements between a photographing lens and an imaging element composed of a plurality of photoelectric conversion elements,
A region dividing step of dividing a captured image acquired by the image sensor into a plurality of first divided regions and a plurality of second divided regions;
A distance information acquisition step of acquiring subject distance information from the first divided region;
A refocus image generation step of generating a reconstructed image set at a predetermined focal position with respect to the second divided region,
In the refocus image generation step, the predetermined focus position is set for each of the second divided areas based on distance information acquired from the first divided area. . An output image from an input image acquired by an imaging device having a microlens array in which one microlens corresponds to the plurality of photoelectric conversion elements between a photographing lens and an imaging element composed of a plurality of photoelectric conversion elements. An image processing device for generating
Area dividing means for dividing the input image into a plurality of first divided areas and a plurality of second divided areas;
Distance information acquisition means for acquiring distance information of a subject from the first divided region;
Refocus image generation means for generating a reconstructed image set at a predetermined focal position with respect to the second divided region;
The image processing apparatus, wherein the refocus image generation unit sets the predetermined focus position for each of the second divided areas based on distance information acquired from the first divided area. An output image from an input image acquired by an imaging device having a microlens array in which one microlens corresponds to the plurality of photoelectric conversion elements between a photographing lens and an imaging element composed of a plurality of photoelectric conversion elements. An image generation method for generating
A region dividing step of dividing the input image into a plurality of first divided regions and a plurality of second divided regions;
A distance information acquisition step of acquiring subject distance information from the first divided region;
A refocus image generation step of generating a reconstructed image set at a predetermined focal position with respect to the second divided region;
In the refocus image generation step, the predetermined focus position is set for each of the second divided regions based on distance information acquired from the first divided region.   A computer-executable program in which the procedure of the image generation method according to claim 8 is described.   A computer-readable storage medium storing a program for causing a computer to execute each step of the image generation method according to claim 8.

Images