JP2018050147A - Imaging apparatus, imaging method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging apparatus capable of generating a composite image reducing blur due to camera shake, etc., when synthesizing multiple images captured by changing the focus position.SOLUTION: An imaging apparatus has synthesis means for generating a composite image by synthesizing multiple images captured while changing the focus position of an optical system, focus position setting means for setting the focus position of the optical system, and refocus means performing reconstitution of an image where the focus position of the captured image was differentiated. The refocus means performs reconstitution of an image where the focus position was differentiated, based on the difference of the set focus position and the focus position when capturing the image.SELECTED DRAWING: Figure 9

Description

  The present invention relates to an imaging apparatus that performs image composition.

  When shooting multiple subjects with different distances from the camera, or when shooting subjects that are long in the depth direction, the depth of field in the imaging optical system is insufficient. It may not be possible. Therefore, a plurality of images are picked up by changing the focus position, only a focused area is extracted from each image, and a single image is synthesized to generate a composite image focused on the entire imaging area. Technology (hereinafter referred to as depth synthesis) is known (see Patent Document 1). By using this depth synthesis technique, it is possible to obtain an image in which the entire intended subject is in focus.

JP2015-216480A

  However, when imaging is performed using the technique of depth synthesis, the following problems may occur when camera shake or the like occurs.

  FIG. 11 is a diagram for explaining camera shake in depth synthesis. FIG. 11A is a diagram for explaining imaging using ideal depth synthesis. In FIG. 11A, the user places the digital camera on 1101 and performs imaging a plurality of times while moving the focus position by a predetermined distance 1102.

  FIG. 11B shows a case where camera shake has occurred when the third image is captured in the imaging in FIG. Since the imaging position 1101 is shifted rearward in the optical axis direction, the focus position comes closer to the position that should be originally captured. Furthermore, as shown in FIG. 11C, when a fourth image is captured, a case where camera shake occurs in a direction opposite to the direction of camera shake in FIG. In this case, the fourth image is farther from the focus position that should be captured. When combining processing is performed using a plurality of images captured in FIG. 11C, there is no focused image in the area 1103, and a blurred portion remains in the combined image.

  The present invention has been made in view of the above problems, and when combining using a plurality of images picked up by changing the focus position, a composite image that corrects the focus position of the image and reduces blur due to camera shake or the like. It is an object to provide an imaging device capable of generating

  The present invention combines a plurality of images captured while changing the focus position of the optical system, generates a composite image, a focus position setting means for setting the focus position of the optical system, Refocusing means for reconstructing an image with a different focus position of the captured image, and the refocusing means is different from the set focus position and the focus position when capturing the image. The present invention provides an image pickup apparatus that reconstructs an image with a different focus position.

  According to the present invention, when combining using a plurality of images picked up by changing the focus position, it is possible to generate a combined image that reduces blur due to camera shake or the like.

It is a block diagram which shows the structure of the digital camera in embodiment of this invention. It is a figure for demonstrating the array of the sensor which comprises the imaging part in embodiment of this invention. In embodiment of this invention, it is a figure for demonstrating incidence | injection to the pixel of an optical signal. (A) is the figure which looked at the opening of the imaging lens from the optical axis direction, (b) is the figure which looked at one micro lens and the pixel arrangement | positioning arrange | positioned behind it from the optical axis direction. It is a figure for demonstrating the calculation of the refocus surface in embodiment of this invention. It is a figure for demonstrating the refocus range in embodiment of this invention. It is a flowchart for demonstrating embodiment of this invention. It is a figure for demonstrating the production | generation of the refocus image in embodiment of this invention. It is a figure for demonstrating an example of the refocus process when it is judged that refocusing is possible in embodiment of this invention. It is a figure for demonstrating an example of reset of a focus position when it is judged that refocusing is not possible in the embodiment of the present invention, and subsequent refocus processing. It is a figure for demonstrating the camera shake in depth composition.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description, a digital camera will be described as an example. However, the present invention is not limited to a digital camera, and an imaging apparatus capable of taking in focus position information and correcting the focus position of a captured image is described. If used, the present invention can be applied.

(Example of embodiment of the present invention)
FIG. 1 is a block diagram showing the structure of a digital camera according to this embodiment. The digital camera 100 can capture a still image, records information on the in-focus position, and can calculate a contrast value and synthesize an image. Furthermore, the digital camera 100 can perform an enlargement process or a reduction process on an image captured and stored or an image input from the outside.

  The control unit 101 is a signal processor such as a CPU or an MPU, for example, and controls each part of the digital camera 100 while reading a program built in the ROM 105 described later. For example, as will be described later, the control unit 101 instructs the imaging unit 104 described later to start and end imaging. Alternatively, an image processing command is issued to an image processing unit 107 described later based on a program built in the ROM 105. A user command is input to the digital camera 100 by an operation unit 110 described later, and reaches each part of the digital camera 100 through the control unit 101.

  The drive unit 102 is configured by a motor or the like, and mechanically operates an optical system 103 to be described later under a command from the control unit 101. For example, based on a command from the control unit 101, the drive unit 102 moves the position of the focus lens included in the optical system 103 and adjusts the focus position of the optical system 103.

  The optical system 103 includes a zoom lens, a focus lens, a diaphragm, and the like. The diaphragm is a mechanism that adjusts the amount of transmitted light. The focus position can be changed by changing the position of the lens.

  The imaging unit 104 is a photoelectric conversion element, and performs photoelectric conversion that converts an incident optical signal into an electrical signal. For example, a CCD or a CMOS sensor can be applied to the imaging unit 104. The imaging unit 104 is provided with a moving image capturing mode, and can capture a plurality of temporally continuous images as each frame of the moving image.

  FIG. 2 is a diagram for explaining an array of sensors constituting the imaging unit 104 in the present embodiment. One microlens 21 is arranged to correspond to the plurality of photoelectric conversion units 201. A plurality of photoelectric conversion units 201 behind one microlens are collectively defined as a pixel array 20. In FIG. 2, the pixel array 20 includes a total of 25 photoelectric conversion units 201 of 5 rows and 5 columns, but is not limited thereto.

  FIG. 3 is a diagram for explaining the incidence of an optical signal on a pixel in the present embodiment.

  FIG. 3 is a diagram in which a state in which light emitted from the imaging lens 31 passes through one microlens 21 and is received by the imaging element 32 is observed from a direction perpendicular to the optical axis. Light emitted from the pupil regions a1 to a5 of the imaging lens 31 and passed through the microlens 21 is imaged by the corresponding photoelectric conversion units p1 to p5 at the rear.

  Using the imaging unit as illustrated in FIG. 2, it is possible to acquire subject distance information using optical signals acquired by a plurality of photoelectric conversion units corresponding to one microlens.

  The ROM 105 is a read-only nonvolatile memory as a recording medium, and stores parameters and the like necessary for the operation of each block in addition to the operation program for each block provided in the digital camera 100. The RAM 106 is a rewritable volatile memory, and is used as a temporary storage area for data output in the operation of each block included in the digital camera 100.

  The image processing unit 107 performs various image processing, such as white balance adjustment, color interpolation, and filtering, on the image output from the imaging unit 104 or image signal data recorded in the internal memory 109 described later. . Further, compression processing is performed on the data of the image signal captured by the imaging unit 104 in accordance with a standard such as JPEG.

  When the image processing unit 107 is used in combination with an imaging unit having a structure in which a plurality of photoelectric conversion units correspond to one microlens as shown in FIG. 2, an image focused on a focus position within a certain range is obtained. It is possible to reconstruct (refocus) after imaging. The “certain range” here is called a refocus range, and details will be described later.

  The image processing unit 107 includes an integrated circuit (ASIC) that collects circuits for performing specific processing. Alternatively, the control unit 101 may perform some or all of the functions of the image processing unit 107 by performing processing according to a program read from the ROM 105. When the control unit 101 has all the functions of the image processing unit 107, it is not necessary to have the image processing unit 107 as hardware.

  The display unit 108 is an image temporarily stored in the RAM 106, an image stored in an internal memory 109, which will be described later, or a liquid crystal display or an organic EL display for displaying a setting screen of the digital camera 100. is there.

  The built-in memory 109 is a place for recording an image captured by the image capturing unit 104, an image obtained by the processing of the image processing unit 107, information on a focus position at the time of image capturing, and the like. A memory card or the like may be used instead of the built-in memory.

  The operation unit 110 is, for example, a button, switch, key, or mode dial attached to the digital camera 100, or a touch panel that is also used as the display unit 108. A command from the user reaches the control unit 101 via the operation unit 110.

  The apparatus motion detection unit 111 is configured by a gyro sensor and is a device that detects the motion of the digital camera 100, and detects the motion of the digital camera 100 in the yaw direction and the pitch direction based on an angular change per unit time, that is, an angular velocity. .

<How to refocus>
Here, a method for calculating a focus position (refocus plane) corresponding to a subject within a certain range will be described.

  FIG. 4A is a view of the aperture of the imaging lens 31 as viewed from the optical axis direction. FIG. 4B is a view of one microlens 21 and the pixel array 20 disposed behind the microlens 21 as viewed from the optical axis direction. As shown in FIG. 4A, when the pupil region of the imaging lens 31 is divided into the same number of regions as the photoelectric conversion units behind one micro lens, one pupil division of the imaging lens 31 is included in one photoelectric conversion unit. Light emitted from the region is imaged. However, here, it is assumed that the numbers shown on the imaging lens 31 and the microlens 21 are substantially the same.

  The correspondence between the pupil division regions a11 to a55 of the imaging lens 31 shown in FIG. 4A and the pixels p11 to p55 shown in FIG. 4B is point-symmetric when viewed from the optical axis Z direction. Therefore, the light emitted from the pupil division area a11 of the imaging lens 31 forms an image on the pixel p11 in the pixel array 20 behind the microlens. Similarly, light emitted from the pupil division area a11 and passing through another microlens 21 forms an image on the pixel p11 in the pixel array 20 behind the microlens.

  Each photoelectric conversion unit of the pixel array 20 receives light that has passed through different pupil regions with respect to the imaging lens 31. By combining a plurality of pixel signals from these divided signals, a pair of signals divided into pupils in the horizontal direction is generated.

式（１）は、ある画素配列２０の各光電変換部について、撮像レンズ３１の射出瞳の左側領域（瞳領域ａ１１〜ａ５２）を通過した光を積分したものである。これを水平方向に並ぶ複数の画素配列２０に適用し、これらの出力信号群で構成した被写体像をＡ像とする。また、式（２）は、ある画素配列２０の各光電変換部について、撮像レンズ３１の射出瞳の右側領域（瞳領域ａ１４〜ａ５５）を通過した光を積分したものである。これを水平方向に並ぶ複数の画素配列２０に適用し、これらの出力信号群で構成した被写体像をＢ像とする。Ａ像とＢ像に対して相関演算を施し、像のずれ量（瞳分割位相差）を検出する。さらに、像のずれ量に対して撮像レンズ３１の焦点位置と光学系から決まる変換係数を乗じることで、画面内の被写体に対応したピント位置を算出することができる。

Expression (1) is obtained by integrating the light passing through the left area (pupil areas a11 to a52) of the exit pupil of the imaging lens 31 for each photoelectric conversion unit of a certain pixel array 20. This is applied to a plurality of pixel arrays 20 arranged in the horizontal direction, and a subject image constituted by these output signal groups is defined as an A image. Expression (2) is obtained by integrating light passing through the right region (pupil regions a14 to a55) of the exit pupil of the imaging lens 31 for each photoelectric conversion unit of a certain pixel array 20. This is applied to a plurality of pixel arrays 20 arranged in the horizontal direction, and a subject image constituted by these output signal groups is defined as a B image. A correlation operation is performed on the A image and the B image, and an image shift amount (pupil division phase difference) is detected. Furthermore, the focus position corresponding to the subject in the screen can be calculated by multiplying the image shift amount by a conversion coefficient determined by the focal position of the imaging lens 31 and the optical system.

  Next, an image reconstruction process on the refocus plane that is the set focus position with respect to the imaging data acquired by the imaging unit 104 will be described.

  FIG. 5 is a diagram for explaining the calculation of the refocus plane. In FIG. 5, the light passing through a certain pixel on the set refocus plane is emitted from which pupil division region of the imaging lens and to which microlens is viewed from the direction perpendicular to the optical axis Z. In FIG. 5, the position of the pupil division area of the imaging lens 31 is the coordinates (u, v), the pixel position on the refocus plane is the coordinates (x, y), and the position of the microlens on the microlens array is the coordinate (x ′ , Y ′). The distance from the imaging lens to the microlens array is F, and the distance from the imaging lens to the refocus plane is αF. α is a refocus coefficient for determining the position of the refocus plane and can be set by the user. In FIG. 5, only the directions of u, x, and x ′ are shown, and v, y, and y ′ are omitted. As shown in FIG. 5, the light passing through the coordinates (u, v) and the coordinates (x, y) reaches the coordinates (x ′, y ′) on the microlens array. The coordinates (x ′, y ′) can be expressed as in Equation (3).

そして、この光を受光する画素の出力をＬ（ｘ’，ｙ’，ｕ，ｖ）とすると、リフォーカス面上の座標（ｘ，ｙ）で得られる出力Ｅ（ｘ，ｙ）は、Ｌ（ｘ’，ｙ’，ｕ，ｖ）を撮像レンズの瞳領域に関して積分したものとなるため、式（４）のようになる。

When the output of the pixel receiving this light is L (x ′, y ′, u, v), the output E (x, y) obtained with the coordinates (x, y) on the refocus plane is L Since (x ′, y ′, u, v) is integrated with respect to the pupil region of the imaging lens, Equation (4) is obtained.

式（４）において、リフォーカス係数αはユーザによって決定されるため、（ｘ，ｙ）、（ｕ，ｖ）を与えれば、光の入射するマイクロレンズの位置（ｘ’，ｙ’）がわかる。そして、そのマイクロレンズに対応する画素配列２０から（ｕ，ｖ）の位置に対応する画素がわかる。この画素の出力がＬ（ｘ’，ｙ’，ｕ，ｖ）となる。これをすべての瞳分割領域について行い、求めた画素出力を積分することでＥ（ｘ，ｙ）が算出できる。なお、（ｕ，ｖ）を撮像レンズの瞳分割領域の代表座標とすれば、式（４）の積分は、単純加算により計算することができる。

In equation (4), since the refocus coefficient α is determined by the user, if (x, y) and (u, v) are given, the position (x ′, y ′) of the microlens where the light enters can be known. . Then, the pixel corresponding to the position (u, v) is known from the pixel array 20 corresponding to the microlens. The output of this pixel is L (x ′, y ′, u, v). E (x, y) can be calculated by performing this operation for all pupil division regions and integrating the obtained pixel outputs. If (u, v) is a representative coordinate of the pupil division area of the imaging lens, the integral of equation (4) can be calculated by simple addition.

  The refocusing method has been described above. However, a correct refocus image cannot be generated unless the refocus plane described above is set from the focus position at the time of capturing the original image to the focus position in the refocus range. The reason is that the angular distribution of light rays incident on the image sensor, that is, the parallax amount of the parallax image is limited by the aperture system using the imaging lens and the diaphragm, the pixel pitch of the image sensor in the imaging unit, and the like. Next, a method for calculating the refocus range will be described.

<Calculation method of refocus range>
Here, the two-dimensional intensity distribution of light is called a light field spatial component. At this time, the refocus range is determined by the sampling pitch Δy of the spatial component and the sampling pitch Δu of the angular component, and the coefficient α ± is given by the following equation (5).

図６は、リフォーカス範囲を説明するための図である。式（１）を用いて表される像側のリフォーカス範囲（α＋）・ｓ２〜（α−）・ｓ２と、結像光学系６１に対して共役な範囲が、物体側のリフォーカス範囲となる。ここでｓ２は、結像光学系６１の像側主平面と被写体面６２に対する結像光学系６１の像側共役面との間の距離である。

FIG. 6 is a diagram for explaining the refocus range. The refocus range (α +) · s2 to (α−) · s2 on the image side represented by the expression (1) and the range conjugate to the imaging optical system 61 are the refocus range on the object side. Become. Here, s2 is the distance between the image-side main plane of the imaging optical system 61 and the image-side conjugate plane of the imaging optical system 61 with respect to the subject surface 62.

  In the configuration example shown in FIG. 6, since the one-dimensional period of the pixels capturing the same position on the subject surface 62 is three pixels, the sampling pitch Δy of the spatial component is the pixel of the image sensor. 3 times the pitch. The sampling pitch Δu of the angle component is 1/3 of the exit pupil diameter since the exit pupil of the imaging optical system 61 is divided into three (9 in two dimensions). If the refocus range represented by Expression (5) is exceeded, the acquired light field lacks information, and a correct refocus image cannot be generated. Expression (5) can be approximated as the following expression (6) because the pixel pitch Δ of the image sensor is sufficiently small with respect to the pupil distance P of the imaging optical system 61.

ここで、結像光学系６１の瞳距離Ｐとは、結像光学系６１の射出瞳面と被写体面６２に対する結像光学系６１の像側共役面との間の距離である。また、Ｎは結像光学系６１の瞳の１次元分割数、Ｆは結像光学系６１のＦ値、ΔＬＡは画素配列２０のピッチである。

Here, the pupil distance P of the imaging optical system 61 is a distance between the exit pupil plane of the imaging optical system 61 and the image side conjugate plane of the imaging optical system 61 with respect to the subject surface 62. N is the one-dimensional division number of the pupil of the imaging optical system 61, F is the F value of the imaging optical system 61, and ΔLA is the pitch of the pixel array 20.

  The above is the description of the refocus range calculation method. In the following description, “refocus range” refers to a refocus range on the object side unless otherwise specified.

<Flowchart of this embodiment>
FIG. 7 is a flowchart for explaining the present embodiment.

  In step S701, the imaging unit 104 acquires distance information of the subject. In step S <b> 702, the control unit 101 sets a focus position based on subject distance information and user settings. As an example, the user first designates an area to be focused using a touch panel or the like, the imaging unit 104 acquires distance information for the area, and the control unit 101 designates an area designated by the user. Based on the above, the focus position of each image is set.

  In step S703, while the optical system 103 changes the focus position, the imaging unit 104 captures images at each set focus position, and acquires distance information of each captured image.

  In step S704, the control unit 101 determines whether camera shake including movement in the optical axis direction has occurred during the imaging in step S703. Specifically, the focus position at the time of imaging is calculated based on the distance information acquired by the imaging unit 104 in step S704. The calculated focus position at the time of imaging is compared with the set focus position to determine whether or not the camera shake has occurred. As an example, a threshold value of a focus position difference is determined, and it is determined that camera shake has occurred when the calculated focus position and the set focus position are equal to or greater than the threshold value. Alternatively, the device motion detection unit 111 detects the motion motion of the device being imaged, and determines that the camera shake has occurred when the width of the motion exceeds a predetermined threshold value. When hand movement is determined based on the movement motion of the device detected by the device motion detection unit 111, it is not necessary to calculate the focus position at the time of imaging in step S704. Instead, at the time of imaging by generating the refocus image in step S705. The focus position of is calculated.

  Here, the control unit 101 may perform fixing detection, and may determine that there is no camera shake when detecting that the digital camera is fixed to a fixing unit such as a tripod.

  If it is determined that camera shake has occurred, the process advances to step S705 to generate a refocus image. If it is determined that there is no camera shake, in order to omit the refocus process and shorten the processing time, the process directly proceeds to step S706 to perform image composition. In the image composition in step S706, a known method may be used, and an example will be described below. First, for alignment, the absolute value sum (SAD: Sum Of Absolute Difference) of the output differences of the pixels of the plurality of images is obtained while shifting the relative positions of the two images. The relative moving amount and moving direction of the two images when the absolute value sum is the smallest are obtained. Then, after calculating the conversion coefficient of the affine transformation or projective transformation according to the obtained movement amount and the movement direction, minimize the error between the movement amount by the conversion coefficient and the movement amount calculated from the absolute value sum. Optimize transform coefficients using the square method. Based on the optimized conversion coefficient, a deformation process is performed on the image to be aligned. The image processing unit 107 performs the above-described alignment and deformation processing on all the images captured by the imaging unit 104 in step S703, and then gives a composition ratio to each region of each image. As an example, the image processing unit 107 gives a 100% composition ratio to pixels included in a corresponding region of an image having a focused region among a plurality of images corresponding to the same region, A synthesis ratio of 0% is given to the pixels included in the corresponding area. Alternatively, the composition ratio may be assigned to the corresponding area of each image in accordance with the degree of focusing between the images in each area. In order to prevent unnaturalness at the synthesis boundary, the image processing unit 107 causes the synthesis ratio to change stepwise between adjacent pixels. Finally, a composite image is generated based on the composite ratio of each pixel.

  Next, the generation of the refocus image in step S705 will be described in detail.

  FIG. 8 is a diagram for explaining generation of a refocus image. In step S801, as described above, the control unit 101 calculates a refocus range. Next, the control unit 101 calculates a refocus correction amount based on the difference between the set focus position and the focus position at the time of imaging. In step S803, the control unit 101 compares the refocus range and the refocus correction amount, and determines whether refocusing is possible. If the refocus correction amount exceeds the refocus range, it is determined that refocus is not possible, and the process proceeds to step S804. If the refocus correction amount is within the refocus range, it is determined that refocusing is possible, and the process directly proceeds to step S806, where the image processing unit 107 performs refocus processing.

  FIG. 9 is a diagram for explaining an example of the refocus process when it is determined in step S803 that refocus is possible. FIG. 9A shows the focus position set by the control unit 101 in step S702. The range 91 is a range in which a portion where the subject is not in focus does not occur, and is determined from the focal length and the allowable confusion circle diameter of the image sensor. In FIG. 9A, the focus position is set for each range 91. FIG. 9B shows a focus position when the imaging unit 104 actually captures an image. This indicates that all the focus positions other than the first focus position are deviated. In the case of No. 2 and No. 3, etc., the distance of the focus position between the images picked up before that may exceed the range 91, and if a composite image is generated as it is, there is a possibility that a blurred portion will appear. is there. Therefore, refocusing is performed. FIG. 9C shows the focus position after the refocus processing. The number written at the focus position in FIG. 9C means the number from which the focus position is obtained in FIG. 9B, and the refocus processing is performed twice for the focus position with the number 2. Indicates that A correction amount 92 indicates a refocus correction amount. As described above, focus correction cannot be performed unless the refocus correction amount is within the refocus range. Here, when the set focus position is within the refocus range of the focus position at the time of imaging, refocusing is performed using the focus position at the time of imaging with a small correction amount. For example, if the set focus position No. 4 falls within one of the refocus ranges of the focus positions for No. 3 and No. 4 imaging, the No. 3 focus position for No. 3 imaging with a smaller correction amount. It is preferable to perform refocusing using.

  If the control unit 101 determines in step S803 that refocusing is not possible, the control unit 101 proceeds to step S804 and resets the set focus position. In step S805, the refocus correction amount is changed based on the reset focus position. Finally, the process proceeds to step S806, and refocus processing is performed.

  FIG. 10 is a diagram for explaining an example of resetting the focus position and subsequent refocus processing when it is determined in step S803 that refocusing is not possible. FIG. 10A shows the focus position set by the control unit 101 in step S702. FIG. 10B shows a focus position when the imaging unit 104 actually captures an image. FIG. 10C shows the focus position after the refocus processing. In this process, the focus position is re-set with the number 2 and 3 focus positions being the limits of the refocus range, and the other focus positions are reset so that they are equally spaced based on the re-set focus position. Set.

  As illustrated in FIG. 10, when the control unit 101 determines that refocusing cannot be performed up to the initially set focus position, the focus position is reset within a refocusable range.

  According to the present embodiment, when combining using a plurality of images picked up by changing the focus position, it is possible to reduce the influence of camera shake or the like by refocusing the focus position and obtain a higher-quality composite image. it can.

(Other embodiments)
The above embodiment has been described based on the implementation with a digital camera, but is not limited to a digital camera. For example, it may be implemented by a portable device with a built-in image sensor or a network camera capable of capturing an image.

  Note that the present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus execute the program. It can also be realized by a process of reading and operating. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

DESCRIPTION OF SYMBOLS 100 Digital camera 101 Control part 102 Drive part 103 Optical system 104 Imaging part 105 ROM
106 RAM
107 Image processing unit 108 Display unit 109 Built-in memory 110 Operation unit 111 Device motion detection unit

Claims (16)

Combining means for combining a plurality of images taken while changing the focus position of the optical system, and generating a combined image;
A focus position setting means for setting a focus position of the optical system;
Refocusing means for reconstructing an image with different focus positions of the captured image,
The imaging apparatus according to claim 1, wherein the refocusing unit reconstructs an image with a different focus position based on a difference between the set focus position and a focus position when the image is captured. An imaging unit that captures the image;
The imaging apparatus according to claim 1, wherein the imaging unit includes a plurality of photoelectric conversion units for one microlens.   The refocusing unit reconstructs an image with a different focus position so that at least a part of the focus position of the captured image becomes one of the set focus positions. The imaging device according to claim 1 or 2.   The refocusing unit reconstructs an image in which the focus position of the image is changed so as to be the set focus position, using a captured image having a focus position closest to the set focus position. The imaging apparatus according to claim 3. It has a camera shake information acquisition means for acquiring camera shake information at the time of imaging,
5. The imaging apparatus according to claim 1, wherein the refocusing unit determines whether to reconstruct an image with a different focus position based on the camera shake information. .   The camera shake information acquisition means determines whether to reconstruct an image with a different focus position based on a difference between the set focus position and the focus position at the time of imaging. The imaging device according to claim 5.   The camera shake information acquisition means reconstructs an image with a different focus position when a difference between the set focus position and the focus position at the time of imaging is equal to or greater than a predetermined threshold. The imaging apparatus according to claim 6. When the image pickup apparatus picks up an image, the image pickup apparatus has a fixed detection means for detecting whether or not the image pickup apparatus is fixed to a fixed portion,
8. The refocusing unit according to claim 1, wherein the refocusing unit does not reconstruct an image obtained by changing the focus position with respect to an image captured while the imaging device is fixed to the fixing unit. The imaging device according to any one of the above.   The imaging apparatus according to claim 8, wherein the fixing unit is a tripod. Having a refocus range calculation means for calculating the refocus range;
10. The imaging apparatus according to claim 1, wherein the refocusing unit reconstructs an image with a different focus position of the image within the refocusing range. . A focus position resetting unit for resetting the focus position of the image;
When the focus position set by the focus position setting means is not within the refocus range of any focus position of the captured image,
The imaging apparatus according to claim 10, wherein the focus position resetting unit resets the set focus position.   The imaging apparatus according to claim 11, wherein the focus position resetting unit resets the set focus position at a position corresponding to a limit of a refocus range of a focus position of the captured image.   The imaging apparatus according to claim 11 or 12, wherein the focus position resetting unit performs the resetting so that at least a part of the focus positions after resetting are at equal intervals. When the set focus position is within the refocus range of the focus positions of the plurality of captured images,
14. The refocusing is performed on the image having the smallest amount of refocus correction among the plurality of the captured images so that the set focus position is obtained. The imaging device described in 1. Combining a plurality of images captured while changing the focus position of the optical system and generating a combined image;
A focus position setting step for setting a focus position of the optical system;
A refocusing step for reconstructing an image with a different focus position of the captured image,
In the refocusing step, an image is reconstructed based on a difference between the set focus position and a focus position when the image is captured, and the focus position is changed. A program for causing a computer of an imaging apparatus to operate,
In the computer,
Combining a plurality of images captured while changing the focus position of the optical system and generating a combined image;
A focus position setting step for setting a focus position of the optical system;
A refocusing step for reconstructing an image with a different focus position of the captured image, and
A computer program characterized in that in the refocusing step, an image with a different focus position is reconstructed based on a difference between the set focus position and a focus position when the image is captured.

Images