JP6728005B2 - Imaging device, imaging method, and program - Google Patents

Description

The present invention relates to an image pickup apparatus that performs image synthesis.

When shooting multiple subjects at different distances from the camera, or when shooting a subject that is long in the depth direction, the depth of field in the imaging optical system is insufficient, so only part of the subject is focused. You may not be able to. Therefore, the focus position is changed, a plurality of images are captured, only the in-focus area is extracted from each image, and one image is combined to generate a combined image in which the entire imaging area is in focus. A technique (hereinafter, referred to as depth synthesis) is known (see Patent Document 1). By using this depth stacking technique, an image in which the entire intended subject is in focus can be obtained.

JP, 2005-216480, A

However, when an image is captured using the depth stacking technique, the following problems may occur if camera shake or the like occurs.

FIG. 11 is a diagram for explaining camera shake in depth stacking. FIG. 11A is a diagram for explaining imaging using ideal depth stacking. In FIG. 11A, the user puts the digital camera on 1101 and takes a plurality of images while moving the focus position by a predetermined distance 1102.

FIG. 11B shows a case where a camera shake occurs when the third image is captured in the image capturing of FIG. 11A. Since the image pickup position 1101 is shifted rearward in the optical axis direction, the focus position comes to a position closer than the distance to be originally imaged. Further, as shown in FIG. 11C, when the fourth image is captured, a case where a camera shake occurs in a direction opposite to the camera shake direction of FIG. 11B is shown. In this case, the fourth image is farther than the focus position that should be originally captured. When the combining process is performed using the plurality of images captured in FIG. 11C, there is no focused image in the area 1103, and a blurred portion remains in the combined image.

The present invention has been made in view of the above problems, and when combining using a plurality of images captured while changing the focus position, a composite image that corrects the focus position of the images and reduces blurring due to camera shake, etc. It is an object of the present invention to provide an imaging device capable of generating

According to the present invention, a synthesizing unit that synthesizes a plurality of images captured while changing a focus position of an optical system to generate a synthetic image, a focus position setting unit that sets a focus position of the optical system, and Refocusing means for reconstructing an image in which the focus position of the captured image is changed, and the refocusing means is configured to detect the difference between the set focus position and the focus position when capturing the image. Based on the above, an image pickup apparatus is provided which is configured to reconstruct an image with a different focus position.

According to the present invention, when a plurality of images captured by changing the focus position are used for composition, it is possible to generate a composite image that reduces blurring due to camera shake or the like.

It is a block diagram which shows the structure of the digital camera in embodiment of this invention. It is a figure for demonstrating the array of the sensor which comprises the imaging part in embodiment of this invention. FIG. 6 is a diagram for explaining how an optical signal enters a pixel in the embodiment of the present invention. (A) is a view of the aperture of the imaging lens as viewed from the optical axis direction, and (b) is a view of one microlens and a pixel array arranged behind it as viewed from the optical axis direction. It is a figure for demonstrating calculation of the refocus surface in embodiment of this invention. It is a figure for demonstrating the refocus range in embodiment of this invention. It is a flow chart for explaining the embodiment of the present invention. FIG. 6 is a diagram for explaining generation of a refocus image in the embodiment of the present invention. FIG. 6 is a diagram for explaining an example of refocus processing when it is determined that refocus is possible in the embodiment of the present invention. FIG. 6 is a diagram for explaining an example of resetting of a focus position and a subsequent refocusing process when it is determined that refocusing is not possible in the embodiment of the present invention. It is a figure for explaining camera shake in depth composition.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description, a digital camera will be described as an example, but the present invention is not limited to a digital camera, and an imaging device capable of incorporating focus position information and correcting the focus position of a captured image is described. If used, the present invention can be applied.

(One Example of Embodiment of the Present Invention)
FIG. 1 is a block diagram showing the structure of a digital camera according to this embodiment. The digital camera 100 is capable of capturing a still image, recording information on a focus position, calculating a contrast value, and combining images. Further, the digital camera 100 can perform enlargement processing or reduction processing on an image captured and stored or an image input from the outside.

The control unit 101 is, for example, a signal processor such as a CPU or MPU, and controls each unit of the digital camera 100 while reading a program stored in a ROM 105 described later in advance. For example, as will be described later, the control unit 101 issues a command to the image capturing unit 104, which will be described later, to start and end image capturing. Alternatively, an image processing command is issued to an image processing unit 107, which will be described later, based on a program stored in the ROM 105. The command from the user is input to the digital camera 100 by the operation unit 110 described later, and reaches each part of the digital camera 100 through the control unit 101.

The drive unit 102 is configured by a motor or the like, and mechanically operates an optical system 103 described later under the command of the control unit 101. For example, based on a command from the control unit 101, the driving unit 102 moves the position of the focus lens included in the optical system 103 to adjust the focus position of the optical system 103.

The optical system 103 includes a zoom lens, a focus lens, a diaphragm, and the like. The diaphragm is a mechanism for adjusting the amount of transmitted light. The focus position can be changed by changing the position of the lens.

The image pickup unit 104 is a photoelectric conversion element, and performs photoelectric conversion for converting an incident optical signal into an electric signal. For example, a CCD, a CMOS sensor, or the like can be applied to the image pickup unit 104. The image capturing unit 104 is provided with a moving image capturing mode and can capture a plurality of temporally consecutive images as each frame of a moving image.

FIG. 2 is a diagram for explaining an array of sensors that constitute the image pickup unit 104 in this embodiment. One microlens 21 is arranged so as to correspond to the plurality of photoelectric conversion units 201. A plurality of photoelectric conversion units 201 behind one microlens are collectively defined as a pixel array 20. In FIG. 2, the pixel array 20 has a total of 25 photoelectric conversion units 201 in 5 rows and 5 columns, but the number is not limited to this.

FIG. 3 is a diagram for explaining the incidence of an optical signal on a pixel in the present embodiment.

FIG. 3 is a diagram in which light emitted from the image pickup lens 31 passes through one microlens 21 and is received by the image pickup element 32, as observed from a direction perpendicular to the optical axis. The light emitted from each of the pupil regions a1 to a5 of the imaging lens 31 and passing through the microlens 21 forms an image in the corresponding photoelectric conversion units p1 to p5 in the rear.

Using the image pickup unit as shown in FIG. 2, it is possible to obtain the information on the subject distance by using the optical signals obtained by the plurality of photoelectric conversion units corresponding to one microlens.

The ROM 105 is a read-only non-volatile memory as a recording medium, and stores, in addition to the operation program of each block included in the digital camera 100, parameters necessary for the operation of each block. The RAM 106 is a rewritable volatile memory and is used as a temporary storage area of data output in the operation of each block included in the digital camera 100.

The image processing unit 107 performs various image processing such as white balance adjustment, color interpolation, and filtering on the image output from the image capturing unit 104 or the image signal data recorded in the internal memory 109 described later. .. Further, the image signal data captured by the image capturing unit 104 is compressed according to a standard such as JPEG.

When the image processing unit 107 is used in combination with the imaging unit having a structure in which a plurality of photoelectric conversion units correspond to one microlens as shown in FIG. 2, an image focused at a focus position within a certain range is obtained. After imaging, it can be reconstructed (refocused). The “certain range” here is called a refocus range, and the details will be described later.

The image processing unit 107 is composed of an integrated circuit (ASIC) that is a collection of circuits that perform specific processing. Alternatively, the control unit 101 may perform some or all of the functions of the image processing unit 107 by processing according to the program read from the ROM 105. When the control unit 101 has all the functions of the image processing unit 107, it is not necessary to have the image processing unit 107 as hardware.

The display unit 108 is an image temporarily stored in the RAM 106, an image stored in a built-in memory 109 described later, or a liquid crystal display or an organic EL display for displaying a setting screen of the digital camera 100 or the like. is there.

The built-in memory 109 is a place for recording an image captured by the image capturing unit 104, an image processed by the image processing unit 107, information on a focus position at the time of capturing an image, and the like. A memory card or the like may be used instead of the built-in memory.

The operation unit 110 is, for example, a button or switch attached to the digital camera 100, a key, a mode dial, or the like, or a touch panel also used as the display unit 108. The command from the user reaches the control unit 101 via the operation unit 110.

The device motion detection unit 111 is a device that is configured by a gyro sensor and that detects the motion of the digital camera 100. The device motion detection unit 111 detects the motion of the digital camera 100 in the yaw direction and the pitch direction based on the angular change per unit time, that is, the angular velocity. ..

<Refocus method>
Here, a method of calculating the focus position (refocus surface) corresponding to the subject within a certain range will be described.

FIG. 4A is a view of the opening of the imaging lens 31 as seen from the optical axis direction. FIG. 4B is a diagram of one microlens 21 and the pixel array 20 arranged behind the microlens 21, as viewed from the optical axis direction. As shown in FIG. 4A, when the pupil area of the imaging lens 31 is divided into the same number of areas as the photoelectric conversion sections behind one microlens, one pupil division of the imaging lens 31 is divided into one photoelectric conversion section. The light emitted from the area is imaged. However, here, it is assumed that the numbers shown on the imaging lens 31 and the microlens 21 are substantially the same.

The correspondence between the pupil division areas a11 to a55 of the imaging lens 31 shown in FIG. 4A and the pixels p11 to p55 shown in FIG. 4B is point symmetric when viewed from the optical axis Z direction. Therefore, the light emitted from the pupil division area a11 of the imaging lens 31 forms an image on the pixel p11 in the pixel array 20 behind the microlenses. Similarly, light emitted from the pupil division area a11 and passing through another microlens 21 is also imaged on the pixel p11 in the pixel array 20 behind the microlens.

Each photoelectric conversion unit of the pixel array 20 receives light that has passed through different pupil regions with respect to the imaging lens 31. By combining a plurality of pixel signals from these divided signals, a pair of signals that are pupil-divided in the horizontal direction are generated.

式（１）は、ある画素配列２０の各光電変換部について、撮像レンズ３１の射出瞳の左側領域（瞳領域ａ１１〜ａ５２）を通過した光を積分したものである。これを水平方向に並ぶ複数の画素配列２０に適用し、これらの出力信号群で構成した被写体像をＡ像とする。また、式（２）は、ある画素配列２０の各光電変換部について、撮像レンズ３１の射出瞳の右側領域（瞳領域ａ１４〜ａ５５）を通過した光を積分したものである。これを水平方向に並ぶ複数の画素配列２０に適用し、これらの出力信号群で構成した被写体像をＢ像とする。Ａ像とＢ像に対して相関演算を施し、像のずれ量（瞳分割位相差）を検出する。さらに、像のずれ量に対して撮像レンズ３１の焦点位置と光学系から決まる変換係数を乗じることで、画面内の被写体に対応したピント位置を算出することができる。

Expression (1) is an integration of the light that has passed through the left side region (pupil regions a11 to a52) of the exit pupil of the imaging lens 31 for each photoelectric conversion unit of a certain pixel array 20. This is applied to a plurality of pixel arrays 20 arranged in the horizontal direction, and a subject image composed of these output signal groups is taken as A image. Further, Expression (2) is an integration of the light that has passed through the right side region (pupil regions a14 to a55) of the exit pupil of the imaging lens 31 for each photoelectric conversion unit of a certain pixel array 20. This is applied to a plurality of pixel arrays 20 arranged in the horizontal direction, and a subject image composed of these output signal groups is taken as a B image. Correlation calculation is performed on the A image and the B image to detect the image shift amount (pupil division phase difference). Further, the focus position corresponding to the subject in the screen can be calculated by multiplying the image shift amount by the conversion coefficient determined by the focal position of the imaging lens 31 and the optical system.

Next, with respect to the imaging data acquired by the imaging unit 104, an image reconstructing process on the refocus plane which is the set focus position will be described.

FIG. 5 is a diagram for explaining the calculation of the refocus surface. In FIG. 5, it is seen from a direction perpendicular to the optical axis Z from which pupil division area of the image pickup lens the light passing through a certain pixel on the set refocusing surface is emitted and into which microlens. In FIG. 5, the position of the pupil division area of the imaging lens 31 is the coordinate (u, v), the pixel position on the refocusing surface is the coordinate (x, y), and the position of the microlens on the microlens array is the coordinate (x′). , Y'). Further, the distance from the imaging lens to the microlens array is F, and the distance from the imaging lens to the refocus surface is αF. α is a refocus coefficient for determining the position of the refocus surface and can be set by the user. In FIG. 5, only u, x, and x'directions are shown, and v, y, and y'are omitted. As shown in FIG. 5, the light passing through the coordinates (u, v) and the coordinates (x, y) reaches the coordinates (x', y') on the microlens array. The coordinates (x', y') can be expressed as in equation (3).

そして、この光を受光する画素の出力をＬ（ｘ’，ｙ’，ｕ，ｖ）とすると、リフォーカス面上の座標（ｘ，ｙ）で得られる出力Ｅ（ｘ，ｙ）は、Ｌ（ｘ’，ｙ’，ｕ，ｖ）を撮像レンズの瞳領域に関して積分したものとなるため、式（４）のようになる。

If the output of the pixel that receives this light is L(x', y', u, v), the output E(x, y) obtained at the coordinates (x, y) on the refocus plane is L. Since (x', y', u, v) is integrated with respect to the pupil area of the imaging lens, it becomes as shown in Expression (4).

式（４）において、リフォーカス係数αはユーザによって決定されるため、（ｘ，ｙ）、（ｕ，ｖ）を与えれば、光の入射するマイクロレンズの位置（ｘ’，ｙ’）がわかる。そして、そのマイクロレンズに対応する画素配列２０から（ｕ，ｖ）の位置に対応する画素がわかる。この画素の出力がＬ（ｘ’，ｙ’，ｕ，ｖ）となる。これをすべての瞳分割領域について行い、求めた画素出力を積分することでＥ（ｘ，ｙ）が算出できる。なお、（ｕ，ｖ）を撮像レンズの瞳分割領域の代表座標とすれば、式（４）の積分は、単純加算により計算することができる。

In the formula (4), the refocusing coefficient α is determined by the user, and therefore, if (x, y) and (u, v) are given, the position (x′, y′) of the microlens on which the light enters can be known. .. Then, from the pixel array 20 corresponding to the microlens, the pixel corresponding to the position (u, v) can be known. The output of this pixel is L(x', y', u, v). This is performed for all pupil division areas, and E(x, y) can be calculated by integrating the obtained pixel output. If (u, v) is the representative coordinate of the pupil division area of the imaging lens, the integral of the equation (4) can be calculated by simple addition.

The method of refocusing has been described above. However, a correct refocus image cannot be generated unless the refocus surface described above is set from the focus position at the time of capturing the original image to the focus position in the refocus range. The reason is that the angular distribution of light rays incident on the image pickup element, that is, the parallax amount of the parallax image is limited by the aperture system of the image pickup lens and the diaphragm, the pixel pitch of the image pickup element in the image pickup unit, and the like. Next, a method of calculating the refocus range will be described.

<Calculation method of refocus range>
Here, the two-dimensional intensity distribution of light is called a light field spatial component. At this time, the refocus range is determined by the sampling pitch Δy of the spatial component and the sampling pitch Δu of the angle component, and its coefficient α± is given by the following equation (5).

図６は、リフォーカス範囲を説明するための図である。式（１）を用いて表される像側のリフォーカス範囲（α＋）・ｓ２〜（α−）・ｓ２と、結像光学系６１に対して共役な範囲が、物体側のリフォーカス範囲となる。ここでｓ２は、結像光学系６１の像側主平面と被写体面６２に対する結像光学系６１の像側共役面との間の距離である。

FIG. 6 is a diagram for explaining the refocus range. The image-side refocus range (α+)·s2 to (α−)·s2 expressed using the equation (1) and the range conjugate to the imaging optical system 61 are the object-side refocus range. Become. Here, s2 is the distance between the image-side principal plane of the imaging optical system 61 and the image-side conjugate plane of the imaging optical system 61 with respect to the subject plane 62.

In the configuration example shown in FIG. 6, the one-dimensional cycle of pixels that image the same position on the subject surface 62 is three pixels, so the sampling pitch Δy of the spatial component is the pixel of the image sensor. 3 times the pitch. The sampling pitch Δu of the angle component is ⅓ of the exit pupil diameter because the exit pupil of the imaging optical system 61 is divided into three (two-dimensionally divided into nine). If the refocus range represented by the equation (5) is exceeded, the acquired light field lacks information, and a correct refocus image cannot be generated. Since the pixel pitch Δ of the image sensor is sufficiently smaller than the pupil distance P of the imaging optical system 61, the equation (5) can be approximated as the following equation (6).

ここで、結像光学系６１の瞳距離Ｐとは、結像光学系６１の射出瞳面と被写体面６２に対する結像光学系６１の像側共役面との間の距離である。また、Ｎは結像光学系６１の瞳の１次元分割数、Ｆは結像光学系６１のＦ値、ΔＬＡは画素配列２０のピッチである。

Here, the pupil distance P of the imaging optical system 61 is the distance between the exit pupil plane of the imaging optical system 61 and the image-side conjugate plane of the imaging optical system 61 with respect to the object plane 62. N is the number of one-dimensional divisions of the pupil of the imaging optical system 61, F is the F value of the imaging optical system 61, and ΔLA is the pitch of the pixel array 20.

The above is the description of the method of calculating the refocus range. In the following, the "refocus range" refers to the refocus range on the object side unless otherwise specified.

<Flowchart of this embodiment>
FIG. 7 is a flowchart for explaining this embodiment.

In step S701, the imaging unit 104 acquires distance information of the subject. In step S702, the control unit 101 sets the focus position based on the subject distance information and the user setting. As an example, the user first uses a touch panel or the like to specify a region to be in focus, the imaging unit 104 acquires distance information for the region, and the control unit 101 sets the region specified by the user. Based on, the focus position of each image is set.

In step S703, while the optical system 103 changes the focus position, the image capturing unit 104 captures an image at each set focus position and acquires distance information of each captured image.

In step S704, the control unit 101 determines whether or not camera shake including movement in the optical axis direction has occurred during the image capturing in step S703. Specifically, the image capturing unit 104 calculates the focus position at the time of image capturing based on the distance information acquired in step S704. The calculated focus position at the time of image capturing is compared with the set focus position to determine whether the camera shake has occurred. As an example, a threshold for the difference between the focus positions is determined, and when the calculated focus position at the time of imaging and the set focus position is equal to or greater than the threshold, it is determined that the camera shake has occurred. Alternatively, the device motion detection unit 111 detects a motion motion of the device during imaging, and when the width of the motion is equal to or greater than a predetermined threshold value, it is determined that the camera shake has been performed. When the camera movement is determined by the movement of the device detected by the device movement detection unit 111, the focus position at the time of image capturing does not have to be calculated at step S704, but instead at the time of image capturing at the generation of the refocus image at step S705. The focus position of is calculated.

Further, here, the control unit 101 may perform fixation detection, and when it is detected that the digital camera is fixed to a fixation unit such as a tripod, the control unit 101 may determine that there is no camera shake.

If it is determined that the camera shake has occurred, the process advances to step S705 to generate a refocus image. If it is determined that the camera shake has not occurred, in order to omit the refocus processing and reduce the processing time, the process directly proceeds to step S706 and image combination is performed. A known method may be used for the image combination in step S706, and an example will be described below. First, for alignment, while shifting the relative positions of the two images, the absolute value sum (SAD: Sum Of Absolute Difference) of the output differences of the pixels of the plurality of images is obtained. The relative movement amount and movement direction of the two images when the value of the sum of absolute values becomes the smallest are obtained. Then, after calculating the conversion coefficient of the affine transformation or projective conversion according to the calculated movement amount and the movement direction, the error is minimized so that the error between the movement amount by the conversion coefficient and the movement amount calculated from the sum of absolute values is minimized. Optimize the transform coefficients using the square method. Based on the optimized conversion coefficient, the deformation process is performed on the image to be aligned. The image processing unit 107 performs the above-described alignment and deformation processing on all the images captured by the image capturing unit 104 in step S703, and then gives a synthesis ratio to each area of each image. As an example, the image processing unit 107 gives a combination ratio of 100% to pixels included in a corresponding area of an image having an in-focus area among a plurality of images corresponding to the same area, and other images A synthesis ratio of 0% is given to the pixels included in the corresponding area. Alternatively, the composition ratio may be assigned to the corresponding area of each image according to the degree of focusing between the images of each area. In addition, in order to prevent unnaturalness at the composition boundary, the image processing unit 107 causes the composition ratio to change stepwise between adjacent pixels. Finally, a composite image is generated based on the composite ratio of each pixel.

Next, the generation of the refocus image in step S705 will be described in detail.

FIG. 8 is a diagram for explaining generation of a refocus image. In step S801, as described above, the control unit 101 calculates the refocus range. Next, the control unit 101 calculates the refocus correction amount based on the difference between the set focus position and the focus position at the time of image capturing. In step S803, the control unit 101 compares the refocus range with the refocus correction amount, and determines whether refocus is possible. When the refocus correction amount exceeds the refocus range, it is determined that refocus is not possible, and the process proceeds to step S804. If the refocus correction amount is within the refocus range, it is determined that refocus is possible, and the process directly proceeds to step S806, where the image processing unit 107 performs the refocus processing.

FIG. 9 is a diagram for explaining an example of the refocusing process when it is determined that refocusing is possible in step S803. FIG. 9A shows the focus position set by the control unit 101 in step S702. The range 91 is a range in which a part where the subject is out of focus does not occur, and is determined from the focal length and the permissible circle of confusion of the image sensor. In FIG. 9A, the focus position is set for each range 91. FIG. 9B shows the focus position when the image capturing unit 104 actually captures an image. It shows that all but the first focus position deviates from the set focus positions. In the case of No. 2 and No. 3, the distance of the focus position from the image captured before that may exceed the range 91, and if the composite image is generated as it is, there is a possibility that a blurred portion may appear. is there. Therefore, refocus is performed. FIG. 9C shows the focus position after the refocus processing. The number written in the focus position of FIG. 9C means from which focus position of FIG. 9B the focus position was obtained, and the focus position of the number 2 is refocused twice. Indicates that The correction amount 92 indicates the refocus correction amount. As described above, focus correction cannot be performed unless the refocus correction amount is within the refocus range. Here, if the set focus position is within the refocus range of a plurality of focus positions at the time of imaging, refocusing is performed using the focus position at the time of imaging with a small correction amount. For example, if the set focus position of No. 4 falls within the refocus range of either the focus position at the time of imaging of No. 3 or No. 4, the focus position at the time of imaging of No. 3 with a smaller correction amount. It is preferable to perform refocusing using.

When it is determined in step S803 that refocusing is not possible, the control unit 101 proceeds to step S804 and resets the set focus position. Next, in step S805, the refocus correction amount is changed based on the reset focus position. Finally, in step S806, refocus processing is performed.

FIG. 10 is a diagram for explaining an example of resetting of the focus position when it is determined that refocusing is not possible in step S803 and subsequent refocusing processing. FIG. 10A shows the focus position set by the control unit 101 in step S702. FIG. 10B shows the focus position when the image capturing unit 104 actually captures an image. FIG. 10C shows the focus position after the refocus processing. In this process, the focus positions of Nos. 2 and 3 are reset to the limits of the refocus range, and other focus positions are reset so that they are evenly spaced based on the reset focus positions. Set.

As shown in FIG. 10, when the control unit 101 determines that the focus position that was initially set cannot be refocused, the focus position is reset within the refocusable range.

According to the present embodiment, when synthesizing using a plurality of images captured by changing the focus position, refocusing the focus position can reduce the influence of camera shake or the like and obtain a higher quality composite image. it can.

(Other embodiments)
The above embodiment has been described based on the implementation in the digital camera, but is not limited to the digital camera. For example, it may be implemented by a mobile device or the like having a built-in image sensor, or may be a network camera or the like capable of capturing an image.

The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus provide the program. Can also be realized by a process of reading out and operating. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100 Digital Camera 101 Control Unit 102 Drive Unit 103 Optical System 104 Imaging Unit 105 ROM
106 RAM
107 image processing unit 108 display unit 109 built-in memory 110 operation unit 111 device motion detection unit

Claims (16)

A synthesizing unit that synthesizes a plurality of images captured while changing the focus position of the optical system to generate a synthesized image,
Focus position setting means for setting the focus position of the optical system,
Refocusing means for reconstructing an image in which the focus position of the captured image is different,
The image pickup apparatus, wherein the refocusing unit reconstructs an image with a different focus position based on a difference between the set focus position and a focus position when the image is captured. An image pickup unit for picking up the image,
The imaging device according to claim 1, wherein the imaging unit has a plurality of photoelectric conversion units for one microlens. The refocusing means reconstructs an image in which the focus position is different so that at least a part of the focus position of the captured image is one of the set focus positions. The image pickup apparatus according to claim 1. The refocusing means reconstructs an image in which the focus position of the image is changed so that the set focus position is obtained, by using the imaged image having the closest focus position to the set focus position. The imaging device according to claim 3, wherein It has a camera shake information acquisition unit that acquires camera shake information at the time of imaging,
The imaging device according to claim 1, wherein the refocusing unit determines whether to reconstruct an image with a different focus position, based on the camera shake information. .. The refocusing unit determines whether to reconstruct an image with a different focus position based on a difference between the set focus position and the focus position when the image is captured. The imaging device according to 5. The refocusing means reconstructs an image in which the focus position is different when the difference between the set focus position and the focus position at the time of imaging is equal to or more than a predetermined threshold value. The imaging device according to claim 6. When the image pickup device picks up an image, it has a fixing detection means for detecting whether or not it is fixed to a fixed portion,
8. The refocusing unit does not reconstruct an image with a different focus position with respect to an image captured while the image pickup device is fixed to the fixing unit. The imaging device according to any one of 1. The imaging device according to claim 8, wherein the fixed portion is a tripod. A refocus range calculation means for calculating the refocus range,
10. The image pickup apparatus according to claim 1, wherein the refocusing unit reconstructs an image with a different focus position of the image within the refocusing range. .. Having a focus position resetting means for resetting the focus position of the image,
When the focus position set by the focus position setting unit is not within the refocus range of any focus position of the captured image,
The imaging apparatus according to claim 10, wherein the focus position resetting unit resets the set focus position. The image pickup apparatus according to claim 11, wherein the focus position resetting unit resets the set focus position at a position within a refocus range limit of the focus position of the captured image. 13. The image pickup apparatus according to claim 11, wherein the focus position resetting unit performs the resetting so that the focus positions after at least a part of the resetting are at equal intervals. When the set focus position is in the refocus range of the focus positions of the plurality of captured images,
14. The refocusing is performed on the image having the smallest refocus correction amount out of the plurality of captured images so that the set focus position is reached. The imaging device according to. A combining step of combining a plurality of images captured while changing the focus position of the optical system to generate a combined image,
A focus position setting step for setting the focus position of the optical system,
A refocusing step of reconstructing an image in which the focus position of the captured image is different,
In the refocusing step, an image is reconstructed in which the focus position is changed based on a difference between the set focus position and a focus position when the image is taken. A program that causes a computer of an imaging device to operate,
On the computer,
A combining step of combining a plurality of images captured while changing the focus position of the optical system to generate a combined image,
A focus position setting step for setting the focus position of the optical system,
A refocus step of reconstructing an image in which the focus position of the captured image is changed,
A computer program characterized in that, in the refocusing step, an image with a different focus position is reconstructed based on a difference between the set focus position and a focus position when the image is captured.

Images