JP2013258449A - Imaging apparatus and control method of the same - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging apparatus capable of capturing a main object at a high resolution.SOLUTION: The imaging apparatus capable of generating a plurality of second images with different focal positions by reconfiguring a first image includes: an image formation optical system; an image pickup device that has a plurality of pixels and captures the first image; a pupil division part that lets rays from the same position on an object surface enter mutually different pixels of the image pickup device according to a pupil area of the image formation optical system, through which the ray passes; a focus adjustment part that adjusts an image formation position of the image formation optical system; and an image processing part that generates the second images by reconfiguring the first image captured by the image pickup device using image pickup condition information of the first image. The focus adjustment part performs a first focus adjustment that adjusts an image formation position of the image formation optical system with respect to a reference object, and performs a second focus adjustment that moves an image formation position of the image formation optical system with respect to a main object according to a deviation amount of a pixel composed by the reconfiguration so as to enhance a resolution of the main object.

Description

  The present invention relates to an imaging apparatus capable of generating a plurality of output images having different focus positions by reconstructing an input image.

  2. Description of the Related Art In recent years, there have been proposed imaging devices that perform various operations on data obtained by an imaging device and perform various digital image processing to output various images. In Non-Patent Document 1 and Non-Patent Document 2, two-dimensional light intensity distribution and parallax information (collectively referred to as “light field”) on the subject surface are simultaneously used by using “Light Field Photography”. An imaging device to acquire is disclosed. According to such an imaging apparatus, it is possible to obtain three-dimensional information of the subject space by acquiring the light field, and by changing the focus position of the image, called refocusing, The depth of field can be adjusted. However, in such an imaging apparatus, it is necessary to assign the pixels of the imaging element not only to the two-dimensional intensity distribution of light but also to save parallax information. For this reason, a spatial resolution falls with respect to the imaging device which preserve | saves only the two-dimensional intensity distribution of light.

  Non-Patent Document 2 discloses a configuration in which a plurality of small images are acquired from a certain point on an image surface formed by an imaging optical system by using a plurality of small lenses constituting a lens array. By reconstructing a plurality of small images obtained in this way, it is possible to increase the resolution of the reconstructed image. Such a high resolution technique is called “pixel shifting super-resolution”.

Ren Ng, et al. , “Light Field Photography with a Hand-held Plenological Camera”, 2005 Computer Science Technical Report CTSR. Todor Georgiev, et al. , “Superresolution with Plenoptic 2.0 Camera”, 2009 Optical Society of America.

  Non-Patent Document 2 discloses a method of obtaining a super-resolution effect by shifting pixels at a specific focus position. However, there is no disclosure regarding the effect of pixel shift super-resolution when generating an image refocused to an arbitrary focus position. Since the effect of pixel-resolved super-resolution changes depending on the focus position, there may be a focus position where the resolution is higher than a specific focus position within a refocusable range. As described above, the subject (main subject) designated by the user at the time of shooting does not necessarily exist at the focus position where the resolution is high.

  Therefore, the present invention provides an imaging apparatus capable of acquiring a main subject with high resolution and a method for controlling the imaging apparatus.

  An imaging apparatus according to one aspect of the present invention is an imaging apparatus that can generate a plurality of second images having different focus positions by reconstructing a first image, and includes an imaging optical system and a plurality of pixels. An image sensor that acquires the first image and a light beam from the same position on the subject surface is incident on different pixels of the image sensor according to a pupil region of the imaging optical system through which the light beam passes. A pupil division unit, a focus adjustment unit that adjusts the imaging position of the imaging optical system, and the first image acquired by the image sensor using the imaging condition information of the first image. An image processing unit that generates the second image by performing the configuration, and the focus adjustment unit includes a first focus adjustment that adjusts an imaging position of the imaging optical system with respect to a reference subject, Depending on the amount of displacement of the pixels synthesized by the configuration, For the main subject so as to increase the resolution of Utsushitai performing a second focus adjustment to move the imaging position of the imaging optical system.

  An imaging apparatus control method according to another aspect of the present invention is an imaging apparatus control method capable of generating a plurality of second images having different focus positions by reconstructing a first image, the imaging apparatus The first focus adjustment for adjusting the imaging position of the imaging optical system included in the imaging optical system with respect to the reference subject, and the resolution of the main subject to increase the resolution of the main subject according to the shift amount of the pixel synthesized by the reconstruction Performing a second focus adjustment for moving an imaging position of the imaging optical system with respect to a main subject; and acquiring the first image by photographing with the imaging device.

  Other objects and features of the present invention are illustrated in the following examples.

  According to the present invention, it is possible to provide an imaging apparatus capable of acquiring a main subject with high resolution and a control method for the imaging apparatus.

1 is a schematic configuration diagram of an imaging optical system in Embodiment 1. FIG. 6 is a schematic configuration diagram of an imaging optical system in Embodiment 2. FIG. 6 is a schematic configuration diagram of an imaging optical system in Embodiment 3. FIG. 1 is a block diagram of an imaging apparatus in Embodiments 1 and 2. FIG. 6 is an explanatory diagram relating to a refocus image generation method according to Embodiment 1. FIG. 6 is an explanatory diagram of a refocus range in Embodiment 1. FIG. It is explanatory drawing of the relationship between the position of a virtual image plane, and the spatial resolution of a refocus image. 6 is a flowchart illustrating a shooting procedure of the imaging apparatus according to the first and second embodiments. FIG. 10 is an explanatory diagram relating to a refocus image generation method according to the second embodiment. FIG. 10 is an explanatory diagram of a refocus range in Embodiment 2. FIG. 10 is an explanatory diagram relating to the number of overlapping pixels during reconstruction processing in the second embodiment. FIG. 10 is a block diagram of an image processing system in Embodiment 3. FIG. 10 is a block diagram of an imaging apparatus in Embodiment 3. 10 is a flowchart illustrating a shooting procedure of the imaging apparatus according to the third embodiment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In each figure, the same members are denoted by the same reference numerals, and redundant description is omitted.

  The imaging apparatus and the imaging apparatus control method of the present embodiment can generate a plurality of second images (output images) having different focus positions by reconstructing the first image (input image). In addition, the imaging apparatus of the present embodiment is configured to be able to acquire a light field by arranging a plurality of optical systems having positive refractive power or arranging a lens array on the image side of the imaging optical system. ing. A 1st image is an image acquired with such an imaging device (imaging element). Further, when generating the second image, an image obtained by performing processing such as demosaicing on the first image may be used.

  1 to 3 are examples of an imaging optical system that constitutes the imaging apparatus according to the present embodiment. The imaging optical system is configured to include an imaging optical system and an imaging element. When a lens array is provided, the imaging optical system includes the lens array. A person or an object does not necessarily exist on the subject surface 201 shown in FIGS. This is because it is possible to focus on a person or object existing behind or in front of the subject surface 201 by the reconstruction process even after shooting. The following description of each embodiment is performed using a one-dimensional system for the sake of simplicity, but the same argument holds for a two-dimensional system.

  First, with reference to FIG. 4, the configuration of the imaging apparatus according to the first embodiment of the present invention will be described. FIG. 4 is a block diagram of the imaging apparatus in the present embodiment. The image processing method of this embodiment is executed by the image processing unit 105 of the imaging apparatus.

  The pupil division unit 102 causes light rays from the same position on the subject surface to enter different pixels of the image sensor 103 according to the pupil region of the imaging optical system 101 through which the light rays pass. The imaging optical system 101, the pupil division unit 102, and the imaging element 103 constitute an imaging optical system. At the time of shooting, light rays from the subject space enter the image sensor 103 via the imaging optical system 101 and the pupil division unit 102. The image sensor 103 is a two-dimensional image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal-Oxide Semiconductor), and includes a plurality of pixels. The energy of light incident on the image sensor 103 via the imaging optical system 101 (main lens group) and the pupil division unit 102 becomes an electric signal (analog signal) and is converted into a digital signal by the A / D converter 104. The digital signal is subjected to predetermined processing by the image processing unit 105 and stored in a predetermined format in an image recording medium 110 such as a semiconductor memory. At this time, the imaging condition information of the imaging device obtained from the state detection unit 108 is also saved. The shooting condition information includes a shooting distance, a diaphragm, a focal length in a zoom lens, and the like. The state detection unit 108 may obtain the shooting condition information directly from the system controller 111, or may obtain information about the imaging optical system from the optical system control unit 107.

  When the image stored in the image recording medium 110 is displayed on the display unit 106, the image processing unit 105 performs a reconstruction process based on the shooting condition information. As a result, the display unit 106 displays an image reconstructed at a desired viewpoint, focus position, and depth of field. Further, for speeding up, desired image settings (viewpoint, focus, depth of field, etc.) are stored in the storage unit 109 in advance, and the reconstructed image is directly displayed on the display unit 106 without using the image recording medium 110. Also good. Furthermore, the image to be recorded on the image recording medium 110 may be a reconstructed image. In this manner, the image processing unit 105 generates a second image by performing reconstruction using the shooting condition information of the first image on the first image acquired by the imaging element 103.

  The user takes a picture by controlling the imaging apparatus using the system controller 111. In this embodiment, a subject that is a reference for focusing before shooting is referred to as a reference subject, and a subject that the user wants to focus at high resolution is referred to as a main subject. In an imaging device that acquires a light field, the focus position can be changed after shooting by refocusing, so the reference subject and the main subject do not have to match.

  The optical system control unit 107 drives and controls the imaging optical system 101. Further, the optical system control unit 107 includes a focus adjustment unit 107 a that drives the focus group 101 a of the imaging optical system 101 and adjusts the imaging position of the imaging optical system 101. The focus adjustment unit 107a drives the focus group 101a of the imaging optical system 101 when a reference subject is designated, and adjusts the imaging position of the imaging optical system 101 with respect to the reference subject to a predetermined position. 1 Adjust the focus. In this embodiment, the focus group 101a of the imaging optical system 101 may be configured by including one or a plurality of lenses, and may constitute the entire imaging optical system 101.

  The focus adjustment unit 107 a drives the focus group 101 a to perform second focus adjustment for moving the imaging position of the imaging optical system 101 by the focus shift amount ζ stored in the storage unit 109. That is, the focus adjustment unit 107a moves the imaging position of the imaging optical system 101 with respect to the main subject so as to increase the resolution of the main subject in accordance with the shift amount of the pixels synthesized by the reconstruction. After the second focus adjustment is performed, shooting is performed. The focus shift amount ζ is an amount determined according to the configuration of the imaging optical system, and a high-resolution image focused on the main subject can be acquired by the second focus adjustment. Details of the first focus adjustment and the second focus adjustment will be described later. In this embodiment, it is assumed that the reference subject matches the main subject.

  The series of control described above is performed by the system controller 111, and mechanical driving of the imaging optical system is performed by the optical system control unit 107 according to an instruction from the system controller 111.

  Next, the configuration of the imaging optical system in the present embodiment will be described with reference to FIG. FIG. 1 is a schematic configuration diagram of an imaging optical system. The imaging optical system includes an imaging optical system 101, a pupil division unit 102 (lens array 102 a), and an imaging element 103. In this embodiment, the imaging optical system 101 includes an aperture stop (not shown).

  In this embodiment, a lens array 102a made of a solid lens is used as the pupil division unit 102. However, the present invention is not limited to this, and other configurations such as a pinhole array may be used. In this embodiment, the lens array 102a is configured by using a plurality of lenses (small lenses), and the small lenses are configured by solid lenses. However, the present embodiment is not limited to this, and the lens array 102a may be configured using a liquid lens, a liquid crystal lens, a diffractive optical element, or the like. The small lens constituting the lens array 102a has convex surfaces on both sides. However, the present embodiment is not limited to this, and the surface on one side may be flat and the surface on the other side may have a convex shape. However, it is preferable that the image side surface of the small lens constituting the lens array 102a has a convex shape. Thereby, the astigmatism of the lens array 102a is reduced, and the image obtained on the image sensor 103 becomes sharp. Conversely, when the image side surface of the small lens is not convex, astigmatism increases, and the peripheral portion of the image formed by each small lens is blurred. If the blurred portion of the image is used for the reconstruction process, the image obtained by the reconstruction may not be sharply formed. Further, it is more preferable that the object-side surface of the small lens constituting the lens array 102a is a flat surface or a convex shape. Thereby, the curvature of the small lens is relaxed, the aberration is reduced, and a sharper image can be obtained.

  The lens array 102a is disposed on the image side focal point of the imaging optical system 101 or on the image side with respect to the image side focal point. The lens array 102a is configured such that the exit pupil of the imaging optical system 101 and the image sensor 103 have a substantially conjugate relationship. The substantially conjugate relationship means not only a strict conjugate relationship but also a relationship that is evaluated to be substantially a conjugate relationship. In this embodiment, the position of the object side main plane of the lens array 102a is defined as the image side focus surface 202. Further, it is assumed that a conjugate plane through the image side focusing surface 202 and the imaging optical system 101 is a surface in which the subject space is in focus. A surface in which the subject space is in focus is referred to as a subject surface 201. A light ray from the subject surface 201 passes through the imaging optical system 101 and the lens array 102a, and then enters a different pixel of the image sensor 103 according to the position and angle of the light ray on the subject surface 201, and the light field. Is acquired. Here, the lens array 102a has a role of preventing light rays that have passed through different positions on the subject surface 201 from entering the same pixel. As a result, the image sensor 103 acquires an image in which pixel groups obtained by photographing the same region on the subject surface 201 from a plurality of viewpoints are arranged. In the configuration of FIG. 1, three pixels (9 pixels in two dimensions) capture the same region on the subject surface 201. Therefore, in the imaging optical system of the present embodiment, the spatial resolution is reduced to １ ／ (1/9 in the two-dimensional system) as compared with the imaging optical system that acquires only the two-dimensional light intensity distribution. This property is the same even if the number of pixels that capture the same region on the subject surface 201 changes.

  Next, refocus processing in the present embodiment will be described. The refocus processing is described in detail in “Fourier Slice Photograph” (see Ren Ng, 2005 ACM Trans. Graph. 24, 735-744), and will be briefly described here. An example of a method for generating a refocus image will be described with reference to FIG. FIGS. 5A and 5B are diagrams showing in detail the lens array 102a and the image sensor 103 in the image pickup optical system shown in FIG. 5 (a) and 5 (b) is a straight line connecting the center of each pixel and the intersection of the optical axis with the principal plane of the small lens corresponding to this pixel. The virtual imaging plane 203 is an image side conjugate plane of the imaging optical system 101 with respect to the object side plane to be focused by refocusing. However, in this embodiment, when the image-side conjugate surface is positioned on the image side with respect to the object-side main plane of the lens array 102a, a surface obtained by moving the image-side conjugate surface to the image side by the main-plane interval of the lens array 102a is virtually connected. It becomes the image plane 203. A refocus image at a desired focus position can be generated by synthesizing overlapping pixels by translating the pixel value obtained by the image sensor 103 to the virtual imaging plane 203 along a one-dot chain line.

  For example, in order to generate an image focused on the subject surface 201 in FIG. 1, as shown in FIG. 5B, a conjugate surface, that is, a lens via the subject surface 201 and the imaging optical system 101 is used. The virtual imaging plane 203 may be set on the main plane (image side main plane) of the array 102a. At this time, the virtual imaging surface 203 coincides with the image side focusing surface 202 or the position moved from the image side focusing surface 202 to the image side by the main plane interval of the lens array 102a. In FIGS. 5A and 5B, the pixels that are moved in parallel at the time of generating the refocus image are represented by broken lines, and are drawn without being overlaid for easy understanding. As shown in FIGS. 5A and 5B, when an arbitrary refocus image is generated, if the pupil regions of the imaging optical system 101 through which the light beams incident on the pixels pass are the same, those pixels Shows that the amount of translation is the same. Therefore, the operation of the pixel when generating the refocus image is determined according to the pupil region of the imaging optical system 101 through which the light beam incident on the pixel has passed.

Next, the range in which refocusing is possible will be described. Since the aperture diameter of the imaging optical system 101 is finite, the angle component of the light field obtained by the image sensor 103, that is, parallax information is also finite. Therefore, the refocusable range is limited to a finite range. Here, the two-dimensional intensity distribution of light is called a light field spatial component. At this time, the refocus range is determined by the sampling pitch Δy of the spatial component (sampling pitch on the surface conjugate with the main subject via the imaging optical system 101) and the sampling pitch Δu of the angular component, and the coefficient α _± is It is given by the following equation (1).

The refocusing range α ₊ s _{2 to} α ₋ s ₂ on the image side expressed using Expression (1) and the range conjugate to the imaging optical system 101 are the refocusing range on the object side. Here, s ₂ is the distance between the image-side main plane of the imaging optical system 101 and the image-side focus surface 202.

  FIG. 6 is an explanatory diagram of the refocus range. In the configuration example shown in FIG. 6, since the one-dimensional period of the lens array 102 a is 3 pixels, the spatial component sampling pitch Δy is three times the pixel pitch of the image sensor 103. The sampling pitch Δu of the angle component is 1/3 of the exit pupil diameter because the exit pupil of the imaging optical system 101 is divided into three (9 in two dimensions). If the refocus range represented by Expression (1) is exceeded, the acquired light field lacks information, and a correct refocus image cannot be generated. Expression (1) can be approximated as the following expression (2) because the pixel pitch Δ of the image sensor 103 is sufficiently small with respect to the pupil distance P of the imaging optical system 101.

Here, the pupil distance P of the imaging optical system 101 is a distance between the exit pupil plane of the imaging optical system 101 and the image side conjugate plane of the imaging optical system 101 with respect to the subject surface 201. Further, N represents 1-dimensional division number of the pupil of the imaging optical system 101 by the lens array 102a, F is F number of the imaging optical system 101, the delta _LA is the pitch of the lens array 102a. Delta _LA is the pupil division unit 102 in the case of such pinhole array becomes the structural period. When the pixel group corresponding to a certain small lens is translated along the alternate long and short dash line in FIG. 5, when the maximum refocus amount of Expression (2) is exceeded, the interval between the pixels becomes larger than Δy and information is lost. The resulting area. In this case, a correct refocus image cannot be generated.

  Next, pixel shift super-resolution at the time of image reconstruction will be described. Referring to FIG. 5A, it can be seen that in the virtual imaging plane 203, the parallelly moved pixels are shifted from each other and overlapped (overlapped). By combining these, the apparent pixel size (apparent pixel pitch) can be reduced. This is called super-resolution by shifting pixels. On the other hand, as shown in FIG. 5B, when the pixels moved in parallel coincide with each other without being shifted, the effect of super-resolution by shifting the pixels cannot be obtained, and the resolution cannot be increased. Since the shift of the overlapping pixels changes according to the position of the virtual imaging plane 203 that translates the pixels, the effect of pixel shifting super-resolution also changes depending on the virtual imaging plane 203.

  Here, it is defined that the largest pixel pitch determines the resolution of the refocused image among the apparent pixel pitches reduced by the pixel-shifted super-resolution. Also, the apparent pixel pitch that determines the resolution is defined as the maximum value of the apparent pixel pitch. At this time, it is assumed that the number of pixels in the one-dimensional direction capturing the same region of the subject surface 201 is n pixels. Here, n corresponds to the number of one-dimensional pupil divisions of the imaging optical system 101. If the pixels are shifted by 1 / n pixels, the maximum apparent pixel pitch is minimum, and the refocused image has the highest resolution.

  The state shown in FIG. 5A is a state where n = 3 and each pixel is shifted by 1/3 pixel on the virtual imaging plane 203, so that the refocus image has the highest resolution. . From FIG. 5, it can be seen that there are four focus positions of the refocus image that can achieve the same effect within the refocus range. On the other hand, when the parallelly moved pixels coincide and overlap as shown in FIG. 5B, pixel-shifted super-resolution cannot be performed. FIG. 7 is a diagram schematically showing this relationship, where the horizontal axis represents the position of the virtual imaging plane 203 and the vertical axis represents the spatial resolution of the refocused image including pixel-resolved super-resolution.

  Next, the focus adjustment unit 107a in the present embodiment will be described. The focus adjustment unit 107a performs the first focus adjustment and the second focus adjustment by driving the focus group 101a of the imaging optical system 101. In the first focus adjustment, the imaging position of the imaging optical system 101 with respect to the reference subject is positioned on the image side focusing surface 202. In this adjustment, for example, the following method is used. In the configuration of this embodiment, a plurality of light beams that have passed through a certain position on the subject surface 201 are incident on the same small lens as shown in FIG. By comparing the signals of the pixel groups corresponding to this small lens, it can be determined whether or not the subject is in focus. When the correlation of the signal of the pixel group that captures the edge portion of the reference subject is high, the main subject exists on the subject surface 201 and is in focus. On the contrary, when the reference subject does not exist on the subject surface 201, the correlation of the signals of the pixel group becomes low.

  Alternatively, the first focus adjustment may be performed from the contrast of the image. The image for calculating the contrast may be an image obtained by extracting only pixels on which light rays that have passed through the same pupil region of the imaging optical system 101 are incident, or the first image obtained by the image sensor 103 is subjected to image processing. It may be a thing. As an example, an image obtained by adding pixel groups corresponding to each small lens to one pixel, a reconstructed image obtained by combining overlapping pixels as shown in FIG. In addition, the image used for calculating the contrast may use the entire angle of view acquired by the image sensor 103, or only the vicinity of the reference subject.

  As described above, as described with reference to FIG. 5B, the image focused on the image-side focusing surface 202 cannot obtain the effect of super-resolution by shifting pixels. However, in this embodiment, the reference subject and the main subject coincide. Therefore, in order to increase the resolution (increase the resolution) of the subject specified by the user (main subject), the imaging position of the imaging optical system 101 is shifted before shooting, and the effect of super-resolution is obtained by shifting the pixels. You can make it. The focus adjustment performed at this time is the second focus adjustment. Here, the shift amount of the focus position on the image side at the time of the second focus adjustment is defined as a focus shift amount ζ. Further, the distance from the image-side focusing surface 202 to the virtual imaging plane 203 of the imaging optical system 101 corresponding to the main subject after shifting the focusing position by the focusing shift amount ζ is η.

FIG. 7 shows an example of η. In this embodiment, since the reference subject and the main subject coincide, ζ = η. However, if the reference subject and the main subject are different, ζ ≠ η unless the distance from the imaging device to both is equal. From FIG. 7, when the virtual imaging surface 203 coincides with the image-side focusing surface 202 (x = s ₂ ), the resolution of the reconstructed image becomes higher when x = s ₂ + ζ = s ₂ + η. I understand that. For this reason, it is possible to generate a high-resolution image in which the main subject is in focus by shifting the imaging position so that the image of the main subject is formed at the position of x = s ₂ + ζ = s ₂ + η. it can. In this embodiment, the focus shift amount ζ is an amount for improving the spatial resolution in the reconstructed image including the pixel shift super-resolution. Therefore, the second focus adjustment has a different physical meaning from the conventional focus correction in which the focus position is adjusted so that MTF (Modulation Transfer Function) becomes higher from the paraxial image plane.

  Up to this point, the configuration, refocus, and focus adjustment of the imaging optical system in the present embodiment have been described. Next, with reference to FIG. 8, the imaging procedure (imaging device control method) using the imaging device in the present embodiment will be described. FIG. 8 is a flowchart of the photographing procedure in the present embodiment. Each step of the flowchart of FIG. 8 is executed by the focus adjustment unit 107a or the like based on a command from the system controller 111.

  First, in step S001, the user designates a reference subject to be focused. Subsequently, in step S002, the focus adjustment unit 107a performs the first focus adjustment according to the reference subject specified in step S001. Specifically, the focus adjustment unit 107 a drives the focus group 101 a of the imaging optical system 101 so that the imaging position of the imaging optical system 101 with respect to the reference subject coincides with the image side focusing surface 202. As a method for determining whether or not the imaging position coincides with the image-side focus surface 202, for example, the method described above is used, but is not limited thereto. Here, the focus position may be corrected so that the MTF of the image by the imaging optical system 101 of the reference subject becomes high on the image-side focus surface 202.

  In step S003, the focus adjustment unit 107a performs second focus adjustment. In other words, the imaging position determined in step S002 (the imaging position of the imaging optical system 101) is shifted by the amount ζ so as to increase the resolution of the main subject in accordance with the shift amount of the pixel synthesized by the reconstruction. Just shift. This is because when the reconstructed image is focused on the main subject, the value obtained by dividing the maximum value of the apparent pixel pitch by the pixel pitch is compared with the state before the shift by the focus shift amount ζ. This means approaching the inverse of the number of overlapping pixels in the configuration. That is, the focus adjustment unit 107a sets the second value so that the value obtained by dividing the maximum value of the apparent pixel pitch formed by the reconstruction by the pixel pitch of the image sensor 103 approaches the reciprocal of the number of overlapping pixels in the reconstruction. Adjust the focus.

  In order for an image focused on the main subject to be generated by refocusing, η needs to be within the refocusing range represented by Equation (2), that is, within the range satisfying −NFΔy ≦ η ≦ NFΔy. There is. For this reason, the focus adjustment unit 107a performs the second focus adjustment so that the distance between the image-side focus surface 202 and the virtual imaging plane is NFΔy or less.

Further, the pixel shifting effect of the super-resolution becomes highest, from the FIG. 5, s ₂ ± FΔ, and a 4-point of α _± s _{2 ±} FΔ. If the diameter of the allowable circle of confusion is ε, the depth of focus of the imaging optical system 101 is approximately expressed by ± Fε. ε is a value determined by the performance required for the imaging apparatus. For example, ε is determined such that the difference between the spread of the point image within the depth of focus and the spread of the point image at the focus position cannot be detected by the user. Therefore, η may be within the range of ± Fε. Here, since ε ≧ Δ is satisfied, s ₂ ± FΔ, which is the focus position where the effect of super-resolution by shifting pixels is the highest, always falls within s ₂ ± Fε. When | η | exceeds Fε, the spread of the point image at the focus position in the refocus image focused on the main subject exceeds ε. For this reason, even if the resolution is increased by super-resolution by shifting pixels, the focus position of the image appears to be blurred, and the sharpness is reduced.

  Moreover, it is desirable that η satisfies the following conditional expression (3).

However, −σ ₂ ≦ η ≦ σ ₂ . Here, σ ₂ is the distance between the image-side main plane of the lens array 102 a and the image sensor 103. Conditional expression (3) expresses the magnitude of the pixel shift super-resolution effect. σ ₂ / (FΔ) represents the number of pixels in the one-dimensional direction capturing the same area of the subject surface 201, and | η / σ ₂ | represents the ratio of deviation of the pixels to be combined. When | η | = FΔ, the value of conditional expression (3) is 1, indicating that the effect of super-resolution by shifting pixels is the highest. As the value of conditional expression (3) goes away from 1, the effect of pixel-shifting super-resolution becomes weaker. If the upper and lower limits of conditional expression (3) are exceeded, sufficient pixel shifting and super-resolution effects cannot be obtained.

  More desirably, satisfying the following conditional expression (3a) makes it possible to obtain a higher-resolution image focused on the main subject.

By satisfying conditional expression (3a), especially when n is small, the reconstructed image has a high resolution. For example, consider a case where the F value of the imaging optical system 101 is 2.800 and the pixel pitch Δ of the image sensor 103 is 0.006 mm. At this time, if ζ = η = 0.025 (mm), the value of conditional expression (3) becomes 1.488, and a high-resolution image in focus on the main subject can be generated. In this embodiment, step S002 and step S003 may be performed simultaneously.

  Subsequently, in step S004, the imaging apparatus performs shooting and acquires a first image. The first image is stored in the image recording medium 110 after predetermined image processing is performed by the image processing unit 105. Alternatively, the image processing unit 105 may generate a second image focused on the main subject from the acquired first image and store the second image in the image recording medium 110. The image processing unit 105 may perform a reconstruction process for changing the viewpoint and the depth of field simultaneously with the refocusing as necessary. Furthermore, at the time of image reconstruction processing, image resolution such as MAP (Maximum a posteriori) estimation may be used in combination to further increase the resolution.

  With the above configuration, according to the present embodiment, it is possible to provide an imaging apparatus and an imaging apparatus control method capable of acquiring a main subject with high resolution.

  Next, a second embodiment of the present invention will be described. Since the basic configuration of the image pickup apparatus in the present embodiment is the same as that of the image pickup apparatus of Embodiment 1 described with reference to FIG. 4, description thereof is omitted.

  With reference to FIG. 2, the structure of the imaging optical system in a present Example is demonstrated. FIG. 2 is a schematic configuration diagram of the imaging optical system. The lens array 102a is disposed closer to the object side than the image-side conjugate plane with respect to the object plane 201 of the imaging optical system 101, and is arranged such that the image-side conjugate plane and the image sensor 103 are in a conjugate relationship via the lens array 102a. Has been. In this embodiment, the image side conjugate plane corresponds to the image side focus plane 202. Light rays from the subject surface 201 pass through the imaging optical system 101 and the lens array 102a, and then enter different pixels of the image sensor 103 according to the position and angle of the light rays on the subject surface 201, and the light field is acquired. Is done. As a result, the image sensor 103 acquires an image in which a plurality of small images having different shooting viewpoints and shooting ranges are arranged.

The imaging optical system shown in FIG. 2 has a lower spatial resolution than the imaging optical system that acquires only the two-dimensional intensity distribution of light. This is because the image formed by the imaging optical system 101 is further reduced and formed on the image sensor 103 when the lens array 102a sees it as a virtual object. The reduction ratio is σ ₂ / σ ₁ times. Here, σ ₁ is the distance between the image-side focal plane 202 and the object-side main plane of the lens array 102 a, and σ ₂ is the distance between the image-side main plane of the lens array 102 and the image sensor 103. Therefore, in the imaging optical system shown in FIG. 2, the two-dimensional spatial resolution is (σ ₂ / σ ₁ ) ² times that of the imaging optical system that acquires only the two-dimensional intensity distribution of light.

  Next, with reference to FIGS. 9A and 9B, a method of generating a refocus image in the present embodiment will be described. FIGS. 9A and 9B are detailed views of the lens array 102a and the image sensor 103 in the configuration of the imaging optical system shown in FIG. In the present embodiment, the lens array 102a is constituted by small lenses whose object-side surface is flat and whose image-side surface is convex. However, like the first embodiment, the shape of the lens array 102a is not limited to this.

  Dotted lines in FIGS. 9A and 9B indicate the angle of view of each small lens. A pixel value obtained by the image sensor 103 is projected onto the virtual imaging plane 203 through a small lens corresponding to the pixel and synthesized, thereby generating a refocus image focused on the virtual imaging plane 203. be able to. For example, in order to generate an image in focus on the subject plane 201 in FIG. 2, the virtual imaging plane 203 may be set on the image side focusing plane 202. In FIGS. 9A and 9B, the pixels projected at the time of generating the refocus image are represented by broken lines and are drawn without being overlaid for easy understanding. The refocus image may be generated by a method of projecting pixels as described above, or a method of synthesizing by moving each pixel in parallel so that the pixels overlap. At this time, when the positions of the lens arrays 102a through which the light beams incident on the pixels have passed are equal, the parallel movement amounts of the pixels are the same. As described above, the operation of the pixel at the time of generating the refocus image in the present embodiment is determined according to the position of the pupil division unit 102 through which the light beam incident on the pixel of the corresponding image sensor 103 has passed.

Next, the refocusable range will be described. The refocus range of the imaging optical system in the present embodiment is also expressed by Expression (1) as in the first embodiment. The relationship is as shown in FIG. FIG. 10 is an explanatory diagram of the refocus range in the present embodiment. In the imaging optical system of the present embodiment, Δy = Δ (σ ₁ / σ ₂ ), Δu = P / (NF), and Δ << P. Therefore, the equation (1) is expressed by the following equation (4). To be rewritten.

As in the first embodiment, when the range of the expression (4) is exceeded, a correct refocus image cannot be generated.

  Next, the improvement of the spatial resolution by pixel shift super-resolution will be described. As shown in FIG. 9A, the pixels projected onto the virtual imaging plane 203 are overlapped. As shown in FIG. 9A, if the projected pixels are shifted from each other, the apparent pixel pitch can be reduced by combining them. On the other hand, as shown in FIG. 9B, when the projected pixel shift is an integer multiple of the pixel, the pixel shift super-resolution effect cannot be obtained. The highest resolution can be achieved by pixel-shifted super-resolution when the pixel shift ratio corresponds to the number of overlapping pixels. Specifically, in the vicinity of y = 0 in FIG. 9A, the number of overlapping pixels is 3, so that the resolution can be maximized when the pixel shift ratio is 1/3 or 2/3. . The relationship between the pixel shift ratio and the number of overlapping pixels will be described in detail later. As described above, also in the present embodiment, the effect of super-resolution by shifting pixels is changed by the virtual imaging plane 203. Therefore, within the refocus range, the resolution of the refocus image changes as shown in FIG.

  Next, a photographing method using the image pickup apparatus of the present embodiment will be described. The imaging method of the present embodiment is represented by the flowchart of FIG. 8, and the description of the same parts as those of the first embodiment is omitted.

  In step S001, the user designates a reference subject, and in step S002, the image of the reference subject formed by the imaging optical system 101 is matched with the image-side focus surface 202 (first focus adjustment). Thereafter, in step S003, second focus adjustment is performed to increase the resolution of the reconstructed image focused on the main subject.

Here, in the present embodiment, conditions that are preferably satisfied by η will be described. Ratio of pixel shift in the refocus image generation time of the virtual imaging plane 203 is represented by the ratio Δ _LA σ _{2 /} divided by the pitch of the pixels projected the delta _LA on the virtual imaging plane 203 (.DELTA..tau) . Here, τ is the distance between the object-side main plane of the imaging optical system 101 and the virtual imaging plane 203. Further, as shown in FIG. 9B, since the shift of the integer multiple of the pixels is meaningless, the integer portion may be considered to be dropped. Therefore, the pixel shift ratio δ is expressed as the following equation (5).

Here, z = mod (x, y) represents that z is equal to the remainder when x is divided by y. Here, a virtual imaging plane 203 having a high pixel resolution and high resolution is obtained. Therefore, first, the number of overlapping pixels on the virtual imaging plane 203 is estimated.

FIG. 11 is an explanatory diagram of the number of overlapping pixels, and is a graph in which the small lens number j shown in FIG. 9A is plotted on the horizontal axis and the coordinate y on the virtual imaging plane 203 is plotted on the vertical axis. Here, j = 0 may be an arbitrary small lens of the lens array 102a. A straight line parallel to the y-axis in FIG. 11 represents coordinates when a set of pixels corresponding to the j-th small lens is projected onto the virtual imaging plane 203. The one connecting the upper limits of these straight lines is the one-dot chain line A, and the one connecting the lower limits is the one-dot chain line B. The one-dot chain line A is given by y = Δ _LA {j + | τ / (2σ ₂ ) |}, and the one-dot chain line B is given by y = Δ _LA {j− | τ / (2σ ₂ ) |}. The number of overlapping pixels corresponds to the interval between the dash-dot line A and the dash-dot line B in the j direction. The number of overlapping pixels among the pixels corresponding to j = 0 is the smallest in the vicinity of y = 0, and the number can be estimated to be about | τ / σ ₂ |.

  Subsequently, a spatial resolution including pixel-shifted super-resolution is obtained. As described above, the highest resolution is obtained when the number of overlapping pixels corresponds to the pixel shift ratio δ represented by Expression (5). For example, when the pixel overlap number is 8 and the pixel deviation ratio δ is 0.45, the pixel deviation of each of the eight pixels is 0, 0.45, 0.90, 0.35, 0.80. , 0.25, 0.70, and 0.15. In this case, the maximum apparent pixel pitch that determines the resolution is 0.70−0.45 = 0.25. Next, consider a case where the number of overlapping pixels is the same and the pixel shift ratio δ is 3/8. At this time, the pixel shift of each of the eight pixels is 0, 3/8, 6/8, 1/8, 4/8, 7/8, 2/8, 5/8. In this case, the maximum value of the apparent pixel pitch is 1/8, which matches the reciprocal of the number of overlapping pixels. Therefore, the highest pixel shift super-resolution effect is obtained. This is the same even when the pixel deviation ratio δ is 1/8, 5/8, or 7/8.

However, when the pixel shift ratio δ is 2/8, 4/8, or 6/8, the pixel shift super-resolution effect decreases. For example, consider a case where the pixel shift ratio δ is 2/8. At this time, the pixel shift of each of the eight overlapping pixels is 0, 2/8, 4/8, 6/8, 0, 2/8, 4/8, and 6/8. The maximum value of the pixel pitch is 2/8 = 1/4. Therefore, the pixel shift super-resolution effect is halved when the pixel shift ratio δ is 1/8, 3/8, 5/8, or 7/8. From this, it can be understood that the maximum pixel shift super-resolution effect can be obtained when the pixel shift ratio δ is equal to m ₀ / M ₀ . Here, _{M 0} is a number from pixels overlapping, _{m 0} is smaller than _{M 0,} and the greatest common divisor of _{M 0} is an integer of 1. As described above, M ₀ can be estimated to be about τ / σ _{2. As} the pixel shift ratio δ is closer to m ₀ / M ₀ , the effect of pixel shift and super-resolution becomes higher.

  As described above, it is desirable that η be within a range that satisfies the following conditional expression (6).

Here, M is an integer that satisfies the following conditional expression (7).

M is an integer that is smaller than M and whose greatest common divisor with M is 1. Conditional expression (6) and conditional expression (7) represent the magnitude of the pixel-shifting super-resolution effect, and satisfying conditional expression (6) and conditional expression (7) allows high resolution to be focused on the main subject. An image reconstructed image is obtained. As the values of Conditional Expression (6) and Conditional Expression (7) are close to 1, the effect of pixel shift super-resolution becomes higher. On the other hand, when the upper limit or lower limit of conditional expression (6) and conditional expression (7) is exceeded, sufficient pixel shift super-resolution effect cannot be obtained, and the spatial resolution is not improved sufficiently.

  Desirably, a reconstructed image with higher resolution that is focused on the main subject can be obtained by setting the range to satisfy the following conditional expression (6a).

More desirably, the resolution can be further increased by setting the range to satisfy the following conditional expression (6b).

  Desirably, by setting a range that satisfies the following conditional expression (7a) and further a range that satisfies the conditional expression (7b), it is possible to obtain an even higher pixel shift super-resolution effect.

For example, the pitch Δ _LA of the lens array 102 a = 4.3559 (mm), the pixel pitch Δ of the image sensor 103 = 0.0043 (mm), σ ₁ = 37.7657 (mm), σ ₂ = 5.4325 (mm) ) At this time, if ζ = η = 0.0243 (mm), the value of conditional expression (6) is 1.04 and the value of conditional expression (7) is 0.7. Here, M = 5 and m = 3. Thereby, a high-resolution image focused on the main subject can be generated. In addition, since ζ or η depends on the pitch and arrangement of the lens array 102a, calibration may be performed at the time of manufacturing the imaging device, and ζ or η corrected for each imaging device may be stored in the storage unit 109 in advance. . The same applies to the case of the first embodiment.

  With the above configuration, according to the present embodiment, it is possible to provide an imaging apparatus and an imaging apparatus control method capable of acquiring a main subject with high resolution.

  Next, Embodiment 3 of the present invention will be described. In this embodiment, an image processing apparatus (image processing system) that performs the above-described image processing method will be described. FIG. 12 is a block diagram of the image processing system in the present embodiment. In this embodiment, it is assumed that the reference subject and the main subject are different from each other.

  As shown in FIG. 12, the image processing system includes an imaging device 301. The imaging device 301 has an imaging optical system shown in FIG. 3, for example. The imaging device 301 has a configuration shown in FIG. 13, for example. The image processing apparatus 302 is a computer device (information processing apparatus) that performs image reconstruction. The image processing device 302 generates a reconstructed image (output image) from the first image (input image) acquired by the imaging device 301. The reconstructed image is not limited to an image in which the main subject is focused, but may be another refocused image or an image in which the depth of field and the viewpoint are adjusted. The result is output to any one or more of the storage medium 303, the display device 304, and the output device 305. The storage medium 303 is, for example, a semiconductor memory, a hard disk, or a server on a network. The display device 304 is, for example, a liquid crystal display or a projector. The output device 305 is a printer, for example. A display device 304 is connected to the image processing apparatus 302, and a reconstructed image is input to the display device 304. The user can perform work while confirming the reconstructed image via the display device 304. Image processing software 306 is installed in the image processing apparatus 302. The image processing software 306 (image processing program) performs the above-described reconstruction processing (image processing method), and also performs development processing and other image processing as necessary.

  Next, the configuration of the imaging device 301 in the present embodiment will be described with reference to FIG. FIG. 13 is a block diagram of the imaging device 301. Since the basic configuration of the imaging apparatus 301 is the same as that of the imaging apparatus of the first embodiment, only the parts different from the first embodiment will be described, and the other description will be omitted.

  The calculation unit 405 includes an image processing unit 405a that performs reconstruction processing and the like, and a distance information acquisition unit 405b that acquires distance information of the subject space. The computing unit 405 calculates a focus shift amount ζ used in the second focus adjustment. The focus shift amount ζ calculated by the calculation unit 405 is used by the focus adjustment unit 407a of the optical system control unit 407.

  The imaging optical system constituting the imaging apparatus 301 of this embodiment is arranged as shown in FIG. The imaging optical system of the present embodiment shown in FIG. 3 has the same configuration as that of the second embodiment shown in FIG. 2 except that the lens array 102a is arranged on the image side with respect to the image side focusing surface 202. is there. The configuration of this embodiment (FIG. 3) is that the lens array 102a sees the image formed by the imaging optical system 101 as a real object and re-images it on the image sensor 103. Different from the configuration. However, the configuration of FIG. 2 and the configuration of FIG. 3 are both essential because the lens array 102a views the image formed by the imaging optical system 101 as an object and forms the image on the image sensor 103. Is the same. Therefore, the refocus image generation method of this embodiment is the same as that of the second embodiment.

  Next, with reference to FIG. 14, a photographing procedure (imaging device control method) by the imaging device 301 of the present embodiment will be described. FIG. 14 is a flowchart showing a photographing procedure in the present embodiment. Each step of the flowchart of FIG. 14 is executed by the focus adjustment unit 407a or the like based on a command from the system controller 111.

  First, in step S101, the user designates a reference subject to be focused. Subsequently, in step S102, the focus adjustment unit 407a performs the first focus adjustment according to the reference subject specified in step S101. In other words, the focus adjustment unit 407a drives the focus group 101a of the imaging optical system 101 so that the imaging position of the imaging optical system 101 with respect to the reference subject coincides with the image-side focusing surface 202.

  In step S103, the user designates a main subject that is a subject to be focused with high resolution. As described above, in this embodiment, the reference subject and the main subject are different from each other, and the main subject is designated after the first focus adjustment is performed on the reference subject. However, the present embodiment is not limited to this, and the designation of the main subject (step S103) may be performed before the designation of the reference subject (step S101).

  Subsequently, in step S104, the distance information acquisition unit 405b of the calculation unit 405 acquires distance information of the subject space. The distance information of the subject space can be acquired using, for example, the following method. That is, the image sensor 103 can obtain parallax information obtained by observing the subject space from different viewpoints. For this reason, a plurality of parallax images can be acquired by performing imaging once and performing reconstruction processing. By comparing the feature points of these parallax images, it is possible to acquire distance information of the subject space. Alternatively, the distance information may be acquired by performing a plurality of shootings while driving the focus group 101a of the imaging optical system. In addition to the imaging optical system, it may have means for acquiring distance information. Here, when the distances from the imaging device 301 to the reference subject and the main subject are the same, the following procedure may be the same as that after step S003 in the second embodiment.

  Next, in step S105, the calculation unit 405 calculates a focus shift amount ζ used when the second focus adjustment is performed by the focus adjustment unit 407a. At this time, the calculation unit 405 calculates the imaging position of the imaging optical system 101 with respect to the main subject from the distance information acquired by the distance information acquisition unit 405b. The behavior of the pixel shift ratio δ with respect to the position of the virtual imaging plane 203 is determined by the configuration of the imaging optical system as expressed by Expression (5). Therefore, the position of the virtual imaging plane 203 with high resolution is stored in the storage unit 109 in advance, and the focus shift amount ζ is set by taking the difference between the position and the imaging position of the imaging optical system 101 with respect to the main subject. Can be calculated. The position of the high-resolution virtual imaging plane 203 stored in advance in the storage unit 109 may be single or plural, and may be a value corrected by calibration at the time of manufacturing the imaging device 301.

  Subsequently, in step S106, the focus adjustment unit 407a performs the second focus adjustment so that the imaging position is changed by the focus shift amount ζ calculated by the calculation unit 405. The refocus range is finite as represented by Equation (4). For this reason, the distance η from the image-side focus surface 202 to the virtual imaging surface 203 corresponding to the main subject after the second focus adjustment needs to be within NFΔy. As in the first embodiment, the spread of the point image in the refocus image focused on the main subject is preferably within ε. Therefore, it is desirable that η be within the depth of focus of the imaging optical system 101. At this time, the focus adjustment unit 407a performs the second focus adjustment so that the distance between the image-side focus surface 202 and the virtual imaging surface is within the focal depth range of the imaging optical system 101. Similarly to the second embodiment, it is desirable that conditional expressions (6), (6a), (6b), (7), (7a), and (7b) are satisfied.

For example, the pitch Δ _LA of the lens array 102 a = 4.3559 (mm), the pixel pitch Δ of the image sensor 103 = 0.0043 (mm), σ ₁ = 37.7657 (mm), σ ₂ = 5.4325 (mm) ) Since the reference subject and the main subject do not match, the position of the image plane corresponding to the main subject is generally different from σ ₁ . Here, the position of the imaging plane is τ = 37.7100 (mm). At this time, if ζ = 0.0100 (mm), then η = −0.0457 (mm), the value of conditional expression (6) is 1.01, and the value of conditional expression (7) is 1.3. Become. Here, M = 9 and m = 8. Thereby, it is possible to generate an image with high resolution and focused on the main subject.

  Finally, in step S107, the imaging device 301 performs shooting and acquires a first image.

  With the above configuration, according to the present embodiment, it is possible to provide an imaging apparatus and an imaging apparatus control method capable of acquiring a main subject with high resolution.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

DESCRIPTION OF SYMBOLS 101 Imaging optical system 102 Pupil division part 103 Image pick-up element 105 Image processing part 107a Focus adjustment part

Claims (12)

An imaging apparatus capable of generating a plurality of second images having different focus positions by reconstructing a first image,
An imaging optical system;
An image sensor comprising a plurality of pixels and acquiring the first image;
A pupil division unit that causes light rays from the same position on the subject surface to enter different pixels of the image sensor according to the pupil region of the imaging optical system through which the light rays pass;
A focus adjustment unit that adjusts the imaging position of the imaging optical system;
An image processing unit that generates the second image by performing the reconstruction using the imaging condition information of the first image with respect to the first image acquired by the imaging element;
The focus adjustment unit is
A first focus adjustment for adjusting an imaging position of the imaging optical system with respect to a reference subject;
And performing a second focus adjustment for moving the imaging position of the imaging optical system with respect to the main subject so as to increase the resolution of the main subject in accordance with the shift amount of the pixel synthesized by the reconstruction. An imaging device.   The focus adjustment unit generates a maximum value of an apparent pixel pitch formed by the reconstruction when generating the second image in focus on a designated virtual imaging plane of the imaging optical system. The image pickup apparatus according to claim 1, wherein the second focus adjustment is performed so that a value divided by a pixel pitch of the image pickup element approaches an inverse number of pixels overlapping in the reconstruction.   The focus adjustment unit is configured such that the F value of the imaging optical system is F, the number of one-dimensional divisions of the pupil of the imaging optical system by the pupil division unit is N, and is conjugate with the main subject via the imaging optical system. 3. The second focus adjustment according to claim 2, wherein the second focus adjustment is performed so that a distance between the image-side focus surface and the virtual imaging surface is equal to or less than NFΔy when a sampling pitch on a smooth surface is Δy. The imaging device described.   The focus adjustment unit performs the second focus adjustment so that a distance between an image-side focus surface and the virtual imaging plane is within a range of a focal depth of the imaging optical system. Item 4. The imaging device according to Item 2 or 3.   The imaging apparatus according to claim 1, wherein the reference subject and the main subject are the same subject.   The imaging apparatus according to claim 1, wherein the reference subject and the main subject are different subjects.   The imaging apparatus according to claim 1, wherein the pupil division unit is a lens array.   The imaging apparatus according to claim 7, wherein the lens array is arranged so that a pupil of the imaging optical system and the imaging element are in a conjugate relationship.   The imaging apparatus according to claim 7, wherein the lens array is disposed so that an image formed by the imaging optical system and the imaging element are in a conjugate relationship. The lens array includes a plurality of small lenses,
The image pickup apparatus according to claim 7, wherein an image side surface of the small lens has a convex shape. The lens array includes a plurality of small lenses,
The imaging apparatus according to claim 10, wherein the object side surface of the small lens is a flat surface or a convex shape. A control method for an imaging apparatus capable of generating a plurality of second images having different focus positions by reconstructing a first image,
Performing a first focus adjustment for adjusting an imaging position with respect to a reference object of an imaging optical system included in the imaging apparatus;
Performing a second focus adjustment for moving the imaging position of the imaging optical system with respect to the main subject so as to increase the resolution of the main subject in accordance with the shift amount of the pixel synthesized by the reconstruction;
And a step of acquiring the first image by photographing with the imaging device.