JP2015033032A - Optical equipment and pixel deviated image acquisition method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To acquire a plurality of pixel deviated images each having a high-accuracy pixel deviation amount while using one imaging device.SOLUTION: Optical equipment 1 used for acquiring a plurality of pixel deviated images to be used for producing a superresolution image while using an imaging device 103 comprises: an imaging optical system 100; opening position variable means 101 capable of changing the position of an opening through which a light flux is passed within a plane that is orthogonal to an optical axis direction; positional relation variable means 100a and 105 capable of changing a positional relation in the optical axis direction between an image forming plane on which the light flux from a subject passed through the imaging optical system and the opening forms an image and the imaging device; and control means 109, 106 and 107 for setting the positional relation in such a manner that the image forming plane and the imaging device have a positional difference in the optical axis direction in accordance with a pixel deviation amount between pixel deviated images, and changing the position of the opening in every imaging for acquiring the pixel deviated images, respectively.

Description

  The present invention relates to an optical apparatus used for synthesizing a plurality of pixel-shifted images to obtain a super-resolution image.

  As a method for acquiring a plurality of pixel-shifted images used for super-resolution image synthesis, there are a method in which a plurality of image pickup elements are shifted from each other, and a method disclosed in Patent Document 1. In Patent Document 1, a plurality of subject images respectively formed by a plurality of light beams that have passed through openings at different positions by changing the position of the aperture through which the light beam from the subject passes or by providing a plurality of openings. A method for imaging a plurality of image sensors with a plurality of image sensors is disclosed.

  However, when a plurality of image sensors are used as in the method disclosed in Patent Document 1, it is difficult to reduce the size and cost of the optical device. For this reason, Patent Documents 2 and 3 disclose a method of acquiring a pixel-shifted image by reducing the number of imaging elements. In Patent Document 2, a reflecting mirror is disposed between an optical system and an image sensor, and a plurality of pixels are obtained by shifting the subject image formed on one image sensor by minutely displacing the reflector. A method for acquiring a shifted image has been proposed. Further, Patent Document 3 proposes a method of acquiring a plurality of pixel-shifted images by rotating a decentered lens to shift a pixel of a subject image formed on one image sensor.

JP 07-222038 A Japanese Patent Laid-Open No. 2002-077696 JP 2004-187047 A

  However, in the method using the reflecting mirror capable of minute displacement disclosed in Patent Document 2, it is difficult to obtain a desired pixel shift amount with high accuracy by the back of the mechanism for displacing the reflecting mirror. In addition, it is difficult to mount such a reflector in a compact digital camera in which the imaging optical system to the image sensor are integrated.

  In addition, the method disclosed in Patent Document 3 requires a radial space for rotating the decentered lens in the optical device, and thus there is a problem that the optical device is easily increased in size.

  The present invention provides an optical device and a pixel-shifted image acquisition method capable of obtaining a plurality of pixel-shifted images having a high-accuracy pixel shift amount using a single image sensor while avoiding an increase in size of the optical device. I will provide a.

  An optical apparatus according to one aspect of the present invention is used when acquiring a plurality of pixel-shifted images used for generating a super-resolution image using an imaging element. The optical apparatus includes an imaging optical system, aperture position changing means capable of changing the position of the aperture through which the light beam passes in a plane orthogonal to the optical axis direction, and the light flux from the imaging optical system and the subject passing through the aperture. The positional relationship variable means that can change the positional relationship between the imaging surface and the image sensor in the optical axis direction, and the image plane and the image sensor are pixel-shifted and light according to the pixel shift amount between the images The positional relationship is set so as to have a positional difference in the axial direction, and control means for changing the position of the opening for each imaging for acquiring each pixel-shifted image is provided.

  A pixel shifted image acquisition method according to another aspect of the present invention is used for acquiring a plurality of pixel shifted images used for generating a super-resolution image using an imaging element. The pixel-shifted image acquisition method can change the position of an aperture through which a light beam passes within a plane orthogonal to the optical axis direction, and an imaging optical system and an imaging surface on which a light beam from a subject that has passed through the aperture forms an image. The positional relationship in the optical axis direction with the image sensor can be changed, and the imaging plane and the image sensor are shifted in pixels so that the position in the optical axis direction according to the pixel shift amount between the images is changed. The relationship is set, and the position of the opening is changed for each imaging for obtaining each pixel-shifted image.

  In the present invention, the position difference between the imaging surface and the image sensor is set according to the pixel shift amount, and imaging for acquiring a plurality of pixel shifted images is performed while changing the position of the aperture through which the light beam passes. For this reason, according to the present invention, it is possible to obtain a plurality of pixel-shifted images having a high-precision pixel shift amount using one image sensor, and to avoid an increase in the size of the optical device.

1 is a block diagram illustrating a configuration of an imaging apparatus that is Embodiment 1 of the present invention. 6 is a flowchart illustrating the operation of the imaging apparatus according to the first embodiment. FIG. 3 is a diagram illustrating the shape of an opening in the imaging apparatus according to the first embodiment. 3 is a flowchart illustrating defocus control in the imaging apparatus according to the first exemplary embodiment. FIG. 6 is a diagram illustrating a relationship between a pixel shift amount obtained by the imaging apparatus according to the first embodiment and other parameters. 3 is a flowchart illustrating super-resolution processing in the imaging apparatus according to the first embodiment. FIG. 3 is a block diagram illustrating a configuration of an imaging apparatus that is Embodiment 2 of the present invention. FIG. 6 is a diagram illustrating the shape of an opening in an imaging apparatus according to Embodiment 2. 9 is a flowchart illustrating defocus control in the imaging apparatus according to the second embodiment. FIG. 6 is a diagram illustrating a relationship between a pixel shift amount obtained by the imaging apparatus of Embodiment 2 and other parameters.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 shows a configuration of an imaging apparatus (optical apparatus) such as a video camera or a digital still camera that is Embodiment 1 of the present invention. Reference numeral 100 denotes an imaging optical system, which forms a subject on the imaging element 103. The imaging optical system 100 includes a focus lens (focus element) 100a that can move in the optical axis direction. A focus control unit 105 controls the movement of the focus lens 100a. The position in the optical axis direction with respect to the imaging element 103 of the imaging surface on which the light beam that has passed through the imaging optical system 100 (and the aperture of the variable aperture stop 101 described later) is imaged by the movement of the focus lens 100a, that is, the imaging surface And the image sensor 103 in the same direction can be changed. The focus lens 100a and the focus control unit 105 constitute a positional relationship variable unit.

  A variable aperture stop 101 as an aperture position varying unit is disposed between the imaging optical system 100 and the imaging element 103 and at a position on the pupil plane of the imaging optical system 100. The variable aperture stop 101 is composed of a liquid crystal element or the like, and can form an aperture having an arbitrary shape at an arbitrary position on the pupil plane. The light flux from the subject passes through the imaging optical system 100 and then passes through the aperture of the variable aperture stop 101 and travels toward the imaging element 103. Reference numeral 102 denotes a shutter disposed between the variable aperture stop 101 and the image sensor 103, and controls the exposure of the image sensor 103 by opening and closing.

  The image sensor 103 is configured by a photoelectric conversion element such as a CCD sensor or a CMOS sensor that converts an object image into an electric signal and outputs the electric signal. Reference numeral 104 denotes an A / D converter that converts an analog image signal output from the image sensor 103 into a digital image signal.

  A defocus control unit 106 receives a command from an imaging control unit 109 (to be described later), and focuses to change the position of the imaging surface of the subject image with respect to the image sensor 103 according to the pixel shift amount between the image shift images. A command is issued to the control unit 105. Reference numeral 107 denotes an aperture control unit, which receives a command from the imaging control unit 109 and controls the position and shape of the aperture formed in the variable aperture stop 101. An exposure control unit 108 controls the aperture size of the variable aperture stop 101 and the opening / closing operation of the shutter 102 through the aperture control unit 107.

  The imaging control unit 109 manages the overall operation of the imaging apparatus 1. The imaging control unit 109 also determines exposure conditions including the aperture size of the variable aperture stop 101 and the opening / closing speed (shutter speed) of the shutter 102 from the luminance component of the digital imaging signal output from the A / D conversion unit 104. . Further, the imaging control unit 109 generates an evaluation value indicating the contrast of the image by extracting a high frequency component from the digital imaging signal from the A / D conversion unit 104, and the position of the focus lens 100a at which the evaluation value reaches a peak. The in-focus position is detected. When the focus lens 100 a is moved to this focus position, a focus state in which the position of the imaging surface of the subject image by the imaging optical system 100 matches the position of the imaging surface of the image sensor 103 is obtained. That is, autofocus is performed. The imaging control unit 109, the defocus control unit 106, and the aperture control unit 107 described above constitute a control unit.

  A memory 110 includes a flash ROM or the like that stores control data and processing programs used by the imaging control unit 109. Reference numeral 111 denotes a nonvolatile memory such as an EEPROM which stores information such as various adjustment values.

  A super-resolution processing unit 112 generates a pixel-shifted image using a digital image pickup signal from the A / D conversion unit 104, and combines a plurality of pixel-shifted images to generate a super-resolution image (also referred to as a super-resolution image). Super-resolution processing is generated. The super-resolution image is an image having a super-resolution that is higher than the resolution of each pixel-shifted image corresponding to the number of pixels of the image sensor 103. The super-resolution processing unit 112 corresponds to an image generation unit.

  Reference numeral 113 denotes an image processing unit that performs processing to make the generated super-resolution image a more visually preferable image. Specifically, auto gain control, auto white balance, pixel interpolation processing, color conversion processing, and the like are performed on the super-resolution image.

  Reference numeral 114 denotes a compression unit that compresses the super-resolution image using a compression method such as JPEG. Reference numeral 115 denotes a first frame memory which stores several frames of the image digitized by the A / D conversion unit 104. Reference numeral 116 denotes a second frame memory, which temporarily stores an image generated by the super-resolution processor 112 and processed by the image processor 113. Reference numeral 117 denotes a memory control unit that controls input / output of images to the first and second frame memories 110 and 111. Reference numeral 118 denotes an image output unit for outputting the generated super-resolution image to a display device or recording device (not shown).

  Next, imaging operations performed by the imaging apparatus 1 (imaging control unit 109, defocus control unit 106, aperture control unit 107, and super-resolution processing unit 112) according to the present embodiment will be described with reference to the flowchart of FIG. The imaging operation includes an operation of capturing a subject and acquiring a plurality of pixel-shifted images (a pixel-shifted image acquisition method), and generating a super-resolution image by combining the plurality of pixel-shifted images to display a display device or recording Output to the device. This imaging operation is executed by the imaging control unit 109, the defocus control unit 106, the aperture control unit 107, and the super-resolution processing unit 112 as a computer according to a pixel shift image acquisition program and a super-resolution processing program that are computer programs. .

  In step S101, the imaging control unit 109 calculates (determines) imaging conditions including an exposure condition and a focus condition for acquiring a pixel-shifted image when an imaging start button (not shown) is pressed by the user. At this time, the imaging control unit 109 determines the aperture shape of the variable aperture stop 101 according to the number of necessary pixel shifted images stored in the nonvolatile memory 111. The number of pixel shifted images may be set by the user.

  In the following description, four pixel-shifted images are acquired (pixel-shifted imaging is performed) in order to generate a super-resolution image having a resolution twice as long as the number of pixels (resolution) of the image sensor 103. Will be described. However, the four pixel-shifted images are merely examples, and another number of pixel-shifted images may be acquired.

  In FIG. 3A, the variable aperture stop 101 formed of a liquid crystal element is divided into four quarter-circle regions (a shape obtained by dividing the pupil plane into four) by two lines passing through the center and orthogonal to each other. An example of division is shown. In FIG. 3A, one of these four regions is set in a transmissive state (light passing state) to form a 1/4 circular opening, and the other three regions are in a light shielding state. When acquiring four pixel-shifted images, the light flux that has passed through four different apertures is changed by changing the region to be transmitted for each imaging to acquire one pixel-shifted image. The four subject images formed by the above are sequentially picked up.

  Note that the shape of the opening does not necessarily have to be a quarter circle as shown in FIG. For example, when a square array image sensor is used, any shape such as a circle as shown in FIG. 3B can be used as long as the centers of the four openings are arranged to be the apexes of a square. It can be. Further, the shape and size of the opening are determined according to the exposure conditions, and may be a shape and size in which the four openings do not overlap each other as shown in FIGS. The shape and size may overlap. That is, the positions of the four apertures only need to be different from each other in the center (or center of gravity) in a plane orthogonal to the optical axis direction (in the pupil plane).

  In step S <b> 102, the imaging control unit 109 causes the defocus control unit 106 to perform defocus control for performing pixel shift imaging of a necessary pixel shift amount stored in the nonvolatile memory 111. The operation of this defocus control will be described using the flowchart of FIG.

  In step S201, the defocus control unit 106 determines the pixel size μ of the image sensor 103, the focal length f of the imaging optical system 100, and the subject distance (the distance from the front principal point of the imaging optical system 100 to the subject on the focus plane). ) Get a. The subject distance a may be calculated from the in-focus position of the focus lens 100a or may be measured using a distance measuring sensor. In addition, the defocus control unit 106 acquires an aperture distance R that is a distance from the optical axis of the imaging optical system 100 to the center of the aperture of the variable aperture stop 101.

  Then, the defocus control unit 106 corrects the position of the focus surface of the imaging optical system 100 necessary for shifting the pixels according to the pixel size μ, the focal length f, the subject distance a, and the aperture distance R ( Hereinafter, d is calculated. By correcting the position of the focus surface on the subject side on which the imaging optical system 100 is in focus and bringing the imaging optical system 100 into a defocused state with respect to the subject, the imaging surface of the subject image is changed to the imaging element 103 (imaging surface). ). The opening distance R may be calculated according to the opening shape determined in step S101 and stored in advance in the nonvolatile memory 111.

  The reason for correcting the position of the focus surface of the imaging optical system 100 is as follows. As indicated by a solid line in FIG. 5A, the light beam from a certain point on the focus plane is changed to a point on the same image plane (that is, the imaging plane) even if the aperture position of the variable aperture stop 101 is changed. Since the image is formed, the pixels cannot be shifted. On the other hand, the principal ray at the variable aperture of the light beam from a certain point on the surface (defocus surface) defocused by the imaging optical system 100 is the aperture of the variable aperture stop 101 as indicated by the dotted line in FIG. By changing the position, it reaches a different position on the imaging surface, so that the pixels can be shifted. In the case where the pixel position is shifted by changing the position of the aperture in the variable aperture stop 101, the pixel shift amount S on the imaging surface is the amount before and after the change of the principal ray passing through the opening when the position of the focus surface is changed. This corresponds to the amount of deviation of the image plane.

  At this time, the image of the focus plane is blurred on the imaging plane due to the defocusing of the imaging optical system 100, but by setting the pixel shift amount to the sub-pixel level, the blur amount becomes minute and the influence on the image quality is small. In the case of a Bayer array such as RGB, the pixel shift amount may be set to one pixel, but even in this case, the blur amount is small and the influence on the image is small.

  As described above, in order to perform imaging by shifting the pixel with respect to the subject on the focus surface, the imaging optical system 100 is brought into focus with respect to the subject, and then the subject is subjected to the required amount of pixel shift. The position of the focus surface is corrected so that the imaging optical system 100 is defocused.

  The pixel shift amount S is expressed by the following equation (1). In Expression (1), a is the subject distance to the focus plane, and b is the rear principal point of the imaging optical system 100 focused on the focus plane (in FIG. 5B, the front principal point and the rear principal point). The distance from the image plane to the image plane is the image distance. Further, a ′ is a subject distance to the defocus plane, and b ′ is an image distance when it is assumed that the defocus plane is in focus. At this time

It is.

  When acquiring four single-color pixel-shifted images, the pixel shift amount of the other three images with respect to one image is half pixel only in the horizontal direction, half pixel only in the vertical direction, and horizontal direction. It is desirable to use half pixels in the vertical direction. In this case, the pixel shift amount is half pixel × √2 in the diagonal direction. FIG. 5A shows the locus of the principal ray at the aperture located in the diagonal direction in the variable aperture stop 101. The pixel shift amount twice as large as the pixel shift amount S shown in FIG. 5A corresponds to half pixel × √2, which is the pixel shift amount in the diagonal direction. Therefore, if the pixel size is μ, the half pixel is μ / 2, and the pixel shift amount S is

It becomes.

  FIG. 5B shows the principal ray from the corrected focus plane (the focus plane after defocusing the subject on the focus plane before correction), that is, on the imaging plane when performing pixel-shifted imaging. The principal ray to be imaged is indicated by a solid line. By correcting the position of the focus surface in this way, the principal ray from the subject on the focus surface before correction reaches a position shifted from the arrival position before correction on the imaging surface. The amount of shift of the arrival position corresponds to the pixel shift amount S.

  As can be seen from FIG. 5B, the focus correction amount d, which is the correction amount of the focus surface on the subject side, is

It is. Therefore, from the expressions (1) to (3), the focus correction amount d on the subject side when performing pixel-shifted imaging for obtaining four pixel-shifted images each having a pixel shift amount of half a pixel is given by

It can be expressed as. From equation (4), the focus correction amount d can be uniquely calculated by determining the subject distance a.

  In step S202, the defocus control unit 106 changes the position of the focus lens 100a in the optical axis direction via the focus control unit 105 according to the focus correction amount d calculated in step S201.

  In this embodiment, the case where the focus correction amount on the subject side is obtained and the defocus control of the imaging optical system 100 is performed has been described, but the focus correction amount on the image side is obtained and the defocus control is performed. Good. That is, the imaging plane of the light beam that has passed through the aperture of the variable aperture stop 101 and the imaging element 103 (imaging plane) have a positional relationship that has a positional difference (defocus amount) in the optical axis direction according to the pixel shift amount. The position of the focus lens 100a is controlled.

  Returning to FIG. 2, in step S <b> 103, the imaging control unit 109 determines the position and shape of the aperture of the variable aperture stop 101 via the aperture control unit 107 according to the imaging condition calculated in step S <b> 101 and the number of times of pixel shifting imaging. To control.

  In step S104, the imaging control unit 109 performs pixel shift imaging at the shutter speed determined in step S101. An analog imaging signal obtained by photoelectrically converting the subject image is output from the imaging element 103, and the analog imaging signal is converted into a digital imaging signal by the A / D conversion unit 104, and the frame is passed through the memory control unit 117. Stored in the memory 115.

  Next, in step S105, the imaging control unit 109 determines whether or not the number of pixel-shifted imaging is N (the number of acquired pixel-shifted images is N), and is four (four) in the present embodiment. Determine. If the number of pixel-shifted imaging has not reached N (= 4) times, the process returns to step S103, the aperture position at the variable aperture stop 101 is changed, and the process returns to step S104 again to perform pixel-shifted imaging. When the number of pixel-shifted imaging has reached N (= 4) times, the process proceeds to the next step S106.

  In step S106, the imaging control unit 109 performs super-resolution processing on the super-resolution processing unit 112 using the acquired N (= 4) pixel-shifted images to generate a super-resolution image. This super-resolution processing will be described with reference to the flowchart of FIG. Here, a case where reconfigurable super-resolution processing is performed as super-resolution processing will be described.

  First, in step S301, the super-resolution processing unit 112 performs N pixel-shifted images (a reference image as a target image for super-resolution processing and the other three images) acquired up to step S105. Receive input.

  Then, in the same step, the super-resolution processing unit 112 performs an enlargement process according to the set enlargement magnification on the input reference image to obtain an initial estimated super-resolution image (hereinafter, initial estimated super-resolution). Image). As the enlargement processing, any method such as simple linear interpolation or spline interpolation may be used.

  Next, in step S302, the super-resolution processing unit 112 calculates a degradation process for the initial estimated super-resolution image generated in step S301, and generates N low-resolution images. The deterioration process referred to here is an image deterioration process by an imaging system including the imaging optical system 100 and the imaging element 103. For the calculation of the deterioration process, an observation model obtained by modeling the deterioration process is used. The observation model is given by the following equation (5) using the response function PSF (point spread function) of the imaging system.

Here, X represents a super-resolution image, and Y ′ _k represents the k-th low-resolution image after the deterioration process. * Indicates convolution operation (convolution), and n indicates additional noise. PSF _k represents a response function of the imaging system for the k-th pixel shifted image.

  The N pixel-shifted images acquired up to step S105 are obtained by capturing a subject image formed by a light beam that has passed through a different position in the imaging optical system 100 as the aperture position of the variable aperture stop 101 is changed. It is an image. For this reason, images captured using the same imaging optical system 100 are deteriorated by different PSFs. The calculation of the deterioration process is performed using the first term on the right side of Equation (5). Note that the observation model is not limited to that given by the above equation (5), and may be determined as appropriate according to the characteristics of the imaging system and the algorithm of super-resolution processing. Further, when calculating the degradation process, Expression (5) is used by using the response function of the imaging system corresponding to the divided area including the pixel to be calculated among the plurality of divided areas obtained by dividing the image into a plurality of parts. Is calculated.

Next, in step S303, the super-resolution processing unit 112 calculates an evaluation value (hereinafter referred to as a super-resolution evaluation value) using N low-resolution images generated by the calculation of the deterioration process.
The super-resolution evaluation value J is calculated by, for example, the following formula (6).

Here, Y _k represents the k-th pixel-shifted image (captured image), and Y ′ _k represents the k-th low-resolution image. α represents a constraint parameter, and c represents a matrix representing prior information for the estimated super-resolution image. Also,

Indicates the L2 norm.

The first term on the right side of Equation (6) is a term for evaluating the estimated super-resolution image by comparing the captured image with a low-resolution image generated by degrading the estimated super-resolution image. The second term on the right side of Equation (6) is a term for evaluating the likelihood of the estimated super-resolution image, and is evaluated using known information and properties of the estimated super-resolution image. The constraint parameter α is appropriately set according to the image and the imaging system, and c is a matrix representing a high-pass filter or the like. Note that, as the super-resolution evaluation value J, instead of the one represented by the equation (6),
“Multiframe Demosaicing and Super-Resolution of ColorImages” (IEEE TRANSACTION ON IMAGE PROCESSING, VOL.15, NO.1, JANUARY 2006)
A method for evaluating using a bilateral filter or a method for evaluating using color correlation may be used.

  Next, in step S304, the super-resolution processing unit 112 determines whether or not the super-resolution evaluation value J calculated in step S303 is equal to or less than the threshold value β. The threshold β is appropriately set according to the type of image, the performance of the imaging system, and the purpose of super-resolution processing. If J> β, the process proceeds to step S305. If J ≦ β, the super-resolution processing is terminated and the estimated super-resolution image is set as a super-resolution image.

  In step S305, the super-resolution processing unit 112 updates the estimated super-resolution image in a direction in which the super-resolution evaluation value J calculated in step S303 is optimized. As an update method, any optimization method can be used, but as an example, using the steepest descent method,

You can update as Here, X ′ represents an updated super-resolution image, and γ is a constant for setting an update step size. Thereafter, X ′ is assumed to be an estimated super-resolution image, and the processing from step S302 is repeated until J ≦ β in step S304 to obtain a super-resolution image.

  Returning to FIG. 2, in step S <b> 107, the imaging control unit 109 stores the super-resolution image generated in step S <b> 106 in the memory 116. At this time, if necessary, the super resolution image is compressed by the compression unit 114 and then stored in the memory 116.

  According to the present embodiment, the imaging optical system 100 is defocused with respect to the subject using one imaging element 103, and the position of the aperture of the variable aperture stop 101 is changed to change the light beam passage position. As a result, a plurality of pixel-shifted images can be acquired. Thereby, it is possible to acquire a plurality of pixel shifted images to which a desired pixel shift amount is given with high accuracy without increasing the size of the imaging device. Therefore, a good super-resolution image can be generated by super-resolution processing using these pixel shifted images.

  In this embodiment, since the shape of the aperture of the variable aperture stop 101 is also variable, various pixel shift amounts can be handled.

  In this embodiment, the case of generating a super-resolution image as a still image using a pixel-shifted image as a still image has been described, but high-speed imaging is performed while changing the position of the opening for a moving image. Thus, it is possible to generate a super-resolution image as a moving image.

  FIG. 7 shows a configuration of an image pickup apparatus 2 that is Embodiment 2 of the present invention. In the present embodiment, the same components as those in the first embodiment are denoted by the same reference numerals as those in the first embodiment, and the description is omitted. The flow of the imaging operation in the present embodiment is basically the same as the imaging operation described in Embodiment 1 with reference to FIG. However, in this embodiment, the variable aperture stop 202 has a structure capable of selectively opening and closing a plurality of apertures. In this embodiment, the image sensor 103 is configured to be movable in the optical axis direction, and the contents of the defocus control during the imaging operation in relation to this are described with reference to FIG. 4 in the first embodiment. This is different from the defocus control.

  As shown in FIG. 8, the variable aperture stop 201 has a substrate 201a having a circular opening in each of four regions divided by two lines passing through the center and orthogonal to each other, and a light shielding for opening and closing each opening. And blades 201b. A plurality of light shielding blades 201b are provided for each opening, and opening the light shielding blade 201b causes the opening to pass light, and closing the light shielding blade 201b causes the opening to be in a light shielding state. An actuator (not shown) that opens and closes a plurality of light shielding blades 201b provided in each opening is attached to the substrate 201a. By switching between one opening for opening the light shielding blade 201b and three openings for closing the light shielding blade 201b, the position of the opening through which the light beam from the subject passes can be changed in the variable aperture stop 201. Further, by changing the position of the light shielding blade 201b provided in each opening in the opening and closing direction, the size of the opening can be changed and the amount of light passing through can be adjusted.

  The defocus control unit 206 controls the image sensor moving unit 219 configured with a piezo element or the like to move the image sensor 103 in the optical axis direction. In the present embodiment, the movement of the image sensor 103 causes the position in the optical axis direction of the image sensor 103 relative to the imaging plane on which the light flux that has passed through the apertures of the image pickup optical system 100 and the variable aperture stop 201 forms an image, that is, the connection. The positional relationship between the image plane and the image sensor 103 in the same direction can be changed. The image sensor 103 and the image sensor moving unit 219 constitute a positional relationship variable means. Upon receiving a command from the imaging control unit 209, the defocus control unit 206 determines the positional difference (defocus amount) in the optical axis direction according to the pixel shift amount between the imaging surface and the imaging element 103 (imaging surface). The position of the image sensor 103 is controlled so as to satisfy the positional relationship.

  Note that the imaging optical system 100 of the present embodiment includes a focus lens 100a as in the first embodiment. However, the focus lens 100a is moved in the optical axis direction only in order to perform focusing by autofocus (or manual focus) by the focus control unit 205.

  Next, the operation of the defocus control performed by the defocus control unit 206 in this embodiment will be described with reference to the flowchart of FIG.

  In step S401, the defocus control unit 206 calculates an image sensor movement amount d that is a movement amount (correction amount) of the image sensor 103 to obtain a desired pixel shift amount. The imaging element moving amount d is calculated by the following equation (8) from the relationship shown in FIG.

Here, S is a pixel shift amount, and b is an image distance. R is the distance from the optical axis of the imaging optical system 100 to the center of the aperture of the variable aperture stop 201.

  In this embodiment, a Bayer array color image (color pixel shifted image) is acquired as N (= 4) pixel shifted images. Of the four color pixel shifted images, the pixel shift amount of the other three images with respect to one image is one pixel only in the horizontal direction, one pixel only in the vertical direction, and one pixel in the horizontal and vertical directions. It is desirable to do. In this case, the pixel shift amount is 1 pixel × √2 in the diagonal direction. Then, when the pixel size is μ as in the equation (2) described in the first embodiment, the pixel shift amount S is expressed by the following equation (9).

  By substituting equation (9) into equation (8), the image sensor movement amount d when photographing four color pixel shifted images is calculated by the following equation (10).

  In step S402, the defocus control unit 206 moves the image sensor 103 in the optical axis direction by the image sensor movement amount d calculated in step S401 via the image sensor movement unit 219.

  Note that as the subject distance increases, the image sensor movement amount d becomes smaller and exceeds the position resolution capable of controlling the position of the image sensor 103. For this reason, a threshold value may be provided for the subject distance, and when the subject distance exceeds the threshold value, defocus control by moving the image sensor 103 may not be performed.

  In this embodiment, by using the variable aperture stop 201 having a plurality of openings that can be opened and closed by a mechanical opening and closing structure, it is possible to perform imaging with a pixel shift with a simpler configuration than in the first embodiment. Although the degree of freedom of the opening shape is reduced, there is also an advantage that it is superior in light-shielding properties compared to liquid crystal. In this embodiment, the pixel shift by defocusing is performed by moving the image sensor, but this method is also applicable to the first embodiment.

  In each of the above-described embodiments, the imaging apparatus integrally including the imaging optical system has been described. However, the present invention can also be applied to an optical apparatus such as an interchangeable lens that can be attached to and detached from the imaging apparatus.

  Each embodiment described above is only a representative example, and various modifications and changes can be made to each embodiment in carrying out the present invention.

  It is possible to provide a small optical apparatus that can acquire a good pixel-shifted image.

DESCRIPTION OF SYMBOLS 1, 2 Imaging apparatus 100 Imaging optical system 101,201 Variable aperture stop 103 Imaging element 106,206 Defocus control part 107,207 Aperture control part 109,209 Imaging control part 112 Super-resolution processing part

Claims (8)

An optical device used when acquiring a plurality of pixel-shifted images used for generating a super-resolution image using an imaging element,
An imaging optical system;
Aperture position varying means capable of changing the position of the aperture through which the light beam passes in a plane perpendicular to the optical axis direction;
A positional relationship variable means capable of changing a positional relationship in an optical axis direction between an imaging surface on which a light flux from a subject passing through the imaging optical system and the aperture forms an image; and
The positional relationship is set so that the imaging plane and the image sensor have a positional difference in the optical axis direction according to the pixel shift amount between the pixel shifted images, and each of the pixel shifted images is acquired. And a control unit that changes the position of the opening for each imaging to be performed.   The optical apparatus according to claim 1, wherein the control unit changes the position of the opening for each imaging after the setting of the positional relationship.   The optical apparatus according to claim 1, wherein the opening position changing unit is capable of changing a shape of the opening.   The opening position varying means has a plurality of regions in a plane orthogonal to the optical axis direction, and a region where the opening is formed among the plurality of regions is set as a light passing state, and the other regions are set as a light shielding state. The optical apparatus according to any one of claims 1 to 3, wherein:   5. The position relation variable unit changes a position of the imaging plane with respect to the image pickup element by moving a focus element included in the image pickup optical system in an optical axis direction. 6. The optical apparatus according to one item.   5. The optical according to claim 1, wherein the positional relationship variable unit changes the position of the image sensor with respect to the imaging plane by moving the image sensor in an optical axis direction. 6. machine. An image sensor;
7. The image generating device according to claim 1, further comprising image generation means for generating a plurality of pixel-shifted images by an output from the image sensor and generating a super-resolution image using the plurality of pixel-shifted images. The optical apparatus as described in any one. A pixel-shifted image acquisition method used when acquiring a plurality of pixel-shifted images used for generating a super-resolution image using an imaging element,
The position of the aperture through which the light beam passes can be changed in a plane perpendicular to the optical axis direction,
It is possible to change the positional relationship in the optical axis direction between the imaging element and the imaging surface on which the light beam from the subject that has passed through the imaging optical system and the aperture forms an image,
The positional relationship is set so that the imaging plane and the image sensor have a positional difference in the optical axis direction according to the pixel shift amount between the pixel shifted images, and each of the pixel shifted images is acquired. A pixel-shifted image acquisition method characterized in that the position of the opening is changed for each imaging to be performed.