JP2020202555A - Imaging apparatus - Google Patents

Abstract

To create a high-resolution image from photographic images acquired when an imaging apparatus is in a hand-held state.SOLUTION: An imaging apparatus 10 comprises: an image sensor 12; an image sensor driving unit 22 that drives the image sensor 12; a high-resolution image creation unit that creates a high-resolution image with a predetermined number of photographic images in which respective subjects are shifted from each other and captured in a predetermined positional relationship as original images; and a shaking amount detection unit 24 that detects the amount of shaking of the imaging apparatus 10. During the execution of a hand-held high resolution mode, the image sensor driving unit 22 drives the image sensor 12 with a supplemented change amount in which a change amount for correction based on the shaking amount is supplemented with an additional change amount, and a high-resolution image is created from photographic images acquired by the image sensor 12.SELECTED DRAWING: Figure 8

Description

The present disclosure relates to an imaging device capable of generating a high resolution image from a plurality of captured images.

For example, Patent Document 1 discloses an imaging device capable of generating a high-resolution image (high-resolution image) having a number of pixels higher than the number of pixels of these captured images by using a plurality of captured images. There is.

The image pickup apparatus described in Patent Document 1 includes a moving mechanism for moving at least one of a lens and an image sensor by a predetermined pixel. When at least one of the lens and the image sensor is moved by the moving mechanism, the image of the subject on the light receiving surface of the image sensor is displaced by a predetermined pixel. A high-resolution image is generated by the image sensor acquiring a plurality of captured images in which the subject is shifted by a predetermined pixel and synthesizing the plurality of captured images.

Japanese Unexamined Patent Publication No. 6-225317

By the way, in the case of the image pickup apparatus described in Patent Document 1, in order for the image sensor to acquire a plurality of captured images necessary for generating a high resolution image, the image pickup apparatus needs to be fixed with a tripod or the like so as not to shake. .. On the other hand, it is desired to generate a high-resolution image from a captured image acquired when the image pickup device is in a handheld state.

Therefore, an object of the present disclosure is to provide an imaging device capable of generating a high-resolution image from a captured image acquired in a handheld state.

According to one aspect of the present disclosure
It is an imaging device
An image sensor that acquires a captured image of the subject, and
An image sensor driving unit that drives the image sensor so that at least one of the position and the posture changes,
A high-resolution image creation unit that creates a high-resolution image from a predetermined number of original images by using a predetermined number of captured images in which each subject is displaced from each other and appears in a predetermined positional relationship.
A blur amount detection unit that detects the blur amount of the image pickup apparatus,
It has an operation unit for instructing the user to start execution of the handheld high resolution mode for creating the high resolution image from the captured image acquired when the imaging device is in the handheld state.
During the execution of the handheld high resolution mode, the image sensor driving unit is driving the image sensor with an augmented change amount in which an additional change amount is supplemented to the change amount for blur correction calculated based on the blur amount. Provided is an imaging device in which the high-resolution image creating unit creates the high-resolution image from the captured image acquired by the image sensor.

According to the present disclosure, the image pickup apparatus can generate a high resolution image from a captured image acquired in a handheld state.

Schematic perspective view of the image pickup apparatus according to the embodiment of the present disclosure. Block diagram of the image pickup device Diagram showing the positional relationship of the subjects in each of a predetermined number of original images The figure which shows the process of creating red pixel data, green pixel data, and blue pixel data for a high resolution image from a predetermined number of original image data. The figure which shows the process of creating the high resolution image data from the red pixel data, the green pixel data, and the blue pixel data for a high resolution image. The figure which shows the positional relationship of the subject which appears in each of a plurality of captured images acquired during execution of the handheld high resolution mode. The figure which shows an example of the augmentation change amount which changes the position of an image sensor in the X-axis direction. The figure which shows an example of the process from the start of the handheld high resolution mode by the user until the high resolution image is created. Block diagram of an image pickup apparatus according to another embodiment of the present disclosure. The figure which shows another example of the augmentation change amount which changes the position of an image sensor in the X-axis direction. Timing chart showing changes in the position of the image sensor to acquire multiple original images

Hereinafter, embodiments will be described in detail with reference to the drawings as appropriate. However, more detailed explanation than necessary may be omitted. For example, detailed explanations of already well-known matters and duplicate explanations for substantially the same configuration may be omitted. This is to avoid unnecessary redundancy of the following description and to facilitate the understanding of those skilled in the art.

It should be noted that the inventors (or others) intend to limit the subject matter described in the claims by those skilled in the art by providing the accompanying drawings and the following description in order to fully understand the present disclosure. It is not something to do.

The information processing apparatus according to the embodiment of the present disclosure will be described below with reference to FIGS. 1 to 8.

FIG. 1 is a perspective view schematically showing an imaging device according to an embodiment of the present disclosure. FIG. 2 is a block diagram of the imaging device.

As shown in FIGS. 1 and 2, the image pickup apparatus 10 according to the present embodiment includes an image sensor 12 that acquires a captured image of a subject, and a plurality of image sensors 12 that form an image of the subject on a light receiving surface 12a of the image sensor 12. It is a so-called digital camera having an interchangeable lens 14 including a lens.

Further, as shown in FIG. 2, the image pickup device 10 includes an operation unit 16 such as a shutter button for the user to operate the image pickup device 10, and a display unit 18 used for the user to confirm a captured image. It has a storage unit 20 that stores a photographed image or the like acquired by the image sensor 12.

Further, in the case of the present embodiment, the image pickup device 10 has a sensor actuator (image sensor drive unit) 22 for driving the image sensor 12 and an angular velocity sensor (shake amount) for detecting the shake amount of the image pickup device 10 on the main body 10a. It has a detection unit) 24 and a position sensor 25 that detects the position of the image sensor 12.

The image sensor 12 includes a light receiving surface 12a. In the case of the present embodiment, a plurality of photodiodes that detect red light, a plurality of photodiodes that detect green light, and a plurality of photodiodes that detect blue light are Bayer-arranged on the light receiving surface 12a. There is.

The interchangeable lens 14 is a unit that can be attached to and detached from the main body 10a of the image pickup apparatus 10, and forms an image of a subject on the light receiving surface 12a of the image sensor 12. The interchangeable lens 14 includes a focus lens 14a, a zoom lens 14b, and the like. Further, the interchangeable lens 14 includes actuators 14c and 14d for driving a focus lens and a zoom lens, and a lens controller 14e for controlling the actuators. By connecting the lens mounter 14f provided on the interchangeable lens 14 and the body mounter 10b provided on the main body 10a of the image pickup device 10, the interchangeable lens 14 is detachably fixed to the main body 10 of the image pickup device 10. It is also electrically connected.

The operation unit 16 is composed of a plurality of buttons such as the shutter button 16a. In the case of the present embodiment, the operation unit 16 is configured to receive instructions from the user to start execution of each of the camera shake correction mode, the normal high resolution mode, and the handheld high resolution mode as the operation mode of the imaging device 10. For example, the operation unit 16 includes a button for starting each mode.

The image stabilization mode is a mode for suppressing blurring of a subject in a captured image caused by blurring of the image pickup apparatus 10. The normal high-resolution mode is a mode for creating a high-resolution image from a plurality of captured images acquired when the image pickup device 10 is fixed by a tripod or the like. The handheld high resolution mode is a mode for creating a high resolution image from a plurality of captured images acquired when the image pickup device 10 is in the handheld state. The details of these modes will be described later.

The display unit 18 is, for example, a liquid crystal display, which is a device for displaying information such as a captured image and a mode currently being executed.

A part of the operation unit 16 and the display unit 18 may be provided by a touch screen display.

The storage unit 20 is, for example, a memory card that is detachably and detachably attached to the image pickup device 10 via a memory built in the image pickup device 10 or a card slot (not shown).

The sensor actuator 22 is a device that drives the image sensor 12 to change its state, and in the case of the present embodiment, it changes its position as a state. In order to change the position of the image sensor 12, the sensor actuator 22 moves the image sensor 12 in two directions (X-axis direction and Y-axis direction) orthogonal to the optical axis C, for example. The X-axis direction is the width direction of the image pickup device 10, the Y-axis direction is the height direction of the image pickup device 10, and the Z-axis direction is the extension direction of the optical axis C.

The angular velocity sensor 24 detects the amount of blurring of the imaging device 10, for example, the amount of blurring (angular velocity) that occurs when the imaging device 10 is used in a state of being held by the user. In the case of this embodiment, the angular velocity sensor 24 detects the amount of blurring in the X-axis direction and the Y-axis direction.

The position sensor 25 detects the position of the image sensor 12. In the case of the present embodiment, the position sensor 25 detects the position of the image sensor 12 in the X-axis direction, the Y-axis direction, and the Roll direction.

The image pickup device 10 also has a camera controller 30 that controls the operation of the image pickup device 10. For example, the camera controller 30 outputs a signal corresponding to a focus operation or a zoom operation to the user's operation unit 16 to the lens controller 14e of the interchangeable lens 14, and the lens controller 14e sends the signals to the actuators 14c and 14d based on the signal. It drives the focus lens 14a and the zoom lens 14b. Further, in the case of the present embodiment, the camera controller 30 has a sensor actuator control unit 32 that controls the sensor actuator 22 and a blur correction unit 34 for suppressing blurring of the subject in the captured image caused by blurring of the image pickup device 10. And a high-resolution image creation unit 36 that creates a high-resolution image.

The camera controller 30 is, for example, a processor such as a CPU, and controls various operations of the image pickup apparatus 10 by operating according to a program stored in the flash memory (storage unit 20). In the case of the present embodiment, the processor functions as a sensor actuator control unit 32, a blur correction unit 34, and a high-resolution image creation unit 36 by operating according to the program. The camera controller 30 uses the DRAM (storage unit 20) as a work memory during a control operation or an image processing operation. The camera controller 30 may be configured by a hard-wired electronic circuit. Further, the camera controller 30 may be composed of one semiconductor chip together with an image processing unit such as a high resolution image creation unit 36 and a DRAM, or a sensor actuator control unit 32, a blur correction unit 34, and a high resolution image creation unit 36. It may be composed of a separate semiconductor chip that functions as.

The sensor actuator control unit 32 controls the sensor actuator 22 as described later when the camera shake correction mode, the normal high resolution mode, and the handheld high resolution mode described above are executed. Based on the control, the sensor actuator 22 drives the image sensor 12.

The image stabilization unit 34 cooperates with the sensor actuator control unit 32 so that the subject is captured in the captured image without substantially blurring even if the image pickup device 10 is blurred during the execution of the image stabilization mode. It is configured to change the position of the sensor 12. That is, the blur correction unit 34 and the sensor actuator control unit 22 so that the image of the subject continues to be imaged on substantially the same portion of the light receiving surface 12a of the image sensor 12 even if the image pickup device 10 is blurred. The position of the image sensor 12 is changed in cooperation with each other. For example, when the image pickup apparatus 10 vibrates periodically, the image sensor 12 is vibrated with substantially the same period and amplitude.

Therefore, the blur correction unit 34 is for forming an image of the subject on substantially the same portion of the light receiving surface 12a of the image sensor 12 based on the blur amount of the image pickup device 10 detected by the angular velocity sensor 24. The amount of change in the position of the required image sensor 12 is calculated. For example, when the image pickup device 10 vibrates periodically as the amount of blur that changes with the passage of time, the blur correction unit 34 calculates the blur frequency and amplitude based on the detection result of the angular velocity sensor 24. , Is output to the sensor actuator control unit 32. The sensor actuator control unit 32 determines the amount of change for blur correction for deviating the position of the image sensor 12 based on the input blur frequency and amplitude and the current position of the image sensor 12 detected by the position sensor 25. calculate. Then, based on the calculated change amount for blur correction, the sensor actuator control unit 32 controls the sensor actuator 22 so that the sensor actuator 22 drives the image sensor 12 with the change amount for blur correction. With such an image stabilization mode, even if the image pickup device 10 is shaken, it is possible to acquire a captured image in which the subject is captured without being substantially shaken. Here, the position sensor 25 may be a Hall element or other position sensor.

The high-resolution image creation unit 36 sets a captured image (RAW image) that is acquired by the image sensor 12 during execution of the normal high-resolution mode and that the respective subjects are displaced from each other and are captured in a predetermined positional relationship, and is high-resolution from the original image. Create an image. A high-resolution image is an image having a higher resolution (a large number of pixels) than a captured image.

First, when the normal high-resolution mode is being executed, that is, when the image pickup device 10 is fixed via a tripod or the like, when the user operates the shutter button 16a, a predetermined number of captured images (original images) are taken instead of one. Is acquired via the image sensor 12. Specifically, when the shutter button 16a is operated, the sensor actuator 22 drives the image sensor 12, and drives and stops the image sensor 12 at each of a plurality of different positions (original image acquisition positions). Then, the image sensor 12 acquires the original image at each of the plurality of original image acquisition positions. As a result, a predetermined number of original images in which the respective subjects are displaced from each other and appear in a predetermined positional relationship are acquired.

FIG. 3 shows the positional relationship of the subjects reflected in each of a predetermined number of original images.

FIG. 3 shows the positions (required positions) Np (1) to Np (8) of the subject appearing in each of the eight original images required for creating the high-resolution image. The plurality of required positions Np (1) to Np (8) are, for example, positions on the captured image in which the feature points of the subject are captured. In the case of the present embodiment, the plurality of required positions Np (1) to Np (8) exist within the range of 1.5 pixels × 1.5 pixels on the XY plane.

The sensor actuator 22 is a change amount (change amount for acquiring an original image) for capturing a subject at each of the required positions Np (1) to Np (8) by being controlled by the sensor actuator control unit 32, and is an image sensor 12 Change the position of. For example, the sensor actuator 22 is an image obtained by superimposing a vibration that periodically changes the position in the X-axis direction and a vibration that periodically changes the position in the Y-axis direction as the amount of change for acquiring the original image. Drive the sensor 12.

For example, as shown in FIG. 3, when the image sensor 12 stops at the position for acquiring the original image, first, the first original image in which the subject appears at the first required position Np (1) is earned. By moving the image sensor 12 by one pixel in the minus direction of the X axis from the original image acquisition position where the first original image was acquired, the second original image in which the subject appears at the required position Np (2) is acquired. Will be done. By moving the image sensor 12 by one pixel in the Y-axis plus direction from the original image acquisition position where the second original image was acquired, the third original image in which the subject appears at the required position Np (3) is acquired. To. By moving the image sensor 12 by one pixel in the plus direction of the X axis from the original image acquisition position where the third original image was acquired, the fourth original image in which the subject appears at the required position Np (4) is acquired. To. The image sensor 12 moves by half a pixel in the minus direction of the X axis and half a pixel in the minus direction of the Y axis from the original image acquisition position where the fourth original image is acquired, so that the subject appears at the required position Np (5). The fifth original image is acquired. By moving the image sensor 12 by one pixel in the minus direction of the X axis from the original image acquisition position where the fifth original image was acquired, the sixth original image in which the subject appears at the required position Np (6) is acquired. To. By moving the image sensor 12 by one pixel in the plus direction of the Y axis from the original image acquisition position where the sixth original image was received, the seventh original image in which the subject appears at the required position Np (7) is acquired. To. Then, the image sensor 12 moves by one pixel in the X-axis plus direction from the original image acquisition position where the seventh original image is acquired, so that the eighth original image in which the subject appears at the required position Np (8) is displayed. To be acquired.

The order of acquiring the original images described above is the order in which the movement amount of the image sensor 12 is minimized, that is, the original images are acquired in a short time. As long as eight original images of the subject appearing at the required positions Np (1) to Np (8) can be acquired, the order of acquisition and the driving direction are not limited.

By driving the image sensor 12 in this way, eight original images in which the respective subjects are displaced from each other and are captured in a predetermined positional relationship are acquired. The high-resolution image creation unit 36 creates one high-resolution image using the eight original images acquired in this way. The production method will be described with reference to FIGS. 4 and 5. In order to facilitate understanding, an original image having a size of 4 pixels (w) × 4 pixels (h) will be taken as an example. w indicates the width of the image, and h indicates the height of the image.

FIG. 4 shows a process of creating red pixel data, green pixel data, and blue pixel data for a high resolution image from the original image data. FIG. 5 shows a process of creating high-resolution image data from red pixel data, green pixel data, and blue pixel data for a high-resolution image.

As shown in FIG. 4, the high-resolution image creation unit 36 first doubles the number of pixels of each of the eight original image (RAW image) data. That is, RAW image data of 8 pixels (2w) × 8 pixels (2h) is created from RAW image data of 4 pixels (w) × 4 pixels (h).

The high-resolution image creation unit 36 separates each of the RAW image data whose number of pixels is doubled into RGB, and selects the red pixel data indicating the position of the red pixel, the green pixel data indicating the position of the green pixel, and the position of the blue pixel. Create the blue pixel data shown. The red pixel data created from the nth original image (the original image in which the subject appears at the required position Np (n)) is Rn, the green pixel data is Gn, and the blue pixel data is Bn.

As shown in FIG. 4, the high-resolution image creation unit 36 next creates one high-resolution red pixel data Rs that collectively shows the positions of the red pixels in each of the eight red pixel data R1 to R8. Similarly, high-resolution green pixel data Gs is created from eight green pixel data G1 to G8, and high-resolution blue pixel data Bs is created from eight blue data B1 to B8.

As shown in FIG. 5, the high resolution image creation unit 36 creates red interpolation data Rc for demoizing high resolution red pixel data Rs based on the interpolation algorithm, and similarly demosizes high resolution blue pixel data Bs. Blue interpolation data Bc is created for this purpose.

Then, the high-resolution image creation unit 36 synthesizes the red interpolation data Rc, the high-resolution green pixel data Gs, and the blue interpolation data Bc to create high-resolution image data. The high-resolution image data created by the high-resolution image creation unit 36 is stored in the storage unit 20. The high-resolution image generation method is not limited to this method.

The high-resolution image creation unit 36 creates a high-resolution image not only during the execution of the normal high-resolution mode but also during the execution of the handheld high-resolution mode. However, the method of creating a high-resolution image in the handheld high-resolution mode, particularly the method of acquiring an original image necessary for creating a high-resolution image, is different from that during execution of the normal high-resolution mode. This handheld high resolution mode will be described.

The handheld high-resolution mode is a mode in which a high-resolution image is created from a captured image acquired by the image sensor 12 during the blur correction while performing the blur correction (blurring correction mode) by the blur correction unit 34 described above.

As described above, the blur correction unit 34 forms an image of the subject on substantially the same portion of the light receiving surface 12a of the image sensor 12 based on the blur amount of the image pickup device 10 detected by the angular velocity sensor 24. The position of the image sensor 12 is changed as described above. As a matter of course, when the image of the subject is completely formed on the same portion of the light receiving surface 12a of the image sensor 12, the subject does not shift in the plurality of captured images. However, in reality, when the amount of blurring of the imaging device 10 is several hundred pixels or more, the image of the subject is imaged with a deviation of about several pixels from the light receiving surface 12a of the image sensor 12 even if the blurring correction is executed. As a result, the position of the subject is displaced by several pixels between the plurality of captured images. The handheld high-resolution mode is a mode that utilizes the misalignment of the subject of about several pixels.

In the case of the present embodiment, when the user presses the shutter button 16a while the handheld high resolution mode is being executed, the image sensor 12 acquires a number of captured images (for example, 16) exceeding a predetermined number required for creating the high resolution image. To.

At this time, the position of the image sensor 12 is changed by the blur correction unit 34 so that the image of the subject is formed on substantially the same portion of the light receiving surface 12a of the image sensor 12 (for example, it is vibrated). ). However, in reality, the image of the subject is imaged with a deviation of about several pixels from the light receiving surface 12a of the image sensor 12. As a result, a number of captured images exceeding a predetermined number of images in which the subjects are displaced from each other are acquired.

The high-resolution image creation unit 36 detects the positional relationship of the subjects captured in each of the 16 original images, for example, by performing image analysis processing on each of the predetermined number of images in which the subjects are displaced from each other. For example, the first captured image is used as a reference captured image, and the displacement of the subject captured in each of the 2nd to 16th original images with respect to the position of the subject captured in the reference captured image is detected.

FIG. 6 shows the positional relationship of the subjects captured in each of the plurality of captured images acquired during the execution of the handheld high resolution mode.

FIG. 6 shows the positions Cp (1) to Cp (16) of the subject (for example, its feature point) captured in each of the 16 captured images acquired during the execution of the handheld high resolution mode. The position of the subject reflected in the m-th photographed image is C (m). Further, FIG. 6 shows the required positions Np (1) to Np (8) of the subject appearing in each of the eight original images required for creating the high-resolution image.

The high-resolution image creation unit 36 selects an original image to be used for creating a high-resolution image from 16 captured images. As shown in FIG. 6, the selection is made between the positions Cp (1) to Cp (16) of the subject appearing in each of the 16 captured images and the subject appearing in each of the eight original images required for creating the high-resolution image. It is executed based on the required positions Np (1) to Np (8).

Specifically, the high-resolution image creation unit 36 sets the m-th captured image in which the position Cp (m) of the subject is within the first range A1 with respect to each of the required positions Np (n) of the original image. Select as the nth original image. For example, the seventh captured image in which the subject appears at the position Cp (7) of A1 within the first range with respect to the required position Np (2) is selected as the second original image. Further, for example, a plurality of captured images (12th and 13th images) in which a subject (subject at position Cp (12) and position Cp (13)) is captured within the first range A1 such as the required position Np (7). If there is a photographed image of the eye), the thirteenth photographed image in which the subject appears at the closest position Cp (13) is selected as the seventh original image.

Note that, as in the required position Np (1) shown in FIG. 6, there may be no captured image in which the subject is captured within the first range A1 with respect to the required position Np (n) of the original image. In this case, the first original image in which the subject appears at the required position Np (1) is insufficient, and a high-resolution image cannot be created. Therefore, in the case of the present embodiment, the high-resolution image creation unit 36 creates the missing original image by synthesizing at least two captured images.

For example, as shown in FIG. 6, the high-resolution image creation unit 36 is second with respect to the required position Np (1) in order to create the first original image in which the subject appears at the required position Np (1). A plurality of captured images in which the subject appears within the range A2 of are used. Specifically, the first to third captured images in which the subject appears at the positions Cp (1), Cp (2), and Cp (3) are combined to create the first original image. The second range A2 is a larger range than the first range A1. At this time, regarding the composition, it is also possible to perform the composition by weighting according to the positional relationship of the images used for the composition.

In addition, in order to create an original image in which the subject appears at a required position with high accuracy, the high-resolution image creation unit 36 may preprocess at least two captured images to be used.

For example, each captured image is divided into a plurality of areas. Next, the amount of misalignment (for example, displacement vector) is detected for each region of the remaining captured images with reference to each region of the captured image first acquired in at least two captured images to be preprocessed. Based on the amount of misalignment detected for each region, the total amount of misalignment and the amount of rotation of each of the remaining captured images with respect to the entire first captured image are calculated. Based on the calculated total displacement amount and rotation amount, each subject appears at substantially the same position for each of at least two captured images to be preprocessed, that is, appears near the required position of the original image. As described above, translation correction and rotation correction are performed. By using a plurality of captured images in which such preprocessing has been performed, it is possible to create an original image in which the subject appears at substantially a required position.

As described above, in the handheld high-resolution mode, even if the blur correction is executed, the image of the subject is imaged with a deviation of about several pixels from the light receiving surface 12a of the image sensor 12, which is necessary for creating a high-resolution image. It is possible to acquire a predetermined number of original images in which the respective subjects are displaced from each other and appear in a predetermined positional relationship. However, when the amount of blurring of the image pickup apparatus 10 is small, for example, 100 pixels or less, when the blurring correction is executed, the image of the subject is displaced on the light receiving surface 12a of the image sensor 12 by the number of pixels of 1 pixel or less and imaged. there's a possibility that. Due to the blur correction, the image of the subject may be completely imaged on the same portion of the light receiving surface of the image sensor 12 without being displaced. In this case, as shown in FIG. 3, the subjects are displaced from each other in the range of 1.5 pixels × 1.5 pixels and are captured in a predetermined positional relationship (the subjects are captured at the required positions Np (1) to Np (8), respectively. ) It may not be possible to obtain the specified number of original images.

Therefore, during the execution of the handheld high resolution mode, when the amount of displacement of the subject between a plurality of captured images is smaller than the amount of displacement of a predetermined threshold (for example, one pixel or less), the sensor actuator control unit 32 uses the sensor actuator 22. The sensor actuator 22 is controlled so that the position of the image sensor 12 is changed by the augmented change amount supplemented with the additional change amount for the blur correction change amount.

FIG. 7 shows an example of the augmented change amount that changes the position of the image sensor in the X-axis direction.

As shown in FIG. 7, when the execution of the handheld high resolution mode is started (timing t0), the sensor actuator 22 changes the position of the image sensor 12 by the amount of change for blur correction. For example, the sensor actuator 22 vibrates the image sensor 12 with a vibration having a frequency of about several Hz corresponding to the blur of the image pickup apparatus 10 as a change amount for blur correction. As a result, the image of the subject is imaged on the light receiving surface 12a of the image sensor 12 with a deviation amount of one pixel or less, that is, a deviation amount at which the original image cannot be acquired.

When the shutter button 16a is operated by the user's finger (timing t1) during the execution of the handheld high resolution mode, the image sensor 12 starts acquiring a plurality of captured images. During each exposure, the sensor is driven in response to the change for blur correction, but at a timing other than the exposure time, the sensor actuator 22 is imaged with the augmented change amount in which the additional change amount is supplemented with the change amount for blur correction. The position of the sensor 22 is changed. The additional change amount is, for example, a vibration having a lower frequency and a lower amplitude than the change amount for blur correction, and has a frequency and an amplitude of about several pixels that cause a shift between images in continuous shooting of the imaging device 10. Is.

By replenishing such an additional change amount, the image of the subject is larger than before the additional change amount is replenished, and a deviation amount that enables acquisition of the original image necessary for creating a high-resolution image, for example. An image is formed on the light receiving surface 12a of the image sensor 12 with a deviation amount of several pixels or more. As a result, as shown in FIG. 3, the subjects are displaced from each other in the range of 1.5 pixels × 1.5 pixels and are captured in a predetermined positional relationship (the subjects are captured at the required positions Np (1) to Np (8), respectively. ) It becomes possible to acquire a predetermined number of original images.

The additional change amount may be a change amount that can acquire a predetermined number of original images required for creating a high-resolution image (for example, periodic vibration or random vibration), and the change for original image acquisition in the normal high-resolution mode. You may use the quantity. Further, by simply weakening the degree of blur correction, an amount of deviation that enables acquisition of the original image necessary for creating a high-resolution image may be generated.

Further, when the acquisition of the plurality of captured images is completed (timing tx), the replenishment of the additional change amount is completed.

The shutter button 16a is operated by the user to determine whether or not to replenish the additional amount of change, that is, whether or not the amount of displacement of the subject between a plurality of captured images is smaller than the amount of displacement of a predetermined threshold. It may be performed using the first and second photographed images acquired immediately after the image is taken.

Specifically, the high-resolution image creation unit 36 detects the amount of misalignment of the subject captured in each of the first and second captured images. When the detected misalignment amount is smaller than the predetermined threshold misalignment amount (for example, 1 pixel or less), that is, when the first captured image and the second captured image are substantially the same, the sensor The actuator 22 starts driving by replenishing the change amount for blur correction with the additional change amount (the image sensor 12 starts to be driven by the augmented change amount). In this case, a predetermined number of original images used for creating a high-resolution image are selected from the third and subsequent captured images.

Instead of this, the sensor actuator 22 replenishes the change amount for blur correction with the additional change amount after the start of the handheld high resolution mode (timing t0) until the shutter button 16a is pressed (timing t1). It may start (the image sensor 12 may start to be driven by the augmented change amount). In this case, when the amount of blur detected by the angular velocity sensor 24 immediately before or after the start of the handheld high resolution mode is smaller than the predetermined amount of blur corresponding to the predetermined amount of threshold misalignment, the sensor actuator 22 Drives the image sensor 12 with the augmented change amount. At this time, the angular velocity sensor 24 is used as the sensor used for the determination, but an acceleration sensor, a determination using the movement of the image, or the like may be used.

From here on, the process from the start of the handheld high-resolution mode by the user to the creation of the high-resolution image will be described with reference to FIG.

FIG. 8 shows an example of the process from the start of the handheld high resolution mode by the user to the creation of the high resolution image.

As shown in FIG. 8, when the handheld high resolution mode is started by the user, in step S100, the camera controller 30 determines the amount of change for blur correction based on the amount of blur of the image pickup device 10 detected by the angular velocity sensor 24. calculate.

Next, in step S110, the sensor actuator 22 starts driving the image sensor 12 with the amount of change for blur correction calculated in step S100 (the position of the image sensor 12 starts to change). As a result, blur correction is started.

Subsequently, in step S120, it is determined whether or not the user operates the shutter button 16a. When the shutter button 16a is operated by the user, the process proceeds to the next step S130.

In step S130, the image sensor 12 whose position is changed by the amount of blur correction displacement starts to acquire the captured image.

In step S140, the camera controller 30 detects the amount of displacement of the subject between the first and second captured images.

In step S150, it is determined whether or not the misalignment amount detected in step S140 is larger than the predetermined threshold misalignment amount. If it is larger, the process proceeds to step S160. If not, the process proceeds to step S190.

In step S160, the image sensor 12 acquires a plurality of captured images including the first and second captured images, for example, 16 captured images.

In step S170, the camera controller 30 selects a predetermined number of original images to be used for creating a high-resolution image from the acquired plurality of captured images. For example, 8 original images are selected from 16 captured images.

In step S180, the camera controller 30 creates a high-resolution image from a predetermined number of original images selected in step 170.

When the high resolution image is created in step S180, it is determined whether or not the user has instructed to end the handheld high resolution mode. When the user instructs the end of the handheld high resolution mode, the camera controller 30 ends the handheld high resolution mode. As a result, the drive of the image sensor by the sensor actuator 22 is stopped. If there is no end instruction, the process returns to step S100.

On the other hand, when the misalignment amount detected in step S140 in step S150 is smaller than the predetermined misalignment amount, in step S200, the sensor actuator 22 changes the blur correction calculation calculated in step S100 in addition to the exposure time. The image sensor 12 is started to be driven by the augmented change amount supplemented with the additional change amount. This makes it possible to acquire a captured image that can be selected as an original image.

In the following step S210, the image sensor 12 acquires a plurality of captured images excluding the first and second captured images. Then, the process proceeds to step S170.

According to the present embodiment as described above, the image pickup apparatus 10 can generate a high resolution image from the captured image acquired in the handheld state.

Although the present disclosure has been described above with reference to the above-described embodiments, the embodiments of the present disclosure are not limited thereto.

For example, in the case of the above-described embodiment, even if the image pickup device 10 is blurred, the position of the image sensor 12 is changed so that the subject is captured in the captured image without being substantially blurred. However, the embodiments of the present disclosure are not limited to this.

FIG. 9 is a block diagram of an imaging device according to another embodiment of the present disclosure.

As shown in FIG. 9, the image pickup device 110 according to another embodiment has the image sensor 12, the optical system 14, the operation unit 16, and the display, similarly to the image pickup device 10 according to the above-described embodiment shown in FIG. It has a unit 18, a storage unit 20, a sensor actuator 22, an angular speed sensor 24, a position sensor 25, and a camera controller 130. In addition to this, the interchangeable lens 114 has a blur correction lens 114g, a blur correction lens actuator (lens drive unit) 114h that drives the blur correction lens 114g so as to change the position, an angular speed sensor 114i, and a blur correction lens 114g. It has a position sensor 114k that detects the position of. Further, the lens controller 114e of the interchangeable lens 114 includes a blur correction lens actuator control unit 114j that controls the blur correction lens actuator 114h.

In the case of the image pickup apparatus 110 according to another embodiment shown in FIG. 9, in the camera shake correction mode and the camera shake high resolution mode, the image of the subject continues to be imaged on substantially the same portion on the light receiving surface of the image sensor 12. As the position of the image sensor 12 is changed, the position of the image stabilization lens 114g is also changed. Therefore, the blur correction unit 134 first calculates the blur amount (blur frequency and amplitude) of the imaging device 110 based on the detection result of the angular velocity sensor 24, and the blur correction unit 134 of the interchangeable lens 114 based on the detection result of the angular velocity sensor 114j. The amount of blur (blur frequency and amplitude) is calculated and output to the sensor actuator control unit 32 and the blur correction lens actuator control unit 114j. The sensor actuator control unit 32 calculates the amount of change for blur correction for deviating the position of the image sensor 12 based on the input blur frequency and amplitude and the current position of the image sensor 12 detected by the position sensor 25. To do. Further, the correction lens actuator control unit 114j deviates the position of the blur correction lens 114g based on the input blur frequency and amplitude and the current position of the blur correction lens 114g detected by the position sensor 114k. Calculate the amount of change for blur correction. Then, while the sensor actuator 22 changes the position of the image sensor 12 by the amount of change for blur correction, the blur correction lens actuator 114h changes the position of the blur correction lens 114g by the amount of change for the second blur correction. Here, the second amount of change in blur correction for the blur correction lens 114 g may be calculated using the signal from the angular velocity sensor 24 on the main body 10a side of the image pickup apparatus 110, or may be provided on the interchangeable lens 114. The calculation may be performed using the signal of the angular velocity sensor 114i. In the former case, a signal corresponding to the second amount of blur correction change is transmitted from the camera controller 130 to the lens controller 114e via the body mounter 10b and the lens mounter 14f.

In the case of this other embodiment, the amount of change for blur correction for the image sensor 12 can be zero. That is, the image of the subject may continue to be imaged on substantially the same portion of the light receiving surface of the image sensor 12 simply by changing the position of the blur correction lens 126. In this case, in the handheld high resolution mode, when the amount of displacement of the subject is smaller than the amount of displacement of the predetermined threshold, the image sensor 12 increases the amount of change by supplementing the amount of change for blur correction to zero. That is, the position is changed by the amount of additional change.

Further, in the case of the image pickup apparatus 10 according to the embodiment shown in FIG. 2 and the image pickup apparatus 110 according to another embodiment shown in FIG. 9, the amount of change for blur correction and the amount of augmentation change are determined by the image sensor 12. The amount of change in position. However, the embodiments of the present disclosure are not limited to this.

Explaining with reference to FIG. 1, in the image pickup apparatus 10 (also in the image pickup apparatus 110), the image sensor 12 is changed with respect to the position in the X-axis direction and the position in the Y-axis direction. In addition to or instead of this, the image sensor 12 may change its rotational posture about the Z axis extending in the extending direction of the optical axis C and orthogonal to the light receiving surface 12a. In this case, the amount of change in the rotational posture of the image sensor 12 is included in the amount of change for blur correction and the amount of augmented change. In addition, an angular velocity sensor is used to detect the amount of fluctuation in the posture of the image pickup device 10.

Furthermore, in the case of the above-described embodiment, as shown in FIGS. 7 and 8, when the amount of displacement of the subject between captured images is smaller than the amount of predetermined threshold displacement in the handheld high resolution mode. The sensor actuator (image sensor drive unit) drives the image sensor with an augmented change amount in which an additional change amount is supplemented with the change amount for blur correction. However, the embodiments of the present disclosure are not limited to this.

For example, in the handheld high-resolution mode, the sensor actuator (image sensor drive unit) constantly supplements the amount of change for blur correction with an additional amount of change regardless of the amount of displacement of the subject between captured images. The image sensor may be driven by the amount.

For example, the user may fix the image pickup device with a tripod or the like during the handheld high resolution mode. In this case, the angular velocity sensor (shake amount detection unit) that detects the shake amount of the image pickup device does not detect the shake amount of the image pickup device, specifically, a predetermined threshold value that can determine that the image pickup device is shaken. The above amount of blur is not detected. Therefore, the amount of change for blur correction of the image sensor in which the blur correction unit is substantially zero and does not change is calculated. As a result, the amount of misalignment of the subject between the captured images can be substantially zero, that is, a plurality of substantially the same captured images are acquired. As a countermeasure, in the handheld high resolution mode, the sensor actuator (image sensor drive unit) constantly replenishes the change amount for blur correction with the additional change amount regardless of the position shift amount of the subject between the captured images. The image sensor is driven by the augmented change amount.

FIG. 10 shows another example of the augmented change amount that changes the position of the image sensor in the X-axis direction.

As shown in FIG. 10, when the image pickup device is fixed by a tripod or the like in the handheld high resolution mode, for example, the amount of augmentation change that changes the position of the image sensor in the X-axis direction is substantially zero and does not change. The amount of change for blur correction is constantly supplemented with the amount of additional change. With such an augmented change amount, a predetermined number of original images required for creating a high-resolution image can be acquired even if the image pickup device is fixed by a tripod or the like in the handheld high-resolution mode or the image pickup device is in the hand-held state. can do.

Instead of this, if the image pickup device is fixed with a tripod or the like in the handheld high resolution mode, that is, if the angular velocity sensor (shake amount detection unit) does not detect a shake amount exceeding a predetermined threshold value, the handheld The mode may be switched from the high resolution mode to the normal high resolution mode.

Further, in the case of the above-described embodiment, in the handheld high resolution mode, for example, as shown in FIGS. 3 and 6, the subjects are displaced from each other in a range of 1.5 pixels × 1.5 pixels and are captured in a predetermined positional relationship. (The subject appears at each of the required positions Np (1) to Np (8).) The additional change amount is supplemented to the blur correction change amount in order to acquire a predetermined number of original images. However, the plurality of original images are not limited to images in which the subjects are displaced from each other in a range of 1.5 pixels × 1.5 pixels and are captured in a predetermined positional relationship.

That is, considering the Bayer arrangement, the original image may be an original image in which the subjects are displaced from each other in a predetermined positional relationship in the range of (1.5 + 2k (k is an integer)) pixel × (1.5 + 2k) pixel.

FIG. 11 is a timing chart showing a change in the position of the image sensor for acquiring a plurality of original images.

As shown in FIG. 11, for example, the first original image in which the subject appears at the first required position Np (1) is acquired. The image sensor 12 moves by (1 + 2k) pixels in the minus direction of the X-axis from the original image acquisition position where the first original image was acquired, so that the subject appears at the required position Np (2). Is obtained. By moving the image sensor 12 by (1 + 2k) pixels in the Y-axis plus direction from the original image acquisition position where the second original image was acquired, the third original image in which the subject appears at the required position Np (3) is displayed. To be acquired. By moving the image sensor 12 by (1 + 2k) pixels in the X-axis plus direction from the original image acquisition position where the third original image was acquired, the fourth original image in which the subject appears at the required position Np (4) is displayed. To be acquired. The subject moves by moving the image sensor 12 by (0.5 + 2k) pixels in the minus direction of the X axis and (0.5 + 2k) pixels in the minus direction of the Y axis from the original image acquisition position where the fourth original image is acquired. The fifth original image captured at the required position Np (5) is acquired. By moving the image sensor 12 by (1 + 2k) pixels in the minus direction of the X axis from the original image acquisition position where the fifth original image was acquired, the sixth original image in which the subject appears at the required position Np (6) is displayed. To be acquired. By moving the image sensor 12 by (1 + 2k) pixels in the Y-axis plus direction from the original image acquisition position that received the 6th original image, the 7th original image in which the subject appears at the required position Np (7) is displayed. To be acquired. Then, the image sensor 12 moves by (1 + 2k) pixels in the X-axis plus direction from the original image acquisition position where the 7th original image is acquired, so that the subject is captured at the required position Np (8). The image is acquired.

That is, in a broad sense, the image pickup apparatus according to the embodiment of the present disclosure is driven by an image sensor that acquires a captured image of a subject and an image sensor that drives the image sensor so that at least one of a position and an attitude changes. A high-resolution image creation unit that creates a high-resolution image from a predetermined number of original images, and a predetermined number of captured images in which the subjects are displaced from each other and are captured in a predetermined positional relationship. A blur detection unit that detects the blur amount of the device, and an operation unit for instructing the user to start execution of the handheld high resolution mode that creates the high resolution image from the captured image acquired when the imaging device is in the handheld state. And, during the execution of the handheld high resolution mode, the image sensor driving unit uses the augmented change amount in which the additional change amount is supplemented to the blur correction change amount calculated based on the blur amount. In the driven state, the high-resolution image creating unit creates the high-resolution image from the captured image acquired by the image sensor.

As described above, a plurality of embodiments have been described as examples of the techniques in the present disclosure. To that end, the accompanying drawings and detailed description are provided.

Therefore, among the components described in the attached drawings and the detailed description, not only the components essential for solving the problem, but also the components not essential for solving the problem in order to illustrate the technique. Can also be included. Therefore, the fact that these non-essential components are described in the accompanying drawings or detailed description should not immediately determine that those non-essential components are essential.

Further, since the above-described embodiment is for exemplifying the technique in the present disclosure, various changes, replacements, additions, omissions, etc. can be made within the scope of claims or the equivalent scope thereof.

The present disclosure is applicable to an imaging device such as a smartphone having a camera shake correction function.

The present disclosure can be applied not only to the high resolution function but also to the real resolution function that improves the resolution and color reproducibility while keeping the number of pixels as it is.

10 Imaging device 12 Image sensor 22 Image sensor drive unit (sensor actuator)
24 Shake amount detector (angular velocity sensor)

Claims (10)

It is an imaging device
An image sensor that acquires a captured image of the subject, and
An image sensor driving unit that drives the image sensor so that at least one of the position and the posture changes,
A high-resolution image creation unit that creates a high-resolution image from a predetermined number of original images by using a predetermined number of captured images in which each subject is displaced from each other and appears in a predetermined positional relationship.
A blur amount detection unit that detects the blur amount of the image pickup apparatus,
It has an operation unit for instructing the user to start execution of the handheld high resolution mode for creating the high resolution image from the captured image acquired when the imaging device is in the handheld state.
During the execution of the handheld high resolution mode, the image sensor driving unit is driving the image sensor with an augmented change amount in which an additional change amount is supplemented to the change amount for blur correction calculated based on the blur amount. An imaging device in which the high-resolution image creating unit creates the high-resolution image from the captured image acquired by the image sensor.
During the execution of the handheld high resolution mode, when the amount of displacement of the subject between captured images is smaller than the amount of displacement of a predetermined threshold, the image sensor driving unit adds an amount of change to the amount of change for blur correction. The imaging device according to claim 1, wherein the image sensor is driven by the supplemented augmented change amount.
2. The image sensor driving unit drives the image sensor with the augmented change amount when the blurring amount is smaller than the predetermined thresholding amount corresponding to the predetermined threshold misalignment amount. The imaging apparatus according to.
The imaging device according to claim 2 or 3, wherein the predetermined threshold misalignment amount is 1 pixel or less.
The imaging device according to claim 1, wherein the additional amount of change is constantly replenished with the amount of change for blur correction during execution of the handheld high-resolution mode.
During execution of the handheld high resolution mode,
The image sensor acquires a plurality of captured images exceeding the predetermined number of images, and
The high resolution image creation unit
Detecting the positional relationship of the subject in each of the plurality of captured images,
Based on the detected positional relationship, the predetermined number of original images are selected from the plurality of captured images.
The imaging device according to any one of claims 1 to 5, which creates the high-resolution image from the selected number of original images.
6. When the high-resolution image creating unit lacks the original image, the shortage original image is created by synthesizing at least two of the plurality of captured images that are not selected as the original image. The imaging apparatus according to.
The operation unit is configured to be able to receive an instruction to start execution of the normal high resolution mode for creating the high resolution image from the captured image acquired when the imaging device is in the fixed state.
During execution of the normal high resolution mode,
While the image sensor driving unit drives the image sensor with an amount of change for acquiring the original image for acquiring each of the predetermined number of original images, the high resolution image creating unit performs the image. The imaging device according to any one of claims 1 to 7, wherein the high-resolution image is created from the captured image acquired by the sensor.
The imaging device according to claim 9, wherein the additional change amount is the change amount for acquiring the original image.
A blur correction lens included in the optical system that forms a subject on the light receiving surface of the image sensor,
It has a lens driving unit that drives the blur correction lens so that at least one of the position and the posture changes.
During execution of the handheld high resolution mode,
The amount of change for blur correction and the amount of change for second blur correction are calculated based on the amount of blur.
The imaging device according to any one of claims 1 to 9, wherein the lens driving unit drives the blur correction lens with the change amount for the second blur correction.

Images