JP2006345147A - Electronic camera with generating function for pixel-shifted image and imaging method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To generate a pixel-shifted image including two-dimensional resolution information in a well-balanced state. <P>SOLUTION: An electronic camera according to the present invention has an imaging section, a direction detection section, a driving section, and an imaging control section. The imaging section photoelectrically converts a subject image formed through a taking lens on an imaging surface to generate image data. The direction detection section detects the movement direction of the subject image. The driving section determines a driving direction according to the detected movement direction so that orthogonal direction components of the movement direction are included and relatively moves the subject image on the imaging surface in the driving direction. The imaging control section performs photoelectric conversion a plurality of times by driving the imaging section as the subject image is relatively moved to acquire a plurality of image data (pixel-shifted image). <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an electronic camera having a pixel shifted image generation function and an imaging method.

Japanese Patent Application Laid-Open Publication No. 2004-228561 discloses a technique for generating a plurality of pixels-shifted images by manually scanning an image input device. In these pixel-shifted images, the pixel sampling positions are shifted little by little. Therefore, fine resolution information exceeding the resolution limit of the image input device can be obtained. Therefore, a high-resolution image exceeding the resolution limit (hereinafter referred to as “super-resolution image”) can be obtained by combining the plurality of pixel-shifted images.
Japanese Patent Laid-Open No. 10-69537

By the way, the above-described conventional technique has a problem that the resolution limit of the super-resolution image differs in the two-dimensional direction due to the direction of movement.
For example, when an image is input while moving in the horizontal direction using an image input device having a vertical and horizontal pixel arrangement, pixel shifting is performed only in the horizontal direction. In this case, the resolution limit is increased in the horizontal direction, but the substantial resolution limit cannot be increased in the vertical direction.

  When the resolution limit is different in the two-dimensional direction in this way, distortion such as the difference in the thickness of the contour line in the image between the vertical and horizontal directions occurs, so that an image with an unnatural impression tends to occur. In addition, since the direction with a low resolution limit seems to be smoothed only in that direction, it tends to be an unnatural image with no direction uniformity.

  In addition, even in the case of horizontal one-dimensional movement, a gentle movement in the vertical direction can be expected. However, especially when the amount of movement is small, there are other problems, such as the need to increase the number of pixels shifted to multiple times in order to obtain further vertical resolution information from this gentle vertical movement. Arise.

  Therefore, an object of the present invention is to generate a pixel-shifted image including resolution information in a two-dimensional direction with a good balance.

<< 1 >> The electronic camera of the present invention includes an imaging unit, a direction detection unit, a drive unit, and an imaging control unit.
The imaging unit photoelectrically converts the subject image formed through the photographing lens on the imaging surface to generate image data.
The direction detection unit detects the movement direction of the subject image.
The driving unit determines a driving direction so as to include a component orthogonal to the moving direction according to the detected moving direction, and relatively moves the subject image in the driving direction with respect to the imaging surface.
The imaging control unit drives the imaging unit with a relative movement of the subject image, performs a plurality of photoelectric conversions, and acquires a plurality of image data (pixel shifted images).

<< 2 >> Also preferably, the direction detection unit detects the direction of movement from the output of a sensor that detects the shake of the electronic camera or the photographing lens.

<< 3 >> Preferably, the direction detection unit detects the movement direction by analyzing the movement of the image data.

<< 4 >> Preferably, a data processing unit is provided that generates a super-resolution image having a resolution higher than the two-dimensional resolution of the imaging unit based on resolution information obtained from a plurality of pixel-shifted images. .

<< 5 >> Preferably, the driving unit moves the subject image by controlling and driving an optical camera shake correction mechanism mounted on the electronic camera or the photographing lens.

<6> Preferably, the optical camera shake correction mechanism causes the subject image to be substantially stationary on the imaging surface by camera shake correction during the exposure period of the imaging surface, and the subject image is placed on the imaging surface between the exposure period and the exposure period. The relative movement is made in the driving direction.

<< 7 >> The imaging method of the present invention is an imaging method that uses an imaging unit that photoelectrically converts an object image formed through a photographic lens on an imaging surface to generate image data, and executes the following operation steps. .
(Step 1) The movement direction of the subject image is detected.
(Step 2) The drive direction is determined so as to include the orthogonal component of the motion direction according to the detected motion direction, and the subject image is moved relative to the imaging surface in the drive direction.
(Step 3) Along with the relative movement of the subject image, the imaging unit is driven to perform photoelectric conversion a plurality of times to obtain a plurality of image data (pixel shifted images).

  In the present invention, the movement direction of the subject image is detected, and the subject image is relatively moved in a direction including a direction orthogonal to the movement direction. As a result, the subject image moves relative to the imaging surface so as to draw a two-dimensional locus. In this state, by taking a plurality of images, it is possible to obtain a pixel-shifted image that includes resolution information in a two-dimensional direction with a good balance.

<< Configuration Description of Embodiment >>
FIG. 1 is a block diagram showing a configuration of the electronic camera 11.
In FIG. 1, a photographing lens 12 is attached to an electronic camera 11. The optical system of the photographing lens 12 is provided with a camera shake correction optical system 12a. The camera shake correction optical system 12a is driven in the biaxial direction by the shake correction unit 12b. By this driving operation, the subject image generated by the photographic lens 12 moves up and down, left and right in a two-dimensional direction. The imaging surface of the imaging device 13 is disposed at the imaging position of the subject image. The imaging element 13 is driven by the imaging control unit 13a and outputs image data. The output image data is processed through the signal processing unit 14, the image memory 14a, and the data processing unit 15. The processed image data is output to the monitor display unit 16 and the compression / decoding unit 17. The monitor display unit 16 performs monitor display of image data. The compression / decoding unit 17 compresses the image data and stores it in the recording medium 18.

In addition, a motion sensor 24 is disposed in the electronic camera 11 or the photographing lens 12, and detects biaxial movement (angular velocity, angular acceleration, acceleration, etc.). Further, the electronic camera 11 is provided with an operation member 25 including a release button, a menu button, and the like.
The electronic camera 11 is provided with a control unit 21. The control unit 21 processes signals from the motion sensor 24 and the operation member 25 and controls operations of the shake correction unit 12b, the imaging control unit 13a, the data processing unit 15, and the like.

<< Description of Operation of Embodiment >>
The electronic camera 11 described above has a super-resolution shooting mode selectable in addition to the normal shooting mode.
FIG. 2 is a flowchart for explaining the super-resolution photographing mode. Hereinafter, the operation in the super-resolution imaging mode will be described along the step numbers shown in FIG.

Step S11: The control unit 21 waits for a release operation from the user. When the control unit 21 detects a release operation through the operation member 25, the control unit 21 proceeds to step S12.

Step S12: The control unit 21 instructs the imaging control unit 13a to start continuous shooting (continuous shooting). The imaging control unit 13a exposes a subject image on the imaging surface of the imaging element 13 by performing aperture control in the imaging lens 12, opening / closing control of a mechanical shutter (not shown), and electronic shutter control.
Note that during this exposure period, the shake correction unit 12b performs shake correction control using the camera shake correction optical system 12a so that the subject image remains at the image position at the time of exposure (the image position that has been displaced in step S15). Is preferred.
After the exposure period ends, the imaging control unit 13a outputs a readout pulse to the imaging element 13 and reads out the generated image data. The image data is subjected to processing such as digitization via the signal processing unit 14, and then stored in the image memory 14a.
The imaging control unit 13a repeats this imaging operation at a predetermined continuous shooting speed (frame rate) for a period up to step S18 described later. By this continuous shooting operation, a plurality of image data is accumulated in the image memory 14a.

Step S13: Based on the biaxial angular acceleration output from the motion sensor 24, the control unit 21 detects the movement direction of the subject image with respect to the imaging surface. Note that the control unit 21 may detect this movement direction by analyzing the motion of image data that is continuously shot (block matching processing or the like).
The movement direction here may be an instantaneous movement direction or an average movement direction obtained by taking a moving average of the instantaneous movement directions. Alternatively, the trajectory of the past subject image may be extended to obtain the motion direction of the subject image at the next exposure by prediction.

Step S14: The control unit 21 determines the driving direction of the subject image so as to include an orthogonal direction component with respect to the movement direction obtained in step S13. For example, the direction orthogonal to the movement direction may be used as the driving direction as it is. Further, for example, a direction close to a direction orthogonal to the movement direction may be selected from preset directions such as an up / down direction, a left / right direction, a right diagonal direction, and a left diagonal direction.

Step S15: The control unit 21 drives the camera shake correction optical system 12a via the shake correction unit 12b to shake the subject image in the driving direction. By this operation, the subject image is displaced not only in the moving direction but also in the orthogonal direction. The displacement width may be equal to or less than the pixel interval or may exceed the pixel interval. Specifically, the pixel positions may be set so as not to overlap in a plurality of pixel shifted images.
The operation of shaking the subject image (the operation of displacing the subject image on the imaging surface) is performed in a period (period from the completion of exposure to the start of the next exposure) excluding the exposure period described in step S12. This is preferable from the viewpoint of preventing imaging blur.
Further, the driving of the subject image may be a reciprocating motion (vibration motion, for example, vertical vibration motion). Further, the movement may be driven only in one direction (for example, only upward).

Step S16: When the controller 21 detects release of the release operation, the operation proceeds to step S18. On the other hand, when the release operation is continued, the control unit 21 shifts the operation to step S17.

Step S17: If the scheduled number of continuous shots is custom-set in advance in the electronic camera 11, the control unit 21 determines whether or not the scheduled number of images has been captured.
If the planned number is not reached, the controller 21 returns the operation to step S12.
On the other hand, when the planned number is reached, the control unit 21 shifts the operation to step S18.

Step S18: The control unit 21 instructs the imaging control unit 13a to end the continuous shooting operation. Through the series of operations described above, a plurality of pixel-shifted images are accumulated in the image memory 14a.
Incidentally, in order to obtain the resolution information in a two-dimensional direction with good balance, it is preferable that the image positions at the time of imaging are not aligned on all three or more pixel-shifted images. Note that if the imaging operation is performed three or more times along with the movement of the image of the two-dimensional locus, all of the image positions will not line up unless a coincidence occurs.

Step S19: The control unit 21 instructs the data processing unit 15 to perform image composition. The data processing unit 15 reads out a plurality of pixel-shifted images in the image memory 14a, and performs alignment with finer accuracy than the pixel interval of the image sensor 13 so that the positions of the pattern of the subject image coincide.
This alignment may be performed by image analysis processing such as image matching. In this case, it is preferable to realize alignment accuracy finer than the pixel interval of the image sensor 13 by performing fitting so that each result of alignment is placed on the movement trajectory of the known imaging position. Further, by performing image matching after interpolating the pixel-shifted image in advance, a finer alignment accuracy than the pixel interval may be realized.
In addition, for each pixel-shifted image, the image position of the subject image (such as the combination of the movement direction and the drive direction) during the exposure period may be obtained, and the pixel-shifted image may be aligned according to this image position.

Step S20: By aligning a plurality of pixel-shifted images, a large number of pixels finely dispersed in the two-dimensional direction as shown in FIGS. 4, 6, and 7 can be obtained. The data processing unit 15 resamples (interpolates) these finely dispersed pixels to obtain, for example, a super-resolution image in which the sample point structure is formed into a square lattice. The pixel density for resampling here is preferably set to a value close to the average pixel density of finely dispersed pixels. In addition, as an interpolation process at the time of resampling, a well-known interpolation process (For example, a bilinear method, a bicubic method, a nearest neighbor method, a linear interpolation method etc.) can be used.

  With the series of processes described above, it is possible to generate a super-resolution image that exceeds the imaging limit of the imaging device 13 and has an excellent balance of resolution in the two-dimensional direction.

<< Effects of the embodiment >>
Hereinafter, the effect of this embodiment will be described with reference to FIGS. 3 to 7, the circled numbers shown in the drawings indicate the frame numbers of the pixel shifted images. Further, each position of a circled number indicates a pixel position of these pixel shifted images.

  First, FIG. 3 shows a case where image driving in the vertical direction (step S15) is not performed in a situation in which the subject image is moving substantially in the horizontal direction of the screen. In this case, the pixel density is high in the horizontal direction of the screen, and fine resolution information can be obtained. However, since the pixel density hardly increases in the vertical direction of the screen, the resolution ability is not improved. Therefore, from these pixel-shifted images, only unequal resolution information can be obtained in the vertical and horizontal directions, and a super-resolution image with an unnatural resolution is created.

  On the other hand, FIG. 4 shows a case where image driving in the vertical direction (step S15) is performed in the same situation as FIG. In this case, pixel positions are dispersed in a two-dimensional direction, and a pixel-shifted image including resolution information in a two-dimensional direction with good balance can be obtained. As a result, it is possible to obtain a super-resolution image with almost uniform resolution in the two-dimensional direction.

  FIG. 5 shows a case where driving in the driving direction (step S15) is not performed in a situation where the subject image is moving in an obliquely downward right direction. In this case, the pixel density is high in the lower right direction, and fine resolution information can be obtained. However, the pixel density does not increase in the orthogonal upward direction, and the resolution is not improved. Therefore, only two-dimensionally non-uniform resolution information can be obtained from these pixel-shifted images, and a super-resolution image with an unnatural resolution is created.

  On the other hand, FIG. 6 shows a case in which driving in the upward driving direction (step S15) is performed in the same situation as FIG. In this case, pixel positions are dispersed in a two-dimensional direction, and a pixel-shifted image including resolution information in a two-dimensional direction with good balance can be obtained. As a result, it is possible to obtain a super-resolution image with almost uniform resolution in the two-dimensional direction.

  FIG. 7 shows a case where the movement of the subject image is faster than the continuous shooting speed and the pixel shift is performed beyond the pixel interval of the image sensor 13 in the same situation as FIG. Also in this case, with respect to the overlapping range of the plurality of pixel shifted images, a pixel shifted image including resolution information in a two-dimensional direction with a good balance can be obtained. As a result, it is possible to obtain a super-resolution image with almost uniform resolution in the two-dimensional direction.

  Further, in the present embodiment, the subject image is moved using the camera shake correction optical system of the photographing lens 12. Therefore, it is not necessary to newly provide another mechanism for moving the image. Therefore, it is possible to easily realize the super-resolution image imaging function.

  In the present embodiment, the movement direction of the subject image is detected using the movement sensor 24. The motion sensor 24 is also preferably a motion sensor mounted for correcting camera shake. In this case, it is not necessary to newly provide another sensor for detecting the moving direction of the subject image with respect to the imaging surface, and the super-resolution image imaging function can be easily realized.

  Further, in the present embodiment, camera shake correction is performed during the exposure period of the image sensor 13, and relative movement of the subject image is realized during the period excluding the exposure period. As a result, image blurring with respect to the pixel shifted image can be reduced, and a clearer pixel shifted image can be obtained. Therefore, the sharpness of the super-resolution image can be further enhanced.

  In the present embodiment, various super-resolution images are generated by combining the movement of the subject image due to subject movement, camera shake, or manual scanning (manual movement of the imaging range) and image movement in the orthogonal direction. be able to. For example, by expanding the scanning range of manual scanning, it is possible to generate a super-resolution image of a wide screen or a panoramic screen.

<< Additional items of embodiment >>
In the above-described embodiment, the subject image on the imaging surface is optically moved. However, the embodiment is not limited to this. For example, the relative movement of the subject image relative to the imaging surface may be realized by driving the imaging surface of the imaging element 13 in the surface direction.

  Furthermore, in the above-described embodiment, the example in which the super-resolution image is generated from the image obtained by the continuous shooting operation has been described. However, a super-resolution image can be generated from the moving image obtained by the moving image capturing operation by the same method.

  In the above-described embodiment, the movement direction of the subject image is detected, and the subject image is moved in the driving direction including the orthogonal direction, thereby realizing a two-dimensional locus-like image movement. However, the embodiment is not limited to this. For example, the subject image may be moved relative to the imaging surface in a two-dimensional locus. In this case, in addition to the wave-like locus as described above, a relatively free locus as shown in FIGS. 8A and 8B can be set. Such an operation makes it possible to obtain a pixel-shifted image that two-dimensionally includes resolution information in a well-balanced manner even when there is no movement such as camera shake or manual scanning in the subject image.

  In the above-described embodiment, the electronic camera 11 generates a super resolution image. However, the embodiment is not limited to this. For example, on the electronic camera 11 side, a plurality of pixel-shifted images may be generated and a super-resolution image may be generated by post-processing such as a computer.

  As described above, the present invention is a technique that can be used for an electronic camera that generates a plurality of pixel-shifted images.

2 is a block diagram illustrating a configuration of an electronic camera 11. FIG. 4 is a flowchart illustrating a super-resolution shooting mode of the electronic camera 11. It is a figure explaining the pixel shift state when not driving an image in the driving direction. It is a figure explaining the pixel shift state at the time of image driving in the drive direction. It is a figure explaining the pixel shift state when not driving an image in the driving direction. It is a figure explaining the pixel shift state at the time of image driving in the drive direction. It is a figure explaining the pixel shift state when a subject image moves quickly. It is explanatory drawing at the time of performing pixel shift in the shape of a two-dimensional locus.

Explanation of symbols

DESCRIPTION OF SYMBOLS 11 ... Electronic camera, 12 ... Shooting lens, 12a ... Camera shake correction optical system, 12b ... Shake correction part, 13 ... Imaging device, 13a ... Imaging control part, 14 ... Signal processing part, 14a ... Image memory, 15 ... Data processing part , 16 ... Monitor display part, 21 ... Control part, 24 ... Motion sensor

Claims (7)

An imaging unit that photoelectrically converts an object image formed through the imaging lens on an imaging surface and generates image data;
A direction detection unit for detecting a movement direction of the subject image;
A driving unit that determines a driving direction so as to include an orthogonal direction component of the moving direction according to the detected moving direction, and relatively moves the subject image in the driving direction with respect to the imaging surface;
An imaging control unit that drives the imaging unit to perform a plurality of photoelectric conversions and acquires a plurality of image data (pixel-shifted images) in association with the relative movement of the subject image. An electronic camera characterized by that. The electronic camera according to claim 1,
The said direction detection part is a means to detect the said movement direction from the output of the sensor which detects the shake of the said electronic camera or the said imaging lens. The electronic camera characterized by the above-mentioned. The electronic camera according to claim 1,
The electronic camera according to claim 1, wherein the direction detection unit is means for analyzing the movement of the image data to detect the movement direction. The electronic camera according to any one of claims 1 to 3,
An electronic camera comprising: a data processing unit that generates a super-resolution image having a resolution higher than the two-dimensional resolution of the imaging unit based on resolution information obtained from a plurality of pixel-shifted images. The electronic camera according to any one of claims 1 to 4,
The electronic camera characterized in that the drive unit moves the subject image by controlling and driving an optical camera shake correction mechanism mounted on the electronic camera or the photographing lens. The electronic camera according to claim 5,
The optical camera shake correction mechanism causes the subject image to be substantially stationary on the imaging surface by camera shake correction during an exposure period of the imaging surface, and the subject image is captured between the exposure period and the exposure period. An electronic camera, wherein the electronic camera is moved relative to the driving direction. An imaging method using an imaging unit that photoelectrically converts an object image formed through a photographic lens on an imaging surface to generate image data,
Detecting a movement direction of the subject image;
Determining a driving direction according to the detected moving direction so as to include an orthogonal direction component of the moving direction, and relatively moving the subject image in the driving direction with respect to the imaging surface;
A step of driving the imaging unit to perform a plurality of photoelectric conversions in accordance with relative movement of the subject image, and acquiring a plurality of image data (pixel shifted images). A characteristic imaging method.

Images