JP2000069352A - Method and device for image input - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image input method and an image input device thereof by which an image blur is corrected and white flaws are corrected. SOLUTION: This method is provided with a step, where plural images are photographed timewise and continuously, a step where image data required for the photographed image are stored in a memory, a step where characteristics of the photographed image are extracted based on the image data, a step where the overlapped position of plural images is adjusted based on the extracted characteristics, and a step where the overlapped images are composited to obtain fewer number of images. Thus, the image is formed with proper exposure, regardless of less camera-shake by compositing image data photographed plural number of times with an exposure time shorter than the proper exposure. Moreover, an image having less white flaws is obtained by compositing image data photographed plural number of times with the short exposure time.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image input technique, and more particularly to an image input technique capable of preventing blurring of an image at the time of photographing and reducing noise included in an image signal. The present invention relates to an image input technique capable of combining a panoramic image and the like.

[0002]

2. Description of the Related Art In a digital still camera, which is an image input device, an image may be blurred due to hand shake during photographing or the like. In order to prevent such blurring, a high shutter speed (normally, when the focal length of the lens is f,
It is desirable to perform shooting at 1 / f second or less). However, since the optimum shutter speed is uniquely determined from the relationship between the field luminance and the aperture value, for example, when the field luminance is low, the required exposure amount is obtained even when the aperture value is opened. In some cases, the shutter speed needs to be slower than 1 / f second. In such a case, in order to prevent the camera shake, a measure such as fixing the camera using a tripod or increasing the field brightness using a strobe is often adopted.

[0003]

However, the photographer does not always have a tripod, and in photographing using a lot of strobe light, the battery is quickly consumed. Therefore, it is desired to achieve blur prevention without using a tripod or a strobe.

On the other hand, as a method for preventing blurring in a video camera, for example, an image sensor having a larger number of pixels than an actual output pixel is provided, the movement of the camera is detected by an acceleration sensor or the like, and the image reading position is changed. There is known a method for preventing a large deviation from occurring in an image for each frame.

However, such a method merely changes the cutout position of the image for each frame so that the position of the image is substantially the same on the screen. It can be said that blurring of a single image cannot be prevented.

Further, Japanese Patent Application Laid-Open No. 3-166869 discloses a video camera with an image blur correction function. This publication discloses a camera in which a switch for performing image shake correction and a shutter speed are linked to prevent image shake.

However, the prior art only discloses that the optimum shutter speed is automatically selected in response to the detection of the image blur by the image blur detecting means. There is no disclosure as to how to compensate for the decrease in exposure amount due to the increase. Further, the related art relates to a video camera, not to a still camera.

Further, as one mode of blur correction in a still image, there is known a method of detecting a blur component from an image and strongly applying edge enhancement in a blur direction in image processing. However, information necessary for restoring the original image has already been lost in the image data of the blurred image, and it is difficult to completely restore even if post-processing such as edge enhancement is carefully performed. In some cases, especially when the image is largely blurred, edge enhancement itself may not be possible. The image processing of the digital camera is described in N.I.
Takizawa et al., The development
of 1M-Pixel Digital Still
l camera, IS &T's 1998 PICS
Conference, pp. 64-66 (199
It is explained in 8).

On the other hand, a solid-state image pickup device provided in a digital still camera as an image input device uses a large number of two-dimensionally arranged pixels to form an optical image of a subject formed on the pixels.
It has a function of converting it into a charge amount (electric signal) and outputting it. By the way, since such a pixel has a defect (pixel defect) based on adhesion of dust, a crystal defect, and the like,
Some may not be able to output a normal signal. For such a pixel defect, a signal that is obtained by adding an extra signal component to an output signal that should be output in accordance with the luminance of the subject is output, and a white flaw that makes the image whitish, There is a black flaw that outputs a signal obtained by subtracting a certain signal component from an output signal that should be output in accordance with the luminance, thereby making an image blackish.

[0010] When many pixel defects occur, there is a possibility that the image quality is remarkably deteriorated when reproducing an image picked up using such a solid-state image pickup device. On the other hand, since a solid-state imaging device that has recently been used has at least several hundred thousand pixels, it can be said that it is actually difficult to manufacture a solid-state imaging device having no pixel defect. Therefore, it is required to use a solid-state imaging device on the assumption that a certain amount of pixel defects always exist.

Based on this premise, a digital still camera which has a correction circuit for correcting an electric signal output from a pixel having a pixel defect by post-processing to improve the image quality has already been developed. According to such a digital still camera, in a production process of the digital still camera, a pixel having a pixel defect (defective pixel) of the solid-state imaging device is detected using a pixel defect inspection device, and the position thereof is determined, for example, by the digital still camera. By storing the information as information in a ROM mounted on the device, an output signal from the defective pixel is appropriately corrected at the time of actual imaging.

However, it has been found that the above-mentioned white flaws are pixel defects based on crystal defects, and may increase depending on the use environment of the solid-state imaging device. For example, white flaws tend to increase as the exposure time increases. It is conceivable to correct such image data based on white flaws by image processing. However, since such image processing requires a relatively long processing time, a method for simply correcting white spots is desired.

Further, a so-called panoramic image having a wide width is easy to express a vast landscape or the like, and is sometimes desired even when photographing with a digital still camera. However, unlike an image captured using a silver halide film, an image from a digital still camera is relatively easy to combine. Therefore, it is common practice to join images using software installed in a personal computer to form a panoramic image. However, with this approach, for users without personal computers,
It is generally difficult to synthesize a panoramic image, and there is a need for a method that can more easily capture a panoramic image.

The present invention has been made in view of the above-mentioned problems of the prior art, and provides an image input method and an image input apparatus which can correct image blur, correct white spots, and can easily obtain a panoramic image or the like. The purpose is to provide.

[0015]

In order to achieve the above-mentioned object, an image input method according to the present invention comprises the steps of: taking a plurality of images continuously in time; storing image data of the taken images in a memory; Accumulating the image data, extracting the feature amount of the captured image based on the image data, adjusting the overlapping position of the plurality of images based on the extracted feature amount, Combining into a smaller number of images.

According to the image input method of the present invention, the step of rotating the image pickup means, the step of detecting the rotation angle of the image pickup means, and the step of, when the rotation angle becomes a predetermined angle,
It is characterized by comprising a step of taking a picture, and a step of extracting a feature quantity of an image obtained by the picture taking and performing image composition based on the feature quantity.

[0017]

According to the image input method of the present invention, a step of photographing a plurality of images consecutively in time, a step of storing image data relating to the photographed images in a memory, Extracting a feature amount of the photographed image; adjusting a superposition position of the plurality of images based on the extracted feature amount; and combining the superimposed images into a smaller number of images. Therefore, for example, by synthesizing image data of an image photographed a plurality of times with an exposure time shorter than an appropriate exposure time, it is possible to form an image with an appropriate exposure amount despite less camera shake. it can. Further, in an image taken a plurality of times with a short exposure time, since the positions of the white flaws hardly overlap, the image data relating to the image part where the white flaw occurs is transferred to the same image part where the white flaw does not occur. Correction can be made based on such image data, whereby an image with less white spots can be obtained.

According to the image input method of the present invention, a step of rotating the image pickup means, a step of detecting a rotation angle of the image pickup means, and a step of photographing when the rotation angle reaches a predetermined angle Extracting the feature amount of the image obtained by the shooting, and performing image synthesis based on the feature amount, for example, by rotating the digital still camera by the user, automatically release, The photographed images can be combined, so that a panoramic image or the like can be easily formed without any trouble such as joining the images in post-processing.

[0019]

DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described below with reference to embodiments. FIG. 1 is a block diagram of a digital still camera as an image input device according to an embodiment of the present invention. In FIG. 1, an optical image of a subject passes through a lens 1 and an aperture 2 and forms an image on an image forming surface of an imaging unit 3 (CCD, C-M0S, etc.). The formed optical image is photoelectrically converted and temporarily stored in the memory 4. Such image data is subjected to demosaic processing, color conversion, and gradation conversion processing by the arithmetic unit 5, performs JPEG compression and the like, and is written to a large-capacity memory device 6 such as a memory card or a floppy disk. .

Next, the operation of the digital still camera according to the present embodiment will be described below. First, in response to the photographer operating a release button (not shown), the arithmetic unit 5 first obtains a necessary exposure amount from the field luminance and the aperture value, and then calculates the number of continuous shootings from the focal length of the lens. To determine. Assuming that the focal length of the lens 1 is 35 mm in terms of a 35 mm plate, the shutter speed 1 / f second is a limit value at which a normal user does not cause blurring (hereinafter referred to as a camera shake limit value). .

Here, the focal length of the lens 1 is 50 mm in terms of a 35 mm plate, and the shutter speed for obtaining a necessary exposure amount is 1/30 from the field luminance and the current aperture value.
If it is seconds, the MPU 5 determines that the camera shake limit value is 1 /
Since the time is 50 seconds, it is determined that shooting is performed twice at a shutter speed of 1/60 seconds. In such a case, the same exposure amount can be obtained by performing shooting at a shutter speed of 1/240 sec eight times. However, if the shutter speed is increased and the exposure time is shortened too much, noise increases in the image. However, it may be difficult to extract features described later. Therefore,
It is desirable that the shutter speed be 1/8 or more of the camera shake limit value.

In addition, the digital still camera operates continuously (for example, at 1/60 second intervals) at such a shutter speed.
A picture is taken and the image data is written to the memory. The images captured in this manner are shown in FIGS.
(B). Next, the arithmetic unit 5 detects the feature amount of the image stored in the memory 4. Although various methods are known as this method, it is considered that binarizing image data is relatively easy, and therefore, this embodiment is used in this embodiment.

The arithmetic unit 5 moves the two images little by little on the basis of the binarized image data, and searches for the most overlapping point. The moving amount of the image is generally １ ／ to の of the angle of view.
It is considered within the extent, but it is of course possible to move more. In addition, if the relative movement of the images is only the parallel movement, the processing can be performed quickly. However, when high accuracy is required by superimposition, the images may be relatively rotated. Further, for a moving subject, higher-precision superposition can be achieved by adding enlargement or reduction in addition to parallel and rotational movement.

The arithmetic unit 5 aligns the two images in this way, and averages the image data before binarization in the overlapped image portion. The non-overlapping portion is calculated by doubling the image data value to match the exposure amount.
When the number of images to be combined is large, processing is performed so that the average brightness is uniform. It should be noted that such a process may cause a clear boundary between the overlapping portion and the non-overlapping portion of the image due to, for example, a difference (doubling) in the noise amount. Such unnaturalness can be eliminated or reduced to some extent.

FIG. 3 is a diagram for explaining the weighting process according to the present embodiment. In FIG. 3, an image G1 and an image G
2 do not overlap at the position A, but overlap at the position B. In such a case, the image G1
The weight of the data value of the image G2 is gradually decreased as the position moves from the position A to the position B, and the weight of the data value of the image G2 is gradually increased as the position moves from the position A to the position B. Therefore, by such weighting processing, the overlapped portion and the non-overlapped portion of the image
It can be blurred so that no clear boundaries appear. still,
With the processing according to the present embodiment, image quality may be reduced to some extent in non-overlapping images (periphery of the screen). However, since the main subject is usually located at the center of the screen, there is a major problem. It is not expected to occur.

Since the images formed in this way have different overlapping amounts at the partial portions, several methods can be considered as a method of cutting out (forming one image) as follows. Since each process is well known, the details will not be described below. (1) Trimming from a part where a plurality of images are all overlapped into a single image (however, the number of effective pixels is reduced, and an image of a part that does not overlap is discarded, so efficiency is low). (2) Extract the entire range of the captured image (however, since the image data value of the non-overlapping image portion is simply doubled, noise may increase). (3) Trim the entire or entire range of the synthesized image (effective for panoramic shooting described later).

The above-mentioned image synthesizing process is most suitable after so-called demosaic (color interpolation) processing. If color processing conversion and gradation conversion are performed after the amount of image data is reduced by superimposing images, efficient processing can be achieved. It can be performed. The arithmetic unit 5 records the image data on which the image processing has been performed as described above on the large-capacity recording medium 6.

The arithmetic unit 5 can determine the blur direction and the blur amount when determining the overlapping position of the images, and can store the amounts. The arrows shown in FIG. 2C indicate the blur direction and the blur amount. In this way, if the blur direction and the blur amount are known, the synthesized image (FIG. 2C)
By applying spatial frequency processing such as edge emphasis or blur processing in the blur direction, it is possible to reduce the deviation (blurring) of the apparent image in the frame.

It is desirable that the continuous shooting described above is automatically switched between single shooting in accordance with the brightness of the subject and the focal length of the lens. That is, assuming that the focal length of the lens is fmm in 35 mm conversion, the camera shake limit value 1 / f
When a shutter speed slower than a second is required, the mode may be set to the continuous shooting mode of the present embodiment.
In the case of a zoom lens or interchangeable lens camera,
Information on the focal length may be received from the lens, and continuous shooting and single shooting may be automatically switched based on the information and the brightness of the subject. In the continuous shooting mode, the resolution may decrease due to a slight shift of the image. For this reason, at the time of output, the memory can be used effectively if the number of pixels is adjusted to the reduction in resolution.

On the other hand, if the required shutter speed is n / f seconds, a switch is provided so that the user can arbitrarily set the shutter speed. , N images may be synthesized. As a result, a synthesized image with a better S / N ratio can be obtained. Further, as this application example, long-time exposure can be considered.

At the time of long-time exposure used for photographing night scenes,
In many cases, a digital still camera is fixed on a tripod or the like for photographing, and in that case, there is no risk of camera shake. Therefore,
The camera shake limit value is theoretically infinite. Thereby, the shutter speed is set to an arbitrary value (for example, 1 /
2 seconds). However, white spots due to pixel defects of the image pickup unit 3 tend to be more prominent during long-time exposure. According to the prior art, such white spots are corrected by image processing.

However, even if a pixel cannot be used because of a white defect at a shutter speed of 1/2 second, it may function as a normal pixel at a shutter speed of 1/8 second. Therefore, according to an application example of the present embodiment, a night view or the like is photographed four times at a shutter speed of 1/8 second, for example, so that the four images are synthesized to obtain a necessary exposure amount, and a white defect is obtained. Image can be obtained.

Next, a modified example of this embodiment will be described. FIG. 4 is a diagram illustrating a state in which a digital still camera according to the present modification is taking an image, and FIG. 5 is a diagram illustrating an image obtained by photographing using the digital still camera according to the present modification. . In FIG. 4, the digital still camera 10 includes an angular velocity sensor 11.

The user rotates the digital still camera 10 horizontally at a constant speed while pointing the lens of the digital still camera 10 at the subject. At this time, the angular velocity sensor 11 emits a signal when the rotation angle has reached a predetermined angle, and the release operation is performed accordingly. In this modification, it is assumed that three releases have been made.

Since three images have been obtained from the three releases, the arithmetic unit 5 synthesizes the images in the manner described above. The synthesized image obtained in this manner is a so-called panoramic image having a horizontal width as shown in FIG. In addition, if the digital still camera 10 is rotated in the vertical direction, it becomes possible to photograph a high-rise building without using a wide-angle lens or the like. Further, while the user rotates the digital still camera 10, image data is taken in at predetermined intervals, a feature amount of the image is obtained, and if it is determined that the image data is unnecessary for forming a panoramic image, the image data is discarded. If image data necessary for forming a panoramic image is obtained by repeating the process of taking in new image data and storing the image data and performing image synthesis, It is not necessary to use a simple angular velocity sensor.

According to the above-described embodiment, since one image is synthesized from a plurality of images, there is an advantage that noise is reduced and image quality is improved. Further, even when a dark subject is photographed with a long focus lens, blurring can be prevented. Furthermore, the accuracy of the composition can be improved because the images of one sheet do not become extremely dark. In addition, the calculation can be performed faster by obtaining the feature amount instead of superimposing the images as they are.

According to the present embodiment, all the pixels of the imaging means can be used. Although noise increases in the periphery, it is considered that there is no particular problem because the number of main subjects is small. In addition, the weighting process prevents the noise amount from suddenly changing between a portion having a small overlap and a portion having a large overlap.

According to the present embodiment, the shutter speed is automatically optimized according to the focal length of the lens / brightness of the subject. In addition, although the image quality is likely to be deteriorated if the original pixels are used for the interpolation calculation, the calculation cost and the image quality can be balanced by reducing the number of pixels. Further, the post-processing can be facilitated by obtaining the shift direction and the shift amount of the image at the time of image synthesis.

According to the present embodiment, a slight shift of the image in one frame can be apparently reduced. In addition, the provision of the means capable of changing the shutter speed allows the user to select one of the shooting that can obtain an image with a good S / N ratio and the anti-shake shooting of the present invention.

Although the present invention has been described with reference to the embodiments, it should be understood that the present invention should not be construed as being limited to the above-described embodiments, and that modifications and improvements can be made as appropriate. is there.

[0041]

According to the image input method of the present invention, a step of photographing a plurality of images successively in time, a step of storing image data relating to the photographed images in a memory, Extracting a feature amount of the captured image based on the extracted feature amount, and adjusting a superposition position of the plurality of images based on the extracted feature amount;
Compositing the superimposed images into a smaller number of images, for example, by synthesizing image data of an image photographed a plurality of times with an exposure time shorter than an appropriate exposure time, thereby reducing camera shake. Nevertheless, an image with an appropriate exposure amount can be formed. Further, in an image taken a plurality of times with a short exposure time, since the positions of the white flaws hardly overlap, the image data relating to the image part where the white flaw occurs is transferred to the same image part where the white flaw does not occur. Correction can be made based on such image data, whereby an image with less white spots can be obtained.

According to the image input method of the present invention, a step of rotating the image pickup means, a step of detecting a rotation angle of the image pickup means, and a step of photographing when the rotation angle reaches a predetermined angle Extracting the feature amount of the image obtained by the shooting, and performing image synthesis based on the feature amount, for example, by rotating the digital still camera by the user, automatically release, The photographed images can be combined, so that a panoramic image or the like can be easily formed without any trouble such as joining the images in post-processing.

[Brief description of the drawings]

FIG. 1 is a block diagram of a digital still camera as an image input device according to an embodiment of the present invention.

FIG. 2 is a diagram illustrating image composition.

FIG. 3 is a diagram illustrating a weighting process according to the embodiment;

FIG. 4 is a diagram showing a state in which a digital still camera according to the present modification is taking a picture.

FIG. 5 is a diagram showing an image obtained by photographing with a digital still camera according to the present modification.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Lens 2 Aperture 3 Imaging means 4 Memory 5 Arithmetic unit 6 Large-capacity recording medium 10 Digital still camera 11 Angular velocity sensor G1-G3 Image

Continued on the front page F term (reference) 5B057 CA01 CA08 CA12 CA16 CB01 CB08 CB12 CB16 CD05 CE02 CE03 CE04 CE08 CH08 DA08 DB02 DB06 DB09 DC05 5C022 AA13 AB01 AB12 AB37 AB55 AB62 AB68 AC69 5C023 AA11 AA37 BA11 BA17 CA03 A04A08 A08 AA22 BA06

Claims (17)

[Claims] 1. A photographing step of photographing a plurality of images successively in time, an accumulating step of accumulating image data relating to the photographed images in a memory, and a characteristic of the photographed image based on the image data. An extracting step of extracting an amount, a position adjusting step of adjusting an overlapping position of the plurality of images based on the extracted feature amount, and a combining step of combining the overlapped images into a smaller number of images. An image input method comprising: 2. The image input method according to claim 1, wherein a single exposure time in the photographing is shorter than an exposure time required to obtain an appropriate exposure amount. 3. The image according to claim 1, wherein an exposure time of one time in the photographing is 1/8 or more of an exposure time required to obtain an appropriate exposure amount. input method. 4. The image input device according to claim 1, wherein, in the position adjusting step, the main subjects of the plurality of shot images are adjusted so as to overlap each other based on the characteristics of the images. Method. 5. The image input method according to claim 4, wherein, in the position adjusting step, a plurality of images are relatively moved in parallel so as to adjust so that a main subject of each image overlaps. 6. The image input device according to claim 4, wherein in the position adjusting step, a plurality of images are relatively rotated and moved to adjust the main subjects of each image so as to overlap each other. Method. 7. The method according to claim 4, wherein, in the position adjusting step, a plurality of images are relatively enlarged or reduced so that the main subjects of each image are adjusted so as to overlap each other. Image input method described. 8. The image input method according to claim 1, wherein, in the synthesizing step, the number of overlapping images is adjusted so as to have approximately the same brightness at different places. 9. The image input method according to claim 1, wherein in the synthesizing step, weighting is adjusted so that a portion where the number of overlapping images changes gradually changes. 10. The image input method according to claim 1, wherein a portion where the images overlap is used as a final output. 11. The method according to claim 1, further comprising the step of selecting one of one-time shooting and a plurality of temporally continuous shootings according to at least one of a focal length of a lens and brightness of a subject. The image input method according to claim 1. 12. The image input method according to claim 1, wherein photographing is performed using an image pickup device having a relatively low number of pixels. 13. The image input method according to claim 1, wherein information relating to a shift in a plurality of images is detected. 14. The image input method according to claim 13, wherein edge emphasis is applied in the shift direction in the composite image. 15. The method according to claim 1, further comprising a step of controlling an exposure time during the photographing.
15. The image input method according to any one of claims 1 to 14. 16. A step of rotating an image pickup unit, a step of detecting a rotation angle of the image pickup unit, a step of taking an image when the rotation angle reaches a predetermined angle, and an image obtained by the image pickup Extracting the characteristic amount of the image and performing image synthesis based on the characteristic amount. 17. An image input apparatus for performing the image input method according to claim 1.