JP4469309B2 - Camera shake correction method, camera shake correction apparatus, and imaging apparatus - Google Patents

Description

  The present invention relates to a camera shake correction method, and more particularly to a technique for correcting camera shake of a digital still camera and a portable information terminal device.

In recent years, digital still cameras have been increased in size and size. Along with this, the effect on image quality due to camera shake is increasing. Therefore, as a method of correcting the camera shake, there is a method of mechanically shifting the lens and the CCD. However, this method causes an increase in cost, and a new mechanism is provided for correcting the camera shake, thereby preventing a reduction in size. ing.
As a conventional technique, Patent Document 1 discloses that a plurality of images are continuously shot, a shift vector of digital image data corresponding to each shot image is detected, and at least one image is based on the detected shift vector. A method of correcting camera shake by correcting data is disclosed.
Further, as other conventional techniques, Patent Documents 2, 3, and 4 continuously photograph a plurality of images, detect a deviation vector of digital image data corresponding to each photographed image, and correct the detected deviation vector. However, a method for correcting camera shake by combining the images is disclosed.
Patent Document 5 discloses a method of generating a PSF based on shake trajectory data and performing camera shake correction.
JP 2002-247444 A JP 2004-266648 A JP 2000-341577 A JP 2003-087647 A Japanese Patent Application Laid-Open No. 11-134481

However, in the conventional technique disclosed in Patent Document 1, since the influence on image quality due to camera shake is suppressed only by shortening the image acquisition time, the correction error cannot be sufficiently suppressed.
In addition, in the conventional techniques disclosed in Patent Documents 2 to 4, camera shake cannot be suppressed for each image data.
In addition, the conventional technique disclosed in Patent Document 5 needs to acquire blur locus data, and thus has a problem of preventing miniaturization, and the influence of the estimated PSF error cannot be sufficiently suppressed.
In view of such problems, the present invention calculates and corrects a PSF based on a deviation vector, thereby further reducing the influence on image quality due to camera shake, and reducing the cost and reducing the size of the camera shake correction method. It is another object of the present invention to provide an imaging device and a portable information terminal device using the same.

In order to solve the above-described problems, the present invention provides an acquisition procedure for acquiring a plurality of pieces of image data captured continuously, and a reference detected as reference data from the image data acquired by the acquisition procedure. A data detection procedure, a shift vector detection procedure for detecting a shift vector between the reference data detected by the reference data detection procedure and other image data, and a point spread based on the shift vector detected by the shift vector detection procedure PSF estimation procedure for estimating a function, correction procedure for correcting each image data by a point spread function estimated by the PSF estimation procedure, and a synthesis procedure for combining the image data corrected by the correction procedure and the reference data It is characterized by having provided.
In the camera shake correction procedure according to the present invention, a plurality of pieces of image data are acquired by the acquisition procedure, and image data with the least camera shake among the image data acquired by the reference data detection procedure is used as reference data. Then, a deviation vector between the reference data and the other image data is detected by a deviation vector detection procedure, and PSF estimation is performed based on the deviation vector detected by the PSF estimation procedure. Next, each image data is corrected by the PSF estimated by the correction procedure. Finally, the reference data and the corrected image data are synthesized by the synthesis procedure. Note that PSF is a point spread function, which is a function representing how light emitted from one point spreads.
According to a second aspect of the present invention, in the shift vector detection procedure, image data to be used for camera shake correction processing is selected based on the acquired information.
According to a third aspect of the present invention, the deviation vector detection procedure is based on a deviation vector between the target image data acquired by the acquisition procedure and the image data acquired immediately before the target image data, or a maximum value of a cross-correlation. And selecting image data to be used for camera shake correction processing.
As a calculation method of the shift vector detection procedure of the present invention, there is a cross-correlation degree or the like. When detecting by the cross-correlation degree, the difference in position of the maximum value of the cross-correlation degree between the reference data and the other image data is set as a shift vector. By selecting the image data to be used for camera shake correction processing based on the deviation vector between the target image data and the previously acquired image data and the maximum value of the cross-correlation, the effect of the image with large camera shake Can be suppressed.

According to a fourth aspect of the present invention, in the reference data detection procedure, image data with the smallest camera shake is detected as reference data among the image data acquired by the acquisition procedure.
It is desirable to use the image data with the smallest camera shake as the reference data from the acquired image data. As a result, the shift vector can be detected with higher accuracy.
According to a fifth aspect of the present invention, the reference data detection procedure detects image data having the highest contrast among the image data acquired by the acquisition procedure as reference data.
As a method for detecting the image data with the smallest camera shake, the image data with the highest contrast is made the image data with the smallest camera shake. The reason for this is that it can be considered that the image data with larger camera shake lowers the contrast. As a result, the shift vector can be detected with higher accuracy.
According to a sixth aspect of the present invention, in the shift vector detection procedure, a shift vector is detected for a partial area included in the image data.
Since the speed can be increased by limiting the detection range by the shift vector detection procedure, it is preferable to detect the shift vector for a part of the region included in the image data.
A seventh aspect of the present invention is characterized in that the deviation vector detection procedure and the correction procedure perform coordinate transformation of the image data.
By performing the coordinate conversion of the image data, the shift vector is detected with higher accuracy, and better PSF estimation is possible. There is a possibility that a change in tilt due to camera shake of image data can be corrected and super-resolution can be realized.

An eighth aspect of the present invention includes the camera shake correction method according to any one of the first to seventh aspects.
By providing the camera shake correction method of the present invention in the camera shake correction device, it is possible to realize low cost and downsizing of the device.
According to a ninth aspect of the present invention, when the camera shake correction mode is set, the process of the camera shake correction method is performed.
The camera shake correction apparatus according to the present invention includes a camera shake correction mode in addition to the normal mode, and performs processing of the camera shake correction method when the mode is set.
According to a tenth aspect of the present invention, the camera shake correction apparatus according to the eighth or ninth aspect is provided.
By providing the digital still camera and the portable information terminal device with the camera shake correction device of the present invention, it is possible to perform excellent camera shake correction despite the small size and low cost.

According to the first aspect of the present invention, camera shake is increased by 1 / n times by shortening the image acquisition time for each of a plurality of sheets. Furthermore, since the PSF is estimated based on the shift vector, it is possible to correct camera shake better.
According to the second and third aspects of the present invention, it is possible to suppress the influence of image data with a large amount of camera shake, and to correct camera shake more satisfactorily.
According to the fourth and fifth aspects of the present invention, camera shake can be corrected more satisfactorily by improving the deviation vector detection accuracy.
According to the invention of claim 6, faster correction can be performed by making the detection region a part.
According to the seventh aspect of the invention, there is a possibility that camera shake can be further corrected and super-resolution can be realized by detecting the shift vector with higher accuracy and correcting the tilt component.
According to the eighth and ninth aspects of the present invention, in the camera shake correction apparatus, it is possible to correct camera shake better, to realize cost reduction and downsizing, and to select whether or not to perform camera shake correction processing. Can do.
According to the invention of claim 10, in a digital still camera or a portable information terminal device having a camera shake correction device, camera shake can be corrected further, cost reduction and size reduction can be realized.

の関係があるとする。ここで、ｈ(x、y、x’、y’)はＰＳＦ、ｎ(x、y)はランダムノイズである。ＰＳＦとは点広がり関数（Point Spread Function）であり、一点から出た光がどのように広がるかを表す。ここで、ＰＳＦが場所に依存しないと考えると、

となる。手ぶれは、シフトによる影響が大きいと考えられ、場所への依存度が小さいと言える。
本発明は、連続して撮影された複数枚の画像データを取得し、取得された画像データのうちの１枚を参照データとして検出し、参照データとその他の画像データとのずれベクトルを検出し、検出されたずれベクトルを基にＰＳＦを推定し、推定したＰＳＦにより画像データ各々を補正し、参照データと補正された画像データを合成することにより手ぶれを補正する方法である。 Hereinafter, the present invention will be described in detail with reference to embodiments shown in the drawings. However, the components, types, combinations, shapes, relative arrangements, and the like described in this embodiment are merely illustrative examples and not intended to limit the scope of the present invention only unless otherwise specified. .
First, the PSF used in the camera shake correction method of the present invention will be described below.
When f (x, y) is ideal image data and g (x, y) is acquired image data,

Suppose there is a relationship. Here, h (x, y, x ′, y ′) is PSF, and n (x, y) is random noise. PSF is a point spread function, which represents how light emitted from one point spreads. Here, considering that PSF is location independent,

It becomes. Camera shake is considered to be greatly affected by shifts, and it can be said that the degree of dependence on the location is small.
The present invention acquires a plurality of image data continuously shot, detects one of the acquired image data as reference data, and detects a shift vector between the reference data and other image data. In this method, PSF is estimated based on the detected shift vector, each image data is corrected by the estimated PSF, and the camera shake is corrected by combining the reference data and the corrected image data.

In the conventional Patent Documents 2, 3, and 4, camera shake for each image data cannot be suppressed. However, in the camera shake correction method according to the present invention, PSF is estimated and correction is performed on each image data, so that the influence of camera shake can be suppressed. In addition, it is considered that the influence of the PSF error estimated by combining can be suppressed.
FIG. 1 is a diagram showing a difference between PSF estimation from one image and PSF estimation in the image stabilization method according to the present invention. FIG. 1A is a diagram based on PSF estimation from one image. Since PSF estimation from one image is complicated, estimation is difficult. However, in the camera shake correction method according to the present invention, the PSF is estimated from each of a plurality of acquired image data, and therefore the PSF in the range indicated by the arrow 1 in FIG. Therefore, the estimation is easy because the camera shake is small. Further, since the PSF is estimated from the deviation vector, it is not necessary to provide a new mechanism.

FIG. 2 is a flowchart showing an example of the procedure of the camera shake correction method according to the present invention. An image stabilization method of the present invention includes an acquisition procedure for acquiring a plurality of image data captured continuously and a reference data detection procedure for detecting one of the image data acquired by the acquisition means as reference data. A shift vector detection procedure for detecting a shift vector between the reference data detected by the reference data detection procedure and other image data, and a point spread function is estimated based on the shift vector detected by the shift vector detection procedure A PSF estimation procedure, a correction procedure for correcting each image data by a point spread function estimated by the PSF estimation procedure, and a synthesis procedure for synthesizing the image data corrected by the correction procedure and the reference data. The
First, in the acquisition procedure, a plurality of pieces of image data taken continuously are acquired (S1). When shooting n images, the image acquisition time is 1 / n times and the sensitivity is n times that when shooting a single image. Therefore, one hand shake in a plurality of sheets is 1 / n times, and noise is n times.
Next, in the reference data detection procedure, image data with the smallest camera shake is detected as reference data from the acquired image data (S2). As the method, the image data with the highest contrast is made the image data with the smallest camera shake. In other words, it can be considered that the image data with greater hand movement has lower contrast. As a result, the shift vector can be detected with higher accuracy. Further, by selecting image data to be used for the camera shake correction process based on the contrast, it is possible to suppress the influence of an image with a large camera shake.
Next, in the shift vector detection procedure, a shift vector between the reference data and other image data is detected (S3). As a calculation method, there is a cross-correlation degree or the like. When detecting by the cross-correlation degree, the difference in position of the maximum value of the cross-correlation degree between the reference data and the other image data is set as a shift vector. By selecting the image data to be used for camera shake correction processing based on the deviation vector between the target image data and the previously acquired image data and the maximum value of the cross-correlation, the effect of the image with large camera shake Can be suppressed. Further, the speed can be increased by limiting the detection range.

は求めたい対象の前に撮影された軌跡と参照データからのずれベクトルで、

は求めたい対象の軌跡と参照データからのずれベクトルである。検出される参照データからのずれベクトルは軌跡の中心であると考えられることから、求めたい対象の軌跡は、

・・・・・・（１）
となる。このように求めた軌跡よりＰＳＦを算出する。他にも、ずれベクトルを基にＰＳＦを推定するすべての方法が適用可能である。
次にずれベクトル補正手順では、推定したＰＳＦにより画像データ各々を補正する（Ｓ５）。例えば、推定したＰＳＦにより画像データ各々を擬似ウィナーフィルタで補正することを行う。擬似ウィナーフィルタの式は以下で与えられる。

・・・・・・（２）

は推定したＰＳＦの空間周波数、

は推定したＰＳＦの空間周波数の複素共役、

は定数である。式（２）と画像データの空間周波数の積をフーリエ変換すると補正される。他にも、推定したＰＳＦで補正されるすべての方法が適用可能である。
最後に合成手順では、参照データと補正された画像データを合成する（Ｓ６）。合成する方法として、平均化処理が挙げられる。合成することにより、ＰＳＦ推定誤差による影響を抑えることができる。
以上のように画像データの座標変換を行うことにより、より精度良くずれベクトル検出がされ、より良いＰＳＦ推定が可能となる。画像データの手ぶれによる傾き変化に対して補正することや、超解像も期待できる。また、カラーの場合はＲＧＢそれぞれにおいて補正処理を行い、それらの情報を統合して手ぶれ補正を行うこともできる。 Next, in the PSF estimation procedure, the PSF is estimated based on the detected deviation vector (S4). For example, if it is assumed that the image acquisition interval is 0, the image acquisition time is sufficiently short, and the camera shake is constant and linear, the PSF can be estimated by the method shown in FIG. That is, according to FIG.

Is the deviation vector from the trajectory and the reference data taken before the object you want to find,

Is a deviation vector from the trajectory of the target to be obtained and the reference data. Since the deviation vector from the detected reference data is considered to be the center of the trajectory, the trajectory of the target to be obtained is

(1)
It becomes. The PSF is calculated from the trajectory thus obtained. In addition, all methods for estimating the PSF based on the shift vector are applicable.
Next, in the shift vector correction procedure, each image data is corrected by the estimated PSF (S5). For example, each image data is corrected by a pseudo Wiener filter based on the estimated PSF. The formula for the pseudo Wiener filter is given below.

(2)

Is the estimated spatial frequency of the PSF,

Is the complex conjugate of the estimated PSF spatial frequency,

Is a constant. Correction is performed by Fourier transform of the product of the expression (2) and the spatial frequency of the image data. In addition, all methods corrected by the estimated PSF are applicable.
Finally, in the synthesis procedure, the reference data and the corrected image data are synthesized (S6). An averaging process is an example of the synthesis method. By synthesizing, the influence of the PSF estimation error can be suppressed.
By performing the coordinate conversion of the image data as described above, the shift vector is detected with higher accuracy, and better PSF estimation is possible. It can be expected to correct the tilt change due to camera shake of the image data and to achieve super-resolution. In the case of color, correction processing can be performed for each of RGB, and the information can be integrated to perform camera shake correction.

FIG. 4 is a diagram illustrating an example of a camera shake correction apparatus according to an embodiment of the present invention. The camera shake correction apparatus 100 includes an imaging device 10 that acquires a plurality of pieces of image data captured continuously, a buffer memory 14 that temporarily stores the image data acquired by the imaging device 10, and one of the image data. A reference data detector 11 that detects a sheet as reference data, a shift vector detector 12 that detects a shift vector between the reference data detected by the reference data detector 11 and other image data, and a shift vector detector 12. A PSF estimator 13 that estimates a point spread function based on the detected shift vector, a corrector 15 that corrects each image data by the point spread function estimated by the PSF estimator 13, and an image corrected by the corrector 15 It comprises a synthesizer 16 that synthesizes data and reference data, and a memory 17 that stores the synthesized image.
FIG. 5 is an example of a digital still camera having a camera shake correction device. FIG. 5A is a front view, and FIG. 5B is a rear view. The digital still camera 110 includes a camera body 50, a lens 51, a viewfinder 52 for looking into a subject, a flash 53 for flashing the subject, a shutter button 54, a power switch 55, a camera shake correction processing selection button 56, an LCD, and the like. Display 57, a menu selection button 58, and a memory slot 59 into which a memory for storing images is inserted. Since the operation of the digital still camera is known, the description thereof is omitted here.

FIG. 6 is a diagram showing an embodiment from the deviation vector detection procedure to the correction procedure of the present invention. r ′ and d ′ are deviation vectors from the trajectory photographed before the object to be obtained and the reference data, and r and d are deviation vectors from the object locus to be obtained and the reference data.
When the ideal image data (a), the reference image data (b), the image data (c) photographed before the target, and the target image data (d) are in the situation shown in the figure, the reference image data ( The maximum value of the cross-correlation between the image data (c) photographed before b) and the object is the first row and the third column (reference numeral 20), and the reference image data (b) and the object image data (d). Since the maximum value of the degree of cross-correlation is the second row and third column (reference numeral 21), deviation vectors d ′ and d from the reference data are obtained. From the PSF of the image data (c) photographed before the object and the deviation vectors d ′ and d from the reference image data (b), the locus r of the object to be obtained recursively is obtained. The PSF is estimated from the locus r and the cross-correlation degree distribution, and the target image data is corrected. In this embodiment, the corrected image data (g) is ideal image data (a). By combining, it is possible to suppress the influence of PSF estimation error, noise, and the like.

It is a figure which shows the difference by PSF estimation in the PSF estimation from one image, and the camera-shake correction method by this invention. It is a flowchart which shows an example of the procedure of the camera-shake correction method by this invention. It is a figure explaining the method of estimating PSF of the present invention. It is a figure which shows an example of the camera-shake correction apparatus concerning one Embodiment of this invention. It is a figure which shows an example of the digital still camera which has a camera shake correction apparatus. It is a figure explaining one Example from the shift vector detection procedure of this invention to a correction procedure.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 10 Imaging device 11 Reference data detector 12 Vector detector 13 PSF estimator 14 Buffer memory 15 Corrector 16 Synthesizer 17 Memory 50 Camera body 51 Lens 52 Finder 53 Flash 54 Shutter Button, 55 power switch, 56 camera shake correction processing selection button, 57 display, 58 menu selection button, 59 memory slot, 100 camera shake correction device, 110 digital still camera

Claims (10)

  An acquisition procedure for acquiring a plurality of image data captured continuously, a reference data detection procedure for detecting as reference data from the image data acquired by the acquisition procedure, and reference data detected by the reference data detection procedure Vector detection procedure for detecting a deviation vector between the image and other image data, a PSF estimation procedure for estimating a point spread function based on the deviation vector detected by the deviation vector detection procedure, and estimation by the PSF estimation procedure A camera shake correction method comprising: a correction procedure for correcting each image data by a point spread function; and a synthesis procedure for combining the image data corrected by the correction procedure and the reference data. The camera shake correction method according to claim 1, wherein the shift vector detection procedure selects image data used for camera shake correction processing based on the acquired information.   The deviation vector detection procedure is a camera shake correction process based on the deviation vector between the target image data acquired by the acquisition procedure and the image data acquired immediately before the target image data and the maximum value of the cross-correlation. The image stabilization method according to claim 2, wherein image data to be used is selected.   The camera shake correction method according to claim 1, wherein the reference data detection procedure detects, as reference data, image data having the smallest camera shake among the image data acquired by the acquisition procedure.   5. The camera shake correction method according to claim 4, wherein the reference data detection procedure detects image data having the highest contrast among the image data acquired by the acquisition procedure as reference data.   6. The camera shake correction method according to claim 1, wherein the shift vector detection procedure detects a shift vector for a part of a region included in the image data.   The camera shake correction method according to claim 1, wherein the shift vector detection procedure and the correction procedure perform coordinate conversion of the image data.   A camera shake correction apparatus comprising the camera shake correction method according to claim 1.   The camera shake correction apparatus according to claim 8, wherein the camera shake correction method is processed when a camera shake correction mode is set.   An image pickup apparatus comprising the camera shake correction apparatus according to claim 8.

Images