JP4345940B2 - Camera shake image correction method, recording medium, and imaging apparatus - Google Patents

Description

式中、θは回転角度、σ_x，σ_yはｘ，ｙ方向の平行移動量を表す。手ぶれによる回転角度θは一般に十分に小さいため、（１）式は（２）式のように近似してもよいし、（３）式のようにさらに強い近似をしてもよい。
【００２６】
【数２】

【数３】

【００２７】
ステップ１０１においては、（１）式、（２）式又は（３）式の座標変換行列の３つのパラメータ（θ，σ_x，σ_y）を決定する。入力画像をカメラで撮影する際に、加速度センサやジャイロセンサ等を利用し、カメラの光軸のぶれ量と光軸周りの回転角度を検出することができる場合には、その検出結果を例えばステップ１００で画像の付加情報として入力し、その付加情報に基づき直接的に３つのパラメータ（θ，σ_x，σ_y）を決定できる。本発明の一実施例によれば、そのような方法が採用される。
【００２８】
しかし、そのようなセンサが装備されていない一般的なカメラが利用される場合には、上に述べたような方法は利用不可能である。そこで、本発明の他の一実施例によれば、入力画像上の２カ所以上における（ｘ，ｙ）と（Ｘ，Ｙ）の組を推定し、換言すれば、図２にＢ１，Ｂ２，Ｂ３として示すような手ぶれの方向と大きさ（手ぶれベクトル）を推定し、それを（１）式、（２）式又は（３）式に代入することにより、座標変換行列のパラメータ（θ，σ_x，σ_y）を決定する。その具体的な処理手順の一例を以下に述べる。
【００２９】
最初に、入力画像上の適当な点を少なくとも２つ選ぶ。これは、処理プログラムによって自動的に行われるようにしてもよいし、画像を表示させた適当なディスプレイの画面上で、ユーザがポインティングデバイスを利用して指定するようにしてもよい。
【００３０】
次に、選ばれた各点における、手ぶれの方向を推定する。例えば、選ばれた各点の近傍領域について空間周波数分布を測定し、その高周波数分布の減少率が最大の方向を手ぶれの方向と推定する。このような方法は、本出願人の特願平９−１８０８４６号（特開平１１−０２７５７４号）、あるいは前記のR.Fabianらの文献等に詳しい。
【００３１】
次に、選ばれた各点における手ぶれの大きさを推定する。例えば、選ばれた各点の近傍領域について、推定された手ぶれの方向に画像の自己相関関数をとり、これを手ぶれ方向に微分すると、手ぶれの大きさに相当する間隔で微分値が極小を示すため、その極小点の間隔から、ぶれの大きさを推定する。この方法は、前記のY.Yitzhakyらの文献等に詳しい。
【００３２】
このような推定を終わると、選ばれた各点のｘ，ｙ座標値（ｘ，ｙ）と、それを推定した手ぶれ方向へ、推定した手ぶれの大きさだけ移動した点のｘ，ｙ座標値（Ｘ，Ｙ）との組を、前記（１）式、（２）式又は（３）式に代入することによりパラメータ（θ，σ_x，σ_y）を求める。これで、（１）式、（２）式又は（３）式の座標変換行列が決定された。
【００３３】
なお、入力画像を適当なディスプレイの画面に拡大表示し、ユーザが画面上で手ぶれによる画像のぶれを判断し、適当な２点以上と、そのぶれにより移動した点とをポインティングデバイスを利用して直接的に指示し、指示された座標値を用いて、パラメータ（θ，σ_x，σ_y）を求めるという方法も可能である。この簡便な方法によっても、熟練したユーザであれば、それなりの精度でパラメータの決定が可能である。
【００３４】
《各画素位置の手ぶれベクトルの推定：ステップ１０２》
次にステップ１０２においては、ステップ１０１によって決定された座標変換行列を用い、入力画像上の各画素位置（ｘ，ｙ）における手ぶれベクトルＢ（ｘ，ｙ）（手ぶれによる動きベクトル）を推定する。すなわち、（１）式、（２）式又は（３）式に（ｘ，ｙ）を代入して（Ｘ，Ｙ）を計算し、次式によりＢ（ｘ，ｙ）を求める。
【００３５】
【数４】

【００３６】
《ボケ補正：ステップ１０３》
ステップ１０３においては、入力画像の各画素値に対し、その位置の推定された手ぶれベクトルに応じたボケ補正を施し、手ぶれによるボケを補償した補正画像を生成する。
【００３７】
手ぶれが一様な動きによって導入された場合、手ぶれによるボケは（５）式のような１次元の点広がり関数で表され、また、このような手ぶれによるボケは（６）式のようなボケ補正関数で補正できることが知られている。この２式中、ｔは手ぶれの距離である。ａはボケの長さであり、これは手ぶれベクトルの長さに相当する。ただし、手ぶれの速度が一様でないと点広がり関数及びボケ補正関数は多少違ったものになるので、より高精度な手ぶれ補正のためには、手ぶれのパターンに応じてボケ補正関数を選択又は修正できるようにするのが望ましいが、これについては後述する。
【００３８】
【数５】

【数６】

【００３９】
本発明の一実施例によれば、入力画像上の各画素（ｉ_c，ｊ_c）に関し、その位置の手ぶれベクトルＢに沿った１次元の補間演算を行って補正画素値Ｖ（ｉ_c，ｊ_c）を計算する。具体的には
【数７】

式中、θは手ぶれベクトルＢのｘ軸に対しなす角度である。また、次の関係が成り立つ。
【数８】

Ｓはボケの及ぶ範囲の画素であり、ａ_x，ａ_yを手ぶれベクトルＢのｘ，ｙ成分の長さとすると、（７）式を次のように書くことができる。
【数９】

【００４０】
（７）式及び（９）式におけるｐ（）は、手ぶれベクトルＢに沿った前後の画素値から算出される、手ぶれベクトルＢ上又はその延長線上の補間画素値であり、これを（９）式に代入することにより補正画素値Ｖ（ｉ_c，ｊ_c）が求まる。なお、補間関数
Ｉ（ｘ）＝ｈ^-1（ｘ）
が成り立つものとする。
【００４１】
もし、注目画素の隣接画素のみを利用するのであれば、補間関数Ｉ（ｘ）は、図３に記号「＊」で示すような、手ぶれベクトルＢに沿った両側の画素から算出する。ここで、補間画素値ｐ（）は、図３に記号「＝」で示した仮想画素の値である。この画素値も、補間関数Ｉ（ｘ）によって求めることができる（ただし、このときはａ＝１）。ｘ，ｙの値は一般的には任意でよいが、効率的な計算を行うには、画素ピッチを１とし、
【数１０】

と置くことができる（図３は、そのような場合である）。したがって、（９）式は次のように書き直すことができる。
【数１１】

また、手ぶれベクトルＢの両側の画素のみを利用するならば、条件式 ｉ，ｊ Ａ
は次のように表すことができる。
【数１２】

式中、［ｘ］はｘを越えない整数を表す。
【００４２】
以上に述べた本発明による手ぶれ画像補正方法は、画像内に不均一な手ぶれ（回転と平行移動）がある場合にも、実空間処理により高精度な手ぶれ補正が可能である。画像の回転等の前処理や後処理を含まないため、そのような処理に伴う画質劣化が生じない。また、そのような前処理や後処理を含まないうえに、手ぶれベクトルに沿った補間画素値をテーブル参照方式で高速に計算できるから、全体として高速な処理が可能である。
【００４３】
さて、（５）式の点広がり関数は、図４の（Ａ）のようなグラフとして表すことができる。図４において、横軸はぶれの距離ｔ、縦軸はレスポンスの大きさである。実際の手ぶれは速度が一様でないことが多く、点広がり関数は図４（Ａ）に示したグラフとは多少異なったものになる。しかし、手ぶれのパターンはいくつかの類型に分類可能である。いくつかの典型的な手ぶれパターンに対応した点広がり関数のグラフを図４の（Ｃ），（Ｄ）及び（Ｅ）に示す。（Ｃ）のグラフは最初はゆっくりで次第に速くなる手ぶれパターンに対応した点広がり関数を表し、（Ｄ）のグラフは、それと逆に最初は速く、次第に遅くなる手ぶれパターンに対応した点広がり関数を表す。（Ｅ）のグラフは、最初と最後がゆっくりで中間で動きが速い手ぶれパターンに対応した点広がり関数を表す。慣れてくると、画像を観測すれば、かなりの正確さで、どのような手ぶれパターンに該当するか見当をつけることができる。
【００４４】
本発明の一実施例によれば、図４の（Ａ），（Ｃ），（Ｄ），（Ｅ）のような複数の典型的な手ぶれパターンの点広がり関数に対応したボケ補正関数が例えばテーブルの形で用意され、その中からボケ補正に利用するためのボケ補正関数がユーザによって選択される。図１に示した処理フローとの関連では、そのようなボケ補正関数の選択のためのステップは、ボケ補正処理ステップ１０３の内部、又はそれより前の段階に置かれる。
【００４５】
また、本発明の別の実施例によれば、そのようなボケ補正関数の選択が自動的に行われる。そのための手順の一例を図５に示す。
【００４６】
図５のステップ２００において、入力画像の手ぶれによる点広がり関数を推定する。光軸のぶれを検出するための加速度センサ等を装備したカメラが入力画像の撮影に利用される場合には、そのセンサによって光軸の経時変化を検出し、カメラのシャッター開放のタイミングと重畳させて、直接的に入力画像の点広がり関数を推定できる。そのようなカメラが利用されない場合は、入力画像から点広がり関数を推定する。具体的には、被写体上の１点のぶれによって形成される尾引き画像に沿った濃度（輝度）分布を観測すればよい。より正確な推定のためには、例えば、画像中の滑らかな背景部分に隣接する高コントラスト部分（通常、エッジと呼ばれる部分）に、前記のY.Yitzhakyらの論文に述べられている方法を利用すればよい。
【００４７】
次のステップ２０１において、予め用意された複数の典型的手ぶれパターンに対応する点広がり関数の中から、前ステップで推定された点広がり関数に最も類似したものを選択する。この選択の基準としては距離を利用できる。例えば次のような距離Ｄ_jが最小の点広がり関数ｈ_j（ｔ）が選択される。
【数１３】

式中、ｈ（ｔ）は推定した点広がり関数、ｈ_j（ｔ）は予め用意された点広がり関数である。
【００４８】
予め用意された典型的手ぶれパターンの点広がり関数のそれぞれに対応した複数のボケ補正関数も予め用意される。ステップ２０２において、用意された複数のボケ補正関数の中から、前ステップで選択された点広がり関数に対応したボケ補正関数が選択される。このようにして選択されたボケ補正関数が、図１のボケ補正処理ステップ１０３で利用されることになる。
【００４９】
本発明の一実施例によれば、ユーザが、ディスプレイ上で、点広がり関数を確認して必要な編集をすることにより、ボケ補正処理に利用するためのボケ補正関数を修正することができる。その手順を図６に示す。
【００５０】
図６のステップ３００において、予め用意されている点広がり関数を適当なディスプレイの画面に表示する。ステップ３０１において、ユーザは適当なポインティングデバイスを利用して、画面上で点広がり関数に対し必要な編集を行う。例えば、図４の（Ａ）に示すような点広がり関数のグラフが表示されている場合に、ポインティングデバイスを利用して、例えば図４の（Ｂ）に示すようにグラフを編集することができる。なお、ステップ３００で表示される点広がり関数を、ユーザが複数の点広がり関数の中から選択できるようにしてもよい。
【００５１】
ステップ３０２において、前ステップにおける点広がり関数の編集内容を反映させるように、予め用意されているボケ補正関数を修正する。このようにして修正後のボケ補正関数が図１のボケ編集処理ステップ１０３に利用される。
【００５２】
本発明の別の実施例について、図６を援用して説明する。ステップ３００において、入力画像の手ぶれによる点広がり関数をグラフィック表示する。具体的は、例えば、入力画像中の均一な背景部分について、ぶれ方向に直交する方向に輝度を投影することにより図４に示したようなグラフを直接得られるので、それを適当なディスプレイに表示する。このようにして測定された点広がり関数は、しばしば測定誤差を伴うので、ステップ３０１において、ユーザはポインティングデバイスを利用してディスプレイの画面上で点広がり関数に対し必要な修正を行う。ステップ３０２において、修正後の点広がり関数に対応したボケ補正関数を生成する。この生成されたボケ補正関数がボケ補正処理に用いられる。
【００５３】
本発明は、例えば図７に示すようなＣＰＵ４００、メモリ４０１、ディスプレイ４０２、マウス等のポインティングデバイス４０３、ＣＤ−ＲＯＭ等の記録媒体４０４のためのドライブ４０５、ハードディスク等の補助記憶装置４０６、外部インターフェース４０７等をシステムバス４０９で相互接続したような構成のコンピュータを利用し、ソフトウェアにより実施し得る。図１、図５及び図６に関連して説明した処理のためのプログラム４１０、図５又は図６に関連して説明したような点広がり関数やボケ補正関数のテーブル４１１は、例えば、それらが記録された記録媒体４０４よりドライブ４０５によってメモリ４０１に読み込まれる。あるいは、プログラム４１０及びテーブル４１１は補助記憶装置４０６に一旦保存され、処理実行時に補助記憶装置４０６からメモリ４０１に読み込まれる。処理対象の画像データは、例えば、それを撮影したデジタルカメラ４０８より、外部インターフェース４０７を通じてメモリ４０１に読み込まれる。図６に関連して説明した点広がり関数のグラフィック表示及びその編集には、ディスプレイ４０２とポインティングデバイス４０３が利用される。
【００５４】
本発明の別の実施の形態によれば、図１，図５及び図６に関連して説明した処理のための機能がデジタルカメラ等の撮像装置に実装される。図８は、そのようなデジタルカメラの一例の概略ブロック図である。図８において、５００は不図示の光学系を介して光学像が結像されるＣＣＤイメージセンサ、５０２はＣＣＤイメージセンサ５００から出力されるアナログ画像信号の増幅やデジタル信号への変換等を行う信号回路部、５０２は画像データ等の記憶のためのデータメモリ、５０３はデジタル画像データの圧縮／伸長やビデオ信号への変換等を行うデジタル信号プロセッサ、５０４は液晶ディスプレイパネル等を用いたモニタディスプレイである。５０６はカメラ内部の制御等を行うためのマイクロコンピュータであり、ＣＰＵ５１０やＲＯＭ５１１、ＲＡＭ５１２、その他の不図示の外部インターフェース等からなる。５０７はユーザがマイクロコンピュータ５０６へ各種指示を入力するための操作部である。
【００５５】
操作部５０７の撮影ボタンが押されない状態では、デジタル信号プロセッサ５０３は、信号回路部５０１から連続的に出力されるデジタル画像データをビデオ信号に変換してモニタディスプレイ５０４に出力するため、ユーザは視野内の映像をモニタディスプレイ５０４で監視することができる。ユーザが操作部５０７上の撮影ボタンを押すと、マイクロコンピュータ５０６の制御により、信号回路部５０１より出力される画像１枚分のデジタル画像データがデータメモリ５０２に一時的に保存される。次に、この画像データがデジタル信号プロセッサ５０３へ転送され、必要な階調補正等が施された後に圧縮される。得られた圧縮画像データはデータメモリ５０２へ転送される。外部のパソコン等への画像データ出力が操作部５０７より指示されると、マイクロコンピュータ５０６の制御により、データメモリ５０２内の圧縮画像データがデジタル信号プロセッサ５０３へ転送されて伸長される。伸長された画像データは再びデータメモリ５０２に書き込まれる。そして、この画像データがマイクロプロセッサ５０６の制御により外部のパソコン等へ転送される。このようなデジタルカメラ内部の一般的な制御は、ＲＯＭ５１１に格納されたプログラムに従って遂行される。
【００５６】
ここに示すデジタルカメラにおいては、図１，図５及び図６に関連して説明した処理ステップのための手段がソフトウェアにより実装される。すなわち、図１、図５及び図６に関連して説明した各処理ステップをマイクロコンピュータ５０６に実行させるためのプログラム５２０や、図５又は図６に関連して説明したような点広がり関数やボケ補正関数のテーブル５２１もＲＯＭ５１１（又はＲＡＭ５１２）に格納されている。ユーザが、操作部５０７の操作により手ぶれ補正を指示すると、マイクロコンピュータ５０６は、プログラム５２０に従って、データメモリ５０２上の圧縮画像データをデジタル信号プロセッサ５０３へ転送して伸長させ、伸長された画像データをデータメモリ５０２に格納させる。この伸長された画像データのビデオ信号もデジタル信号プロセッサ５０３より出力されるので、その画像データの画像をモニタディスプレイ５０４上でユーザは確認できる。そして、マイクロプロセッサ５０６は、データメモリ５０２上の伸長された画像データに対し、以上に説明したような本発明による手ぶれ画像補正処理を実行する。データメモリ５０２上に得られた補正後の画像データはデジタル信号プロセッサ５０３へ転送され、そのビデオ信号がモニタディプレイ５０４へ与えられ、ユーザは手ぶれ補正後の画像を確認できる。ユーザが、操作部５０７で指示することにより、前述のようなボケ補正関数の選択や点広がり関数の編集を行うことができる。この場合、マイクロコンピュータ５０６は、ボケ補正関数の選択のためのイメージや点広がり関数をデジタル信号プロセッサ５０３を介してモニタディスプレイ５０４に表示させる。ユーザは、操作部５０７に設けられたポインティングデバイス（不図示）を操作することにより、モニタディスプレイ５０４の画面上で、ボケ補正関数の選択や点広がり関数の編集を指示することができる。
【００５７】
【発明の効果】
以上の説明から明らかなように、請求項１乃至６、８乃至１３の各項記載の発明によれば、不均一な回転及び平行移動を含む手ぶれが生じた場合にも、適切な手ぶれ補正が可能である。また、手ぶれ補正のために画像の回転等の前処理や後処理を必要としないため、そのような処理に伴う画質劣化も生じない。したがって、不均一な手ぶれ画像から、手ぶれを補正した高画質の画像を復元可能である。請求項６又は１３記載の発明によれば、手ぶれ補正のための計算をテーブル参照方式等で高速に行うことができ、しかも画像の回転等の前処理や後処理も不要であるため、全体として高速な処理が可能である。請求項５記載の発明によれば、画像の複数位置での手ぶれの方向及び大きさを、画像そのものから推定するため、加速度センサ等を持たないカメラで撮影された画像に対しても、適切な手ぶれ補正が可能である。請求項１２記載の発明によれば、撮像装置に加速度センサ等を設けなくとも、撮像した画像に対し手ぶれ補正を適切に行うことができる。請求項１又は８記載の発明によれば、ユーザが、画像を観察し、適切と判断したボケ補正関数を選択して手ぶれ補正を行わせ、また、必要と判断した場合にはボケ補正関数を選び直すことができるため、様々な手ぶれパターンに対応可能になる。また、請求項２又は９によれば、ユーザが関与することなく、画像撮影時の手ぶれパターンに対応したボケ補正関数を利用し適切な手ぶれ補正が可能になる。また、請求項３，４、１０又は１１記載の発明によれば、ユーザが、特開平７−２２６９０５号に関連して説明したような撮影環境条件に応じた補正の違いや、手ぶれパターンの違いに応じて、点広がり関数を任意に編集しボケ補正関数を最適化することができる。請求項７記載の発明によれば、上に述べたような手ぶれ画像補正やボケ補正関数の最適化等をコンピュータを利用し実施可能になる、等々の効果を得られる。
【図面の簡単な説明】
【図１】 本発明による手ぶれ画像補正の処理手順の一例を示すフローチャートである。
【図２】 座標変換行列の決定に関連した、画像上の複数の位置における手ぶれベクトルを示す図である。
【図３】 手ぶれベクトルに沿った１次元の補間演算の説明のための図である。
【図４】 典型的な手ぶれパターンに対応した点広がり関数の例及び点広がり関数の編集例を示す図である。
【図５】 ボケ補正関数の自動選択手順の一例を説明するためのフローチャートである。
【図６】 点広がり関数の編集によりボケ補正関数を修正する手順を説明するためのフローチャートである。
【図７】 本発明のソフトウェアによる実施のためのコンピュータの一例を示すブロック図である。
【図８】 本発明によるデジタルカメラの一例を示すブロック図である。
【符号の説明】
４００ ＣＰＵ
４０１ メモリ
４０２ ディスプレイ装置
４０３ ポインティングデバイス
４１０ 手ぶれ画像補正のためのプログラム
４１１ 点広がり関数及びボケ補正関数のテーブル
５００ ＣＣＤイメージセンサ
５０１ 信号回路部
５０２ データメモリ
５０３ デジタル信号プロセッサ
５０４ モニタディスプレイ
５０６ マイクロコンピュータ
５０７ ポインティングデバイスを含む操作部
５２０ 手ぶれ画像補正のためのプログラム
５２１ 点広がり関数及びボケ補正関数のテーブル[0001]
BACKGROUND OF THE INVENTION
  The present invention relates to a technique for correcting image degradation due to camera shake.
[0002]
[Prior art]
  When a subject is photographed with a digital camera or the like, camera movement during exposure, that is, blur due to camera shake occurs in the photographed image. Various techniques have been developed in the past to avoid image degradation due to camera shake.
[0003]
  For example, the camera is equipped with an acceleration sensor or the like for detecting the movement of camera shake, the detection signal of the sensor is converted into an image movement, and the camera optical system (prism, A method of mechanically moving a lens or an image pickup device is known (for example, Japanese Patent No. 2637464).
[0004]
  The movement of a virtual point-like subject on the image due to camera shake corresponds to blurring, and is often called a point spread function. Also known is a method for correcting camera shake by estimating the point spread function from the change over time in the aperture ratio of the shutter at the time of shooting and the movement of the camera optical axis detected by an acceleration sensor, etc., and applying the inverse correction to the entire image. (JP-A-7-226905).
[0005]
  Furthermore, research for estimating camera shake from an image without using an acceleration sensor or the like has been made for a long time. For example, in the document “A. About Alib and L. M. Silverman; Rotation of Motion Degraded Images; IEEE Trans. Circuits and Systems, p. Are listed. Also, in the direction of camera shake, if the camera shake can be approximated to be uniform, the density should be uniform (brightness). Therefore, taking the autocorrelation function of the image in the camera shake direction and differentiating this will yield the amount of camera shake. The differential value shows a minimum at a corresponding interval and shows a maximum at the zero point in the direction without blurring, so that it is possible to estimate the magnitude of the blur, “Y. Yitzhaky and NS Kopeika; Identification of Blur Parameters from Motion Blurred Images; Graphical Models and Image Processing Vol.59, pp.310-320, 1997 ". Regarding estimation of the blur direction, for example, the document “R. Fabian and D. Malah; Robust Identification and Out-Of-Focus Blur Parameters from Blurred and Noisy Images; Graphical Models and Image Processing, Vol. 53, pp. 403-412, 1991 ”describes a method using a spatial frequency distribution.
[0006]
[Problems to be solved by the invention]
  When the camera rotates around the optical axis at the time of shooting, camera shake on the image becomes non-uniform. It is an object of the present invention to provide a camera shake that can restore a sharp image without mechanical operation as in the above-mentioned Japanese Patent No. 2637464 even when such a camera shake including uneven rotation and translation occurs. An image correction method and an imaging apparatus are provided. Another object of the present invention is a camera shake image correction method capable of accurately correcting non-uniform camera shake even with respect to an image taken with an ordinary camera that does not have an acceleration sensor or the like for detecting shake of an optical axis. Is to provide. Another object of the present invention is to provide an imaging apparatus capable of obtaining an image in which non-uniform camera shake is accurately corrected without providing an acceleration sensor or the like for detecting optical axis shake.
[0007]
  The speed of camera shake is not always uniform from the beginning to the end of camera shake. As the speed of camera shake changes with time, the shape of the point spread function also changes. Another object of the present invention is to provide a camera shake image correction method and an imaging apparatus capable of appropriate camera shake correction even when the camera shake speed fluctuates with time.
[0008]
  In some cases, such as Japanese Patent Application Laid-Open No. 7-226905, which presupposes that image data that directly reflects a change in the amount of light due to a change in the aperture ratio of the shutter as it is, good camera shake correction may not be expected. This problem is particularly noticeable in digital cameras that have become widespread in recent years. In a general digital camera, an optical image formed by a lens system is converted into an electric signal by an image pickup device such as a CCD. This electric signal is converted to an external brightness or color deviation (red in a clear sky evening) Because it is corrected according to the shooting environment conditions such as strong blue and strong blue under fluorescent lamps), it is not possible to obtain image data that directly reflects the change in light quantity due to the change in shutter aperture over time. is there. Another object of the present invention is to provide a camera shake image correction method and an imaging apparatus capable of performing appropriate camera shake correction even when correction according to the shooting environment conditions described above is performed. .
[0009]
[Means for Solving the Problems]
  According to the first to seventh aspects of the present invention, an improved camera shake image correction method is provided. Its characteristics are as follows.
[0010]
  A camera shake image correction method according to the first aspect of the present invention includes:
A first step of determining a coordinate transformation matrix representing rotation and translation of pixels on the target image due to camera shake;
A second step of estimating a camera shake vector at each pixel position on the target image using the coordinate transformation matrix determined in the first step;
A third step of performing blur correction on each pixel value on the target image in accordance with the camera shake vector estimated in the second step at the position;
Among the plurality of blur correction functions corresponding to the point spread function of a plurality of typical camera shake patterns including a camera shake pattern with a non-uniform camera shake speed prepared in advance, the blur correction function is used in the third step. Including a fourth step in which a user selects a blur correction function.
[0011]
  A camera shake image correction method according to a second aspect of the present invention includes:
A first step of determining a coordinate transformation matrix representing rotation and translation of pixels on the target image due to camera shake;
A second step of estimating a camera shake vector at each pixel position on the target image using the coordinate transformation matrix determined in the first step;
A third step of performing blur correction on each pixel value on the target image in accordance with the camera shake vector estimated in the second step at the position;
A fourth step of estimating a point spread function due to camera shake of the target image;
A fifth step of selecting a point spread function most similar to the point spread function estimated in the fourth step from a plurality of typical point spread functions prepared in advance;
A sixth step of selecting a blur correction function corresponding to the typical point spread function selected in the fifth step from a plurality of blur correction functions prepared in advance.
Including
The blur function selected in the sixth step is used for blur correction in the third step.
[0012]
  According to a third aspect of the present invention, there is provided a camera shake image correction method.
A first step of determining a coordinate transformation matrix representing rotation and translation of pixels on the target image due to camera shake;
A second step of estimating a camera shake vector at each pixel position on the target image using the coordinate transformation matrix determined in the first step;
A third step of performing blur correction on each pixel value on the target image in accordance with the camera shake vector estimated by the second step at the position;
A fourth step of displaying a graph of a point spread function prepared in advance on a display screen;
A fifth step of editing the point spread function graph displayed on the screen according to an operation of a pointing device;
Corresponding to the point spread function graph after editing in the fifth step, a sixth step of correcting a blur correction function used for blur correction in the third step is included.
[0013]
  According to a fourth aspect of the present invention, there is provided a camera shake image correction method.
A first step of determining a coordinate transformation matrix representing rotation and translation of pixels on the target image due to camera shake;
A second step of estimating a camera shake vector at each pixel position on the target image using the coordinate transformation matrix determined in the first step;
A third step of performing blur correction on each pixel value on the target image in accordance with the camera shake vector estimated in the second step at the position;
A fourth step of displaying a graph of a point spread function due to camera shake of the target image on a display screen;
A fifth step of editing the point spread function graph displayed on the screen according to an operation of a pointing device;
A sixth step of generating a blur correction function corresponding to the point spread function after editing in the fifth step;
The blur correction function generated in the sixth step is used for blur correction in the third step.
[0014]
  According to a fifth aspect of the present invention, in the camera shake image correction method according to any one of the first to fourth aspects, the first step includes a direction of camera shake at a plurality of positions on the target image, and Including a step of estimating the size based on the target image, and based on the direction and size of the camera shake estimated by the step, a coordinate transformation matrix representing rotation and translation of the pixel on the target image due to the camera shake Is to decide.
[0015]
  According to a sixth aspect of the present invention, in the camera shake image correction method according to any one of the first to fifth aspects, blur correction is performed in the third step by an interpolation operation along a camera shake vector. It is to be.
[0016]
  According to the eighth to thirteenth aspects of the present invention, there is provided an improved image pickup apparatus that picks up a subject with an image pickup device and obtains a digital image. Its characteristics are as follows.
[0017]
  An image pickup apparatus according to the invention described in claim 8 provides:
A first means for determining a coordinate transformation matrix representing rotation and translation of pixels on a digital image due to camera shake;
Second means for estimating a camera shake vector at each pixel position on the digital image using the coordinate transformation matrix determined by the first means;
Third means for performing blur correction on each pixel value on the digital target image according to the hand movement vector estimated by the second means at the position;
Among the plurality of blur correction functions prepared in advance and corresponding to the point spread function of a plurality of typical camera shake patterns including a camera shake pattern having a non-uniform camera shake speed, the third means uses the blur correction function in the third means. Fourth means for the user to select a blur correction function for
It is characterized by including.
[0018]
  An image pickup apparatus according to a ninth aspect of the present invention provides:
A first means for determining a coordinate transformation matrix representing rotation and translation of pixels on a digital image due to camera shake;
Second means for estimating a camera shake vector at each pixel position on the digital image using the coordinate transformation matrix determined by the first means;
Third means for performing blur correction on each pixel value on the digital image according to the hand movement vector estimated by the second means at the position;
A fourth means for estimating a point spread function due to camera shake in a digital image;
A fifth means for selecting a point spread function most similar to the point spread function estimated by the fourth means from a plurality of typical point spread functions prepared in advance;
Sixth means for selecting a blur correction function corresponding to the typical point spread function selected by the fifth means from a plurality of blur correction functions prepared in advance.
Including
The blur function selected by the sixth means is used for blur correction in the third means.
[0019]
  An imaging apparatus according to the invention of claim 10 is provided.
A first means for determining a coordinate transformation matrix representing rotation and translation of pixels on a digital image due to camera shake;
Second means for estimating a camera shake vector at each pixel position on the digital image using the coordinate transformation matrix determined by the first means;
Third means for performing blur correction on each pixel value on the digital image according to the hand movement vector estimated by the second means at the position;
A fourth means for displaying a graph of a point spread function prepared in advance on a display screen;
A fifth means for editing the graph of the point spread function displayed on the screen according to an operation of a pointing device;
Corresponding to the point spread function graph after editing by the fifth means, sixth means for correcting the blur correction function used for blur correction in the third means
It is characterized by including.
[0020]
  An imaging device according to the invention of claim 11 is provided.
A first means for determining a coordinate transformation matrix representing rotation and translation of pixels on a digital image due to camera shake;
Second means for estimating a camera shake vector at each pixel position on the digital image using the coordinate transformation matrix determined by the first means;
Third means for performing blur correction on each pixel value on the digital image according to the hand movement vector estimated by the second means at the position;
A fourth means for displaying a graph of a point spread function due to camera shake of a digital image on a display screen;
A fifth means for editing the graph of the point spread function displayed on the screen according to an operation of a pointing device;
Sixth means for generating a blur correction function corresponding to the point spread function after editing by the fifth means
Including
The blur correction function generated by the sixth means is used for blur correction in the third means.
[0021]
  According to a twelfth aspect of the present invention, in the imaging apparatus according to any one of the eighth to eleventh aspects, the first means includes a direction and a size of camera shake at a plurality of positions on the digital image. Is determined based on the digital image, and a coordinate transformation matrix representing rotation and translation of the pixel on the digital image due to the camera shake is determined based on the direction and size of the camera shake estimated by the means. That is.
[0022]
  According to a thirteenth aspect of the present invention, in the imaging apparatus according to any one of the eighth to twelfth aspects, blur correction is performed by an interpolation operation along a camera shake vector in the third means. .
[0023]
DETAILED DESCRIPTION OF THE INVENTION
  FIG. 1 shows an example of a processing procedure for camera shake image correction according to the present invention. In FIG. 1, 100 is a step in which an image in which blur is caused by camera shake is input, 101 is a step in which a coordinate transformation matrix representing rotation and translation of pixels on the input image due to camera shake is determined, and 102 is the coordinates A step of estimating a camera shake vector (direction and magnitude of “blurring” due to camera shake) at each pixel position on the input image using the transformation matrix, and 103 for each pixel value on the input image A step in which blur correction according to the estimated camera shake vector is performed and an image in which blur due to camera shake is corrected is generated, and 104 is a step in which a corrected image is output. Note that the camera shake image correction processing procedure according to the present invention can be variously changed as will be described later.
[0024]
  << Determination of coordinate transformation matrix: Step101》
  Step101The specific processing content of will be described.
  Pixels on the image move from their original positions due to camera shake when taking an image. Since such minute blur can be expressed by affine transformation, the following equation is obtained between the original pixel coordinate value (x, y) and the coordinate value (X, Y) changed by the camera shake. It holds.
[0025]
[Expression 1]

Where θ is the rotation angle and σ_x, Σ_yRepresents the amount of parallel movement in the x and y directions. Since the rotation angle θ due to camera shake is generally sufficiently small, the equation (1) may be approximated as in the equation (2) or may be further approximated as in the equation (3).
[0026]
[Expression 2]

[Equation 3]

[0027]
  Step101In the three parameters (θ, σ) of the coordinate transformation matrix of the formula (1), (2) or (3)_x, Σ_y). When taking an input image with a camera and using an acceleration sensor, a gyro sensor or the like to detect the shake amount of the optical axis of the camera and the rotation angle around the optical axis, the detection result is, for example, a step. 100 is input as additional information of the image, and based on the additional information, three parameters (θ, σ_x, Σ_y) Can be determined. According to one embodiment of the present invention, such a method is employed.
[0028]
  However, when a general camera not equipped with such a sensor is used, the above-described method cannot be used. Therefore, according to another embodiment of the present invention, a set of (x, y) and (X, Y) at two or more places on the input image is estimated, in other words, in FIG. By estimating the direction and magnitude (shake vector) of the camera shake as shown by B3 and substituting it into the formula (1), (2) or (3), the parameters (θ, σ) of the coordinate transformation matrix are calculated._x, Σ_y). An example of a specific processing procedure will be described below.
[0029]
  First, select at least two appropriate points on the input image. This may be automatically performed by a processing program, or may be designated by a user using a pointing device on an appropriate display screen on which an image is displayed.
[0030]
  Next, the direction of camera shake at each selected point is estimated. For example, the spatial frequency distribution is measured in the vicinity of each selected point, and the direction in which the reduction rate of the high frequency distribution is maximum is estimated as the direction of camera shake. Such a method is described in detail in Japanese Patent Application No. 9-180846 (Japanese Patent Laid-Open No. 11-027574) or the above-mentioned R. Fabian et al.
[0031]
  Next, the magnitude of camera shake at each selected point is estimated. For example, taking the autocorrelation function of the image in the direction of the estimated camera shake for the region near each selected point and differentiating it in the direction of the camera shake, the differential value is minimal at intervals corresponding to the size of the camera shake. Therefore, the size of the shake is estimated from the interval between the local minimum points. This method is detailed in the above-mentioned literature by Y. Yitzhaky et al.
[0032]
  When such estimation is finished, the x and y coordinate values (x, y) of each selected point and the x and y coordinate values of the point moved by the estimated camera shake direction in the estimated camera shake direction. By substituting the pair with (X, Y) into the above formula (1), (2) or (3), the parameters (θ, σ_x, Σ_y) Thus, the coordinate transformation matrix of the formula (1), the formula (2) or the formula (3) is determined.
[0033]
  The input image is enlarged and displayed on an appropriate display screen, and the user determines whether the image is shaken due to camera shake on the screen, and uses a pointing device to find the appropriate two or more points and the points moved by the shake. Directly instructing and using the indicated coordinate values, parameters (θ, σ_x, Σ_y) Is also possible. Even with this simple method, a skilled user can determine parameters with reasonable accuracy.
[0034]
  << Estimation of Shake Vector at Each Pixel Position: Step 102 >>
  Next, in step 102, the hand movement vector B (x, y) (motion vector due to hand movement) at each pixel position (x, y) on the input image is estimated using the coordinate transformation matrix determined in step 101. That is, (X, Y) is calculated by substituting (x, y) into the equation (1), (2) or (3), and B (x, y) is obtained by the following equation.
[0035]
[Expression 4]

[0036]
  << Bokeh correction: Step 103 >>
  In step 103, each pixel value of the input image is subjected to blur correction according to a camera shake vector whose position is estimated, and a corrected image in which blur due to camera shake is compensated is generated.
[0037]
  When camera shake is introduced by uniform movement, blur due to camera shake is expressed by a one-dimensional point spread function as shown in Equation (5), and blur due to such camera shake is as shown in Equation (6). It is known that correction can be performed with a correction function. In these two formulas, t is the distance of camera shake. a is the length of the blur, which corresponds to the length of the hand movement vector. However, if the camera shake speed is not uniform, the point spread function and the blur correction function will be slightly different. For more accurate camera shake correction, select or correct the blur correction function according to the camera shake pattern. It is desirable to be able to do this, but this will be described later.
[0038]
[Equation 5]

[Formula 6]

[0039]
  According to one embodiment of the present invention, each pixel (i_c, J_c) For the corrected pixel value V (i) by performing a one-dimensional interpolation operation along the hand movement vector B at that position._c, J_c). In particular
[Expression 7]

In the equation, θ is an angle formed with respect to the x axis of the shake vector B. In addition, the following relationship holds.
[Equation 8]

S is a pixel in a blurred range, and a_x, A_yIs the length of the x and y components of the hand movement vector B, the equation (7) can be written as follows.
[Equation 9]

[0040]
  P () in the equations (7) and (9) is an interpolated pixel value on the camera shake vector B or an extension line thereof calculated from the previous and next pixel values along the camera shake vector B, and is expressed as (9) By substituting into the equation, the corrected pixel value V (i_c, J_c) Is obtained. Interpolation function
  I (x) = h^-1(X)
Is assumed to hold.
[0041]
  If only the pixel adjacent to the target pixel is used, the interpolation function I (x) is calculated from the pixels on both sides along the camera shake vector B as indicated by the symbol “*” in FIG. Here, the interpolated pixel value p () is the value of the virtual pixel indicated by the symbol “=” in FIG. This pixel value can also be obtained by the interpolation function I (x) (in this case, a = 1). The values of x and y may generally be arbitrary. However, in order to perform efficient calculation, the pixel pitch is set to 1,
[Expression 10]

(FIG. 3 shows such a case). Therefore, equation (9) can be rewritten as follows.
## EQU11 ##

  If only pixels on both sides of the hand movement vector B are used, the conditional expression i, j A
Can be expressed as:
[Expression 12]

In the formula, [x] represents an integer not exceeding x.
[0042]
  The above-described camera shake image correction method according to the present invention can perform highly accurate camera shake correction by real space processing even when there is uneven camera shake (rotation and translation) in an image. Since pre-processing and post-processing such as image rotation are not included, image quality degradation associated with such processing does not occur. In addition, since such pre-processing and post-processing are not included, the interpolation pixel value along the camera shake vector can be calculated at high speed by the table reference method, so that high-speed processing as a whole is possible.
[0043]
  Now, the point spread function of equation (5) is represented as a graph as shown in FIG.ThisYou can. In FIG. 4, the horizontal axis represents the blur distance t, and the vertical axis represents the response magnitude. Actual camera shake often has a non-uniform velocity, and the point spread function is slightly different from the graph shown in FIG. However, camera shake patterns can be classified into several types. Graphs of point spread functions corresponding to some typical camera shake patterns are shown in FIGS. 4 (C), (D) and (E). The graph in (C) represents a point spread function corresponding to a blurring pattern that is slow and gradually fast at first, and the graph in (D) is a point spread function corresponding to a blurring pattern that is fast at first and gradually slows down. To express. The graph of (E) represents a point spread function corresponding to a camera shake pattern in which the beginning and the end are slow and the motion is fast in the middle. Once you get used to it, you can figure out what kind of hand-shake pattern it is, with considerable accuracy, by observing the image.
[0044]
  According to one embodiment of the present invention, a blur correction function corresponding to a point spread function of a plurality of typical camera shake patterns such as (A), (C), (D), and (E) in FIG. A blur correction function that is prepared in the form of a table and is used for blur correction is selected by the user. In the context of the processing flow shown in FIG. 1, the step for selecting such a blur correction function is placed inside or before the blur correction processing step 103.
[0045]
  According to another embodiment of the present invention, such a blur correction function is automatically selected. An example of the procedure for this is shown in FIG.
[0046]
  In step 200 of FIG. 5, a point spread function due to camera shake of the input image is estimated. When a camera equipped with an acceleration sensor to detect optical axis shake is used to capture an input image, the sensor detects a change in the optical axis over time and superimposes it with the shutter release timing of the camera. Thus, the point spread function of the input image can be estimated directly. When such a camera is not used, a point spread function is estimated from the input image. Specifically, it is only necessary to observe a density (luminance) distribution along the tail image formed by the blurring of one point on the subject. For more accurate estimation, for example, the method described in the above-mentioned Y. Yitzhaky et al. Paper is used for a high-contrast portion (usually called an edge) adjacent to a smooth background portion in an image. do it.
[0047]
  In the next step 201, the one most similar to the point spread function estimated in the previous step is selected from the point spread functions corresponding to a plurality of typical hand movement patterns prepared in advance. Distance can be used as a criterion for this selection. For example, the following distance D_jHas the smallest point spread function h_j(T) is selected.
[Formula 13]

Where h (t) is the estimated point spread function, h_j(T) is a point spread function prepared in advance.
[0048]
  A plurality of blur correction functions corresponding to respective point spread functions of typical hand movement patterns prepared in advance are also prepared in advance. In step 202, a blur correction function corresponding to the point spread function selected in the previous step is selected from the prepared blur correction functions. The blur correction function selected in this way is used in the blur correction processing step 103 in FIG.
[0049]
  According to an embodiment of the present invention, a user can correct a blur correction function for use in blur correction processing by checking a point spread function on a display and performing necessary editing. The procedure is shown in FIG.
[0050]
  In step 300 of FIG. 6, a point spread function prepared in advance is displayed on an appropriate display screen. In step 301, the user makes necessary edits to the point spread function on the screen using an appropriate pointing device. For example, when a point spread function graph as shown in FIG. 4A is displayed, the graph can be edited as shown in FIG. 4B using a pointing device, for example. . Note that the point spread function displayed in step 300 may be selected by the user from a plurality of point spread functions.
[0051]
  In step 302, the blur correction function prepared in advance is corrected so as to reflect the edited content of the point spread function in the previous step. Thus, the corrected blur correction function is used in the blur edit processing step 103 in FIG.
[0052]
  Another embodiment of the present invention will be described with reference to FIG. In step 300, a point spread function due to camera shake of the input image is displayed graphically. Specifically, for example, by projecting the luminance in a direction orthogonal to the blur direction for a uniform background portion in the input image, a graph as shown in FIG. 4 can be directly obtained, so that it is displayed on an appropriate display. To do. Since the point spread function measured in this way often involves measurement errors, in step 301, the user makes necessary corrections to the point spread function on the screen of the display using a pointing device. In step 302, a blur correction function corresponding to the corrected point spread function is generated. The generated blur correction function is used for the blur correction process.
[0053]
  The present invention includes a CPU 400 as shown in FIG. 7, a memory 401, a display 402, a pointing device 403 such as a mouse, a drive 405 for a recording medium 404 such as a CD-ROM, an auxiliary storage device 406 such as a hard disk, and an external interface. It can be implemented by software using a computer configured such that 407 and the like are interconnected by a system bus 409. The program 410 for the processing described in relation to FIG. 1, FIG. 5 and FIG. 6, and the table 411 of the point spread function and blur correction function as described in relation to FIG. 5 or FIG. The data is read into the memory 401 by the drive 405 from the recorded recording medium 404. Alternatively, the program 410 and the table 411 are temporarily stored in the auxiliary storage device 406 and are read from the auxiliary storage device 406 into the memory 401 at the time of processing execution. For example, the image data to be processed is read into the memory 401 through the external interface 407 from the digital camera 408 that captured the image data. The display 402 and the pointing device 403 are used for the graphic display and editing of the point spread function described with reference to FIG.
[0054]
  According to another embodiment of the present invention, the processing functions described in relation to FIGS. 1, 5, and 6 are implemented in an imaging device such as a digital camera.FIG.FIG. 2 is a schematic block diagram of an example of such a digital camera. In FIG. 8, 500 is a CCD image sensor on which an optical image is formed through an optical system (not shown), and 502 is a signal for amplifying an analog image signal output from the CCD image sensor 500, converting it to a digital signal, or the like. A circuit unit 502 is a data memory for storing image data and the like, 503 is a digital signal processor for compressing / decompressing digital image data and converting it into a video signal, and 504 is a monitor display using a liquid crystal display panel or the like. is there. Reference numeral 506 denotes a microcomputer for controlling the inside of the camera, and includes a CPU 510, a ROM 511, a RAM 512, and other external interfaces (not shown). An operation unit 507 is used by the user to input various instructions to the microcomputer 506.
[0055]
  In a state where the shooting button of the operation unit 507 is not pressed, the digital signal processor 503 converts the digital image data continuously output from the signal circuit unit 501 into a video signal and outputs the video signal to the monitor display 504. The video inside can be monitored on the monitor display 504. When the user presses the shooting button on the operation unit 507, digital image data for one image output from the signal circuit unit 501 is temporarily stored in the data memory 502 under the control of the microcomputer 506. Next, the image data is transferred to the digital signal processor 503 and compressed after necessary gradation correction and the like. The obtained compressed image data is transferred to the data memory 502. When the operation unit 507 instructs to output image data to an external personal computer or the like, the compressed image data in the data memory 502 is transferred to the digital signal processor 503 and decompressed under the control of the microcomputer 506. The decompressed image data is written into the data memory 502 again. The image data is transferred to an external personal computer or the like under the control of the microprocessor 506. Such general control inside the digital camera is performed according to a program stored in the ROM 511.
[0056]
  In the digital camera shown here, the means for processing steps described in relation to FIGS. 1, 5 and 6 are implemented by software. That is, a program 520 for causing the microcomputer 506 to execute each processing step described with reference to FIG. 1, FIG. 5, and FIG. 6, a point spread function and a blur as described with reference to FIG. The correction function table 521 is also stored in the ROM 511 (or RAM 512). When the user instructs camera shake correction by operating the operation unit 507, the microcomputer 506 transfers the compressed image data on the data memory 502 to the digital signal processor 503 according to the program 520 and decompresses the decompressed image data. The data is stored in the data memory 502. Since the video signal of the decompressed image data is also output from the digital signal processor 503, the user can confirm the image of the image data on the monitor display 504. Then, the microprocessor 506 executes the camera shake image correction processing according to the present invention as described above on the decompressed image data on the data memory 502. The corrected image data obtained on the data memory 502 is transferred to the digital signal processor 503, the video signal is given to the monitor display 504, and the user can check the image after the camera shake correction. The user can select the blur correction function or edit the point spread function as described above by giving an instruction using the operation unit 507. In this case, the microcomputer 506 displays an image for selecting the blur correction function and a point spread function on the monitor display 504 via the digital signal processor 503. The user can instruct selection of a blur correction function or editing of a point spread function on the screen of the monitor display 504 by operating a pointing device (not shown) provided in the operation unit 507.
[0057]
【The invention's effect】
  As is apparent from the above description, the claims1 to 6, 8 to 13According to the inventions described in the above items, even when a camera shake including non-uniform rotation and translation occurs, an appropriate camera shake correction is possible. In addition, pre-processing and post-processing such as image rotation are not required for camera shake correction, and image quality degradation associated with such processing does not occur. Therefore, it is possible to restore a high-quality image in which camera shake is corrected from a non-uniform camera shake image. Claim6 or 13According to the described invention, calculation for camera shake correction can be performed at high speed by a table reference method and the like, and pre-processing and post-processing such as image rotation are not necessary, so that high-speed processing as a whole is possible. It is. Claim5According to the described invention, since the direction and size of camera shake at a plurality of positions of the image are estimated from the image itself, an appropriate camera shake correction can be performed even for an image taken with a camera having no acceleration sensor or the like. Is possible. Claim12According to the described invention, camera shake correction can be appropriately performed on a captured image without providing an acceleration sensor or the like in the imaging device. Claim1 or 8According to the described invention, the user can observe the image, select the blur correction function determined to be appropriate, perform camera shake correction, and reselect the blur correction function when determined to be necessary. Therefore, it is possible to cope with various hand movement patterns. Claims2 or 9According to this, it is possible to perform appropriate camera shake correction using a blur correction function corresponding to a camera shake pattern at the time of image capturing without involving the user. Claims3, 4, 10 or 11According to the described invention, the user can arbitrarily edit the point spread function according to the difference in the correction according to the shooting environment condition and the difference in the shake pattern as described in JP-A-7-226905. The blur correction function can be optimized. Claim7According to the described invention, it is possible to perform the above-described camera shake image correction, blur correction function optimization, and the like using a computer.
[Brief description of the drawings]
FIG. 1 is a flowchart illustrating an example of a camera shake image correction processing procedure according to the present invention.
FIG. 2 is a diagram showing camera shake vectors at a plurality of positions on an image related to determination of a coordinate transformation matrix.
FIG. 3 is a diagram for explaining a one-dimensional interpolation operation along a camera shake vector;
FIG. 4 is a diagram illustrating an example of a point spread function corresponding to a typical camera shake pattern and an example of editing the point spread function.
FIG. 5 is a flowchart for explaining an example of a procedure for automatically selecting a blur correction function.
FIG. 6 is a flowchart for explaining a procedure for correcting a blur correction function by editing a point spread function;
FIG. 7 is a block diagram illustrating an example of a computer for implementation by software of the present invention.
FIG. 8 is a block diagram showing an example of a digital camera according to the present invention.
[Explanation of symbols]
  400 CPU
  401 memory
  402 Display device
  403 pointing device
  410 Program for image stabilization
  411 Point spread function and blur correction function table
  500 CCD image sensor
  501 Signal circuit
  502 data memory
  503 Digital signal processor
  504 Monitor display
  506 Microcomputer
  507 Operation unit including pointing device
  520 Program for image stabilization
  521 Table of point spread function and blur correction function

Claims (13)

A first step of determining a coordinate transformation matrix representing rotation and translation of pixels on the target image due to camera shake;
A second step of estimating a camera shake vector at each pixel position on the target image using the coordinate transformation matrix determined in the first step;
A third step of performing blur correction on each pixel value on the target image in accordance with the camera shake vector estimated in the second step at the position;
Among the plurality of blur correction functions corresponding to the point spread function of a plurality of typical camera shake patterns including a camera shake pattern with a non-uniform camera shake speed prepared in advance, the blur correction function is used in the third step. A camera shake image correction method comprising: a fourth step in which a user selects a blur correction function for the purpose. A first step of determining a coordinate transformation matrix representing rotation and translation of pixels on the target image due to camera shake;
A second step of estimating a camera shake vector at each pixel position on the target image using the coordinate transformation matrix determined in the first step;
A third step of performing blur correction on each pixel value on the target image in accordance with the camera shake vector estimated in the second step at the position;
A fourth step of estimating a point spread function due to camera shake of the target image;
A fifth step of selecting a point spread function most similar to the point spread function estimated in the fourth step from a plurality of typical point spread functions prepared in advance;
Includes a sixth step of selecting from among said fifth blur correction function corresponding to the typical point spread function selected in step, a plurality of blur correction function prepared in advance,
A blurring image correction method, wherein the blur function selected in the sixth step is used for blur correction in the third step . A first step of determining a coordinate transformation matrix representing rotation and translation of pixels on the target image due to camera shake;
A second step of estimating a camera shake vector at each pixel position on the target image using the coordinate transformation matrix determined in the first step;
A third step of performing blur correction on each pixel value on the target image in accordance with the camera shake vector estimated by the second step at the position;
A fourth step of displaying a graph of a point spread function prepared in advance on a display screen;
A fifth step of editing the point spread function graph displayed on the screen according to an operation of a pointing device;
A camera shake image correction comprising: a sixth step of correcting a blur correction function used for blur correction in the third step in correspondence with the point spread function graph after editing in the fifth step Method. A first step of determining a coordinate transformation matrix representing rotation and translation of pixels on the target image due to camera shake;
A second step of estimating a camera shake vector at each pixel position on the target image using the coordinate transformation matrix determined in the first step;
A third step of performing blur correction on each pixel value on the target image in accordance with the camera shake vector estimated in the second step at the position;
A fourth step of displaying a graph of a point spread function due to camera shake of the target image on a display screen;
A fifth step of editing the point spread function graph displayed on the screen according to an operation of a pointing device;
A sixth step of generating a blur correction function corresponding to the point spread function after editing in the fifth step ;
The blur image correction method , wherein the blur correction function generated in the sixth step is used for blur correction in the third step . The first step includes the step of estimating the direction and size of camera shake at a plurality of positions on the target image based on the target image, and based on the direction and size of the camera shake estimated by the step, 5. The camera shake image correction method according to claim 1, wherein a coordinate transformation matrix representing rotation and translation of a pixel on a target image due to camera shake is determined. 6. The camera shake image correction method according to claim 1 , wherein in the third step, blur correction is performed by an interpolation calculation along a camera shake vector.   A computer-readable recording medium on which a program for causing a computer to execute each step of the camera shake image correction method according to any one of claims 1 to 6 is recorded. An imaging apparatus that obtains a digital image by imaging a subject with an imaging device,
A first means for determining a coordinate transformation matrix representing rotation and translation of pixels on a digital image due to camera shake;
Second means for estimating a camera shake vector at each pixel position on the digital image using the coordinate transformation matrix determined by the first means;
Third means for performing blur correction on each pixel value on the digital target image according to the hand movement vector estimated by the second means at the position;
Among the plurality of blur correction functions prepared in advance and corresponding to the point spread function of a plurality of typical camera shake patterns including a camera shake pattern having a non-uniform camera shake speed, the third means uses the blur correction function in the third means. An image pickup apparatus comprising: a fourth means for a user to select a blur correction function for use. An imaging apparatus that obtains a digital image by imaging a subject with an imaging device,
A first means for determining a coordinate transformation matrix representing rotation and translation of pixels on a digital image due to camera shake;
Second means for estimating a camera shake vector at each pixel position on the digital image using the coordinate transformation matrix determined by the first means;
Third means for performing blur correction on each pixel value on the digital image according to the hand movement vector estimated by the second means at the position;
A fourth means for estimating a point spread function due to camera shake in a digital image;
A fifth means for selecting a point spread function most similar to the point spread function estimated by the fourth means from a plurality of typical point spread functions prepared in advance;
A sixth means for selecting a blur correction function corresponding to the typical point spread function selected by the fifth means from a plurality of blur correction functions prepared in advance;
An image pickup apparatus, wherein the blur function selected by the sixth means is used for blur correction by the third means. An imaging apparatus that obtains a digital image by imaging a subject with an imaging device,
A first means for determining a coordinate transformation matrix representing rotation and translation of pixels on a digital image due to camera shake;
Second means for estimating a camera shake vector at each pixel position on the digital image using the coordinate transformation matrix determined by the first means;
Third means for performing blur correction on each pixel value on the digital image according to the hand movement vector estimated by the second means at the position;
A fourth means for displaying a graph of a point spread function prepared in advance on a display screen;
A fifth means for editing the graph of the point spread function displayed on the screen according to an operation of a pointing device;
An image pickup apparatus comprising: sixth means for correcting a blur correction function used for blur correction in the third means in correspondence with the point spread function graph after editing by the fifth means. An imaging apparatus that obtains a digital image by imaging a subject with an imaging device,
A first means for determining a coordinate transformation matrix representing rotation and translation of pixels on a digital image due to camera shake;
Second means for estimating a camera shake vector at each pixel position on the digital image using the coordinate transformation matrix determined by the first means;
Third means for performing blur correction on each pixel value on the digital image according to the hand movement vector estimated by the second means at the position;
A fourth means for displaying a graph of a point spread function due to camera shake of a digital image on a display screen;
A fifth means for editing the graph of the point spread function displayed on the screen according to an operation of a pointing device;
A sixth means for generating a blur correction function corresponding to the point spread function after editing by the fifth means;
An image pickup apparatus, wherein the blur correction function generated by the sixth means is used for blur correction in the third means.   The first means includes means for estimating the direction and size of camera shake at a plurality of positions on the digital image based on the digital image, and based on the direction and size of camera shake estimated by the means, The imaging apparatus according to claim 8, wherein a coordinate transformation matrix representing rotation and translation of a pixel on a digital image due to camera shake is determined.   13. The imaging apparatus according to claim 8, wherein in the third means, blur correction is performed by an interpolation calculation along a camera shake vector.

Images