JP3899144B2 - Image processing device - Google Patents

Description

【００２６】
ここで、ｉ，ｊは画像の空間座標値、α，βは定数である。最小自乗係数演算装置２７は自己相関関数演算装置２８により算出された自己相関関数と、関数形記憶装置２７に記憶されている関数との自乗誤差が最小となるように（１）式における係数α，βを定める。相関値演算装置３０は最小自乗係数演算装置２７により算出した係数α，βを用いた（１）式から、相関関数の値を求め、順次出力する。
【００２７】
図１における推定係数算出装置１５は、空間的相関情報算出装置１４により算出された相関係数の値を用いて自己回帰モデルに基づく画像情報推定式（２）の係数Ｃ_ijを算出する。この自己回帰モデルは、推定画像情報が近傍画像情報の線形和で表わされるとしたモデルであり、空間座標（ｍ，ｎ）における正しい画像情報ｆ（ｍ，ｎ）の推定値ｇ（ｍ，ｎ）は、
【００２８】
【数２】

と表わされる。ここでＤはｇ（ｍ，ｎ）を推定するために用いる画像情報ｆ（ｍ，ｎ）の空間座標の範囲（近傍領域）、ｃ_ijは推定画像情報の空間座標から相対的に座標値がｍｎ方向にそれぞれｉ，ｊに位置する画像情報ｆ（ｍ−ｉ，ｎ−ｊ）の推定係数を示す。このとき、推定値ｇ（ｍ，ｎ）と正しい画像ｆ（ｍ，ｎ）との自乗誤差Ｅrrは、
【００２９】
【数３】

と表わされる。ここで、Ａは領域抽出画像の範囲に対応する。自乗誤差Ｅrrを最小にする推定係数ｃ_ijはＥrrをｃ_ijで偏微分することにより近似的に、
【００３０】
【数４】

【００３１】
ここで、Ｒ（ｉ−ｐ，ｊ−ｐ）、Ｒ（ｐ，ｑ）は空間的相関情報算出装置１４により算出される既知の値であるから、式（４）は、推定係数ｃ_ijを未知数とする連立方程式となる。推定係数算出装置１５は、推定係数範囲記憶装置１６に記憶された近傍領域Ｄの情報を用いて（４）式の関係から推定係数ｃ_ijを算出する。
【００３２】
図１における画像信号推定装置１７は、推定係数算出装置１５により算出された推定係数ｃ_ijを用いて式（２）で表わされる自己回帰モデルの画像情報ｆ（ｍ，ｎ）の推定値ｇ（ｍ，ｎ）を算出する。
【００３３】
図１における補正画像記憶装置１８は、画像信号推定装置１７により順次算出された推定色情報の各推定位置の推定画像情報を順次記憶する。
そして、領域抽出装置１２による領域抽出画像の決定から推定画像情報の補正画像記憶装置１８への記憶までの処理をＧ，Ｂ，Ｒの色情報について全推定位置で行い、全ての画素をＲＧＢ色情報を有する画像情報として補正画像記憶装置１８に記憶する。なお、上記した第１実施形態の構成は、各種変形、変更が可能である。例えばデジタルカメラのフィルタはストライプ状に配置してもよく、また、原色系の分光特性を有するもの以外にＣＭＹの補色系のものを用いてもよい。さらに、ＲＧＢのようにフィルタの種類が３枚ではなく任意の枚数のフィルタからなるものについても適用可能である。
【００３４】
図５は本発明の第２実施形態が適用される画像処理システムの構成を示す図である。図５において、領域抽出装置１２により第１実施形態と同様の方法で抽出された領域抽出画像は領域抽出装置１２内の領域抽出記憶装置２６から図５における構造情報抽出装置３１に読み込まれる。構造情報抽出装置３１は、領域抽出画像のエッジ強度Ｉを算出する。エッジ強度Ｉは領域抽出画像を３×３画素の矩形画像とすると、次式で定義される。
【００３５】
【数５】

【００３６】
ここで、ａ〜ｉは図６に示す領域抽出画像の画素位置における画素値を表わし、Ｘ，Ｙはそれぞれｘ方向，ｙ方向の微分値に相当する。エッジ強度Ｉは用いる画像の濃度レンジに応じて０〜１に正規化される。任意の形状の領域抽出画像についても適当にエッジ強度Ｉを再定義することにより、エッジ強度Ｉの算出が可能である。
【００３７】
図５に示す空間的相関情報算出装置１４は図７に示すような構成を有する。第１実施形態と同様に、最小自乗係数演算装置２９により相関関数の関数形の係数α，βを算出する。最小自乗係数補正装置３２はこの係数α，βを、構造情報抽出装置３１により算出された抽出領域のエッジ強度Ｉと、補正係数記憶装置３３に予め記憶された補正係数Ｐ，Ｑとを用いて次式によりα′，β′に補正する。
【００３８】
α′＝α（ＰＩ＋１）
β′＝β（ＱＩ＋１） （７）
（７）式の補正によりα，βは、エッジ部ではＩ＝１となり、それぞれＰ，Ｑ％増加される。補正された係数α′，β′を用いて相関値演算装置３０により相関値が計算される。以下第１実施形態と同様の処理により推定画像情報が求められ、補正画像記憶装置１８に記憶される。
【００３９】
なお、上記した第２実施形態の構成は、各種変形、変更が可能である。例えばＸ，Ｙの値を用いてエッジ強度のみならずエッジの方向をも算出し、相関関数の補正を行うことも可能である。また、エッジの強度、向きなどに応じて相関関数の算出方法を選択することも可能である。
【００４０】
図８は本発明の第３実施形態が適用される画像処理システムの構成を示す図である。図８において、構造情報抽出装置３１は上記した第２実施形態と同様の方法により領域抽出装置１２により抽出された領域抽出画像のエッジ強度Ｉを算出する。推定方法切り替え装置３３は、構造情報算出装置３１により算出されたエッジ強度Ｉの値から推定位置における画像情報の推定方法を選択する。すなわち、
Ｉ＞Ｔ
の場合は推定方法として第一の推定方法を選択する。また、
Ｉ≦Ｔ
の場合は第二の推定方法を選択する。Ｔは実験的に予め定めた定数である。
【００４１】
第一の推定方法は上記した第１実施形態と同様の自己回帰モデルに基づき、領域抽出画像の自己相関関数情報を用いて画像情報を推定する方法である。また、第二の推定方法は簡易線形補間装置３５により、予め定められた補間位置近傍の画素情報の線形和として補間位置の画素情報を推定するものである。
推定位置の推定画素情報ｇ（ｍ，ｎ）は、推定位置を中心とする３×３の領域内の画像情報ｆ（ｍ，ｎ）の平均値として次式で算出される。
【００４２】
【数６】

【００４３】
ここで、Ｅは（ｍ，ｎ）を中心とする（ｍ，ｎ）以外の３×３の矩形領域を表わす。簡易線形補間装置３５で（８）式により算出された推定画像情報は、補正画像記憶装置１８に記憶される。
【００４４】
なお、上記した第３実施形態の構成は、各種の変形、変更が可能である。例えば、第二の画像信号の推定方法は、線形補間に基づく方法に限らず目的に応じて任意の方法を用いることができる。また、切り替え可能な推定方法の数は２つに限らず複数の推定方法を用意することで複数の切り替え方法が選択可能となる。また、切り替え方法は、構造解析によるものに限らず、処理装置にマニュアルで設定することも可能である。
【００４５】
図９は本発明の第４実施形態が適用される画像処理システムの構成を示す図である。図９において、上記した第１実施形態と同様の方法により領域抽出装置１２により抽出された領域抽出画像は構造情報抽出装置３１により解析される。
【００４６】
図１０は構造情報抽出装置３１の構成を示す図である。図１０において、パターン記憶装置３８には領域抽出画像と同サイズのパターン画像が予め数十種類記憶されている。パターンマッチング演算装置３９はこのパターン画像を用いて、領域抽出装置１２内の領域抽出記憶装置２６からの領域抽出画像とのパターンマッチングを行う。そして、パターンマッチング演算装置３９において算出された相関領域画像とパターン画像との類似度を類似度評価演算装置４０にて評価値として算出する。
【００４７】
図９におけるメモリカード３６には、パターン画像の種類に対応して、推定係数がテーブルデータとして与えられている。メモリカード３６に記憶されている推定係数は、予め構造情報抽出装置３１に記憶されているパターン画像について上記した第１実施形態と同様の処理により計算された自己相関関数から算出された値である。
【００４８】
推定係数選択装置３７は構造情報抽出装置３１により解析された領域抽出画像と類似する数種類のパターン画像に対応する推定係数をメモリカード３６内のテーブルデータより読み出し、読み出した推定係数を類似度評価演算装置４０により算出された評価値を用いて算出する。画像信号推定装置１７は推定係数選択装置３７により算出された推定係数を用いて上記した第１実施形態と同様の方法により推定画像情報を求め、補正画像記憶装置１８に記憶する。
【００４９】
なお、上記した第４実施形態の構成は、各種変形、変更が可能である。例えば、メモリカード３６内のテーブルデータは、領域抽出画像のパターンに対応した推定係数を与えるのではなく、相関関数の関数形のパラメータに対応した推定係数を与えるものとすることも可能である。また、メモリカード３６は、被撮影画像の種類に応じて入れ替え可能か、もしくは複数の異なるテーブルデータを保存しておき被撮影画像の種類に応じて選択可能とする。
【００５０】
以下に本発明の第５実施形態を説明する。上記した第１実施形態における推定位置決定装置１１は、第５実施形態では、一画素のみではなく５×５の矩形の推定領域を定めるものとする。推定領域の決定は、推定領域内に含まれる画素が重複しないようにＲＧＢ各色の情報について画像内全体について行われる。
【００５１】
領域抽出装置１２は、推定領域を中心とする１０×１０の矩形領域を画像情報から領域抽出画像として抽出する。空間的相関情報算出装置１４及び推定係数算出装置１５は第１実施形態と同様の処理を行い、画像信号推定装置１７は内部の制御装置により推定領域内の推定色情報の欠落した画素についてのみ、推定係数算出装置１５により算出された推定係数を用いて画像情報を推定し、補正画像記憶装置１８に記憶する。
【００５２】
なお、上記した第５実施形態は、各種変形、変更が可能である。例えば、推定領域は予め定めた領域ではなく、第２実施形態と同様の構造情報抽出装置３１により推定係数が一定とみなせる領域を推定して推定領域とすることも可能である。この場合、領域抽出装置１２により抽出された領域抽出画像を構造情報抽出装置３１により解析し、推定位置決定装置１１により推定領域を定める。それ以降の処理は上記した第５実施形態と同様である。
【００５３】
図１１は本発明の第６実施形態が適用される画像処理システムの構成を示す図である。図１１において、構造情報抽出装置３１は上記した第１実施形態と同様の方法により領域抽出装置１２によって抽出された領域抽出画像の微分画像を作成する。相関領域抽出装置４１は自己相関関数の値が一定とみなせる領域を推定し、新たに相関領域画像として抽出する。領域抽出装置１２において抽出される領域抽出画像は、予め定めた矩形領域画像とする。相関領域抽出装置４１は、領域抽出装置１２により抽出された領域抽出画像の微分画像を作成し、微分画像濃度の分布の類似性から自己相関関数が一定とみなせる領域を推定する。推定された位置の集合を包含するような矩形領域を新たに相関領域画像とする。以降は上記した第１実施形態と同様の方法により相関領域画像を用いて画像情報を推定し、補正画像記憶装置１８に記憶する。
【００５４】
なお、上記した第６実施形態の構成は、各種変形、変更が可能である。例えば、相関領域画像の作成方法は微分画像に基づく方法に限られず、実際に抽出領域画像の相関関数を計算し、その類似性から相関領域を定めることも可能である。
【００５５】
図１２は本発明の第７実施形態が適用される画像処理システムの構成を示す図である。図１２において、推定位置選択装置１００は、上記した第１実施形態と同様の方法により推定位置決定装置１１により定められた推定位置のうち、画像を予め定めた矩形領域に分割した推定領域の中心点に最も近い位置のみを選択する。推定位置選択装置１００により選択された選択推定位置における自己相関関数の関数形の係数を空間的相関情報算出装置１４により算出する。以上の処理を画像全体について行い、全ての選択推定位置における相関関数の関数形の係数を求め、相関係数記憶装置４２に記憶する。
【００５６】
空間的相関情報補間装置４３は、記憶された選択推定位置における相関関数の関数形の係数を用いて、全ての位置における相関関数の関数形の係数を補間演算により推定する。補間演算は推定位置を囲む３つの選択位置における相関係数を用いることを基本として補間演算を行なう。相関関数の関数形の係数を求めた各推定位置について上記した第１実施形態と同様の方法により画像情報を推定し、補正画像記憶装置１８に記憶する。
【００５７】
なお、上記した第７実施形態の構成は、各種変形、変更が可能である。例えば、推定位置選択装置１００は、予め定めた矩形領域より選択推定位置を選択するのではなく、上記した第２実施形態と同様の構造情報抽出装置３１により解析した画像から相関関数が一定とみなせる領域を推定して定めることも可能である。また、選択推定位置における相関関数の関数形の係数から他の位置の相関関数の関数形の係数を補間するのではなく、選択推定位置における推定係数から他の位置の推定係数を補間することも可能である。
【００５８】
図１３は本発明の第８実施形態が適用される画像処理システムの構成を示す図である。図１３において、本発明の画像処理装置６５は、図２に示すような配置のＲ，Ｇ，Ｂの色フィルタを有する単板式デジタルカメラ４４で撮像された画像情報と、コンピュータ４５内部に記憶された単板式デジタルカメラ４４のフィルタ配置情報より輝度信号を生成する。
【００５９】
単板式デジタルカメラ４４により撮像された画像情報ｇ（ｍ，ｎ）は、カラー画像入力端子４６を経由し、画像処理装置６５内の画像メモリ４７に伝送、記憶される。また、画像情報ｇ（ｍ，ｎ）が画像メモリ４７へ記憶されるのと同時に、コンピュータ４５に予め記憶された単板式デジタルカメラ４４のフィルタ配置情報が、色成分マスクＨ_c （ｍ，ｎ）（ｃ＝ｒ，ｇ，ｂ）に関する情報としてマスク情報入力端子５４を経由して色成分マスクバッファ５５に伝送、記憶される。
【００６０】
色成分マスクバッファ５５はＲ成分マスクバッファ５５ａ、Ｇ成分マスクバッファ５５ｂ、Ｂ成分マスクバッファ５５ｃから構成されており、それぞれ対応する色成分のマスク情報が記憶される。色成分マスクＨ_c （ｍ，ｎ）は、単板式デジタルカメラ４４の画素位置（ｍ，ｎ）の色フィルタがマスクの色成分ｃの場合は１、その他の色成分の場合は０の値を持つ。
【００６１】
図１４に図２のフィルタ配置に対応する色成分マスクＨ_r （ｍ，ｎ）、Ｈ_g （ｍ，ｎ）の例を示す。またマスク情報の伝送と同時に予めコンピュータ４５内に記憶されている輝度信号演算係数β_c （ｃ＝ｒ，ｇ，ｂ）がマスク情報入力端子５４を経由して画像処理装置６５内の輝度ウエイトバッファ５６に伝送、記憶される。輝度Ｙ（ｍ，ｎ）は、輝度信号演算係数β_c と、各画素においてＲ，Ｇ，Ｂ全色情報を有する画像情報ｆ_c （ｍ，ｎ）（ｃ＝ｒ，ｇ，ｂ）とを用いて、
【００６２】
【数７】

と表わされるものとなる。
【００６３】
ブロックＩ／Ｏ部４８は、色成分マスクバッファ５５への情報伝送が完了した後、色成分マスクバッファ５５の情報を用いた制御装置５２の制御により部分ブロック画像ｇ_c （ｍ，ｎ）（ｃ＝ｒ，ｇ，ｂ）を画像メモリ４７内の画像情報ｇ（ｍｎ）からブロックバッファ４９へ書き込む。ブロックバッファ４９はＲ成分ブロックバッファ４９ａ、Ｇ成分ブロックバッファ４９ｂ、Ｂ成分ブロックバッファ４９ｃから成り、各色成分ブロックバッファには、各色成分ブロックバッファの色成分に対応する色情報を有する画素以外のｇ（ｍ，ｎ）を０とした色成分ブロック画像ｇ_c （ｍ，ｎ）が記憶される。画像メモリ内の画像情報ｇ（ｍ，ｎ）は色成分マスクＨ_c （ｍ，ｎ）及び色成分画像ｆ_c （ｍ，ｎ）を用いて、
【００６４】
【数８】

と表わされる。
【００６５】
以上の処理により、単板式デジタルカメラ４４及びコンピュータ４５から画像処理装置６５への情報伝送が完了し、以下、制御装置５２による制御のもとで輝度信号推定処理が各画素ごとに順次行われる。輝度信号Ｙ（ｍ，ｎ）は、ＡＲモデルを拡張した次式により推定される。
【００６６】
【数９】

【００６７】
ここではＤはＹ（ｍ，ｎ）を推定するために用いる画像メモリ内の画像情報ｇ（ｍ，ｎ）の空間座標の範囲（近傍領域）、ｃ_ijは推定画像情報の空間座標から相対的に座標値がｍ，ｎ方向にそれぞれｉ，ｊに位置する画像情報ｇ（ｍ−ｉ，ｎ−ｊ）の推定係数を示す。このとき、（１１）式による推定値と正しい輝度成分Ｙ（ｍ，ｎ）との所定の領域Ａにおける自乗誤差Ｅrrは、
【００６８】
【数１０】

を得る。
【００６９】
（１４）式より、近似的にｃ_klを求め、（１１）式より、輝度信号Ｙ（ｍ，ｎ）を推定する。
まず、制御装置５２は輝度推定の推定画素の位置を定める。
【００７０】
【数１１】

を計算し、記憶する。例えば、単板式デジタルカメラ４４のＲ成分マスクＨ_r （ｍ，ｎ）とＧ成分マスクＨ_g （ｍ，ｎ）が図１４に示すような５×５画素の矩形領域の場合、ＲＧ色間相互相関Ｒ^h _rg（ｉ，ｊ）は図１５のようになり、
Ｓ_pqは、
Ｓ_pq＝｛（−２，−１，４），（−２，１，４），（−１，−２，４），
（−１，０，６），（−１，２，４），（０，−１，６），（０，１，６），（１，−２，４），（１，０，６），（１，２，４），（２，−１，４），（２，１，４）｝となる。また、マスク相関計算装置５７は、各色成分マスクＨ_c （ｍ，ｎ）の値が１である数、すなわち対応する色フィルタを有する画素数α_c を記憶する。例えば図１４に示したＲ成分マスクＨ_r （ｍ，ｎ）とＧ成分マスクＨ_g （ｍ，ｎ）の場合、α_r は９、α_g は１２となる。
【００７１】
色間相関計算装置５０は部分色間相関計算部５１、相関値補間部５３、相関関数パラメータ記憶部５２から構成されている。
部分色間相関計算部５１は各色成分ブロックバッファ４９ａ，４９ｂ，４９ｃから制御装置５２の制御により各色成分の所定領域Ａの部分画像を読み込み、Ｒ，Ｇ，Ｂの重複しない組み合わせ
【００７２】
【数１２】

に対して、次式で定義される部分画像間の相互相関Ｒ_pq′（ｉ，ｊ）
を計算する。
【００７３】
【数１３】

【００７４】
ここで、ｇ_p （ｍ，ｎ）、ｇ_q （ｍ−ｉ，ｎ−ｊ）は色フィルタの対応していない画素における情報の欠落があるため、一般には次式で定義される相関値Ｒ_pq（ｉ，ｊ）、
【００７５】
【数１４】

とは一致しない。ただし、ｆ_p （ｍ，ｎ）、ｆ_q （ｍ−ｉ，ｎ−ｊ）は、それぞれ全画素において色情報ｐ，ｑを有する画像情報を表わす。そこで、部分色間相関計算部５１は、マスク相関計算装置５７で計算された集合Ｓ_pqを用いることによりＲ_pq′（ｉ，ｊ）を補正し、Ｒ_pq（ｉ，ｊ）の推定値を算出する。集合Ｓ_pqに含まれる点の座標（ｉ，ｊ）はＲ^h _pq（ｉ，ｊ）がゼロとならない座標値であるから、（１７）式の右辺には０にならない項ｇ_p （ｍ′，ｎ′）ｇ_q （ｍ′−ｉ，ｎ′−ｊ）が存在し、その値は正しい相関値を与える（１８）式における項の値ｆ_p （ｍ′，ｎ′）ｆ_q （ｍ′−ｉ，ｎ′−ｊ）と一致する。ただし、そのような項はブロックの全画素数に対し、集合Ｓ_pqの（ｉ，ｊ）に対応する項に限られる。
【００７６】
例えば図１５に示した例の場合、（１７）式の２５個の項に対して正しい値が得られる項は最大６項であり、その他の項は０となる。そこで（１７）式で得られる値Ｒ_pq′（ｉ，ｊ）と、正しい相関値Ｒ_pq（ｉ，ｊ）との比が近似的に全領域において１の値をもつマスク間の相関値Ｒ₀ ^h _pq（ｉ，ｊ）とマスク相関値Ｒ^h _pq（ｉ，ｊ）との比になると仮定し、Ｒ_pq（ｉ，ｊ）の推定値Ｒ″_pq（ｉ，ｊ）を次式より求める。
Ｒ″_pq(i,j) ＝Ｒ′_pq(i,j) Ｒ₀ ^h _pq(i,j) ／Ｒ^h _pq(i,j) （19）
また、画像メモリ４７内の画像情報ｇ（ｍ，ｎ）を用いて、
【００７７】
【数１５】

を計算して内部に記憶する。
【００７８】
相関値補間部５３は部分色間相関計算部５１において計算、補正された相関関数Ｒ″_pq（ｉ，ｊ）の補間を行い、近似相関関数として出力する。Ｒ″_pq（ｉ，ｊ）は、画素情報の欠落した画像間の自己相関であるため、図１６に示すように離散的位置にのみデータを有する。相関値補間部５３は、相関関数パラメータ記憶部５２に記憶されている近似関数形、及びそのパラメータを用いて相関関数Ｒ″_pq（ｉ，ｊ）を既知の関数形に最小自乗近似する。近似されたＲ″_pq（ｉ，ｊ）は相関値補間部５３内部に記憶される。
【００７９】
補正係数算出装置６４は相関値加算部５８、相関値バッファ５９、及び補正係数計算部６０から成る。
相関値加算部５８はマスク相関計算装置５７により算出されたα_c 、輝度ウエイトバッファ５６から得られた輝度信号演算係数β_c 、及び相関値補間部５３より得られた近似されたＲ″_pq（ｉ，ｊ）を用いて次式を計算、記憶する。
【００８０】
【数１６】

【００８１】
相関値バッファ５９は、部分色間相関計算部５１において算出された（２０）式、及び相関値加算部５８において算出された（２１）式の値を記憶する。
補正係数計算部６０は、相関値バッファ５９に記憶された（２０）式、（２１）式の値を用いて、（１４）式を満たすｃ_klを算出し、記憶する。ここで、（１４）式の左辺において次式の近似、
【００８２】
【数１７】

を用いた。
【００８３】
輝度成分復元装置６１は画像メモリ４７に記憶された画像情報ｆ（ｍ，ｎ）と補正係数計算部６０に記憶された補正係数ｃ_klを用いて、（１１）式から推定画素の輝度成分Ｙ（ｍ，ｎ）を算出し、出力画像メモリ６２に記憶する。
【００８４】
全ての画素について以上の処理が行われる。補正された画像は輝度画像出力端子６３を経由して外部への読み出しが可能となる。
なお、上記した実施形態は各種の変形、変更が可能である。例えば、推定画素は一画素ごとに行うとしたが、近似精度がよいと判断された場合はブロック全画素の輝度成分を求めることも可能である。また、個々の色間の相関を計算する方法としては様々な相関推定手法が利用可能であり、先見的情報を利用することも可能である。また、ここでは輝度信号の推定を行ったが特に輝度に限らず色信号の線形和で計算される任意の画像情報が復元可能である。また、この例では単板モザイク色フィルタＣＣＤにより撮像された画像の補正を取り上げたが、
【００８５】
【数１８】

とすることで、例えば色収差を含んだ画像の補正などにも適用できる。また、色間の相互相関は、観測画像から得られる情報を基に適当な関数形を当てはめるなどして求めてもよい。また、対象の統計的性質があらかじめ判っている場合はそのような情報を利用することも有効である。
【００８６】
なお、上記した具体的実施形態には以下の構成を有する発明が含まれており、各発明に対応する実施形態と効果は次の通りである。
（１） 画像信号を生成する撮像装置に関する情報に基づき、推定すべき画像信号の位置を決定する推定位置決定手段と、
この推定位置決定手段から得られた上記画像信号の位置を包含する所定サイズの局所領域を抽出する領域抽出手段と、
この領域抽出手段から得られた局所領域に関する空間的相関情報を算出する空間的相関情報算出手段と、
この空間的相関情報算出手段によって算出された空間的相関情報に基づき、上記推定位置決定手段から得られた上記画像信号の位置に対応する画像信号を推定する画像信号推定手段と、
を具備することを特徴とする画像処理装置。
（２） 上記推定位置決定手段は、上記撮像装置の色フィルタの配置情報に基づき推定すべき画像信号の位置を決定することを特徴とする構成（１）に記載の画像処理装置。
（３） 上記領域抽出手段は、ｎ，ｍを２以上の整数とするｎ×ｍ画素のブロック領域を抽出することを特徴とする構成（１）または（２）に記載の画像処理装置。
（４） 上記空間的相関情報算出手段は、上記局所領域の自己相関関数に基づき上記空間的相関情報を算出することを特徴とする構成（１）〜（３）のいずれか１つに記載の画像処理装置。
（５） 上記空間的相関情報算出手段は、上記局所領域の自己相関関数を指数関数で近似することに基づき上記空間的相関情報を算出することを特徴とする構成（１）〜（４）のいずれか１つに記載の画像処理装置。
（６） 上記画像信号推定手段は、上記空間的相関情報算出手段から算出された空間的相関情報に基づき、上記領域抽出手段から得られた局所領域に関する推定係数を算出する推定係数算出手段を具備し、この推定係数算出手段は、自己回帰モデルに基づき上記局所領域の推定係数を算出することを特徴とする構成（１）〜（５）のいずれか１つに記載の画像処理装置。
（７） 上記領域抽出手段から得られた局所領域に関する構造情報を抽出する構造情報抽出手段と、
この構造情報抽出手段で抽出された構造情報に基づき、上記空間的相関情報算出手段による空間的相関情報算出の方法を変更する変更手段と、
をさらに具備することを特徴とする構成（１）に記載の画像処理装置。
（８） 上記構造情報抽出手段は、明度情報に基づき上記構造情報を抽出することを特徴とする構成（７）に記載の画像処理装置。
（９） 上記構造情報抽出手段は、微分処理、パターンマッチに基づき上記構造情報を抽出することを特徴とする構成（７）に記載の画像処理装置。
（１０） 画像信号を生成する撮像装置に関する情報に基づき、推定すべき画像信号の位置を決定する推定位置決定手段と、
この推定位置決定手段から得られた位置を包含する所定サイズの局所領域を抽出する領域抽出手段と、
この領域抽出手段から得られた局所領域に関する構造情報を抽出する構造情報抽出手段と、
上記領域抽出手段から得られた局所領域に関する空間的相関情報から推定係数を算出し、上記推定位置決定手段から得られた上記画像信号の位置に対応する画像信号を推定する第一画像信号推定手段と、
上記領域抽出手段から得られた局所領域において、上記推定位置決定手段から得られた上記画像信号の位置に対応する他の画像信号又はその位置の周囲に位置する画像信号を用いて、上記画像信号の位置に対応する画像信号を推定する第二画像信号推定手段と、
上記構造情報抽出手段で抽出された上記構造情報に基づき、上記第一画像信号推定手段と上記第二画像信号推定手段を切り換える切換手段と、
を具備することを特徴とする画像処理装置。
（１１） 上記第二画像信号推定手段は、上記領域抽出手段から得られた局所領域において、上記推定位置決定手段から得られた上記画像信号の位置の周囲の位置の画像信号からの線形補間、２次、高次関数による補間によって推定を行なうことを特徴とする構成（１０）に記載の画像処理装置。
（１２） 上記第二画像信号推定手段は、上記領域抽出手段から得られた局所領域において、上記推定位置決定手段から得られた上記画像信号の位置に対応する他の画像信号との相関により、上記画像信号を推定することを特徴とする構成（１０）に記載の画像処理装置。
（１３） 画像信号を生成する撮像装置に関する情報に基づき、推定すべき画像信号の位置を決定する推定位置決定手段と、
この推定位置決定手段から得られた前記画像信号の位置を包含する所定サイズの局所領域を抽出する領域抽出手段と、
この領域抽出手段から得られた上記局所領域に関する構造情報を抽出する構造情報抽出手段と、
複数の所定の構造情報に対応する推定係数が保存された推定係数保存手段と、
上記構造情報抽出手段から抽出された上記構造情報に基づき、上記推定係数保存手段から推定係数を選択する推定係数選択手段と、
を具備することを特徴とする画像処理装置。
（１４） 上記推定係数保存手段は、磁気ディスク、メモリカードなどの交換可能な記録媒体からなることを特徴とする構成（１３）に記載の画像処理装置。
（１５） 上記空間的相関情報算出手段から算出された上記空間的相関情報を上記推定位置決定手段から得られた複数の位置で共有して使用させる制御手段をさらに有することを特徴とする構成（１）〜（６）のいずれか１つに記載の画像処理装置。
（１６） 上記領域抽出手段から得られた上記局所領域に関する構造情報を抽出する構造情報抽出手段と、
この構造情報抽出手段で抽出された構造情報に基づき、上記領域抽出手段から得られた局所領域から新たな局所領域を抽出する第二領域抽出手段と、
をさらに具備することを特徴とする構成（１）〜（６）のいずれか１つに記載の画像処理装置。
（１７） 上記第二領域抽出手段は、自己相関関数が近似的に一定とみなせる領域を抽出することを特徴とする構成（１６）に記載の画像処理装置。
（１８） 画像信号を生成する撮像装置に関する情報に基づき、推定すべき画像信号の位置を決定する推定位置決定手段と、
この推定位置決定手段から得られた複数の前記画像信号の位置を所定間隔以上で疎に再抽出する推定位置再抽出手段と、
この推定位置再抽出手段から得られた前記画像信号の位置を包含する所定サイズの局所領域を抽出する領域抽出手段と、
この領域抽出手段から得られた上記局所領域に関し、上記推定位置再抽出手段で再抽出された前記画像信号の位置に対応する上記局所領域の空間的相関情報を算出する空間的相関情報算出手段と、
この空間的相関情報算出手段から算出された上記空間的相関情報に基づき、上記推定位置決定手段から得られた全ての前記画像信号の位置に関する空間的相関情報を補間する空間的相関情報補間手段と、
この空間的相関情報補間手段によって補間された空間的相関情報に基づき、上記推定位置決定手段から得られた前記画像信号の位置に対応する画像信号を推定する画像信号推定手段と、
を具備することを特徴とする画像処理装置。
（１９） 複数の色信号からなる画像信号を生成する撮像装置に関する情報に基づき、推定すべき画像信号の位置を決定する推定位置決定手段と、
この推定位置決定手段から得られた上記画像信号の位置を包含する所定サイズの局所領域を複数の色信号を含む画像信号から抽出する領域抽出手段と、
この領域抽出手段から得られた局所領域に関する空間的相関情報を算出する空間的相関情報算出手段と、
上記領域抽出手段から得られた局所領域に関する色相関情報を算出する色相関情報算出手段と、
上記空間的相関情報算出手段によって算出された空間的相関情報と上記色相関情報算出手段から算出された色相関情報に基づき、上記推定位置決定手段から得られた上記画像信号の位置に対応する画像信号を推定する画像信号推定手段と、
を具備することを特徴とする画像処理装置。
（２０） 上記色相関情報算出手段は、上記所定サイズの局所領域に関する複数の色信号の空間的相関に対する先見的な情報を保持する先見的色相関情報保持手段を有することを特徴とする構成（１９）に記載の画像処理装置。
（２１） 上記色相関情報算出手段は、上記所定サイズの局所領域に関する複数の色信号の空間的相関情報をパラメータ表記された所定の関数に近似する関数近似手段を有することを特徴とする構成（１９）に記載の画像処理装置。
（２２） 上記領域抽出手段は、ｎ，ｍを２以上の整数とするｎ×ｍのブロック領域を抽出することを特徴とする構成（１９）に記載の画像処理装置。
（構成（１）〜（６）に対応する発明の実施の形態）
これらの発明は上記した第１実施形態に対応する。すなわち、構成中の撮像装置は図１の単板式デジタルカメラ４に、推定位置決定手段は図１の推定位置決定装置１１に、領域抽出手段は図１の領域抽出装置１２に、空間的相関情報算出手段は図１の空間的相関情報算出装置１４に、推定係数算出手段は図１の推定係数算出装置１５に、画像信号推定手段は図１の画像信号推定装置１７に対応する。また、撮像装置の色フィルタの配置情報は撮像装置情報記憶装置９に記憶された図２のＣＣＤ上の色フィルタ配置情報に対応し、ｎ×ｍ画素のブロック領域は３×３のブロック領域に対応する。
【００８７】
図１において、単板式デジタルカメラ４で撮像され、内部の画像メモリ７に記憶された画像情報は画像処理装置１９内の画像情報記憶装置１０に伝送、記憶される。これと同時に単板式デジタルカメラ４内部のＲＯＭ６に記憶されている単板式デジタルカメラ４のＣＣＤ３上に配置された色フィルタ２の配置情報が、撮像装置情報記憶装置９に伝送、記憶される。撮像装置情報記憶装置９に記憶されたフィルタ配置情報から、推定位置決定装置１１により推定すべき色情報に対応する画像情報記憶装置１０内の画像情報の全画素位置を算出、記憶する。記憶された一画素の位置について、領域抽出装置１２により相関範囲記憶装置１３に記憶された領域に応じて、画像情報記憶装置１０から領域抽出画像を抽出する。空間的相関情報算出装置１４は領域抽出画像に関する相関情報を算出し、推定係数算出装置１５は相関情報を用いて、モデルに基づく画像信号を推定する推定式の推定係数を算出する。画像信号推定装置１７は推定係数を用いて画像信号の推定値を算出し、補正画像記憶装置１８に記憶する。以上の処置を推定位置決定装置１１内部に記憶された全ての推定位置について行うことにより、全画像内で各画素がＲＧＢ色情報を有する画像として補正画像記憶装置１８に記憶される。
（構成（１）〜（６）の効果）
推定すべき画像信号の局所領域に関する空間的相関情報を用いて画像構造に依存した適応的推定を行なうようにしたので、任意の構造の画像に対して最適な推定を行なうことができる。例えば、推定位置近傍の領域抽出画像の自己相関情報を用いて、自己回帰モデルによる画像情報推定を行うことにより、推定の誤差が最少になるような推定係数を自動的に算出することができ、これによって、任意の画像構造に依存した画像処理を行なうことができる。
（構成（７）〜（９）に対応する発明の実施の形態）
この発明には上記した第２実施形態が対応する。構成中の構造情報抽出手段は図５の構造情報抽出装置３１に、変更手段は空間的相関情報算出装置１４内の最小自乗係数補正装置３２（図７）にそれぞれ対応する。この発明では、構造情報抽出装置３１の解析により得られた推定位置近傍の画像構造に応じて、空間的相関情報算出装置１４により算出された相関係数を補正することにより、より正確な空間的相関情報を算出する。構造情報抽出装置３１は、領域抽出装置１２により抽出された領域抽出画像のエッジ強度を解析する。空間的相関情報算出装置１４内の最小自乗係数補正装置３２は、領域抽出画像のエッジ強度に従い最小自乗係数演算装置２９（図７）により算出された相関係数を補正する。補正はエッジ強度が大きいほど係数を大きくする。以下、第１の実施形態と同様の処理により画像情報を推定、記憶する。
（構成（７）〜（９）の効果）
空間的相関情報算出装置１４により算出された相関関数の形状は、領域抽出画像の補間処理や、関数形の近似のため必ずしも正確に求めることができない。そこで、特に推定の難しいエッジ領域において相関関数の形状を、経験的に良好な結果が得られるように補正することで、より精度の高い推定ができる。
（構成（１０）〜（１２）に対応する発明の実施の形態）
この発明には上記した第３実施形態が対応する。構成中の第一画像信号推定手段は上記した自己回帰モデルに基づき領域抽出画像の自己相関関数情報を用いて画像情報を推定する手段に、第二画像信号推定手段は上記した簡易線形補間装置３５により予め定められた補間位置近傍の画素情報の線形和として補間位置の画素情報を推定する手段に、切換手段は上記した図８の推定方法切り替え装置３３にそれぞれ対応する。
【００８８】
この発明では、構造情報抽出装置３１の解析により得られた推定位置近傍の画像構造に応じて、平坦領域での簡易線形補間とエッジ領域での自己回帰モデルに基づく方法とを使いわける。構造情報抽出装置３１は領域抽出装置１２により抽出された領域抽出画像の構造を解析し、推定方法切り替え装置３３において推定方法を選択するために用いるパラメータを算出する。推定方法切り替え装置３３は、構造情報解析の結果から領域抽出画像のエッジ強度が所定の値よりも大きい場合に第一の画像信号推定手段、小さい場合に第二の画像信号推定手段を用いて画像信号を推定する。
（構成（１０）〜（１２）の効果）
領域抽出画像の構造に応じて画像信号推定方法を切り替えることにより、推定精度を維持しながら計算量の少ない画像信号推定を行なうことができる。例えば推定が容易な画像構造が平坦な領域では、計算量の多い自己回帰モデルに基づく推定方法を用いるのではなく、計算量の比較的少ない簡易線形補間装置３５による推定を行なう。また、構造情報解析装置３１により得られた情報を第一の画像信号推定方法の処理補正に組み込むか、もしくは第二の画像信号推定方法として用いることで、より高精度な推定が可能となる。
（構成（１３）、（１４）に対応する発明の実施の形態）
この発明には上記した第４実施形態が対応する。構成中の推定係数保存手段は図９のメモリカード３６に、推定係数選択手段は推定係数選択装置３７にそれぞれ対応する。この発明では、上記した第１実施形態において計算量の多い推定係数の演算を予め複数の所定のパターンについて求めておき、推定位置近傍の画像構造と推定係数のテーブルを作成、記憶しておく。画像情報の推定の際にこのテーブル情報を利用することで演算を高速化させる。図９の構造情報抽出装置３１は、領域抽出画像とパターン記憶装置３８内に記憶された所定のパターン画像とのパターンマッチング演算を行ない、類似度を評価する。そして、類似度の高いパターンに対応する推定係数をメモリカード３６に記憶されているテーブルデータより求め、類似度の評価値に基づく適当な重みづけにより推定係数を算出する。求めた推定係数を用いて画像信号推定装置１７により画像信号を推定し、補正画像記憶装置１８に記憶する。
（構成（１３）、（１４）の効果）
複数の所定の構造情報に対応する推定係数を保存しておくことで、領域抽出画像からの推定係数を算出するための演算を行なうことなく推定係数を求めることができる。このように計算量の多い演算部分を予め所定の画像についてのみ演算してテーブルデータとして保存しておくことで、実際の処理の際にはテーブルデータを用いて計算量を削減することができる。
（構成（１５）〜（１７）に対応する発明の実施の形態）
この発明には、上記した第６実施形態が対応する。構成中の第二領域抽出手段は図１１の相関領域抽出装置４１に対応する。この発明では、ＡＲモデルがよく成り立つ推定位置近傍の領域を推定して相関領域を定めることで、効率的に高精度な画像情報の推定を行う。図１１の構造情報抽出装置３１は領域抽出画像の微分画像を作成し、相関領域抽出装置４１により微分値がほぼ一定の領域を相関値も一定の領域と推定して相関領域とする。相関領域の自己相関関数を空間的相関情報算出装置１４により求め、以下、上記した第１実施形態と同様の処理により画像信号の推定記憶を行なう。
（構成（１５）〜（１７）の効果）
相関領域抽出装置４１により、相関が一定とみなせる領域で推定係数を算出することにより、より精度の高い推定を行なうことができる。また、計算量の多い推定係数算出の演算過程において適切な相関領域を設定することで計算量の削減が可能となる。
（構成（１８）に対応する発明の実施の形態）
この発明には、上記した第７実施形態が対応する。構成中の推定位置再抽出手段は推定位置選択装置１００に、空間的相関情報補間手段は空間的相関情報補間装置４３にそれぞれ対応する。
【００８９】
この発明では、上記した第１実施形態のように一画素ごとに推定位置の相関係数を実際に相関をとることにより求めるのではなく、適当なサンプル位置についてのみ相関計算から相関係数を求め、サンプル位置間における相関係数はサンプル位置における相関係数からの補間演算により求める。これは、相関関数の形状は大部分空間的に連続的に変化しているという推定に基づく。補間によって相関係数を求めることにより、計算量を削減できる。推定位置選択装置１００は、予め定められた画像情報の複数の所定の位置のうち最も近くに位置する推定位置を選択し、選択位置として保存する。領域抽出手段は全ての選択位置について空間的相関情報算出装置１４により相関関数の関数形の係数を算出し、相関係数記憶装置４２に記憶する。空間的相関情報補間装置４３は、相関係数記憶装置４２に記憶された推定位置における相関関数の関数形の係数から画像内全ての推定位置における相関関数の関数形の係数を補間により求める。以下第１の実施形態と同様の処理により画像信号を求め、補正画像記憶装置１８に記憶する。
（構成（１８）の効果）
推定すべき画像信号の複数の位置を所定間隔以上で疎に再抽出し、この再抽出された位置のみに対応する局所領域の空間的相関情報を算出し、全ての位置に関する空間的相関情報を補間している。すなわち、相関関数の関数形の係数を推定位置選択装置１００により選択された選択位置についてのみ求め、他の推定位置における相関関数の関数形の係数を選択位置における相関関数の関数形の係数からの補間演算により求めることにより、計算量を削減することができる。
（構成（１９）〜（２２）に対応する発明の実施の形態）
この発明には、上記した第８実施形態が対応する。構成中の補正位置決定手段は図１３の制御装置５２に、領域抽出手段は図１３の制御装置５２と部分色間相関計算部５１に、色相関情報算出手段は図１３の色間相関計算装置５０にそれぞれ対応する。補正係数算出手段は図１３の補正係数算出装置６４に対応する。画像信号補正手段は図１３の輝度成分復元装置６１に対応する。先見的色相関情報保持手段は図１３の相関関数パラメータ記憶部５２に対応する。
【００９０】
この発明では、第１実施形態と同様に、ＡＲモデルにより推定位置における輝度信号を周囲の画像情報の線形和として推定する。ただし、輝度信号はＲ，Ｇ，Ｂ信号値の既知の線形和として表わされるものとする。輝度信号を推定するための推定係数は、ＲＧＢ各色の自己相関情報のみならず色間の相互相関情報が必要となるが、ここでは近似的に求めた色間相互相関を用いる。
【００９１】
図１３において、図２に示すようなフィルタを有する単板式デジタルカメラ４４により撮像された画像情報はカラー画像入力端子４６を通して画像処理装置６５内の画像メモリ４７に伝送、記憶される。また、コンピュータ４５内に予め記憶された単板式デジタルカメラ４４のフィルタ配置に関する情報及び輝度信号をＲ，Ｇ，Ｂ色信号の線形和で表わすための輝度信号演算係数をそれぞれ色成分マスクバッファ５５及び輝度ウエイトバッファ５６に伝送、記憶する。図１３の制御装置５２は色成分マスクバッファ５５内に記憶されたフィルタ配置情報を用いてブロックＩ／Ｏ部４８を制御して画像メモリ４７に記憶された画像情報から部分ブロック画像をブロックバッファ４９に記録する。
【００９２】
以上で、単板式デジタルカメラ４４及びコンピュータ４５からの情報伝送、記憶が完了し、以下一画素ごとに輝度信号が推定され、出力画像メモリ６２に記憶される。色間相関計算装置５０は、画像メモリ４７内の画像情報、ブロックバッファ４９内に記憶されたブロック画像、マスク相関計算装置５７で計算されたフィルタ配置の相関情報を用いて、輝度信号推定係数を算出するために必要となる色間相互相関を算出する。補正係数算出装置６４は、色間相関計算装置５０において得られた色間相互相関、輝度ウエイトバッファ５６に記憶された輝度信号推定係数、マスク相関計算装置５７により得られたフィルタ配置の相関情報を用いて補正係数を算出する。輝度成分復元装置６１は補正係数算出装置６４により算出された補正係数を用いて輝度信号を算出し、出力画像メモリ６２に記憶する。全ての画素について輝度信号の推定から出力画像メモリ６２への記憶を行い、処理を完了する。出力画像メモリ６２に記憶された輝度信号画像は、制御装置５２の制御のもとで、輝度画像出力端子６３を通して外部の表示装置、出力装置等に伝送される。
（構成（１９）〜（２２）の効果）
複数の色信号からなる画像信号に対し、各色信号間の空間的相関を利用することで近傍の全ての画素から画像情報を推定することが可能となり、各色信号を独立に回復する場合に比較して高い精度で画像情報の推定ができる。
【００９３】
特定の対象を被写体とする場合には、各色信号間の空間的相関に関する先見的な知識を予め求めておき有効に生かすことができる。先見的情報は画像信号の種類に応じて切り替えて用いることで、より高精度な推定が可能となる。
【００９４】
所定領域内の色信号から得られる部分的、あるいは近似的な各色信号間の相互相関の精度は領域により異なる。そこで、画像情報から推定された相互相関に色間の相互相関の一般的関数形や画像の種類に応じて求めた色間の相互相関の関数形を当てはめることで、画像情報の推定精度の局所的な低下を抑制することができる。また、相互相関の推定精度が低いと思われる位置において相互相関を周囲の相互相関から補間により求めたり、あるいは相互相関が特定の性質をもつように規制することによって推定を高精度化することも可能である。
【００９５】
【発明の効果】
本発明によれば、推定すべき画像信号の局所領域に関する空間的相関情報を用いて画像構造に依存した適応的推定を行なうようにしたので、任意の構造の画像に対して最適な推定を行なうことができるようになる。
【図面の簡単な説明】
【図１】本発明の第１実施形態が適用される画像処理システムの構成を示す図である。
【図２】ＣＣＤ上の色フィルタの配置を示す図である。
【図３】図１に示す領域抽出装置の構成を示す図である。
【図４】図１に示す空間的相関情報算出装置の構成を示す図である。
【図５】本発明の第２実施形態が適用される画像処理システムの構成を示す図である。
【図６】エッジ検出のための領域抽出画像を示す図である。
【図７】図５に示す空間的相関情報算出装置の構成を示す図である。
【図８】本発明の第３実施形態が適用される画像処理システムの構成を示す図である。
【図９】本発明の第４実施形態が適用される画像処理システムの構成を示す図である。
【図１０】図９に示す構造情報抽出装置の構成を示す図である。
【図１１】本発明の第６実施形態が適用される画像処理システムの構成を示す図である。
【図１２】本発明の第７実施形態が適用される画像処理システムの構成を示す図である。
【図１３】本発明の第８実施形態が適用される画像処理システムの構成を示す図である。
【図１４】色成分マスクの一例を示す図である。
【図１５】赤、緑成分マスクの相互相関について説明するための図である。
【図１６】相互相関関数のデータ分布を示す図である。
【符号の説明】
１…レンズ、２…色フィルタ、３…ＣＣＤ、４…単板式デジタルカメラ、５…Ａ／Ｄ変換装置、６…ＲＯＭ、７…画像メモリ、８…画像信号、９…撮像装置情報記憶装置、１０…画像情報記憶装置、１１…推定位置決定装置、１２…領域抽出装置、１３…相関範囲記憶装置、１４…空間的相関情報算出装置、１５…推定係数算出装置、１６…推定係数範囲記憶装置、１７…画像信号推定装置、１８…補正画像記憶装置。[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image processing apparatus, and more particularly to an image processing apparatus that estimates an image signal based on spatial correlation information calculated for a local region.
[0002]
[Prior art]
In recent years, an imaging system using a solid-state imaging device such as a CCD instead of a silver salt film conventionally used is rapidly spreading. In such an imaging system, color image information is stored as color information of three colors or luminance and color difference information. For example, in an imaging system such as a digital camera, in principle, three types of filters having different spectral sensitivities corresponding to three colors are arranged on the front surface of the CCD, and each color signal information is obtained in principle. In order to acquire signal information, three types of sensors are required for each pixel.
[0003]
As described above, when realizing the imaging system, a three-plate type imaging apparatus including an optical system for dividing incident light from a subject and three CCDs having three types of filters having different spectral sensitivities is used. However, using three CCDs and a splitting optical system results in an expensive device.
[0004]
Therefore, a single-plate digital camera that appropriately arranges each color filter in each pixel of one CCD and obtains image information having different color information in each pixel is widely used as an inexpensive device. In this case, image information having color information of three colors at each pixel is estimated by post-processing such as spatial interpolation processing.
[0005]
As the most basic method of color information interpolation in a single-plate imaging device, linear interpolation from pixel information in the vicinity of an interpolation pixel is used. This interpolation processing changes gradually between pixels used for interpolation. Therefore, in a region including many high spatial frequency components such as an edge portion of an image, a false color is generated. Several methods for estimating image information for improving such problems have been proposed.
[0006]
The first is to cope with a structure that generally has a large estimation error by recognizing an edge of an image or a specific image structure and adaptively changing the processing method according to the image structure. For example, in Japanese Patent Laid-Open No. 6-303617, three types of interpolation methods are switched and used in accordance with the luminance change in the vicinity of the estimated pixel. In Japanese Patent Laid-Open No. 61-501423, the interpolation method is switched by matching between a pattern prepared in advance and an estimated pixel neighborhood image.
[0007]
An estimation method using the correlation between the color information of the three colors has also been proposed. For example, in Japanese Patent Application Laid-Open No. 5-56446, it is assumed that there is a high correlation between low spatial frequency components for each color, and acquired color information × (estimated color of low spatial frequency component / acquired color information of estimated pixel of low spatial frequency component) ) To obtain estimated color information. In Japanese Patent Laid-Open No. 4-502096, estimation is performed by linear interpolation from the vicinity of an estimated pixel in accordance with a change in first color information sampled at a high spatial frequency.
[0008]
[Problems to be solved by the invention]
None of the above-described conventional techniques proposes an estimation method using spatial correlation information regarding a local region of an image signal to be estimated as an adaptive estimation method depending on the image structure.
[0009]
Therefore, the image processing apparatus of the present invention performs optimal estimation for an image having an arbitrary structure by performing adaptive estimation depending on the image structure using spatial correlation information regarding the local region of the image signal to be estimated. An object of the present invention is to provide an image processing apparatus capable of performing the above.
[0010]
[Means for Solving the Problems]
  In order to achieve the above object, an image processing apparatus according to a first invention is an imaging apparatus that generates an image signal.Color filter placement informationBased onThe position where the image signal of the predetermined color type is not obtainedImage signal position to be estimatedAsEstimated position determining means for determining, and this estimated position determining meansDetermined byA local area of a predetermined size including the position of the image signalImage signalRegion extracting means for extracting the region and the region extracting meansCalculates an autocorrelation function for an image signal of a predetermined color type from the image signal extracted bySpatial correlation information calculating means;Based on the autoregressive model, using the autocorrelation function calculated by the spatial correlation information calculating unit, the predetermined color type at the position of the image signal to be estimated obtained from the estimated position determining unitImage signal estimating means for estimating the image signal.
[0011]
  An image processing apparatus according to the second invention isEstimated position determining means for determining a position where an image signal of a predetermined color type is not obtained as the position of the image signal to be estimated based on the color filter arrangement information of the imaging device that generates the image signal, and the estimated position determining means Region extracting means for extracting an image signal of a local area of a predetermined size including the position of the image signal determined by the above, and autocorrelation for an image signal of a predetermined color type from the image signal extracted by the area extracting means Spatial correlation information calculating means for calculating a function;In the local region obtained from the region extraction meansThe differential information is calculated as structural informationStructure information extraction means;Based on the autoregressive model, the image signal of the predetermined color type at the position of the image signal to be estimated obtained from the estimated position determining means using the autocorrelation function calculated by the spatial correlation information calculating means The image signal of the predetermined color type at the position of the image signal to be estimated obtained from the estimated position determining means by a method different from the first image signal estimating means and the first image signal estimating means Based on the structure information calculated by the second image signal estimating means and the structure information extracting means, the result of either the first image signal estimating means or the second image signal estimating means is Selecting means for selecting whether to use as an estimation result of the image signal of the predetermined color type at the position of the image signal to be estimated obtained from the estimated position determining means;It has.
[0012]
  An image processing apparatus according to a third aspect of the invention is an imaging apparatus that generates an image signal.Color filter placement informationBased onThe position where the image signal of the predetermined color type is not obtainedImage signal position to be estimatedAsEstimated position determining means for determining, and this estimated position determining meansDetermined byA local area of a predetermined size including the position of the image signalImage signalExtracting region extracting means and the region extracting meansCalculates an autocorrelation function for an image signal of a predetermined color type from the image signal extracted bySpatial correlation information calculating means and the region extracting meansThe inter-color cross-correlation function between the image signals of two predetermined color types is calculated from the image signal extracted byColor correlation information calculating means;Based on the autoregressive model, using the autocorrelation function calculated by the spatial correlation information calculation means and the cross-correlation function between colors calculated by the color correlation information calculation means,Obtained from the estimated position determination meansThe predetermined color type at the position of the image signal to be estimated;Image signal estimating means for estimating the image signal.
[0013]
  In other words, the image processing apparatus according to the first invention is an imaging apparatus that generates an image signal.Color filter placement informationBased on the estimated position determination meansThe position where the image signal of the predetermined color type is not obtainedImage signal position to be estimatedAsDecideAn image signal of a local area having a predetermined size including the position of the image signal determined by the estimated position determining means is extracted by the area extracting means. From the image signal extracted by the region extracting means, an autocorrelation function relating to an image signal of a predetermined color type is calculated by the spatial correlation information calculating means, and calculated by the spatial correlation information calculating means based on the autoregressive model. Using the calculated autocorrelation function, the predetermined color type at the position of the image signal to be estimated obtained from the estimated position determining means.The image signal is estimated by image signal estimation means.
[0014]
  An image processing apparatus according to the second invention isBased on the color filter arrangement information of the imaging device that generates the image signal, a position where an image signal of a predetermined color type is not obtained is determined by the estimated position determination means as the position of the image signal to be estimated. An image signal of a local area having a predetermined size including the position of the image signal determined by the estimated position determining means is extracted by the area extracting means, and a predetermined color type is extracted from the image signal extracted by the area extracting means. An autocorrelation function relating to the image signal is calculated by the spatial correlation information calculating means. With respect to the local region obtained from the region extraction means, the differential information is calculated as structure information by the structure information extraction means, and based on the autoregressive model, the autocorrelation function calculated by the spatial correlation information calculation means is used. The image signal of the predetermined color type at the position of the image signal to be estimated obtained from the estimated position determining means is estimated by the first image signal estimating means. The second image signal estimating means estimates the image signal of the predetermined color type at the position of the image signal to be estimated obtained from the estimated position determining means by a method different from the first image signal estimating means. Based on the structure information calculated by the structure information extraction means, the result of either the first image signal estimation means or the second image signal estimation means is obtained from the estimated position determination means. The selection means selects whether to use the image signal of the predetermined color type at the position of the image signal to be used.
[0015]
  An image processing apparatus according to a third invention isBased on the color filter arrangement information of the imaging device that generates the image signal, the position where the image signal of the predetermined color type is not obtained is determined by the estimated position determining means as the position of the image signal to be estimated. An image signal of a local area having a predetermined size including the position of the image signal determined by the means is extracted by the area extracting means. From the image signal extracted by the region extracting means, an autocorrelation function relating to an image signal of a predetermined color type is calculated by the spatial correlation information calculating means. From the image signal extracted by the region extracting means, a color cross-correlation function between the image signals of two predetermined color types is calculated by the color correlation information calculating means, and the spatial correlation information is based on the autoregressive model. The predetermined color at the position of the image signal to be estimated obtained from the estimated position determining means using the autocorrelation function calculated by the calculating means and the inter-color cross-correlation function calculated by the color correlation information calculating means. The type of image signal is estimated by the image signal estimation means.
[0016]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. FIG. 1 is a diagram showing an image processing system to which the first embodiment of the present invention is applied. The image processing system includes a single-plate digital camera 4 and an image processing device 19. The single-plate digital camera 4 includes a lens 1, a single-plate CCD 3, three types of color filters 2 (R, G, B) having different spectral sensitivities arranged in a mosaic pattern with respect to the CCD 3, and an A / D The conversion device 5 is composed of an image memory 7 and a ROM 6. Here, as schematically shown in FIG. 2, each of the R, G, and B color filters 2 includes one R and B filter and two G filters in a 2 × 2 pixel rectangular area. The unit structure has a structure in which the unit structure is periodically repeated.
[0017]
The image processing device 19 includes an imaging device information storage device 9, an image information storage device 10, an estimated position determination device 11, a region extraction device 12, a correlation range storage device 13, and a spatial correlation information calculation device 14. And an estimation coefficient calculation device 15, an estimation coefficient range storage device 16, an image signal estimation device 17, and a corrected image storage device 18.
[0018]
In the configuration described above, image information obtained by imaging the subject with the single-plate digital camera 4 is output from the CCD 3, converted into digital image information by the A / D conversion device 5, and stored in the image memory 7. The The image information stored in the image memory 7 is transmitted and stored as an image signal 8 to the image information storage device 10 in the image processing device 19. Here, the image information stored in the image memory 7 is stored in the image information storage device 10 as image information in which each pixel has only color information corresponding to the filter arrangement of FIG.
[0019]
At this time, information relating to the arrangement of the color filter 2 stored in the ROM 6 inside the single-plate digital camera 4 is also transmitted and stored in the imaging device information storage device 9 in the image processing device 19.
[0020]
The image processing device 19 obtains image information having only color information corresponding to any one of the color filters 2 related to the pixels R, G, and B captured by the single-plate digital camera 4 as information in the image and the imaging device. By estimating only from the information regarding the image information, the image information having R, G, B color information for all pixels is corrected. The estimation of the color information is sequentially performed on the image information stored in the image information storage device 10 in the order of G, B, and R colors.
[0021]
That is, the estimated position determination device 11 obtains the position of the estimated pixel corresponding to the estimated color information of the image read from the ROM 6 and stored in the imaging device information storage device 9. The estimated position is obtained from the information regarding the filter arrangement stored in the imaging device information storage device 9 as the pixel position where the estimated color information is missing. For example, if the estimated color information is G, the estimated position is the position of the pixel where the R or B color filter is arranged. All estimated positions obtained in the image for each estimated color information are stored in the estimated position determination device 11. The region extraction device 12 obtains a region for calculating an autocorrelation function described later and stores it as a region extraction image in a region extraction storage device described later.
[0022]
The region extraction by the region extraction device 12 is performed as follows. That is, in FIG. 3, the region extraction calculation device 25 is based on the estimated pixel position information 24 determined by the estimated position determination device 11 and the correlation range shape information 23 stored in advance in the correlation range storage device 13. A predetermined region extraction image is extracted from the image information 20 stored in the image information storage device 10 and stored in the region extraction storage device 26.
[0023]
The region extracted image at this time becomes image information corresponding to the correlation range shape of the image information stored in the image information storage device 10 centered on the estimated position. In addition, for pixels that do not have estimated color information in the region extraction image of the image information stored in the image information storage device 10, the pixel information of the neighboring region that has estimated color information in advance is used before extracting the region. Estimated color information is obtained by interpolation processing such as simple linear interpolation, and all the pixels in the region extracted image are assumed to have estimated color information. That is, in FIG. 3, in the image information 20 stored in the image information storage device 10, the pixel of the portion lacking the G component is replaced with an interpolated pixel (G^* ) In the interpolated image 21 obtained by interpolation in step 22, assuming that the coordinates of the estimated pixel position are (i, j) and the correlation range shape is a 3 × 3 rectangular region, the region extraction image is an image of the region shown in FIG. It is stored in the area extraction storage device 26.
[0024]
The spatial correlation information calculation device 14 shown in FIG. 1 calculates an autocorrelation function of the region extraction image obtained by the region extraction device 12, and outputs the result by approximating the least squares to a function form stored in advance. That is, in FIG. 4, the autocorrelation function of the region extracted image obtained by the region extracting device 12 is calculated in the autocorrelation function calculating device 28 and input to the least square coefficient calculating device 27. The function storage device 27 stores a function form that can approximate a statistical image autocorrelation function such as an exponential function expressed by the following equation.
[0025]
[Expression 1]

[0026]
Here, i and j are spatial coordinate values of the image, and α and β are constants. The least square coefficient computing device 27 is a coefficient α in the equation (1) so that the square error between the autocorrelation function calculated by the autocorrelation function computing device 28 and the function stored in the function-type storage device 27 is minimized. , Β are determined. The correlation value calculating device 30 obtains the value of the correlation function from the equation (1) using the coefficients α and β calculated by the least square coefficient calculating device 27 and sequentially outputs them.
[0027]
The estimation coefficient calculation device 15 in FIG. 1 uses the coefficient C of the image information estimation formula (2) based on the autoregressive model using the correlation coefficient value calculated by the spatial correlation information calculation device 14._ijIs calculated. This autoregressive model is a model in which estimated image information is represented by a linear sum of neighboring image information, and an estimated value g (m, n) of correct image information f (m, n) in spatial coordinates (m, n). )
[0028]
[Expression 2]

It is expressed as Here, D is a spatial coordinate range (neighboring region) of image information f (m, n) used for estimating g (m, n), c_ijIndicates the estimation coefficient of the image information f (m−i, n−j) whose coordinate values are located at i and j in the mn direction relative to the spatial coordinates of the estimated image information. At this time, the square error Err between the estimated value g (m, n) and the correct image f (m, n) is
[0029]
[Equation 3]

It is expressed as Here, A corresponds to the range of the region extraction image. Estimation coefficient c that minimizes the square error Err_ijErr c_ijApproximately by partial differentiation with
[0030]
[Expression 4]

[0031]
Here, since R (ip, jp) and R (p, q) are known values calculated by the spatial correlation information calculation device 14, equation (4) can be expressed as an estimation coefficient c._ijIt becomes a simultaneous equation with the unknown as. The estimation coefficient calculation device 15 uses the information of the neighborhood region D stored in the estimation coefficient range storage device 16 to estimate the estimation coefficient c from the relationship of equation (4)._ijIs calculated.
[0032]
The image signal estimation device 17 in FIG. 1 has an estimation coefficient c calculated by the estimation coefficient calculation device 15._ijIs used to calculate the estimated value g (m, n) of the image information f (m, n) of the autoregressive model expressed by the equation (2).
[0033]
The corrected image storage device 18 in FIG. 1 sequentially stores estimated image information at each estimated position of estimated color information sequentially calculated by the image signal estimating device 17.
Then, the processing from the determination of the region extraction image by the region extraction device 12 to the storage of the estimated image information in the corrected image storage device 18 is performed at all the estimated positions for the color information of G, B, R, and all the pixels are converted to the RGB color. The corrected image storage device 18 stores the image information as information. Note that the configuration of the first embodiment described above can be variously modified and changed. For example, the filters of a digital camera may be arranged in a stripe pattern, or a CMY complementary color system may be used in addition to a primary color system spectral characteristic. Further, the present invention can be applied to a filter composed of an arbitrary number of filters instead of three, such as RGB.
[0034]
FIG. 5 is a diagram showing a configuration of an image processing system to which the second embodiment of the present invention is applied. In FIG. 5, the region extraction image extracted by the region extraction device 12 in the same manner as in the first embodiment is read from the region extraction storage device 26 in the region extraction device 12 to the structure information extraction device 31 in FIG. The structure information extraction device 31 calculates the edge strength I of the region extraction image. The edge strength I is defined by the following equation when the region extraction image is a rectangular image of 3 × 3 pixels.
[0035]
[Equation 5]

[0036]
Here, a to i represent pixel values at the pixel positions of the region extraction image shown in FIG. 6, and X and Y correspond to differential values in the x and y directions, respectively. The edge intensity I is normalized to 0 to 1 according to the density range of the image used. The edge strength I can be calculated by appropriately redefining the edge strength I for a region extracted image having an arbitrary shape.
[0037]
The spatial correlation information calculation device 14 shown in FIG. 5 has a configuration as shown in FIG. Similar to the first embodiment, the least squares coefficient computing device 29 calculates coefficients α and β of the function form of the correlation function. The least square coefficient correction device 32 uses the coefficients α and β by using the edge intensity I of the extraction region calculated by the structure information extraction device 31 and the correction coefficients P and Q stored in advance in the correction coefficient storage device 33. It is corrected to α ′ and β ′ by the following equation.
[0038]
α ′ = α (PI + 1)
β ′ = β (QI + 1) (7)
By correcting the equation (7), α and β become I = 1 at the edge portion, and are increased by P and Q%, respectively. A correlation value is calculated by the correlation value calculation device 30 using the corrected coefficients α ′ and β ′. Thereafter, estimated image information is obtained by the same processing as in the first embodiment, and is stored in the corrected image storage device 18.
[0039]
The configuration of the second embodiment described above can be variously modified and changed. For example, it is possible to calculate not only the edge intensity but also the edge direction using the values of X and Y, and correct the correlation function. It is also possible to select a correlation function calculation method according to the edge strength, orientation, and the like.
[0040]
FIG. 8 is a diagram showing the configuration of an image processing system to which the third embodiment of the present invention is applied. In FIG. 8, the structure information extraction device 31 calculates the edge strength I of the region extraction image extracted by the region extraction device 12 by the same method as in the second embodiment. The estimation method switching device 33 selects an image information estimation method at the estimated position from the value of the edge strength I calculated by the structure information calculation device 31. That is,
I> T
In this case, the first estimation method is selected as the estimation method. Also,
I ≦ T
In the case of (2), the second estimation method is selected. T is an experimentally predetermined constant.
[0041]
The first estimation method is a method for estimating image information using autocorrelation function information of a region extraction image based on the same autoregressive model as that of the first embodiment described above. In the second estimation method, the simple linear interpolation device 35 estimates the pixel information at the interpolation position as a linear sum of the pixel information in the vicinity of the predetermined interpolation position.
The estimated pixel information g (m, n) of the estimated position is calculated by the following equation as an average value of the image information f (m, n) in the 3 × 3 region centered on the estimated position.
[0042]
[Formula 6]

[0043]
Here, E represents a 3 × 3 rectangular area other than (m, n) centered on (m, n). The estimated image information calculated by the simple linear interpolation device 35 using the equation (8) is stored in the corrected image storage device 18.
[0044]
The configuration of the above-described third embodiment can be variously modified and changed. For example, the second image signal estimation method is not limited to a method based on linear interpolation, and an arbitrary method can be used depending on the purpose. The number of estimation methods that can be switched is not limited to two, and a plurality of switching methods can be selected by preparing a plurality of estimation methods. Further, the switching method is not limited to the structure analysis, and can be manually set in the processing apparatus.
[0045]
FIG. 9 is a diagram showing the configuration of an image processing system to which the fourth embodiment of the present invention is applied. In FIG. 9, the region extraction image extracted by the region extraction device 12 by the same method as in the first embodiment is analyzed by the structure information extraction device 31.
[0046]
FIG. 10 is a diagram illustrating a configuration of the structure information extraction device 31. In FIG. 10, dozens of types of pattern images having the same size as the region extraction image are stored in the pattern storage device 38 in advance. The pattern matching calculation device 39 uses this pattern image to perform pattern matching with the region extraction image from the region extraction storage device 26 in the region extraction device 12. Then, the similarity between the correlation area image and the pattern image calculated by the pattern matching calculation device 39 is calculated as an evaluation value by the similarity evaluation calculation device 40.
[0047]
The memory card 36 in FIG. 9 is given estimation coefficients as table data corresponding to the type of pattern image. The estimation coefficient stored in the memory card 36 is a value calculated from the autocorrelation function calculated by the same process as in the first embodiment described above for the pattern image stored in advance in the structure information extraction device 31. .
[0048]
The estimation coefficient selection device 37 reads estimation coefficients corresponding to several kinds of pattern images similar to the region extraction image analyzed by the structure information extraction device 31 from the table data in the memory card 36, and calculates the read estimation coefficients by the similarity evaluation calculation. The evaluation value calculated by the device 40 is used for calculation. The image signal estimation device 17 obtains estimated image information by the same method as in the first embodiment using the estimation coefficient calculated by the estimation coefficient selection device 37 and stores it in the corrected image storage device 18.
[0049]
The configuration of the fourth embodiment described above can be variously modified and changed. For example, the table data in the memory card 36 may give an estimation coefficient corresponding to a functional parameter of the correlation function, instead of giving an estimation coefficient corresponding to the pattern of the region extraction image. In addition, the memory card 36 can be replaced according to the type of the image to be captured, or a plurality of different table data can be stored and selected according to the type of the image to be captured.
[0050]
The fifth embodiment of the present invention will be described below. In the fifth embodiment, the estimated position determination device 11 in the first embodiment described above determines a 5 × 5 rectangular estimation area instead of only one pixel. The estimation area is determined for the entire image in the RGB color information so that the pixels included in the estimation area do not overlap.
[0051]
The region extraction device 12 extracts a 10 × 10 rectangular region centered on the estimated region from the image information as a region extraction image. The spatial correlation information calculation device 14 and the estimation coefficient calculation device 15 perform the same processing as in the first embodiment, and the image signal estimation device 17 performs only the pixel lacking the estimated color information in the estimation region by the internal control device. Image information is estimated using the estimated coefficient calculated by the estimated coefficient calculating device 15 and stored in the corrected image storage device 18.
[0052]
The fifth embodiment described above can be variously modified and changed. For example, the estimation area is not a predetermined area, but an area where the estimation coefficient can be regarded as constant can be estimated by the structure information extraction device 31 similar to that of the second embodiment, and can be set as the estimation area. In this case, the region extraction image extracted by the region extraction device 12 is analyzed by the structure information extraction device 31, and the estimated region is determined by the estimated position determination device 11. The subsequent processing is the same as in the fifth embodiment.
[0053]
FIG. 11 is a diagram showing the configuration of an image processing system to which the sixth embodiment of the present invention is applied. In FIG. 11, the structure information extraction device 31 creates a differential image of the region extraction image extracted by the region extraction device 12 by the same method as in the first embodiment. The correlation region extraction device 41 estimates a region where the value of the autocorrelation function can be regarded as constant, and newly extracts a correlation region image. The region extraction image extracted by the region extraction device 12 is a predetermined rectangular region image. The correlation region extraction device 41 creates a differential image of the region extraction image extracted by the region extraction device 12, and estimates a region where the autocorrelation function can be regarded as constant from the similarity of the distribution of the differential image density. A rectangular area including a set of estimated positions is newly set as a correlation area image. Thereafter, image information is estimated using the correlation region image by the same method as in the first embodiment, and is stored in the corrected image storage device 18.
[0054]
Note that the configuration of the above-described sixth embodiment can be variously modified and changed. For example, the method of creating the correlation area image is not limited to the method based on the differential image, and it is also possible to actually calculate the correlation function of the extraction area image and determine the correlation area from the similarity.
[0055]
FIG. 12 is a diagram showing a configuration of an image processing system to which the seventh embodiment of the present invention is applied. In FIG. 12, the estimated position selecting device 100 is the center of the estimated area obtained by dividing the image into predetermined rectangular areas among the estimated positions determined by the estimated position determining apparatus 11 by the same method as in the first embodiment. Select only the position closest to the point. The spatial correlation information calculation device 14 calculates a function-type coefficient of the autocorrelation function at the selected estimated position selected by the estimated position selection device 100. The above processing is performed on the entire image, and the coefficient of the function form of the correlation function at all selected estimated positions is obtained and stored in the correlation coefficient storage device 42.
[0056]
The spatial correlation information interpolating apparatus 43 estimates the function form coefficients of the correlation function at all positions by interpolation using the stored function form coefficients of the correlation function. Interpolation calculation is performed based on the use of correlation coefficients at three selected positions surrounding the estimated position. Image information is estimated by the same method as in the first embodiment described above for each estimated position for which the function form coefficient of the correlation function has been obtained, and stored in the corrected image storage device 18.
[0057]
The configuration of the seventh embodiment described above can be variously modified and changed. For example, the estimated position selection device 100 does not select the selected estimated position from a predetermined rectangular area, but can consider that the correlation function is constant from the image analyzed by the structure information extraction device 31 similar to the above-described second embodiment. It is also possible to estimate and determine the area. It is also possible to interpolate the estimation coefficient at another position from the estimation coefficient at the selected estimated position, instead of interpolating the coefficient of the function form of the correlation function at another position from the coefficient of the correlation function at the selected estimated position. Is possible.
[0058]
FIG. 13 is a diagram showing the configuration of an image processing system to which the eighth embodiment of the present invention is applied. In FIG. 13, the image processing apparatus 65 of the present invention stores image information captured by a single-plate digital camera 44 having R, G, and B color filters arranged as shown in FIG. A luminance signal is generated from the filter arrangement information of the single-plate digital camera 44.
[0059]
Image information g (m, n) captured by the single-plate digital camera 44 is transmitted and stored in the image memory 47 in the image processing device 65 via the color image input terminal 46. At the same time that the image information g (m, n) is stored in the image memory 47, the filter arrangement information of the single-plate digital camera 44 stored in advance in the computer 45 is the color component mask H._c Information relating to (m, n) (c = r, g, b) is transmitted and stored in the color component mask buffer 55 via the mask information input terminal 54.
[0060]
The color component mask buffer 55 includes an R component mask buffer 55a, a G component mask buffer 55b, and a B component mask buffer 55c, and stores mask information of corresponding color components. Color component mask H_c (M, n) has a value of 1 when the color filter at the pixel position (m, n) of the single-plate digital camera 44 is the mask color component c, and 0 when the color filter is other color components.
[0061]
FIG. 14 shows a color component mask H corresponding to the filter arrangement of FIG._r (M, n), H_g An example of (m, n) is shown. In addition, the luminance signal calculation coefficient β stored in the computer 45 in advance simultaneously with the transmission of the mask information._c (C = r, g, b) is transmitted and stored in the luminance weight buffer 56 in the image processing device 65 via the mask information input terminal 54. The luminance Y (m, n) is the luminance signal calculation coefficient β_c And image information f having R, G, B all color information in each pixel_c (M, n) (c = r, g, b)
[0062]
[Expression 7]

It will be expressed as
[0063]
After the information transmission to the color component mask buffer 55 is completed, the block I / O unit 48 controls the partial block image g under the control of the control device 52 using the information of the color component mask buffer 55._c (M, n) (c = r, g, b) is written to the block buffer 49 from the image information g (mn) in the image memory 47. The block buffer 49 includes an R component block buffer 49a, a G component block buffer 49b, and a B component block buffer 49c. In each color component block buffer, g () other than the pixel having color information corresponding to the color component of each color component block buffer. color component block image g with m, n) set to 0_c (M, n) is stored. The image information g (m, n) in the image memory is stored in the color component mask H_c (M, n) and color component image f_c Using (m, n)
[0064]
[Equation 8]

It is expressed as
[0065]
With the above processing, information transmission from the single-panel digital camera 44 and the computer 45 to the image processing device 65 is completed, and thereafter, luminance signal estimation processing is sequentially performed for each pixel under the control of the control device 52. The luminance signal Y (m, n) is estimated by the following equation that extends the AR model.
[0066]
[Equation 9]

[0067]
Here, D is a spatial coordinate range (neighboring region) of image information g (m, n) in the image memory used for estimating Y (m, n), c_ijIndicates an estimation coefficient of image information g (m−i, n−j) whose coordinate values are located at i and j in the m and n directions, respectively, relative to the spatial coordinates of the estimated image information. At this time, the square error Err in the predetermined region A between the estimated value by the equation (11) and the correct luminance component Y (m, n) is
[0068]
[Expression 10]

Get.
[0069]
From equation (14), approximately c_klAnd the luminance signal Y (m, n) is estimated from equation (11).
First, the control device 52 determines the position of the estimated pixel for luminance estimation.
[0070]
## EQU11 ##

Is calculated and memorized. For example, the R component mask H of the single-plate digital camera 44_r (M, n) and G component mask H_g When (m, n) is a rectangular area of 5 × 5 pixels as shown in FIG.^h _rg(I, j) is as shown in FIG.
S_pqIs
S_pq= {(-2, -1,4), (-2,1,4), (-1, -2,4),
(-1, 0, 6), (-1, 2, 4), (0, -1, 6), (0, 1, 6), (1, -2, 4), (1, 0, 6 ), (1, 2, 4), (2, -1, 4), (2, 1, 4)}. In addition, the mask correlation calculation device 57 receives each color component mask H_c A number where the value of (m, n) is 1, that is, the number of pixels α having a corresponding color filter_c Remember. For example, the R component mask H shown in FIG._r (M, n) and G component mask H_g In the case of (m, n), α_r Is 9, α_g Becomes 12.
[0071]
The inter-color correlation calculation device 50 includes a partial inter-color correlation calculation unit 51, a correlation value interpolation unit 53, and a correlation function parameter storage unit 52.
The partial color correlation calculation unit 51 reads partial images of a predetermined area A of each color component from the respective color component block buffers 49a, 49b, 49c under the control of the control device 52, and a combination of R, G, B that does not overlap.
[0072]
[Expression 12]

, The cross-correlation R between the partial images defined by_pq′ (I, j)
Calculate
[0073]
[Formula 13]

[0074]
Where g_p (M, n), g_q Since (m−i, n−j) lacks information in pixels that do not correspond to the color filter, the correlation value R defined by the following equation is generally used._pq(I, j),
[0075]
[Expression 14]

Does not match. Where f_p (M, n), f_q (Mi, nj) represents image information having color information p, q in all pixels, respectively. Therefore, the partial color correlation calculation unit 51 sets the set S calculated by the mask correlation calculation device 57._pqBy using R_pq′ (I, j) is corrected and R_pqAn estimated value of (i, j) is calculated. Set S_pqThe coordinates (i, j) of the points included in^h _pqSince (i, j) is a coordinate value that is not zero, the term g that does not become zero is present on the right side of equation (17)._p (M ′, n ′) g_q (M′−i, n′−j) exists, and its value gives the correct correlation value. The value f of the term in equation (18)_p (M ′, n ′) f_q (M'-i, n'-j). However, such a term is a set S for the total number of pixels of the block._pqIt is limited to the term corresponding to (i, j).
[0076]
For example, in the example shown in FIG. 15, the maximum number of terms that can be obtained for the 25 terms in the equation (17) is six terms, and the other terms are zero. Therefore, the value R obtained by equation (17)_pq′ (I, j) and the correct correlation value R_pqCorrelation value R between masks whose ratio to (i, j) is approximately 1 in all regions₀ ^h _pq(I, j) and mask correlation value R^h _pqAssuming a ratio with (i, j), R_pqEstimated value R ″ of (i, j)_pq(I, j) is obtained from the following equation.
R ″_pq(i, j) = R '_pq(i, j) R₀ ^h _pq(i, j) / R^h _pq(i, j) (19)
Also, using the image information g (m, n) in the image memory 47,
[0077]
[Expression 15]

Is calculated and stored internally.
[0078]
The correlation value interpolation unit 53 calculates and corrects the correlation function R ″ calculated and corrected by the inter-color correlation calculation unit 51._pqInterpolate (i, j) and output as an approximate correlation function. R ″_pqSince (i, j) is an autocorrelation between images lacking pixel information, it has data only at discrete positions as shown in FIG. The correlation value interpolating unit 53 uses the approximate function form stored in the correlation function parameter storage unit 52 and its parameters, and the correlation function R ″._pq(I, j) is approximated to a known function form by least squares approximation. Approximated R ″_pq(I, j) is stored in the correlation value interpolation unit 53.
[0079]
The correction coefficient calculation device 64 includes a correlation value addition unit 58, a correlation value buffer 59, and a correction coefficient calculation unit 60.
The correlation value adding unit 58 is calculated by the mask correlation calculation device 57._c , The luminance signal calculation coefficient β obtained from the luminance weight buffer 56_c , And the approximated R ″ obtained from the correlation value interpolation unit 53_pqThe following equation is calculated and stored using (i, j).
[0080]
[Expression 16]

[0081]
The correlation value buffer 59 stores the value of the equation (20) calculated by the partial color correlation calculation unit 51 and the value of the equation (21) calculated by the correlation value addition unit 58.
The correction coefficient calculation unit 60 satisfies the expression (14) using the values of the expressions (20) and (21) stored in the correlation value buffer 59._klIs calculated and stored. Here, on the left side of the equation (14), an approximation of the following equation:
[0082]
[Expression 17]

Was used.
[0083]
The luminance component restoration device 61 includes the image information f (m, n) stored in the image memory 47 and the correction coefficient c stored in the correction coefficient calculation unit 60._klIs used to calculate the luminance component Y (m, n) of the estimated pixel from the equation (11) and store it in the output image memory 62.
[0084]
The above processing is performed for all pixels. The corrected image can be read out to the outside via the luminance image output terminal 63.
The above-described embodiment can be variously modified and changed. For example, the estimated pixels are set for each pixel, but if it is determined that the approximation accuracy is good, the luminance components of all the pixels in the block can be obtained. Various correlation estimation methods can be used as a method for calculating the correlation between individual colors, and foresight information can also be used. In addition, although the luminance signal is estimated here, it is possible to restore arbitrary image information calculated not only by the luminance but also by a linear sum of color signals. In this example, correction of an image captured by the single-panel mosaic color filter CCD is taken up.
[0085]
[Expression 18]

Thus, for example, it can be applied to correction of an image including chromatic aberration. The cross-correlation between colors may be obtained by applying an appropriate function form based on information obtained from the observed image. It is also effective to use such information when the statistical properties of the object are known in advance.
[0086]
The above-described specific embodiments include inventions having the following configurations, and embodiments and effects corresponding to the respective inventions are as follows.
(1) Estimated position determining means for determining the position of the image signal to be estimated based on information relating to the imaging device that generates the image signal;
Area extracting means for extracting a local area of a predetermined size including the position of the image signal obtained from the estimated position determining means;
Spatial correlation information calculating means for calculating spatial correlation information regarding the local area obtained from the area extracting means;
Image signal estimation means for estimating an image signal corresponding to the position of the image signal obtained from the estimated position determination means based on the spatial correlation information calculated by the spatial correlation information calculation means;
An image processing apparatus comprising:
(2) The image processing apparatus according to the configuration (1), wherein the estimated position determining unit determines a position of an image signal to be estimated based on color filter arrangement information of the imaging device.
(3) The image processing apparatus according to (1) or (2), wherein the region extraction unit extracts a block region of n × m pixels in which n and m are integers of 2 or more.
(4) The spatial correlation information calculating unit calculates the spatial correlation information based on the autocorrelation function of the local region, according to any one of configurations (1) to (3), Image processing device.
(5) The spatial correlation information calculating means calculates the spatial correlation information based on approximating the autocorrelation function of the local region with an exponential function. The image processing apparatus according to any one of the above.
(6) The image signal estimation unit includes an estimation coefficient calculation unit that calculates an estimation coefficient related to the local region obtained from the region extraction unit based on the spatial correlation information calculated from the spatial correlation information calculation unit. And this estimated coefficient calculation means calculates the estimated coefficient of the said local area based on an autoregressive model, The image processing apparatus as described in any one of the structure (1)-(5) characterized by the above-mentioned.
(7) structure information extraction means for extracting structure information on the local area obtained from the area extraction means;
Based on the structure information extracted by the structure information extracting means, changing means for changing the method of calculating the spatial correlation information by the spatial correlation information calculating means,
The image processing apparatus according to configuration (1), further comprising:
(8) The image processing apparatus according to (7), wherein the structure information extraction unit extracts the structure information based on lightness information.
(9) The image processing apparatus according to (7), wherein the structure information extraction unit extracts the structure information based on differentiation processing and pattern matching.
(10) Estimated position determining means for determining the position of the image signal to be estimated based on information relating to the imaging device that generates the image signal;
Area extracting means for extracting a local area of a predetermined size including the position obtained from the estimated position determining means;
Structure information extraction means for extracting structure information about the local area obtained from the area extraction means;
First image signal estimation means for calculating an estimation coefficient from spatial correlation information about the local area obtained from the area extraction means and estimating an image signal corresponding to the position of the image signal obtained from the estimated position determination means When,
In the local area obtained from the area extracting means, the image signal is obtained using another image signal corresponding to the position of the image signal obtained from the estimated position determining means or an image signal located around the position. Second image signal estimation means for estimating an image signal corresponding to the position of
Switching means for switching between the first image signal estimating means and the second image signal estimating means based on the structure information extracted by the structure information extracting means;
An image processing apparatus comprising:
(11) The second image signal estimation means includes linear interpolation from image signals at positions around the position of the image signal obtained from the estimated position determination means in the local area obtained from the area extraction means, The image processing apparatus according to the configuration (10), wherein estimation is performed by interpolation using a quadratic or higher-order function.
(12) The second image signal estimation unit is configured to correlate with another image signal corresponding to the position of the image signal obtained from the estimated position determination unit in the local region obtained from the region extraction unit. The image processing apparatus according to (10), wherein the image signal is estimated.
(13) Estimated position determining means for determining the position of the image signal to be estimated based on information relating to the imaging device that generates the image signal;
Area extracting means for extracting a local area of a predetermined size including the position of the image signal obtained from the estimated position determining means;
Structure information extracting means for extracting structure information on the local area obtained from the area extracting means;
An estimated coefficient storage means in which estimated coefficients corresponding to a plurality of predetermined structure information are stored;
Estimation coefficient selection means for selecting an estimation coefficient from the estimation coefficient storage means based on the structure information extracted from the structure information extraction means;
An image processing apparatus comprising:
(14) The image processing apparatus according to (13), wherein the estimation coefficient storage unit is composed of a replaceable recording medium such as a magnetic disk or a memory card.
(15) A configuration further comprising control means for sharing and using the spatial correlation information calculated from the spatial correlation information calculation means at a plurality of positions obtained from the estimated position determination means ( The image processing apparatus according to any one of 1) to (6).
(16) Structure information extraction means for extracting structure information about the local area obtained from the area extraction means;
Based on the structure information extracted by the structure information extracting means, a second area extracting means for extracting a new local area from the local area obtained from the area extracting means;
The image processing apparatus according to any one of configurations (1) to (6), further comprising:
(17) The image processing apparatus according to (16), wherein the second region extraction unit extracts a region in which the autocorrelation function can be regarded as approximately constant.
(18) Estimated position determining means for determining the position of the image signal to be estimated based on information relating to the imaging device that generates the image signal;
Estimated position re-extracting means for re-extracting the positions of the plurality of image signals obtained from the estimated position determining means sparsely at a predetermined interval or more;
Area extracting means for extracting a local area of a predetermined size including the position of the image signal obtained from the estimated position re-extracting means;
Spatial correlation information calculating means for calculating spatial correlation information of the local area corresponding to the position of the image signal re-extracted by the estimated position re-extracting means with respect to the local area obtained from the area extracting means; ,
Spatial correlation information interpolation means for interpolating spatial correlation information related to the positions of all the image signals obtained from the estimated position determination means based on the spatial correlation information calculated from the spatial correlation information calculation means; ,
Image signal estimation means for estimating an image signal corresponding to the position of the image signal obtained from the estimated position determination means based on the spatial correlation information interpolated by the spatial correlation information interpolation means;
An image processing apparatus comprising:
(19) Estimated position determining means for determining the position of an image signal to be estimated based on information relating to an imaging device that generates an image signal composed of a plurality of color signals;
Area extracting means for extracting a local area of a predetermined size including the position of the image signal obtained from the estimated position determining means from an image signal including a plurality of color signals;
Spatial correlation information calculating means for calculating spatial correlation information regarding the local area obtained from the area extracting means;
Color correlation information calculating means for calculating color correlation information regarding the local area obtained from the area extracting means;
Based on the spatial correlation information calculated by the spatial correlation information calculation means and the color correlation information calculated by the color correlation information calculation means, an image corresponding to the position of the image signal obtained from the estimated position determination means Image signal estimating means for estimating a signal;
An image processing apparatus comprising:
(20) The color correlation information calculation means includes a prior color correlation information holding means for holding a priori information with respect to a spatial correlation of a plurality of color signals related to the local area of the predetermined size ( The image processing apparatus according to 19).
(21) The color correlation information calculating means includes function approximating means for approximating spatial correlation information of a plurality of color signals related to the local area of the predetermined size to a predetermined function indicated by a parameter ( The image processing apparatus according to 19).
(22) The image processing apparatus according to (19), wherein the area extracting unit extracts n × m block areas in which n and m are integers of 2 or more.
(Embodiments of the invention corresponding to configurations (1) to (6))
These inventions correspond to the first embodiment described above. That is, the imaging apparatus in the configuration is the single-plate digital camera 4 in FIG. 1, the estimated position determining means is in the estimated position determining apparatus 11 in FIG. 1, and the area extracting means is in the area extracting apparatus 12 in FIG. The calculation means corresponds to the spatial correlation information calculation device 14 in FIG. 1, the estimation coefficient calculation means corresponds to the estimation coefficient calculation device 15 in FIG. 1, and the image signal estimation means corresponds to the image signal estimation device 17 in FIG. Further, the color filter arrangement information of the imaging device corresponds to the color filter arrangement information on the CCD of FIG. 2 stored in the imaging device information storage device 9, and the block area of n × m pixels is changed to a 3 × 3 block area. Correspond.
[0087]
In FIG. 1, image information captured by the single-plate digital camera 4 and stored in the internal image memory 7 is transmitted and stored in the image information storage device 10 in the image processing device 19. At the same time, the arrangement information of the color filter 2 arranged on the CCD 3 of the single-plate digital camera 4 stored in the ROM 6 inside the single-plate digital camera 4 is transmitted and stored in the imaging device information storage device 9. From the filter arrangement information stored in the imaging device information storage device 9, all pixel positions of the image information in the image information storage device 10 corresponding to the color information to be estimated by the estimation position determination device 11 are calculated and stored. The region extraction image is extracted from the image information storage device 10 in accordance with the region stored in the correlation range storage device 13 by the region extraction device 12 for the stored position of one pixel. The spatial correlation information calculation device 14 calculates correlation information related to the region extraction image, and the estimation coefficient calculation device 15 calculates an estimation coefficient of an estimation formula for estimating an image signal based on the model using the correlation information. The image signal estimation device 17 calculates an estimated value of the image signal using the estimation coefficient and stores it in the corrected image storage device 18. By performing the above processing for all estimated positions stored in the estimated position determining device 11, each pixel in the entire image is stored in the corrected image storage device 18 as an image having RGB color information.
(Effects of configurations (1) to (6))
Since adaptive estimation depending on the image structure is performed using the spatial correlation information regarding the local region of the image signal to be estimated, optimal estimation can be performed for an image having an arbitrary structure. For example, by using the autocorrelation information of the region extraction image near the estimated position and performing image information estimation using an autoregressive model, an estimation coefficient that minimizes the estimation error can be automatically calculated. As a result, image processing depending on an arbitrary image structure can be performed.
(Embodiments of the invention corresponding to configurations (7) to (9))
The second embodiment described above corresponds to this invention. The structure information extraction means in the configuration corresponds to the structure information extraction apparatus 31 in FIG. 5, and the change means corresponds to the least square coefficient correction apparatus 32 (FIG. 7) in the spatial correlation information calculation apparatus 14. In the present invention, by correcting the correlation coefficient calculated by the spatial correlation information calculation device 14 in accordance with the image structure in the vicinity of the estimated position obtained by the analysis of the structure information extraction device 31, a more accurate spatial Correlation information is calculated. The structure information extraction device 31 analyzes the edge strength of the region extraction image extracted by the region extraction device 12. The least square coefficient correction device 32 in the spatial correlation information calculation device 14 corrects the correlation coefficient calculated by the least square coefficient calculation device 29 (FIG. 7) according to the edge strength of the region extraction image. The correction increases the coefficient as the edge strength increases. Hereinafter, the image information is estimated and stored by the same processing as in the first embodiment.
(Effects of configurations (7) to (9))
The shape of the correlation function calculated by the spatial correlation information calculation device 14 cannot always be obtained accurately due to the interpolation processing of the region extraction image and the approximation of the function shape. Therefore, by correcting the shape of the correlation function in an edge region that is particularly difficult to estimate so that a good result can be obtained empirically, more accurate estimation can be performed.
(Embodiments of the invention corresponding to configurations (10) to (12))
The third embodiment described above corresponds to this invention. The first image signal estimation means in the configuration is a means for estimating image information using the autocorrelation function information of the region extracted image based on the autoregressive model, and the second image signal estimation means is the simple linear interpolator 35 described above. The switching means corresponds to the means for estimating the pixel information at the interpolation position as a linear sum of the pixel information in the vicinity of the interpolation position determined in advance by the estimation method switching device 33 in FIG.
[0088]
In the present invention, the simple linear interpolation in the flat region and the method based on the autoregressive model in the edge region are selectively used according to the image structure in the vicinity of the estimated position obtained by the analysis of the structure information extracting device 31. The structure information extraction device 31 analyzes the structure of the region extraction image extracted by the region extraction device 12 and calculates parameters used for selecting an estimation method in the estimation method switching device 33. The estimation method switching device 33 uses the first image signal estimation means when the edge strength of the region extraction image is larger than a predetermined value based on the result of the structural information analysis, and uses the second image signal estimation means when the edge strength is small. Estimate the signal.
(Effects of configurations (10) to (12))
By switching the image signal estimation method according to the structure of the region extraction image, it is possible to perform image signal estimation with a small amount of calculation while maintaining the estimation accuracy. For example, in a region where the image structure that is easy to estimate is flat, an estimation method based on an autoregressive model with a large amount of calculation is not used, but an estimation is performed by the simple linear interpolation device 35 with a relatively small amount of calculation. Further, by incorporating the information obtained by the structural information analysis device 31 into the processing correction of the first image signal estimation method or using it as the second image signal estimation method, it is possible to estimate with higher accuracy.
(Embodiments of the invention corresponding to configurations (13) and (14))
The fourth embodiment described above corresponds to this invention. The estimated coefficient storage means in the configuration corresponds to the memory card 36 in FIG. 9, and the estimated coefficient selection means corresponds to the estimated coefficient selection device 37. In this invention, in the first embodiment described above, calculation of an estimation coefficient having a large amount of calculation is obtained in advance for a plurality of predetermined patterns, and a table of image structures and estimation coefficients in the vicinity of the estimated position is created and stored. The calculation is speeded up by using this table information when estimating the image information. The structure information extraction device 31 in FIG. 9 performs a pattern matching operation between the region extraction image and a predetermined pattern image stored in the pattern storage device 38, and evaluates the similarity. Then, an estimation coefficient corresponding to a pattern having a high similarity is obtained from the table data stored in the memory card 36, and the estimation coefficient is calculated by appropriate weighting based on the evaluation value of the similarity. An image signal is estimated by the image signal estimation device 17 using the obtained estimation coefficient, and stored in the corrected image storage device 18.
(Effects of configurations (13) and (14))
By storing the estimation coefficients corresponding to a plurality of predetermined structure information, the estimation coefficient can be obtained without performing an operation for calculating the estimation coefficient from the region extraction image. In this way, by calculating only a predetermined image for a calculation part having a large calculation amount in advance and storing it as table data, the calculation amount can be reduced using the table data in actual processing.
(Embodiments of the invention corresponding to configurations (15) to (17))
The sixth embodiment described above corresponds to this invention. The second area extracting means in the configuration corresponds to the correlation area extracting device 41 of FIG. According to the present invention, highly accurate image information is efficiently estimated by estimating a region in the vicinity of an estimated position where the AR model is well established and determining a correlation region. The structure information extraction device 31 in FIG. 11 creates a differential image of the region extraction image, and the correlation region extraction device 41 estimates a region having a substantially constant differential value as a region having a constant correlation value as a correlation region. The autocorrelation function of the correlation region is obtained by the spatial correlation information calculation device 14, and the image signal is estimated and stored by the same processing as in the first embodiment.
(Effects of configurations (15) to (17))
By calculating the estimation coefficient in the region where the correlation can be regarded as constant by the correlation region extracting device 41, more accurate estimation can be performed. In addition, the calculation amount can be reduced by setting an appropriate correlation region in the calculation process of calculating the estimation coefficient with a large calculation amount.
(Embodiment of the invention corresponding to the configuration (18))
The seventh embodiment described above corresponds to this invention. The estimated position re-extraction means in the configuration corresponds to the estimated position selection apparatus 100, and the spatial correlation information interpolation means corresponds to the spatial correlation information interpolation apparatus 43.
[0089]
In the present invention, the correlation coefficient of the estimated position is not obtained by actually taking the correlation for each pixel as in the first embodiment, but the correlation coefficient is obtained from the correlation calculation only for an appropriate sample position. The correlation coefficient between the sample positions is obtained by interpolation calculation from the correlation coefficient at the sample position. This is based on the assumption that the shape of the correlation function changes continuously in most space. The amount of calculation can be reduced by obtaining the correlation coefficient by interpolation. The estimated position selection device 100 selects an estimated position that is closest to a plurality of predetermined positions of predetermined image information, and stores it as a selected position. The area extracting means calculates the function form coefficient of the correlation function by the spatial correlation information calculating device 14 for all selected positions, and stores it in the correlation coefficient storage device 42. The spatial correlation information interpolating device 43 obtains the function form coefficients of the correlation function at all estimated positions in the image from the correlation function function coefficients at the estimated positions stored in the correlation coefficient storage device 42 by interpolation. Thereafter, an image signal is obtained by the same processing as in the first embodiment, and is stored in the corrected image storage device 18.
(Effect of configuration (18))
A plurality of positions of the image signal to be estimated are sparsely re-extracted at a predetermined interval or more, the spatial correlation information of the local region corresponding only to the re-extracted position is calculated, and the spatial correlation information regarding all positions is calculated. Interpolated. That is, the coefficient of the function form of the correlation function is obtained only for the selected position selected by the estimated position selection device 100, and the function form coefficient of the correlation function at the other estimated position is obtained from the function form coefficient of the correlation function at the selected position. By calculating by interpolation, the amount of calculation can be reduced.
(Embodiments of the invention corresponding to configurations (19) to (22))
The eighth embodiment described above corresponds to this invention. The correction position determining means in the configuration is the control device 52 in FIG. 13, the region extraction means is in the control device 52 and the partial color correlation calculation unit 51 in FIG. 13, and the color correlation information calculation means is in the color correlation calculation device in FIG. 50 respectively. The correction coefficient calculation means corresponds to the correction coefficient calculation device 64 of FIG. The image signal correcting means corresponds to the luminance component restoring device 61 of FIG. The a priori color correlation information holding unit corresponds to the correlation function parameter storage unit 52 of FIG.
[0090]
In the present invention, as in the first embodiment, the luminance signal at the estimated position is estimated as a linear sum of surrounding image information by the AR model. However, the luminance signal is expressed as a known linear sum of R, G, and B signal values. The estimation coefficient for estimating the luminance signal requires not only autocorrelation information for each color of RGB but also cross-correlation information between colors. Here, the cross-correlation obtained approximately is used.
[0091]
In FIG. 13, image information captured by a single-plate digital camera 44 having a filter as shown in FIG. 2 is transmitted and stored in an image memory 47 in the image processing device 65 through a color image input terminal 46. Further, information relating to the filter arrangement of the single-plate digital camera 44 stored in advance in the computer 45 and luminance signal calculation coefficients for representing the luminance signal as a linear sum of R, G, and B color signals are respectively shown in the color component mask buffer 55 and It is transmitted to the luminance weight buffer 56 and stored. The control device 52 in FIG. 13 controls the block I / O unit 48 using the filter arrangement information stored in the color component mask buffer 55 and converts the partial block image from the image information stored in the image memory 47 to the block buffer 49. To record.
[0092]
The information transmission and storage from the single-panel digital camera 44 and the computer 45 are completed as described above, and the luminance signal is estimated for each pixel and stored in the output image memory 62. The inter-color correlation calculation device 50 uses the image information in the image memory 47, the block image stored in the block buffer 49, and the correlation information of the filter arrangement calculated by the mask correlation calculation device 57 to calculate the luminance signal estimation coefficient. A cross-correlation between colors necessary for calculation is calculated. The correction coefficient calculation device 64 uses the intercolor correlation obtained by the intercolor correlation calculation device 50, the luminance signal estimation coefficient stored in the luminance weight buffer 56, and the correlation information of the filter arrangement obtained by the mask correlation calculation device 57. To calculate the correction coefficient. The luminance component restoration device 61 calculates a luminance signal using the correction coefficient calculated by the correction coefficient calculation device 64 and stores it in the output image memory 62. The luminance signal is estimated and stored in the output image memory 62 for all the pixels, and the processing is completed. The luminance signal image stored in the output image memory 62 is transmitted to an external display device, output device or the like through the luminance image output terminal 63 under the control of the control device 52.
(Effects of configurations (19) to (22))
For image signals consisting of multiple color signals, it is possible to estimate image information from all nearby pixels by using the spatial correlation between each color signal, compared to when each color signal is recovered independently. Image information can be estimated with high accuracy.
[0093]
When a specific target is a subject, a priori knowledge about the spatial correlation between the color signals can be obtained in advance and used effectively. By using the a priori information by switching according to the type of the image signal, more accurate estimation can be performed.
[0094]
The accuracy of the cross-correlation between partial or approximate color signals obtained from the color signals in the predetermined region varies depending on the region. Therefore, by applying the general function form of cross-correlation between colors and the function form of cross-correlation between colors obtained according to the type of image to the cross-correlation estimated from image information, Reduction can be suppressed. In addition, the cross-correlation can be obtained by interpolation from the surrounding cross-correlation at a position where the cross-correlation estimation accuracy seems low, or the cross-correlation can be regulated so that it has a specific property. Is possible.
[0095]
【The invention's effect】
According to the present invention, the adaptive estimation depending on the image structure is performed using the spatial correlation information regarding the local region of the image signal to be estimated, so that the optimum estimation is performed for an image having an arbitrary structure. Will be able to.
[Brief description of the drawings]
FIG. 1 is a diagram showing a configuration of an image processing system to which a first embodiment of the present invention is applied.
FIG. 2 is a diagram illustrating an arrangement of color filters on a CCD.
FIG. 3 is a diagram illustrating a configuration of the region extraction device illustrated in FIG. 1;
4 is a diagram showing a configuration of a spatial correlation information calculation apparatus shown in FIG.
FIG. 5 is a diagram showing a configuration of an image processing system to which a second embodiment of the present invention is applied.
FIG. 6 is a diagram showing a region extraction image for edge detection.
7 is a diagram showing a configuration of the spatial correlation information calculation apparatus shown in FIG.
FIG. 8 is a diagram showing a configuration of an image processing system to which a third embodiment of the present invention is applied.
FIG. 9 is a diagram showing a configuration of an image processing system to which a fourth embodiment of the present invention is applied.
10 is a diagram showing a configuration of the structure information extracting device shown in FIG. 9;
FIG. 11 is a diagram showing a configuration of an image processing system to which a sixth embodiment of the present invention is applied.
FIG. 12 is a diagram illustrating a configuration of an image processing system to which a seventh embodiment of the present invention is applied.
FIG. 13 is a diagram showing a configuration of an image processing system to which an eighth embodiment of the present invention is applied.
FIG. 14 is a diagram illustrating an example of a color component mask.
FIG. 15 is a diagram for explaining cross-correlation between red and green component masks.
FIG. 16 is a diagram illustrating a data distribution of a cross-correlation function.
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 1 ... Lens, 2 ... Color filter, 3 ... CCD, 4 ... Single plate type digital camera, 5 ... A / D converter, 6 ... ROM, 7 ... Image memory, 8 ... Image signal, 9 ... Imaging device information storage device, DESCRIPTION OF SYMBOLS 10 ... Image information storage device, 11 ... Estimated position determination device, 12 ... Area extraction device, 13 ... Correlation range storage device, 14 ... Spatial correlation information calculation device, 15 ... Estimation coefficient calculation device, 16 ... Estimation coefficient range storage device , 17 ... Image signal estimation device, 18 ... Corrected image storage device.

Claims (3)

An estimated position determining means for determining a position where an image signal of a predetermined color type is not obtained as the position of the image signal to be estimated based on the color filter arrangement information of the imaging device that generates the image signal ;
A region extracting unit for extracting an image signal of a local region of a predetermined size including the position of the image signal determined by the estimated position determining unit;
Spatial correlation information calculating means for calculating an autocorrelation function relating to an image signal of a predetermined color type from the image signal extracted by the region extracting means;
Based on the autoregressive model, the image signal of the predetermined color type at the position of the image signal to be estimated obtained from the estimated position determining means using the autocorrelation function calculated by the spatial correlation information calculating means Image signal estimating means for estimating
An image processing apparatus comprising: An estimated position determining means for determining a position where an image signal of a predetermined color type is not obtained as the position of the image signal to be estimated based on the color filter arrangement information of the imaging device that generates the image signal;
A region extracting unit for extracting an image signal of a local region of a predetermined size including the position of the image signal determined by the estimated position determining unit;
Spatial correlation information calculating means for calculating an autocorrelation function relating to an image signal of a predetermined color type from the image signal extracted by the region extracting means;
A structure information extracting means for calculating a differential information as structural information regarding the local region obtained from the region extracting means,
Based on the autoregressive model, the image signal of the predetermined color type at the position of the image signal to be estimated obtained from the estimated position determining means using the autocorrelation function calculated by the spatial correlation information calculating means First image signal estimating means for estimating
Second image signal estimating means for estimating the image signal of the predetermined color type at the position of the image signal to be estimated obtained from the estimated position determining means by a method different from the first image signal estimating means;
Based on the structure information calculated by the structure information extraction means, the result of either the first image signal estimation means or the second image signal estimation means is estimated using the estimated position determination means. A selection means for selecting whether to use as an estimation result of the image signal of the predetermined color type at the position of the power image signal;
An image processing apparatus comprising: An estimated position determining means for determining a position where an image signal of a predetermined color type is not obtained as the position of the image signal to be estimated based on the color filter arrangement information of the imaging device that generates the image signal ;
A region extracting unit for extracting an image signal of a local region of a predetermined size including the position of the image signal determined by the estimated position determining unit;
Spatial correlation information calculating means for calculating an autocorrelation function relating to an image signal of a predetermined color type from the image signal extracted by the region extracting means;
Color correlation information calculating means for calculating a cross-correlation function between colors of image signals of two predetermined color types from the image signal extracted by the region extracting means;
Based on the autoregressive model, using the colors cross-correlation function calculated by the autocorrelation function and the color correlation information calculation unit calculated by the spatial correlation information calculation unit, the estimates obtained from the estimated position determining means Image signal estimating means for estimating the image signal of the predetermined color type at the position of the image signal to be;
An image processing apparatus comprising:

Images